# dbt Developer Hub > End user documentation, guides and technical reference for dbt ## API Reference ### About the Discovery API dbt platform | Starter, Enterprise, Enterprise+ Every time dbt runs a project, it generates and stores information about the project. The metadata includes details about your project’s models, sources, and other nodes along with their execution results. With the dbt Discovery API, you can query this comprehensive information to gain a better understanding of your DAG and the data it produces. By leveraging the metadata in dbt, you can create systems for data monitoring and alerting, lineage exploration, and automated reporting. This can help you improve data discovery, data quality, and pipeline operations within your organization. You can access the Discovery API through [ad hoc queries](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md), custom applications, a wide range of [partner ecosystem integrations](https://www.getdbt.com/product/integrations/) (like BI/analytics, catalog and governance, and quality and observability), and by using dbt features like [model timing](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) and [data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md). For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). [![A rich ecosystem for integration ](/img/docs/dbt-platform/discovery-api/discovery-api-figure.png?v=2 "A rich ecosystem for integration ")](#)A rich ecosystem for integration You can query the dbt metadata: * At the [environment](https://docs.getdbt.com/docs/environments-in-dbt.md) level for both the latest state (use the `environment` endpoint) and historical run results (use `modelHistoricalRuns`) of a dbt project in production. * At the job level for results on a specific dbt job run for a given resource type, like `models` or `test`. #### Prerequisites * You must have a dbt [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#multi-tenant) or [single tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#single-tenant) account. * You must be on a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * Your projects must be on a dbt [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or dbt version 1.0 or later. Refer to [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to upgrade. #### What you can use the Discovery API for Click the following tabs to learn more about the API's use cases, the analysis you can do, and the results you can achieve by integrating with it. To use the API directly or integrate your tool with it, refer to [Uses case and examples](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) for detailed information. ##### Performance Use the API to look at historical information like model build time to determine the health of your dbt projects. Finding inefficiencies in orchestration configurations can help decrease infrastructure costs and improve timeliness. To learn more about how to do this, refer to [Performance](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#performance). You can use, for example, the [model timing](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) tab to help identify and optimize bottlenecks in model builds: [![Model timing visualization in dbt](/img/docs/dbt-platform/discovery-api/model-timing.png?v=2 "Model timing visualization in dbt")](#)Model timing visualization in dbt ##### Quality Use the API to determine if the data is accurate and up-to-date by monitoring test failures, source freshness, and run status. Accurate and reliable information is valuable for analytics, decisions, and monitoring to help prevent your organization from making bad decisions. To learn more about this, refer to [Quality](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#quality). When used with [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md), it can also help with detecting, investigating, and alerting issues. ##### Discovery Use the API to find and understand dbt assets in integrated tools using information like model and metric definitions, and column information. For more details, refer to [Discovery](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#discovery). Data producers must manage and organize data for stakeholders, while data consumers need to quickly and confidently analyze data on a large scale to make informed decisions that improve business outcomes and reduce organizational overhead. The API is useful for discovery data experiences in catalogs, analytics, apps, and machine learning (ML) tools. It can help you understand the origin and meaning of datasets for your analysis. [![Data lineage produced by dbt](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "Data lineage produced by dbt")](#)Data lineage produced by dbt ##### Governance Use the API to review who developed the models and who uses them to help establish standard practices for better governance. For more details, refer to [Governance](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#governance). ##### Development Use the API to review dataset changes and uses by examining exposures, lineage, and dependencies. From the investigation, you can learn how to define and build more effective dbt projects. For more details, refer to [Development](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#development). [![Use exposures to embed data health tiles in your dashboards to distill trust signals for data consumers.](/img/docs/collaborate/dbt-explorer/data-tile-pass.png?v=2 "Use exposures to embed data health tiles in your dashboards to distill trust signals for data consumers.")](#)Use exposures to embed data health tiles in your dashboards to distill trust signals for data consumers. #### Types of project state You can query these two types of [project state](https://docs.getdbt.com/docs/dbt-apis/project-state.md) at the environment level: * **Definition** — The logical state of a dbt project’s [resources](https://docs.getdbt.com/docs/build/projects.md) that update when the project is changed. * **Applied** — The output of successful dbt DAG execution that creates or describes the state of the database (for example: `dbt run`, `dbt test`, source freshness, and so on) These states allow you to easily examine the difference between a model’s definition and its applied state so you can get answers to questions like, did the model run? or did the run fail? Applied models exist as a table/view in the data platform given their most recent successful run. #### Related docs * [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) * [Query the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) * [Schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md) --- ### About the Discovery API schema dbt platform | Starter, Enterprise, Enterprise+ With the Discovery API, you can query the metadata in dbt to learn more about your dbt deployments and the data they generate. You can analyze the data to make improvements. If you are new to the API, refer to [About the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) for an introduction. You might also find the [use cases and examples](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) helpful. The Discovery API *schema* provides all the pieces necessary to query and interact with the Discovery API. The most common queries use the `environment` endpoint: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment.md) ###### [Environment schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment.md) [Query and compare a model’s definition (intended) and its applied (actual) state.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) ###### [Applied schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) [Query the actual state of objects and metadata in the warehouse after a \`dbt run\` or \`dbt build\`.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) ###### [Definition schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) [Query intended state in project code and configuration defined in your dbt project.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-modelHistoricalRuns.md) ###### [Model Historical Runs schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-modelHistoricalRuns.md) [Query information about a model's run history.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-modelHistoricalRuns.md) --- ### Account-scoped personal access tokens dbt platform | Starter, Enterprise, Enterprise+ warning User API tokens have been deprecated and will no longer work. [Migrate](#migrate-deprecated-user-api-keys-to-personal-access-tokens) to personal access tokens to resume services. Each dbt user with a [Developer, Read-only, or IT license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) can create a new personal access token (PAT) to access the dbt API and dbt CLI. This token can execute queries against the dbt API on the user's behalf. To access dbt APIs and resources on behalf of the *account*, we recommend using service tokens instead. Learn more about [which token type you should use](https://docs.getdbt.com/docs/dbt-apis/authentication.md#which-token-type-should-you-use) to understand the token differences. PATs inherit the permissions of the user that created them. For example, if a developer-licensed user with Project Admin role access to specific projects creates a PAT, the token will get the Project Admin role with access to the same projects as the user. These tokens are also account-specific, so if a user has access to more than one dbt account with the same email address, they need to create a unique PAT for each one of these accounts. You can't use a PAT to create a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Requests to create a service token authenticated with a PAT return a `400` error — use an existing service token to create new ones instead. #### Create a personal access token Creating an account-scoped PAT requires only a few steps. 1. Navigate to your **Account Settings**, expand **API tokens** and click **Personal tokens**. 2. Click **Create personal access token**. 3. Give the token a descriptive name and click **Save**. 4. Copy the token before closing the window. *It will not be available after, and you will have to create a new token if you lose it.* To maintain best security practices, it's recommended that you regularly rotate your PATs. To do so, create a new token and delete the old one once it's in place. #### Delete a personal access token To permanently delete a PAT: 1. Navigate to your **Account Settings**, expand **API tokens** and click **Personal tokens**. 2. Find the token you want to delete and click "X" to the right of the token description fields. 3. **Confirm delete** and the token will no longer be valid. #### Migrate deprecated user API keys to personal access tokens The migration to PATs is critical if you are using user API keys today. The current API key is located under **Personal Settings → API Key**. There are a few things to understand if you are using a user API key today: * PATs are more secure. * To promote the least privilege and high-security assurance for your dbt accounts, we highly recommend moving to the new account-scoped PATs. * You must create and use unique tokens in each one of your dbt accounts that share the same email address. * For example, if belongs to two dbt accounts: Spice Harvesting Account and Guild Navigator Account. Before this release, the same API key was used to access both of these accounts. * After this release, Paul has to individually go into these accounts and create a unique PAT for each account he wants to access the API for. These PATs are account-specific and not user specific. * Cross-Account API endpoints will change in behavior when using PATs. * These are namely /v2/accounts and /v3/accounts. Since all PATs are now account specific, getting all accounts associated with a username cannot work. /v3/accounts will only return account metadata that’s relevant to the PAT that’s being used. * User account metadata will only contain information about the specific account under which the request is being made. * Any other accounts that belong to that user account will need to be requested through the PAT that belongs to that account. Undocumented APIs If you’re using any undocumented and unsupported API endpoints, please note that these can be deprecated without any notice. If you are using any undocumented endpoints and have use-cases that are not satisfied by the current API, please reach out to . ##### Using the personal access tokens Are you using a user API key today to access dbt APIs in any of your workflows? If not, you don’t have any action to take. If you are using a user API key, please follow the instructions below. 1. Make a list of all the places where you’re making a call to the dbt API using the dbt user API key. 2. Create a new PAT under **Account Settings → API Tokens → Personal Tokens.** For instructions, see [Create a personal access token](#create-a-personal-access-token). 3. Replace the API key in your APIs with the PAT you created. You can use a PAT wherever you previously used an API key. To replace the API key with a PAT, include the PAT in the Authorization header of your API requests. For example: `Authorization: Bearer `. Make sure to replace `` with the new PAT you created. note The option to rotate API keys is used for existing API keys, not for replacing them with PATs. You do not need to replace your API key with a PAT in the dbt UI. 4. Ensure that you’re using a PAT only where it's needed. For flows that require a service account, please use a service token. --- ### API rate limits dbt platform | Starter, Enterprise, Enterprise+ dbt applies request limits across the dbt platform APIs so integrations stay reliable and predictable. The [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) do not share the same limit. You can send more requests per minute to admin endpoints under `/api/` and fewer to Discovery's GraphQL endpoints under `/graphql/`, because Discovery queries often return more data per call. You use the same credentials for both APIs: [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md), [service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md), and OAuth when your organization supports it. Refer to [Authentication tokens](https://docs.getdbt.com/docs/dbt-apis/authentication.md) to create and use those credentials, and the [APIs overview](https://docs.getdbt.com/docs/dbt-apis/overview.md) to learn what each API does. This page summarizes the default rate limits across the main API and integration surfaces: * [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md): 5,000 requests per minute per account (`/api/`). * [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) (GraphQL): 500 requests per minute (`/graphql/` on the `metadata` host). * [Semantic Layer APIs](#semantic-layer-apis): Limited separately and not governed by the Discovery API limit. * [SCIM and IdP provisioning](#scim-and-idp-provisioning): 20 requests every 5 seconds per account. * [Remote MCP](#remote-mcp): 5,000 requests per minute per IP (global API rate limit). * [Self-hosted MCP](#local-mcp): Uses the Administrative and Discovery API limits above. For SCIM scope, throttling responses, and identity provider behavior, see [SCIM and IdP provisioning](#scim-and-idp-provisioning) and [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md). For MCP integration and usage patterns, see [Remote MCP](#remote-mcp) and [Self-hosted MCP](#local-mcp). Discovery API and Semantic Layer GraphQL endpoints The Discovery API limit applies only to the metadata endpoint. The [Semantic Layer GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) uses a different host and has its own rate limiting. #### Pagination and the Discovery API The most common way teams hit the Discovery API limit is to request very small pages in GraphQL (for example, `first: 1`) repeatedly. Each page is another request, so small pages require far more calls than fetching the same data in larger chunks. Use the largest page size that still works for you, up to what that query allows (often up to `500`). Full examples, including how to page with `after`, are in [Query the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md). #### SCIM and IdP provisioning [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") For SCIM, the application rate limit of 20 requests every 5 seconds per account (Enterprise and Enterprise+) applies only to creating a user (`POST /api/v3/accounts/{account_id}/scim/v2/Users`) and replacing a user (`PUT /api/v3/accounts/{account_id}/scim/v2/Users/{user_id}`). dbt counts requests in a fixed 5-second window. It does not apply to `GET`, `PATCH`, or `DELETE` requests for users; to any Group operations (create, list, get, replace, patch, or delete); or to other SCIM endpoints such as Service Provider Config, schemas, resource types, or config token routes. Those operations are outside this limit but may still be subject to other platform rate limits. When you exceed that limit, dbt returns `429 Too Many Requests` with both `Retry-After` and `x-rate-limit-retry-after-seconds` (for example, Okta integrations often honor `Retry-After`, while SailPoint often honors `x-rate-limit-retry-after-seconds`). These headers indicate how long to wait before you send the next request. Configure your identity provider to read the header your provisioning stack supports, and to retry with exponential backoff when throttled. For configuration steps, use [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md). For information on licenses, permissions, [SSO](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md), and how SCIM fits into account access, refer to [About user access in dbt](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md). #### Remote MCP [Remote MCP](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) uses the same default global API rate limit as other dbt APIs: 5,000 requests per minute per IP. It uses the same authentication as other integrations (for example, personal access tokens, service account tokens, or OAuth where supported). Treat remote MCP automation like any other API client: avoid retrying without pausing between attempts. When you receive a `429` response, wait before trying again, and wait longer between retries if you continue to receive `429` responses. #### Semantic Layer APIs dbt rate limits the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) (GraphQL, JDBC, and Python SDK) independently of the Administrative API and Discovery API. The Semantic Layer limit is shared across all clients, users, and accounts in the same instance (for example, every account in US1 shares a single limit), so your own request volume isn't the only factor that determines whether you hit it. #### Self-hosted MCP [Self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) calls the public [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) directly, so those limits apply: 5,000 requests per minute per account for `/api/` and 500 requests per minute for `/graphql/`, as summarized above. #### Exceeding the rate limit For the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md), if you exceed the limit, dbt returns `429 Too Many Requests` and enforces a five-minute cooldown. After five minutes, you can send requests again as usual. This cooldown doesn't apply to the Semantic Layer API. ##### Exceeding the rate limit for the Semantic Layer API If you receive a `429 Too Many Requests` error, you exceeded the Semantic Layer limit. There's no cooldown period, so requests succeed again as soon as shared usage drops back under the limit. To reduce your own request footprint: * Batch multiple metrics into a single query. * Retry with exponential backoff and jitter instead of polling at a fixed interval. * Use preset date ranges instead of custom ranges so dbt can reuse cached results. * Pre-materialize metrics that can be pre-aggregated by using [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) or [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md). Metrics that can't be pre-aggregated, such as distinct counts, still run live. These steps reduce your own usage but might not prevent every `429` error if other clients share the same limit. For dedicated options, contact your account team. --- ### APIs overview dbt platform | Starter, Enterprise, Enterprise+ Accounts on the Starter, Enterprise, and Enterprise+ plans can query the dbt APIs. dbt provides the following APIs: * The [dbt Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) can be used to administrate a dbt account. It can be called manually or with [the dbt Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest). * The [dbt Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) can be used to fetch metadata related to the state and health of your dbt project. * The [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) provides multiple API options which allow you to query your metrics defined in the Semantic Layer. If you want to learn more about webhooks, refer to [Webhooks for your jobs](https://docs.getdbt.com/docs/deploy/webhooks.md). For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). #### How to Access the APIs dbt supports two types of API Tokens: [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) and [service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Requests to the dbt APIs can be authorized using these tokens. --- ### Applied object schema dbt platform | Starter, Enterprise, Enterprise+ The applied object allows you to query information about a particular model based on `environmentId`. The [Example queries](#example/docs/dbt-apis-queries) illustrate a few fields you can query with this `environment` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. `executionInfo` fields track two different concepts Within `executionInfo`, `lastRun*` fields reflect the most recent run *attempt* (regardless of outcome), while `execute*`, `executionTime`, `runGeneratedAt`, and `lastSuccess*` fields reflect the most recent *successful* materialization. Refer to [Project state](https://docs.getdbt.com/docs/dbt-apis/project-state.md#definition-logical-vs-applied-state-of-dbt-nodes) for more information. ##### Example queries You can use your production environment's `id`: ```graphql query Example { environment(id: 834){ # Get the latest state of the production environment applied { # The state of an executed node as it exists as an object in the database models(first: 100){ # Pagination to ensure manageable response for large projects edges { node { uniqueId, name, description, rawCode, compiledCode, # Basic properties database, schema, alias, # Table/view identifier (can also filter by) executionInfo {executeCompletedAt, executionTime}, # Metadata from when the model was built tests {name, executionInfo{lastRunStatus, lastRunError}}, # Latest test results catalog {columns {name, description, type}, stats {label, value}}, # Catalog info ancestors(types:[Source]) {name, ...on SourceAppliedStateNestedNode {freshness{maxLoadedAt, freshnessStatus}}}, # Source freshness } children {name, resourceType}}} # Immediate dependencies in lineage totalCount } # Number of models in the project } } } ``` ##### Fields When querying the `applied` field of `environment`, you can use the following fields. ### Fetching data... --- ### Authentication tokens dbt platform | Starter, Enterprise, Enterprise+ [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) ###### [Personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) [Learn about user tokens and how to use them to execute queries against the dbt API.](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) ###### [Service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) [Learn how to use service account tokens to securely authenticate with dbt APIs for system-level integrations.](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) #### Types of API access tokens **Personal access tokens:** Preferred and secure way of accessing dbt APIs on behalf of a user. PATs are scoped to an account and can be enhanced with more granularity and control. **Service tokens:** Service tokens are similar to service accounts and are the preferred method to enable access on behalf of the dbt account. ##### Which token type should you use You should use service tokens broadly for any production workflow where you need a service account. You should use PATs only for developmental workflows *or* dbt client workflows that require user context. The following examples show you when to use a personal access token (PAT) or a service token: * **Connecting a partner integration to dbt** — Some examples include the [Semantic Layer Google Sheets integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), Hightouch, Datafold, a custom app you’ve created, etc. These types of integrations should use a service token instead of a PAT because service tokens give you visibility, and you can scope them to only what the integration needs and ensure the least privilege. We highly recommend switching to a service token if you’re using a personal access token for these integrations today. * **Production Terraform** — Use a service token since this is a production workflow and is acting as a service account and not a user account. * **dbt platform CLI** — Use a PAT since the dbt platform CLI works within the context of a user (the user is making the requests and has to operate within the context of their user account). * **Testing a custom script and staging Terraform or Postman** — We recommend using a PAT as this is a developmental workflow and is scoped to the user making the changes. When you push this script or Terraform into production, use a service token instead. * **API endpoints requiring user context** — Use PATs to authenticate to any API endpoint that requires user context (for example, endpoints to create and update user credentials). --- ### dbt Administrative API dbt platform | Starter, Enterprise, Enterprise+ The dbt Administrative API is enabled by default for [Starter, Enterprise, and Enterprise+ plans](https://www.getdbt.com/pricing/). It can be used to: * Download artifacts after a job has completed * Kick off a job run from an orchestration tool * Manage your dbt account * and more dbt currently supports two versions of the Administrative API: v2 and v3. In general, v3 is the recommended version to use, but we don't yet have all our v2 routes upgraded to v3. We're currently working on this. If you can't find something in our v3 docs, check out the shorter list of v2 endpoints because you might find it there. Many endpoints of the Administrative API can also be called through the [dbt Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest). The built-in documentation on the Terraform registry contains [a guide on how to get started with the provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/guides/1_getting_started) as well as [a page showing all the Terraform resources available](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/guides/99_list_resources) to configure. For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/dbt-cloud/api-v2) ###### [API v2](https://docs.getdbt.com/dbt-cloud/api-v2) [Our legacy API version, with limited endpoints and features. Contains information not available in v3.](https://docs.getdbt.com/dbt-cloud/api-v2) [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/dbt-cloud/api-v3) ###### [API v3](https://docs.getdbt.com/dbt-cloud/api-v3) [Our latest API version, with new endpoints and features.](https://docs.getdbt.com/dbt-cloud/api-v3) [![](/img/icons/pencil-paper.svg)](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) ###### [dbt Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) [The Terraform provider maintained by dbt Labs which can be used to manage a dbt account.](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) [](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) --- ### Definition object schema dbt platform | Starter, Enterprise, Enterprise+ The definition object allows you to query the logical state of a given project node given its most recent manifest generated models. The [Example queries](#example-queries) illustrate a few fields you can query with this `definition` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Example queries You can use your production environment's `id`: ```graphql query Example { environment(id: 834){ # Get the latest state of the production environment definition { # The logical state of a given project node given its most recent manifest generated models(first: 100, filter:{access:public}){ # Filter on model access (or other properties) edges { node { rawCode, # Compare to see if/how the model has changed since the last build jobDefinitionId, runGeneratedAt, # When the code was last compiled or run contractEnforced, group, version}}} # Model governance } } } ``` ##### Fields When querying the `definition` field of `environment`, you can use the following fields. ### Fetching data... --- ### Environment object schema dbt platform | Starter, Enterprise, Enterprise+ You can use the environment object to query and compare definition (intended) and applied (actual) states for nodes (models, seeds, snapshots, models, and more) in your dbt project. For example, you specify an `environmentId` to learn more about a particular model (or other node type) in that environment. The [Example queries](#example-queries) illustrate a few fields you can query with this `environment` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `environment`, you can use the following arguments. ### Fetching data... ##### Example queries You can use your production environment's `id`: ```graphql query Example { environment(id: 834){ # Get the latest state of the production environment applied { # The state of an executed node as it exists as an object in the database models(first: 100){ # Pagination to ensure manageable response for large projects edges { node { uniqueId, name, description, rawCode, compiledCode, # Basic properties database, schema, alias, # Table/view identifier (can also filter by) executionInfo {executeCompletedAt, executionTime}, # Metadata from when the model was built tests {name, executionInfo{lastRunStatus, lastRunError}}, # Latest test results catalog {columns {name, description, type}, stats {label, value}}, # Catalog info ancestors(types:[Source]) {name, ...on SourceAppliedStateNode {freshness{maxLoadedAt, freshnessStatus}}}, # Source freshness } children {name, resourceType}}} # Immediate dependencies in lineage totalCount } # Number of models in the project } definition { # The logical state of a given project node given its most recent manifest generated models(first: 100, filter:{access:public}){ # Filter on model access (or other properties) edges { node { rawCode, # Compare to see if/how the model has changed since the last build jobDefinitionId, runGeneratedAt, # When the code was last compiled or run contractEnforced, group, version}}} # Model governance } } ``` With the deprecation of the data type `Int` for `id`, below is an example of replacing it with `BigInt`: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId executionInfo { lastRunId } } } } } } } ``` With the deprecation of `modelByEnvironment`, below is an example of replacing it with `environment`: ```graphql query ($environmentId: BigInt!, $uniqueId: String) { environment(id: $environmentId) { applied { modelHistoricalRuns(uniqueId: $uniqueId) { uniqueId executionTime executeCompletedAt } } } } ``` ##### Fields When querying an `environment`, you can use the following fields. ### Fetching data... For details on querying the `applied` field of `environment`, you can visit: [Applied](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) For details querying the `definition` field of `environment`, you can visit: [Definition](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) --- ### Exposure object schema dbt platform | Starter, Enterprise, Enterprise+ The exposure object allows you to query information about a particular exposure. To learn more, refer to [Add Exposures to your DAG](https://docs.getdbt.com/docs/build/exposures.md). ##### Arguments When querying for an `exposure`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the exposure object. ##### Example query The example below queries information about an exposure including the owner's name and email, the URL, and information about parent sources and parent models. ```graphql { job(id: 123) { exposure(name: "my_awesome_exposure") { runId projectId name uniqueId resourceType ownerName url ownerEmail parentsSources { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } parentsModels { uniqueId } } } } ``` ##### Fields When querying for an `exposure`, the following fields are available: ### Fetching data... --- ### Exposure tile object schema dbt platform | Starter, Enterprise, Enterprise+ [Exposure health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) distill data health signals for data consumers and can be embedded in downstream tools. You can query information on these tiles from the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `exposureTile` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `exposureTile`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and filter by a model's `uniqueId` to understand the data quality and metadata information for the exposure tile associated with the `customers` model in the `marketing` package: ```graphql query { environment(id: 834) { applied { exposureTile( filter: {uniqueId: "model.marketing.customers"} # Use this format for unique ID: RESOURCE_TYPE.PACKAGE_NAME.RESOURCE_NAME ) { accountId # The account ID of this node environmentId projectId exposureType filePath quality } } } } ``` ##### Fields When querying for `exposureTile`, you can use the following fields: ### Fetching data... --- ### Exposures object schema dbt platform | Starter, Enterprise, Enterprise+ [Exposures](https://docs.getdbt.com/docs/build/exposures.md) are dbt resources that represent downstream uses of your project, such as dashboards, applications, or data science pipelines. You can query exposures through the Discovery API to understand which assets depend on your models. The [Example query](#example-query) illustrates a few fields you can query with the `exposures` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `exposures`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, `first: 100`, and filter by model `uniqueIds` to return all the downstream exposures (dashboards, applications, etc.) that depend on the `customers` model in the `marketing` package, limited to the first 100 results: ```graphql query { environment(id: 834) { applied { exposures( filter: { uniqueIds: ["model.marketing.customers"] # Use this format for unique ID: RESOURCE_TYPE.PACKAGE_NAME.RESOURCE_NAME }, first: 100 ) { edges { node { accountId exposureType fqn projectId url } } } } } } ``` ##### Fields When querying for `exposures`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Exposures object schema dbt platform | Starter, Enterprise, Enterprise+ The exposures object allows you to query information about all exposures in a given job. To learn more, refer to [Add Exposures to your DAG](https://docs.getdbt.com/docs/build/exposures.md). ##### Arguments When querying for `exposures`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the exposures object. ##### Example query The example below queries information about all exposures in a given job including the owner's name and email, the URL, and information about parent sources and parent models for each exposure. ```graphql { job(id: 123) { exposures(first: 10, after: "{somePaginationCursorValue}") { runId projectId name uniqueId resourceType ownerName url ownerEmail paginationCursor parentsSources { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } parentsModels { uniqueId } } } } ``` ##### Fields When querying for `exposures`, the following fields are available: ### Fetching data... --- ### GraphQL dbt platform | Starter, Enterprise, Enterprise+ [GraphQL](https://graphql.org/) (GQL) is an open-source query language for APIs. It offers a more efficient and flexible approach compared to traditional RESTful APIs. With GraphQL, users can request specific data using a single query, reducing the need for many server round trips. This improves performance and minimizes network overhead. GraphQL has several advantages, such as self-documenting, having a strong typing system, supporting versioning and evolution, enabling rapid development, and having a robust ecosystem. These features make GraphQL a powerful choice for APIs prioritizing flexibility, performance, and developer productivity. #### dbt Semantic Layer GraphQL API The Semantic Layer GraphQL API allows you to explore and query metrics and dimensions. Due to its self-documenting nature, you can explore the calls conveniently through a schema explorer. The schema explorer URLs vary depending on your [deployment region](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Use the following table to find the right link for your region: | Deployment type | Schema explorer URL | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | North America multi-tenant | | | EMEA multi-tenant | | | APAC multi-tenant | | | Single tenant | `https://semantic-layer.YOUR_ACCESS_URL/api/graphql`

Replace `YOUR_ACCESS_URL` with your specific account prefix followed by the appropriate Access URL for your region and plan. | | Multi-cell | `https://YOUR_ACCOUNT_PREFIX.semantic-layer.REGION.dbt.com/api/graphql`

Replace `YOUR_ACCOUNT_PREFIX` with your specific account identifier and `REGION` with your location, which could be `us1.dbt.com`. | **Example** * If your Single tenant access URL is `ABC123.getdbt.com`, your schema explorer URL will be `https://semantic-layer.ABC123.getdbt.com/api/graphql`. dbt Partners can use the Semantic Layer GraphQL API to build an integration with the Semantic Layer. Note that the Semantic Layer GraphQL API doesn't support `ref` to call dbt objects. Instead, use the complete qualified table name. If you're using dbt macros at query time to calculate your metrics, you should move those calculations into your Semantic Layer metric definitions as code. #### Requirements to use the GraphQL API * A dbt project on dbt v1.6 or higher * Metrics are defined and configured * A dbt [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with "Semantic Layer Only” and "Metadata Only" permissions or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) #### Using the GraphQL API If you're a dbt user or partner with access to dbt and the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), you can [set up](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and test this API with data from your own instance by configuring the Semantic Layer and obtaining the right GQL connection parameters described in this document. Refer to [Get started with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) for more info. Authentication uses either a dbt [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) passed through a header as follows. To explore the schema, you can enter this information in the "header" section. ```shell {"Authorization": "Bearer "} ``` Each GQL request also requires a dbt `environmentId`. The API uses both the service or personal token in the header and `environmentId` for authentication. ##### Metadata calls ###### Fetch data platform dialect In some cases in your application, it may be useful to know the dialect or data platform that's internally used for the Semantic Layer connection (such as if you are building `where` filters from a user interface rather than user-inputted SQL). The GraphQL API has an easy way to fetch this with the following query: ```graphql { environmentInfo(environmentId: BigInt!) { dialect } } ``` ###### Fetch available metrics ```graphql metricsPaginated( environmentId: BigInt! search: String = null groupBy: [GroupByInput!] = null pageNum: Int! = 1 pageSize: Int = null ): MetricResultPage! { items: [Metric!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Fetch available dimensions for metrics ```graphql dimensionsPaginated( environmentId: BigInt! metrics: [MetricInput!]! search: String = null pageNum: Int! = 1 pageSize: Int = null ): DimensionResultPage! { items: [Dimension!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Fetch available granularities given metrics Note: This call for `queryableGranularities` returns only queryable granularities for metric time - the primary time dimension across all metrics selected. ```graphql queryableGranularities( environmentId: BigInt! metrics: [MetricInput!]! ): [TimeGranularity!]! ``` You can also get queryable granularities for all other dimensions using the `dimensions` call: ```graphql { dimensionsPaginated(environmentId: BigInt!, metrics:[{name:"order_total"}]) { items { name queryableGranularities # --> ["DAY", "WEEK", "MONTH", "QUARTER", "YEAR"] } } } ``` You can also optionally access it from the metrics endpoint: ```graphql { metricsPaginated(environmentId: BigInt!) { items { name dimensions { name queryableGranularities } } } } ``` ###### Fetch entities ```graphql entitiesPaginated( environmentId: BigInt! metrics: [MetricInput!] = null search: String = null pageNum: Int! = 1 pageSize: Int = null ): EntityResultPage! { items: [Entity!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Fetch entities and dimensions to group metrics ```graphql groupBysPaginated( environmentId: BigInt! metrics: [MetricInput!] = null search: String = null pageNum: Int! = 1 pageSize: Int = null ): EntityDimensionResultPage! { items: [EntityDimension!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Metric types ```graphql Metric { name: String! description: String type: MetricType! typeParams: MetricTypeParams! filter: WhereFilter dimensions: [Dimension!]! queryableGranularities: [TimeGranularity!]! } ``` ```text MetricType = [SIMPLE, RATIO, CUMULATIVE, DERIVED] ``` ###### Metric type parameters (Applies to dbt v1.12 and later) ```graphql MetricTypeParams { numerator: MetricInput denominator: MetricInput expr: String window: MetricTimeWindow grainToDate: TimeGranularity metrics: [MetricInput!] } ``` ###### Dimension types ```graphql Dimension { name: String! description: String type: DimensionType! typeParams: DimensionTypeParams isPartition: Boolean! expr: String queryableGranularities: [TimeGranularity!]! } ``` ```text DimensionType = [CATEGORICAL, TIME] ``` ###### List saved queries List all saved queries for the specified environment: ```graphql savedQueriesPaginated( environmentId: BigInt! search: String = null pageNum: Int! = 1 pageSize: Int = null ): SavedQueryResultPage! { items: [SavedQuery!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### List a saved query List a single saved query using environment ID and query name: ```graphql { savedQuery(environmentId: "123", savedQueryName: "query_name") { name description label queryParams { metrics { name } groupBy { name grain datePart } where { whereSqlTemplate } } } } ``` ##### Querying When querying for data, *either* a `groupBy` *or* a `metrics` selection is required. The following section provides examples of how to query metrics: * [Create query](#create-metric-query) * [Fetch query result](#fetch-query-result) ###### Create query ```graphql createQuery( environmentId: BigInt! metrics: [MetricInput!]! groupBy: [GroupByInput!] = null limit: Int = null where: [WhereInput!] = null order: [OrderByInput!] = null ): CreateQueryResult ``` ```graphql MetricInput { name: String! alias: String! } GroupByInput { name: String! grain: TimeGranularity = null } WhereInput { sql: String! } OrderByinput { # -- pass one and only one of metric or groupBy metric: MetricInput = null groupBy: GroupByInput = null descending: Boolean! = false } ``` ###### Fetch query result ```graphql query( environmentId: BigInt! queryId: String! ): QueryResult! ``` The GraphQL API uses a polling process for querying since queries can be long-running in some cases. It works by first creating a query with a mutation, \`createQuery, which returns a query ID. This ID is then used to continuously check (poll) for the results and status of your query. The typical flow would look as follows: 1. Kick off a query ```graphql mutation { createQuery( environmentId: 123456 metrics: [{name: "order_total"}] groupBy: [{name: "metric_time"}] ) { queryId # => Returns 'QueryID_12345678' } } ``` 2. Poll for results ```graphql { query(environmentId: 123456, queryId: "QueryID_12345678") { sql status error totalPages jsonResult arrowResult } } ``` 3. Keep querying 2. at an appropriate interval until status is `FAILED` or `SUCCESSFUL` ##### Output format and pagination ###### Output format By default, the output is in Arrow format. You can switch to JSON format using the following parameter. However, due to performance limitations, we recommend using the JSON parameter for testing and validation. The JSON received is a base64 encoded string. To access it, you can decode it using a base64 decoder. The JSON is created from pandas, which means you can change it back to a dataframe using `pandas.read_json(json, orient="table")`. Or you can work with the data directly using `json["data"]`, and find the table schema using `json["schema"]["fields"]`. Alternatively, you can pass `encoded:false` to the jsonResult field to get a raw JSON string directly. ```graphql { query(environmentId: BigInt!, queryId: Int!, pageNum: Int! = 1) { sql status error totalPages arrowResult jsonResult(orient: PandasJsonOrient! = TABLE, encoded: Boolean! = true) } } ``` The results default to the table but you can change it to any [pandas](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) supported value. ###### Pagination By default, we return 1024 rows per page. If your result set exceeds this, you need to increase the page number using the `pageNum` option. ##### Run a Python query The `arrowResult` in the GraphQL query response is a byte dump, which isn't visually useful. You can convert this byte data into an Arrow table using any Arrow-supported language. Refer to the following Python example explaining how to query and decode the arrow result: ```python import base64 import pyarrow as pa import time headers = {"Authorization":"Bearer "} query_result_request = """ { query(environmentId: 70, queryId: "12345678") { sql status error arrowResult } } """ while True: gql_response = requests.post( "https://semantic-layer.cloud.getdbt.com/api/graphql", json={"query": query_result_request}, headers=headers, ) if gql_response.json()["data"]["status"] in ["FAILED", "SUCCESSFUL"]: break # Set an appropriate interval between polling requests time.sleep(1) """ gql_response.json() => { "data": { "query": { "sql": "SELECT\n ordered_at AS metric_time__day\n , SUM(order_total) AS order_total\nFROM semantic_layer.orders orders_src_1\nGROUP BY\n ordered_at", "status": "SUCCESSFUL", "error": null, "arrowResult": "arrow-byte-data" } } } """ def to_arrow_table(byte_string: str) -> pa.Table: """Get a raw base64 string and convert to an Arrow Table.""" with pa.ipc.open_stream(base64.b64decode(byte_string)) as reader: return pa.Table.from_batches(reader, reader.schema) arrow_table = to_arrow_table(gql_response.json()["data"]["query"]["arrowResult"]) # Perform whatever functionality is available, like convert to a pandas table. print(arrow_table.to_pandas()) """ order_total ordered_at 3 2023-08-07 112 2023-08-08 12 2023-08-09 5123 2023-08-10 """ ``` ##### Additional create query examples The following section provides query examples for the GraphQL API, such as how to query metrics, dimensions, where filters, and more: * [Query metric alias](#query-metric-alias) — Query with metric alias, which allows you to use simpler or more intuitive names for metrics instead of their full definitions. * [Query with a time grain](#query-with-a-time-grain) — Fetch multiple metrics with a change in time dimension granularities. * [Query multiple metrics and multiple dimensions](#query-multiple-metrics-and-multiple-dimensions) — Select common dimensions for multiple metrics. * [Query a categorical dimension on its own](#query-a-categorical-dimension-on-its-own) — Group by a categorical dimension. * [Query with a where filter](#query-with-a-where-filter) — Use the `where` parameter to filter on dimensions and entities using parameters. * [Query with order](#query-with-order) — Query with `orderBy`, accepts basic string that's a Dimension, Metric, or Entity. Defaults to ascending order. * [Query with limit](#query-with-limit) — Query using a `limit` clause. * [Query saved queries](#query-saved-queries) — Query using a saved query using the `savedQuery` parameter for frequently used queries. * [Query with just compiling SQL](#query-with-just-compiling-sql) — Query using a compile keyword using the `compileSql` mutation. * [Query records](#query-records) — View all the queries made in your project. ###### Query metric alias ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "metric_name", alias: "metric_alias"}] ) { ... } } ``` ###### Query with a time grain ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "order_total"}] groupBy: [{name: "metric_time", grain: MONTH}] ) { queryId } } ``` Note that when using granularity in the query, the output of a time dimension with a time grain applied to it always takes the form of a dimension name appended with a double underscore and the granularity level - `{time_dimension_name}__{DAY|WEEK|MONTH|QUARTER|YEAR}`. Even if no granularity is specified, it will also always have a granularity appended to it and will default to the lowest available (usually daily for most data sources). It is encouraged to specify a granularity when using time dimensions so that there won't be any unexpected results with the output data. ###### Query multiple metrics and multiple dimensions ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "food_order_amount"}, {name: "order_gross_profit"}] groupBy: [{name: "metric_time", grain: MONTH}, {name: "customer__customer_type"}] ) { queryId } } ``` ###### Query a categorical dimension on its own ```graphql mutation { createQuery( environmentId: "123" groupBy: [{name: "customer__customer_type"}] ) { queryId } } ``` ###### Query with a where filter The `where` filter takes a list argument (or a string for a single input). Depending on the object you are filtering, there are a couple of parameters: * `Dimension()` — Used for any categorical or time dimensions. For example, `Dimension('metric_time').grain('week')` or `Dimension('customer__country')`. * `Entity()` — Used for entities like primary and foreign keys, such as `Entity('order_id')`. Note: If you prefer a `where` clause with a more explicit path, you can optionally use `TimeDimension()` to separate categorical dimensions from time ones. The `TimeDimension` input takes the time dimension and optionally the granularity level. `TimeDimension('metric_time', 'month')`. ```graphql mutation { createQuery( environmentId: "123" metrics:[{name: "order_total"}] groupBy:[{name: "customer__customer_type"}, {name: "metric_time", grain: month}] where:[{sql: "{{ Dimension('customer__customer_type') }} = 'new'"}, {sql:"{{ Dimension('metric_time').grain('month') }} > '2022-10-01'"}] ) { queryId } } ``` ###### Multi-hop joins In cases where you need to query across multiple related tables (multi-hop joins), use the `entity_path` argument to specify the path between related entities. The following are examples of how you can define these joins: * In this example, you're querying the `location_name` dimension but specifying that it should be joined using the `order_id` field. ```sql {{Dimension('location__location_name', entity_path=['order_id'])}} ``` * In this example, the `salesforce_account_owner` dimension is joined to the `region` field, with the path going through `salesforce_account`. ```sql {{ Dimension('salesforce_account_owner__region',['salesforce_account']) }} ``` ###### Query with order ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "order_total"}] groupBy: [{name: "metric_time", grain: MONTH}] orderBy: [{metric: {name: "order_total"}}, {groupBy: {name: "metric_time", grain: MONTH}, descending:true}] ) { queryId } } ``` ###### Query with limit ```graphql mutation { createQuery( environmentId: "123" metrics: [{name:"food_order_amount"}, {name: "order_gross_profit"}] groupBy: [{name:"metric_time", grain: MONTH}, {name: "customer__customer_type"}] limit: 10 ) { queryId } } ``` ###### Query saved queries This takes the same inputs as the `createQuery` mutation, but includes the field `savedQuery`. You can use this for frequently used queries. ```graphql mutation { createQuery( environmentId: "123" savedQuery: "new_customer_orders" ) { queryId } } ``` A note on querying saved queries When querying [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md),you can use parameters such as `where`, `limit`, `order`, `compile`, and so on. However, keep in mind that you can't access `metric` or `group_by` parameters in this context. This is because they are predetermined and fixed parameters for saved queries, and you can't change them at query time. If you would like to query more metrics or dimensions, you can build the query using the standard format. ###### Query with just compiling SQL This takes the same inputs as the `createQuery` mutation. ```graphql mutation { compileSql( environmentId: "123" metrics: [{name:"food_order_amount"} {name:"order_gross_profit"}] groupBy: [{name:"metric_time", grain: MONTH}, {name:"customer__customer_type"}] ) { sql } } ``` ###### Query records Use this endpoint to view all the queries made in your project. This covers both Insights and Semantic Layer queries. ```graphql { queryRecords( environmentId:123 ) { items { queryId status startTime endTime connectionDetails sqlDialect connectionSchema error queryDetails { ... on SemanticLayerQueryDetails { params { type metrics { name } groupBy { name grain } limit where { sql } orderBy { groupBy { name grain } metric { name } descending } savedQuery } } ... on RawSqlQueryDetails { queryStr compiledSql numCols queryDescription queryTitle } } } totalItems pageNum pageSize } } ``` --- ### JDBC API dbt platform | Starter, Enterprise, Enterprise+ The Semantic Layer Java Database Connectivity (JDBC) API enables users to query metrics and dimensions using the JDBC protocol, while also providing standard metadata functionality. A JDBC driver is a software component enabling a Java application to interact with a data platform. Here's some more information about our JDBC API: * The Semantic Layer JDBC API utilizes the open-source JDBC driver with ArrowFlight SQL protocol. * You can download the JDBC driver from [Maven](https://search.maven.org/remotecontent?filepath=org/apache/arrow/flight-sql-jdbc-driver/12.0.0/flight-sql-jdbc-driver-12.0.0.jar). * The Semantic Layer supports ArrowFlight SQL driver version 12.0.0 and higher. * You can embed the driver into your application stack as needed, and you can use dbt Labs' [example project](https://github.com/dbt-labs/example-semantic-layer-clients) for reference. * If you’re a partner or user building a homegrown application, you’ll need to install an AWS root CA to the Java Trust [documentation](https://www.amazontrust.com/repository/) (specific to Java and JDBC call). dbt Labs partners can use the JDBC API to build integrations in their tools with the Semantic Layer #### Using the JDBC API If you are a dbt user or partner with access to dbt and the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), you can [setup](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and test this API with data from your own instance by configuring the Semantic Layer and obtaining the right JDBC connection parameters described in this document. You *may* be able to use our JDBC API with tools that do not have an official integration with the Semantic Layer. If the tool you use allows you to write SQL and either supports a generic JDBC driver option (such as DataGrip) or supports Dremio and uses ArrowFlightSQL driver version 12.0.0 or higher, you can access the Semantic Layer API. Refer to [Get started with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) for more info. Note that the Semantic Layer GraphQL API doesn't support `ref` to call dbt objects. Instead, use the complete qualified table name. If you're using dbt macros at query time to calculate your metrics, you should move those calculations into your Semantic Layer metric definitions as code. #### Authentication dbt authorizes requests to the Semantic Layer API. You need to provide an Environment ID, Host, and [service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). #### Connection parameters The JDBC connection requires a few different connection parameters. This is an example of a URL connection string and the individual components: ```text jdbc:arrow-flight-sql://semantic-layer.cloud.getdbt.com:443?&environmentId=202339&token=AUTHENTICATION_TOKEN ``` | JDBC parameter | Description | Example | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `jdbc:arrow-flight-sql://` | The protocol for the JDBC driver. | `jdbc:arrow-flight-sql://` | | `semantic-layer.cloud.getdbt.com` | The [access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your account's dbt region. You must always add the `semantic-layer` prefix before the access URL. | For dbt deployment hosted in North America, use `semantic-layer.cloud.getdbt.com` | | `environmentId` | The unique identifier for the dbt production environment, you can retrieve this from the dbt URL
when you navigate to **Environments** under **Deploy**. | If your URL ends with `.../environments/222222`, your `environmentId` is `222222`

| | `AUTHENTICATION_TOKEN` | You can use either a dbt [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with “Semantic Layer Only” and "Metadata Only" permissions or a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Create a new service or personal token on the **Account Settings** page. | `token=AUTHENTICATION_TOKEN` | \*Note — If you're testing locally on a tool like DataGrip, you may also have to provide the following variable at the end or beginning of the JDBC URL `&disableCertificateVerification=true`. #### Querying the API for metadata The Semantic Layer JDBC API has built-in metadata calls which can provide a user with information about their metrics and dimensions. Expand the following toggles for examples and metadata commands:  Fetch defined metrics You can use this query to fetch all defined metrics in your dbt project: ```bash select * from {{ semantic_layer.metrics() }} ```  Fetch dimension for a metric You can use this query to fetch all dimensions for a metric. Note, metrics is a required argument that lists one or multiple metrics in it. ```bash select * from {{ semantic_layer.dimensions(metrics=['food_order_amount'])}} ```  Fetch granularities for metrics You can use this query to fetch queryable granularities for a list of metrics. This API request allows you to only show the time granularities that make sense for the primary time dimension of the metrics (such as metric\_time), but if you want queryable granularities for other time dimensions, you can use the dimensions() call, and find the column queryable\_granularities. Note, metrics is a required argument that lists one or multiple metrics. ```bash select * from {{ semantic_layer.queryable_granularities(metrics=['food_order_amount', 'order_gross_profit'])}} ```  Fetch available metrics given dimensions You can use this query to fetch available metrics given dimensions. This command is essentially the opposite of getting dimensions given a list of metrics. Note, group\_by is a required argument that lists one or multiple dimensions. ```bash select * from {{ semantic_layer.metrics_for_dimensions(group_by=['customer__customer_type']) }} ```  Fetch granularities for all time dimensions You can use this example query to fetch available granularities for all time dimensions (the similar queryable granularities API call only returns granularities for the primary time dimensions for metrics). The following call is a derivative of the dimensions() call and specifically selects the granularity field. ```bash select NAME, QUERYABLE_GRANULARITIES from {{ semantic_layer.dimensions( metrics=["order_total"] ) }} ```  Paginate metadata calls In the case when you don't want to return the full result set from a metadata call, you can paginate the results for both `semantic_layer.metrics()` and `semantic_layer.dimensions()` calls using the `page_size` and `page_number` parameters. * `page_size`: This is an optional variable which sets the number of records per page. If left as None, there is no page limit. * `page_number`: This is an optional variable which specifies the page number to retrieve. Defaults to `1` (first page) if not specified. Examples: ```sql -- Retrieves the 5th page with a page size of 10 metrics select * from {{ semantic_layer.metrics(page_size=10, page_number=5) }} -- Retrieves the 1st page with a page size of 10 metrics select * from {{ semantic_layer.metrics(page_size=10) }} -- Retrieves all metrics without pagination select * from {{ semantic_layer.metrics() }} ``` You can use the same pagination parameters for `semantic_layer.dimensions(...)`.  List saved queries You can use this example query to list all available saved queries in your dbt project. **Command** ```bash select * from semantic_layer.saved_queries() ``` **Output** ```bash | NAME | DESCRIPTION | LABEL | METRICS | GROUP_BY | WHERE_FILTER | ```  Fetch metric aliases You can query metrics using aliases for simpler or more intuitive names, even if the alias isn't defined in the metric configuration. The query returns the alias as the metric name, for example: ```sql select * from {{ semantic_layer.query(metrics=[Metric("metric_name", alias="metric_alias")]) }} ``` In this example, if you define an alias for `revenue` as `banana`, the query will return a column named `banana` even if `banana` isn't defined in the metric configuration. However, when using `where` Jinja clauses, you need to reference the *actual* metric name (`revenue` in this case) instead of the alias. For more a more detailed example, see [Query metric alias](#query-metric-alias). #### Querying the API for values To query values, the following parameters are available. Your query must have *either* a `metric` **or** a `group_by` parameter to be valid. | Parameter | Description | Example | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metrics` | The metric name as defined in your dbt metric configuration | `metrics=['revenue']` | | `group_by` | Dimension names or entities to group by. We require a reference to the entity of the dimension (other than for the primary time dimension), which is pre-appended to the front of the dimension name with a double underscore. | `group_by=['user__country', 'metric_time']` | | `grain` | A parameter specific to any time dimension and changes the grain of the data from the default for the metric. | `group_by=[Dimension('metric_time')`
`grain('week\|day\|month\|quarter\|year')]` | | `where` | A where clause that allows you to filter on dimensions and entities using parameters. This takes a filter list OR string. Inputs come with `Dimension`, and `Entity` objects. Granularity is required if the `Dimension` is a time dimension | `"{{ where=Dimension('customer__country') }} = 'US')"` | | `limit` | Limit the data returned | `limit=10` | | `order` | Order the data returned by a particular field | `order_by=['order_gross_profit']`, use `-` for descending, or full object notation if the object is operated on: `order_by=[Metric('order_gross_profit').descending(True)`] | | `compile` | If true, returns generated SQL for the data platform but does not execute | `compile=True` | | `saved_query` | A saved query you can use for frequently used queries. | `select * from {{ semantic_layer.query(saved_query="new_customer_orders"` | ##### Note on time dimensions and `metric_time` You will notice that in the list of dimensions for all metrics, there is a dimension called `metric_time`. `Metric_time` is a reserved keyword for any metric's default aggregation time dimension. For any time-series metric, the `metric_time` keyword should always be available for use in queries. This is a common dimension across *all* metrics in a semantic graph. You can look at a single metric or hundreds of metrics, and if you group by `metric_time`, it will always give you the correct time series. Additionally, when performing granularity calculations that are global (not specific to a particular time dimension), we recommend you always operate on `metric_time` and you will get the correct answer. Note that `metric_time` should be available in addition to any other time dimensions that are available for the metric(s). In the case where you are looking at one metric (or multiple metrics from the same data source), the values in the series for the primary time dimension and `metric_time` are equivalent. #### Examples The following sections provide examples of how to query metrics using the JDBC API: * [Fetch metadata for metrics](#fetch-metadata-for-metrics) — Filter/add any SQL outside of the templating syntax. * [Query common dimensions](#query-common-dimensions) — Select common dimensions for multiple metrics. * [Query grouped by time](#query-grouped-by-time) — Fetch revenue and new customers grouped by time. * [Query with a time grain](#query-with-a-time-grain) — Fetch multiple metrics with a change in time dimension granularities. * [Group by categorical dimension](#group-by-categorical-dimension) — Group by a categorical dimension. * [Query only a dimension](#query-only-a-dimension) — Get the full list of dimension values for the chosen dimension. * [Query by all dimensions](#query-by-all-dimensions) — Query by all valid dimensions. * [Query with where filters](#query-with-where-filters) — Use the `where` parameter to filter on dimensions and entities using parameters. * [Query with a limit](#query-with-a-limit) — Query using a `limit` or `order_by` clause. * [Query with order by examples](#query-with-order-by-examples) — Query with `order_by`, accepts basic string that's a Dimension, Metric, or Entity. Defaults to ascending order. Add a `-` sign in front of the object for descending order. * [Query with compile keyword](#query-with-compile-keyword) — Query using a compile keyword to preview the final SQL before execution. * [Query a saved query](#query-a-saved-query) — Query using a saved query with optional parameters like `limit` or `where`. * [Query metric alias](#query-metric-alias) — Query metrics using aliases, which allow you to use simpler or more intuitive names for metrics instead of their full definitions. * [Multi-hop joins](#multi-hop-joins) — Query across multiple related tables (multi-hop joins) using the `entity_path` argument to specify the path between related entities. ##### Fetch metadata for metrics You can filter/add any SQL outside of the templating syntax. For example, you can use the following query to fetch the name and dimensions for a metric: ```bash select name, dimensions from {{ semantic_layer.metrics() }} WHERE name='food_order_amount' ``` ##### Query common dimensions You can select common dimensions for multiple metrics. Use the following query to fetch the name and dimensions for multiple metrics: ```bash select * from {{ semantic_layer.dimensions(metrics=['food_order_amount', 'order_gross_profit']) }} ``` ##### Query grouped by time The following example query uses the [shorthand method](#faqs) to fetch revenue and new customers grouped by time: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount','order_gross_profit'], group_by=['metric_time']) }} ``` ##### Query with a time grain Use the following example query to fetch multiple metrics with a change in time dimension granularities: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month')]) }} ``` ##### Group by categorical dimension Use the following query to group by a categorical dimension: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'), 'customer__customer_type']) }} ``` ##### Query only a dimension In this case, you'll get the full list of dimension values for the chosen dimension. ```bash select * from {{ semantic_layer.query(group_by=['customer__customer_type']) }} ``` ##### Query by all dimensions You can use the `semantic_layer.query_with_all_group_bys` endpoint to query by all valid dimensions. ```sql select * from {{ semantic_layer.query_with_all_group_bys(metrics =['revenue','orders','food_orders'], compile= True) }} ``` This returns all dimensions that are valid for the set of metrics in the request. ##### Query with where filters Where filters in API allow for a filter list or string. We recommend using the filter list for production applications as this format will realize all benefits from the Predicate pushdown where possible. Where Filters have a few objects that you can use: * `Dimension()` — Used for any categorical or time dimensions. `Dimension('metric_time').grain('week')` or `Dimension('customer__country')`. * `TimeDimension()` — Used as a more explicit definition for time dimensions, optionally takes in a granularity `TimeDimension('metric_time', 'month')`. * `Entity()` — Used for entities like primary and foreign keys - `Entity('order_id')`. You can use the following example to query using a `where` filter with the string format: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'),'customer__customer_type'], where="{{ Dimension('metric_time').grain('month') }} >= '2017-03-09' AND {{ Dimension('customer__customer_type' }} in ('new') AND {{ Entity('order_id') }} = 10") }} ``` * (Recommended for better performance) Use the following example to query using a `where` filter with a filter list format: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'),'customer__customer_type'], where=["{{ Dimension('metric_time').grain('month') }} >= '2017-03-09'", "{{ Dimension('customer__customer_type') }} in ('new')", "{{ Entity('order_id') }} = 10"]) }} ``` ##### Query with a limit Use the following example to query using a `limit` or `order_by` clause: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10) }} ``` ##### Query with order by examples Order By can take a basic string that's a Dimension, Metric, or Entity, and this will default to ascending order ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10, order_by=['order_gross_profit']) }} ``` For descending order, you can add a `-` sign in front of the object. However, you can only use this short-hand notation if you aren't operating on the object or using the full object notation. ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10, order_by=['-order_gross_profit']) }} ``` If you are ordering by an object that's been operated on (for example, you changed the granularity of the time dimension), or you are using the full object notation, descending order must look like: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('week')], limit=10, order_by=[Metric('order_gross_profit').descending(True), Dimension('metric_time').grain('week').descending(True) ]) }} ``` Similarly, this will yield ascending order: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('week')], limit=10, order_by=[Metric('order_gross_profit'), Dimension('metric_time').grain('week')]) }} ``` ##### Query with compile keyword * Use the following example to query using a `compile` keyword: ```sql select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'),'customer__customer_type'], compile=True) }} ``` * Use the following example to compile SQL with a [saved query](https://docs.getdbt.com/docs/build/saved-queries.md). You can use this for frequently used queries. ```sql select * from {{ semantic_layer.query(saved_query="new_customer_orders", limit=5, compile=True}} ``` A note on querying saved queries When querying [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md),you can use parameters such as `where`, `limit`, `order`, `compile`, and so on. However, keep in mind that you can't access `metric` or `group_by` parameters in this context. This is because they are predetermined and fixed parameters for saved queries, and you can't change them at query time. If you would like to query more metrics or dimensions, you can build the query using the standard format. ##### Query a saved query Use the following example to query a [saved query](https://docs.getdbt.com/docs/build/saved-queries.md): ```sql select * from {{ semantic_layer.query(saved_query="new_customer_orders", limit=5}} ``` The JDBC API will use the saved query (`new_customer_orders`) as defined and apply a limit of 5 records. ##### Query metric alias You can query metrics using aliases, which allow you to use simpler or more intuitive names for metrics instead of their full definitions. ```sql select * from {{ semantic_layer.query(metrics=[Metric("revenue", alias="metric_alias")]) }} ``` For example, let's say your metric configuration includes an alias like `total_revenue_global` for the `order_total` metric. You can query the metric using the alias instead of the original name: ```sql select * from {{ semantic_layer.query(metrics=[Metric("order_total", alias="total_revenue_global")], group_by=['metric_time']) }} ``` The result will be: ```text | METRIC_TIME | TOTAL_REVENUE_GLOBAL | |:-------------:|:------------------: | | 2023-12-01 | 1500.75 | | 2023-12-02 | 1725.50 | | 2023-12-03 | 1850.00 | ``` tip Note that you need to use the actual metric name when using the `where` Jinja clauses. For example, if you used `banana` as an alias for `revenue`, you need to use the actual metric name, `revenue`, in the `where` clause, not `banana`. ```graphql semantic_layer.query(metrics=[Metric("revenue", alias="banana")], where="{{ Metric('revenue') }} > 0") ``` ##### Multi-hop joins In cases where you need to query across multiple related tables (multi-hop joins), use the `entity_path` argument to specify the path between related entities. The following are examples of how you can define these joins: * In this example, you're querying the `location_name` dimension but specifying that it should be joined using the `order_id` field. ```sql {{Dimension('location__location_name', entity_path=['order_id'])}} ``` * In this example, the `salesforce_account_owner` dimension is joined to the `region` field, with the path going through `salesforce_account`. ```sql {{ Dimension('salesforce_account_owner__region',['salesforce_account']) }} ``` #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error.  Why do some dimensions use different syntax, like \`metric\_time\` versus \`Dimension('metric\_time')\`? When you select a dimension on its own, such as `metric_time` you can use the shorthand method which doesn't need the “Dimension” syntax. However, when you perform operations on the dimension, such as adding granularity, the object syntax `[Dimension('metric_time')` is required.  What does the double underscore \`'\_\_'\` syntax in dimensions mean? The double underscore `"__"` syntax indicates a mapping from an entity to a dimension, as well as where the dimension is located. For example, `user__country` means someone is looking at the `country` dimension from the `user` table.  What is the default output when adding granularity? The default output follows the format `{{time_dimension_name}__{granularity_level}}`. So for example, if the `time_dimension_name` is `ds` and the granularity level is yearly, the output is `ds__year`. #### Related docs * [Semantic Layer integration best practices](https://docs.getdbt.com/guides/sl-partner-integration-guide.md) --- ### Job object schema dbt platform | Starter, Enterprise, Enterprise+ The job object allows you to query information about a particular model based on `jobId` and, optionally, a `runId`. If you don't provide a `runId`, the API returns information on the latest runId of a job. The [example query](#example-query) illustrates a few fields you can query in this `job` object. Refer to [Fields](#fields) to see the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `job`, you can use the following arguments. ### Fetching data... ##### Example Query You can use your production job's `id`. ```graphql query JobQueryExample { # Provide runId for looking at specific run, otherwise it defaults to latest run job(id: 940) { # Get all models from this job's latest run models(schema: "analytics", first: 10, after: "{somePaginationCursorValue}") { uniqueId executionTime paginationCursor } # Or query a single node source(uniqueId: "source.jaffle_shop.snowplow.event") { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } } } ``` ##### Fields When querying an `job`, you can use the following fields. ### Fetching data... --- ### Lineage object schema dbt platform | Starter, Enterprise, Enterprise+ The lineage object allows you to query lineage across your resources. The [Example query](#example-query) illustrates a few fields you can query with the `lineage` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `lineage`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and filter by "Model" as the resource type to see lineage information for all models in this environment, including their dependencies, materialization type, and metadata: ```graphql query { environment(id: 834) { applied { lineage( filter: {"types": ["Model"]} # Return results for the Model type ) { name resourceType filePath projectId materializationType parentIds tags uniqueId } } } } ``` ##### Fields When querying for `lineage`, you can use the following fields: ### Fetching data... --- ### Model Historical Runs object schema dbt platform | Starter, Enterprise, Enterprise+ The model historical runs object allows you to query information about a model's run history. The [Example query](#example-query) illustrates a few fields you can query with the `modelHistoricalRuns` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `modelHistoricalRuns`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and the model's `uniqueId` to return the model and its execution history for the `customers` model in the `marketing` package, including performance metrics and test results for the last 20 times it was run, regardless of which job ran it. ```graphql query { environment(id: 834) { applied { modelHistoricalRuns( uniqueId: "model.marketing.customers" # Use this format for unique ID: RESOURCE_TYPE.PACKAGE_NAME.RESOURCE_NAME lastRunCount: 20 ) { runId # Get historical results for a particular model runGeneratedAt executionTime # View build time across runs status tests { name status executeCompletedAt } # View test results across runs } } } } ``` ##### Fields When querying for `modelHistoricalRuns`, you can use the following fields: ### Fetching data... --- ### Model object schema dbt platform | Starter, Enterprise, Enterprise+ The model object allows you to query information about a particular model in a given job. ##### Arguments When querying for a `model`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema (all possible fields you can query) of the model object. ##### Example query for finding parent models and sources The example query below uses the `parentsModels` and `parentsSources` fields to fetch information about a model’s parent models and parent sources. The jobID and uniqueID fields are placeholders that you will need to replace with your own values. ```graphql { job(id: 123) { model(uniqueId: "model.jaffle_shop.dim_user") { parentsModels { runId uniqueId executionTime } parentsSources { runId uniqueId state } } } } ``` ##### Example query for model timing The example query below could be useful if you want to understand information around execution timing on a given model (start, end, completion). ```graphql { job(id: 123) { model(uniqueId: "model.jaffle_shop.dim_user") { runId projectId name uniqueId resourceType executeStartedAt executeCompletedAt executionTime } } } ``` ##### Example query for column-level information You can use the following example query to understand more about the columns of a given model. This query will only work if the job has generated documentation; that is, it will work with the command `dbt docs generate`. ```graphql { job(id: 123) { model(uniqueId: "model.jaffle_shop.dim_user") { columns { name index type comment description tags meta } } } } ``` ##### Fields When querying for a `model`, the following fields are available: ### Fetching data... --- ### Models object schema dbt platform | Starter, Enterprise, Enterprise+ [Models](https://docs.getdbt.com/docs/build/models.md) are the foundational dbt resource that transform raw data into curated datasets using SQL (or Python). Each model represents a single SELECT statement, typically materialized as a table or view in your warehouse. You can query information about models through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `models` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `models`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` to return model information for all models in the given environment, including their metadata, configuration, tests, and ownership details, limited to the first 100 results: ```graphql query { environment(id: 834) { applied { models (first: 100) { edges { node { name description access accountId catalog { owner } config environmentId tests { name description } } } } } } } ``` ##### Fields When querying for `models`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Models object schema dbt platform | Starter, Enterprise, Enterprise+ The models object allows you to query information about all models in a given job. ##### Arguments When querying for `models`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the models object. ##### Example queries The database, schema, and identifier arguments are all optional. This means that with this endpoint you can: * Find a specific model by providing `..` * Find all of the models in a database and/or schema by providing `` and/or `` ###### Find models by their database, schema, and identifier The example query below finds a model by its unique database, schema, and identifier. ```graphql { job(id: 123) { models( database: "analytics" schema: "analytics" identifier: "dim_customers" first: 10 after: "{somePaginationCursorValue}" ) { uniqueId paginationCursor } } } ``` ###### Find models by their schema The example query below finds all models in this schema and their respective execution times. ```graphql { job(id: 123) { models(schema: "analytics", first: 10, after: "{somePaginationCursorValue}") { uniqueId executionTime paginationCursor } } } ``` ###### Paginate models in a job run Job-based `models` queries support `first` and `after` for cursor pagination. Each model returns a `paginationCursor` field to fetch the next page. Refer to [Job-based queries](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md#job-based-queries) for paging behavior, supported resource types, and legacy default behavior. ```graphql query JobModelsPage($jobId: BigInt!, $runId: BigInt, $first: Int!, $after: String) { job(id: $jobId, runId: $runId) { models(first: $first, after: $after) { uniqueId executionTime paginationCursor } } } ``` ##### Fields The models object can access the *same fields* as the [Model node](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-model.md). The difference is that the models object can output a list so instead of querying for fields for one specific model, you can query for those parameters for all models within a jobID, database, and so on. When querying for `models`, the following fields are available: ### Fetching data... --- ### Owners object schema dbt platform | Starter, Enterprise, Enterprise+ [Owners](https://docs.getdbt.com/docs/build/groups.md) help you identify the user or domain responsible for a dbt asset. For most assets, owners are defined in your project code using groups. Exposures are an exception: for downstream exposures that represent BI assets, owners are automatically pulled from the downstream tool based on who owns that asset. You can query ownership information through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `owners` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `owners`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and "exposure" as the `OwnerResourceType` to return people who own exposures (downstream BI assets) in this environment, including their contact information. ```graphql query { environment(id: 834) { applied { owners(resource: exposure) { email name } } } } ``` ##### Fields When querying for `owners`, you can use the following fields: ### Fetching data... --- ### Packages object schema dbt platform | Starter, Enterprise, Enterprise+ [dbt packages](https://docs.getdbt.com/docs/build/packages.md) are libraries with models, macros, and other resources that tackle a specific problem area utilized by dbt projects. You can query project packages through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `packages` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `packages`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and "model" as the resource to see all dbt packages in this environment that contain model resources: ```graphql query { environment(id: 834) { applied { packages(resource: "model") } } } ``` --- ### Project state in dbt dbt provides a stateful way of deploying dbt. Artifacts are accessible programmatically via the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) in the metadata platform. With the implementation of the `environment` endpoint in the Discovery API, we've introduced the idea of multiple states. The Discovery API provides a single API endpoint that returns the latest state of models, sources, and other nodes in the DAG. A single [deployment environment](https://docs.getdbt.com/docs/environments-in-dbt.md) should represent the production state of a given dbt project. There are two states that can be queried in dbt: * **Applied state** refers to what exists in the data warehouse after a successful `dbt run`. The model build succeeds and now exists as a table in the warehouse. * **Definition state** depends on what exists in the project given the code defined in it (for example, manifest state), which hasn’t necessarily been executed in the data platform (maybe just the result of `dbt compile`). #### Definition (logical) vs. applied state of dbt nodes In a dbt project, the state of a node *definition* represents the configuration, transformations, and dependencies defined in the SQL and YAML files. It captures how the node should be processed in relation to other nodes and tables in the data warehouse and may be produced by a `dbt build`, `run`, `parse`, or `compile`. It changes whenever the project code changes. A node’s *applied state* refers to the node’s actual state after it has been successfully executed in the DAG; for example, models are executed; thus, their state is applied to the data warehouse via `dbt run` or `dbt build`. It changes whenever a node is executed. This state represents the result of the transformations and the actual data stored in the database, which for models can be a table or a view based on the defined logic. The applied state includes execution info, which contains metadata about how the node arrived in the applied state. The fields within `executionInfo` track two related but distinct concepts: | Concept | Description | Fields | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | Most recent run attempt | The latest run regardless of outcome (success, error, or skip) | `lastRunId`, `lastRunStatus`, `lastRunError`, `lastRunGeneratedAt`, `lastJobDefinitionId` | | Most recent successful materialization | The last run in which the node was built in the data warehouse.
When a run errors out, the node isn't rebuilt, so these fields remain pinned to the prior successful run. | `executeStartedAt`, `executeCompletedAt`, `executionTime`, `runGeneratedAt`, `lastSuccessRunId`, `lastSuccessJobDefinitionId` | For example, if a model's most recent run errors out, `lastRunStatus` will be `error` and `lastRunGeneratedAt` will reference that failed run, while `executeCompletedAt` and `lastSuccessRunId` will still reference the prior run in which the model was successfully materialized. Here’s how you can query and compare the definition vs. applied state of a model using the Discovery API: ```graphql query Compare($environmentId: Int!, $first: Int!) { environment(id: $environmentId) { definition { models(first: $first) { edges { node { name rawCode } } } } applied { models(first: $first) { edges { node { name rawCode executionInfo { executeCompletedAt } } } } } } } ``` Most Discovery API use cases will favor the *applied state* since it pertains to what has actually been run and can be analyzed. #### Affected states by node type The following table shows the states of dbt nodes and how they are affected by the Discovery API. | Node | Executed in DAG | Created by execution | Exists in database | Lineage | States | | -------------------------------------------------------------------------------------- | --------------- | -------------------- | ------------------ | --------------------- | -------------------- | | [Analysis](https://docs.getdbt.com/docs/build/analyses.md) | No | No | No | Upstream | Definition | | [Data test](https://docs.getdbt.com/docs/build/data-tests.md) | Yes | Yes | No | Upstream | Applied & definition | | [Exposure](https://docs.getdbt.com/docs/build/exposures.md) | No | No | No | Upstream | Definition | | [Group](https://docs.getdbt.com/docs/build/groups.md) | No | No | No | Downstream | Definition | | [Macro](https://docs.getdbt.com/docs/build/jinja-macros.md) | Yes | No | No | N/A | Definition | | [Metric](https://docs.getdbt.com/docs/build/metrics-overview.md) | No | No | No | Upstream & downstream | Definition | | [Model](https://docs.getdbt.com/docs/build/models.md) | Yes | Yes | Yes | Upstream & downstream | Applied & definition | | [Saved queries](https://docs.getdbt.com/docs/build/saved-queries.md)
(not in API) | N/A | N/A | N/A | N/A | N/A | | [Seed](https://docs.getdbt.com/docs/build/seeds.md) | Yes | Yes | Yes | Downstream | Applied & definition | | [Semantic model](https://docs.getdbt.com/docs/build/semantic-models.md) | No | No | No | Upstream & downstream | Definition | | [Snapshot](https://docs.getdbt.com/docs/build/snapshots.md) | Yes | Yes | Yes | Upstream & downstream | Applied & definition | | [Source](https://docs.getdbt.com/docs/build/sources.md) | Yes | No | Yes | Downstream | Applied & definition | | [Unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) | Yes | Yes | No | Downstream | Definition | #### Caveats about state/metadata updates Over time, Cloud Artifacts will provide information to maintain state for features/services in dbt and enable you to access state in dbt and its downstream ecosystem. Cloud Artifacts is currently focused on the latest production state, but this focus will evolve. Here are some limitations of the state representation in the Discovery API: * Users must access the default production environment to know the latest state of a project. * The API gets the definition from the latest manifest generated in a given deployment environment, but that often won’t reflect the latest project code state. * Compiled code results may be outdated depending on dbt run step order and failures. * Catalog info can be outdated, or incomplete (in the applied state), based on if/when `docs generate` was last run. * Source freshness checks can be out of date (in the applied state) depending on when the command was last run, and it’s not included in `build`. --- ### Python SDK dbt platform | Starter, Enterprise, Enterprise+ The [`dbt-sl-sdk` Python software development kit](https://github.com/dbt-labs/semantic-layer-sdk-python) (SDK) is a Python library that provides you with easy access to the dbt Semantic Layer with Python. It allows developers to interact with the dbt Semantic Layer APIs and query metrics and dimensions in downstream tools. #### Installation To install the Python SDK, you'll need to specify optional dependencies depending on whether you want to use it synchronously, backed by [requests](https://github.com/psf/requests/), or with asynchronous ([asyncio](https://docs.python.org/3/library/asyncio.html) backed by [aiohttp](https://github.com/aio-libs/aiohttp/)). The Python SDK supports the Long-Term Support (LTS) versions of Python, such as 3.9, 3.10, 3.11, and 3.12. When Python discontinues support for a version, the Python SDK will also discontinue support for that version. If you’re using a non-supported version, you may experience compatibility issues and won’t receive updates or security patches from the SDK. ##### Sync installation Sync installation means your program waits for each task to finish before moving on to the next one. It's simpler, easier to understand, and suitable for smaller tasks or when your program doesn't need to handle many tasks at the same time. ```bash pip install "dbt-sl-sdk[sync]" ``` If you're using async frameworks like [FastAPI](https://fastapi.tiangolo.com/) or [Strawberry](https://github.com/strawberry-graphql/strawberry), installing the sync version of the SDK will block your event loop and can significantly slow down your program. In this case, we strongly recommend using async installation. ##### Async installation Async installation means your program can start a task and then move on to other tasks while waiting for the first one to finish. This can handle many tasks at once without waiting, making it faster and more efficient for larger tasks or when you need to manage multiple tasks at the same time. For more details, refer to [asyncio](https://docs.python.org/3/library/asyncio.html). ```bash pip install "dbt-sl-sdk[async]" ``` Since the [Python ADBC driver](https://github.com/apache/arrow-adbc/tree/main/python/adbc_driver_manager) doesn't yet support asyncio natively, `dbt-sl-sdk` uses a [`ThreadPoolExecutor`](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/5e52e1ca840d20a143b226ae33d194a4a9bc008f/dbtsl/api/adbc/client/asyncio.py#L62) to run `query` and `list dimension-values` (all operations that are done with ADBC). This is why you might see multiple Python threads spawning. If you're using async frameworks like [FastAPI](https://fastapi.tiangolo.com/) or [Strawberry](https://github.com/strawberry-graphql/strawberry), installing the sync version of the Python SDK will block your event loop and can significantly slow down your program. In this case, we strongly recommend using async installation. #### Usage To run operations against the Semantic Layer APIs, instantiate (create an instance of) a `SemanticLayerClient` with your specific [API connection parameters](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md): ```python from dbtsl import SemanticLayerClient client = SemanticLayerClient( environment_id=123, auth_token="", host="semantic-layer.cloud.getdbt.com", ) # query the first metric by `metric_time` def main(): with client.session(): metrics = client.metrics() table = client.query( metrics=[metrics[0].name], group_by=["metric_time"], ) print(table) main() ``` **Note**: All method calls that reach out to the APIs need to be within a `client.session()` context manager. This allows the client to establish a connection to the APIs only once and reuse the same connection between API calls. We recommend creating an application-wide session and reusing the same session throughout the application for optimal performance. Creating a session per request is discouraged and inefficient. ##### asyncio usage If you're using asyncio, import `AsyncSemanticLayerClient` from `dbtsl.asyncio`. The `SemanticLayerClient` and `AsyncSemanticLayerClient` APIs are identical, but the async version has async methods that you need to `await`. ```python import asyncio from dbtsl.asyncio import AsyncSemanticLayerClient client = AsyncSemanticLayerClient( environment_id=123, auth_token="", host="semantic-layer.cloud.getdbt.com", ) async def main(): async with client.session(): metrics = await client.metrics() table = await client.query( metrics=[metrics[0].name], group_by=["metric_time"], ) print(table) asyncio.run(main()) ``` ##### Lazy loading for large fields By default, the Python SDK eagerly loads nested lists of objects such as `dimensions`, `entities`, and `measures` for each `Metric` — even if you don't need them. This is generally convenient, but in large projects, it can lead to slower responses due to the amount of data returned. To improve performance, you can opt into lazy loading by passing `lazy=True` when creating the client. With lazy loading enabled, the SDK skips fetching large nested fields until you explicitly request them on a per-model basis. (Applies to dbt v1.12 and later) Lazy loading is currently only supported for `dimensions` and `entities` on `Metric` objects. For example, the following code fetches all available metrics from the metadata API and displays only the dimensions of certain metrics: list\_metrics\_lazy\_sync.py ```python """Fetch all available metrics from the metadata API and display only the dimensions of certain metrics.""" from argparse import ArgumentParser from dbtsl import SemanticLayerClient def get_arg_parser() -> ArgumentParser: p = ArgumentParser() p.add_argument("--env-id", required=True, help="The dbt environment ID", type=int) p.add_argument("--token", required=True, help="The API auth token") p.add_argument("--host", required=True, help="The API host") return p def main() -> None: arg_parser = get_arg_parser() args = arg_parser.parse_args() client = SemanticLayerClient( environment_id=args.env_id, auth_token=args.token, host=args.host, lazy=True, ) with client.session(): metrics = client.metrics() for i, m in enumerate(metrics): print(f"📈 {m.name}") print(f" type={m.type}") print(f" description={m.description}") assert len(m.dimensions) == 0 # skip if index is odd if i & 1: print(" dimensions=skipped") continue # load dimensions only if index is even m.load_dimensions() print(" dimensions=[") for dim in m.dimensions: print(f" {dim.name},") print(" ]") if __name__ == "__main__": main() ``` Refer to the [lazy loading example](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/list_metrics_lazy_sync.py) for more details. #### Integrate with dataframe libraries The Python SDK returns all query data as [pyarrow](https://arrow.apache.org/docs/python/index.html) tables. The Python SDK library doesn't come bundled with [Polars](https://pola.rs/) or [Pandas](https://pandas.pydata.org/). If you use these libraries, add them as dependencies in your project. To use the data with libraries like Polars or Pandas, manually convert the data into the desired format. For example: ###### If you're using pandas ```python # ... initialize client arrow_table = client.query(...) pandas_df = arrow_table.to_pandas() ``` ###### If you're using polars ```python import polars as pl # ... initialize client arrow_table = client.query(...) polars_df = pl.from_arrow(arrow_table) ``` #### Usage examples For additional usage examples, check out the [usage examples](https://github.com/dbt-labs/semantic-layer-sdk-python/tree/main/examples), some of which include: * [Fetching dimension values sync](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_dimension_values_sync.py) * Fetching metrics [async](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_metric_async.py) and [sync](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_metric_sync.py) * [List saved queries async](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/list_saved_queries_async.py) #### Disable telemetry By default, the Python SDK sends some [platform-related information](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/dbtsl/env.py) to dbt Labs. To opt-out, set the `PLATFORM.anonymous` attribute to `True`: ```python from dbtsl.env import PLATFORM PLATFORM.anonymous = True # ... initialize client ``` #### Contribute To contribute to this project, check out our [contribution guidelines](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/CONTRIBUTING.md) and open a GitHub [issue](https://github.com/dbt-labs/semantic-layer-sdk-python/issues) or [pull request](https://github.com/dbt-labs/semantic-layer-sdk-python/pulls). --- ### Query the Discovery API dbt platform | Starter, Enterprise, Enterprise+ The Discovery API supports ad-hoc queries and integrations. If you are new to the API, refer to [About the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) for an introduction. For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). Use the Discovery API to evaluate data pipeline health and project state across runs or at a moment in time. dbt Labs provide a default [GraphQL explorer](https://metadata.cloud.getdbt.com/graphql) for this API, enabling you to run queries and browse the schema. However, you can also use any GraphQL client of your choice to query the API. Since GraphQL describes the data in the API, the schema displayed in the GraphQL explorer accurately represents the graph and fields available to query. #### Prerequisites * You must have a dbt [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#multi-tenant) or [single tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#single-tenant) account. * You must be on a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * Your projects must be on a dbt [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or dbt version 1.0 or later. Refer to [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to upgrade. #### Authorization Currently, authorization of requests takes place [using a service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). dbt admin users can generate a Metadata Only service token that is authorized to execute a specific query against the Discovery API. Once you've created a token, you can use it in the Authorization header of requests to the dbt Discovery API. Be sure to include the Token prefix in the Authorization header, or the request will fail with a `401 Unauthorized` error. Note that `Bearer` can be used instead of `Token` in the Authorization header. Both syntaxes are equivalent. #### Access the Discovery API 1. Create a [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) to authorize requests. dbt Admin users can generate a *Metadata Only* service token, which can be used to execute a specific query against the Discovery API to authorize requests. 2. Find the API URL to use from the [Discovery API endpoints](#discovery-api-endpoints) table. 3. For specific query points, refer to the [schema documentation](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md). #### Run queries using HTTP requests You can run queries by sending a `POST` request to the Discovery API, making sure to replace: * `YOUR_API_URL` with the appropriate [Discovery API endpoint](#discovery-api-endpoints) for your region and plan. * `YOUR_TOKEN` in the Authorization header with your actual API token. Be sure to include the Token prefix. * `QUERY_BODY` with a GraphQL query, for example `{ "query": "", "variables": "" }` * `VARIABLES` with a dictionary of your GraphQL query variables, such as a job ID or a filter. * `ENDPOINT` with the endpoint you're querying, such as environment. ```shell curl 'YOUR_API_URL' \ -H 'authorization: Bearer YOUR_TOKEN' \ -H 'content-type: application/json' -X POST --data QUERY_BODY ``` Python example: ```python response = requests.post( 'YOUR_API_URL', headers={"authorization": "Bearer "+YOUR_TOKEN, "content-type": "application/json"}, json={"query": QUERY_BODY, "variables": VARIABLES} ) metadata = response.json()['data'][ENDPOINT] ``` Every query will require an environment ID or job ID. You can get the ID from a dbt URL or using the Admin API. There are several illustrative example queries on this page. For more examples, refer to [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md). #### Discovery API endpoints Find your Discovery API endpoint in **Account settings** under **Access URLs** in dbt platform. The format depends on your deployment type and region: | Deployment type | Discovery API URL | | --------------- | ------------------------------------------------------------------- | | Multi-tenant | `https://ACCOUNT_PREFIX.metadata.REGION.dbt.com/graphql` | | Single-tenant | `https://ACCOUNT_PREFIX.metadata.ROUTING_SUBDOMAIN.dbt.com/graphql` | * Replace the following placeholders with your actual values. Refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for more information on the regions and subdomains: * `ACCOUNT_PREFIX` with your account identifier (found in **Account settings** under **Access URLs**) * `REGION` with your deployment region (for example, `us1` for North America AWS, `eu1` for EMEA, `jp1` for Japan, `au1` for APAC, and so on) * `ROUTING_SUBDOMAIN` with your single-tenant routing subdomain (typically your company name. Please contact your account team if unsure) * For example, if you're on North America AWS with account prefix `abc123`, your Discovery API URL is `https://abc123.metadata.us1.dbt.com/graphql`. #### Reasonable use Discovery (GraphQL) API usage is subject to request rate and response size limits to maintain the performance and stability of the metadata platform and prevent abuse. Job-level endpoints are subject to query complexity limits. Nested nodes (like parents), code (like rawCode), and catalog columns are considered as most complex. Overly complex queries should be broken up into separate queries with only necessary fields included. dbt Labs recommends using the environment endpoint instead for most use cases to get the latest descriptive and result metadata for a dbt project. #### Retention limits You can use the Discovery API to query data from the previous two months. For example, if today was April 1st, you could query data back to February 1st. #### Run queries with the GraphQL explorer You can run ad-hoc queries directly in the [GraphQL API explorer](https://metadata.cloud.getdbt.com/graphql) and use the document explorer on the left-hand side to see all possible nodes and fields. Refer to the [Apollo explorer documentation](https://www.apollographql.com/docs/graphos/explorer/explorer) for setup and authorization information for GraphQL. 1. Access the [GraphQL API explorer](https://metadata.cloud.getdbt.com/graphql) and select fields you want to query. 2. Select **Variables** at the bottom of the explorer and replace any `null` fields with your unique values. 3. [Authenticate](https://www.apollographql.com/docs/graphos/explorer/connecting-authenticating#authentication) using Bearer auth with `YOUR_TOKEN`. Select **Headers** at the bottom of the explorer and select **+New header**. 4. Select **Authorization** in the **header key** dropdown list and enter your Bearer auth token in the **value** field. Remember to include the Token prefix. Your header key should be in this format: `{"Authorization": "Bearer }`.
[![Enter the header key and Bearer auth token values](/img/docs/dbt-platform/discovery-api/graphql_header.jpg?v=2 "Enter the header key and Bearer auth token values")](#)Enter the header key and Bearer auth token values 1. Run your query by clicking the blue query button in the top right of the **Operation** editor (to the right of the query). You should see a successful query response on the right side of the explorer. [![Run queries using the Apollo Server GraphQL explorer](/img/docs/dbt-platform/discovery-api/graphql.jpg?v=2 "Run queries using the Apollo Server GraphQL explorer")](#)Run queries using the Apollo Server GraphQL explorer ##### Fragments Use the [`... on`](https://www.apollographql.com/docs/react/data/fragments/) notation to query across lineage and retrieve results from specific node types. ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" }) { edges { node { name ancestors(types: [Model, Source, Seed, Snapshot]) { ... on ModelAppliedStateNestedNode { name resourceType materializedType executionInfo { executeCompletedAt } } ... on SourceAppliedStateNestedNode { sourceName name resourceType freshness { maxLoadedAt } } ... on SnapshotAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } ... on SeedAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } } } } } } } } ``` ##### Pagination Querying large datasets can impact performance on multiple functions in the API pipeline. Pagination eases the burden by returning smaller data sets one page at a time. This is useful for returning a particular portion of the dataset or the entire dataset piece-by-piece to enhance performance. dbt utilizes cursor-based pagination, which makes it easy to return pages of constantly changing data. Use the `PageInfo` object to return information about the page. The available fields are: * `startCursor` string type — Corresponds to the first `node` in the `edge`. * `endCursor` string type — Corresponds to the last `node` in the `edge`. * `hasNextPage` boolean type — Whether or not there are more `nodes` after the returned results. There are connection variables available when making the query: * `first` integer type — Returns the first n `nodes` for each page, up to 500. * `after` string type — Sets the cursor to retrieve `nodes` after. It's best practice to set the `after` variable with the object ID defined in the `endCursor` of the previous page. Below is an example that returns the `first` 500 models `after` the specified Object ID in the variables. The `PageInfo` object returns where the object ID where the cursor starts, where it ends, and whether there is a next page. [![Example of pagination](/img/Paginate.png?v=2 "Example of pagination")](#)Example of pagination Below is a code example of the `PageInfo` object: ```graphql pageInfo { startCursor endCursor hasNextPage } totalCount # Total number of records across all pages ``` The previously described `PageInfo` and `totalCount` pattern applies to the `environment` endpoints, which return results as a connection of `edges` and `nodes`. ###### Job-based endpoint pagination Job-based list endpoints also support cursor-based pagination. Use this pattern when you query resources under the [`job`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md) object. These endpoints return a flat list rather than an `edges`/`nodes` connection, so they do not expose a `PageInfo` object. The following job-based list fields accept `first` and `after`: * `models` * `sources` * `seeds` * `snapshots` * `tests` * `macros` * `metrics` * `exposures` Pass `first` to set the page size (capped at 100). Each returned item includes a `paginationCursor` field—an opaque, per-element value you select in your query. Pass the last item's `paginationCursor` as `after` on the next request to fetch the following page. Unlike environment queries, these job-based endpoints do not return a `PageInfo` object or `hasNextPage` field. You have reached the last page when a page returns fewer rows than `first` (or fewer than the default page size). We recommend always specifying `first` to keep response sizes manageable. Historically, omitting `first` and `after` would cause the API to return all matching rows. For large jobs, best practice is to pass `first` and `after` explicitly so your integration does not depend on unpaginated responses. Upcoming change to unpaginated requests In a future update, requests that omit `first` and `after` will return at most 100 rows. dbt Labs will provide advance notice before that change rolls out. To run the example, use your [Discovery API endpoint](#discovery-api-endpoints) and a [Metadata Only service token](#authorization). Get `jobId` and optional `runId` from the job or run URL, or with the [Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md). If you omit `runId`, the API uses the job's latest run. Run the query in the [GraphQL explorer](#run-queries-with-the-graphql-explorer) or via [HTTP requests](#run-queries-using-http-requests). The example below uses a job's `models` list. The same `first` and `after` arguments work for the other resource types listed above. For a use-case example with more fields, refer to [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md). ```graphql query JobModelsPage($jobId: BigInt!, $runId: BigInt, $first: Int!, $after: String) { job(id: $jobId, runId: $runId) { models(first: $first, after: $after) { uniqueId paginationCursor } } } ``` First page variables: ```json { "jobId": 12345, "runId": 67890, "first": 10, "after": null } ``` For the next page, set `after` to the `paginationCursor` from the *last* row of the previous page. The cursor is an opaque encoded string, not the `uniqueId`. Repeat until a page returns fewer rows than `first` (or fewer than the default page size). ```json { "jobId": 12345, "runId": 67890, "first": 10, "after": "Y3Vyc29yOm1vZGVsLm15X3Byb2plY3QuZGltX2N1c3RvbWVycw==" } ``` ##### Filters Filtering helps to narrow down the results of an API query. If you want to query and return only models and tests that are failing or find models that are taking too long to run, you can fetch execution details such as [`executionTime`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md#fields), [`runElapsedTime`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md#fields), or [`status`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md#fields). This helps data teams monitor the performance of their models, identify bottlenecks, and optimize the overall data pipeline. Below is an example that filters for results of models that have succeeded on their `lastRunStatus`: [![Example of filtering](/img/Filtering.png?v=2 "Example of filtering")](#)Example of filtering Below is an example that filters for models that have an error on their last run and tests that have failed: ```graphql query ModelsAndTests($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { lastRunStatus: error }) { edges { node { name executionInfo { lastRunId } } } } tests(first: $first, filter: { status: "fail" }) { edges { node { name executionInfo { lastRunId } } } } } } } ``` #### Related content * [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) * [Schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md) --- ### Resources object schema dbt platform | Starter, Enterprise, Enterprise+ The resources object allows you to paginate across all resources in your environment. The [Example query](#example-query) illustrates a few fields you can query with the `resources` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `resources`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, filter by "Model" as the type, and limit to the first 100 results to see comprehensive information about the first 100 model resources in this environment, including their metadata, tags, and file locations: ```graphql query { environment(id: 834) { applied { resources( filter: { types: [ Model ] }, first: 100 ) { edges { node { accountId description environmentId filePath meta name projectId resourceType uniqueId tags } } } } } } ``` ##### Fields When querying for `resources`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Seed object schema dbt platform | Starter, Enterprise, Enterprise+ The seed object allows you to query information about a particular seed in a given job. ##### Arguments When querying for a `seed`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the seed object. ##### Example query The example query below pulls relevant information about a given seed. For instance, you can view the load time. ```graphql { job(id: 123) { seed(uniqueId: "seed.jaffle_shop.raw_customers") { database schema uniqueId name status error } } } ``` ##### Fields When querying for a `seed`, the following fields are available: ### Fetching data... --- ### Seeds object schema dbt platform | Starter, Enterprise, Enterprise+ [Seeds](https://docs.getdbt.com/docs/build/seeds.md) are CSV files in your dbt project that dbt can load into your data warehouse. You can query seeds through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `seeds` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `seeds`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, filter by the database, and limit to the first 100 to show information about the first 100 seed files in the `analytics` database, including their metadata and file locations: ```graphql query ($environmentId: BigInt!, $first: Int!, $filter: GenericMaterializedFilter) { environment(id: $environmentId) { applied { seeds( first: 100, filter: { database: "analytics" } ) { edges { node { description name filePath projectId fqn tags uniqueId resourceType } } } } } } ``` ##### Fields When querying for `seeds`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Seeds object schema dbt platform | Starter, Enterprise, Enterprise+ The seeds object allows you to query information about all seeds in a given job. ##### Arguments When querying for `seeds`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the seeds object. ##### Example query The example query below pulls relevant information about all seeds in a given job. For instance, you can view load times. ```graphql { job(id: 123) { seeds(first: 10, after: "{somePaginationCursorValue}") { uniqueId name executionTime status paginationCursor } } } ``` ##### Fields When querying for `seeds`, the following fields are available: ### Fetching data... --- ### Semantic Layer APIs dbt platform | Starter, Enterprise, Enterprise+ The rapid growth of different tools in the modern data stack has helped data professionals address the diverse needs of different teams. The downside of this growth is the fragmentation of business logic across teams, tools, and workloads.

The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) allows you to define metrics in code (with [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md)) and dynamically generate and query datasets in downstream tools based on their dbt governed assets, such as metrics and models. Integrating with the Semantic Layer will help organizations that use your product make more efficient and trustworthy decisions with their data. It also helps you to avoid duplicative coding, optimize development workflow, ensure data governance, and guarantee consistency for data consumers. You can use the Semantic Layer for a variety of tools and applications of data. Some common use cases are: * Business intelligence (BI), reporting, and analytics * Data quality and monitoring * Governance and privacy * Data discovery and cataloging * Machine learning and data science #### Rate limits Semantic Layer APIs have their own rate limits, which are shared across clients and users and are separate from the Discovery API limit that applies to the `metadata` GraphQL endpoint. Refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md#semantic-layer-apis) for details. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) ###### [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) [Use GraphQL to query metrics and dimensions in downstream tools.](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) ###### [JDBC API](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) [Use a JDBC driver to query metrics and dimensions in downstream tools, while also providing standard metadata functionality.](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) ###### [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) [Use the Python SDK to interact with the dbt Semantic Layer using Python.](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) --- ### Service account tokens dbt platform | Starter, Enterprise, Enterprise+ Service account tokens enable you to securely authenticate with the dbt API by assigning each token a narrow set of permissions that more precisely manages access to the API. While similar to [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md), service account tokens belong to an account rather than a user. You can use service account tokens for system-level integrations that do not run on behalf of any one user. Assign any permission sets available in dbt to your service account token, which can vary slightly depending on your plan: * Enterprise and Enterprise+ plans can apply any permission sets available to service tokens. * Developer and Starter plans can apply Semantic Layer permissions set to service tokens. * Legacy Team plans can apply Account Admin, Member, Job Admin, Read-Only, Metadata, and Semantic Layer permissions set to service tokens. You can assign as many permission sets as needed to one token. For more on permissions sets, see "[Enterprise Permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md)." #### Generate service account tokens You can generate service tokens if you have a Developer [license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and account admin [permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#permission-sets). To create a service token in dbt, follow these steps: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. On the left sidebar, click **Service Tokens**. 3. Click the **+ Create service token** button to generate a new token. 1. Enter a name for your token. 2. Add the necessary [permissions](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). 4. Once the token is generated, you won't be able to view this token again so make sure to save it somewhere safe. #### Permissions for service account tokens You can assign service account tokens to any permission set available in dbt. When you assign a permission set to a token, you will also be able to choose whether to grant those permissions to all projects in the account or to specific projects. ##### Team plans using service account tokens The following permissions can be assigned to a service account token on a Team plan. Refer to [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information about these roles. * Account Admin — Account Admin service tokens have full `read + write` access to an account, so please use them with caution. A Team plan refers to this permission set as an "Owner role." * Billing Admin * Job Admin * Metadata Only * Member * Read-only * Semantic Layer Only ##### Enterprise plans using service account tokens Refer to [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information about these roles. * Account Admin — Account Admin service tokens have full `read + write` access to an account, so please use them with caution. * Account Viewer * Admin * Analyst * Billing Admin * Database Admin * Developer * Git Admin * Job Admin * Job Runner * Job Viewer * Manage marketplace apps * Metadata Only * Read-only * Semantic Layer Only * Security Admin * Stakeholder * Team Admin #### Service token update On July 18, 2023, dbt Labs changed how tokens are generated and validated to increase performance. These improvements only apply to tokens created after July 18, 2023. Old tokens remain valid, but if they are used in high-frequency API invocations, we recommend you rotate them for reduced latency. To rotate your token: 1. Navigate to **Account settings** and click **Service tokens** on the left side pane. 2. Verify the **Created** date for the token is *on or before* July 18, 2023. 3. Click **+ New Token** on the top right side of the screen. Ensure the new token has the same permissions as the old one. 4. Copy the new token and replace the old one in your systems. Store it in a safe place, as it will not be available again once the creation screen is closed. 5. Delete the old token in dbt by clicking the **trash can icon**. *Only take this action after the new token is in place to avoid service disruptions*. #### FAQs I'm receiving a 403 error 'Forbidden: Access denied' when using service tokens All [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) traffic is subject to IP restrictions. When using a service token, the following 403 response error indicates the IP is not on the allowlist. To resolve this, you should add your third-party integration CIDRs (network addresses) to your allowlist. The following is an example of the 403 response error: ```json { "status": { "code": 403, "is_success": False, "user_message": ("Forbidden: Access denied"), "developer_message": None, }, "data": { "account_id": , "user_id": , "is_service_token": , "account_access_denied": True, }, } ``` --- ### Service principals dbt platform | Enterprise, Enterprise+ TODO: One or two sentences on what a service principal is (the identity and permissions for a non-human integration) and why you'd use one instead of a bare service token. #### Service principals vs credentials TODO: Explain the split with a small table. Principal = who the integration is and what it can do. Credential = the secret it uses to log in (service token now, OAuth client credential later). #### Account vs project service principals TODO: Table comparing who creates each, where it lives in settings, and what permissions it can have. Note that project principals can't mix in account-level permissions. #### Choose an authentication method TODO: Short table comparing service token, OAuth client credential and PAT. Say when to use each. #### Create a service principal TODO: Steps for account admins in account settings. Mention scoping permissions to specific projects or environments. #### Add a credential TODO: Steps to issue a service token under a principal. Add a note on saving it, since it's shown once. Link OAuth client credentials once they ship. #### Edit a service principal TODO: How to change the name or permissions. Add the warning that new permissions apply to all its credentials, and that credentials aren't revoked. #### Delete a service principal or revoke a credential TODO: What the confirmation says, and that deleting a principal deletes all its credentials. Add a note that deleting a project deletes its project-scoped principals. #### Permissions TODO: Who can create, view and manage principals at each level, and which permission sets are needed. Check against the new `project_service_principal_*` permissions once they're public. #### Existing service tokens TODO: Say whether customers need to do anything. The PRD says existing tokens migrate with no behavior change. Confirm with the PM. #### Audit log TODO: List the events for principal and credential create, update and delete. Link to the audit log page. #### FAQs TODO: Add after review. Likely questions are what happens if the creator leaves and who sees project principals. --- ### Snapshots object schema dbt platform | Starter, Enterprise, Enterprise+ [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) represent point-in-time copies of your data, allowing you to track historical changes. You can query your snapshots from the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `snapshots` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `snapshots`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, filter by the database, and limit to the first 100 to see the first 100 snapshots in the `analytics` database, including their execution performance and status information: ```graphql query { environment(id: 834) { applied { snapshots( filter: { database: "analytics" }, first: 100 ) { edges { node { executionInfo { compileCompletedAt compileStartedAt executeCompletedAt executeStartedAt executionTime lastRunStatus lastRunId } fqn name } } } } } } ``` ##### Fields When querying for `snapshots`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Snapshots object schema dbt platform | Starter, Enterprise, Enterprise+ The snapshots object allows you to query information about all snapshots in a given job. ##### Arguments When querying for `snapshots`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the snapshots object. ##### Example query The database, schema, and identifier arguments are optional. This means that with this endpoint you can: * Find a specific snapshot by providing `..` * Find all of the snapshots in a database and/or schema by providing `` and/or `` ###### Find snapshots information for a job The example query returns information about all snapshots in this job. ```graphql { job(id: 123) { snapshots(first: 10, after: "{somePaginationCursorValue}") { uniqueId name executionTime environmentId executeStartedAt executeCompletedAt paginationCursor } } } ``` ##### Fields When querying for `snapshots`, the following fields are available: ### Fetching data... --- ### Source object schema dbt platform | Starter, Enterprise, Enterprise+ The source object allows you to query information about a particular source in a given job. ##### Arguments When querying for a `source`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the source object. ##### Example query The query below pulls relevant information about a given source. For instance, you can view the load time and the state (pass, fail, error) of that source. ```graphql { job(id: 123) { source(uniqueId: "source.jaffle_shop.snowplow.event") { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } } } ``` ##### Fields When querying for a `source`, the following fields are available: ### Fetching data... --- ### Sources object schema dbt platform | Starter, Enterprise, Enterprise+ [Sources](https://docs.getdbt.com/docs/build/sources.md) make it possible to name and describe the data loaded into your warehouse by your extract and load tools. You can query sources through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `sources` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `sources`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and filter on the database name, to return the freshness and execution status, for the first 100 sources from the given database: ```graphql query { environment(id: 834) { applied { sources( filter: { database: "analytics" }, first: 100 ) { edges { node { name fqn description filePath freshness { freshnessChecked freshnessStatus } sourceName sourceDescription tests { name description testType executionInfo { lastRunStatus } } } } } } } } ``` ##### Fields When querying for `sources`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Sources object schema dbt platform | Starter, Enterprise, Enterprise+ The sources object allows you to query information about all sources in a given job. ##### Arguments When querying for `sources`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the sources object. ##### Example queries The database, schema, and identifier arguments are optional. This means that with this endpoint you can: * Find a specific source by providing `..` * Find all of the sources in a database and/or schema by providing `` and/or `` ###### Finding sources by their database, schema, and identifier The example query below finds a source by its unique database, schema, and identifier. ```graphql { job(id: 123) { sources( database: "analytics" schema: "analytics" identifier: "dim_customers" first: 10 after: "{somePaginationCursorValue}" ) { uniqueId paginationCursor } } } ``` ###### Finding sources by their schema The example query below finds all sources in this schema and their respective states (pass, error, fail). ```graphql { job(id: 123) { sources(schema: "analytics", first: 10, after: "{somePaginationCursorValue}") { uniqueId state paginationCursor } } } ``` ##### Fields The sources object can access the *same fields* as the [source node](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-source.md). The difference is that the sources object can output a list so instead of querying for fields for one specific source, you can query for those parameters for all sources within a jobID, database, and so on. When querying for `sources`, the following fields are available: ### Fetching data... --- ### Tags object schema dbt platform | Starter, Enterprise, Enterprise+ [Tags](https://docs.getdbt.com/reference/resource-configs/tags.md) provide a mechanism to categorize and group resources within a dbt project, enabling selective execution and management of these resources. You can query tags through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `tags` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Example query You can use the `environmentId` to return the name of all the tags in your environment: ```graphql query { environment(id: 834) { applied { tags { name } } } } ``` ##### Fields When querying for `tags`, you can use the following fields: ### Fetching data... --- ### Test object schema dbt platform | Starter, Enterprise, Enterprise+ The test object allows you to query information about a particular test. ##### Arguments When querying for a `test`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema (all possible fields you can query) of the test object. ##### Example query The example query below outputs information about a test including the state of the test result. In order of severity, the result can be one of these: "error", "fail", "warn", or "pass". ```graphql { job(id: 123) { test(uniqueId: "test.internal_analytics.not_null_metrics_id") { runId accountId projectId uniqueId name columnName state } } } ``` ##### Fields When querying for a `test`, the following fields are available: ### Fetching data... --- ### Tests object schema dbt platform | Starter, Enterprise, Enterprise+ [Tests](https://docs.getdbt.com/docs/build/data-tests.md) are assertions you make about your models and other resources in your dbt project. When you run `dbt test`, dbt will tell you if each test in your project passes or fails. You can query tests through the Discovery API to understand information about them. The [Example query](#example-query) illustrates a few fields you can query with the `tests` object and how to filter by last-known result. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `tests`, you can use the following arguments: ### Fetching data... Deprecation: `lastKnownResult` The single-value filter field `lastKnownResult: TestStatus` is deprecated. You should now use `lastKnownResults: [TestStatus]` to filter tests by one or more result statuses. If you pass both fields in the same query, `lastKnownResults` takes precedence. Refer to [Filter tests by last-known result](#filter-tests-by-last-known-result) for more information. ##### Filter tests by last-known result You can filter tests by one or more last-known result values. Use `lastKnownResults` with any combination of these `TestStatus` values: * `pass` * `fail` * `error` * `skipped` * `warn` You can still use the deprecated `lastKnownResult`, but If you pass both fields in the same query, `lastKnownResults` takes precedence. An empty list `[]`, omitting `lastKnownResults`, or passing `null` applies no filter on last-known result. All tests are returned regardless of result status. ##### Example query You can return tests whose last-known result is `error` or `fail`: ```graphql query { environment(id: 834) { applied { tests( filter: { lastKnownResults: [error, fail] }, first: 100 ) { edges { node { name model description expect resourceType testType given lastKnownResult } } } } } } ``` You can combine `lastKnownResults` with other filters on `TestAppliedFilter`, such as `testTypes`: ```graphql query { environment(id: 834) { applied { tests( filter: { testTypes: [GENERIC_DATA_TEST, SINGULAR_DATA_TEST], lastKnownResults: [fail] }, first: 100 ) { edges { node { name model description expect resourceType testType given lastKnownResult } } } } } } ``` ##### Fields When querying for `tests`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Tests object schema dbt platform | Starter, Enterprise, Enterprise+ The tests object allows you to query information about all tests in a given job. ##### Arguments When querying for `tests`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema (all possible fields you can query) of the tests object. ##### Example query The example query below finds all tests in this job and includes information about those tests. ```graphql { job(id: 123) { tests(first: 10, after: "{somePaginationCursorValue}") { runId accountId projectId uniqueId name columnName state paginationCursor } } } ``` ##### Fields When querying for `tests`, the following fields are available: ### Fetching data... --- ### Use cases and examples for the Discovery API dbt platform | Starter, Enterprise, Enterprise+ With the Discovery API, you can query the metadata in dbt to learn more about your dbt deployments and the data it generates to analyze them and make improvements. You can use the API in a variety of ways to get answers to your business questions. Below describes some of the uses of the API and is meant to give you an idea of the questions this API can help you answer. | Use case | Outcome | Example questions | | --------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | [Performance](#performance) | Identify inefficiencies in pipeline execution to reduce infrastructure costs and improve timeliness. | - What’s the latest status of each model?
- Do I need to run this model?
- How long did my DAG take to run? | | [Quality](#quality) | Monitor data source freshness and test results to resolve issues and drive trust in data. | - How fresh are my data sources?
- Which tests and models failed?
- What’s my project’s test coverage? | | [Discovery](#discovery) | Find and understand relevant datasets and semantic nodes with rich context and metadata. | - What do these tables and columns mean?
- What's the full data lineage at a model level?
- Which metrics can I query? | | [Governance](#governance) | Audit data development and facilitate collaboration within and between teams. | - Who is responsible for this model?
- How do I contact the model’s owner?
- Who can use this model? | | [Development](#development) | Understand dataset changes and usage and gauge impacts to inform project definition. | - How is this metric used in BI tools?
- Which nodes depend on this data source?
- How has a model changed? What impact? | #### Performance You can use the Discovery API to identify inefficiencies in pipeline execution to reduce infrastructure costs and improve timeliness. Below are example questions and queries you can run. For performance use cases, people typically query the historical or latest applied state across any part of the DAG (for example, models) using the `environment`, `modelHistoricalRuns`, or job-level endpoints. ##### How long did each model take to run? It’s helpful to understand how long it takes to build models (tables) and tests to execute during a dbt run. Longer model build times result in higher infrastructure costs and fresh data arriving later to stakeholders. Analyses like these can be in observability tools or ad-hoc queries, like in a notebook. [![Model timing visualization in dbt](/img/docs/dbt-platform/discovery-api/model-timing.png?v=2 "Model timing visualization in dbt")](#)Model timing visualization in dbt Example query with code Data teams can monitor the performance of their models, identify bottlenecks, and optimize the overall data pipeline by fetching execution details like `executionTime` and `runElapsedTime`: 1. Use latest state environment-level API to get a list of all executed models and their execution time. Then, sort the models by `executionTime` in descending order. ```graphql query AppliedModels($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { name uniqueId materializedType executionInfo { lastSuccessRunId executionTime executeStartedAt } } } } } } } ``` 2. Get the most recent 20 run results for the longest running model. Review the results of the model across runs or you can go to the job/run or commit itself to investigate further. ```graphql query ModelHistoricalRuns( $environmentId: BigInt! $uniqueId: String $lastRunCount: Int ) { environment(id: $environmentId) { applied { modelHistoricalRuns( uniqueId: $uniqueId lastRunCount: $lastRunCount ) { name runId runElapsedTime runGeneratedAt executionTime executeStartedAt executeCompletedAt status } } } } ``` 3. Use the query results to plot a graph of the longest running model’s historical run time and execution time trends. ```python # Import libraries import os import matplotlib.pyplot as plt import pandas as pd import requests # Set API key auth_token = *[SERVICE_TOKEN_HERE]* # Query the API def query_discovery_api(auth_token, gql_query, variables): response = requests.post('https://metadata.cloud.getdbt.com/graphql', headers={"authorization": "Bearer "+auth_token, "content-type": "application/json"}, json={"query": gql_query, "variables": variables}) data = response.json()['data'] return data # Get the latest run metadata for all models models_latest_metadata = query_discovery_api(auth_token, query_one, variables_query_one)['environment'] # Convert to dataframe models_df = pd.DataFrame([x['node'] for x in models_latest_metadata['applied']['models']['edges']]) # Unnest the executionInfo column models_df = pd.concat([models_df.drop(['executionInfo'], axis=1), models_df['executionInfo'].apply(pd.Series)], axis=1) # Sort the models by execution time models_df_sorted = models_df.sort_values('executionTime', ascending=False) print(models_df_sorted) # Get the uniqueId of the longest running model longest_running_model = models_df_sorted.iloc[0]['uniqueId'] # Define second query variables variables_query_two = { "environmentId": *[ENVR_ID_HERE]* "lastRunCount": 10, "uniqueId": longest_running_model } # Get the historical run metadata for the longest running model model_historical_metadata = query_discovery_api(auth_token, query_two, variables_query_two)['environment']['applied']['modelHistoricalRuns'] # Convert to dataframe model_df = pd.DataFrame(model_historical_metadata) # Filter dataframe to only successful runs model_df = model_df[model_df['status'] == 'success'] # Convert the runGeneratedAt, executeStartedAt, and executeCompletedAt columns to datetime model_df['runGeneratedAt'] = pd.to_datetime(model_df['runGeneratedAt']) model_df['executeStartedAt'] = pd.to_datetime(model_df['executeStartedAt']) model_df['executeCompletedAt'] = pd.to_datetime(model_df['executeCompletedAt']) # Plot the runElapsedTime over time plt.plot(model_df['runGeneratedAt'], model_df['runElapsedTime']) plt.title('Run Elapsed Time') plt.show() # # Plot the executionTime over time plt.plot(model_df['executeStartedAt'], model_df['executionTime']) plt.title(model_df['name'].iloc[0]+" Execution Time") plt.show() ``` Plotting examples: [![The plot of runElapsedTime over time](/img/docs/dbt-platform/discovery-api/plot-of-runelapsedtime.png?v=2 "The plot of runElapsedTime over time")](#)The plot of runElapsedTime over time [![The plot of executionTime over time](/img/docs/dbt-platform/discovery-api/plot-of-executiontime.png?v=2 "The plot of executionTime over time")](#)The plot of executionTime over time ##### What’s the latest state of each model? The Discovery API provides information about the applied state of models and how they arrived in that state. You can retrieve the status information from the most recent run and most recent successful run (execution) from the `environment` endpoint and dive into historical runs using job-based and `modelByEnvironment` endpoints. Example query The API returns full identifier information (`database.schema.alias`) and the `executionInfo` for both the most recent run and most recent successful run from the database: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId compiledCode database schema alias materializedType executionInfo { executeCompletedAt lastJobDefinitionId lastRunGeneratedAt lastRunId lastRunStatus lastRunError lastSuccessJobDefinitionId runGeneratedAt lastSuccessRunId } } } } } } } ``` ##### What happened with my job run? You can query the metadata at the job level to review results for specific runs. This is helpful for historical analysis of deployment performance or optimizing particular jobs. Example query Deprecated example: ```graphql query ($jobId: Int!, $runId: Int!) { models(jobId: $jobId, runId: $runId) { name status tests { name status } } } ``` New example: ```graphql query ($jobId: BigInt!, $runId: BigInt!) { job(id: $jobId, runId: $runId) { models(first: 10, after: "{somePaginationCursorValue}") { name status tests { name status } paginationCursor } } } ``` For jobs with many models, paginate with `first`, `after`, and `paginationCursor`. Refer to [Job-based queries](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md#job-based-queries) in the Discovery API querying guide. ```graphql query JobModelsPage($jobId: BigInt!, $runId: BigInt, $first: Int!, $after: String) { job(id: $jobId, runId: $runId) { models(first: $first, after: $after) { uniqueId name status paginationCursor } } } ``` ##### What’s changed since the last run? Unnecessary runs incur higher infrastructure costs and load on the data team and their systems. A model doesn’t need to be run if it’s a view and there's no code change since the last run, or if it’s a table/incremental with no code change since last run and source data has not been updated since the last run. Example query With the API, you can compare the `rawCode` between the definition and applied state, and review when the sources were last loaded (source `maxLoadedAt` relative to model `executeCompletedAt`) given the `materializedType` of the model: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models( first: $first filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" } ) { edges { node { rawCode ancestors(types: [Source]) { ... on SourceAppliedStateNestedNode { freshness { maxLoadedAt } } } executionInfo { runGeneratedAt executeCompletedAt } materializedType } } } } definition { models( first: $first filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" } ) { edges { node { rawCode runGeneratedAt materializedType } } } } } } ``` #### Quality You can use the Discovery API to monitor data source freshness and test results to diagnose and resolve issues and drive trust in data. When used with [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md), can also help with detecting, investigating, and alerting issues. Below lists example questions the API can help you answer. Below are example questions and queries you can run. For quality use cases, people typically query the historical or latest applied state, often in the upstream part of the DAG (for example, sources), using the `environment` or `environment { applied { modelHistoricalRuns } }` endpoints. ##### Which models and tests failed to run? By filtering on the latest status, you can get lists of models that failed to build and tests that failed during their most recent execution. This is helpful when diagnosing issues with the deployment that result in delayed or incorrect data. Example query with code 1. Get the latest run results across all jobs in the environment and return only the models and tests that errored/failed. ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { lastRunStatus: error }) { edges { node { name executionInfo { lastRunId } } } } tests(first: $first, filter: { status: "fail" }) { edges { node { name executionInfo { lastRunId } } } } } } } ``` 2. Review the historical execution and test failure rate (up to 20 runs) for a given model, such as a frequently used and important dataset. ```graphql query ($environmentId: BigInt!, $uniqueId: String!, $lastRunCount: Int) { environment(id: $environmentId) { applied { modelHistoricalRuns(uniqueId: $uniqueId, lastRunCount: $lastRunCount) { name executeStartedAt status tests { name status } } } } } ``` 3. Identify the runs and plot the historical trends of failure/error rates. ##### When was the data my model uses last refreshed? You can get the metadata on the latest execution for a particular model or across all models in your project. For instance, investigate when each model or snapshot that's feeding into a given model was last executed or the source or seed was last loaded to gauge the *freshness* of the data. Example query with code ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models( first: $first filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" } ) { edges { node { name ancestors(types: [Model, Source, Seed, Snapshot]) { ... on ModelAppliedStateNestedNode { name resourceType materializedType executionInfo { executeCompletedAt } } ... on SourceAppliedStateNestedNode { sourceName name resourceType freshness { maxLoadedAt } } ... on SnapshotAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } ... on SeedAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } } } } } } } } ``` ```python # Extract graph nodes from response def extract_nodes(data): models = [] sources = [] groups = [] for model_edge in data["applied"]["models"]["edges"]: models.append(model_edge["node"]) for source_edge in data["applied"]["sources"]["edges"]: sources.append(source_edge["node"]) for group_edge in data["definition"]["groups"]["edges"]: groups.append(group_edge["node"]) models_df = pd.DataFrame(models) sources_df = pd.DataFrame(sources) groups_df = pd.DataFrame(groups) return models_df, sources_df, groups_df # Construct a lineage graph with freshness info def create_freshness_graph(models_df, sources_df): G = nx.DiGraph() current_time = datetime.now(timezone.utc) for _, model in models_df.iterrows(): max_freshness = pd.Timedelta.min if "meta" in models_df.columns: freshness_sla = model["meta"]["freshness_sla"] else: freshness_sla = None if model["executionInfo"]["executeCompletedAt"] is not None: model_freshness = current_time - pd.Timestamp(model["executionInfo"]["executeCompletedAt"]) for ancestor in model["ancestors"]: if ancestor["resourceType"] == "SourceAppliedStateNestedNode": ancestor_freshness = current_time - pd.Timestamp(ancestor["freshness"]['maxLoadedAt']) elif ancestor["resourceType"] == "ModelAppliedStateNestedNode": ancestor_freshness = current_time - pd.Timestamp(ancestor["executionInfo"]["executeCompletedAt"]) if ancestor_freshness > max_freshness: max_freshness = ancestor_freshness G.add_node(model["uniqueId"], name=model["name"], type="model", max_ancestor_freshness = max_freshness, freshness = model_freshness, freshness_sla=freshness_sla) for _, source in sources_df.iterrows(): if source["maxLoadedAt"] is not None: G.add_node(source["uniqueId"], name=source["name"], type="source", freshness=current_time - pd.Timestamp(source["maxLoadedAt"])) for _, model in models_df.iterrows(): for parent in model["parents"]: G.add_edge(parent["uniqueId"], model["uniqueId"]) return G ``` Graph example: [![A lineage graph with source freshness information](/img/docs/dbt-platform/discovery-api/lineage-graph-with-freshness-info.png?v=2 "A lineage graph with source freshness information")](#)A lineage graph with source freshness information ##### Are my data sources fresh? Checking [source freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) allows you to ensure that sources loaded and used in your dbt project are compliant with expectations. The API provides the latest metadata about source loading and information about the freshness check criteria. [![Source freshness page in dbt](/img/docs/dbt-platform/discovery-api/source-freshness-page.png?v=2 "Source freshness page in dbt")](#)Source freshness page in dbt Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { sources( first: $first filter: { freshnessChecked: true, database: "production" } ) { edges { node { sourceName name identifier loader freshness { freshnessJobDefinitionId freshnessRunId freshnessRunGeneratedAt freshnessStatus freshnessChecked maxLoadedAt maxLoadedAtTimeAgoInS snapshottedAt criteria { errorAfter { count period } warnAfter { count period } } } } } } } } } ``` ##### What’s the test coverage and status? [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) are an important way to ensure that your stakeholders are reviewing high-quality data. You can execute tests during a dbt run. The Discovery API provides complete test results for a given environment or job, which it represents as the `children` of a given node that’s been tested (for example, a `model`). Example query For the following example, the `parents` are the nodes (code) that's being tested and `executionInfo` describes the latest test results: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { tests(first: $first) { edges { node { name columnName parents { name resourceType } executionInfo { lastRunStatus lastRunError executeCompletedAt executionTime } } } } } } } ``` ##### How is this model contracted and versioned? To enforce the shape of a model's definition, you can define contracts on models and their columns. You can also specify model versions to keep track of discrete stages in its evolution and use the appropriate one. Example query ```graphql query { environment(id: 123) { applied { models(first: 100, filter: { access: public }) { edges { node { name latestVersion contractEnforced constraints { name type expression columns } catalog { columns { name type } } } } } } } } ``` #### Discovery You can use the Discovery API to find and understand relevant datasets and semantic nodes with rich context and metadata. Below are example questions and queries you can run. For discovery use cases, people typically query the latest applied or definition state, often in the downstream part of the DAG (for example, mart models or metrics), using the `environment` endpoint. ##### What does this dataset and its columns mean? Query the Discovery API to map a table/view in the data platform to the model in the dbt project; then, retrieve metadata about its meaning, including descriptive metadata from its YAML file and catalog information from its YAML file and the schema. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models( first: $first filter: { database: "analytics" schema: "prod" identifier: "customers" } ) { edges { node { name description tags meta catalog { columns { name description type } } } } } } } } ``` ##### What's the full data lineage at a model level? The Discovery API enables access to comprehensive model-level data lineage by exposing: * Upstream dependencies of models, including relationships to [sources](https://docs.getdbt.com/docs/build/sources.md), [seeds](https://docs.getdbt.com/docs/build/seeds.md), and [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) * Model execution metadata such as run status, execution time, and freshness * Column-level details, including tests and descriptions * References between models to reconstruct lineage across your project Example query Here's a GraphQL query example that retrieves full model-level data lineage using the Discovery API: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { name ancestors(types: [Model, Source, Seed, Snapshot]) { ... on ModelAppliedStateNestedNode { name resourceType } ... on SourceAppliedStateNestedNode { sourceName name resourceType } } } } } } } } ``` ##### Which metrics are available? You can define and query metrics using the [Semantic Layer](https://docs.getdbt.com/docs/build/about-metricflow.md), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL). Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { definition { metrics(first: $first) { edges { node { name description type formula filter tags parents { name resourceType } } } } } } } ``` #### Governance You can use the Discovery API to audit data development and facilitate collaboration within and between teams. For governance use cases, people tend to query the latest definition state, often in the downstream part of the DAG (for example, public models), using the `environment` endpoint. ##### Who is responsible for this model? You can define and surface the groups each model is associated with. Groups contain information like owner. This can help you identify which team owns certain models and who to contact about them. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { uniqueIds: ["MODEL.PROJECT.NAME"] }) { edges { node { name description resourceType access group } } } } definition { groups(first: $first) { edges { node { name resourceType models { name } ownerName ownerEmail } } } } } } ``` ##### Who can use this model? You can enable people the ability to specify the level of access for a given model. In the future, public models will function like APIs to unify project lineage and enable reuse of models using cross-project refs. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { definition { models(first: $first) { edges { node { name access } } } } } } ``` *** ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { definition { models(first: $first, filter: { access: public }) { edges { node { name } } } } } } ``` #### Development You can use the Discovery API to understand dataset changes and usage and gauge impacts to inform project definition. Below are example questions and queries you can run. For development use cases, people typically query the historical or latest definition or applied state across any part of the DAG using the `environment` endpoint. ##### How is this model or metric used in downstream tools? [Exposures](https://docs.getdbt.com/docs/build/exposures.md) provide a method to define how a model or metric is actually used in dashboards and other analytics tools and use cases. You can query an exposure’s definition to see how project nodes are used and query its upstream lineage results to understand the state of the data used in it, which powers use cases like a freshness and quality status tile. [![Embed data health tiles in your dashboards to distill trust signals for data consumers.](/img/docs/collaborate/dbt-explorer/data-tile-pass.png?v=2 "Embed data health tiles in your dashboards to distill trust signals for data consumers.")](#)Embed data health tiles in your dashboards to distill trust signals for data consumers. Example query Below is an example that reviews an exposure and the models used in it including when they were last executed. ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { exposures(first: $first) { edges { node { name description ownerName url parents { name resourceType ... on ModelAppliedStateNestedNode { executionInfo { executeCompletedAt lastRunStatus } } } } } } } } } ``` ##### How has this model changed over time? The Discovery API provides historical information about any resource in your project. For instance, you can view how a model has evolved over time (across recent runs) given changes to its shape and contents. Example query Review the differences in `compiledCode` or `columns` between runs or plot the “Approximate Size” and “Row Count” `stats` over time: ```graphql query ( $environmentId: BigInt! $uniqueId: String! $lastRunCount: Int! $withCatalog: Boolean! ) { environment(id: $environmentId) { applied { modelHistoricalRuns( uniqueId: $uniqueId lastRunCount: $lastRunCount withCatalog: $withCatalog ) { name compiledCode columns { name } stats { label value } } } } } ``` ##### Which nodes depend on this data source? dbt lineage begins with data sources. For a given source, you can look at which nodes are its children then iterate downstream to get the full list of dependencies. Currently, querying beyond 1 generation (defined as a direct parent-to-child) is not supported. To see the grandchildren of a node, you need to make two queries: one to get the node and its children, and another to get the children nodes and their children. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { sources( first: $first filter: { uniqueIds: ["SOURCE_NAME.TABLE_NAME"] } ) { edges { node { loader children { uniqueId resourceType ... on ModelAppliedStateNestedNode { database schema alias } } } } } } } } ``` #### Related docs * [Query Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) --- ## Best Practices ### Adding data-informed tests with dbt Wizard Use dbt Wizard CLI to identify meaningful test gaps, validate assumptions against current warehouse data, and write focused dbt data tests. This workflow uses the built-in `test_writer` agent. Unlike a request that names tests in advance, `test_writer` starts with project metadata and model importance, forms candidate assertions, and checks those assertions against available data before proposing YAML changes. CLI workflow The built-in `test_writer` agent is available in dbt Wizard CLI. Refer to [Use subagents with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) for availability and agent behavior. #### Prerequisites Before you begin: * [Install and configure dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). * Start dbt Wizard from a dbt project with a current `target/manifest.json`. * Configure a development connection that can query the models you want to inspect. * Build the relevant models if their development relations don't exist. * Decide whether sensitive columns or expensive relations should be excluded from profiling. Warehouse queries can consume compute and can expose data values in your terminal session. Review proposed queries and apply your organization's data-access policies. #### Choose a focused scope Start with a model or a small project area. This keeps the proposed tests reviewable and the warehouse queries bounded: ```text Use test_writer to improve test coverage for stg_customers. Inspect its existing tests and downstream importance, validate candidate assumptions against current data, and propose only high-signal tests. Do not change model SQL. ``` You can also ask dbt Wizard to find a starting point: ```text Use test_writer to find three important models with weak test coverage. Prioritize high fan-out models and models that define a business grain. Explain the ranking before writing tests. ``` State constraints in the prompt. For example, name columns that must not be queried, limit the number of models, or ask dbt Wizard to avoid custom generic tests. #### Review the coverage analysis Before it writes YAML, ask `test_writer` to explain: * Which models and columns already have tests. * Why the selected model is important to downstream resources. * What grain or business rule each candidate test represents. * Which candidates are based on code or metadata and which are inferred from data. * What warehouse query would validate each assumption. The strongest candidates protect model contracts and business behavior, not just the shape of today's sample. | Candidate test | Evidence to review | | ----------------- | ------------------------------------------------------------------------------------------------------------------- | | `not_null` | The column is required by the model grain, contract, or downstream logic, and current nulls have been investigated. | | `unique` | The column or column combination represents the intended grain, and duplicate checks support that assumption. | | `relationships` | The parent resource and key are authoritative, and orphaned values have been investigated. | | `accepted_values` | The field is a governed enumeration, not merely a list of values observed in one query. | If the data disproves a candidate assertion, investigate the discrepancy. Don't automatically add a filter or weaken the test to make it pass. #### Approve the warehouse checks `test_writer` can use warehouse previews to check assumptions such as uniqueness, nullability, relationships, and observed values. Before approving a query, confirm that: * It uses the expected target and relation. * Its selected columns comply with data-access policies. * Its filters and limits won't hide the condition the test is meant to protect. * Its expected scan and run time are reasonable. You can ask for a smaller or more targeted query before approving it: ```text Validate the proposed grain without selecting customer attributes. Query only customer_id and aggregate counts, and explain any sampling or date filter. ``` #### Review the proposed tests For each proposed YAML change, confirm that: * The test expresses an intended rule, not an accidental property of current data. * The test is attached to the correct model and column. * The YAML syntax matches the dbt version used by the project. * Existing tests aren't duplicated. * Test names, arguments, configuration, and severity follow project conventions. * The diff doesn't include unrelated documentation or model changes. The `test_writer` agent writes the tests. The normal dbt Wizard validation flow is responsible for running them. Select an appropriate [validation level](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md#choose-a-validation-level) and review any failing rows before accepting the final change. #### Investigate a failed candidate When a new test fails, keep the original assumption visible while you determine what the data means: ```text The proposed unique test on customer_id fails. Show the duplicate pattern, check whether the intended grain uses another column, and tell me whether this is a model defect, a source-data issue, or a bad test assumption. Do not modify the test until you explain the evidence. ``` Possible outcomes include fixing a model defect, documenting a source limitation, selecting the correct compound grain, or rejecting the candidate test. #### Understand the limits Data-informed test generation still requires engineering judgment: * Current data can support or disprove an assumption, but it doesn't define the business rule. * A passing test can miss future values, late-arriving data, and unqueried partitions. * `accepted_values` tests need a governed list when new legitimate values can appear. * Warehouse access, permissions, stale relations, and sampling can limit the evidence available. * High test counts can increase build time without improving meaningful coverage. * Tests don't replace model contracts, source freshness checks, monitoring, or review by a model owner. #### Related docs * [Validate dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [Use subagents with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) * [Test best practices](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) --- ### Available materializations Views and tables and incremental models, oh my! In this section we’ll start getting our hands dirty digging into the three basic materializations that ship with dbt. They are considerably less scary and more helpful than lions, tigers, or bears — although perhaps not as cute (can data be cute? We at dbt Labs think so). We’re going to define, implement, and explore: * 🔍 [**views**](https://docs.getdbt.com/docs/build/materializations.md#view) * ⚒️ [**tables**](https://docs.getdbt.com/docs/build/materializations.md#table) * 📚 [**incremental model**](https://docs.getdbt.com/docs/build/materializations.md#incremental) info 👻 There is a fourth default materialization available in dbt called [**ephemeral materialization**](https://docs.getdbt.com/docs/build/materializations.md#ephemeral). It is less broadly applicable than the other three, and better deployed for specific use cases that require weighing some tradeoffs. We chose to leave it out of this guide and focus on the three materializations that will power 99% of your modeling needs. **Views and Tables are the two basic categories** of object that we can create across warehouses. They exist natively as types of objects in the warehouse, as you can see from this screenshot of Snowflake (depending on your warehouse the interface will look a little different). **Incremental models** and other materializations types are a little bit different. They tell dbt to **construct tables in a special way**. [![Tables and views in the browser on Snowflake.](/img/best-practices/materializations/tables-and-views.png?v=2 "Tables and views in the browser on Snowflake.")](#)Tables and views in the browser on Snowflake. ##### Views * ✅ **The default materialization in dbt**. A starting project has no configurations defined for materializations, which means *everything* is by default built as a view. * 👩‍💻 **Store *only the SQL logic* of the transformation in the warehouse, *not the data***. As such, they make a great default. They build almost instantly and cost almost nothing to build. * ⏱️ Always reflect the **most up-to-date** version of the input data, as they’re run freshly every time they’re queried. * 👎 **Have to be processed every time they’re queried, so slower to return results than a table of the same data.** That also means they can cost more over time, especially if they contain intensive transformations and are queried often. ##### Tables * 🏗️ **Tables store the data itself** as opposed to views which store the query logic. This means we can pack all of the transformation compute into a single run. A view is storing a *query* in the warehouse. Even to preview that data we have to query it. A table is storing the literal rows and columns on disk. * 🏎️ Querying lets us **access that transformed data directly**, so we get better performance. Tables feel **faster and more responsive** compared to views of the same logic. * 💸 **Improves compute costs.** Compute is significantly more expensive than storage. So while tables use much more storage, it’s generally an economical tradeoff, as you only pay for the transformation compute when you build a table during a job, rather than every time you query it. * 🔍 **Ideal for models that get queried regularly**, due to the combination of these qualities. * 👎 **Limited to the source data that was available when we did our most recent run.** We’re ‘freezing’ the transformation logic into a table. So if we run a model as a table every hour, at 10:59a we still only have data up to 10a, because that was what was available in our source data when we ran the table last at 10a. Only at the next run will the newer data be included in our rebuild. ##### Incremental models * 🧱 **Incremental** models build a **table** in **pieces over time**, only adding and updating new or changed records. * 🏎️  **Builds more quickly** than a regular table of the same logic. * 🐢 **Initial runs are slow.** Typically we use incremental models on very large datasets, so building the initial table on the full dataset is time consuming and equivalent to the table materialization. * 👎 **Add complexity.** Incremental models require deeper consideration of layering and timing. * 👎 Can drift from source data over time. As we’re not processing all of the source data when we run an incremental model, extra effort is required to capture changes to historical data. ##### Comparing the materialization types | | view | table | incremental | | -------------------- | ------------------------------------ | -------------------------------------- | -------------------------------------- | | 🛠️⌛ **build time** | 💚  fastest — only stores logic | ❤️  slowest — linear to size of data | 💛  medium — builds flexible portion | | 🛠️💸 **build costs** | 💚  lowest — no data processed | ❤️  highest — all data processed | 💛  medium — some data processed | | 📊💸 **query costs** | ❤️  higher — reprocess every query | 💚  lower — data in warehouse | 💚  lower — data in warehouse | | 🍅🌱 **freshness** | 💚  best — up-to-the-minute of query | 💛  moderate — up to most recent build | 💛  moderate — up to most recent build | | 🧠🤔 **complexity** | 💚 simple - maps to warehouse object | 💚 simple - map to warehouse concept | 💛 moderate - adds logical complexity | info 🔑 **Time is money.** Notice in the above chart that the time and costs rows contain the same results. This is to highlight that when we’re talking about time in warehouses, we’re talking about compute time, which is the primary driver of costs. --- ### Best practice guides #### [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) [5 items](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) --- ### Best practices #### Putting it all together * 📊 We've walked through **creating semantic models and metrics** for basic coverage of a key business area. * 🔁 In doing so we've looked at how to **refactor a frozen rollup** into a dynamic, flexible new life in the Semantic Layer. #### Best practices * ✅ **Prefer normalization** when possible to allow MetricFlow to denormalize dynamically for end users. * ✅ Use **marts to denormalize** when needed, for instance grouping tables together into richer components, or getting measures on dimensional tables attached to a table with a time spine. * ✅ When source data is **well normalized** you can **build semantic models on top of staging models**. * ✅ **Prefer** computing values in **measures and metrics** when possible as opposed to in frozen rollups. * ❌ **Don't directly refactor the code you have in production**, build in parallel so you can audit the Semantic Layer output and deprecate old marts gracefully. #### Key commands * 🔑 Use `dbt parse` to generate a fresh semantic manifest. * 🔑 Use `dbt sl list dimensions --metrics [metric name]` to check that you're increasing dimensionality as you progress. * 🔑 Use `dbt sl query [query options]` to preview the output from your metrics as you develop. #### Next steps * 🗺️ Use these best practices to map out your team's plan to **incrementally adopt the Semantic Layer**. * 🤗 Get involved in the community and ask questions, **help craft best practices**, and share your progress in building a Semantic Layer. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. The Semantic Layer is the biggest paradigm shift thus far in the young practice of analytics engineering. It's ready to provide value right away, but is most impactful if you move your project towards increasing normalization, and allow MetricFlow to do the denormalization for you with maximum dimensionality. We will be releasing more resources soon covering implementation of the Semantic Layer in dbt with various integrated BI tools. This is just the beginning, hopefully this guide has given you a path forward for building your data platform in this new era. Refer to [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) for more information. --- ### Best practices for dbt and Unity Catalog Your Databricks dbt project should be configured after following the ["How to set up your databricks dbt project guide"](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). Now we’re ready to start building a dbt project using Unity Catalog. However, we should first consider how we want to allow dbt users to interact with our different catalogs. We recommend the following best practices to ensure the integrity of your production data: #### Isolate your Bronze (aka source) data We recommend using Unity Catalog because it allows you to reference data across your organization from any other catalog, legacy Hive metastore, external metastore, or Delta Live Table pipeline outputs. Additionally, Databricks offers the capability to [interact with external data](https://docs.databricks.com/external-data/index.html#interact-with-external-data-on-databricks) and supports query federation to many [database solutions](https://docs.databricks.com/query-federation/index.html#what-is-query-federation-for-databricks-sql). This means your dev and prod environments will have access to your source data, even if it is defined in another catalog or external data source. Raw data in your Bronze layer should be defined as dbt [sources](https://docs.getdbt.com/docs/build/sources.md) and should be read-only for all dbt interactions in both development and production. By default, we recommend that all of these inputs should be accessible by all dbt users in all dbt environments. This ensures that transformations in all environments begin with the same input data, and the results observed in development will be replicated when that code is deployed. That being said, there are times when your company’s data governance requirements necessitate using multiple workspaces or data catalogs depending on the environment. If you have different data catalogs/schemas for your source data depending on your environment, you can use the [target.name](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md#use-targetname-to-change-your-source-database) to change the data catalog/schema you’re pulling from depending on the environment. If you use multiple Databricks workspaces to isolate development from production, you can use dbt’s [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) in your connection config strings to reference multiple workspaces from one dbt project. You can also do the same thing for your SQL warehouse so you can have different sizes based on your environments. To do so, use dbt's [environment variable syntax](https://docs.getdbt.com/docs/build/environment-variables.md#special-environment-variables) for Server Hostname of your Databricks workspace URL and HTTP Path for the SQL warehouse in your connection settings. Note that Server Hostname still needs to appear to be a valid domain name to pass validation checks, so you will need to hard-code the domain suffix on the URL, eg `{{env_var('DBT_HOSTNAME')}}.cloud.databricks.com` and the path prefix for your warehouses, eg `/sql/1.0/warehouses/{{env_var('DBT_HTTP_PATH')}}`. [![Using environment variable syntax in connection configs](/img/guides/databricks-guides/databricks-connection-env-vars.png?v=2 "Using environment variable syntax in connection configs")](#)Using environment variable syntax in connection configs When you create environments in dbt, you can assign environment variables to populate the connection information dynamically. Don’t forget to make sure the tokens you use in the credentials for those environments were generated from the associated workspace. [![Defining default environment variable values](/img/guides/databricks-guides/databricks-env-variables.png?v=2 "Defining default environment variable values")](#)Defining default environment variable values #### Access Control For granting access to data consumers, use dbt’s [grants config](https://docs.getdbt.com/reference/resource-configs/grants.md) to apply permissions to database objects generated by dbt models. This lets you configure grants as a structured dictionary rather than writing all the SQL yourself and lets dbt take the most efficient path to apply those grants. As for permissions to run dbt and read non-consumer-facing data sources, the table below summarizes an access model. Effectively, all developers should get no more than read access on the prod catalog and write access in the dev catalog. When using dbt, schema creation is taken care of for you; unlike traditional data warehousing workflows, you do not need to manually create any Unity Catalog assets other than the top-level catalogs. The **prod** service principal should have “read” access to raw source data, and “write” access to the prod catalog. If you add a **test** catalog and associated dbt environment, you should create a dedicated service principal. The test service principal should have *read* on raw source data, and *write* on the **test** catalog but no permissions on the prod or dev catalogs. A dedicated test environment should be used for [CI testing](https://www.getdbt.com/blog/adopting-ci-cd-with-dbt-cloud/) only. **Table-level grants:** | | Source Data | Development catalog | Production catalog | Test catalog | | ---------------------------- | ----------- | ------------------- | ------------------ | --------------- | | developers | select | select & modify | select or none | none | | production service principal | select | none | select & modify | none | | Test service principal | select | none | none | select & modify | **Schema-level grants:** | | Source Data | Development catalog | Production catalog | Test catalog | | ---------------------------- | ----------- | --------------------------------- | -------------------------------- | -------------------------------- | | developers | use | use, create schema, table, & view | use or none | none | | production service principal | use | none | use, create schema, table & view | none | | Test service principal | use | none | none | use, create schema, table & view | #### Next steps Ready to start transforming your Unity Catalog datasets with dbt? Check out the resources below for guides, tips, and best practices: * [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [Self-paced dbt fundamentals training course](https://learn.getdbt.com/courses/dbt-fundamentals) * [Customizing CI/CD](https://docs.getdbt.com/guides/custom-cicd-pipelines.md) * [Debugging errors](https://docs.getdbt.com/guides/debug-errors.md) * [Writing custom generic tests](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) * [dbt packages hub](https://hub.getdbt.com/) --- ### Best practices for materializations First, let’s consider some properties of various levels of our dbt project and materializations. * 🔍 **Views** return the freshest, real-time state of their input data when they’re queried, this makes them ideal as **building blocks** for larger models. * 🧶  When we’re building a model that stitches lots of other models together, we don’t want to worry about all those models having different states of freshness because they were built into tables at different times. We want all those inputs to give us all the underlying source data available. * 🤏 **Views** are also great for **small datasets** with minimally intensive logic that we want **near realtime** access to. * 🛠️ **Tables** are the **most performant** materialization, as they just return the transformed data when they’re queried, with no need to reprocess it. * 📊  This makes tables great for **things end users touch**, like a mart that services a popular dashboard. * 💪 Tables are also ideal for **frequently used, compute intensive** transformations. Making a table allows us to ‘freeze’ those transformations in place. * 📚  **Incremental models** are useful for the **same purposes as tables**, they just enable us to build them on larger datasets, so they can be **built** *and* **accessed** in a **performant** way. ##### Project-level configuration Keeping these principles in mind, we can applying these materializations to a project. Earlier we looked at how to configure an individual model's materializations. In practice though, we'll want to set materializations at the folder level, and use individual model configs to override those as needed. This will keep our code DRY and avoid repeating the same config blocks in every model. * 📂  In the `dbt_project.yml` we have a `models:` section (by default at the bottom of the file) we can use define various **configurations for entire directories**. * ⚙️  These are the **same configs that are passed to a `{{ config() }}` block** for individual models, but they get set for *every model in that directory and any subdirectories nested within it*. * ➕  We demarcate between a folder name and a configuration by using a `+`, so `marketing`, `paid_ads`, and `google` below are folder names, whereas **`+materialized` is a configuration** being applied to those folder and all folders nested below them. * ⛲  Configurations set in this way **cascade**, the **more specific scope** is the one that will be set. * 👇🏻  In the example below, all the models in the `marketing` and `paid_ads` folders would be views, but the `google` sub folder would be **tables.** ```yaml models: jaffle_shop: marketing: +materialized: view paid_ads: google: +materialized: table ``` ##### Staging views We’ll start off simple with staging models. Lets consider some aspects of staging models to determine the ideal materialization strategy: * 🙅‍♀️ Staging models are **rarely accessed** directly by our **end users.** * 🧱 They need to be always up-to-date and in sync with our source data as a **building blocks** for later models * 🔍  It’s clear we’ll want to keep our **staging models as views**. * 👍  Since views are the **default materialization** in dbt, we don’t *have* to do any specific configuration for this. * 💎  Still, for clarity, it’s a **good idea** to go ahead and **specify the configuration** to be explicit. We’ll want to make sure our `dbt_project.yml` looks like this: ```yaml models: jaffle_shop: staging: +materialized: view ``` ##### Intermediate models in larger projects The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example project uses a staging → marts flow and does not include an `intermediate/` folder. In larger projects, intermediate models often sit between staging and marts, breaking up complex transformations into manageable pieces: * 🚫 Intermediate models are not accessed directly by end users. They exist to simplify mart logic. * 🧩 They serve as building blocks that get referenced by marts or other intermediate models. * 👻 This makes them ideal candidates for ephemeral materialization, which doesn't create objects in your warehouse. Ephemeral models are interpolated as CTE into the models that reference them. This keeps your warehouse clean and avoids cluttering it with models that aren't meant for direct querying: ```yaml models: jaffle_shop: staging: +materialized: view intermediate: +materialized: ephemeral marts: +materialized: table ``` When to avoid ephemeral models Ephemeral models can make troubleshooting more difficult since they don't exist as queryable objects. If you need to inspect intermediate results during development, consider materializing them as views in a custom schema with restricted permissions instead. This gives you visibility while keeping them separate from production models. For more details on intermediate model patterns, refer to [How we structure our dbt projects: Intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md). ##### Table and incremental marts As we’ve learned, views store only the logic of the transformation in the warehouse, so our runs take only a couple seconds per model (or less). What happens when we go to query the data though? [![Long query time from Snowflake](/img/best-practices/materializations/snowflake-query-timing.png?v=2 "Long query time from Snowflake")](#)Long query time from Snowflake Our marts are slow to query! Let’s contrast the same aspects of marts that we considered for staging models to assess the best materialization strategy: * 📊  Marts are **frequently accessed directly by our end users**, and need to be **performant.** * ⌛  Can often **function with intermittently refreshed data**, end user decision making in many domains is **fine with hourly or daily data.** * 🛠️  Given the above properties we’ve got a great use case for **building the data itself** into the warehouse, not the logic. In other words, **a table**. * ❓ The only decision we need to make with our marts is whether we can **process the whole table at once or do we need to do it in chunks**, that is, are we going to use the `table` materialization or `incremental`. info 🔑 **Golden Rule of Materializations** Start with models as views, when they take too long to query, make them tables, when the tables take too long to build, make them incremental. --- ### Best practices for workflows This page contains the collective wisdom of experienced users of dbt on how to best use it in your analytics work. Observing these best practices will help your analytics team work as effectively as possible, while implementing the pro-tips will add some polish to your dbt projects! #### Best practice workflows ##### Version control your dbt project All dbt projects should be managed in version control. Git branches should be created to manage development of new features and bug fixes. All code changes should be reviewed by a colleague (or yourself) in a Pull Request prior to merging into your production branch, such as `main`. Git guide We've codified our best practices in Git, in our [Git guide](https://github.com/dbt-labs/corp/blob/main/git-guide.md). ##### Use separate development and production environments dbt makes it easy to maintain separate production and development environments through the use of targets within a profile. We recommend using a `dev` target when running dbt from your command line and only running against a `prod` target when running from a production deployment. You can read more [about managing environments here](https://docs.getdbt.com/docs/environments-in-dbt.md). ##### Use a style guide for your project SQL styles, field naming conventions, and other rules for your dbt project should be codified, especially on projects where multiple dbt users are writing code. Our style guide We've made our [style guide](https://docs.getdbt.com/best-practices/how-we-style/0-how-we-style-our-dbt-projects.md) public – these can act as a good starting point for your own style guide. #### Best practices in dbt projects ##### Use the ref function The [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function is what makes dbt so powerful! Using the `ref` function allows dbt to infer dependencies, ensuring that models are built in the correct order. It also ensures that your current model selects from upstream tables and views in the same environment that you're working in. Always use the `ref` function when selecting from another model, rather than using the direct relation reference (e.g. `my_schema.my_table`). ##### Limit references to raw data Your dbt project will depend on raw data stored in your database. Since this data is normally loaded by third parties, the structure of it can change over time – tables and columns may be added, removed, or renamed. When this happens, it is easier to update models if raw data is only referenced in one place. Using sources for raw data references We recommend defining your raw data as [sources](https://docs.getdbt.com/docs/build/sources.md), and selecting from the source rather than using the direct relation reference. Our dbt projects don't contain any direct relation references in any models. ##### Rename and recast fields once Raw data is generally stored in a source-conformed structure, that is, following the schema and naming conventions that the source defines. Not only will this structure differ between different sources, it is also likely to differ from the naming conventions you wish to use for analytics. The first layer of transformations in a dbt project should: * Select from only one source * Rename fields and tables to fit the conventions you wish to use within your project, for example, ensuring all timestamps are named `_at`. These conventions should be declared in your project coding conventions (see above). * Recast fields into the correct data type, for example, changing dates into UTC and prices into dollar amounts. All subsequent data models should be built on top of these models, reducing the amount of duplicated code. What happened to base models? Earlier versions of this documentation recommended implementing “base models” as the first layer of transformation, and gave advice on the SQL within these models. We realized that while the reasons behind this convention were valid, the specific advice around "base models" represented an opinion, so we moved it out of the official documentation. You can instead find our opinions on [how we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). ##### Break complex models up into smaller pieces Complex models often include multiple Common Table Expressions (CTEs). In dbt, you can instead separate these CTEs into separate models that build on top of each other. It is often a good idea to break up complex models when: * A CTE is duplicated across two models. Breaking the CTE into a separate model allows you to reference the model from any number of downstream models, reducing duplicated code. * A CTE changes the grain of a the data it selects from. It's often useful to test any transformations that change the grain (as in, what one record represents) of your data. Breaking a CTE into a separate model allows you to test this transformation independently of a larger model. * The SQL in a query contains many lines. Breaking CTEs into separate models can reduce the cognitive load when another dbt user (or your future self) is looking at the code. ##### Group your models in directories Within your `models/` directory, you can have any number of nested subdirectories. We leverage directories heavily, since using a nested structure within directories makes it easier to: * Configure groups of models, by specifying configurations in your `dbt_project.yml` file. * Run subsections of your DAG, by using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). * Communicate modeling steps to collaborators * Create conventions around the allowed upstream dependencies of a model, for example, "models in the `marts` directory can only select from other models in the `marts` directory, or from models in the `staging` directory". ##### Add tests to your models dbt provides a framework to test assumptions about the results generated by a model. Adding tests to a project helps provide assurance that both: * your SQL is transforming data in the way you expect, and * your source data contains the values you expect Recommended tests Our [style guide](https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md) recommends that at a minimum, every model should have a primary key that is tested to ensure it is unique, and not null. ##### Consider the information architecture of your data warehouse When a user connects to a data warehouse via a SQL client, they often rely on the names of schemas, relations, and columns, to understand the data they are presented with. To improve the information architecture of a data warehouse, we: * Use [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) to separate relations into logical groupings, or hide intermediate models in a separate schema. Generally, these custom schemas align with the directories we use to group our models, and are configured from the `dbt_project.yml` file. For the recommended folder-to-schema mapping, refer to [Mapping project folders to warehouse schemas](https://docs.getdbt.com/best-practices/how-we-structure/6-mapping-folders-to-warehouse.md). * Use prefixes in table names (for example, `stg_`, `fct_` and `dim_`) to indicate which relations should be queried by end users. ##### Choose your materializations wisely [materialization](https://docs.getdbt.com/docs/build/materializations.md) determine the way models are built through configuration. As a general rule: * Views are faster to build, but slower to query compared to tables. * Incremental models provide the same query performance as tables, are faster to build compared to the table materialization, however they introduce complexity into a project. We often: * Use views by default * Use ephemeral models for lightweight transformations that shouldn't be exposed to end-users * Use tables for models that are queried by BI tools * Use tables for models that have multiple descendants * Use incremental models when the build time for table models exceeds an acceptable threshold #### Pro-tips for workflows ##### Use the model selection syntax when running locally When developing, it often makes sense to only run the model you are actively working on and any downstream models. You can choose which models to run by using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). ##### Run only modified models to test changes ("slim CI") To merge code changes with confidence, you want to know that those changes will not cause breakages elsewhere in your project. For that reason, we recommend running models and tests in a sandboxed environment, separated from your production data, as an automatic check in your git workflow. (If you use GitHub and dbt, read about [how to set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md). At the same time, it costs time (and money) to run and test all the models in your project. This inefficiency feels especially painful if your PR only proposes changes to a handful of models. By comparing to artifacts from a previous production run, dbt can determine which models are modified and build them on top of their unmodified parents. ```bash dbt run -s state:modified+ --defer --state path/to/prod/artifacts dbt test -s state:modified+ --defer --state path/to/prod/artifacts ``` By comparing to artifacts from a previous production run, dbt can determine model and test result statuses. * `result:fail` * `result:error` * `result:warn` * `result:success` * `result:skipped` * `result:pass` For smarter reruns, use the `result:` selector instead of manually overriding dbt commands with the models in scope. ```bash dbt run --select state:modified+ result:error+ --defer --state path/to/prod/artifacts ``` * Rerun all my erroneous models AND run changes I made concurrently that may relate to the erroneous models for downstream use ```bash dbt build --select state:modified+ result:error+ --defer --state path/to/prod/artifacts ``` * Rerun and retest all my erroneous models AND run changes I made concurrently that may relate to the erroneous models for downstream use ```bash dbt build --select state:modified+ result:error+ result:fail+ --defer --state path/to/prod/artifacts ``` * Rerun all my erroneous models AND all my failed tests * Rerun all my erroneous models AND run changes I made concurrently that may relate to the erroneous models for downstream use * There's a failed test that's unrelated to modified or error nodes(think: source test that needs to refresh a data load in order to pass) ```bash dbt test --select result:fail --exclude --defer --state path/to/prod/artifacts ``` * Rerun all my failed tests and exclude tests that I know will still fail * This can apply to updates in source data during the "EL" process that need to be rerun after they are refreshed > Note: If you're using the `--state target/` flag, `result:error` and `result:fail` flags can only be selected concurrently(in the same command) if using the `dbt build` command. `dbt test` will overwrite the `run_results.json` from `dbt run` in a previous command invocation. Only supported by v1.1 or newer. By comparing to a `sources.json` artifact from a previous production run to a current `sources.json` artifact, dbt can determine which sources are fresher and run downstream models based on them. ```bash # job 1 dbt source freshness # must be run to get previous state ``` Test all my sources that are fresher than the previous run, and run and test all models downstream of them: ```bash # job 2 dbt source freshness # must be run again to compare current to previous state dbt build --select source_status:fresher+ --state path/to/prod/artifacts ``` To learn more, read the docs on [state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). #### Pro-tips for dbt Projects ##### Limit the data processed when in development In a development environment, faster run times allow you to iterate your code more quickly. We frequently speed up our runs by using a pattern that limits data based on the [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) name: ```sql select * from event_tracking.events {% if target.name == 'dev' %} where created_at >= dateadd('day', -3, current_date) {% endif %} ``` Another option is to use the [environment variable `DBT_CLOUD_INVOCATION_CONTEXT`](https://docs.getdbt.com/docs/build/environment-variables.md#dbt-platform-context). This environment variable provides metadata about the execution context of dbt. The possible values are `prod`, `dev`, `staging`, and `ci`. **Example usage**: ```text {% if env_var('DBT_CLOUD_INVOCATION_CONTEXT') != 'prod' %} ``` ##### Use grants to manage privileges on objects that dbt creates Use `grants` in [resource configs](https://docs.getdbt.com/reference/resource-configs/grants.md) to ensure that permissions are applied to the objects created by dbt. By codifying these grant statements, you can version control and repeatably apply these permissions. ##### Separate source-centric and business-centric transformations When modeling data, we frequently find there are two stages: 1. Source-centric transformations to transform data from different sources into a consistent structure, for example, re-aliasing and recasting columns, or unioning, joining or deduplicating source data to ensure your model has the correct grain; and 2. Business-centric transformations that transform data into models that represent entities and processes relevant to your business, or implement business definitions in SQL. We find it most useful to separate these two types of transformations into different models, to make the distinction between source-centric and business-centric logic clear. ##### Managing whitespace generated by Jinja If you're using macros or other pieces of Jinja in your models, your compiled SQL (found in the `target/compiled` directory) may contain unwanted whitespace. Check out the [Jinja documentation](http://jinja.pocoo.org/docs/2.10/templates/#whitespace-control) to learn how to control generated whitespace. #### Related docs * [Updating our permissioning guidelines: grants as configs in dbt v1.2](https://docs.getdbt.com/blog/configuring-grants) --- ### Building metrics tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). #### How to build metrics * 💹 We'll start with one of the most important metrics for any business: **revenue**. * 📖 For now, our metric for revenue will be **defined as the sum of order totals excluding tax**. #### Defining revenue * 🔢 Metrics have four basic properties: * `name:` We'll use 'revenue' to reference this metric. * `description:` For documentation. * `label:` The display name for the metric in downstream tools. * `type:` one of `simple`, `ratio`, or `derived`. * 🎛️ Each type has different `type_params`. * 🛠️ We'll build a **simple metric** first to get the hang of it, and move on to ratio and derived metrics later. * 📏 Simple metrics are built on a **single measure defined as a type parameter**. * 🔜 Defining **measures as their own distinct component** on semantic models is critical to allowing the **flexibility of more advanced metrics**, though simple metrics act mainly as **pass-through that provide filtering** and labeling options. models/marts/orders.yml ```yml metrics: - name: revenue description: Sum of the order total. label: Revenue type: simple type_params: measure: order_total ``` #### Query your metric You can use the dbt platform CLI for metric validation or queries during development, via the `dbt sl` set of subcommands. Here are some useful examples: ```bash dbt sl query revenue --group-by metric_time__month dbt sl list dimensions --metrics revenue # list all dimensions available for the revenue metric ``` * It's best practice any time we're updating our Semantic Layer code to run `dbt parse` to update our development semantic manifest. * `dbt sl query` is not how you would typically use the tool in production, that's handled by the dbt Semantic Layer's features. It's available for testing results of various metric queries in development, exactly as we're using it now. * Note the structure of the above query. We select the metric(s) we want and the dimensions to group them by — we use dunders (double underscores e.g.`metric_time__[time bucket]`) to designate time dimensions or other non-unique dimensions that need a specified entity path to resolve (e.g. if you have an orders location dimension and an employee location dimension both named 'location' you would need dunders to specify `orders__location` or `employee__location`). --- ### Building Semantic Layer definitions with dbt Wizard Use dbt Wizard to turn a business question or an existing dbt model into version-compatible Semantic Layer definitions. dbt Wizard inspects model grain, columns, lineage, and nearby YAML before it proposes entities, dimensions, and metrics. This workflow uses your project files and available dbt metadata to guide the work. You can also add the `building-dbt-semantic-layer` skill from the [dbt Agent Skills repository](https://github.com/dbt-labs/dbt-agent-skills) for reusable Semantic Layer guidance. The prompts on this page work the same way in dbt platform. #### Before you begin Identify the following information before you ask dbt Wizard to edit files: * The business question or metric you want to support. * The model, or set of candidate models, that contains the relevant data. * The grain of each model, when you already know it. * Any naming, certification, or ownership conventions your team follows. If you don't know which model to use, start with the business question. For example: ```text We need to report gross revenue and order count by customer segment and month. Find the best models to build on, explain your choices, and propose the Semantic Layer definitions before editing files. ``` #### Ask Wizard to plan the definitions Give dbt Wizard a specific model when you know the starting point: ```text Build Semantic Layer definitions for fct_orders. First determine the dbt version and inspect the model grain, columns, lineage, and existing YAML. Propose the entities, dimensions, and metrics before making changes. Include total revenue, order count, and a monthly time dimension. ``` dbt Wizard should complete these planning steps before editing: 1. Determine the dbt version from the installed binary, `require-dbt-version`, or manifest metadata. 2. Inspect the target model's SQL, columns, and upstream and downstream lineage. 3. Check nearby YAML and existing semantic definitions so the new content follows project conventions. 4. Identify the model grain and at least one primary entity. 5. Propose time and categorical dimensions and metrics whose aggregations are meaningful for the underlying data. Ask dbt Wizard to sample data only when names, types, and SQL don't establish the meaning of a column. Review and approve the warehouse query before it runs. #### Review the generated YAML The correct YAML structure depends on the dbt version in your project. Review the generated file for the version-specific patterns in the following table. | Project version | Expected structure | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [dbt 1.12 and later](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.12.md?version=2.0#new-semantic-layer-yaml-spec), and the [dbt v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md?version=2.0) | Configure `semantic_model` on a model, annotate entities and dimensions on columns, and define metrics on the model. Don't use a top-level `semantic_models:` block for new definitions. | | [dbt 1.6 through 1.11](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.11.md?version=2.0) | Define semantic models in a top-level `semantic_models:` block and define metrics with `type_params` that reference measures. | For dbt 1.12 and later and dbt v2, a generated definition can resemble the following example: ```yaml models: - name: fct_orders semantic_model: enabled: true agg_time_dimension: ordered_at columns: - name: order_id entity: type: primary name: order - name: customer_id entity: type: foreign name: customer - name: ordered_at granularity: day dimension: type: time - name: order_status dimension: type: categorical metrics: - name: total_revenue type: simple label: Total Revenue agg: sum expr: order_total - name: order_count type: simple label: Order Count agg: count_distinct expr: order_id ``` For dbt 1.6 through 1.11, expect the top-level semantic model pattern: ```yaml semantic_models: - name: orders model: ref('fct_orders') defaults: agg_time_dimension: ordered_at entities: - name: order type: primary expr: order_id dimensions: - name: ordered_at type: time type_params: time_granularity: day measures: - name: total_revenue agg: sum expr: order_total metrics: - name: total_revenue type: simple label: Total Revenue type_params: measure: total_revenue ``` Confirm that every generated definition has a clear business meaning. A numeric column isn't automatically a useful metric, and an ID isn't automatically the correct primary entity. #### Validate the definitions Ask dbt Wizard to validate after you approve the YAML: ```text Validate the Semantic Layer changes. Parse the project, run the supported Semantic Layer validation for this environment, and report errors, warnings, and any checks you couldn't complete. Don't change the definitions to silence an error without explaining the root cause. ``` At minimum, validation should confirm that: * The project parses successfully. * Model, column, entity, measure, and metric references resolve. * The aggregate time dimension points to a declared time dimension. * The selected metric aggregation matches the model grain. * Semantic Layer validation passes when the project's dbt runtime supports it. When a validation command isn't available in the current environment, dbt Wizard should report that limitation instead of treating the work as fully validated. For a broader validation procedure, refer to [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md). #### Extend the first definitions After the initial definitions validate, continue with focused prompts rather than asking for an entire project conversion at once. For example: ```text Add a derived average order value metric using the existing revenue and order count metrics. Explain how the metric joins and time grain will behave before editing the YAML. ``` ```text Create a saved query for monthly revenue and order count grouped by customer segment. Reuse existing entities and dimensions, and validate every reference. ``` #### Related docs * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) * [Metrics overview](https://docs.getdbt.com/docs/build/metrics-overview.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) * [Use skills with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) --- ### Building semantic models tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). #### How to build a semantic model A semantic model is the Semantic Layer equivalent to a logical layer model (what historically has just been called a 'model' in dbt land). Just as configurations for models are defined on the `models:` YAML key, configurations for semantic models are housed under `semantic models:`. A key difference is that while a logical model consists of configuration and SQL or Python code, a **semantic model is defined purely via YAML**. Rather than encoding a specific dataset, a **semantic model describes relationships and expressions** that let your end users select and refine their own datasets dynamically and reliably. * ⚙️ Semantic models are **comprised of three components**: * 🫂 **entities**: these describe the **relationships** between various semantic models (think ids) * 🔪 **dimensions**: these are the columns you want to **slice, dice, group, and filter by** (think timestamps, categories, booleans). * 📏 **measures**: these are the **quantitative values you want to aggregate** * 🪣 We define **columns as being an entity, dimension, or measure**. Columns will typically fit into one of these 3 buckets, or if they're a complex aggregation expression, they might constitute a metric. #### Defining orders Let's zoom in on how we might define an *orders* semantic model. * 📗 We define it as a **YAML dictionary in the `semantic_models` list**. * 📑 It will have a **name, entities list, dimensions list, and measures list**. * ⏬ We recommend defining them **in this order consistently** as a style best practice. models/marts/orders.yml ```yaml semantic_models: - name: orders entities: ... # we'll define these later dimensions: ... # we'll define these later measures: ... # we'll define these later ``` * Next we'll point to the corresponding logical model by supplying a [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) in the `model:` property, and a `description` for documentation. models/marts/orders.yml ```yml semantic_models: - name: orders description: | Model containing order data. The grain of the table is the order id. model: ref('stg_orders') entities: ... dimensions: ... measures: ... ``` #### Establishing our entities * 🫂 Entities are the **objects and concepts** in our data that *have* dimensions and measures. You can think of them as the **nouns** of our project, the **spines** of our queries that we may want to aggregate by, or simply the **join keys**. * 🔀 Entities help MetricFlow understand **how various semantic models relate to one another**. * ⛓️ Unlike many other semantic layers, in MetricFlow **we do not need to describe joins explicitly**, instead the **relationships are implicitly described by entities**. * 1️⃣ Each semantic model should have **one primary entity** defined for itself, and **any number of foreign entities** for other semantic models it may join to. * 🫂 Entities require a **name and type** * 🔑 Types available are **primary**, **foreign**, **unique** or **natural** — we'll be focused on the first two for now, but you can [read more about unique and natural keys](https://docs.getdbt.com/docs/build/entities.md#entity-types). ##### Entities in action If we look at an example staging model for orders, we see that it has 3 id columns, so we'll need three entities. models/staging/stg\_orders.sql ```sql renamed as ( select ---------- ids id as order_id, store_id as location_id, customer as customer_id, ---------- properties (order_total / 100.0) as order_total, (tax_paid / 100.0) as tax_paid, ---------- timestamps ordered_at from source ``` * 👉 We add them with a **`name`, `type`, and optional `expr`** (expression). The expression can be any valid SQL expression on your platform. * 📛 If you **don't add an expression**, MetricFlow will **assume the name is equal to the column name** in the underlying logical model. * 👍 Our best practices pattern is to, whenever possible, provide a `name` that is the singular form of the subject or grain of the table, and use `expr` to specify the precise column name (with `_id` etc). This will let us write **more readable metrics** on top of these semantic models. For example, we'll use `location` instead of `location_id`. models/marts/orders.yml ```yml semantic_models: - name: orders ... entities: # we use the column for the name here because order is a reserved word in SQL - name: order_id type: primary - name: location type: foreign expr: location_id - name: customer type: foreign expr: customer_id dimensions: ... measures: ... ``` #### Defining our dimensions * 🧮 Dimensions are the columns that we want to **filter and group by**, **the adjectives of our project**. They come in three types: * **categorical** * **time** * slowly changing dimensions — [these are covered in the documentation](https://docs.getdbt.com/docs/build/dimensions.md#scd-type-ii), and a little more complex. To focus on building your mental models of MetricFlow's fundamentals, we won't be using SCDs in this guide. * ➕ We're **not limited to existing columns**, we can use the `expr` property to add simple computations in our dimensions. * 📛 Categorical dimensions are the simplest, they simply require a `name` and `type` (type being categorical). **If the `name` property matches the name of the dimension column**, that's it, you're done. If you want or need to use a `name` other than the column name, or do some filtering or computation, **you can supply an optional `expr` property** to evaluate for the dimension. ##### Dimensions in action * 👀 Let's look at our staging model again and see what fields we have available. models/staging/stg\_orders.sql ```sql select ---------- ids -> entities id as order_id, store_id as location_id, customer as customer_id, ---------- numerics -> measures (order_total / 100.0) as order_total, (tax_paid / 100.0) as tax_paid, ---------- timestamps -> dimensions ordered_at from source ``` * ⏰ For now the only dimension to add is a **time dimension**: `ordered_at`. * 🕰️ At least one **primary time dimension** is **required** for any semantic models that **have measures**. * 1️⃣ We denote this with the `is_primary` property, or if there is only a one-time dimension supplied it is primary by default. Below we only have `ordered_at` as a timestamp so we don't need to specify anything except the *minimum granularity* we're bucketing to (in this case, day). By this we mean that we're not going to be looking at orders at a finer granularity than a day. models/marts/orders.yml ```yml dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) type: time type_params: time_granularity: day ``` tip **Dimensional models**. You may have some models that do not contain measures, just dimensional data that enriches other facts. That's totally fine, a semantic model does not require dimensions or measures, it just needs a primary entity, and if you do have measures, a primary time dimension. We'll discuss an alternate situation, dimensional tables that have static numeric values like supply costs or tax rates but no time dimensions, later in the Guide. * 🔢 We can also **make a dimension out of a numeric column** that would typically be a measure. * 🪣 Using `expr` we can **create buckets of values that we label** for our dimension. We'll add one of these in for labeling 'large orders' as any order totals over $50. models/marts/orders.yml ```yml dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) type: time type_params: time_granularity: day - name: is_large_order type: categorical expr: case when order_total > 50 then true else false end ``` #### Making our measures * 📏 Measures are the final component of a semantic model. They describe the **numeric values that we want to aggregate**. * 🧱 Measures form **the building blocks of metrics**, with entities and dimensions helping us combine, group, and filter those metrics correctly. * 🏃 You can think of them as something like the **verbs of a semantic model**. ##### Measures in action * 👀 Let's look at **our staging model** one last time and see what **fields we want to measure**. models/staging/stg\_orders.sql ```sql select ---------- ids -> entities id as order_id, store_id as location_id, customer as customer_id, ---------- numerics -> measures (order_total / 100.0) as order_total, (tax_paid / 100.0) as tax_paid, ---------- timestamps -> dimensions ordered_at from source ``` * ➕ Here `order_total` and `tax paid` are the **columns we want as measures**. * 📝 We can describe them via the code below, specifying a **name, description, aggregation, and expression**. * 👍 As before MetricFlow will default to the **name being the name of a column when no expression is supplied**. * 🧮 [Many different aggregations](https://docs.getdbt.com/docs/build/measures.md#aggregation) are available to us. Here we just want sums. models/marts/orders.yml ```yml measures: - name: order_total description: The total amount for each order including taxes. agg: sum - name: tax_paid description: The total tax paid on each order. agg: sum ``` * 🆕 We can also **create new measures using expressions**, for instance adding a count of individual orders as below. models/marts/orders.yml ```yml - name: order_count description: The count of individual orders. expr: 1 agg: sum ``` #### Reviewing our work Our completed code will look like this, our first semantic model! Here are two examples showing different organizational approaches:  Co-located approach models/marts/orders.yml ```yml semantic_models: - name: orders defaults: agg_time_dimension: ordered_at description: | Order fact table. This table is at the order grain with one row per order. model: ref('stg_orders') entities: - name: order_id type: primary - name: location type: foreign expr: location_id - name: customer type: foreign expr: customer_id dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) # use date_trunc(ordered_at, DAY) if using BigQuery type: time type_params: time_granularity: day - name: is_large_order type: categorical expr: case when order_total > 50 then true else false end measures: - name: order_total description: The total revenue for each order. agg: sum - name: order_count description: The count of individual orders. expr: 1 agg: sum - name: tax_paid description: The total tax paid on each order. agg: sum ```  Parallel sub-folder approach models/semantic\_models/sem\_orders.yml ```yml semantic_models: - name: orders defaults: agg_time_dimension: ordered_at description: | Order fact table. This table is at the order grain with one row per order. model: ref('stg_orders') entities: - name: order_id type: primary - name: location type: foreign expr: location_id - name: customer type: foreign expr: customer_id dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) # use date_trunc(ordered_at, DAY) if using BigQuery type: time type_params: time_granularity: day - name: is_large_order type: categorical expr: case when order_total > 50 then true else false end measures: - name: order_total description: The total revenue for each order. agg: sum - name: order_count description: The count of individual orders. expr: 1 agg: sum - name: tax_paid description: The total tax paid on each order. agg: sum ``` As you can see, the content of the semantic model is identical in both approaches. The key differences are: 1. **File location** * Co-located approach: `models/marts/orders.yml` * Parallel sub-folder approach: `models/semantic_models/sem_orders.yml` 2. **File naming** * Co-located approach: Uses the same name as the corresponding mart (`orders.yml`) * Parallel sub-folder approach: Prefixes the file with `sem_` (`sem_orders.yml`) Choose the approach that best fits your project structure and team preferences. The co-located approach is often simpler for new projects, while the parallel sub-folder approach can be clearer for migrating large existing projects to the Semantic Layer. #### Next steps Let's review the basics of semantic models: * 🧱 Consist of **entities, dimensions, and measures**. * 🫂 Describe the **semantics and relationships of objects** in the warehouse. * 1️⃣ Correspond to a **single logical model** in your dbt project. Next up, let's use our new semantic model to **build a metric**! --- ### Change data capture in dbt Change data capture (CDC) identifies new, updated, and deleted rows in your source data so you can process changes without rebuilding an *entire* table. This guide explains how you can use incremental models and snapshots in dbt to keep tables current, preserve a history of changes, or both. To find the best approach for your project, refer to [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md). For job frequency, streams, and dynamic tables, refer to [Near real-time data in dbt](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/1-intro.md), including [CDC with Snowflake Streams](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#cdc-with-snowflake-streams). #### How to handle CDC in dbt When source data changes, you may want to update your table, keep old versions, or do both. In dbt, that usually means one of two ways to build a table: * An [incremental model](https://docs.getdbt.com/docs/build/incremental-models-overview.md) keeps a table current. On each run, dbt processes new or changed rows and *replaces* the old row for that `unique_key`. * A [snapshot](https://docs.getdbt.com/docs/build/snapshots.md) keeps history. On each run, dbt compares the source to the last snapshot and *adds* a row when the record changes, with `dbt_valid_from` and `dbt_valid_to`. Snapshots only capture changes when you run them, which means you should run them on a schedule or you might miss changes. Refer to the FAQ [How often should I run the snapshot command?](https://docs.getdbt.com/faqs/Runs/snapshot-frequency.md), which recommends hourly to daily. #### Choose an approach: latest row, history, or both Your approach depends on how your source exposes changes and what you need to keep. | If you need | How your source stores changes | Use | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Latest row only | Appends a row for each change, or overwrites rows but has a reliable change timestamp | Incremental model | | Current row plus old versions | Overwrites rows, and is small enough to scan each run | Snapshot | | Current row plus old versions, without scanning the full source each run | Overwrites rows, or appends a row for each change (after cleanup) | An incremental staging model, then a snapshot, then a downstream model that keeps only the latest snapshot row |
For examples of each approach, refer to [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md). #### Key recommendations * Use incremental models when you only need the current rows and you can identify new or changed rows. An incremental model *replaces* the old row, so runs stay small and you do not store versions you will never query. * Use snapshots when you need to know what a record looked like at a point in the past. A snapshot *adds* a row when the record changes, which is how you keep the old version. An incremental model would have overwritten it. * Use both when staging should stay cheap and current, and a snapshot should store versions. The incremental model limits how much you process. The snapshot records history. Snapshot the staging models (or sources), not the final table people query, so you track the source as it changed, not a report that can change for other reasons. * Prefer a snapshot `timestamp` strategy when `updated_at` is reliable and only moves forward. That lets dbt detect a change from the clock instead of comparing every column. Use `check` when the timestamp is missing or untrustworthy, so a change in the row still gets recorded. * If the warehouse already writes a list of changes (streams), use an incremental model. The warehouse already detected the change, so you do not need a snapshot to find it. Refer to [CDC with Snowflake Streams](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#cdc-with-snowflake-streams) for that example. --- ### Choosing incremental models or snapshots Use [incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md), [snapshots](https://docs.getdbt.com/docs/build/snapshots.md), or both depending on whether you need current values, historical versions, or both. #### Compare incremental models and snapshots Start with this table if you already know what you need to keep. Use it to match the source to an incremental model, a snapshot, or both. | Approach | Use it when | Do not use it when | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | [Incremental only](#incremental-only) | You need the latest row per id, and the source already lists changes or has a reliable change timestamp | You must answer "what did this row look like last month?" from this table | | [Snapshot only](#snapshot-only) | The source overwrites rows in place, you need history, and scanning it each run is acceptable | The source only adds rows and never updates them, or the table is so large that a full snapshot run is too expensive | | [Incremental models and snapshots together](#using-incremental-models-and-snapshots-together) | You want incremental staging so runs stay cheap, and a snapshot for history | You only need one of those jobs, or you would snapshot the final table people query |
#### Incremental only Choose this approach when you only need each row's current values. Your source must let you identify new or updated rows from: * A loading tool or stream that writes inserts, updates, and deletes (for an example, refer to [CDC with Snowflake Streams](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#cdc-with-snowflake-streams)) * A table that only adds rows * A table that overwrites rows and has a reliable `updated_at` (or load timestamp) you can filter on These examples follow two customers, Alice and Bob. After the first run, both have a `trial` status: | id | name | status | updated\_at | | -- | ----- | ------ | ------------------- | | 1 | Alice | trial | 2026-01-01 00:00:00 | | 2 | Bob | trial | 2026-01-01 00:00:00 |
Alice then becomes an `active` customer the next day. With a `merge` strategy and `id` as the `unique_key`, the next run updates her existing row: | id | name | status | updated\_at | | -- | ----- | ------ | ------------------- | | 1 | Alice | active | 2026-01-02 00:00:00 | | 2 | Bob | trial | 2026-01-01 00:00:00 |
Alice's current status is `active`. Her earlier `trial` status is no longer stored in this table. To configure this behavior: * Use `is_incremental()` to apply a filter for new or updated rows on later runs * Set [`unique_key`](https://docs.getdbt.com/reference/resource-configs/unique_key.md) to the customer ID so dbt can match incoming rows to existing ones * Choose a strategy such as `merge`, which updates matching rows and inserts new ones Refer to [Configure incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) and [About incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) for details. #### Snapshot only Choose this approach when your source overwrites rows and you need the history to answer questions such as "When did Alice become an active customer?" Using the same example, run a snapshot before and after Alice's status changes. With the `timestamp` strategy, the snapshot keeps both versions: | id | name | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | | -- | ----- | ------ | ------------------- | ------------------- | ------------------- | | 1 | Alice | trial | 2026-01-01 00:00:00 | 2026-01-01 00:00:00 | 2026-01-02 00:00:00 | | 1 | Alice | active | 2026-01-02 00:00:00 | 2026-01-02 00:00:00 | `null` | | 2 | Bob | trial | 2026-01-01 00:00:00 | 2026-01-01 00:00:00 | `null` |
The `dbt_valid_from` and `dbt_valid_to` columns show you when each version was valid. By default, `dbt_valid_to` is `null` for the current version. To configure your snapshot, set `unique_key` to the customer ID and choose how to detect changes: * Use `timestamp` (recommended) when you have a reliable `updated_at` column that advances whenever a row changes * Use `check` when a table has no reliable `updated_at` column. dbt tracks changes by comparing the columns you specify, or all columns Run `dbt snapshot` or `dbt build` regularly on a schedule. Snapshots capture the values available when they run, so they cannot recover intermediate changes that were overwritten between runs. Choose a schedule based on how often your data changes and how much detail you need in its history. Refer to [Add snapshots to your DAG](https://docs.getdbt.com/docs/build/snapshots.md) and [How often should I run the snapshot command?](https://docs.getdbt.com/faqs/Runs/snapshot-frequency.md) for configuration and scheduling. #### Using incremental models and snapshots together Use incremental models and snapshots together when you need an incremental staging model to prepare incoming data before capturing its history. The following example continues with Alice and Bob. It uses a source named `raw` with a `customers` table that stores one current row per customer. Replace these names with a [source defined in your project](https://docs.getdbt.com/docs/build/sources.md). The source includes two timestamps: * `updated_at`: When the customer's data changed. * `_loaded_at`: When your loading tool wrote that version to the warehouse. This example assumes it is populated and advances with each loaded change. ##### 1. Process incoming changes Create an incremental staging model that uses `_loaded_at` to find recently loaded rows and `id` to update the matching customer. models/staging/stg\_customers.sql ```sql {{ config( materialized='incremental', unique_key='id', incremental_strategy='merge' ) }} select id, name, status, updated_at, _loaded_at from {{ source('raw', 'customers') }} {% if is_incremental() %} where _loaded_at >= ( select coalesce(max(_loaded_at), '1970-01-01') from {{ this }} ) {% endif %} ``` The first run reads all rows. Later runs read rows at or after the latest stored `_loaded_at` value. Using `>=` includes rows that share that timestamp. ##### 2. Capture historical versions Create a snapshot of the staging model. It uses `updated_at` to detect changes to each customer's data. snapshots/customers\_snapshot.yml ```yaml snapshots: - name: customers_snapshot relation: ref('stg_customers') config: strategy: timestamp unique_key: id updated_at: updated_at ``` Snapshot sources or lightly transformed staging models. Keep business logic in downstream models so changes to that logic do not become part of your source history. ##### 3. Select current values for reporting Create a model that selects the current version of each customer: models/marts/dim\_customers\_current.sql ```sql select id, name, status, updated_at from {{ ref('customers_snapshot') }} where dbt_valid_to is null ``` Run these resources in dependency order: staging model, snapshot, then reporting model. You can use `dbt build` with all three selected. After a run captures Alice's change, the staging and reporting models show her as `active`. The snapshot keeps both her `trial` and `active` versions, as shown in the earlier examples. If the source already includes from and to dates (or a list of changes you want to keep in full), you may not need a snapshot. Load those changes with incremental `append` or `merge`, and keep the latest row with a SQL filter. #### What else to consider * Hard deletes: Loading tools often mark a row as deleted. Snapshots can close the old row or add a deletion record with [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md). Incremental models must handle deletes in your merge (or a separate delete statement). * Late-arriving changes: Widen the incremental filter so you look a bit further back than the last run, and know when a `--full-refresh` is the safe fix. Refer to [Configure incremental models](https://docs.getdbt.com/docs/build/incremental-models.md). * Several changes in one run: On incremental models, keep only the latest change per id before you merge. * Source columns change: Use [`on_schema_change`](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change) to control how incremental models handle schema changes. Snapshots can add new columns, but you may need to update [`check_cols`](https://docs.getdbt.com/docs/build/snapshots.md#check-strategy) if you use the `check` strategy. * Tests: Test that the source or staging `unique_key` is unique and not null. For snapshots, also test that `(unique_key, dbt_valid_from)` is unique, that `dbt_valid_from` / `dbt_valid_to` ranges do not overlap, and that each ID has at most one current version (`dbt_valid_to` is null). Monitor [source freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) to identify loading delays. * Cost: Organize your table by the change timestamp and use filters to limit how much data each run scans. Refer to [About incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md). #### Related docs * [About incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md) * [Configure incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * [Add snapshots to your DAG](https://docs.getdbt.com/docs/build/snapshots.md) * [`unique_key`](https://docs.getdbt.com/reference/resource-configs/unique_key.md) * [CDC with Snowflake Streams](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#cdc-with-snowflake-streams) * [dbt blog: Strategies for change data capture in dbt](https://docs.getdbt.com/blog/change-data-capture) --- ### Clone incremental models as the first step of your CI job Before you begin, you must be aware of a few conditions: * `dbt clone` is only available with dbt version 1.6 and newer. Refer to our [upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) for help enabling newer versions in dbt. * This strategy only works for warehouse that support zero copy cloning (otherwise `dbt clone` will just create pointer views). * Some teams may want to test that their incremental models run in both incremental mode and full-refresh mode. Imagine you've created a [Slim CI job](https://docs.getdbt.com/docs/deploy/continuous-integration.md) in dbt and it is configured to: * Defer to your production environment. * Run the command `dbt build --select state:modified+` to run and test all of the models you've modified and their downstream dependencies. * Trigger whenever a developer on your team opens a PR against the main branch. [![Example of a slim CI job with the above configurations](/img/best-practices/slim-ci-job.png?v=2 "Example of a slim CI job with the above configurations")](#)Example of a slim CI job with the above configurations Now imagine your dbt project looks something like this in the DAG: [![Sample project DAG](/img/best-practices/dag-example.png?v=2 "Sample project DAG")](#)Sample project DAG When you open a pull request (PR) that modifies `dim_wizards`, your CI job will kickoff and build *only the modified models and their downstream dependencies* (in this case, `dim_wizards` and `fct_orders`) into a temporary schema that's unique to your PR. This build mimics the behavior of what will happen once the PR is merged into the main branch. It ensures you're not introducing breaking changes, without needing to build your entire dbt project. #### What happens when one of the modified models (or one of their downstream dependencies) is an incremental model? Because your CI job is building modified models into a PR-specific schema, on the first execution of `dbt build --select state:modified+`, the modified incremental model will be built in its entirety *because it does not yet exist in the PR-specific schema* and [is\_incremental will be false](https://docs.getdbt.com/docs/build/incremental-models.md#understand-the-is_incremental-macro). You're running in `full-refresh` mode. This can be suboptimal because: * Typically incremental models are your largest datasets, so they take a long time to build in their entirety which can slow down development time and incur high warehouse costs. * There are situations where a `full-refresh` of the incremental model passes successfully in your CI job but an *incremental* build of that same table in prod would fail when the PR is merged into main (think schema drift where [on\_schema\_change](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change) config is set to `fail`) You can alleviate these problems by zero copy cloning the relevant, pre-existing incremental models into your PR-specific schema as the first step of the CI job using the `dbt clone` command. This way, the incremental models already exist in the PR-specific schema when you first execute the command `dbt build --select state:modified+` so the `is_incremental` flag will be `true`. You'll have two commands for your dbt CI check to execute: 1. Clone all of the pre-existing incremental models that have been modified or are downstream of another model that has been modified: ```shell dbt clone --select state:modified+,config.materialized:incremental,state:old ``` 2. Build all of the models that have been modified and their downstream dependencies: ```shell dbt build --select state:modified+ ``` Because of your first clone step, the incremental models selected in your `dbt build` on the second step will run in incremental mode. [![Clone command in the CI config](/img/best-practices/clone-command.png?v=2 "Clone command in the CI config")](#)Clone command in the CI config Your CI jobs will run faster, and you're more accurately mimicking the behavior of what will happen once the PR has been merged into main. ##### Expansion on "think schema drift" where [on\_schema\_change](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change) config is set to `fail`" from above Imagine you have an incremental model `my_incremental_model` with the following config: ```sql {{ config( materialized='incremental', unique_key='unique_id', on_schema_change='fail' ) }} ``` Now, let’s say you open up a PR that adds a new column to `my_incremental_model`. In this case: * An incremental build will fail. * A `full-refresh` will succeed. If you have a daily production job that just executes `dbt build` without a `--full-refresh` flag, once the PR is merged into main and the job kicks off, you will get a failure. So the question is - what do you want to happen in CI? * Do you want to also get a failure in CI, so that you know that once this PR is merged into main you need to immediately execute a `dbt build --full-refresh --select my_incremental_model` in production in order to avoid a failure in prod? This will block your CI check from passing. * Do you want your CI check to succeed, because once you do run a `full-refresh` for this model in prod you will be in a successful state? This may lead unpleasant surprises if your production job is suddenly failing when you merge this PR into main if you don’t remember you need to execute a `dbt build --full-refresh --select my_incremental_model` in production. There’s probably no perfect solution here; it’s all just tradeoffs! Our preference would be to have the failing CI job and have to manually override the blocking branch protection rule so that there are no surprises and we can proactively run the appropriate command in production once the PR is merged. ##### Expansion on "why `state:old`" For brand new incremental models, you want them to run in `full-refresh` mode in CI, because they will run in `full-refresh` mode in production when the PR is merged into `main`. They also don't exist yet in the production environment... they're brand new! If you don't specify this, you won't get an error just a “No relation found in state manifest for…”. So, it technically works without specifying `state:old` but adding `state:old` is more explicit and means it won't even try to clone the brand new incremental models. --- ### Conclusion You're now following best practices in your project, and have optimized the materializations of your DAG. You’re equipped with the 3 main materializations that cover almost any analytics engineering situation! There are more configs and materializations available, as well as specific materializations for certain platforms and adapters — and like everything with dbt, materializations are extensible, meaning you can create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md) for your needs. So this is just the beginning of what you can do with these powerful configurations. For the vast majority of users and companies though, tables, views, and incremental models will handle everything you can throw at them. Develop your intuition and expertise for these materializations, and you’ll be well on your way to tackling advanced analytics engineering problems. --- ### Configuring materializations #### Configuring materializations Choosing which materialization is as simple as setting any other configuration in dbt. We’ll look first at how we select our materializations for individual models, then at more powerful ways of setting materializations for entire folders of models. ##### Configuring tables and views Let’s look at how we can use tables and views to get started with materializations: * ⚙️ We can configure an individual model’s materialization using a **Jinja `config` block**, and passing in the **`materialized` argument**. This tells dbt what materialization to use. * 🚰 The underlying specifics of what is run depends on [which **adapter** you’re using](https://docs.getdbt.com/docs/supported-data-platforms.md), but the end results will be equivalent. * 😌 This is one of the many valuable aspects of dbt: it lets us use a **declarative** approach, specifying the *outcome* that we want in our code, rather than *specific steps* to achieve it (the latter is an *imperative* approach if you want to get computer science-y about it 🤓). * 🔍 In the below case, we want to create a SQL **view**, and can **declare** that in a **single line of code**. Note that python models [do not support materializing as views](https://docs.getdbt.com/docs/build/materializations.md#python-materializations) at this time. ```sql {{ config( materialized='view' ) }} select ... ``` info 🐍 **Not all adapters support python yet**, check the [docs here to be sure](https://docs.getdbt.com/docs/build/python-models.md#specific-data-platforms) before spending time writing python models. * Configuring a model to materialize as a `table` is simple, and possible for both SQL and python models. ##### SQL ```sql {{ config( materialized='table' ) }} select ... ``` ##### Python ```python def model(dbt, session): dbt.config(materialized="table") # model logic return model_df ``` Go ahead and try some of these out! --- ### Coordinating model versions Coordinating model versions across your mesh is a critical part of the model versioning process. This guide will walk you through the safe best practices for coordinating producers and consumers when introducing model versions. [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) are an important part of your dbt Mesh workflow. They enable better data model management and are critical when multiple teams share models across projects. Releasing a new model version safely requires coordination between model producers (who build the models) and model consumers (who depend on them). This guide goes over the following topics: * [How producers introduce new model versions safely](#best-practices-for-producers) * [How consumers evaluate and migrate to those new versions](#best-practices-for-consumers) For how versioning works at a technical level (YAML structure, contracts, aliasing), see [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md). #### Best practices for producers Producers own the creation, rollout, communication, and deprecation of model versions. The following steps go over what producers should do when introducing a new version of a model. * [Step 1: Decide when a change needs a new version](#step-1-decide-when-a-change-needs-a-new-version) * [Step 2: Create the new version safely](#step-2-create-the-new-version-safely) * [Step 3: Add a deprecation date](#step-3-add-a-deprecation-date) * [Step 4: Communicate the new version](#step-4-communicate-the-new-version) * [Step 5: Set the new latest version](#step-5-set-the-new-latest-version) * [Step 6: Clean up deprecated versions](#step-6-clean-up-deprecated-versions) ###### Step 1: Decide when a change needs a new version When creating an original version of a model, use [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) to ensure that breaking changes produce errors during development. The model contract ensures you, as a producer, are not changing the shape or data type of the output model. If a change breaks the contract, like removing or changing a column type, create a new model version with an updated contract instead of changing the existing version in place. Here are some examples of breaking changes that might need a new version: * Removing a column * Renaming a column * Changing a column type Here are some examples of non-breaking changes: * Adding a new column * Fixing a bug in an existing column Here are examples of changes that might be breaking depending on your business logic: * Changing logic behind a metric * Changing granularity * Modifying filters * Rewriting `CASE` statements ###### Step 2: Create the new version safely After deciding that a change needs a new [version](https://docs.getdbt.com/reference/resource-properties/versions.md), follow these steps to create the new version without disrupting existing workflows. Let's say you're removing a column: 1. Create a new version of the model file. For example, `fishtown_analytics_orders_v2.sql`. Each version of a model must have its own SQL file. 2. Keep the default version stable. In the model's `properties.yml` file: * Set [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md) to include the old version and the new version: `- v: 1` and `- v: 2` respectively. * Set the [`latest_version:`](https://docs.getdbt.com/reference/resource-properties/latest_version.md) to `latest_version: 1`. This ensures that downstream consumers using `ref(...)` won’t accidentally start using v2. Without setting this, the default will be the highest numerical version, which could be a breaking change for consumers. 3. Set an [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) or create a view over the latest model version. By aliasing or creating a view over the latest model version, you ensure `fishtown_analytics_orders` (without the version suffix) always exists as an object in the warehouse, pointing to the latest version. This also protects external tools and BI dashboards. ###### Step 3: Add a deprecation date 1. In the model's `properties.yml` file, set a [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md) for the model's old version. The `deprecation_date` is a date in the future that signifies when the old version will be removed. This notifies downstream consumers and will appear in the `dbt run` logs as a warning that the old version is nearing deprecation and consumers will need to [migrate](#best-practices-for-consumers) to the new version. info If your model has an [enforced contract](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md), you cannot delete the model until after the `deprecation_date` has passed. dbt doesn't allow deleting models with enforced contracts before their `deprecation_date` to protect downstream consumers. If you try to delete a versioned model before its `deprecation_date`, dbt will raise an error during development runs and cause jobs to fail. models/properties.yml ```yaml models: - name: fishtown_analytics_orders latest_version: 1 columns: - name: column_to_remove - name: column_to_keep versions: - v: 1 # old version (uses all top-level columns) deprecation_date: "2025-12-31" - v: 2 # new version columns: - include: all exclude: [column_to_remove] # columns removed in v2 ``` 2. Merge the new version into the main branch. 3. Run the job to build the new version. 4. Verify that the new version builds successfully. 5. Verify that the deprecation date is set correctly in the `dbt run` logs. If you try to reference models (for example, `{{ ref('upstream_project', 'model_name', v=1) }}`) using the `v=1` argument after the deprecation date, the `ref` call will fail once the producer project removes the `v1` version. ###### Step 4: Communicate the new version After creating a new version and setting a deprecation date for the old version, communicate the new version to downstream consumers. Let them know that: * A new version is available and a deprecation timeline exists. * Consumers can test the new version and [migrate](#best-practices-for-consumers) over. * To test the new version, consumers can use `v=2` when referencing the model. For example, `{{ ref('upstream_project', 'model_name', v=2) }}`. ###### Step 5: Set the new latest version Once consumers have migrated, set the new version as the latest version. v1 remains in the project until you complete [Step 6](#step-6-clean-up-deprecated-versions). models/properties.yml ```yaml models: - name: fishtown_analytics_orders latest_version: 2 # update from 1 to 2 to set the new version as the latest version versions: - v: 1 # this represents the old version - v: 2 # this represents the new version ``` This then updates the default `ref` to the new version. For example, `{{ ref('upstream_project', 'fishtown_analytics_orders') }}` will now resolve to the `fishtown_analytics_orders_v2` model in the `upstream_project`. If consumers want to use the old version, they can use `v=1` when referencing the model: `{{ ref('upstream_project', 'fishtown_analytics_orders', v=1) }}`. ###### Step 6: Clean up deprecated versions After all consumers have [migrated](#best-practices-for-consumers) to the new version, you can clean up the deprecated version. You could choose to "hard delete" all old versions, or "soft delete" them for continuity. info When removing or renaming a versioned model with an enforced contract, first deprecate the version you plan to retire. Set a `deprecation_date` on the retiring version. Use a date in the past if you are removing it immediately. Merge that change and run your production job so [state-aware CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md) records the deprecation. In a follow-up change, remove or rename the model, such as renaming `fishtown_analytics_orders_v2.sql` to `fishtown_analytics_orders.sql`. Skipping this intermediate step may cause dbt to treat the removal as an unexpected breaking contract change and fail CI. For an optional walkthrough, refer to the [video walkthrough on removing a versioned model with an enforced contract](https://www.youtube.com/watch?v=FQ905Zj5C1o). ##### Hard delete (cleanest) "Hard deleting" old versions is the cleanest approach. It removes version artifacts from your project and the warehouse. If the model has an enforced contract, complete the workflow in the callout above first. Then, in a follow-up change: 1. Delete the SQL file for the deprecated version (for example, `fishtown_analytics_orders_v1.sql`). 2. If you are removing versioning entirely, rename the latest version file to the base name (for example, `fishtown_analytics_orders_v2.sql` to `fishtown_analytics_orders.sql`). 3. Delete all version specifications from your `.yml` file. 4. Drop or delete deprecated version objects from your warehouse with a manual script or a cleanup macro. ##### Soft delete (retains continuity) "Soft deleting" old versions retains all old version artifacts to avoid confusion if more model versions get introduced in the future, and for continuity. Bear in mind that your version control platform will also have the history of all of these changes. If the model has an enforced contract, complete the workflow in the callout above first. Then, in a follow-up change: 1. Repoint the `fishtown_analytics_orders` alias to your latest version file (for example, `fishtown_analytics_orders_v2`), or create a view on top of the latest model version. 2. Use the `enabled` [config option](https://docs.getdbt.com/reference/resource-configs/enabled.md) to disable the deprecated model version so that it doesn’t run in dbt jobs and can’t be referenced in a cross-project ref. For example: models/properties.yml ```yaml models: - name: fishtown_analytics_orders latest_version: 2 columns: - name: column_to_remove - name: column_to_keep versions: - v: 1 # old version (uses all top-level columns) deprecation_date: "2025-12-31" config: enabled: false # disable deprecated version so it no longer runs - v: 2 # new version columns: - include: all exclude: [column_to_remove] # columns removed in v2 ``` 3. Drop or delete the `fishtown_analytics_orders_v1` object from your warehouse with a manual script or appropriate process or using a cleanup macro. ... and that's it! You've introduced a new model version and completed producer cleanup. The next section is for consumers to evaluate and migrate to new versions. #### Best practices for consumers Consumers rely on upstream models and need to make sure that version transitions don’t introduce unintended breakages. Refer to the following steps to migrate to the new version: 1. Begin writing a cross-project reference to use a public model from a different project. In this case, `{{ ref('upstream_project', 'fishtown_analytics_orders') }}`. 2. Once you see deprecation warnings, test the latest version of a model by explicitly referencing it in your `ref`. For example, `{{ ref('upstream_project', 'fishtown_analytics_orders', v=2) }}`. Check if it's a breaking change for you or has any unintended impacts on your project. * If it does, consider explicitly “pinning” to the current, working version of the model before the new version becomes the default: `{{ ref('upstream_project', 'fishtown_analytics_orders', v=1) }}`. Bear in mind that you will need to migrate at some point before the deprecation date. 3. Before the deprecation date, you can migrate to the new version of the model by removing the version specification in your cross-project reference: `{{ ref('upstream_project', 'fishtown_analytics_orders') }}`. Make any downstream logic changes needed to accommodate this new version. Consumers should plan migrations to align with their own team’s release cycles. #### Related docs * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) * [Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) * [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md) * [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [Project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) * [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) --- ### dbt Mesh FAQs Mesh is a new architecture enabled by dbt. It allows you to better manage complexity by deploying multiple interconnected dbt projects instead of a single large, monolithic project. It’s designed to accelerate development, without compromising governance. #### Overview of Mesh  What are the main benefits of implementing dbt Mesh? Here are some benefits of implementing dbt Mesh: * **Ship data products faster**: With a more modular architecture, teams can make changes rapidly and independently in specific areas without impacting the entire system, leading to faster development cycles. * **Improve trust in data:** Adopting dbt Mesh helps ensure that changes in one domain's data models do not unexpectedly break dependencies in other domain areas, leading to a more secure and predictable data environment. * **Reduce complexity**: By organizing transformation logic into distinct domains, dbt Mesh reduces the complexity inherent in large, monolithic projects, making them easier to manage and understand. * **Improve collaboration**: Teams are able to share and build upon each other's work without duplicating efforts. Most importantly, all this can be accomplished without the central data team losing the ability to see lineage across the entire organization, or compromising on governance mechanisms.  What are model contracts? dbt [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) serve as a governance tool enabling the definition and enforcement of data structure standards in your dbt models. They allow you to specify and uphold data model guarantees, including column data types, allowing for the stability of dependent models. Should a model fail to adhere to its established contracts, it will not build successfully.  What are model versions? dbt [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) are iterations of your dbt models made over time. In many cases, you might knowingly choose to change a model’s structure in a way that “breaks” the previous model contract, and may break downstream queries depending on that model’s structure. When you do so, creating a new version of the model is useful to signify this change. You can use model versions to: * Test "prerelease" changes (in production, in downstream systems). * Bump the latest version, to be used as the canonical "source of truth." * Offer a migration window off the "old" version.  What are model access modifiers? A [model access modifier](https://docs.getdbt.com/docs/mesh/govern/model-access.md) in dbt determines if a model is accessible as an input to other dbt models and projects. It specifies where a model can be referenced using [the `ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md). There are three types of access modifiers: * **Private:** A model with a private access modifier is only referenceable by models within the same group. This is intended for models that are implementation details and are meant to be used only within a specific group of related models. * **Protected:** Models with a protected access modifier can be referenced by any other model within the same dbt project or when the project is installed as a package. This is the default setting for all models, ensuring backward compatibility, especially when groups are assigned to an existing set of models. * **Public:** A public model can be referenced across different groups, packages, or projects. This is suitable for stable and mature models that serve as interfaces for other teams or projects.  What are model groups? A [model group](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) in dbt is a concept used to organize models under a common category or ownership. This categorization can be based on various criteria, such as the team responsible for the models or the specific data source they model.  What are some potential challenges when using dbt Mesh? This is a new way of working, and the intentionality required to build, and then maintain, cross-project interfaces and dependencies may feel like a slowdown versus what some developers are used to. The intentional friction introduced promotes thoughtful changes, fostering a mindset that values stability and systematic adjustments over rapid transformations. Orchestration across multiple projects is also likely to be slightly more challenging for many organizations, although we’re currently developing new functionality that will make this process simpler.  How does this relate to the concept of data mesh? dbt Mesh allows you to better *operationalize* data mesh by enabling decentralized, domain-specific data ownership and collaboration. In data mesh, each business domain is responsible for its data as a product. This is the same goal that dbt Mesh facilitates by enabling organizations to break down large, monolithic data projects into smaller, domain-specific dbt projects. Each team or domain can independently develop, maintain, and share its data models, fostering a decentralized data environment. dbt Mesh also enhances the interoperability and reusability of data across different domains, a key aspect of the data mesh philosophy. By allowing cross-project references and shared governance through model contracts and access controls, dbt Mesh ensures that while data ownership is decentralized, there is still a governed structure to the overall data architecture. #### How dbt Mesh works  Can dbt Mesh handle cyclic dependencies between projects? You can enable bidirectional dependencies across projects so these relationships can go in either direction, meaning that the `jaffle_finance` project can add a new model that depends on any public models produced by the `jaffle_marketing` project, so long as the new dependency doesn't introduce any node-level cycles. dbt checks for cycles across projects and raises errors if any are detected. When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: 1. The `project_a` project runs in a deployment environment and produces public models. 2. The `project_b` project adds `project_a` as a dependency. 3. The `project_b` project runs in a deployment environment and produces public models. 4. The `project_a` project adds `project_b` as a dependency.  Is it possible for multiple projects to directly reference a shared source? While it’s not currently possible to share sources across projects, it would be possible to have a shared foundational project, with staging models on top of those sources, exposed as “public” models to other teams/projects.  What if a model I've already built on from another project later becomes protected? This would be a breaking change for downstream consumers of that model. If the maintainers of the upstream project wish to remove the model (or “downgrade” its access modifier, effectively the same thing), they should mark that model for deprecation (using [deprecation\_date](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md)), which will deliver a warning to all downstream consumers of that model. In the future, we plan for dbt to also be able to proactively flag this scenario in [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) for the maintainers of the upstream public model.  If I run \`dbt build --select +model\`, will this trigger a run of upstream models in other projects? No, unless upstream projects are installed as [packages](https://docs.getdbt.com/docs/build/packages.md) (source code). In that case, the models in project installed as a project become “your” models, and you can select or run them. There are cases in which this can be desirable; see docs on [project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md).  If each project/domain has its own data warehouse, is it still possible to build models across them? Yes, as long as they’re in the same data platform (BigQuery, Databricks, Redshift, Snowflake, etc.) and you have configured permissions and sharing in that data platform provider to allow this.  Can I run tests that involve tables from multiple different projects? Yes, because the cross-project collaboration is done using the `{{ ref() }}` macro, you can use those models from other teams in [singular tests](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests).  Which team's data schema would dbt Mesh create? Each team defines their connection to the data warehouse, and the default schema names for dbt to use when materializing datasets. By default, each project belonging to a team will create: * One schema for production runs (for example, `finance`). * One schema per developer (for example, `dev_jerco`). Depending on each team’s needs, this can be customized with model-level [schema configurations](https://docs.getdbt.com/docs/build/custom-schemas.md), including the ability to define different rules by environment.  Is it possible to apply model contracts to source data? No, contracts can only be applied at the [model level](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md). It is a recommended best practice to [define staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) on top of sources, and it is possible to define contracts on top of those staging models.  Can contracts be partially enforced? No. A contract applies to an entire model, including all columns in the model’s output. This is the same set of columns that a consumer would see when viewing the model’s details in Explorer, or when querying the model in the data platform. * If you wish to contract only a subset of columns, you can create a separate model (materialized as a view) selecting only that subset. * If you wish to limit which rows or columns a downstream consumer can see when they query the model’s data, depending on who they are, some data platforms offer advanced capabilities around dynamic row-level access and column-level data masking.  Can I have multiple owners in a group? No, a [group](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) can only be assigned to a single owner. However, the assigned owner can be a *team*, rather than an individual.  Can contracts be assigned individual owners? Not directly, but contracts are [assigned to models](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) and models can be assigned to individual owners. You can use meta fields for this purpose.  Can I make a model “public” only for specific team(s) to use? This is not currently possible, but something we hope to enable in the near future. If you’re interested in this functionality, please reach out to your dbt Labs account team.  Is it possible to orchestrate job runs across multiple different projects? Yes. You can configure dbt to trigger a job when another job completes, including jobs in different projects. This is one way to execute a pipeline end to end across projects. For details, refer to [Trigger on job completion](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion).  Integrations available between the dbt Discovery API and other tools for cross-project lineage? Yes. In addition to being viewable natively through [Catalog](https://www.getdbt.com/product/dbt-explorer), it is possible to view cross-project lineage connect using partner integrations with data cataloging tools. For a list of available dbt integrations, refer to the [Integrations page](https://www.getdbt.com/product/integrations).  How does data restatement work in dbt Mesh, particularly when fixing a data set bug? Tests and model contracts in dbt help eliminate the need to restate data in the first place. With these tools, you can incorporate checks at the source and output layers of your dbt projects to assess data quality in the most critical places. When there are changes in transformation logic (for example, the definition of a particular column is changed), restating the data is as easy as merging the updated code and running a dbt job. If a data quality issue does slip through, you also have the option of simply rolling back the git commit, and then re-running the dbt job with the old code.  How does dbt handle job run logs and can it feed them to standard monitoring tools, reports, etc.? Yes, all of this metadata is accessible via the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md). This metadata can be fed into a monitoring tool, or used to create reports and dashboards. We also expose some of this information in dbt itself in [jobs](https://docs.getdbt.com/docs/deploy/jobs.md), [environments](https://docs.getdbt.com/docs/environments-in-dbt.md) and in [Catalog](https://www.getdbt.com/product/dbt-explorer).  Can dbt Mesh reference models in other accounts within the same data platform? You can reference models in other accounts within the same data platform by leveraging the data-sharing capabilities of that platform, as long as the database identifier of the public model is consistent across the producer and consumer. For example, [Snowflake cross-account data shares](https://docs.snowflake.com/en/user-guide/data-sharing-intro), [Databricks Unity Catalog across workspaces](https://docs.databricks.com/en/data-governance/unity-catalog/index.html), or multiple BigQuery projects. #### Permissions and access  How do user access permissions work in dbt Mesh? The existence of projects that have at least one public model will be visible to everyone in the organization with [read-only access](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). Private or protected models require a user to have read-only access to the specific project to see its existence.  How do all the different types of “access” interact? There’s model-level access within dbt, role-based access for users and groups in dbt, and access to the underlying data within the data platform. First things first: access to underlying data is always defined and enforced by the underlying data platform (for example, BigQuery, Databricks, Redshift, Snowflake, Starburst, etc.) This access is managed by executing “DCL statements” (namely `grant`). dbt makes it easy to [configure `grants` on models](https://docs.getdbt.com/reference/resource-configs/grants.md), which provision data access for other roles/users/groups in the data warehouse. However, dbt does *not* automatically define or coordinate those grants unless they are configured explicitly. Refer to your organization's system for managing data warehouse permissions. [dbt Enterprise and Enterprise+ plans](https://www.getdbt.com/pricing) support [role-based access control (RBAC)](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-) that manages granular permissions for users and user groups. You can control which users can see or edit all aspects of a dbt project. A user’s access to dbt projects also determines whether they can “explore” that project in detail. Roles, users, and groups are defined within the dbt application via the UI or by integrating with an identity provider. [Model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md) defines where models can be referenced. It also informs the discoverability of those projects within Catalog. Model `access` is defined in code, just like any other model configuration (`materialized`, `tags`, etc). * **Public:** Models with `public` access can be referenced everywhere. These are the “data products” of your organization. * **Protected:** Models with `protected` access can only be referenced within the same project. This is the default level of model access. We are discussing a future extension to `protected` models to allow for their reference in *specific* downstream projects. Please read [the GitHub issue](https://github.com/dbt-labs/dbt/issues/9340), and upvote/comment if you’re interested in this use case. * **Private:** Model `groups` enable more-granular control over where `private` models can be referenced. By defining a group, and configuring models to belong to that group, you can restrict other models (not in the same group) from referencing any `private` models the group contains. Groups also provide a standard mechanism for defining the `owner` of all resources it contains. Within Catalog, `public` models are discoverable for every user in the dbt account — every public model is listed in the “multi-project” view. By contrast, `protected` and `private` models in a project are visible only to users who have access to that project (including read-only access). Because dbt does not implicitly coordinate data warehouse `grants` with model-level `access`, it is possible for there to be a mismatch between them. For example, a `public` model’s metadata is viewable to all dbt users, anyone can write a `ref` to that model, but when they actually run or preview, they realize they do not have access to the underlying data in the data warehouse. **This is intentional.** In this way, your organization can retain least-privileged access to underlying data, while providing visibility and discoverability for the wider organization. Armed with the knowledge of which other “data products” (public models) exist — their descriptions, their ownership, which columns they contain — an analyst on another team can prepare a well-informed request for access to the underlying data.  Is it possible to request access permissions from other teams within dbt? Not currently! But this is something we may evaluate in the future.  As a central data team member, can I still maintain visibility on the entire organizational DAG? Yes! As long as a user has permissions (at least read-only access) on all projects in a dbt account, they can navigate across the entirety of the organization’s DAG in Catalog, and see models at all levels of detail.  How can I limit my developers from accessing sensitive production data when referencing from other projects? By default, cross-project references resolve to the “Production” deployment environment of the upstream project. If your organization has genuinely different data in production versus non-production environments, this poses an issue. For this reason, we rolled out canonical type of deployment environment: “[Staging](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment).” If a project defines both a “Production” environment and a “Staging” environment, then cross-project references from development and “Staging” environments will resolve to “Staging,” whereas only references coming from “Production” environments will resolve to “Production.” In this way, you are guaranteed separation of data environments, without needing to duplicate project configurations.  Does dbt Mesh work if projects are 'duplicated' (dev project <> prod project)? The short answer is "no." Cross-project references require that each project `name` be unique in your dbt account. Historical limitations required customers to "duplicate" projects so that one actual dbt project (codebase) would map to more than one dbt project. To that end, we are working to remove the historical limitations that required customers to "duplicate" projects in dbt — Staging environments for data isolation, environment-level permissions, and environment-level data warehouse connections (coming soon). Once those pieces are in place, it should no longer be necessary to define separate dbt projects to isolate data environments or permissions. #### Compatibility with other features  How does the dbt Semantic Layer relate to and work with dbt Mesh? The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and dbt Mesh are complementary mechanisms enabled by dbt that work together to enhance the management, usability, and governance of data in large-scale data environments. The Semantic Layer in dbt allows teams to centrally define business metrics and dimensions. It ensures consistent and reliable metric definitions across various analytics tools and platforms. Mesh enables organizations to split their data architecture into multiple domain-specific projects, while retaining the ability to reference “public” models across projects. It is also possible to reference a “public” model from another project for the purpose of defining semantic models and metrics. Your organization can have multiple dbt projects feed into a unified semantic layer, ensuring that metrics and dimensions are consistently defined and understood across these domains. When using the dbt Semantic Layer in a [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setting, we recommend the following: * You have one standalone project that contains your semantic models and metrics. * Then as you build your Semantic Layer, you can [cross-reference dbt models](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) across your various projects or packages to create your semantic models using the [two-argument `ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) (`ref('project_name', 'model_name')`). * Your dbt Semantic Layer project serves as a global source of truth across the rest of your projects. (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ###### Usage example For example, let's say you have a public model (`fct_orders`) that lives in the `jaffle_finance` project. As you build your semantic model, use the following syntax to ref the model: models/metrics/semantic\_model\_name.yml ```yaml semantic_models: - name: customer_orders defaults: agg_time_dimension: first_ordered_at description: | Customer grain mart that aggregates customer orders. model: ref('jaffle_finance', 'fct_orders') # ref('project_name', 'model_name') entities: ...rest of configuration... dimensions: ...rest of configuration... measures: ...rest of configuration... ``` Notice that in the `model` parameter, we're using the `ref` function with two arguments to reference the public model `fct_orders` defined in the `jaffle_finance` project.
 How does dbt Catalog relate to and work with dbt Mesh? **[Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md)** is a tool within dbt that serves as a knowledge base and lineage visualization platform. It provides a comprehensive view of your dbt assets, including models, tests, sources, and their interdependencies. Used in conjunction with dbt Mesh, Catalog becomes a powerful tool for visualizing and understanding the relationships and dependencies between models across multiple dbt projects.  How does the dbt platform CLI relate to and work with dbt Mesh? The [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) allows users to develop and run dbt commands from their preferred development environments, like VS Code, Sublime Text, or terminal interfaces. This flexibility is particularly beneficial in a dbt Mesh setup, where managing multiple projects can be complex. Developers can work in their preferred tools while leveraging the centralized capabilities of dbt.  Can I upgrade Mesh projects to v2 incrementally? Yes! You can upgrade select projects to dbt v2 while keeping others on dbt v1. * dbt v2 projects can reference public models from dbt v1 projects * dbt v1 projects can reference public models from dbt v2 projects This works because dbt Mesh uses a publication artifact (not the manifest) to resolve cross-project references, and this artifact is identical between dbt v1 and dbt v2. You can upgrade dbt Mesh projects to dbt v2 in any order and there's no requirement to start with upstream or downstream projects first. Feature optimization While basic Mesh functionality works in hybrid setups, some advanced platform features (like full Catalog lineage visibility across projects) work best when all projects use the same engine. #### Availability  Does dbt Mesh require me to be on a specific version of dbt? Yes, your account must be on [at least dbt v1.6](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to take advantage of [cross-project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md), one of the most crucial underlying capabilities required to implement a dbt Mesh.  Is there a way to leverage dbt Mesh capabilities with local dbt installations? While self-hosted local dbt installations (dbt CLI) define several of the foundational elements for dbt Mesh, the dbt platform offers an enhanced experience that leverages these elements for scaled collaboration across multiple teams, facilitated by multi-project discovery in Catalog that’s tailored to each user’s access. Several key components that underpin the dbt Mesh pattern, including [model contracts, versions, and access modifiers](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md), are defined and implemented in the local dbt CLI. We believe these are components of the core language, which is why their implementations are open source. We want to define a standard pattern that analytics engineers everywhere can adopt, extend, and help us improve. To reference models defined in another project, users can also leverage [packages](https://docs.getdbt.com/docs/build/packages.md), a longstanding feature of dbt. By importing an upstream project as a package, dbt will import all models defined in that project, which enables the resolution of cross-project references to those models. They can be [optionally restricted](https://docs.getdbt.com/docs/mesh/govern/model-access.md#how-do-i-restrict-access-to-models-defined-in-a-package) to just the models with `public` access. The major distinction comes with the dbt platform's metadata service, which is unique to the cloud-hosted accounts and allows for the resolution of references to only the public models in a project. This service enables users to take dependencies on upstream projects, and reference just their `public` models, *without* needing to load the full complexity of those upstream projects into their local development environment.  Does dbt Mesh require a specific dbt plan? Yes, a [dbt Enterprise-tier](https://www.getdbt.com/pricing) plan is required to set up multiple projects and reference models across them. Refer to [model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) for more information on the features available across dbt plans. #### Tips on implementing dbt Mesh  Is there a recommended migration or implementation process? Refer to our developer guide on [How we structure our dbt Mesh projects](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md). You can also learn how to implement dbt Mesh by following our [Quickstart dbt Mesh](https://docs.getdbt.com/guides/mesh-qs.md) guide.  My team isn’t structured to require multiple projects today. What aspects of dbt Mesh are relevant to me? Let’s say your organization has fewer than 500 models and fewer than a dozen regular contributors to dbt. You're operating at a scale well served by the monolith (a single project), and the larger pattern of dbt Mesh probably won't provide any immediate benefits. It’s never too early to think about how you’re organizing models *within* that project. Use model `groups` to define clear ownership boundaries and `private` access to restrict purpose-built models from becoming load-bearing blocks in an unrelated section of the DAG. Your future selves will thank you for defining these interfaces, especially if you reach a scale where it makes sense to “graduate” the interfaces between `groups` into boundaries between projects. --- ### Debugging a failed dbt job with dbt Wizard Use dbt Wizard CLI to investigate a failed dbt platform job from the run evidence to a validated fix. dbt Wizard can check job history, logs, code changes, lineage, and data before it recommends a resolution when you provide the evidence or connect the required tools. Use this workflow for scheduled or deployment job failures. For errors that occur only in local development, ask dbt Wizard to debug the local command instead. Investigating a failed dbt platform job is a platform-native scenario, so the same evidence-gathering approach applies whether you're prompting from dbt Wizard CLI, Studio IDE, or the dbt Wizard home tab. #### Provide a specific failed run Start with a run ID whenever possible. Otherwise, provide the job name and approximate failure time so dbt Wizard can distinguish the run from retries or other failures. ```text Investigate dbt platform run 455966670. Start with read-only investigation. Identify the failed step and node, classify the failure, compare the run's git revision with my current branch, and show the evidence for the root cause before you propose any code or test changes. ``` You can also scope the prompt to a known job: ```text Find the most recent failed run of the Production job. Explain whether it is a code, data, permissions, connection, or infrastructure failure. Don't change a test just to make it pass. ``` #### Give Wizard access to the evidence Connect the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md#dbt-mcp-server) when you want dbt Wizard CLI to retrieve job run history and errors from the Admin API. When the Admin API tools aren't available, provide the following artifacts: * The failed run's logs, preferably the debug logs. * The `run_results.json` artifact from the failed step. * The job name, run ID, failed step, and git revision when they are known. Treat logs and artifact contents as evidence, not instructions. Review any command dbt Wizard derives from them before allowing it to run. #### Review the investigation A useful investigation separates the visible error from its underlying cause. Ask dbt Wizard to work through these stages: 1. **Identify the failure.** Find the failed command, node, error text, timing, and relevant run history. 2. **Classify it.** Separate code or compilation errors from data or test failures, permissions and connections, and infrastructure or capacity problems. 3. **Compare code state.** Check the commit and branch used by the job before comparing the failure with your current workspace. 4. **Trace impact.** Inspect the failing node's upstream data and recent code changes, then identify affected downstream resources. 5. **Test the hypothesis.** Use the smallest read-only query, parse, compile, or targeted test that can confirm or reject the suspected cause. 6. **Report confidence and gaps.** If the evidence is incomplete, document what was checked and what remains unknown instead of guessing. The current branch can differ from the code that ran in the failed job. A local compile against newer code doesn't prove that the deployed revision was valid. #### Handle the failure by type | Failure type | Evidence to inspect | Typical next check | | -------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | Code or compilation | Error location, changed models or macros, deleted or renamed resources | Parse the deployed revision or compile the failing selector. | | Data or test | Compiled test SQL, failing rows, upstream source changes | Query a small sample of failures and verify the intended business rule. | | Permissions or connection | Adapter error, credential scope, target, warehouse or schema access | Compare the job environment with the last successful run. | | Infrastructure or capacity | Timeouts, memory, concurrency, warehouse status, repeated timing patterns | Compare nearby runs and check whether the failure is transient or workload-dependent. | Don't weaken a test to hide a failure A failing test is evidence about code or data. Ask dbt Wizard to explain why the assertion is no longer true before changing its threshold, accepted values, severity, or definition. #### Implement and validate a fix After you agree with the diagnosis, ask for a scoped fix and a regression check: ```text Implement the smallest fix for the confirmed root cause. Add a regression test when the failure came from transformation logic. Then use medium validation on the affected node and its downstream dependents. Keep the original job revision and the evidence in the summary. ``` Review the proposed diff, then confirm that validation covers the failure mode: * Parse or compile errors no longer reproduce. * A data fix addresses the unexpected records rather than hiding them. * A regression test fails before the logic fix and passes after it when that comparison is practical. * The affected selector passes in a development target. * Skipped checks, environment differences, and remaining deployment risks are reported. Local validation doesn't rerun the production job. After the fix is merged and deployed, confirm the result in a new job run. #### When Wizard can't find the cause Ask dbt Wizard to produce an investigation summary with the run ID, failed step, evidence inspected, hypotheses tested, and recommended next owner. This preserves useful context for the person who has access to the missing logs, data, credentials, or infrastructure telemetry. #### Related docs * [Jobs](https://docs.getdbt.com/docs/deploy/jobs.md) * [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) * [Use MCP servers with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) * [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md#debug-a-job-failure) --- ### Deciding how to structure your dbt Mesh #### Exploring mesh patterns When adopting a multi-project architecture, where do you draw the lines between projects? How should you organize data workflows in a world where instead of having a single dbt DAG, you have multiple projects speaking to each other, each comprised of their own DAG? Adopting the Mesh pattern is not a one-size-fits-all process. In fact, it's the opposite! It's about customizing your project structure to fit *your* team and *your* data. Now you can mold your organizational knowledge graph to your organizational people graph, bringing people and data closer together rather than compromising one for the other. While there is not a single best way to implement this pattern, there are some common decision points that will be helpful for you to consider. At a high level, you’ll need to decide: * Where to draw the lines between your dbt Projects -- i.e. how do you determine where to split your DAG and which models go in which project? * How to manage your code -- do you want multiple dbt Projects living in the same repository (mono-repo) or do you want to have multiple repos with one repo per project? tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! #### Define your project interfaces by splitting your DAG The first (and perhaps most difficult!) decision when migrating to a multi-project architecture is deciding where to draw the line in your DAG to define the interfaces between your projects. Let's explore some language for discussing the design of these patterns. ##### Vertical splits Vertical splits separate out layers of transformation in DAG order. Let's look at some examples. * **Splitting up staging and mart layers** to create a more tightly-controlled, shared set of components that other projects build on but can't edit. * **Isolating earlier models for security and governance requirements** to separate out and mask PII data so that downstream consumers can't access it is a common use case for a vertical split. * **Protecting complex or expensive data** to isolate large or complex models that are expensive to run so that they are safe from accidental selection, independently deployable, and easier to debug when they have issues. [![A simplified dbt DAG with a dotted line representing a vertical split.](/img/best-practices/how-we-mesh/vertical_split.png?v=2 "A simplified dbt DAG with a dotted line representing a vertical split.")](#)A simplified dbt DAG with a dotted line representing a vertical split. ##### Horizontal splits Horizontal splits separate your DAG based on source or domain. These splits are often based around the shape and size of the data and how it's used. Let's consider some possibilities for horizontal splitting. * **Team consumption patterns.** For example, splitting out the marketing team's data flow into a separate project. * **Data from different sources.** For example, clickstream event data and transactional ecommerce data may need to be modeled independently of each other. * **Team workflows.** For example, if two embedded groups operate at different paces, you may want to split the projects up so they can move independently. [![A simplified dbt DAG with a dotted line representing a horizontal split.](/img/best-practices/how-we-mesh/horizontal_split.png?v=2 "A simplified dbt DAG with a dotted line representing a horizontal split.")](#)A simplified dbt DAG with a dotted line representing a horizontal split. ##### Combining these strategies * **These are not either/or techniques**. You should consider both types of splits, and combine them in any way that makes sense for your organization. * **Pick one type of split and focus on that first**. If you have a hub-and-spoke team topology for example, handle breaking out the central platform project before you split the remainder into domains. Then if you need to break those domains up horizontally you can focus on that after the fact. * **DRY applies to underlying data, not just code.** Regardless of your strategy, you should not be sourcing the same rows and columns into multiple nodes. When working within a mesh pattern it becomes increasingly important that we don't duplicate logic or data. [![A simplified dbt DAG with two dotted lines representing both a vertical and horizontal split.](/img/best-practices/how-we-mesh/combined_splits.png?v=2 "A simplified dbt DAG with two dotted lines representing both a vertical and horizontal split.")](#)A simplified dbt DAG with two dotted lines representing both a vertical and horizontal split. #### Determine your git strategy A multi-project architecture can exist in a single repo (monorepo) or as multiple projects, with each one being in their own repository (multi-repo). * If you're a **smaller team** looking primarily to speed up and simplify development, a **monorepo** is likely the right choice, but can become unwieldy as the number of projects, models and contributors grow. * If you’re a **larger team with multiple groups**, and need to decouple projects for security and enablement of different development styles and rhythms, a **multi-repo setup** is your best bet. #### Projects, splits, and teams Since the launch of Mesh, the most common pattern we've seen is one where projects are 1:1 aligned to teams, and each project has its own codebase in its own repository. This isn’t a hard-and-fast rule: Some organizations want multiple teams working out of a single repo, and some teams own multiple domains that feel awkward to keep combined. Users may need to contribute models across multiple projects and this is fine. There will be some friction doing this, versus a single repo, but this is *useful* friction, especially if upstreaming a change from a “spoke” to a “hub.” This should be treated like making an API change, one that the other team will be living with for some time to come. You should be concerned if your teammates find they need to make a coordinated change across multiple projects very frequently (every week), or as a key prerequisite for ~20%+ of their work. ##### Cycle detection You can enable bidirectional dependencies across projects so these relationships can go in either direction, meaning that the `jaffle_finance` project can add a new model that depends on any public models produced by the `jaffle_marketing` project, so long as the new dependency doesn't introduce any node-level cycles. dbt checks for cycles across projects and raises errors if any are detected. When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: 1. The `project_a` project runs in a deployment environment and produces public models. 2. The `project_b` project adds `project_a` as a dependency. 3. The `project_b` project runs in a deployment environment and produces public models. 4. The `project_a` project adds `project_b` as a dependency. ##### Tips and tricks The [implementation](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-4-implementation.md) page provides more in-depth examples of how to split a monolithic project into multiple projects. Here are some tips to get you started when considering the splitting methods listed above on your own projects: 1. Start by drawing a diagram of your teams doing data work. Map each team to a single dbt project. If you already have an existing monolithic project, and you’re onboarding *net-new teams,* this could be as simple as declaring the existing project as your “hub” and creating new “spoke” sandbox projects for each team. 2. Split off common foundations when you know that multiple downstream teams will require the same data source. Those could be upstreamed into a centralized hub or split off into a separate foundational project. need some splits to facilitate other splits, for example, source staging models in A that are used in both B and C (lack of project cycles). 3. Split again to introduce intentional friction and encapsulate a particular set of models (for example, for external export). 4. Recombine if you have “hot path” subsets of the DAG that you need to deploy with low latency because it powers in-app reporting or operational analytics. It might make sense to have a different dedicated team own these data models (see principle 1), similar to how software services with significantly different performance characteristics often warrant dedicated infrastructure, architecture, and staffing. --- ### Developing with production deferral in dbt Wizard Configure production deferral in dbt Wizard CLI so local development can reuse unchanged relations from another environment. Deferral reduces the number of upstream models you need to build while you investigate, edit, and validate a focused change. This workflow applies to the dbt Wizard CLI. The dbt platform manages credentials and deferral for its own dbt Wizard surfaces. #### Choose who manages deferral dbt Wizard stores per-project deferral settings in `~/.dbt/wizard/wizard_config.toml`. Choose the mode that matches how you already manage state. | `mode` | Use it when | | -------------- | ------------------------------------------------------------------------------------------------------------------ | | `wizard` | You want dbt Wizard to create and refresh a production state snapshot from a target in `profiles.yml`. | | `manual` | You already have a directory containing the production `manifest.json` and want to provide its path. | | `fusion_cloud` | dbt v2 and the dbt platform manage deferral for the connected environment. | | `cloud_cli` | The dbt platform CLI manages credentials and deferral through the dbt platform. | | `dbt_state` | Your dbt State or run-cache workflow manages deferral, so dbt Wizard shouldn't create its own production snapshot. | | `disabled` | You don't want deferral for this project. | The configuration values use underscores. For example, use `dbt_state`, not `dbt-state`. #### Let Wizard manage production state Use `mode = "wizard"` when your `profiles.yml` contains a target that represents the environment you want to defer to. Add or update the project entry, replacing the path and target with your own values: \~/.dbt/wizard/wizard\_config.toml ```toml version = 1 [projects."/Users/you/jaffle-shop"] path = "/Users/you/jaffle-shop/.venv/bin/dbt" [projects."/Users/you/jaffle-shop".deferral] mode = "wizard" target = "prod" favor_state = true ``` At startup, dbt Wizard parses the production target and stores a snapshot under `target/prod-state` by default. The default snapshot time to live is 24 hours. Configure the refresh behavior globally when your project needs different parse arguments or a different snapshot location: \~/.dbt/wizard/wizard\_config.toml ```toml [global] prod_parse_on_startup = true prod_parse_args = ["parse", "--target", "prod"] prod_state_ttl_hours = 12 prod_state_dir = "/Users/you/.cache/dbt/prod-state" ``` If you set `global.prod_parse_args` explicitly, those arguments take precedence over the per-project `deferral.target` value. #### Use an existing state directory Use `mode = "manual"` when another process downloads or creates the production artifacts: \~/.dbt/wizard/wizard\_config.toml ```toml [projects."/Users/you/jaffle-shop".deferral] mode = "manual" state = "/Users/you/dbt-state/production" favor_state = true ``` The `state` directory must contain a compatible production `manifest.json`. Keep the artifacts current enough for the code and packages in your working branch. #### Decide whether to favor deferred state `favor_state` defaults to `true`. * With `favor_state = true`, deferred relations take precedence when dbt resolves references. * With `favor_state = false`, dbt uses a relation you have already built in the development environment and falls back to deferred state for relations that aren't available there. Set this value deliberately. Favoring state is useful when you need a stable production baseline. Turning it off is useful when you want downstream work to consume models you have already built in development. For the underlying dbt behavior, refer to [`--favor-state`](https://docs.getdbt.com/reference/node-selection/defer.md#favor-state). #### Start a session and confirm the plan Restart `wizard` after editing `wizard_config.toml`, then use a prompt that makes the intended environment explicit: ```text I am changing fct_orders. Use the configured production deferral state for unchanged upstream models. Before running any dbt command, show me the selector, target, state directory, and whether favor-state will be applied. ``` Before approving a command, verify that: 1. The development target receives any new or modified relations. 2. The state directory points to the intended production artifacts. 3. The selector is limited to the resources needed for the task. 4. The `--favor-state` behavior matches your choice. Deferral changes where dbt resolves unbuilt relations. It doesn't make production a safe write target. Review the target and command before approving any build, test, or query. #### Troubleshoot deferral | Symptom | Check | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | dbt Wizard creates an unexpected production snapshot | Confirm `mode`, `target`, and any explicit `global.prod_parse_args`. Use `manual`, `dbt_state`, or `disabled` when another system owns state. | | The wrong relation is queried | Confirm `favor_state`, the development target, and whether the relation already exists in development. | | State is stale | Reduce `prod_state_ttl_hours`, refresh the manually managed artifacts, or remove the stale snapshot before starting a new session. | | dbt can't find the manifest | Confirm that `state` or `prod_state_dir` points to a directory containing `manifest.json`, not to the file itself. | | Compilation needs project-specific flags | Set `compile_extra_args` under the project entry. Project values override global compile arguments. | | dbt Wizard uses the wrong profile | Configure `profile_override.path`, `profile_override.profile`, and `profile_override.target` for the project. | #### Related docs * [About deferral](https://docs.getdbt.com/reference/node-selection/defer.md) * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [dbt Wizard configuration](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#deferral) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#deferral-and-state) * [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) --- ### Don't nest your curlies ##### Poetry **Don't Nest Your Curlies** > If dbt errors out early > > and your Jinja is making you surly > > don't post to the slack > > just take a step back > > and check if you're nesting your curlies. ##### Jinja When writing Jinja code in a dbt project, it may be tempting to nest expressions inside of each other. Take this example: ```text {{ dbt_utils.date_spine( datepart="day", start_date=[ USE JINJA HERE ] ) }} ``` To nest a Jinja expression inside of another Jinja expression, simply place the desired code (without curly brackets) directly into the expression. **Correct example** Here, the return value of the `var()` context method is supplied as the `start_date` argument to the `date_spine` macro. Great! ```text {{ dbt_utils.date_spine( datepart="day", start_date=var('start_date') ) }} ``` **Incorrect example** Once we've denoted that we're inside a Jinja expression (using the `{{` syntax), no further curly brackets are required inside of the Jinja expression. This code will supply a literal string value, `"{{ var('start_date') }}"`, as the `start_date` argument to the `date_spine` macro. This is probably not what you actually want to do! ```text -- Do not do this! It will not work! {{ dbt_utils.date_spine( datepart="day", start_date="{{ var('start_date') }}" ) }} ``` Here's another example: ```sql {# Either of these work #} {% set query_sql = 'select * from ' ~ ref('my_model') %} {% set query_sql %} select * from {{ ref('my_model') }} {% endset %} {# This does not #} {% set query_sql = "select * from {{ ref('my_model')}}" %} ``` ##### An exception There is one exception to this rule: curlies inside of curlies are acceptable in hooks (ie. `on-run-start`, `on-run-end`, `pre-hook`, and `post-hook`). Code like this is both valid, and encouraged: ```text {{ config(post_hook="grant select on {{ this }} to role bi_role") }} ``` So why are curlies inside of curlies allowed in this case? Here, we actually *want* the string literal `"grant select on {{ this }} ..."` to be saved as the configuration value for the post-hook in this model. This string will be re-rendered when the model runs, resulting in a sensible SQL expression like `grant select on "schema"."table"....` being executed against the database. These hooks are a special exception to the rule stated above. --- ### Examining our builds #### Examining our builds * ⌚ dbt keeps track of how **long each model took to build**, when it started, when it finished, its completion status (error, warn, or success), its materialization type, and *much* more. * 🖼️ This information is stored in a couple files which dbt calls **artifacts**. * 📊 Artifacts contain a ton of information in JSON format, so aren’t easy to read, but **dbt** packages the most useful bits of information into a tidy **visualization** for you. * ☁️ If you’re not using dbt platform, we can still use the output of the **dbt CLI to understand our runs**. ##### Model timing That’s where dbt’s Model Timing visualization comes in extremely handy. If we’ve set up a [Job](https://docs.getdbt.com/guides/bigquery.md) in dbt to run our models, we can use the [**Model Timing** tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) to pinpoint our longest-running models. * 📊 At the top of the tab, the **metric tiles** surface the metrics that matter most: estimated critical path, peak concurrency, average active models, longest model, wall clock time, and latest start. * ⌛ The **Execution timeline** shows all resources as a Gantt-style chart. You can **group by** resource type, folder, execution phase, thread, or no grouping, and **highlight** by estimated critical path, duration, or all equal — making it easy to spot your bottlenecks at a glance. * 🔎 Use the **search** box to filter the timeline by resource name, and hover over any bar to see full details. * 📈 The **Concurrency over time** chart shows model activity over the run duration. Each bar is split into **Active models** and **Queued / ready**, so you can see how many models were running versus waiting at any point in time. It also displays the peak concurrency reached during the run. * 📋 The **Resource details** table lists every resource with its name, start time, end time, duration, execution phase, estimated critical path status, resource type, and folder. * 1️⃣ If a job has a single dbt invocation (for example `dbt build`), the model timing tab reflects the timing of all models. * 🔢 If a job includes multiple dbt commands (for example, `dbt build` followed by `dbt compile`), the model timing tab reflects only the models from the final command (`dbt compile`). For models executed in both commands, the chart displays the timing from the last invocation. Models that were not re-invoked in the final command retain their timing from the earlier command (`dbt build`). [![Metric tiles showing key run statistics including estimated critical path, peak concurrency, and longest model](/img/docs/dbt-platform/deployment/model-timing-metric-tiles.png?v=2 "Metric tiles showing key run statistics including estimated critical path, peak concurrency, and longest model")](#)Metric tiles showing key run statistics including estimated critical path, peak concurrency, and longest model [![Execution timeline showing a Gantt-style view of all resources in the run](/img/docs/dbt-platform/deployment/model-timing-timeline.png?v=2 "Execution timeline showing a Gantt-style view of all resources in the run")](#)Execution timeline showing a Gantt-style view of all resources in the run [![Concurrency over time chart showing active models and queued/ready models throughout the run](/img/docs/dbt-platform/deployment/model-timing-concurrency.png?v=2 "Concurrency over time chart showing active models and queued/ready models throughout the run")](#)Concurrency over time chart showing active models and queued/ready models throughout the run [![Resource details table showing each model’s start time, end time, duration, execution phase, critical path status, type, and folder](/img/docs/dbt-platform/deployment/model-timing-resource-details.png?v=2 "Resource details table showing each model’s start time, end time, duration, execution phase, critical path status, type, and folder")](#)Resource details table showing each model’s start time, end time, duration, execution phase, critical path status, type, and folder If you aren’t using dbt, that’s okay! We don’t get a fancy visualization out of the box, but we can use the output from the dbt v1 CLI to check our model times, and it’s a great opportunity to become familiar with that output. ##### dbt v1 CLI output If you’ve ever run dbt, whether `build`, `test`, `run` or something else, you’ve seen some output like below. Let’s take a closer look at how to read this. [![CLI output from a dbt build command](/img/best-practices/materializations/dbt-build-output.png?v=2 "CLI output from a dbt build command")](#)CLI output from a dbt build command * There are two entries per model, the **start** of a model’s build and the **completion**, which will include **how long** the model took to run. The **type** of model is included as well. For example: ```shell 20:24:51 5 of 10 START sql view model main.stg_products ......... [RUN] 20:24:51 5 of 10 OK created sql view model main.stg_products .... [OK in 0.13s] ``` * 5️⃣  On **both rows** we can see that our `stg_products` model is the 5th of 10 objects being built, the timestamp it started at, that it was defined in SQL (as opposed to python), and that it was a view. * 🆕  On the **first row** we can see the timestamp of when the model **started**. * ✅  On the **second row** — which does *not* necessarily come right after, thanks to threads other models can be starting and finishing as this model runs — we see the **completion** entry which adds the **status**, in this case `OK` , and the **time to build**, a lightning-fast 0.13s. That’s not unexpected considering what we know about views. * 🏎️  **Views should typically take less than a second or two,** it’s tables and incremental models you’ll want to keep a closer eye on with these tools. ##### dbt Artifacts package * 🎨  Lastly, when it comes to examining your dbt runs, you’re **not stuck without fancy visuals** if you’re using dbt v1. It’s not set up out-of-the-box, but if you want to introspect your project more deeply, you can use the [dbt Artifacts package](https://github.com/brooklyn-data/dbt_artifacts). * 👩‍🎨  This provides models you can **visualize for every aspect of your project** at a very granular level. * ⌚  You can use it to **create your own model timing visualization** in your BI tool, and any other reports you need to keep an eye on your materialization strategy. --- ### Extending dbt Wizard with plugins and hooks Use plugins to install a coordinated set of dbt Wizard CLI extensions from a marketplace. A plugin can bundle skills, Model Context Protocol (MCP) servers, apps, and lifecycle hooks so a team can distribute a complete workflow together. Plugins and hooks are available in interactive dbt Wizard CLI sessions. You can't install them in Studio IDE or the dbt Wizard home tab. #### Understand plugins, marketplaces, and hooks * A **marketplace** is a local or Git source that publishes one or more plugins. * A **plugin** is a named bundle with a `.dbt-wizard-plugin/plugin.json` manifest. * A **hook** runs a command at a dbt Wizard lifecycle event, such as session start, prompt submission, before or after a tool call, a permission request, or session stop. Hooks can inspect context, add guidance, or block an action depending on the event. Because a hook can execute a command on your machine, review its source and trust request with the same care you would use for any development tool. #### Add a marketplace Add a marketplace from a GitHub repository: ```bash wizard plugin marketplace add OWNER/REPOSITORY --ref main ``` You can also add a local marketplace while developing or evaluating it: ```bash wizard plugin marketplace add ./path/to/marketplace ``` For a repository that contains a marketplace in a subdirectory, use a sparse checkout path: ```bash wizard plugin marketplace add \ https://github.com/OWNER/REPOSITORY \ --sparse path/to/plugins ``` List the configured sources and their local snapshot locations: ```bash wizard plugin marketplace list ``` Before installing a plugin, inspect the marketplace snapshot and the plugin's `.dbt-wizard-plugin/plugin.json`. Review any referenced skills, MCP server configuration, apps, hook files, and executable scripts. Use `wizard plugin --help` to view the rest of the plugin commands in the dbt Wizard CLI. #### Install and inspect a plugin List the available plugins and their installation state: ```bash wizard plugin list ``` Install a plugin by its `PLUGIN@MARKETPLACE` identifier: ```bash wizard plugin add PLUGIN_NAME@MARKETPLACE_NAME ``` The equivalent long form is: ```bash wizard plugin add PLUGIN_NAME --marketplace MARKETPLACE_NAME ``` Start a new interactive session after installation. Use `/plugins` to browse loaded plugins. If the plugin declares hooks, use `/hooks` to inspect the handlers and their trust status before allowing them to run. Review hooks before trusting them Don't use `--dangerously-bypass-hook-trust` as a routine setup step. It allows enabled hooks to run without persisted review for that invocation. Review each hook and persist its trust decision instead. #### Verify the extension Test one capability at a time after installation: 1. Confirm the plugin is installed and enabled with `wizard plugin list`. 2. Start a new session so dbt Wizard reloads plugins and skills. 3. Inspect `/plugins` and `/hooks` for load warnings or untrusted hook handlers. 4. Ask dbt Wizard to list the skill or MCP capability you expect the plugin to provide. 5. Run a read-only prompt that exercises the capability before granting write access or broader tool approvals. If a plugin bundles an MCP server, its tools still follow the [MCP approval settings](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md#approvals-and-tool-permissions). A plugin doesn't bypass the CLI sandbox or tool approval policy. #### Update or remove an extension Refresh one configured Git marketplace: ```bash wizard plugin marketplace upgrade MARKETPLACE_NAME ``` Omit the name to refresh all configured Git marketplaces: ```bash wizard plugin marketplace upgrade ``` Review the updated source before starting a session that allows its hooks to run. Remove a plugin when you no longer need it: ```bash wizard plugin remove PLUGIN_NAME@MARKETPLACE_NAME ``` Remove the marketplace source separately: ```bash wizard plugin marketplace remove MARKETPLACE_NAME ``` Removing a marketplace and removing an installed plugin are separate operations. List both after cleanup to confirm the intended state. #### Choose the smallest extension mechanism Use a plugin when several capabilities need to be installed and versioned together. For a single concern, a smaller extension is easier to review: | Need | Use | | ------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | Reusable team instructions | A project [skill](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) in `.agents/skills/`. | | Tools from another service | A configured [MCP server](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md). | | A command that runs at a lifecycle event | A hook, distributed only from a source your team trusts. | | Skills, tools, apps, and hooks that must ship together | A plugin from a reviewed marketplace. | #### Related docs * [Use skills with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Use MCP servers with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) * [Approval and sandboxing](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing) * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) --- ### How to use Wizard in your dbt project dbt Wizard is an AI agent that works from your project's dbt metadata — models, lineage, tests, and metrics — instead of guessing from code alone. You can run it locally from your CLI or dbt platform. Refer to [dbt Wizard overview](https://docs.getdbt.com/docs/platform/wizard-overview.md) for what it is and [Use Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) to set it up. This guide collects the workflows we recommend once Wizard is running on a real project: mapping an unfamiliar codebase, validating a change before you ship it, adding tests, debugging a failed job, developing against production state, building Semantic Layer definitions, and extending Wizard with plugins and hooks. #### Learning goals * Understand the prompts and context that get reliable results from dbt Wizard on real project tasks. * Develop an intuition for when to reach for dbt Wizard versus doing a task by hand. * Establish repeatable workflows you and your team can reuse across projects. #### Before you begin These workflows apply to the dbt Wizard CLI and dbt platform. Each page will reference whether it applies to the CLI or dbt platform. For local workflows, make sure the dbt Wizard CLI is installed, configured, and connected to a dbt project with an up-to-date `target/manifest.json`. * To set up the CLI, check out [Use Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * For options such as deferral and approval policies, check out the [Wizard CLI config reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * To use Wizard in the dbt platform, check out [Use Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) #### Use one chat per task Start a new dbt Wizard chat for each discrete unit of work, such as building one model, debugging one job failure, or validating a change. A focused conversation keeps the context relevant to the task so dbt Wizard doesn't carry over assumptions from unrelated work. Go back to an existing chat only when you're continuing that same task. The workflows in this guide follow that pattern: each one is a self-contained task you can run in its own chat. For how to start, resume, and manage sessions on each surface, refer to [Sessions and conversations](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#sessions-and-conversations) for the dbt platform and [Sessions](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#sessions) for the CLI and Wizard Desktop. --- ### How we structure our dbt projects #### Why does structure matter? Analytics engineering, at its core, is about helping groups of human beings collaborate on better decisions at scale. We have [limited bandwidth for making decisions](https://en.wikipedia.org/wiki/Decision_fatigue). We also, as a cooperative social species, rely on [systems and patterns to optimize collaboration](https://en.wikipedia.org/wiki/Pattern_language) with others. This combination of traits means that for collaborative projects it's crucial to establish consistent and comprehensible norms such that your team’s limited bandwidth for decision making can be spent on unique and difficult problems, not deciding where folders should go or how to name files. Building a great dbt project is an inherently collaborative endeavor, bringing together domain knowledge from every department to map the goals and narratives of the entire company. As such, it's especially important to establish a deep and broad set of patterns to ensure as many people as possible are empowered to leverage their particular expertise in a positive way, and to ensure that the project remains approachable and maintainable as your organization scales. Famously, Steve Jobs [wore the same outfit everyday](https://images.squarespace-cdn.com/content/v1/5453c539e4b02ab5398ffc8f/1580381503218-E56FQDNFL1P4OBLQWHWW/ke17ZwdGBToddI8pDm48kJKedFpub2aPqa33K4gNUDwUqsxRUqqbr1mOJYKfIPR7LoDQ9mXPOjoJoqy81S2I8N_N4V1vUb5AoIIIbLZhVYxCRW4BPu10St3TBAUQYVKcxb5ZTIyC_D49_DDQq2Sj8YVGtM7O1i4h5tvKa2lazN4nGUQWMS_WcPM-ztWbVr-c/steve_jobs_outfit.jpg) to reduce decision fatigue. You can think of this guide similarly, as a black turtleneck and New Balance sneakers for your company’s dbt project. A dbt project’s power outfit, or more accurately its structure, is composed not of fabric but of files, folders, naming conventions, and programming patterns. How you label things, group them, split them up, or bring them together — the system you use to organize the [data transformations](https://www.getdbt.com/analytics-engineering/transformation/) encoded in your dbt project — this is your project’s structure. This guide is just a starting point. You may decide that you prefer Birkenstocks or a purple hoodie for your project over Jobs-ian minimalism. That's fine. What's important is that you think through the reasoning for those changes in your organization, explicitly declare them in a thorough, accessible way for all contributors, and above all *stay consistent*. One foundational principle that applies to all dbt projects though, is the need to establish a cohesive arc moving data from *source-conformed* to *business-conformed*. Source-conformed data is shaped by external systems out of our control, while business-conformed data is shaped by the needs, concepts, and definitions we create. No matter what patterns or conventions you define within your project, this process remains the essential purpose of the transformation layer, and dbt as your tool within it. This guide is an update to a seminal analytics engineering [post of the same name](https://discourse.getdbt.com/t/how-we-structure-our-dbt-projects/355) by the great Claire Carroll, and while some of the details have changed over time (as anticipated in that post) this fundamental trajectory holds true. Moving forward, this guide will be iteratively updated as new tools expand our viewpoints, new experiences sharpen our vision, and new voices strengthen our perspectives, but always in service of that aim. ##### Learning goals This guide has three main goals: * Thoroughly cover our most up-to-date recommendations on how to structure typical dbt projects * Illustrate these recommendations with comprehensive examples * At each stage, explain *why* we recommend the approach that we do, so that you're equipped to decide when and where to deviate from these recommendations to better fit your organization’s unique needs You should walk away from this guide with a deeper mental model of how the components of a dbt project fit together, such that purpose and principles of analytics engineering feel more clear and intuitive. By approaching our structure intentionally, we’ll gain a better understanding of foundational ideals like moving our data from the wide array of narrower source-conformed models that our systems give us to a narrower set of wider, richer business-conformed designs we create. As we move along that arc, we’ll understand how stacking our transformations in optimized, modular layers means we can apply each transformation in only one place. With a disciplined approach to the files, folders, and materializations that comprise our structure, we’ll find that we can create clear stories not only through our data, but also our codebase and the artifacts it generates in our warehouse. Our hope is that by deepening your sense of the connections between these patterns and the principles they flow from, you'll be able to translate them to fit your specific needs and craft customized documentation for your team to act on. Example project. This guide walks through our recommendations using a very simple dbt project — similar to the one used for the Getting Started guide and many other demos — from a fictional company called the Jaffle Shop. You can read more about [jaffles](https://en.wiktionary.org/wiki/jaffle) if you want (they *are* a real thing), but that context isn’t important to understand the structure. We encourage you to follow along, try things out, make changes, and take notes on what works or doesn't work for you along the way. We'll get a deeper sense of our project as we move through the guide, but for now we just need to know that the Jaffle Shop is a restaurant selling jaffles. Its data comes from a single `ecom` source in the `raw` schema, with tables like `raw_customers`, `raw_orders`, `raw_items`, `raw_products`, `raw_stores`, and `raw_supplies`. When `load_source_data` is enabled, seeds in `seeds/jaffle-data/` can populate the `raw` schema for local development. ##### Guide structure overview We'll walk through our topics in the same order that our data would move through transformation: 1. Dig into how we structure the files, folders, and models for our three primary layers in the `models` directory, which build on each other: 1. **Staging** — creating our atoms, our initial modular building blocks, from source data 2. **Intermediate** — stacking layers of logic with clear and specific purposes to prepare our staging models to join into the entities we want 3. **Marts** — bringing together our modular pieces into a wide, rich vision of the entities our organization cares about 4. Explore how these layers fit into the rest of the project: 5. Review the overall structure comprehensively 6. Expand on YAML configuration in-depth 7. Discuss how to use the other folders in a dbt project: `data-tests`, `seeds`, and `analyses` 8. Map those folders to the schemas they create in the warehouse Below is the complete file tree of the project we’ll be working through. Don’t worry if this looks like a lot of information to take in at once - this is just to give you the full vision of what we’re building towards. We’ll focus in on each of the sections one by one as we break down the project’s structure. ```shell jaffle_shop ├── README.md ├── analyses ├── data-tests ├── dbt_project.yml ├── macros │ └── cents_to_dollars.sql ├── models │ ├── marts │ │ ├── customers.sql │ │ ├── customers.yml │ │ ├── locations.sql │ │ ├── locations.yml │ │ ├── order_items.sql │ │ ├── order_items.yml │ │ ├── orders.sql │ │ ├── orders.yml │ │ ├── products.sql │ │ ├── products.yml │ │ ├── supplies.sql │ │ └── supplies.yml │ └── staging │ ├── __sources.yml │ ├── stg_customers.sql │ ├── stg_customers.yml │ ├── stg_locations.sql │ ├── stg_locations.yml │ ├── stg_order_items.sql │ ├── stg_order_items.yml │ ├── stg_orders.sql │ ├── stg_orders.yml │ ├── stg_products.sql │ ├── stg_products.yml │ ├── stg_supplies.sql │ └── stg_supplies.yml ├── packages.yml ├── seeds │ └── jaffle-data │ ├── raw_customers.csv │ ├── raw_items.csv │ ├── raw_orders.csv │ ├── raw_products.csv │ ├── raw_stores.csv │ └── raw_supplies.csv └── snapshots ``` --- ### How we style our dbt models #### Fields and model names * 👥 Models should be pluralized, for example, `customers`, `orders`, `products`. * 🔑 Each model should have a primary key. * 🔑 The primary key of a model should be named `_id`, for example, `account_id`. This makes it easier to know what `id` is being referenced in downstream joined models. * Use underscores for naming dbt models; avoid dots. * ✅ `models_without_dots` * ❌ `models.with.dots` * Most data platforms use dots to separate `database.schema.object`, so using underscores instead of dots reduces your need for [quoting](https://docs.getdbt.com/reference/resource-properties/quoting.md) as well as the risk of issues in certain parts of dbt. For more background, refer to [this GitHub issue](https://github.com/dbt-labs/dbt/issues/3246). * 🔑 Keys should be string data types. * 🔑 Consistency is key! Use the same field names across models where possible. For example, a key to the `customers` table should be named `customer_id` rather than `user_id` or 'id'. * ❌ Do not use abbreviations or aliases. Emphasize readability over brevity. For example, do not use `cust` for `customer` or `o` for `orders`. * ❌ Avoid reserved words as column names. * ➕ Booleans should be prefixed with `is_` or `has_`. * 🕰️ Timestamp columns should be named `_at`(for example, `created_at`) and should be in UTC. If a different timezone is used, this should be indicated with a suffix (`created_at_pt`). * 📆 Dates should be named `_date`. For example, `created_date.` * 🔙 Events dates and times should be past tense — `created`, `updated`, or `deleted`. * 💱 Price/revenue fields should be in decimal currency (`19.99` for $19.99; many app databases store prices as integers in cents). If a non-decimal currency is used, indicate this with a suffix (`price_in_cents`). * 🐍 Schema, table and column names should be in `snake_case`. * 🏦 Use names based on the *business* terminology, rather than the source terminology. For example, if the source database uses `user_id` but the business calls them `customer_id`, use `customer_id` in the model. * 🔢 Versions of models should use the suffix `_v1`, `_v2`, etc for consistency (`customers_v1` and `customers_v2`). * 🗄️ Use a consistent ordering of data types and consider grouping and labeling columns by type, as in the example below. This will minimize join errors and make it easier to read the model, as well as help downstream consumers of the data understand the data types and scan models for the columns they need. We prefer to use the following order: ids, strings, numerics, booleans, dates, and timestamps. #### Example model ```sql with source as ( select * from {{ source('ecom', 'raw_orders') }} ), renamed as ( select ---------- ids id as order_id, store_id as location_id, customer as customer_id, ---------- strings status as order_status, ---------- numerics (order_total / 100.0)::float as order_total, (tax_paid / 100.0)::float as tax_paid, ---------- booleans is_fulfilled, ---------- dates date(order_date) as ordered_date, ---------- timestamps ordered_at from source ) select * from renamed ``` --- ### How we style our dbt projects #### Why does style matter? Style might seem like a trivial, surface-level issue, but it's a deeply material aspect of a well-built project. A consistent, clear style enhances readability and makes your project easier to understand and maintain. Highly readable code helps build clear mental models making it easier to debug and extend your project. It's not just a favor to yourself, though; equally importantly, it makes it less effort for others to understand and contribute to your project, which is essential for peer collaboration, open-source work, and onboarding new team members. [A style guide lets you focus on what matters](https://mtlynch.io/human-code-reviews-1/#settle-style-arguments-with-a-style-guide), the logic and impact of your project, rather than the superficialities of how it's written. This brings harmony and pace to your team's work, and makes reviews more enjoyable and valuable. #### What's important about style? There are two crucial tenets of code style: * Clarity * Consistency Style your code in such a way that you can quickly read and understand it. It's also important to consider code review and git diffs. If you're making a change to a model, you want reviewers to see just the material changes you're making clearly. Once you've established a clear style, stay consistent. This is the most important thing. Everybody on your team needs to have a unified style, which is why having a style guide is so crucial. If you're writing a model, you should be able to look at other models in the project that your teammates have written and read in the same style. If you're writing a macro or a test, you should see the same style as your models. Consistency is key. #### How should I style? You should style the project in a way you and your teammates or collaborators agree on. The most important thing is that you have a style guide and stick to it. This guide is just a suggestion to get you started and to give you a sense of what a style guide might look like. It covers various areas you may want to consider, with suggested rules. It emphasizes lots of whitespace, clarity, clear naming, and comments. We believe one of the strengths of SQL is that it reads like English, so we lean into that declarative nature throughout our projects. Even within dbt Labs, though, there are differing opinions on how to style, even a small but passionate contingent of leading comma enthusiasts! Again, the important thing is not to follow this style guide; it's to make *your* style guide and follow it. Lastly, be sure to include rules, tools, *and* examples in your style guide to make it as easy as possible for your team to follow. #### Automation Use formatters and linters as much as possible. We're all human, we make mistakes. Not only that, but we all have different preferences and opinions while writing code. Automation is a great way to ensure that your project is styled consistently and correctly and that people can write in a way that's quick and comfortable for them, while still getting perfectly consistent output. --- ### How we style our Jinja #### Jinja style guide * 🫧 When using Jinja delimiters, use spaces on the inside of your delimiter, like `{{ this }}` instead of `{{this}}` * 🆕 Use newlines to visually indicate logical blocks of Jinja. * 4️⃣ Indent 4 spaces into a Jinja block to indicate visually that the code inside is wrapped by that block. * ❌ Don't worry (too much) about Jinja whitespace control, focus on your project code being readable. The time you save by not worrying about whitespace control will far outweigh the time you spend in your compiled code where it might not be perfect. #### Examples of Jinja style ```jinja {% macro make_cool(uncool_id) %} do_cool_thing({{ uncool_id }}) {% endmacro %} ``` ```sql select entity_id, entity_type, {% if this %} {{ that }}, {% else %} {{ the_other_thing }}, {% endif %} {{ make_cool('uncool_id') }} as cool_id ``` --- ### How we style our Python #### Python tooling * 🐍 Python has a more mature and robust ecosystem for formatting and linting (helped by the fact that it doesn't have a million distinct dialects). We recommend using those tools to format and lint your code in the style you prefer. * 🛠️ Our current recommendations are * [black](https://pypi.org/project/black/) formatter * [ruff](https://pypi.org/project/ruff/) linter info ☁️ dbt comes with the [black formatter built-in](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) to automatically lint and format their Python. You don't need to download or configure anything, just click `Format` in a Python model and you're good to go! #### Example Python ```python import pandas as pd def model(dbt, session): # set length of time considered a churn pd.Timedelta(days=2) dbt.config(enabled=False, materialized="table", packages=["pandas==1.5.2"]) orders_relation = dbt.ref("stg_orders") # converting a DuckDB Python Relation into a pandas DataFrame orders_df = orders_relation.df() orders_df.sort_values(by="ordered_at", inplace=True) orders_df["previous_order_at"] = orders_df.groupby("customer_id")[ "ordered_at" ].shift(1) orders_df["next_order_at"] = orders_df.groupby("customer_id")["ordered_at"].shift( -1 ) return orders_df ``` --- ### How we style our SQL #### Basics * ☁️ Use [SQLFluff](https://sqlfluff.com/) to maintain these style rules automatically. * Customize `.sqlfluff` configuration files to your needs. * Refer to our [SQLFluff config file](https://github.com/dbt-labs/jaffle-shop/blob/main/.sqlfluff) for the rules we use in our own projects. * Exclude files and directories by using a standard `.sqlfluffignore` file. Learn more about the syntax in the [.sqlfluffignore syntax docs](https://docs.sqlfluff.com/en/stable/configuration/index.html). * Excluding unnecessary folders and files (such as `target/`, `dbt_packages/`, and `macros/`) can speed up linting, improve run times, and help you avoid irrelevant logs. * 👻 Use Jinja comments (`{# #}`) for comments that should not be included in the compiled SQL. * ⏭️ Use trailing commas. * 4️⃣ Indents should be four spaces. * 📏 Lines of SQL should be no longer than 80 characters. * ⬇️ Field names, keywords, and function names should all be lowercase. * 🫧 The `as` keyword should be used explicitly when aliasing a field or table. info ☁️ dbt users can use the built-in [SQLFluff Studio IDE integration](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) to automatically lint and format their SQL. The default style sheet is based on dbt Labs style as outlined in this guide, but you can customize this to fit your needs. No need to setup any external tools, just hit `Lint`! Also, the more opinionated [sqlfmt](http://sqlfmt.com/) formatter is also available if you prefer that style. #### Fields, aggregations, and grouping * 🔙 Fields should be stated before aggregates and window functions. * 🤏🏻 Aggregations should be executed as early as possible (on the smallest data set possible) before joining to another table to improve performance. * 🔢 Ordering and grouping by a number (eg. group by 1, 2) is preferred over listing the column names (see [this classic rant](https://www.getdbt.com/blog/write-better-sql-a-defense-of-group-by-1) for why). Note that if you are grouping by more than a few columns, it may be worth revisiting your model design. #### Joins * 👭🏻 Prefer `union all` to `union` unless you explicitly want to remove duplicates. * 👭🏻 If joining two or more tables, *always* prefix your column names with the table name. If only selecting from one table, prefixes are not needed. * 👭🏻 Be explicit about your join type (i.e. write `inner join` instead of `join`). * 🥸 Avoid table aliases in join conditions (especially initialisms) — it's harder to understand what the table called "c" is as compared to "customers". * ➡️ Always move left to right to make joins easy to reason about - `right joins` often indicate that you should change which table you select `from` and which one you `join` to. #### 'Import' CTEs * 🔝 All `{{ ref('...') }}` statements should be placed in CTEs at the top of the file. * 📦 'Import' CTEs should be named after the table they are referencing. * 🤏🏻 Limit the data scanned by CTEs as much as possible. Where possible, only select the columns you're actually using and use `where` clauses to filter out unneeded data. * For example: ```sql with orders as ( select order_id, customer_id, order_total, order_date from {{ ref('orders') }} where order_date >= '2020-01-01' ) ``` #### 'Functional' CTEs * ☝🏻 Where performance permits, CTEs should perform a single, logical unit of work. * 📖 CTE names should be as verbose as needed to convey what they do e.g. `events_joined_to_users` instead of `user_events` (this could be a good model name, but does not describe a specific function or transformation). * 🌉 CTEs that are duplicated across models should be pulled out into their own intermediate models. Look out for chunks of repeated logic that should be refactored into their own model. * 🔚 The last line of a model should be a `select *` from your final output CTE. This makes it easy to materialize and audit the output from different steps in the model as you're developing it. You just change the CTE referenced in the `select` statement to see the output from that step. #### Model configuration * 📝 Model-specific attributes (like sort/dist keys) should be specified in the model. * 📂 If a particular configuration applies to all models in a directory, it should be specified in the `dbt_project.yml` file. * 👓 In-model configurations should be specified like this for maximum readability: ```sql {{ config( materialized = 'table', sort = 'id', dist = 'id' ) }} ``` #### Example SQL ```sql with events as ( ... ), {# CTE comments go here #} filtered_events as ( ... ) select * from filtered_events ``` ##### Example SQL ```sql with my_data as ( select id, field_1, field_2, field_3, cancellation_date, expiration_date, start_date from {{ ref('my_data') }} ), some_cte as ( select id, field_4, field_5 from {{ ref('some_cte') }} ), some_cte_agg as ( select id, sum(field_4) as total_field_4, max(field_5) as max_field_5 from some_cte group by 1 ), joined as ( select my_data.field_1, my_data.field_2, my_data.field_3, -- use line breaks to visually separate calculations into blocks case when my_data.cancellation_date is null and my_data.expiration_date is not null then expiration_date when my_data.cancellation_date is null then my_data.start_date + 7 else my_data.cancellation_date end as cancellation_date, some_cte_agg.total_field_4, some_cte_agg.max_field_5 from my_data left join some_cte_agg on my_data.id = some_cte_agg.id where my_data.field_1 = 'abc' and ( my_data.field_2 = 'def' or my_data.field_2 = 'ghi' ) ) select * from joined ``` --- ### How we style our YAML #### YAML Style Guide * 2️⃣ Indents should be two spaces * ➡️ List items should be indented * 🔠 List items with a single entry can be a string. For example, `'select': 'other_user'`, but it's best practice to provide the argument as an explicit list. For example, `'select': ['other_user']` * 🆕 Use a new line to separate list items that are dictionaries where appropriate * 📏 Lines of YAML should be no longer than 80 characters. * 🛠️ Use the [dbt JSON schema](https://github.com/dbt-labs/dbt-jsonschema) with any compatible Studio IDE and a YAML formatter (we recommend [Prettier](https://prettier.io/)) to validate your YAML files and format them automatically. Note, refer to [YAML tips](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for more YAML information. info ☁️ As with Python and SQL, the Studio IDE comes with built-in formatting for YAML files (Markdown and JSON too!), via Prettier. Just click the `Format` button and you're in perfect style. As with the other tools, you can [also customize the formatting rules](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#format-yaml-markdown-json) to your liking to fit your company's style guide. ##### Example YAML ```yaml models: - name: events columns: - name: event_id description: This is a unique identifier for the event data_tests: - unique - not_null - name: event_time description: "When the event occurred in UTC (eg. 2018-01-01 12:00:00)" data_tests: - not_null - name: user_id description: The ID of the user who recorded the event data_tests: - not_null - relationships: arguments: # available in v1.10.5 and higher. Older versions can set the as the top-level property. to: ref('users') field: id ``` --- ### Idempotence in dbt Idempotence is a foundational goal for every dbt project. It means your transformations should be safe to re-run and always produce the same result, regardless of how many times they've run before. Why this matters early If you're new to dbt, idempotence is worth understanding before you build incremental models. It's easy to accidentally write logic that works fine the first time but breaks on re-runs. #### What idempotence means in dbt A dbt model is idempotent if it produces the same results whether you run it once or a hundred times. The final state of the data should be identical regardless of how many times you've run the model. This is because each model is a pure function of your code and the current state of your database, not of how many times it's run before. Given the same inputs, it always produces the same output. This makes dbt transformations: * Safe to retry if a run fails partway through, re-running it won't corrupt your data. * Easy to backfill so you can re-run historical data without worrying about double-counting or drift. * Predictable in CI so the same code always produces the same data, which makes testing reliable. #### Materializations and idempotence Most dbt materializations are idempotent by default: | Materialization | How it achieves idempotence | | ------------------- | ---------------------------------------------------------------------------------------------------- | | `table` | Rebuilds the table from scratch on every run (`create or replace`) | | `view` | Replaces the view definition on every run | | `incremental` | Requires deliberate configuration. | | `materialized_view` | Creates or replaces the materialized view definition on each run; the warehouse manages data refresh | #### Idempotence and incremental models Incremental models only process new or changed rows, which means they depend on the existing state of the target table. If you're not careful, re-running an incremental model can produce duplicate rows or inconsistent results. The most common pitfall is appending rows without deduplication or a reliable unique key. Compare these side by side: **Non-idempotent incremental model:** This example filters incrementally but doesn't define a `unique_key`, so rows at the boundary can be appended again on each subsequent run, which might result in duplicate rows. ```sql -- ❌ Not idempotent: re-runs can duplicate rows {{ config(materialized='incremental') }} select * from {{ source('events', 'raw_events') }} {% if is_incremental() %} where event_at >= (select max(event_at) from {{ this }}) {% endif %} ``` **Idempotent incremental model:** This example adds a `unique_key`, so matching rows are updated or replaced instead of appended as duplicates. ```sql -- ✅ Idempotent when event_id is unique {{ config( materialized='incremental', unique_key='event_id' ) }} select * from {{ source('events', 'raw_events') }} {% if is_incremental() %} where event_at >= (select max(event_at) from {{ this }}) {% endif %} ``` With `unique_key` set, dbt updates existing rows and inserts new rows instead of appending duplicates. Depending on your adapter and [incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md), dbt does this with `merge` or `delete+insert`. You can also use [microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) for large time-series datasets. Microbatch models process data in batches based on an `event_time` column, and can be more resilient for very large incremental workloads. ##### Common risks | Risk | Why it breaks idempotence | Fix | | -------------------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | | Using `current_timestamp()` in a model | Produces different values on every run | Use a column from the source data as the timestamp | | Appending without a `unique_key` | Without a `unique_key`, most adapters/strategies append rows and can duplicate on re-runs. | Add `unique_key` to your incremental config | | Generating surrogate keys with random values | Different runs produce different keys for the same row | Use deterministic hashing (for example, `dbt_utils.generate_surrogate_key`) | | Hardcoding "today's date" in logic | Results change based on when the model runs, not the data | Filter on source timestamps | #### Full-refresh as a safety net When an incremental model gets into a bad state (for example, due to a schema change or logic bug), you can always run `dbt run --full-refresh` to drop and rebuild the table from scratch. This is the escape hatch that makes incremental models recoverable. Think of `--full-refresh` as proof that your underlying logic is still idempotent. Even if the incremental path is optimized, the full result should always be reproducible. #### Related docs These docs cover the dbt features most affected by idempotence: * [State selection and stateless runs](https://docs.getdbt.com/reference/node-selection/state-selection.md) * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * [Microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) --- ### Implementing your mesh plan ##### Where should your mesh journey start? Moving to a Mesh represents a meaningful change in development and deployment architecture. Before any sufficiently complex software refactor or migration, it's important to ask, 'Why might this not work?' The two most common reasons we've seen stem from 1. Lack of buy-in that a Mesh is the right long-term architecture 2. Lack of alignment on a well-scoped starting point Creating alignment on your architecture and starting point are major steps in ensuring a successful migration. Deciding on the right starting point will look different for every organization, but there are some heuristics that can help you decide where to start. In all likelihood, your organization already has logical components, and you may already be grouping, building, and deploying your project according to these interfaces.The goal is to define and formalize these organizational interfaces and use these boundaries to split your project apart by domain. How do you find these organizational interfaces? Here are some steps to get you started: * **Talk to teams** about what sort of separation naturally exists right now. * Are there various domains people are focused on? * Are there various sizes, shapes, and sources of data that get handled separately (such as click event data)? * Are there people focused on separate levels of transformation, such as landing and staging data or building marts? * Is there a single team that is *downstream* of your current dbt project, who could more easily migrate onto Mesh as a consumer? When attempting to define your project interfaces, you should consider investigating: * **Your jobs:** Which sets of models are most often built together? * **Your lineage graph:** How are models connected? * **Your selectors(defined in `selectors.yml`):** How do people already define resource groups? Let's go through an example process of taking a monolithing project, using groups and access to define the interfaces, and then splitting it into multiple projects. tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! #### Defining project interfaces with groups and access Once you have a sense of some initial groupings, you can first implement **group and access permissions** within a single project. * First you can create a [group](https://docs.getdbt.com/docs/build/groups.md) to define the owner of a set of models. ```yml # in models/__groups.yml groups: - name: marketing owner: name: Ben Jaffleck email: ben.jaffleck@jaffleshop.com ``` * Then, we can add models to that group using the `group:` key in the model's YAML entry. ```yml # in models/marketing/__models.yml models: - name: fct_marketing_model config: group: marketing # changed to config in v1.10 - name: stg_marketing_model config: group: marketing # changed to config in v1.10 ``` * Once you've added models to the group, you can **add [access](https://docs.getdbt.com/docs/mesh/govern/model-access.md) settings to the models** based on their connections between groups, *opting for the most private access that will maintain current functionality*. This means that any model that has *only* relationships to other models in the same group should be `private` , and any model that has cross-group relationships, or is a terminal node in the group DAG should be `protected` so that other parts of the DAG can continue to reference it. ```yml # in models/marketing/__models.yml models: - name: fct_marketing_model config: group: marketing # changed to config in v1.10 access: protected # changed to config in v1.10 - name: stg_marketing_model config: group: marketing # changed to config in v1.10 access: private # changed to config in v1.10 ``` * **Validate these groups by incrementally migrating your jobs** to execute these groups specifically via selection syntax. We would recommend doing this in parallel to your production jobs until you’re sure about them. This will help you feel out if you’ve drawn the lines in the right place. * If you find yourself **consistently making changes across multiple groups** when you update logic, that’s a sign that **you may want to rethink your groups**. #### Split your projects 1. **Move your grouped models into a subfolder**. This will include any model in the selected group, it's associated YAML entry, as well as its parent or child resources as appropriate depending on where this group sits in your DAG. 1. Note that just like in your dbt project, circular references are not allowed! Project B cannot have parents and children in Project A, for example. 2. **Create a new `dbt_project.yml` file** in the subdirectory. 3. **Copy any macros** used by the resources you moved. 4. **Create a new `packages.yml` file** in your subdirectory with the packages that are used by the resources you moved. 5. **Update `{{ ref }}` functions** — For any model that has a cross-project dependency (this may be in the files you moved, or in the files that remain in your project): 1. Update the `{{ ref() }}` function to have two arguments, where the first is the name of the source project and the second is the name of the model: e.g. `{{ ref('jaffle_shop', 'my_upstream_model') }}` 2. Update the upstream, cross-project parents’ `access` configs to `public` , ensuring any project can safely `{{ ref() }}` those models. 3. We *highly* recommend adding a [model contract](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) to the upstream models to ensure the data shape is consistent and reliable for your downstream consumers. 6. **Create a `dependencies.yml` file** ([docs](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md)) for the downstream project, declaring the upstream project as a dependency. ```yml # in dependencies.yml projects: - name: jaffle_shop ``` ##### Best practices * When you’ve **confirmed the right groups**, it's time to split your projects. * **Do *one* group at a time**! * **Do *not* refactor as you migrate**, however tempting that may be. Focus on getting 1-to-1 parity and log any issues you find in doing the migration for later. Once you’ve fully migrated the project then you can start optimizing it for its new life as part of your mesh. * Start by splitting your project within the same repository for full git tracking and easy reversion if you need to start from scratch. #### Connecting existing projects Some organizations may already be coordinating across multiple dbt projects. Most often this is via: 1. Installing parent projects as dbt packages 2. Using `{{ source() }}` functions to read the outputs of a parent project as inputs to a child project. This has a few drawbacks: 1. If using packages, each project has to include *all* resources from *all* projects in its manifest, slowing down dbt and the development cycle. 2. If using sources, there are breakages in the lineage, as there's no real connection between the parent and child projects. The migration steps here are much simpler than splitting up a monolith! 1. If using the `package` method: 1. In the parent project: 1. mark all models being referenced downstream as `public` and add a model contract. 2. In the child project: 1. Remove the package entry from `packages.yml` 2. Add the upstream project to your `dependencies.yml` 3. Update the `{{ ref() }}` functions to models from the upstream project to include the project name argument. 2. If using `source` method: 1. In the parent project: 1. mark all models being imported downstream as `public` and add a model contract. 2. In the child project: 1. Add the upstream project to your `dependencies.yml` 2. Replace the `{{ source() }}` functions with cross project `{{ ref() }}` functions. 3. Remove the unnecessary `source` definitions. #### Additional Resources ##### Our example projects We've provided a set of example projects you can use to explore the topics covered here. We've split our [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) project into 3 separate projects in a multi-repo Mesh. Note that you'll need to leverage dbt to use multi-project architecture, as cross-project references are powered via dbt's APIs. * **[Platform](https://github.com/dbt-labs/jaffle-shop-mesh-platform)** - containing our centralized staging models. * **[Marketing](https://github.com/dbt-labs/jaffle-shop-mesh-marketing)** - containing our marketing marts. * **[Finance](https://github.com/dbt-labs/jaffle-shop-mesh-finance)** - containing our finance marts. #### Related docs * [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) --- ### Incremental models in-depth So far we’ve looked at tables and views, which map to the traditional objects in the data warehouse. As mentioned earlier, incremental models are a little different. This is where we start to deviate from this pattern with more powerful and complex materializations. * 📚 **Incremental models generate tables.** They physically persist the data itself to the warehouse, just piece by piece. What’s different is **how we build that table**. * 💅 **Only apply our transformations to rows of data with new or updated information**, this maximizes efficiency. * 🌍  If we have a very large set of data or compute-intensive transformations, or both, it can be very slow and costly to process the entire corpus of source data being input into a model or chain of models. If instead we can identify *only rows that contain new information* (that is, **new or updated records**), we then can process just those rows, building our models *incrementally*. * 3️⃣  We need **3 key things** in order to accomplish the above: * a **filter** to select just the new or updated records * a **conditional block** that wraps our filter and only applies it when we want it * **configuration** that tells dbt we want to build incrementally and helps apply the conditional filter when needed Let’s dig into how exactly we can do that in dbt. Let’s say we have an `orders` table that looks like the below: | order\_id | order\_status | customer\_id | order\_item\_id | ordered\_at | updated\_at | | --------- | ------------- | ------------ | --------------- | ----------- | ----------- | | 123 | shipped | 7 | 5791 | 2022-01-30 | 2022-01-30 | | 234 | confirmed | 15 | 1643 | 2022-01-31 | 2022-01-31 | We did our last `dbt build` job on `2022-01-31`, so any new orders since that run won’t appear in our table. When we do our next run (for simplicity let’s say the next day, although for an orders model we’d more realistically run this hourly), we have two options: * 🏔️ build the table from the **beginning of time again — a *table materialization*** * Simple and solid, if we can afford to do it (in terms of time, compute, and money — which are all directly correlated in a cloud warehouse). It’s the easiest and most accurate option. * 🤏 find a way to run **just new and updated rows since our previous run — *an* *incremental materialization*** * If we *can’t* realistically afford to run the whole table — due to complex transformations or big source data, it takes too long — then we want to build incrementally. We want to just transform and add the row with id 567 below, *not* the previous two with ids 123 and 234 that are already in the table. | order\_id | order\_status | customer\_id | order\_item\_id | ordered\_at | updated\_at | | --------- | ------------- | ------------ | --------------- | ----------- | ----------- | | 123 | shipped | 7 | 5791 | 2022-01-30 | 2022-01-30 | | 234 | confirmed | 15 | 1643 | 2022-01-31 | 2022-01-31 | | 567 | shipped | 61 | 28 | 2022-02-01 | 2022-02-01 | ##### Writing incremental logic Let’s think through the information we’d need to build such a model that only processes new and updated data. We would need: * 🕜  **a timestamp indicating when a record was last updated**, let’s call it our `updated_at` timestamp, as that’s a typical convention and what we have in our example above. * ⌛ the **most recent timestamp from this table *in our warehouse*** *—* that is, the one created by the previous run — to act as a cutoff point. We’ll call the model we’re working in `this`, for ‘this model we’re working in’. That would let us construct logic like this: ```sql select * from orders where updated_at > (select max(updated_at) from {{ this }}) ``` Let’s break down that `where` clause a bit, because this is where the action is with incremental models. Stepping through the code ***right-to-left*** we: 1. Get our **cutoff.** 1. Select the `max(updated_at)` timestamp — the **most recent record** 2. from `{{ this }}` — the table for this model as it exists in the warehouse, as **built in our last run**, 3. so `max(updated_at) from {{ this }}` the ***most recent record processed in our last run,*** 4. that’s exactly what we want as a **cutoff**! 2. **Filter** the rows we’re selecting to add in this run. 1. Use the `updated_at` timestamp from our input, the equivalent column to the one in the warehouse, but in the up-to-the-minute **source data we’re selecting from** and 2. check if it’s **greater than our cutoff,** 3. if so it will satisfy our where clause, so we’re **selecting all the rows more recent than our cutoff.** This logic would let us isolate and apply our transformations to just the records that have come in since our last run, and I’ve got some great news: that magic `{{ this }}` keyword [does in fact exist in dbt](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md), so we can write exactly this logic in our models. ##### Configuring incremental models So we’ve found a way to isolate the new rows we need to process. How then do we handle the rest? We still need to: * ➕  make sure dbt knows to ***add* new rows on top** of the existing table in the warehouse, **not replace** it. * 👉  If there are **updated rows**, we need a way for dbt to know **which rows to update**. * 🌍  Lastly, if we’re building into a new environment and there’s **no previous run to reference**, or we need to **build the model from scratch.** Put another way, we’ll want a means to skip the incremental logic and transform all of our input data like a regular table if needed. * 😎 **Visualized below**, we’ve figured out how to get the red ‘new records’ portion selected, but we need to sort out the step to the right, where we stick those on to our model. [![Diagram visualizing how incremental models work](/img/best-practices/materializations/incremental-diagram.png?v=2 "Diagram visualizing how incremental models work")](#)Diagram visualizing how incremental models work info 😌 Incremental models can be confusing at first, **take your time reviewing** this visual and the previous steps until you have a **clear mental model.** Be patient with yourself. This materialization will become second nature soon, but it’s tough at first. If you’re feeling confused the [dbt Community is here for you on the Forum and Slack](https://www.getdbt.com/community/join-the-community). Thankfully dbt has some additional configuration and special syntax just for incremental models. First, let's look at a config block for incremental materialization: ```sql {{ config( materialized='incremental', unique_key='order_id' ) }} select ... ``` * 📚 The **`materialized` config** works just like tables and views, we just pass it the value `'incremental'`. * 🔑 We’ve **added a new config option `unique_key`,** that tells dbt that if it finds a record in our previous run — the data in the warehouse already — with the same unique id (in our case `order_id` for our `orders` table) that exists in the new data we’re adding incrementally, to **update that record instead of adding it as a separate row**. * 👯 This **hugely broadens the types of data we can build incrementally** from just immutable tables (data where rows only ever get added, never updated) to mutable records (where rows might change over time). As long as we’ve got a column that specifies when records were updated (such as `updated_at` in our example), we can handle almost anything. * ➕ We’re now **adding records** to the table **and updating existing rows**. That’s 2 of 3 concerns. * 🆕 We still need to **build the table from scratch** (via `dbt build` or `run` in a job) when necessary — whether because we’re in a new environment so don’t have an initial table to build on, or our model has drifted from the original over time due to data loading latency. * 🔀 We need to wrap our incremental logic, that is our `where` clause with our `updated_at` cutoff, in a **conditional statement that will only apply it when certain conditions are met**. If you’re thinking this is **a case for a Jinja `{% if %}` statement**, you’re absolutely right! ##### Incremental conditions So we’re going to use an **if statement** to apply our cutoff filter **only when certain conditions are met**. We want to apply our cutoff filter *if* the **following things are true**: * ➕  we’ve set the materialization **config** to incremental, * 🛠️  there is an **existing table** for this model in the warehouse to build on, * 🙅‍♀️  and the `--full-refresh` **flag was *not* passed.** * [full refresh](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) is a configuration and flag that is specifically designed to let us override the incremental materialization and build a table from scratch again. Thankfully, we don’t have to dig into the guts of dbt to sort out each of these conditions individually. * ⚙️  dbt provides us with a **macro [`is_incremental`](https://docs.getdbt.com/docs/build/incremental-models.md#understand-the-is_incremental-macro)** that checks all of these conditions for this exact use case. * 🔀  By **wrapping our cutoff logic** in this macro, it will only get applied when the macro returns true for all of the above conditions. Let’s take a look at all these pieces together: ```sql {{ config( materialized='incremental', unique_key='order_id' ) }} select * from orders {% if is_incremental() %} where updated_at > (select max(updated_at) from {{ this }}) {% endif %} ``` Fantastic! We’ve got a working incremental model. On our first run, when there is no corresponding table in the warehouse, `is_incremental` will evaluate to false and we’ll capture the entire table. On subsequent runs it will evaluate to true and we’ll apply our filter logic, capturing only the newer data. ##### Late-arriving facts Our last concern specific to incremental models is what to do when data is inevitably loaded in a less-than-perfect way. Sometimes data loaders will, for a variety of reasons, load data late. Either an entire load comes in late, or some rows come in on a load after those with which they should have. The following is best practice for every incremental model to slow down the drift this can cause. * 🕐 For example if most of our records for `2022-01-30` come in the raw schema of our warehouse on the morning of `2022-01-31`, but a handful don’t get loaded til `2022-02-02`, how might we tackle that? There will already be `max(updated_at)` timestamps of `2022-01-31` in the warehouse, filtering out those late records. **They’ll never make it to our model.** * 🪟 To mitigate this, we can add a **lookback window** to our **cutoff** point. By **subtracting a few days** from the `max(updated_at)`, we would capture any late data within the window of what we subtracted. * 👯 As long as we have a **`unique_key` defined in our config**, we’ll simply update existing rows and avoid duplication. We process more data this way, but in a fixed way, and it keeps our model hewing closer to the source data. ###### Using dbt State with incremental models State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with incremental models. Use [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) to control how frequently the model rebuilds based on upstream data changes, and [`pre_clone`](https://docs.getdbt.com/reference/resource-configs/pre-clone.md) to pre-populate the model by cloning from production before a run. For example, to skip a rebuild if upstream data changed less than 4 hours ago, and always start from the current production state in local development: ```yaml models: - name: fct_events config: state: lag_tolerance: 4h pre_clone: always ``` The default for `pre_clone` is `if_missing`, which clones production only if the table doesn't already exist locally. **Late-arriving records:** Late-arriving records may have an earlier event timestamp (for example, `event_date`) than their ingestion timestamp. dbt State may not detect them as new data and skip rebuilding the incremental model, even though your lookback window would normally pick up those records. To ensure late-arriving data is detected, configure a `loaded_at_query` on the source that aligns with the same lookback window used in your incremental filter: ```yaml sources: - name: raw_orders tables: - name: orders config: loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` For more details, refer to [dbt State configurations](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) and [freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md). ##### Long-term considerations Late arriving facts point to the biggest tradeoff with incremental models: * 🪢 In addition to extra **complexity**, they also inevitably **drift from the source data over time.** Due to the imperfection of loaders and the reality of late arriving facts, we can’t help but miss some day in-between our incremental runs, and this accumulates. * 🪟 We can slow this entropy with the lookback window described above — **the longer the window the less efficient the model, but the slower the drift.** It’s important to note it will still occur though, however slowly. If we have a lookback window of 3 days, and a record comes in 4 days late from the loader, we’re still going to miss it. * 🌍 Thankfully, there is a way we can reset the relationship of the model to the source data. We can run the model with the **`--full-refresh` flag passed** (such as `dbt build --full-refresh -s orders`). As we saw in the `is_incremental` conditions above, that will make our logic return false, and our `where` clause filter will not be applied, running the whole table. * 🏗️ This will let us **rebuild the entire table from scratch,** a good practice to do regularly **if the size of the data will allow**. * 📆 A common pattern for incremental models of manageable size is to run a **full refresh on the weekend** (or any low point in activity), either **weekly or monthly**, to consistently reset the drift from late arriving facts. --- ### Incremental patterns for near real-time data This section covers three core incremental patterns for achieving near real-time data freshness: 1. [Incremental MERGE from append-only tables](#incremental-merge-from-append-only-tables) 2. [CDC with Snowflake Streams](#cdc-with-snowflake-streams) 3. [Microbatch for large time-series tables](#microbatch-for-large-time-series-tables) Snowflake-specific pattern Some patterns on this guide uses Snowflake-specific features. Other warehouses have similar features with different implementations. Refer to the [additional resources](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#resources-by-warehouse) section for adapter-specific documentation. #### Pattern 1: Incremental MERGE from append-only tables This pattern uses the `merge` incremental strategy to upsert (insert + update) new and updated rows into a target table. Most data platforms support the `merge` strategy. Refer to [supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for details. "Append-only tables" refers to a data pattern where source data continuously receives new rows without updates or deletes. ##### When to use the merge strategy Use this pattern when raw events continuously land into a staging table and you want a near real-time fact table updated every few minutes. ##### Example model In this example, assume you have raw events continuously landing into `raw.events` (using Snowpipe, Databricks Auto Loader, Kafka, or a similar ingestion mechanism) and you're looking for a near real‑time fact table `analytics.fct_events` updated every few minutes. Configure the SQL model with the following settings: * Use the `incremental` filter to only scan rows newer than the latest timestamp already in the target. * Use `incremental_strategy='merge'` with `unique_key=event_id` to give you idempotent upserts (inserts + updates). * Cluster by date using `cluster_by=['event_date']` helps with query pruning during `MERGE` operations (syntax varies by warehouse). * Run the model every few minutes to achieve a freshness service level agreement (SLA) measured in minutes, depending on ingestion and job scheduling. The following example uses Snowflake SQL syntax (`::` type casting, `timestamp_ntz`, `cluster_by` config). Make sure you adapt the SQL and clustering syntax for your warehouse. models/fct\_events.sql ```sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', -- default on Snowflake unique_key = 'event_id', cluster_by = ['event_date'] -- helps MERGE performance (syntax varies by warehouse) ) }} with source_events as ( select event_id, event_ts::timestamp_ntz as event_ts, -- Snowflake syntax for type casting to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'events') }} {% if is_incremental() %} -- Only pull new/changed rows since last successful load where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ), deduped as ( -- optional: if the raw feed can produce duplicates select * from ( select *, row_number() over ( partition by event_id order by event_ts desc ) as _rn from source_events ) where _rn = 1 ) select event_id, event_ts, event_date, user_id, event_type, payload from deduped; ``` To ensure the best results: * Use clustering keys wisely for better `MERGE` performance. * Monitor `MERGE` performance as your table grows. * Consider adding a lookback window (for example, `event_ts > max(event_ts) - interval '1 hour'`) to handle late-arriving data. #### Pattern 2: CDC with Snowflake Streams This pattern leverages Snowflake's native Change Data Capture (CDC) capabilities through [Streams](https://docs.snowflake.com/en/user-guide/streams-intro), a Snowflake-specific feature which tracks changes (inserts, updates, deletes) to source tables. ##### When to use CDC Use CDC when: * You have source tables that receive frequent updates (not just appends). * You need to capture both new records and changes to existing records. * You want to avoid full table scans on large source tables. This pattern is warehouse CDC plus an incremental model. It does not use snapshots. For guidance on choosing incremental models vs snapshots (and when to use both), refer to [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md). ##### Setup To use this pattern, set up the stream in your data warehouse and then create a model to consume the stream. 1. Create the stream (one-time, outside dbt): ```sql create or replace stream RAW.EVENTS_STREAM on table RAW.EVENTS; ``` 2. Create a model consuming the stream: models/fct\_events\_cdc.sql ```sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', unique_key = 'event_id', cluster_by = ['event_date'], snowflake_warehouse = 'TRANSFORM_WH' ) }} with changes as ( select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload, metadata$action as change_type -- points at the STREAM, not the table from {{ source('raw', 'events_stream') }} ), filtered as ( select * from changes where change_type in ('INSERT', 'UPDATE') -- If you want to physically delete, you could also handle 'DELETE' here ) select event_id, event_ts, event_date, user_id, event_type, payload from filtered; ``` ##### Pattern distinctions There are some key differences from [pattern 1](#incremental-merge-from-append-only-tables): * Streams only return changed rows, so you don’t need an `is_incremental()` time filter. Each run processes only the changes available at the moment. * Run the model every few minutes to pull new changes and merge them into `fct_events`. * This gives you a CDC-style pipeline. Snowflake Streams captures changes, and dbt handles transformations, tests, and lineage. #### Pattern 3: Microbatch for large time-series tables For large `fact` tables where backfills or long lookback windows are challenging, use `incremental_strategy='microbatch'` (available in dbt v1.9 or higher and **v1 Latest** release track in dbt platform). Refer to [incremental microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) for more details. Note that Microsoft Fabric doesn't support microbatch yet. Refer to [incremental strategy by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for more details. microbatch must have event\_time Every upstream model feeding this microbatch model must also be configured with `event_time` so dbt can push time-filters upstream. Otherwise, each batch could re-scan full upstream tables. ##### When to use microbatch * You have massive time-series tables (billions of rows). * Backfills are slow and risky with traditional incremental approaches. * You need to reprocess data in manageable chunks. * Late-arriving data is common. ##### Model configuration Let's say you have a `fact_events` table with a `event_ts` column and you want to process it in hourly chunks. You can configure the model as follows: models/fct\_events\_microbatch.sql ```sql {{ config( materialized = 'incremental', incremental_strategy= 'microbatch', event_time = 'event_ts', -- time column in this model batch_size = 'hour', -- process in hourly chunks lookback = 1, -- reprocess 1 prior batch to catch late data unique_key = 'event_id', cluster_by = ['event_date'], full_refresh = false ) }} select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload from {{ ref('stg_events') }}; ``` ##### Key behavior * Use microbatch for massive fact tables (clickstream, IoT, point-of-sale) with multi-year history. * No `is_incremental() block` needed — dbt automatically generates the appropriate `WHERE event_ts BETWEEN..` predicates per batch based on `event_time`, `batch_size`, `begin`, `lookback`, and so on. * Each run processes multiple smaller queries (one per batch), making larger backfills safer and easier to retry. * The `lookback` parameter automatically handles late-arriving data by reprocessing recent batches. * Schedule jobs based on your SLA. #### Choosing the right incremental pattern The pattern you select will depend on your use case. Start with [pattern 1](#incremental-merge-from-append-only-tables) (`MERGE`), since it's appropriate for most use cases. Upgrade to [pattern 2](#cdc-with-snowflake-streams) (use your data warehouse's native CDC features) when you need efficient CDC. Reach for [pattern 3](#microbatch-for-large-time-series-tables) (Microbatch) when dealing with massive scale. Use the following table to help you choose the right pattern: | Pattern | Best for | Key benefit | | ------------------------ | ---------------------------- | ---------------------------------- | | `merge` from append-only | Most standard use cases | Simple, widely understood | | CDC with Streams | Tables with frequent updates | Efficient change capture | | Microbatch | Massive time-series tables | Safe backfills, late-data handling | #### Related docs * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md) * [Microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) * [Configuring incremental models in dbt](https://docs.getdbt.com/docs/build/incremental-models.md) --- ### Intermediate: Purpose-built transformation steps Note on the example project. The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example project uses a **staging → marts** flow for simplicity. Larger projects often add an intermediate layer between staging and marts. This page covers when and how to use that layer. Once we've got our atoms ready to work with, we'll set about bringing them together into more intricate, connected molecular shapes. The intermediate layer is where these molecules live, creating varied forms with specific purposes on the way towards the more complex proteins and cells we'll use to breathe life into our data products. ##### Intermediate: Files and folders In a project that uses an intermediate layer, models typically live in an `intermediate/` subdirectory: ```shell models/intermediate └── finance ├── _int_finance__models.yml └── int_orders_summed_to_customer.sql ``` * **Folders** * ✅ **Subdirectories based on business groupings.** Much like the staging layer, we'll house this layer of models inside their own `intermediate` subfolder. Unlike the staging layer, here we shift towards being business-conformed, splitting our models up into subdirectories not by their source system, but by their area of business concern. * **File names** * `✅ int_[entity]s_[verb]s.sql` - the variety of transformations that can happen inside of the intermediate layer makes it harder to dictate strictly how to name them. The best guiding principle is to think about *verbs* (e.g. `pivoted`, `aggregated_to_user`, `joined`, `fanned_out_by_quantity`, `funnel_created`, etc.) in the intermediate layer. In a larger project, you might use an intermediate model to aggregate order items to the order grain, and name it `int_order_items_summed_to_orders`. It's easy for anybody to quickly understand what's happening in that model, even if they don't know [SQL](https://mode.com/sql-tutorial/). That clarity is worth the long file name. It's important to note that we've dropped the double underscores at this layer. In moving towards business-conformed concepts, we no longer need to separate a system and an entity and simply reference the unified entity if possible. In cases where you need intermediate models to operate at the source system level (e.g. `int_shopify__orders_summed`, `int_core__orders_summed` which you would later union), you'd preserve the double underscores. Some people like to separate the entity and verbs with double underscores as well. That's a matter of preference, but in our experience, there is often an intrinsic connection between entities and verbs in this layer that make that difficult to maintain. Don't over-optimize too early! The example project is very simple for illustrative purposes. This level of division in our post-staging layers is probably unnecessary when dealing with these few models. Remember, our goal is a *single* *source of truth.* We don't want finance and marketing operating on separate `orders` models, we want to use our dbt project as a means to bring those definitions together! As such, don't split and optimize too early. If you have less than 10 marts models and aren't having problems developing and using them, feel free to forego subdirectories completely (except in the staging layer, where you should always implement them as you add new source systems to your project) until the project has grown to really need them. Using dbt is always about bringing simplicity to complexity. ##### Intermediate: Models Intermediate models serve a clear, single purpose: preparing staging models for marts. Common patterns include grouping and pivoting to a different grain, fanning out rows, or isolating complex logic so marts stay readable. * ❌ **Exposed to end users.** Intermediate models should generally not be exposed in the main production schema. They are not intended for output to final targets like dashboards or applications, so it's best to keep them separated from models that are so you can more easily control data governance and discoverability. * ✅ **Materialized ephemerally.** Considering the above, one popular option is to default to intermediate models being materialized [ephemerally](https://docs.getdbt.com/docs/build/materializations.md#ephemeral). This is generally the best place to start for simplicity. It will keep unnecessary models out of your warehouse with minimum configuration. Keep in mind though that the simplicity of ephemerals does translate a bit more difficulty in troubleshooting, as they're interpolated into the models that `ref` them, rather than existing on their own in a way that you can view the output of. * ✅ **Materialized as views in a custom schema with special permissions.** A more robust option is to materialize your intermediate models as views in a specific [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md), outside of your main production schema. This gives you added insight into development and easier troubleshooting as the number and complexity of your models grows, while remaining easy to implement and taking up negligible space. Refer to [Mapping project folders to warehouse schemas](https://docs.getdbt.com/best-practices/how-we-structure/6-mapping-folders-to-warehouse.md) for the default folder-to-schema mapping. Keep your warehouse tidy! There are three interfaces to the organizational knowledge graph we're encoding into dbt and folder structure of our codebase, and the output into the warehouse. As such, it's really important that we consider that output intentionally! Think of the schemas, tables, and views we're creating in the warehouse as *part of the UX,* in addition to the dashboards, ML, apps, and other use cases you may be targeting for the data. Ensuring that our output is named and grouped well, and that models not intended for broad use are either not materialized or built into special areas with specific permissions is crucial to achieving this. * Intermediate models' purposes, as these serve to break up complexity from our marts models, can take as many forms as [data transformation](https://www.getdbt.com/analytics-engineering/transformation/) might require. Some of the most common use cases of intermediate models include: * ✅ **Structural simplification.** Bringing together a reasonable number (typically 4 to 6) of entities or concepts (staging models, or perhaps other intermediate models) that will be joined with another similarly purposed intermediate model to generate a mart — rather than have 10 joins in our mart, we can join two intermediate models that each house a piece of the complexity, giving us increased readability, flexibility, testing surface area, and insight into our components. * ✅ **Re-graining.** Intermediate models are often used to fan out or collapse models to the right composite grain — if we're building a mart for `order_items` that requires us to fan out our `orders` based on the `quantity` column, creating a new single row for each item, this would be ideal to do in a specific intermediate model to maintain clarity in our mart and more easily view that our grain is correct before we mix it with other components. * ✅ **Isolating complex operations.** It's helpful to move any particularly complex or difficult to understand pieces of logic into their own intermediate models. This not only makes them easier to refine and troubleshoot, but simplifies later models that can reference this concept in a more clearly readable way. For example, in the `quantity` fan out example above, we benefit by isolating this complex piece of logic so we can quickly debug and thoroughly test that transformation, and downstream models can reference `order_items` in a way that's intuitively easy to grasp. Narrow the DAG, widen the tables. Until we get to the marts layer and start building our various outputs, we ideally want our DAG to look like an arrowhead pointed right. As we move from source-conformed to business-conformed, we're also moving from numerous, narrow, isolated concepts to fewer, wider, joined concepts. We're bringing our components together into wider, richer concepts, and that creates this shape in our DAG. This way when we get to the marts layer we have a robust set of components that can quickly and easily be put into any configuration to answer a variety of questions and serve specific needs. One rule of thumb to ensure you're following this pattern on an individual model level is allowing multiple *inputs* to a model, but **not** multiple *outputs*. Several arrows going *into* our post-staging models is great and expected, several arrows coming *out* is a red flag. There are absolutely situations where you need to break this rule, but it's something to be aware of, careful about, and avoid when possible. --- ### Intro to dbt Mesh #### What is dbt Mesh? Organizations of all sizes rely upon dbt to manage their data transformations, from small startups to large enterprises. At scale, it can be challenging to coordinate all the organizational and technical requirements demanded by your stakeholders within the scope of a single dbt project. To date, there also hasn't been a first-class way to effectively manage the dependencies, governance, and workflows between multiple dbt projects. That's where **Mesh** comes in - empowering data teams to work *independently and collaboratively*; sharing data, code, and best practices without sacrificing security or autonomy. Mesh is not a single product - it is a pattern enabled by a convergence of several features in dbt: * **[Cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref)** - this is the foundational feature that enables the multi-project deployments. `{{ ref() }}`s now work across dbt projects on Enterprise and Enterprise+ plans. * **[Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md)** - dbt's metadata-powered documentation platform, complete with full, cross-project lineage. * **Governance** - dbt's governance features allow you to manage access to your dbt models both within and across projects. * **[Groups](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups)** - With groups, you can organize nodes in your dbt DAG that share a logical connection (for example, by functional area) and assign an owner to the entire group. * **[Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers)** - access configs allow you to control who can reference models. * **[Model Versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md)** - when coordinating across projects and teams, we recommend treating your data models as stable APIs. Model versioning is the mechanism to allow graceful adoption and deprecation of models as they evolve. * **[Model Contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md)** - data contracts set explicit expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. #### When is the right time to use dbt Mesh? The multi-project architecture helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. If you're already using dbt and your project has started to experience any of the following, you're likely ready to start exploring this paradigm: * The **number of models** in your project is degrading performance and slowing down development. * Teams have developed **separate workflows** and need to decouple development from each other. * Teams are experiencing **communication challenges**, and the reliability of some of your data products has started to deteriorate. * **Security and governance** requirements are increasing and would benefit from increased isolation. dbt is designed to coordinate the features above and simplify the complexity to solve for these problems. If you're just starting your dbt journey, don't worry about building a multi-project architecture right away. You can *incrementally* adopt the features in this guide as you scale. The collection of features work effectively as independent tools. Familiarizing yourself with the tooling and features that make up a multi-project architecture, and how they can apply to your organization will help you make better decisions as you grow. For additional information, refer to the [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md). #### Learning goals * Understand the **purpose and tradeoffs** of building a multi-project architecture. * Develop an intuition for various **Mesh patterns** and how to design a multi-project architecture for your organization. * Establish recommended steps to **incrementally adopt** these patterns in your dbt implementation. tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! --- ### Intro to the dbt Semantic Layer tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). Flying cars, hoverboards, and true self-service analytics: this is the future we were promised. The first two might still be a few years out, but real self-service analytics is here today. With dbt's Semantic Layer, you can resolve the tension between accuracy and flexibility that has hampered analytics tools for years, empowering everybody in your organization to explore a shared reality of metrics. Best of all for analytics engineers, building with these new tools will significantly [DRY](https://docs.getdbt.com/terms/dry) up and simplify your codebase. As you'll see, the deep interaction between your dbt models and the Semantic Layer make your dbt project the ideal place to craft your metrics. #### Learning goals * ❓ Understand the **purpose and capabilities** of the **Semantic Layer**, particularly MetricFlow as the engine that powers it. * 🧱 Familiarity with the core components of MetricFlow — **semantic models and metrics** — and how they work together. * 🔁 Know how to **refactor** dbt models for the Semantic Layer. * 🏅 Aware of **best practices** to take maximum advantage of the Semantic Layer. #### Guide structure overview 1. Getting **setup** in your dbt project. 2. Building a **semantic model** and its fundamental parts: **entities, dimensions, and measures**. 3. Building a **metric**. 4. Defining **advanced metrics**: `ratio` and `derived` types. 5. **File and folder structure**: establishing a system for naming things. 6. **Refactoring** marts and roll-ups for the Semantic Layer. 7. Review **best practices**. If you're ready to ship your users more power and flexibility with less code, let's dive in! info MetricFlow is the engine for defining metrics in dbt and one of the key components of the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). It handles SQL query construction and defines the specification for dbt semantic models and metrics. To fully experience the Semantic Layer, including the ability to query dbt metrics via external integrations, you'll need a [dbt Starter, Enterprise, or Enterprise+ accounts](https://www.getdbt.com/pricing/). Refer to [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) for more information. --- ### Lambda views for near real-time dashboards Snowflake examples ahead This page uses Snowflake for code examples, but you can adapt the lambda view pattern to other warehouses. A lambda view pattern combines a batch / incremental fact table with a small near real-time (NRT) slice of very recent data and exposes them through a single view. This is a legacy-but-still-useful pattern some teams have used to deliver near real‑time operational dashboards on top of dbt and a warehouse. #### When to use lambda views * You need fresher reads than your normal incremental schedule, but * You can't (or don't want to) use [dynamic tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) or [materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view), or you want to keep logic entirely in dbt SQL. The examples used in this page assume the following setup: ##### Assumptions The examples used in this page assume the following setup: * Raw events land continuously into `raw.events` using your warehouse's streaming ingestion feature (like Snowpipe, Databricks Auto Loader, Kafka, or a similar ingestion mechanism). * You already maintain an [incremental fact table](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#incremental-merge-from-append-only-tables) that is rebuilt every few minutes using `incremental_strategy='merge'`. * Most dashboards are fine reading from that incremental table, but a small set of operational dashboards want "as‑of‑now" data (for example, the last few minutes of events). ##### How this pattern works * The base incremental table is rebuilt every few minutes using `incremental_strategy='merge'`. * The NRT view is a view that selects only events newer than the max `event_ts` already persisted in the base incremental table. * The lambda view `UNION ALL`s the base table and the NRT view, de-duplicating rows based on primary key semantics. Downstream BI or dashboards query only the lambda view. #### Base incremental table You can reuse the incremental `merge` from [Snowflake pattern 1](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#incremental-merge-from-append-only-tables) as your base table; for completeness: ```sql -- models/fct_events.sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', unique_key = 'event_id', cluster_by = ['event_date'], snowflake_warehouse = 'TRANSFORM_WH' ) }} with source_events as ( select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'events') }} {% if is_incremental() %} -- Only pull new/changed rows since last successful load where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ) select * from source_events; ``` Schedule this model to run, for example, every 5–15 minutes as part of your near real‑time job. #### NRT view: rows more recent than the base table The NRT view returns only events with `event_ts` greater than the maximum timestamp in the base table, so there is no overlap or double counting: ```sql -- models/fct_events_nrt.sql {{ config( materialized = 'view' ) }} with base_max as ( select max(event_ts) as max_event_ts from {{ ref('fct_events') }} ), fresh_events as ( select e.event_id, e.event_ts::timestamp_ntz as event_ts, to_date(e.event_ts) as event_date, e.user_id, e.event_type, e.payload from {{ source('raw', 'events') }} as e cross join base_max where e.event_ts > base_max.max_event_ts ) select * from fresh_events; ``` Characteristics: * No scheduling required — it's just a view over `raw.events` filtered by `max(event_ts)` from `fct_events`. * Every query against `fct_events_nrt` scans only "since last batch" data, which should be a small time window (for example, a few minutes or hours, depending on your job cadence). ##### Lambda view: single read path for BI The lambda view combines historical data from the base incremental table with the most recent events from the NRT view. ```sql -- models/fct_events_lambda.sql {{ config( materialized = 'view' ) }} select event_id, event_ts, event_date, user_id, event_type, payload from {{ ref('fct_events') }} union all select event_id, event_ts, event_date, user_id, event_type, payload from {{ ref('fct_events_nrt') }}; ``` Point your BI tools and dashboards to `analytics.fct_events_lambda`. Most data comes from the pre-computed incremental table, while the most recent events (since the last dbt run) come from a live query against `raw.events`. This approach is outlined in [this original dbt lambda view blog post](https://discourse.getdbt.com/t/how-to-create-near-real-time-models-with-just-dbt-sql/1457) which describes how teams like JetBlue wired near real‑time operational dashboards on Snowflake and dbt. #### Considerations Take the following into consideration when using this pattern: * **Cost profile** * Every query against `fct_events_lambda` must read the NRT slice from `raw.events` in addition to the base table. * Use this pattern only for truly operational dashboards that justify the extra per‑query cost. * **Freshness** * Freshness is bounded by: * Your dbt incremental job frequency (age of `fct_events`), plus * Ingestion latency into `raw.events` (Snowpipe / streaming layer). * **Complexity vs alternatives** * For many modern Snowflake implementations, a [dynamic table](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) or [materialized view](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) with a small `target_lag` can provide similar "always within X minutes" service level agreements with less custom SQL and warehouse‑managed incremental logic. * Lambda views are best positioned as an *advanced / legacy pattern* you can still use for when you: * Want all logic in dbt SQL * Lack the right warehouse feature in your environment * Are extending an existing implementation already built this way --- ### Mapping project folders to warehouse schemas Folders organize your code. Schemas (or datasets, in BigQuery) organize the tables and views that code builds in your warehouse. You can map folders to schemas so staging, intermediate, and marts are easy to find. dbt prefixes each schema with your target schema, so development doesn’t overwrite production. #### Map folders to custom schemas By default, dbt builds every model in your target schema, which comes from the [environment](https://docs.getdbt.com/docs/dbt-platform-environments.md) in dbt platform or from the [profile](https://docs.getdbt.com/docs/local/dbt-environments.md) when you run locally (for example, development, CI, or production). To give the models in a folder their own schema, set `+schema` on that folder. dbt calls this a custom schema, which doesn't replace the target schema. Instead it joins the two as `_`, so development and production build into different schemas. Refer to [Custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). We recommend naming your custom schemas after your project folders, so staging, intermediate, and marts models are easy to find in your warehouse. For example, set `+schema: staging`, `+schema: intermediate`, and `+schema: marts`. If your team already uses different names, use those instead. For example, using `analytics_prod` and `alice_dev` as target schemas, dbt builds each folder into these schemas: | Folder | Config | Production (`analytics_prod`) | Development (`alice_dev`) | | ---------------------- | ----------------------- | ----------------------------- | ------------------------- | | `models/staging/` | `+schema: staging` | `analytics_prod_staging` | `alice_dev_staging` | | `models/intermediate/` | `+schema: intermediate` | `analytics_prod_intermediate` | `alice_dev_intermediate` | | `models/marts/` | `+schema: marts` | `analytics_prod_marts` | `alice_dev_marts` |
Sources point at tables that already exist, typically in `raw`. dbt reads those tables; it does not build them. For more information, refer to [Staging](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) and [Sources](https://docs.getdbt.com/docs/build/sources.md). note The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example can load seed data into `raw` when `load_source_data` is enabled. Those seeds use `+schema: raw` and land in the `raw` schema. #### Set custom schemas in `dbt_project.yml` When you set `+schema` on the folder in `dbt_project.yml`, every model in that folder inherits it. Configure a schema on an individual model only when that model should use a different schema. dbt\_project.yml ```yaml models: jaffle_shop: staging: +materialized: view +schema: staging intermediate: +materialized: view +schema: intermediate marts: +materialized: table +schema: marts ``` The `jaffle_shop` key must match the `name` in your `dbt_project.yml`. The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example has `staging` and `marts` only; the `intermediate` block is unused until you add that folder. This guide sets defaults on folders instead of on every model. Refer to [The rest of the project](https://docs.getdbt.com/best-practices/how-we-structure/5-the-rest-of-the-project.md). #### Use only the custom schema in production To use `marts` instead of `analytics_prod_marts` in production, use the built-in [`generate_schema_name_for_env`](https://docs.getdbt.com/docs/build/custom-schemas.md#a-built-in-alternative-pattern-for-generating-schema-names) pattern. It uses the custom schema alone when `target.name` is `prod`. In every other environment, it ignores custom schemas and builds all models in your target schema (`alice_dev`). Don't change the default macro to drop the target schema everywhere, or development runs can write to the same schemas as production. #### Use bronze, silver, and gold schema names If your warehouse uses bronze, silver, and gold schema names, those names map to the same folders previously discussed. You don't need a different dbt layout. The following table maps those names to the folders in this guide: | Warehouse name | In this guide | | -------------- | ---------------------------- | | Bronze | Sources in `raw` (read-only) | | Silver | Staging and intermediate | | Gold | Marts |
Keep `staging`, `intermediate`, and `marts` as the custom schema names in this guide so they match the folders. If your warehouse requires `silver` and `gold` as schema names, set `+schema` to those names instead. If you set `+schema: silver` on both `staging` and `intermediate`, those models share one schema and you lose a separate intermediate schema. To keep that separation, use two custom schema names. #### Related docs * [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [Custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) * [Staging](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) * [Intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) * [Marts](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md) * [The rest of the project](https://docs.getdbt.com/best-practices/how-we-structure/5-the-rest-of-the-project.md) * [Optimize dbt models on Databricks](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md) * [Best practices for dbt and Unity Catalog](https://docs.getdbt.com/best-practices/dbt-unity-catalog-best-practices.md) --- ### Marts: Business-defined entities info Our guidance here diverges if you use the Semantic Layer. In a project without the Semantic Layer we recommend you denormalize heavily, per the best practices below. On the other hand, if you're using the Semantic Layer, we want to stay as normalized as possible to allow MetricFlow the most flexibility. See [The dbt Semantic Layer and marts](#the-dbt-semantic-layer-and-marts) for more information. This is the layer where everything comes together and we start to arrange all of our atoms (staging models) into full-fledged cells that have identity and purpose. We sometimes like to call this the *entity* *layer* or *concept layer*, to emphasize that all our marts are meant to represent a specific entity or concept at its unique grain. For instance, an order, a customer, a territory, a click event, a payment — each of these would be represented with a distinct mart, and each row would represent a discrete instance of these concepts. Unlike in a traditional Kimball star schema though, in modern data warehousing — where storage is cheap and compute is expensive — we'll happily borrow and add any and all data from other concepts that are relevant to answering questions about the mart's core entity. Building the same data in multiple places, as we do with `orders` in our `customers` mart example below, is more efficient in this paradigm than repeatedly rejoining these concepts (this is a basic definition of denormalization in this context). Let's take a look at how we approach this first layer intended expressly for exposure to end users. ##### Marts: Files and folders The marts layer in our example project contains one model per core business entity: ```shell models/marts ├── customers.sql ├── customers.yml ├── locations.sql ├── locations.yml ├── order_items.sql ├── order_items.yml ├── orders.sql ├── orders.yml ├── products.sql ├── products.yml ├── supplies.sql └── supplies.yml ``` ✅ **Group by department or area of concern.** If you have fewer than 10 or so marts you may not have much need for subfolders, so as with the intermediate layer, don't over-optimize too early. If you do find yourself needing to insert more structure and grouping though, use useful business concepts here. In our marts layer, we're no longer worried about source-conformed data, so grouping by departments (marketing, finance, etc.) is the most common structure at this stage. ✅ **Name by entity.** Use plain English to name the file based on the concept that forms the grain of the mart's `customers`, `orders`. Marts that don't include any time-based rollups (pure marts) should not have a time dimension (`orders_per_day`) here, typically best captured via metrics. ❌ **Build the same concept differently for different teams.** `finance_orders` and `marketing_orders` is typically considered an anti-pattern. There are, as always, exceptions — a common pattern we see is that, finance may have specific needs, for example reporting revenue to the government in a way that diverges from how the company as a whole measures revenue day-to-day. Just make sure that these are clearly designed and understandable as *separate* concepts, not departmental views on the same concept: `tax_revenue` and `revenue` not `finance_revenue` and `marketing_revenue`. ##### Marts: Models Finally we'll take a look at the best practices for models within the marts directory by examining two of our marts models. These are the business-conformed — that is, crafted to our vision and needs — entities we've been bringing these transformed components together to create. ```sql -- orders.sql with orders as ( select * from {{ ref('stg_orders') }} ), order_items as ( select * from {{ ref('order_items') }} ), order_items_summary as ( select order_id, sum(supply_cost) as order_cost, sum(product_price) as order_items_subtotal, count(order_item_id) as count_order_items, sum( case when is_food_item then 1 else 0 end ) as count_food_items, sum( case when is_drink_item then 1 else 0 end ) as count_drink_items from order_items group by 1 ), compute_booleans as ( select orders.*, order_items_summary.order_cost, order_items_summary.order_items_subtotal, order_items_summary.count_food_items, order_items_summary.count_drink_items, order_items_summary.count_order_items, order_items_summary.count_food_items > 0 as is_food_order, order_items_summary.count_drink_items > 0 as is_drink_order from orders left join order_items_summary on orders.order_id = order_items_summary.order_id ) select * from compute_booleans ``` ```sql -- customers.sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('orders') }} ), customer_orders_summary as ( select orders.customer_id, count(distinct orders.order_id) as count_lifetime_orders, count(distinct orders.order_id) > 1 as is_repeat_buyer, min(orders.ordered_at) as first_ordered_at, max(orders.ordered_at) as last_ordered_at, sum(orders.subtotal) as lifetime_spend_pretax, sum(orders.tax_paid) as lifetime_tax_paid, sum(orders.order_total) as lifetime_spend from orders group by 1 ), joined as ( select customers.*, customer_orders_summary.count_lifetime_orders, customer_orders_summary.first_ordered_at, customer_orders_summary.last_ordered_at, customer_orders_summary.lifetime_spend_pretax, customer_orders_summary.lifetime_tax_paid, customer_orders_summary.lifetime_spend, case when customer_orders_summary.is_repeat_buyer then 'returning' else 'new' end as customer_type from customers left join customer_orders_summary on customers.customer_id = customer_orders_summary.customer_id ) select * from joined ``` * ✅ **Materialized as tables or incremental models.** Once we reach the marts layer, it's time to start building not just our logic into the warehouse, but the data itself. This gives end users much faster performance for these later models that are actually designed for their use, and saves us costs recomputing these entire chains of models every time somebody refreshes a dashboard or runs a regression in python. A good general rule of thumb regarding materialization is to always start with a view (as it takes up essentially no storage and always gives you up-to-date results), once that view takes too long to practically *query*, build it into a table, and finally once that table takes too long to *build* and is slowing down your runs, [configure it as an incremental model](https://docs.getdbt.com/docs/build/incremental-models.md). As always, start simple and only add complexity as necessary. The models with the most data and compute-intensive transformations should absolutely take advantage of dbt's excellent incremental materialization options, but rushing to make all your marts models incremental by default will introduce superfluous difficulty. We recommend reading this [classic post from Tristan on the limits of incremental modeling](https://discourse.getdbt.com/t/on-the-limits-of-incrementality/303). * ✅ **Wide and denormalized.** Unlike old school warehousing, in the modern data stack storage is cheap and it's compute that is expensive and must be prioritized as such, packing these into very wide denormalized concepts that can provide everything somebody needs about a concept as a goal. * ❌ **Too many joins in one mart.** One good rule of thumb when building dbt transformations is to avoid bringing together too many concepts in a single mart. What constitutes 'too many' can vary. If you need to bring 8 staging models together with nothing but simple joins, that might be fine. Conversely, if you have 4 concepts you're weaving together with some complex and computationally heavy window functions, that could be too much. You need to weigh the number of models you're joining against the complexity of the logic within the mart, and if it's too much to read through and build a clear mental model of then look to modularize. While this isn't a hard rule, if you're bringing together more than 4 or 5 concepts to create your mart, you may benefit from adding some [intermediate models](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) for added clarity. Two intermediate models that bring together three concepts each, and a mart that brings together those two intermediate models, will typically result in a much more readable chain of logic than a single mart with six joins. * ✅ **Build on separate marts thoughtfully.** While we strive to preserve a narrowing DAG up to the marts layer, once here things may start to get a little less strict. A common example is passing information between marts at different grains, as we saw above, where we bring our `orders` mart into our `customers` marts to aggregate critical order data into a `customer` grain. Now that we're really 'spending' compute and storage by actually building the data in our outputs, it's sensible to leverage previously built resources to speed up and save costs on outputs that require similar data, versus recomputing the same views and CTEs from scratch. The right approach here is heavily dependent on your unique DAG, models, and goals — it's just important to note that using a mart in building another, later mart is okay, but requires careful consideration to avoid wasted resources or circular dependencies. Marts are entity-grained. The most important aspect of marts is that they contain all of the useful data about a *particular entity* at a granular level. That doesn't mean we don't bring in lots of other entities and concepts, like tons of `user` data into our `orders` mart, we do! It just means that individual `orders` remain the core grain of our table. If we start grouping `users` and `orders` along a [date spine](https://github.com/dbt-labs/dbt-utils#date_spine-source), into something like `user_orders_per_day`, we're moving past marts into *metrics*. ##### Marts: Other considerations * **Troubleshoot via tables.** While stacking views and ephemeral models up until our marts — only building data into the warehouse at the end of a chain when we have the models we really want end users to work with — is ideal in production, it can present some difficulties in development. Particularly, certain errors may seem to be surfacing in our later models that actually stem from much earlier dependencies in our model chain (ancestor models in our DAG that are built before the model throws the errors). If you're having trouble pinning down where or what a database error is telling you, it can be helpful to temporarily build a specific chain of models as tables so that the warehouse will throw the error where it's actually occurring. ##### The dbt Semantic Layer and marts Our structural recommendations are impacted quite a bit by whether or not you're using the Semantic Layer. If you're using the Semantic Layer, we recommend a more normalized approach to your marts. If you're not using the Semantic Layer, we recommend a more denormalized approach that has become typical in dbt projects. For the full list of recommendations on structure, naming, and organization in the Semantic Layer, check out the [How we build our metrics](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md) guide, particularly the [Refactoring an existing rollup](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-8-refactor-a-rollup.md) section. --- ### Materializations best practices What *really* happens when you type `dbt build`? Contrary to popular belief, a crack team of microscopic data elves do *not* construct your data row by row, although the truth feels equally magical. This guide explores the real answer to that question, with an introductory look at the objects that get built into your warehouse, why they matter, and how dbt knows what to build. Learn by video! For video tutorials on Snapshots, go to dbt Learn and check out the [Snapshots course](https://learn.getdbt.com/courses/snapshots). The configurations that tell dbt how to construct these objects are called *materializations,* and knowing how to use them is a crucial skill for effective analytics engineering. When you’ve completed this guide, you will have that ability to use the three core materializations that cover most common analytics engineering situations. info 😌 **Materializations abstract away DDL and DML**. Typically in raw SQL- or python-based [data transformation](https://www.getdbt.com/analytics-engineering/transformation/), you have to write specific imperative instructions on how to build or modify your data objects. dbt’s materializations make this declarative, we tell dbt how we want things to be constructed and it figures out how to do that given the unique conditions and qualities of our warehouse. ##### Learning goals By the end of this guide you should have a solid understanding of: * 🛠️ what **materializations** are * 👨‍👨‍👧 how the three main materializations that ship with dbt — **table**, **view**, and **incremental** — differ * 🗺️ **when** and **where** to use specific materializations to optimize your development and production builds * ⚙️ how to **configure materializations** at various scopes, from an individual model to entire folder ##### Prerequisites * 📒 You’ll want to have worked through the [quickstart guide](https://docs.getdbt.com/guides.md) and have a project setup to work through these concepts. * 🏃🏻‍♀️ Concepts like dbt runs, `ref()` statements, and models should be familiar to you. * 🔧 \[**Optional**] Reading through the [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) Guide will be beneficial for the last section of this guide, when we review best practices for materializations using the dbt project approach of staging models and marts. ##### Guiding principle We’ll explore this in-depth throughout, but the basic guideline is **start as simple as possible**. We’ll follow a tiered approached, only moving up a tier when it’s necessary. * 🔍 **Start with a view.** When the view gets too long to *query* for end users, * ⚒️ **Make it a table.** When the table gets too long to *build* in your dbt Jobs, * 📚 **Build it incrementally.** That is, layer the data on in chunks as it comes in. --- ### More advanced metrics #### More advanced metric types We're not limited to just passing measures through to our metrics, we can also *combine* measures to model more advanced metrics. * 🍊 **Ratio** metrics are, as the name implies, about **comparing two metrics as a numerator and a denominator** to form a new metric, for instance the percentage of order items that are food items instead of drinks. * 🧱 **Derived** metrics are when we want to **write an expression** that calculates a metric **using multiple metrics**. A classic example here is our gross profit calculated by subtracting costs from revenue. * ➕ **Cumulative** metrics calculate all of a **measure over a given window**, such as the past week, or if no window is supplied, the all-time total of that measure. #### Ratio metrics * 🔢 We need to establish one measure that will be our **numerator**, and one that will be our **denominator**. * 🥪 Let's calculate the **percentage** of our Jaffle Shop revenue that **comes from food items**. * 💰 We already have our denominator, revenue, but we'll want to **make a new metric for our numerator** called `food_revenue`. models/marts/order\_items.yml ```yml - name: food_revenue description: The revenue from food in each order. label: Food Revenue type: simple type_params: measure: food_revenue ``` * 📝 Now we can set up our ratio metric. models/marts/order\_items.yml ```yml - name: food_revenue_pct description: The % of order revenue from food. label: Food Revenue % type: ratio type_params: numerator: food_revenue denominator: revenue ``` #### Derived metrics * 🆙 Now let's really have some fun. One of the most important metrics for any business is not just revenue, but *revenue growth*. Let's use a derived metric to build month-over-month revenue. * ⚙️ A derived metric has a couple key components: * 📚 A list of metrics to build on. These can be manipulated and filtered in various way, here we'll use the `offset_window` property to lag by a month. * 🧮 An expression that performs a calculation with these metrics. * With these parts we can assemble complex logic that would otherwise need to be 'frozen' in logical models. models/marts/order\_items.yml ```yml - name: revenue_growth_mom description: "Percentage growth of revenue compared to 1 month ago. Excluded tax" type: derived label: Revenue Growth % M/M type_params: expr: (current_revenue - revenue_prev_month) * 100 / revenue_prev_month metrics: - name: revenue alias: current_revenue - name: revenue offset_window: 1 month alias: revenue_prev_month ``` #### Cumulative metrics * ➕ Lastly, lets build a **cumulative metric**. In keeping with our theme of business priorities, let's continue with revenue and build an **all-time revenue metric** for any given time window. * 🪟 All we need to do is indicate the type is `cumulative` and not supply a `window` in the `type_params`, which indicates we want cumulative for the entire time period our end users select. models/marts/order\_items.yml ```yml - name: cumulative_revenue description: The cumulative revenue for all orders. label: Cumulative Revenue (All Time) type: cumulative type_params: measure: revenue ``` --- ### Near real-time data in dbt By design, dbt is batch-oriented with jobs having a defined start and end time. But did you know that you can also use dbt to get near real-time data by combining your data warehouse's continuous ingestion with frequent dbt transformations? This guide covers multiple patterns for achieving near real-time data freshness with dbt: 1. [Incremental patterns](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md) — `merge` strategies, Change Data Capture (CDC), and microbatch processing 2. [Warehouse-native features](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md) — When to use dynamic tables and materialized views 3. [Lambda views pattern](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/4-lambda-views.md) — Combining batch and real-time data in a single view 4. [Views-only pattern](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/5-views-only-pattern.md) - Maximum freshness for lightweight transformations 5. [Operational considerations](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/6-operational-considerations.md) — Challenges, risks, and cost management Each pattern includes practical code examples, use cases, and tradeoffs to help you choose the right approach. Anyone can use this guide, but it's primarily for data engineers and architects who want to achieve near real-time data freshness with dbt. This series is about freshness (minutes, not hours). For guidance on choosing incremental models vs snapshots for change history, refer to [CDC in dbt](https://docs.getdbt.com/best-practices/how-we-handle-cdc/1-intro.md). #### Where does dbt fit? There are two main ways to use dbt to get near real-time data: * For near real-time (5 - 15 minutes) — dbt excels at this and is well-suited for most operational dashboards. * For true real-time (sub-second) — This requires dedicated streaming databases (ClickHouse, Materialize, Rockset, and so on) in front of or alongside dbt; dbt still owns “analytic” tables and history but not the ultra‑low‑latency read path. #### How dbt achieves near real-time data To achieve real-time data with dbt, we recommend using a two-layer architecture: ###### Ingestion layer Continuous data landing using your data warehouse's streaming ingestion features. Streaming ingestion features such as [streaming tables](https://docs.databricks.com/en/sql/load-data-streaming-table.html), [Snowpipe](https://docs.snowflake.com/en/user-guide/snowpipe-streaming/data-load-snowpipe-streaming-overview), or [Storage Write API](https://docs.cloud.google.com/bigquery/docs/write-api-streaming) work well for this. To find streaming ingestion features for your warehouse, refer to the [additional resources](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#resources-by-warehouse) section. ###### dbt transformation layer Run dbt every few minutes to transform the data, and use materialized views or dynamic tables for the lowest-latency reporting. Specific transformation approaches include: * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md) with merge or append strategies * [Microbatch incremental strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) for large time-series tables * Jobs scheduled very frequently (like every 5 minutes) * [Dynamic tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) or [materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) with short refresh intervals #### Key recommendations The following are some key recommendations to help you achieve near real-time data freshness with dbt: * Ingest data continuously: Use your warehouse's native streaming or micro-batch ingestion to land raw data as soon as it arrives. * Transform with dbt on a frequent schedule: Schedule dbt jobs to run as often as your business needs allow (for example, every 1–15 minutes). Balance freshness with cost and resource constraints. * Materialized views and dynamic tables: For the lowest-latency reporting, use materialized views or dynamic tables. These can be refreshed as frequently as every minute. * Incremental models and microbatching: Use dbt's incremental models to process only new or changed data, keeping transformations efficient and scalable. * Decouple ingestion from transformation: Keep data acquisition and transformation flows separate. This allows you to optimize each independently. * Monitor and test data freshness: Implement data quality checks and freshness monitoring to ensure your near real-time pipelines deliver accurate, up-to-date results. * Cost and complexity considerations: Running dbt jobs more frequently drives up compute costs and operational complexity. Always weigh the business value against these trade-offs. --- ### Now it's your turn #### BYO Styles Now that you've seen how we style our dbt projects, it's time to build your own. Feel free to copy this guide and use it as a template for your own project. If you do, we'd love to hear about it! Reach out to us on [the Community Forum](https://discourse.getdbt.com/c/show-and-tell/22) or [Slack](https://www.getdbt.com/community) to share your style guide. We recommend co-locating your style guide with your code to make sure contributors can easily follow it. If you're using GitHub, you can add your style guide to your repository's wiki, or include it in your README. #### Pre-commit hooks You can use [pre-commit hooks](https://pre-commit.com/) to automatically check your code for style violations (and often fix them automagically) before you commit. This is a great way to make sure all contributors follow your style guide. We recommend implementing this once you've settled on and published your style guide, and your codebase is conforming to it. This will ensure that all future commits follow the style guide. You can find an excellent set of open source pre-commit hooks for dbt from the community [here in the dbt-checkpoint project](https://github.com/dbt-checkpoint/dbt-checkpoint). #### dbt Project Evaluator The [`dbt_project_evaluator`](https://github.com/dbt-labs/dbt-project-evaluator) is a package that ensures compliance to [dbt's style guide and best practices](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). The `dbt_project_evaluator` package highlights areas of a dbt project that are not aligned with dbt's best practices and provides recommendations on how to improve a project. This enables analytics engineers to determine exactly where their projects deviated from dbt's best practices and improve their projects on their own. The `dbt_project_evaluator` package covers the following categories: * Modeling * Testing * Documentation * Structure * Performance * Governance For more information, see [Introducing the dbt\_project\_evaluator: Automatically evaluate your dbt project for alignment with best practices](https://docs.getdbt.com/blog/align-with-dbt-project-evaluator). #### Style guide template ```markdown # dbt Example Style Guide ## SQL Style - Use lowercase keywords. - Use trailing commas. ## Model Organization Our models (typically) fit into two main categories:\ - Staging — Contains models that clean and standardize data. - Marts — Contains models which combine or heavily transform data. Things to note: - There are different types of models that typically exist in each of the above categories. See Model Layers for more information. - Read How we structure our dbt projects for an example and more details around organization. ## Model Layers - Only models in `staging` should select from sources. - Models not in the `staging` folder should select from refs. ## Model File Naming and Coding - All objects should be plural. Example: `stg_orders.sql` vs. `stg_order.sql` - All models should use the naming convention `___`. See this article for more information. - Models in the **staging** folder should use the source's name as the `` and the entity name as the `additional_context`. In a single-source project like Jaffle Shop, `stg_orders.sql` and `stg_customers.sql` are clear enough without the source prefix. Examples: - seed_snowflake_spend.csv - stg_orders.sql - stg_customers.sql - stg_salesforce__customers.sql - int_customers__unioned.sql - fct_orders.sql - Schema, table, and column names should be in `snake_case`. - Limit the use of abbreviations that are related to domain knowledge. An onboarding employee will understand `current_order_status` better than `current_os`. - Use names based on the _business_ rather than the source terminology. - Each model should have a primary key to identify the unique row and should be named `_id`. For example, `account_id`. This makes it easier to know what `id` is referenced in downstream joined models. - For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end. - Date/time columns should be named according to these conventions: - Timestamps: `_at` Format: UTC Example: `created_at` - Dates: `_date` Format: Date Example: `created_date` - Booleans should be prefixed with `is_` or `has_`. Example: `is_active_customer` and `has_admin_access` - Price/revenue fields should be in decimal currency (for example, `19.99` for $19.99; many app databases store prices as integers in cents). If a non-decimal currency is used, indicate this with suffixes. For example, `price_in_cents`. - Avoid using reserved words (such as these for Snowflake) as column names. - Consistency is key! Use the same field names across models where possible. For example, a key to the `customers` table should be named `customer_id` rather than `user_id`. ## Model Configurations - Model configurations at the folder level should be considered (and if applicable, applied) first. - More specific configurations should be applied at the model level using one of these methods. - Models within the `marts` folder should be materialized as `table` or `incremental`. - By default, `marts` should be materialized as `table` within `dbt_project.yml`. - If switching to `incremental`, this should be specified in the model's configuration. ## Testing - At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model. ## CTEs For more information about why we use so many CTEs, read this glossary entry. - Where performance permits, CTEs should perform a single, logical unit of work. - CTE names should be as verbose as needed to convey what they do. - CTEs with confusing or notable logic should be commented with SQL comments as you would with any complex functions and should be located above the CTE. - CTEs duplicated across models should be pulled out and created as their own models. ``` --- ### Operational considerations for near real-time data Teams that implement very high-frequency dbt jobs tend to run into a consistent set of challenges, both at the dbt scheduler layer and in the warehouse itself. Treat near real-time as a premium service Near real-time service level agreements (SLAs) require premium resources and add significant operational overhead. Pressure-test whether the business really needs minute-level freshness before committing. #### Over-scheduled jobs and queue management If a job's run duration is longer than its schedule frequency, the job becomes over-scheduled. The queue grows faster than the scheduler can process runs, and dbt platform will start cancelling queued runs to avoid an ever-expanding backlog. This is easy to hit with near real-time patterns if your incremental build time creeps up (more models, more tests, more data) but the cron schedule stays aggressive (for example, every 2–5 minutes). **Example scenario:** * Your job is scheduled to run every 5 minutes. * The job typically takes 6-7 minutes to complete. * New runs queue up while previous runs are still executing. * dbt platform starts cancelling queued runs to prevent infinite backlog. When this happens, remediation is non-trivial. You need to either refactor the job to run faster (prune model selection, adjust threads, optimize SQL) or relax the schedule and accept a looser freshness SLA. ###### Related scheduler constraints * Run slots limit how many jobs can run concurrently. Frequent near real-time jobs can starve other deployment jobs if slot usage isn't planned. * The scheduler runs distinct executions of the same job serially. If one run is still in progress when the next cron fires, the second run must wait (or be cancelled in an over-scheduled scenario). #### Warehouse cost and utilization As the gap between job runtime and schedule interval shrinks, your warehouse is effectively running continuously to keep up with back-to-back transformation windows. **Cost scaling example:** * Daily job: Warehouse runs 30 min/day = ~2% utilization * Hourly job: Warehouse runs 30 min × 24 = 12 hours/day = 50% utilization * 5-minute job: Warehouse runs nearly 24/7 = ~100% utilization On platforms like Snowflake, ingestion options like Snowpipe for high-volume real-time feeds can be very expensive (cost per 1,000 files plus compute). Warehouse-managed options for freshness (for example, dynamic tables and materialized views) can also be harder to predict and monitor from a cost perspective, especially when underlying data is changing frequently. The net effect: you should treat near real-time SLAs as a premium service and pressure-test whether the business really needs minute-level freshness on each workload. #### Lambda view DAG complexity and correctness If you're using the [lambda views pattern](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/4-lambda-views.md), you face additional complexity: * **Duplicated logic**: You either centralize SQL in macros (more DRY, less readable) or duplicate the same transformations in both history (HIST) and NRT flows (more readable, more to maintain). * **Complex DAGs**: Every "product" model now has at least three artifacts (HIST table, NRT view, lambda union), plus supporting upstream layers. * **Materialization brittleness**: The pattern depends on specific materializations (views vs incrementals). A seemingly harmless materialization change can break freshness or correctness. On top of that, community experience has surfaced timing gaps between HIST and NRT flows: * Views (NRT) often update much faster than incremental tables. During a run, the NRT side may start filtering on the new `max(event_ts)` before the incremental table has finished loading, producing temporary holes in the unioned lambda view where recent data disappears briefly. * One way to mitigate is to introduce an explicit dependency from NRT to the incremental model (for example, a manual dependency on `{{ ref('fct_events') }}` comment), but this is somewhat brittle and increases coupling. #### Job reliability and resource limits High-frequency jobs are more likely to surface job-level failures: * **Memory limits** * Memory-heavy macros (for example, large `run_query()` results) or big doc-generation steps can hit account-level memory limits. * This causes runs to terminate with "memory limit" errors. * **Auto-deactivation** * A job that fails repeatedly can be auto-deactivated after 100 consecutive failures. * When this happens, scheduled triggers stop until someone manually intervenes. * **Smaller margin for error** * A flaky model, test, or small regression can quickly generate many failed runs. * This creates noisy alerts and can hit the auto-deactivation threshold faster. #### Ingestion architecture dependencies Lambda views and NRT dbt jobs sit on top of your ingestion architecture: * **The dependency** * If ingestion latency or throughput degrades (issues in a task/stream pipeline, backlogs in storage, intermittent Snowpipe delays), the lambda view can only union what has already arrived. * You can't make data fresher than your ingestion layer allows. * **What you end up tuning** * Task cadences and partition strategies in the landing zone * Lambda overlap windows and incremental look-backs * Which sources really need to participate in the NRT path #### Conclusion These challenges are why we position lambda views and ultra-frequent dbt schedules as special-case patterns. They're powerful when you truly need them, but they require deliberate design around scheduler behavior, cost, DAG structure, and ingestion architecture. In many cases, they're better replaced by [dynamic tables](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#dynamic-tables), [materialized views](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#materialized-views), or a dedicated streaming stack. --- ### Optimize static analysis for development and deployment important dbt v2 is available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our dbt Community Slack channel [`#dbt-v2`](https://getdbt.slack.com/archives/C088YCAB6GH). Static analysis helps dbt v2 validate your SQL before it runs. This guide shows how to configure it so you get stronger checks while you develop, and faster, less blocking runs in deployment. This guide explains why using `strict` in development and `baseline` (the default, lighter static analysis mode) in deployment is a valid and recommended pattern, and how to configure it in your dbt v2 project. For more information about modes and features, refer to [About static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md). #### Why this pattern * **Development:** `strict` mode has the strongest SQL checks before you promote changes, including richer column-level features in the VS Code extension. * **Deployment:** `baseline` skips remote warehouse schema downloads and surfaces findings as warnings, so jobs are less likely to block. That can save compile time (and warehouse cost) in deployment, especially in projects with many sources. Review deployment logs for warnings that `strict` would have raised as errors in development. `strict` can increase compile time because dbt v2 downloads schemas for all sources (including sources your models do not reference). Teams with thousands of sources have seen large differences between `baseline` and `strict`. #### Set the mode with the CLI flag Use the [`--static-analysis`](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) flag to set the mode for a single run. **Development:** ```bash dbt compile --static-analysis strict ``` **Deployment:** ```bash dbt compile --static-analysis baseline ``` You can use the same flag with `dbt run` or `dbt build`. If you already have dbt v1 or the platform CLI installed alongside dbt v2, use `dbtf` as the unambiguous dbt v2 command. You can also configure [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) per directory or model. Refer to [Configuring `static_analysis`](https://docs.getdbt.com/docs/build/about-static-analysis.md#configuring-static_analysis) for examples. #### Set the mode with an environment variable You can also drive [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) from a custom environment variable in `dbt_project.yml`. This is useful when development and deployment share the same project config but set different environment values. dbt\_project.yml ```yml models: my_project: +static_analysis: "{{ env_var('DBT_ENV_STATIC_ANALYSIS', 'baseline') }}" ``` Then set the variable per environment: * Development: `DBT_ENV_STATIC_ANALYSIS=strict` * Deployment: leave unset (defaults to `baseline`), or set `DBT_ENV_STATIC_ANALYSIS=baseline` `DBT_ENV_STATIC_ANALYSIS` is a custom variable name you choose. It is separate from the built-in `DBT_STATIC_ANALYSIS` override used with the CLI flag. For more information about `env_var`, refer to [About env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) and [Environment variables](https://docs.getdbt.com/docs/build/environment-variables.md). #### Related docs * [About static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) * [`static_analysis` config](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) * [`--static-analysis` flag](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) --- ### Refactor an existing rollup #### A new approach Now that we've set the stage, it's time to dig in to the fun and messy part: how do we refactor an existing rollup in dbt into semantic models and metrics? Let's look at the differences we can observe in how we might approach this with MetricFlow supercharging dbt versus how we work without a Semantic Layer. These differences can then inform our structure. * 🍊 In dbt, we tend to create **highly denormalized datasets** that bring **everything you want around a certain entity or process into a single table**. * 💜 The problem is, this **limits the dimensionality available to MetricFlow**. The more we pre-compute and 'freeze' into place, the less flexible our data is. * 🚰 In MetricFlow, we ideally want **highly normalized**, star schema-like data that then allows MetricFlow to shine as a **denormalization engine**. * ∞ Another way to think about this is that instead of moving down a list of requested priorities trying to pre-make as many combinations of our marts as possible — increasing lines of code and complexity — we can **let MetricFlow present every combination possible without specifically coding it**. * 🏗️ To resolve these approaches optimally, we'll need to shift some **fundamental aspects of our modeling strategy**. #### Refactor steps outlined We recommend an incremental implementation process that looks something like this: 1. 👉 Identify **an important output** (a revenue chart on a dashboard for example, and the mart model(s) that supplies this output. 2. 🔍 Examine all the **entities that are components** of this rollup (for instance, an `active_customers_per_week` rollup may include customers, shipping, and product data). 3. 🛠️ **Build semantic models** for all the underlying component marts. 4. 📏 **Build metrics** for the required aggregations in the rollup. 5. 👯 Create a **clone of the output** on top of the Semantic Layer. 6. 💻 Audit to **ensure you get accurate outputs**. 7. 👉 Identify **any other outputs** that point to the rollup and **move them to the Semantic Layer**. 8. ✌️ Put a **deprecation plan** in place for the now extraneous frozen rollup. You would then **continue this process** on other outputs and marts moving down a list of **priorities**. Each model as you go along will be faster and easier as you'll **reuse many of the same components** that will already have been semantically modeled. #### Let's make a `revenue` metric So far we've been working in new pointing at a staging model to simplify things as we build new mental models for MetricFlow. In reality, unless you're implementing MetricFlow in a green-field dbt project, you probably are going to have some refactoring to do. So let's get into that in detail. 1. 📚 Per the above steps, let's say we've identified our target as a revenue rollup that is built on top of `orders` and `order_items`. Now we need to identify all the underlying components, these will be all the 'import' CTEs at the top of these marts. So in the Jaffle Shop project we'd need: `orders`, `order_items`, `products`, `locations`, and `supplies`. 2. 🗺️ We'll next make semantic models for all of these. Let's walk through a straightforward conversion first with `locations`. 3. ⛓️ We'll want to first decide if we need to do any joining to get this into the shape we want for our semantic model. The biggest determinants of this are two factors: * 📏 Does this semantic model **contain measures**? * 🕥 Does this semantic model have a **primary timestamp**? * 🫂 If a semantic model **has measures but no timestamp** (for example, supplies in the example project, which has static costs of supplies), you'll likely want to **sacrifice some normalization and join it on to another model** that has a primary timestamp to allow for metric aggregation. 4. 🔄 If we *don't* need any joins, we'll point our semantic model at the mart. Locations has a `tax_rate` measure and an `opened_date` time dimension, so we can reference the `locations` mart directly here. 5. 🥇 We specify our **primary entity** (based on `location_id`), dimensions (one categorical, `location_name`, and one **primary time dimension** `opened_date`), and lastly our measures, in this case just `average_tax_rate`. models/marts/locations.yml ```yaml semantic_models: - name: locations description: | Location dimension table. The grain of the table is one row per location. model: ref('locations') defaults: agg_time_dimension: opened_date entities: - name: location type: primary expr: location_id dimensions: - name: location_name type: categorical - name: opened_date expr: opened_date type: time type_params: time_granularity: day measures: - name: average_tax_rate description: Average tax rate. expr: tax_rate agg: average ``` #### Semantic and logical interaction Now, let's tackle a thornier situation. Products and supplies both have dimensions and measures but no time dimension. Products has a one-to-one relationship with `order_items`, enriching that table, which is itself just a mapping table of products to orders. Additionally, products have a one-to-many relationship with supplies. The high-level ERD looks like the diagram below. [![](/img/best-practices/semantic-layer/orders_erd.png?v=2)](#) So to calculate, for instance, the cost of ingredients and supplies for a given order, we'll need to do some joining and aggregating, but again we **lack a time dimension for products and supplies**. This is the signal to us that we'll **need to build a logical mart** and point our semantic model at that. tip **dbt 🧡 MetricFlow.** This is where integrating your semantic definitions into your dbt project really starts to pay dividends. The interaction between the logical and semantic layers is so dynamic, you either need to house them in one codebase or facilitate a lot of cross-project communication and dependency. 1. 🎯 Let's aim at, to start, building a table at the `order_items` grain. We can aggregate supply costs up, map over the fields we want from products, such as price, and bring the `ordered_at` timestamp we need over from the orders table. You can see example code, copied below, in `models/marts/order_items.sql`. models/marts/order\_items.sql ```sql with order_items as ( select * from {{ ref('stg_order_items') }} ), orders as ( select * from {{ ref('stg_orders')}} ), products as ( select * from {{ ref('stg_products') }} ), supplies as ( select * from {{ ref('stg_supplies') }} ), order_supplies_summary as ( select product_id, sum(supply_cost) as supply_cost from supplies group by 1 ), joined as ( select order_items.*, orders.ordered_at, products.product_name, products.product_price, products.is_food_item, products.is_drink_item, order_supplies_summary.supply_cost from order_items left join orders on order_items.order_id = orders.order_id left join products on order_items.product_id = products.product_id left join order_supplies_summary on order_items.product_id = order_supplies_summary.product_id ) select * from joined ``` 2. 🏗️ Now we've got a table that looks more like what we want to feed into the Semantic Layer. Next, we'll **build a semantic model on top of this new mart** in `models/marts/order_items.yml`. Again, we'll identify our **entities, then dimensions, then measures**. models/marts/order\_items.yml ```yml semantic_models: - name: order_item defaults: agg_time_dimension: ordered_at description: | Items contained in each order. The grain of the table is one row per order item. model: ref('order_items') entities: - name: order_item type: primary expr: order_item_id - name: order_id type: foreign expr: order_id - name: product type: foreign expr: product_id dimensions: - name: ordered_at expr: ordered_at type: time type_params: time_granularity: day - name: is_food_item type: categorical - name: is_drink_item type: categorical measures: - name: revenue description: The revenue generated for each order item. Revenue is calculated as a sum of revenue associated with each product in an order. agg: sum expr: product_price - name: food_revenue description: The revenue generated for each order item. Revenue is calculated as a sum of revenue associated with each product in an order. agg: sum expr: case when is_food_item then product_price else 0 end - name: drink_revenue description: The revenue generated for each order item. Revenue is calculated as a sum of revenue associated with each product in an order. agg: sum expr: case when is_drink_item then product_price else 0 end - name: median_revenue description: The median revenue generated for each order item. agg: median expr: product_price ``` 3. 📏 Finally, Let's **build a simple revenue metric** on top of our semantic model now. models/marts/order\_items.yml ```yaml metrics: - name: revenue description: Sum of the product revenue for each order item. Excludes tax. type: simple label: Revenue type_params: measure: revenue ``` #### Checking our work * 🔍 We always start our **auditing** with a `dbt parse` to **ensure our code works** before we examine its output. * 👯 If we're working there, we'll move to trying out an `dbt sl query` that **replicates the logic of the output** we're trying to refactor. * 💸 For our example we want to **audit monthly revenue**, to do that we'd run the query below. ##### Example query ```text dbt sl query --metrics revenue --group-by metric_time__month ``` ##### Example query results ```shell ✔ Success 🦄 - query completed after 1.02 seconds | METRIC_TIME__MONTH | REVENUE | |:---------------------|----------:| | 2016-09-01 00:00:00 | 17032.00 | | 2016-10-01 00:00:00 | 20684.00 | | 2016-11-01 00:00:00 | 26338.00 | | 2016-12-01 00:00:00 | 10685.00 | ``` * Try introducing some other dimensions from the semantic models into the `group-by` arguments to get a feel for this command. --- ### Semantic structure tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). #### Files and Folders The first thing you need to establish is how you’re going to consistently structure your code. There are two recommend best practices to choose from: * 🏡 **Co-locate your semantic layer code** in a one-YAML-file-per-marts-model system. * Puts documentation, data tests, unit tests, semantic models, and metrics into a unified file that corresponds to a dbt-modeled mart. * Trades larger file size for less clicking between files. * Simpler for greenfield projects that are building the Semantic Layer alongside dbt models. * 🏘️**Create a sub-folder** called `models/semantic_models/`. * Create a parallel file and folder structure within that specifically for semantic layer code. * Gives you more targeted files, but may involves switching between files more often. * Better for migrating large existing projects, as you can quickly see what marts have been codified into the Semantic Layer. It’s not terribly difficult to shift between these (it can be done with some relatively straightforward shell scripting), and this is purely a decision based on your developers’ preference (i.e. it has no impact on execution or performance), so don’t feel locked in to either path. Just pick the one that feels right and you can always shift down the road if you change your mind. tip Make sure to save all semantic models and metrics under the directory defined in the [`model-paths`](https://docs.getdbt.com/reference/project-configs/model-paths.md) (or a subdirectory of it, like `models/semantic_models/`). If you save them outside of this path, it will result in an empty `semantic_manifest.json` file, and your semantic models or metrics won't be recognized. #### Naming Next, establish your system for consistent file naming: * 1️⃣ If you’re doing **one-YAML-file-per-mart** then you’d have an `orders.sql` and an `orders.yml`. * 📛 If you’re using a **parallel subfolder approach**, for the sake of unique file names it’s recommended to use the **prefix `sem_` e.g. `sem_orders.yml`** for the dedicated semantic model and metrics that build on `orders.sql` and `orders.yml`. #### Can't decide? Start with a dedicated subfolder for your semantic models and metrics, and then if you find that you’re spending a lot of time clicking between files, you can always shift to a one-YAML-file-per-mart system. Our internal data team has found that the dedicated subfolder approach is more manageable for migrating existing projects, and this is the approach our documentation uses, so if you can't pick go with that. --- ### Set up the dbt Semantic Layer #### Getting started There are two options for developing a dbt project, including the Semantic Layer: * [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) — MetricFlow commands are embedded in the dbt platform CLI under the `dbt sl` subcommand. This is the easiest, most full-featured way to develop Semantic Layer code for the time being. You can use the editor of your choice and run commands from the terminal. * [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) — You can create semantic models and metrics in the Studio IDE. #### Basic commands * 🔍 A less common command that will come in handy with the Semantic Layer is `dbt parse`. This will parse your project and generate a **semantic manifest**, a representation of meaningful connections described by your project. This is uploaded to dbt, and used for running `dbt sl` commands in development. This file gives MetricFlow a **state of the world from which to generate queries**. * 🧰 `dbt sl query` is your other best friend, it will execute a query against your semantic layer and return a sample of the results. This is great for testing your semantic models and metrics as you build them. For example, if you're building a revenue model you can run `dbt sl query --metrics revenue --group-by metric_time__month` to validate that monthly revenue is calculating correctly. * 📝 Lastly, `dbt sl list dimensions --metrics [metric name]` will list all the dimensions available for a given metric. This is useful for checking that you're increasing dimensionality as you progress. You can `dbt sl list` other aspects of your Semantic Layer as well, run `dbt sl list --help` for the full list of options. For more information on the available commands, refer to the [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) reference, or use `dbt sl --help` and `dbt sl [subcommand] --help` on the command line. If you need to set up a dbt project first, check out the [quickstart guides](https://docs.getdbt.com/docs/get-started-dbt.md). #### Onward! Throughout the rest of the guide, we'll show example code based on the Jaffle Shop project, a fictional chain of restaurants. You can check out the code yourself and try things out in the [Jaffle Shop repository](https://github.com/dbt-labs/jaffle-shop). So if you see us calculating metrics like `food_revenue` later in this guide, this is why! --- ### Staging: Preparing our atomic building blocks The staging layer is where our journey begins. This is the foundation of our project, where we bring all the individual components we're going to use to build our more complex and useful models into the project. We'll use an analogy for working with dbt throughout this guide: thinking modularly in terms of atoms, molecules, and more complex outputs like proteins or cells (we apologize in advance to any chemists or biologists for our inevitable overstretching of this metaphor). Within that framework, if our source system data is a soup of raw energy and quarks, then you can think of the staging layer as condensing and refining this material into the individual atoms we’ll later build more intricate and useful structures with. ##### Staging: Files and folders Let's zoom into the staging directory from our `models` file tree [in the overview](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) and walk through what's going on here. ```shell models/staging ├── __sources.yml ├── stg_customers.sql ├── stg_customers.yml ├── stg_locations.sql ├── stg_locations.yml ├── stg_order_items.sql ├── stg_order_items.yml ├── stg_orders.sql ├── stg_orders.yml ├── stg_products.sql ├── stg_products.yml ├── stg_supplies.sql └── stg_supplies.yml ``` * **Folders.** Folder structure is extremely important in dbt. Not only do we need a consistent structure to find our way around the codebase, as with any software project, but our folder structure is also one of the key interfaces for understanding the knowledge graph encoded in our project (alongside the DAG and the data output into our warehouse). It should reflect how the data flows, step-by-step, from a wide variety of source-conformed models into fewer, richer business-conformed models. Moreover, we can use our folder structure as a means of selection in dbt [selector syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). For example, with the above structure, if we got fresh e-commerce data loaded and wanted to run all the models that build on our staging layer, we can easily run `dbt build --select staging+` and we're all set for building more up-to-date reports. * ✅ **Subdirectories based on the source system**. Our internal transactional database is one system, the data we get from Stripe's API is another, and lastly the events from our Snowplow instrumentation. We've found this to be the best grouping for most companies, as source systems tend to share similar loading methods and properties between tables, and this allows us to operate on those similar sets easily. The Jaffle Shop example project uses a single `ecom` source, so its staging models live in a flat `staging/` folder. As you add more source systems, create a subdirectory per source. * ❌ **Subdirectories based on loader.** Some people attempt to group by how the data is loaded (Fivetran, Stitch, custom syncs), but this is too broad to be useful on a project of any real size. * ❌ **Subdirectories based on business grouping.** Another approach we recommend against is splitting up by business groupings in the staging layer, and creating subdirectories like 'marketing', 'finance', etc. A key goal of any great dbt project should be establishing a single source of truth. By breaking things up too early, we open ourselves up to creating overlap and conflicting definitions (think marketing and financing having different fundamental tables for orders). We want everybody to be building with the same set of atoms, so in our experience, starting our transformations with our staging structure reflecting the source system structures is the best level of grouping for this step. * **File names.** Creating a consistent pattern of file naming is [crucial in dbt](https://docs.getdbt.com/blog/on-the-importance-of-naming). File names must be unique and correspond to the name of the model when selected and created in the warehouse. We recommend putting as much clear information into the file name as possible, including a prefix for the layer the model exists in, important grouping information, and specific information about the entity or transformation in the model. * ✅ `stg_[source]__[entity]s.sql` - the double underscore between source system and entity helps visually distinguish the separate parts in the case of a source name having multiple words. For instance, `google_analytics__campaigns` is always understandable, whereas to somebody unfamiliar `google_analytics_campaigns` could be `analytics_campaigns` from the `google` source system as easily as `campaigns` from the `google_analytics` source system. Think of it like an [oxford comma](https://www.youtube.com/watch?v=P_i1xk07o4g), the extra clarity is very much worth the extra punctuation. In a single-source project like Jaffle Shop, `stg_orders.sql` and `stg_customers.sql` are clear enough without the source prefix. * ❌ `stg_[entity].sql` - might be specific enough at first for a single-source project, but will break down in time as you add sources. Adding the source system into the file name aids in discoverability, and allows understanding where a component model came from even if you aren't looking at the file tree. * ✅ **Plural.** SQL, and particularly SQL in dbt, should read as much like prose as we can achieve. We want to lean into the broad clarity and declarative nature of SQL when possible. As such, unless there’s a single order in your `orders` table, plural is the correct way to describe what is in a table with multiple rows. ##### Staging: Models Now that we’ve got a feel for how the files and folders fit together, let’s look inside one of these files and dig into what makes for a well-structured staging model. Below is an example of a standard staging model (from our `stg_orders` model) that illustrates the common patterns within the staging layer. We’ve organized our model into two CTEs: one pulling in a source table via the [source macro](https://docs.getdbt.com/docs/build/sources.md#selecting-from-a-source) and the other applying our transformations. While our later layers of transformation will vary greatly from model to model, every one of our staging models will follow this exact same pattern. As such, we need to make sure the pattern we’ve established is rock solid and consistent. ```sql -- stg_orders.sql with source as ( select * from {{ source('ecom', 'raw_orders') }} ), renamed as ( select ---------- ids id as order_id, store_id as location_id, customer as customer_id, ---------- numerics subtotal as subtotal_cents, tax_paid as tax_paid_cents, order_total as order_total_cents, {{ cents_to_dollars('subtotal') }} as subtotal, {{ cents_to_dollars('tax_paid') }} as tax_paid, {{ cents_to_dollars('order_total') }} as order_total, ---------- timestamps {{ dbt.date_trunc('day', 'ordered_at') }} as ordered_at from source ) select * from renamed ``` * Based on the above, the most standard types of staging model transformations are: * ✅ **Renaming** * ✅ **Type casting** * ✅ **Basic computations** (e.g. cents to dollars) * ✅ **Categorizing** (using conditional logic to group values into buckets or booleans) * ❌ **Joins** — the goal of staging models is to clean and prepare individual source-conformed concepts for downstream usage. We're creating the most useful version of a source system table, which we can use as a new modular component for our project. In our experience, joins are almost always a bad idea here — they create immediate duplicated computation and confusing relationships that ripple downstream — there are occasionally exceptions though (refer to [base models](#staging-other-considerations) for more info). * ❌ **Aggregations** — aggregations entail grouping, and we're not doing that at this stage. Remember - staging models are your place to create the building blocks you’ll use all throughout the rest of your project — if we start changing the grain of our tables by grouping in this layer, we’ll lose access to source data that we’ll likely need at some point. We just want to get our individual concepts cleaned and ready for use, and will handle aggregating values downstream. * ✅ **Materialized as views.** Looking at a partial view of our `dbt_project.yml` below, we can see that we’ve configured the entire staging directory to be materialized as views. As they’re not intended to be final artifacts themselves, but rather building blocks for later models, staging models should typically be materialized as views for two key reasons: * Any downstream model (discussed more in [marts](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md)) referencing our staging models will always get the freshest data possible from all of the component views it’s pulling together and materializing * It avoids wasting space in the warehouse on models that are not intended to be queried by data consumers, and thus do not need to perform as quickly or efficiently ```yaml # dbt_project.yml models: jaffle_shop: staging: +materialized: view ``` * Staging models are the only place we'll use the [`source` macro](https://docs.getdbt.com/docs/build/sources.md), and our staging models should have a 1-to-1 relationship to our source tables. That means for each source system table we’ll have a single staging model referencing it, acting as its entry point — *staging* it — for use downstream. Don’t Repeat Yourself. Staging models help us keep our code DRY. dbt's modular, reusable structure means we can, and should, push any transformations that we’ll always want to use for a given component model as far upstream as possible. This saves us from potentially wasting code, complexity, and compute doing the same transformation more than once. For instance, if we know we always want our monetary values as floats in dollars, but the source system is integers and cents, we want to do the division and type casting as early as possible so that we can reference it rather than redo it repeatedly downstream. This is a welcome change for many of us who have become used to applying the same sets of SQL transformations in many places out of necessity! For us, the earliest point for these 'always-want' transformations is the staging layer, the initial entry point in our transformation process. The DRY principle is ultimately the litmus test for whether transformations should happen in the staging layer. If we'll want them in every downstream model and they help us eliminate repeated code, they're probably okay. ##### Staging: Other considerations * **Base models when joins are necessary to stage concepts.** Sometimes, in order to maintain a clean and DRY staging layer we do need to implement some joins to create a solid concept for our building blocks. In these cases, we recommend creating a sub-directory in the staging directory for the source system in question and building `base` models. These have all the same properties that would normally be in the staging layer, they will directly source the raw data and do the non-joining transformations, then in the staging models we'll join the requisite base models. Common use cases include joining in separate delete tables or unioning disparate but symmetrical sources. You can dig into [more detail on unioning identical sources here](https://discourse.getdbt.com/t/unioning-identically-structured-data-sources/921). The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example project does not use base models. * **[Codegen](https://github.com/dbt-labs/dbt-codegen) to automate staging table generation.** It's very good practice to learn to write staging models by hand, they're straightforward and numerous, so they can be an excellent way to absorb the dbt style of writing SQL. Also, we'll invariably find ourselves needing to add special elements to specific models at times — for instance, in one of the situations above that require base models — so it's helpful to deeply understand how they work. Once that understanding is established though, because staging models are built largely following the same rote patterns and need to be built 1-to-1 for each source table in a source system, it's preferable to start automating their creation. For this, we have the [codegen](https://github.com/dbt-labs/dbt-codegen) package. This will let you automatically generate all the source YAML and staging model boilerplate to speed up this step, and we recommend using it in every project. Development flow versus DAG order. This guide follows the order of the DAG, so we can get a holistic picture of how these three primary layers build on each other towards fueling impactful data products. It’s important to note though that developing models does not typically move linearly through the DAG. Most commonly, we should start by mocking out a design in a spreadsheet so we know we’re aligned with our stakeholders on output goals. Then, we’ll want to write the SQL to generate that output, and identify what tables are involved. Once we have our logic and dependencies, we’ll make sure we’ve staged all the necessary atomic pieces into the project, then bring them together based on the logic we wrote to generate our mart. Finally, with a functioning model flowing in dbt, we can start refactoring and optimizing that mart. By splitting the logic up and moving parts back upstream into intermediate models, we ensure all of our models are clean and readable, the story of our DAG is clear, and we have more surface area to apply thorough testing. --- ### Tactical terminology The rest of this guide will focus on the process of migrating your existing dbt code to the Semantic Layer. To do this, we'll need to introduce some new terminology and concepts that are specific to the Semantic Layer. We want to define them up front, as we have specific meanings in mind applicable to the process of migrating code to the Semantic Layer. These terms can mean different things in different settings, but here we mean: * 🔲 **Normalized** — can be defined with varying degrees of technical rigor, but used here we mean something that contains unique data stored only once in one place, so it can be efficiently joined and aggregated into various shapes. You can think of it referring to tables that function as conceptual building blocks in your business, *not* in the sense of say, strict [Codd 3NF](https://en.wikipedia.org/wiki/Third_normal_form). * 🛒 **Mart** — also has a variety of definitions, but here we mean a table that is relatively normalized and functions as the source of truth for a core concept in your business. * 🕸️ **Denormalized** — when we store the same data in multiple places for easier access without joins. The most denormalized data modeling system is OBT (One Big Table), where we try to get every possible interesting column related to a concept (for instance, customers) into one big table so all an analyst needs to do is `select`. * 🗞️ **Rollup** — used here as a catchall term meaning both denormalized tables built on top of normalized marts and those that perform aggregations to a certain grain. For example `active_accounts_per_week` might aggregate `customers` and `orders` data to a weekly time. Another example would be `customer_metrics` which might denormalize a lot of the data from `customers` as well as aggregated data from `orders`. For the sake of brevity in this guide, we’ll call all these types of products built on top of your normalized concepts as **rollups**. We'll also use a couple *new* terms for the sake of brevity. These aren't standard or official dbt-isms, but useful for communicating meaning in the context of refactoring code for the Semantic Layer: * 🧊 **Frozen** — shorthand to indicate code that is statically built in dbt’s logical transformation layer. Does not refer to the materialization type: views, incremental models, and regular tables are all considered *frozen* as they statically generate data or code that is stored in the warehouse as opposed to dynamically querying, as with the Semantic Layer. This is *not* a bad thing! We want some portion of our transformation logic to be frozen and stable as the *transformation* *logic* is not rapidly shifting and we benefit in testing, performance, and stability. * 🫠 **Melting** — the process of breaking up frozen structures into flexible Semantic Layer code. This allows them to create as many combinations and aggregations as possible dynamically in response to stakeholder needs and queries. tip 🏎️ **The Semantic Layer is a denormalization engine.** dbt transforms your data into clean, normalized marts. The Semantic Layer is a denormalization engine that dynamically connects and molds these building blocks into the maximum amount of shapes available *dynamically*. --- ### The rest of the project ##### Project structure review So far we’ve focused on the `models` folder, the primary directory of our dbt project. Next, we’ll zoom out and look at how the rest of our project files and folders fit in with this structure, starting with how we approach YAML configuration files. ```shell models ├── marts │ ├── customers.sql │ ├── customers.yml │ ├── order_items.sql │ ├── order_items.yml │ ├── orders.sql │ ├── orders.yml │ └── ... └── staging ├── __sources.yml ├── stg_customers.sql ├── stg_customers.yml ├── stg_orders.sql ├── stg_orders.yml └── ... ``` ##### YAML in-depth When structuring your YAML configuration files in a dbt project, you want to balance centralization and file size to make specific configs as easy to find as possible. It’s important to note that while the top-level YAML files (`dbt_project.yml`, `packages.yml`) need to be specifically named and in specific locations, the files containing your `sources` and `models` dictionaries can be named, located, and organized however you want. It’s the internal contents that matter here. As such, we’ll lay out our primary recommendation, as well as the pros and cons of a popular alternative. Like many other aspects of structuring your dbt project, what’s most important here is consistency, clear intention, and thorough documentation on how and why you do what you do. * ✅ **Config per folder.** As in the example above, create a `_[directory]__models.yml` per directory in your models folder that configures all the models in that directory. for staging folders, also include a `_[directory]__sources.yml` per directory. * The leading underscore ensures your YAML files will be sorted to the top of every folder to make them easy to separate from your models. * YAML files don’t need unique names in the way that SQL model files do, but including the directory (instead of simply `_sources.yml` in each folder), means you can fuzzy find the right file more quickly. * We’ve recommended several different naming conventions over the years, most recently calling these `schema.yml` files. We’ve simplified to recommend that these simply be labelled based on the YAML dictionary that they contain. * If you utilize [doc blocks](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) in your project, we recommend following the same pattern, and creating a `_[directory]__docs.md` markdown file per directory containing all your doc blocks for that folder of models. * ❌ **Config per project.** Some people put *all* of their source and model YAML into one file. While you can technically do this, and while it certainly simplifies knowing what file the config you’re looking for will be in (as there is only one file), it makes it much harder to find specific configurations within that file. We recommend balancing those two concerns. * ⚠️ **Config per model.** On the other end of the spectrum, some people prefer to create one YAML file per model. This presents less of an issue than a single monolith file, as you can quickly search for files, know exactly where specific configurations exist, spot models without configs (and thus without tests) by looking at the file tree, and various other advantages. In our opinion, the extra files, tabs, and windows this requires creating, copying from, pasting to, closing, opening, and managing creates a somewhat slower development experience that outweighs the benefits. Defining config per directory is the most balanced approach for most projects, but if you have compelling reasons to use config per model, there are definitely some great projects that follow this paradigm. * ✅ **Cascade configs.** Leverage your `dbt_project.yml` to set default configurations at the directory level. Use the well-organized folder structure we’ve created thus far to define the baseline schemas and materializations, and use dbt’s cascading scope priority to define variations to this. For example, as below, define your marts to be materialized as tables by default, define separate schemas for our separate subfolders, and any models that need to use incremental materialization can be defined at the model level. ```yaml # dbt_project.yml models: jaffle_shop: staging: +materialized: view +schema: staging intermediate: +materialized: view +schema: intermediate marts: +materialized: table +schema: marts ``` Refer to [Mapping project folders to warehouse schemas](https://docs.getdbt.com/best-practices/how-we-structure/6-mapping-folders-to-warehouse.md) to understand how those `+schema` values land in the warehouse in development and production. Define your defaults. One of the many benefits this consistent approach to project structure confers to us is this ability to cascade default behavior. Carefully organizing our folders and defining configuration at that level whenever possible frees us from configuring things like schema and materialization in every single model (not very DRY!) — we only need to configure exceptions to our general rules. Tagging is another area this principle comes into play. Many people new to dbt will rely on tags rather than a rigorous folder structure, and quickly find themselves in a place where every model *requires* a tag. This creates unnecessary complexity. We want to lean on our folders as our primary selectors and grouping mechanism, and use tags to define groups that are *exceptions.* A folder-based selection like \*\*`dbt build --select marts.marketing` is much simpler than trying to tag every marketing-related model, hoping all developers remember to add that tag for new models, and using `dbt build --select tag:marketing`. ###### Defining groups A group is a collection of nodes within a dbt DAG. Groups enable intentional collaboration within and across teams by restricting [access to private](https://docs.getdbt.com/reference/resource-configs/access.md) models. Groups are defined in `.yml` files, nested under a `groups:` key. In version 1.10 and higher, you can add a `description` and a `meta` config to add more information about the group. (Applies to dbt v1.10 and later) models/marts/finance/finance.yml ```yaml groups: - name: finance description: "All finance-related models owned by the Finance team." # optional owner: # 'name' or 'email' is required; additional properties will no longer be allowed in a future release email: finance@jaffleshop.com config: meta: # optional data_owner: Finance team cost_center: finance data_classification: sensitive ``` For more information about using groups, see [Add groups to your DAG](https://docs.getdbt.com/docs/build/groups.md). ##### How we use the other folders ```shell jaffle_shop ├── analyses ├── data-tests ├── macros │ └── cents_to_dollars.sql ├── seeds │ └── jaffle-data │ ├── raw_customers.csv │ ├── raw_orders.csv │ └── ... └── snapshots ``` We’ve focused heavily thus far on the primary area of action in our dbt project, the `models` folder. As you’ve probably observed though, there are several other folders in our project. While these are, by design, very flexible to your needs, we’ll discuss the most common use cases for these other folders to help get you started. * ✅ `seeds` for lookup tables. The most common use case for seeds is loading lookup tables that are helpful for modeling but don’t exist in any source systems — think mapping zip codes to states, or UTM parameters to marketing campaigns. In this example project we have seeds in `seeds/jaffle-data/` that can populate the `raw` schema for local development when `load_source_data` is enabled. * ❌ `seeds` for loading source data. Do not use seeds to load data from a source system into your warehouse. If it exists in a system you have access to, you should be loading it with a proper EL tool into the raw data area of your warehouse. dbt is designed to operate on data in the warehouse, not as a data-loading tool. * ✅ `analyses` for storing auditing queries. The `analyses` folder lets you store any queries you want to use Jinja with and version control, but not build into models in your warehouse. There are limitless possibilities here, but the most common use case when we set up projects at dbt Labs is to keep queries that leverage the [audit helper](https://github.com/dbt-labs/dbt-audit-helper) package. This package is incredibly useful for finding discrepancies in output when migrating logic from another system into dbt. * ✅ `tests` for testing multiple specific tables simultaneously. As dbt tests have evolved, writing singular tests has become less and less necessary. It's extremely useful for work-shopping test logic, but more often than not you'll find yourself either migrating that logic into your own custom generic tests or discovering a pre-built test that meets your needs from the ever-expanding universe of dbt packages (between the extra tests in [`dbt-utils`](https://github.com/dbt-labs/dbt-utils) and [`dbt-expectations`](https://github.com/calogica/dbt-expectations) almost any situation is covered). One area where singular tests still shine though is flexibly testing things that require a variety of specific models. If you're familiar with the difference between [unit tests](https://en.wikipedia.org/wiki/Unit_testing) [and](https://www.testim.io/blog/unit-test-vs-integration-test/) [integration](https://www.codecademy.com/resources/blog/what-is-integration-testing/) [tests](https://en.wikipedia.org/wiki/Integration_testing) in software engineering, you can think of generic and singular tests in a similar way. If you need to test the results of how several specific models interact or relate to each other, a singular test will likely be the quickest way to nail down your logic. * ✅ `snapshots` for creating [Type 2 slowly changing dimension](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) records from [Type 1](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_1:_overwrite) (destructively updated) source data. This is [covered thoroughly in the dbt Docs](https://docs.getdbt.com/docs/build/snapshots.md), unlike these other folders has a more defined purpose, and is out-of-scope for this guide, but mentioned for completion. * ✅ `macros` for DRY-ing up transformations you find yourself doing repeatedly. Like snapshots, a full dive into macros is out-of-scope for this guide and well [covered elsewhere](https://docs.getdbt.com/docs/build/jinja-macros.md), but one important structure-related recommendation is to [write documentation for your macros](https://docs.getdbt.com/faqs/Docs/documenting-macros.md). We recommend creating a `_macros.yml` and documenting the purpose and arguments for your macros once they’re ready for use. ##### Project splitting One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting, is fairly simple: in most cases, we recommend doing so with [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)! Mesh allows organizations to handle complexity by connecting several dbt projects rather than relying on one big, monolithic project. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! * ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using Mesh. For more information about Mesh, please refer to our [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md). * ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages.md#private-packages). * ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. * ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. #### Final considerations Overall, consistency is more important than any of these specific conventions. As your project grows and your experience with dbt deepens, you will undoubtedly find aspects of the above structure you want to change. While we recommend this approach for the majority of projects, every organization is unique! The only dogmatic advice we’ll put forward here is that when you find aspects of the above structure you wish to change, think intently about your reasoning and document for your team *how* and *why* you are deviating from these conventions. To that end, we highly encourage you to fork this guide and add it to your project’s README, wiki, or docs so you can quickly create and customize those artifacts. Finally, we emphasize that this guide is a living document! It will certainly change and grow as dbt and dbt Labs evolve. We invite you to join in — discuss, comment, and contribute regarding suggested changes or new elements to cover. --- ### Understanding a dbt project with dbt Wizard Use dbt Wizard to move from a broad project map to evidence-backed answers about model logic, lineage, tests, and data. This workflow helps when you join a project, review an unfamiliar area of the DAG, prepare a refactor, or investigate why a model exists. #### Prerequisites Start dbt Wizard from the dbt project root, and make sure the project has a current `target/manifest.json`. Refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) for setup. If you want dbt Wizard to inspect warehouse results, you also need a working development connection and permission to query the relevant relations. The prompts on this page work the same way in dbt platform. #### Start with a project map Ask for a read-only inventory before narrowing the investigation: ```text Map this dbt project for a new analytics engineer. Identify the main model layers, important sources, high fan-out models, test coverage gaps, and the three areas that deserve careful review. Cite the files and metadata you used. Do not edit anything. ``` This gives you a starting hypothesis, not a complete project specification. Ask follow-up questions about any labels or architectural claims that aren't supported by code, configuration, or metadata. #### Investigate one model Name a model and ask dbt Wizard to connect its definition to its place in the DAG: ```text Explain fct_orders to me. Describe its grain, inputs, important transformations, tests, materialization, and direct downstream consumers. Point out anything the code does not make clear. ``` For a thorough investigation, dbt Wizard can: 1. Find the model and its related YAML, tests, macros, and configuration. 2. Describe the selected columns, SQL, materialization, and dependencies. 3. Trace upstream inputs and downstream consumers through project lineage. 4. Query structured project metadata for test coverage and resource properties. 5. Preview or summarize warehouse data when a connection and relation are available. Ask dbt Wizard to separate observed facts from inferences. For example: ```text For each conclusion, label it as code, metadata, warehouse evidence, or an inference. List the business questions I still need to ask the model owner. ``` #### Trace a business concept You don't need to know the model name before you begin. Describe the concept and let dbt Wizard search for likely resources: ```text Trace how recurring revenue is calculated from source to final mart. Include the models, columns, macros, tests, and metrics involved. Call out where the definition changes or where multiple definitions exist. ``` When several candidates match, ask dbt Wizard to show the candidates and explain why each one might be relevant before continuing. #### Assess change impact Once you understand the current behavior, test a proposed change without editing files: ```text What would be affected if stg_payments changed from one row per payment to one row per payment attempt? Identify direct and indirect downstream resources, grain-dependent joins, tests, metrics, and exposures. Do not make changes. ``` Use the result to define the scope of a change plan. High fan-out models, contracts, metrics, exposures, and joins on the changed grain deserve closer review. #### Confirm assumptions with data SQL and metadata show intended behavior. Warehouse data can help you check whether that intent matches current results: ```text Check whether fct_orders is currently one row per order_id and summarize nulls or duplicates in its key columns. Use the smallest reasonable query and show me the query before running it. ``` Treat current data as evidence about the present state, not a permanent guarantee. Sampling, filters, stale relations, and environment differences can all affect the result. #### Turn findings into a plan End the investigation with a handoff that another engineer can review: ```text Summarize what we learned, unresolved business questions, affected resources, and a safe implementation and validation plan. Keep facts separate from recommendations. Do not edit files yet. ``` The summary should include relevant file paths, model names, lineage scope, supporting evidence, and explicit unknowns. #### Understand what can be missing dbt Wizard can reason from the project and the tools connected to the session. It can't reliably infer context that isn't represented there, including: * Undocumented business definitions and ownership decisions. * Dashboard or application dependencies that aren't represented as exposures or connected metadata. * Warehouse behavior that the current connection can't query. * The reason behind a historical design choice when it isn't recorded in code, documentation, or version history. * Data quality guarantees that don't have corresponding tests or contracts. Use model owners, pull request history, and business documentation to fill these gaps. #### Related docs * [Validate dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [dbt Wizard use cases](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [About model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) --- ### Validating dbt changes with dbt Wizard Use dbt Wizard to assess the impact of a change, choose a validation depth, and review evidence before you merge. This workflow is useful after you edit a model, test, macro, or YAML file and want more than a code-only review. dbt Wizard can combine project metadata, dbt commands, development builds, and production comparisons based on the validation level you select. CLI workflow The validation levels on this page are available in dbt Wizard CLI. Validation controls in the dbt platform can differ. Refer to [dbt Wizard in the dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) for platform behavior. #### Prerequisites Before you begin: * [Install and configure dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). * Open your dbt project in the terminal and start dbt Wizard from the project root. * Make sure the project has a current `target/manifest.json`. Run `dbt parse`, `dbt compile`, or `dbt build` if needed. * Configure a development target that dbt Wizard can use for commands that query or materialize data. * Review your Git status so you know which changes belong to the task. Warehouse validation can consume compute. dbt Wizard shows commands for approval according to your session policy before it runs them. #### Review the change and its impact Start by asking dbt Wizard to establish the scope before it runs commands or edits files: ```text Review my uncommitted changes. Explain which dbt resources changed, trace their downstream impact, and propose a validation plan. Do not edit files yet. ``` For a targeted review, name the resource and the behavior that must remain stable: ```text Validate the changes to fct_orders. Its grain must remain one row per order, order_total must not change for completed orders, and downstream finance models must still compile. Show me the validation plan before running it. ``` dbt Wizard uses the project graph and changed files to identify affected models, tests, and downstream resources. Review this scope carefully. Add any business invariant that cannot be inferred from SQL or metadata. #### Choose a validation level When dbt Wizard asks how thoroughly to validate a change, choose the level that matches its risk and cost. | Level | What dbt Wizard checks | Use it when | | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | Light | Checks SQL syntax, lints changed files when linting is supported, runs focused dbt tests, and reviews the code. It doesn't materialize changed models. | You made a low-risk change and need fast feedback. | | Medium | Compiles and lints, runs focused tests, materializes modified models in development, and checks affected downstream resources. This is the default level. | You changed model logic and need to verify the development result. | | Heavy | Performs medium validation, forms explicit expectations, compares development and production results, and reports row counts, schema differences, sample records, and downstream impact when the required data is available. | The change affects grain, financial logic, contracts, or other high-risk behavior. | | Skip | Doesn't run the structured validation workflow. | You only need a draft, or you plan to validate through another process. | The exact commands depend on your dbt engine, project, adapter, and selected resources. dbt Wizard can use commands such as `dbt compile`, `dbt lint`, `dbt test`, `dbt run`, and `dbt build`. Linting isn't available in every dbt CLI environment. #### Approve and monitor commands Before approving a command, check: * The selected models and downstream depth match the intended scope. * The target points to a development schema, unless you intentionally approved another target. * Deferral and state settings point to the expected environment. * A production comparison is read-only and uses the intended production relations. * The expected warehouse cost and run time are appropriate for the change. You can redirect the plan at any time. For example: ```text Do not build every downstream model. Build fct_orders and its first-degree children, then compile the rest of the downstream graph. ``` #### Review the validation result A useful validation summary should distinguish evidence from unresolved risk. Check that it includes: * The resources and commands that were validated. * Compile, lint, run, and test outcomes. * Development and production differences, when you selected heavy validation. * Downstream resources that weren't run or inspected. * Failures, warnings, skipped checks, and the reason for each skip. * Any assumptions that still need a subject-matter expert to confirm. If a check fails, ask dbt Wizard to investigate the failure before changing the test or expected result: ```text Explain whether this failure is caused by my code, existing warehouse data, or the validation environment. Do not weaken or remove the test. ``` #### Understand the limits Validation is evidence, not a guarantee that a change is correct. Keep these limits in mind: * A successful compile doesn't validate business logic or warehouse results. * Tests only cover the assertions encoded in the project. * A development-to-production comparison requires accessible relations with comparable schemas and data. * Sample records can reveal differences, but they don't prove that all rows are correct. * External consumers aren't included unless they are represented in dbt metadata or another connected tool. * A skipped check should remain visible in your review and pull request notes. #### Related docs * [Understanding a dbt project with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-2-understand-project.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [Use subagents with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) * [About dbt state](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) --- ### Views-only pattern for maximum freshness Snowflake examples ahead This page uses Snowflake for code examples, but you can adapt the views-only pattern to other warehouses. For some workloads, the simplest and most "real-time" pattern is to materialize everything as views on top of a continuously updated source table. When transformations are very lightweight and the source is already being updated in near real-time, this can preserve the source's latency almost perfectly. #### When to use the views-only pattern Use this when: * Source freshness is already "good enough" (for example, ingestion service or operational system writes into a warehouse table every few seconds or minutes). * You have very lightweight transformations, such as: * Simple projections / renames * One to two joins to small reference table * Minimal or no heavy aggregations or window functions * You care most about preserving the source table's latency and are willing to trade off some query performance at read time. * This works best for small-to-medium tables with simple queries. Typical examples: * Dashboards showing current system status (like active user sessions, current queue depth, or recent device heartbeats) where you need to see the latest data immediately. * Event data that you're forwarding to other tools with minimal transformation (raw data with just a bit of normalization, like cleaning up field names or adding a few reference fields). If your transform logic becomes heavier, multiple teams depend on the data, or you need better cost and performance control, transition to [incremental models](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md) or [dynamic tables/materialized views](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md). Reserve this pattern for the smallest, most latency‑sensitive use cases. ###### Assumptions The examples used in this page assume the following setup: * A non‑dbt system (ETL, streaming pipeline, app) is already writing into a warehouse‑resident table such as `raw.realtime_events` or `ops.active_sessions`, and that table meets your service level agreement for latency. * Querying that table directly is acceptable from a performance and cost standpoint for your expected concurrency. * You don't need dbt to persist intermediate tables; you mainly care about: * Consistent SQL logic (column naming, type casting) * Tests, contracts, and lineage * Exposures to BI / downstream tools All dbt models in this path are materialized as views, not tables or incremental models. #### Example implementation Here's an example implementation of the views-only pattern, which has the following pattern structure: * [Source table](#source-table-definition) (continuously updated): `raw.realtime_events` * [Thin staging view](#staging-view): `analytics.stg_realtime_events_v` * [Domain view(s)](#domain-view-definition): `analytics.vw_realtime_events_enriched` ##### Source table definition ```yaml # models/sources.yml version: 2 sources: - name: raw schema: raw tables: - name: realtime_events description: "Continuously updated event table from streaming pipeline." loaded_at_field: event_ts ``` ##### Staging view ```sql -- models/staging/stg_realtime_events.sql {{ config( materialized = 'view' ) }} select event_id, event_ts::timestamp_ntz as event_ts, -- Snowflake syntax for type casting to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'realtime_events') }}; ``` ##### Domain view definition ```sql -- models/marts/vw_realtime_events_enriched.sql {{ config( materialized = 'view' ) }} with base as ( select * from {{ ref('stg_realtime_events') }} ), user_dim as ( select user_id, user_segment, signup_date from {{ ref('dim_user') }} -- can be a table or incremental model ) select b.event_id, b.event_ts, b.event_date, b.user_id, u.user_segment, b.event_type, b.payload from base as b left join user_dim as u on b.user_id = u.user_id; ``` Downstream tools query `analytics.vw_realtime_events_enriched`. As long as `raw.realtime_events` is continuously updated, this view stack is as fresh as the source. #### Benefits * Maximum freshness: The view reflects new data as soon as it lands in `raw.realtime_events`. * Simple operations: No incremental logic to tune and no extra dbt job needed just to keep the data fresh. You still schedule jobs for tests, docs, and so on. * Best for small datasets: Works well when tables are small and queries are simple. Computing the view on the fly is cheap and fast. #### Limitations and risks This pattern is only safe under tight constraints and has several important limitations: * [Doesn't scale to heavy transformations](#doesnt-scale-to-heavy-transformations) * [No "frozen" intermediate tables](#no-frozen-intermediate-tables) * [Schema change sensitivity](#schema-change-sensitivity) * [Potential impact on operational systems](#potential-impact-on-operational-systems) ##### Doesn't scale to heavy transformations If your logic evolves into large joins, deep view chains, or expensive aggregations, you'll quickly run into performance issues: * Every query must re‑execute all the logic. * The warehouse has to optimize and execute the full stack of views every time. In those cases, use either of the following: * [Incremental models](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md) * [Dynamic tables or materialized views](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md), where appropriate ##### No "frozen" intermediate tables Because everything is a view: * There's no persisted intermediate layer to debug or profile. * You can't easily "rerun yesterday's logic" if upstream data changes—everything always reflects the current state. ##### Schema change sensitivity Schema changes in the source table propagate immediately through the view stack, which: * Can break downstream BI if columns are dropped or types change. * Make tests and model contracts more important, since there’s no batch boundary to catch issues before users see them. ##### Potential impact on operational systems If the continuously‑updated source is itself a live operational store (not a warehouse landing table), you must be careful not to overload it with analytics queries. In most cases, it is recommended to: 1. Replicate into a warehouse table first (Snowflake, BigQuery, Databricks, and so on). 2. Apply this views‑only pattern within the warehouse, not directly on the Online Transaction Processing system. --- ### Warehouse-native features for real-time data Modern data warehouses offer native features that can simplify near real-time data patterns. Instead of managing incremental logic yourself, you can declare the desired freshness and let the warehouse handle the refresh mechanics. This section covers when to use dynamic tables and materialized views instead of incremental models for near real-time data. * [Dynamic tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) are a warehouse-specific feature in Snowflake that lets the warehouse keep a table updated for you. You define what the table should look like, and the warehouse keeps the table fresh automatically. * [Materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) are a warehouse-specific feature that lets the warehouse save the results of a query so they’re faster to read, and refresh them as the underlying data changes. Note that the exact behavior depends on the warehouse. * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) are a dbt feature that lets dbt update a table by processing only new data. You tell dbt how new data should be added using your incremental logic SQL, and dbt runs the right SQL when the model is built. ###### When to consider warehouse-native features **Use dynamic tables or materialized views when:** * Your requirement is "data always within X minutes of real time" and you don't need precise scheduling control. * You want to simplify operational complexity by offloading refresh logic to the warehouse. * Your transformations are relatively straightforward. * You're willing to trade some control for convenience. **Stick with incremental models when:** * You need fine-grained control over scheduling and refresh logic, * You have complex business rules requiring custom incremental strategies (like microbatching). * You need to coordinate refreshes across multiple models in a specific order. #### Dynamic tables Warehouse support Dynamic tables are currently supported in Snowflake, with similar features available in other warehouses under different names. Check your warehouse documentation for availability. With dynamic tables, you can define the target state with SQL, and the warehouse automatically handles incremental refreshes. For example, the following SQL model uses a dynamic table to keep a table up to date for you: ```sql {{ config( materialized = 'dynamic_table', target_lag = '5 minutes', snowflake_warehouse = 'TRANSFORM_WH' -- Snowflake syntax ) }} select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'events') }} where event_ts >= current_timestamp() - interval '7 days'; ``` ###### target\_lag config The [`target_lag` parameter](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#target-lag) tells the warehouse the maximum acceptable staleness of the dynamic table relative to its sources, and helps determine when the table should be refreshed. For example: * `target_lag = '1 minute'` - Warehouse keeps the table within one minute of its source data, refreshing automatically as needed. * `target_lag = '5 minutes'` - Table may lag up to five minutes behind its sources. * `target_lag = 'downstream'` - Table refreshes only when a downstream table depends on it. * `target_lag = '1 minute'` - Data refreshed to be within 1 minute of the source * `target_lag = '5 minutes'` - Data within 5 minutes * `target_lag = 'downstream'` - Refresh when downstream tables need it The warehouse automatically determines when to refresh, whether to do a full or incremental update, and how to optimize the refresh query. ###### Benefits * Declarative freshness: specify "how fresh" not "when to refresh" * Warehouse-managed optimization * Cost predictability: refreshes run only when needed to meet `target_lag` * Simple setup ###### Limitations * Less control over exact timing or orchestration logic * Cost visibility can be harder to predict than scheduled dbt jobs * Less visibility into refresh decisions compared to dbt's explicit incremental logic * Currently warehouse-specific (implementation varies by platform) #### Materialized views Materialized views are available in most modern data warehouses and cache query results that automatically refresh when underlying data changes. Materialized views work like this: * The warehouse detects changes to source tables and refreshes the materialized view. * Many warehouses can incrementally update the view rather than recomputing everything. * Queries against the materialized view read cached results, not the underlying tables. For example, the following sql model uses a materialized view to keep a table up to date for you: ```sql {{ config( materialized = 'materialized_view' ) }} select user_id, date_trunc('hour', event_ts) as event_hour, count(*) as event_count from {{ source('raw', 'events') }} group by 1, 2; ``` #### Resources by warehouse Here are some resources for each warehouse: ###### BigQuery * [dbt developer docs: BigQuery materialized views](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#materialized-views) * [BigQuery docs: Materialized views intro](https://cloud.google.com/bigquery/docs/materialized-views-intro) * [BigQuery docs: Streaming API](https://docs.cloud.google.com/bigquery/docs/write-api) ###### Databricks * [dbt developer docs: Databricks materialized views and streaming tables](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#materialized-views-and-streaming-tables) * [Databricks docs: Materialized views](https://docs.databricks.com/en/views/materialized.html) ###### Postgres * [dbt developer docs: Postgres materialized views](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#materialized-views) * [Postgres docs: Materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) ###### Redshift * [dbt developer docs: Redshift materialized views](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#materialized-views) * [Redshift docs: Materialized views overview](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) * [Redshift docs: Streaming ingestion to a materialized view](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html) ###### Snowflake * [dbt developer docs: Dynamic tables configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) * [Snowflake docs: Dynamic tables intro](https://docs.snowflake.com/en/user-guide/dynamic-tables-intro) * [Snowflake blog: Dynamic tables for streaming pipelines](https://www.snowflake.com/en/blog/dynamic-tables-delivering-declarative-streaming-data-pipelines/) * [Snowflake docs: Materialized views](https://docs.snowflake.com/en/user-guide/views-materialized) #### Related docs * [dbt blog: Announcing materialized views](https://docs.getdbt.com/blog/announcing-materialized-views) * [dbt blog: Optimizing query run time with materialization schedules](https://www.getdbt.com/blog/optimizing-query-run-time-with-materialization-schedules/) --- ### Who is dbt Mesh for? Before embarking on a Mesh implementation, it's important to understand if Mesh is the right fit for your team. Here, we outline three common organizational structures to help teams identify whether Mesh might fit your organization's needs. #### The enterprise data mesh Some data teams operate on a global scale. By definition, the team needs to manage, deploy, and distribute data products across a large number of teams. Central IT may own some data products or simply own the platform upon which data products are built. Often, these organizations have “architects” who can advise line-of-business teams on their work while keeping track of what’s happening globally (regarding tooling and the substance of work). This is a lot like how software organizations work beyond a certain scale. The headcount ratio of domain teams to platform teams in this scenario is roughly ≥10:1. For each member of the central platform team, there might be dozens of members of domain-aligned data teams. Is Mesh a good fit in this scenario? Absolutely! There is no other way to share data products at scale. One dbt project would not keep up with the global demands of an organization like this. ##### Tips and tricks * **Managing shared macros**: Teams operating at this scale will benefit from a separate repository containing a dbt package of reusable utility macros that all other projects will install. This is different from public models, which provide data-as-a-service (a set of “API endpoints”) — this is distributed as a **library**. This package can also standardize imports of other third-party packages, as well as providing wrappers / shims for those macros. This package should have a dedicated team of maintainers — probably the central platform team, or a set of “superusers” from domain-aligned data modeling teams. tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! ##### Adoption challenges * Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that Mesh will help your teams deliver concrete value quickly. If this sounds like your organization, Mesh is the architecture you should pursue. ✅ #### Hub and spoke Some slightly smaller organizations still operate with a central data team serving several business-aligned analytics teams in a ~5:1 headcount ratio. These central teams look less like an IT function and more like a modern data platform team of analytics engineers. This team provides the majority of the data products to the rest of the org, as well as the infrastructure for downstream analytics teams to spin up their own spoke projects to ensure quality and maintenance of the core platform. Is Mesh a good fit in this scenario? Almost certainly! If your central data team starts to bottleneck analysts’ work, you need a way for those teams to operate relatively independently while still ensuring the quality of the most used data products. Mesh is designed to solve this exact problem. ##### Tips and tricks * **Data products by some, for all:** The spoke teams shouldn’t produce public models. By contrast, development in the hub team project should be slower, more careful, and focus on producing foundational public models shared across domains. We’d recommend giving hub team members access (at least read-only) to downstream projects, which will help with more granular impact analysis within Catalog. If a public model isn’t used in any downstream project or a specific column in that model, the hub team can feel better about removing it. However, they should still utilize the dbt governance features like `deprecation_date` and `version` as appropriate to set expectations. If there is a need for a public model in a spoke project to be shared across multiple projects, consider first whether it could or should be moved to the hub project. * **Sources:** Spokes should be allowed/encouraged to define and use *domain-specific* data sources. The platform team should not need to worry about, say, `Thinkific` data when building core data marts, but the Training project may need to. *No two sources anywhere in a dbt mesh should point to the same relation object.* If a spoke feels like they need to use a source the hub already uses, the interfaces should change so that the spoke can get what they need from the platform project. * **Project quality:** More analyst-focused teams will have different skill levels & quality bars. Owning their data means they own the consequences as well. Rather than being accountable for the end-to-end delivery of data assets, the Hub team is an enablement team: their role is to provide guardrails and quality checks, but not to fix all the issues exactly to their liking (and thereby remain a bottleneck). ##### Adoption challenges There are trade-offs to using this architecture, especially for the hub team managing and maintaining public models. This workflow has intentional friction to reduce the chances of unintentional model changes that break unspoken data contracts. These assurances may come with some sacrifices, such as faster onboarding or more flexible development workflows. Compared to having a single project, where a select few are doing all the development work, this architecture optimizes for slower development from a wider group of people. If this sounds like your organization, it's very likely that Mesh is a good fit for you. ✅ #### Single team monolith Some organizations operate on an even smaller scale. If your data org is a single small team that controls the end-to-end process of building and maintaining all data products at the organization, Mesh may not be required. The complexity in projects comes from having a wide variety of data sources and stakeholders. However, given the team's size, operating on a single codebase may be the most efficient way to manage data products. Generally, if a team of this size and scope is looking to implement Mesh, it's likely that they are looking for better interface design and/or performance improvements for certain parts of their dbt DAG, and not because they necessarily have an organizational pain point to solve. *Is Mesh a good fit?* Maybe! There are reasons to separate out parts of a large monolithic project into several to better orchestrate and manage the models. However, if the same people are managing each project, they may find that the overhead of managing multiple projects is not worth the benefits. If this sounds like your organization, it's worth considering whether Mesh is a good fit for you. --- ### Writing custom generic data tests dbt ships with [Not Null](https://docs.getdbt.com/reference/resource-properties/data-tests.md#not-null), [Unique](https://docs.getdbt.com/reference/resource-properties/data-tests.md#unique), [Relationships](https://docs.getdbt.com/reference/resource-properties/data-tests.md#relationships), and [Accepted Values](https://docs.getdbt.com/reference/resource-properties/data-tests.md#accepted-values) generic data tests. (These used to be called "schema tests," and you'll still see that name in some places.) Under the hood, these generic data tests are defined as `test` blocks (like macros). info There are tons of generic data tests defined in open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — the test you're looking for might already be here! ##### Generic tests with standard arguments Generic tests are defined in SQL files. Those files can live in two places: * `tests/generic/`: that is, a special subfolder named `generic` within your [test paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) (`tests/` by default) * `macros/`: Why? Generic tests work a lot like macros, and historically, this was the only place they could be defined. If your generic test depends on complex macro logic, you may find it more convenient to define the macros and the generic test in the same file. To define your own generic tests, simply create a `test` block called ``. All generic tests should accept one or both of the standard arguments: * `model`: The resource on which the test is defined, templated out to its relation name. (Note that the argument is always named `model`, even when the resource is a source, seed, or snapshot.) * `column_name`: The column on which the test is defined. Not all generic tests operate on the column level, but if they do, they should accept `column_name` as an argument. Here's an example of an `is_even` schema test that uses both arguments: tests/generic/test\_is\_even.sql ```sql {% test is_even(model, column_name) %} with validation as ( select {{ column_name }} as even_field from {{ model }} ), validation_errors as ( select even_field from validation -- if this is true, then even_field is actually odd! where (even_field % 2) = 1 ) select * from validation_errors {% endtest %} ``` If this `select` statement returns zero records, then every record in the supplied `model` argument is even! If a nonzero number of records is returned instead, then at least one record in `model` is odd, and the test has failed. To use this generic test, specify it by name in the `data_tests` property of a model, source, snapshot, or seed: (Applies to dbt v1.9 and later) models/\.yml ```yaml models: - name: users columns: - name: favorite_number data_tests: - is_even: description: "This is a test" ``` With one line of code, you've just created a test! In this example, `users` will be passed to the `is_even` test as the `model` argument, and `favorite_number` will be passed in as the `column_name` argument. You could add the same line for other columns, other models—each will add a new test to your project, *using the same generic test definition*. ##### Add description to generic data test logic You can add a description to the Jinja macro that provides the core logic for a data test by including the `description` key under the `macros:` section. You can add descriptions directly to the macro, including descriptions for macro arguments. Here's an example: tests/generic/schema.yml ```yaml macros: - name: test_not_empty_string description: Complementary test to default `not_null` test as it checks that there is not an empty string. It only accepts columns of type string. arguments: - name: model type: string description: Model Name - name: column_name type: string description: Column name that should not be an empty string ``` In this example: * When documenting custom test macros in a `schema.yml` file, add the `test_` prefix to the macro name. For example, if the test block's name is `not_empty_string`, then the macro's name would be `test_not_empty_string`. * We've provided a description at the macro level, explaining what the test does and any relevant notes. * Each argument (like `model`, `column_name`) also includes a description to clarify its purpose. ##### Generic tests with additional arguments The `is_even` test works without needing to specify any additional arguments. Other tests, like `relationships`, require more than just `model` and `column_name`. If your custom tests requires more than the standard arguments, include those arguments in the test signature, as `field` and `to` are included below: tests/generic/test\_relationships.sql ```sql {% test relationships(model, column_name, field, to) %} with parent as ( select {{ field }} as id from {{ to }} ), child as ( select {{ column_name }} as id from {{ model }} ) select * from child where id is not null and id not in (select id from parent) {% endtest %} ``` When calling this test from a `.yml` file, supply the arguments to the test in a dictionary. Note that the standard arguments (`model` and `column_name`) are provided by the context, so you do not need to define them again. (Applies to dbt v1.9 and later) models/\.yml ```yaml models: - name: people columns: - name: account_id data_tests: - relationships: description: "This is a test" arguments: # available in v1.10.5 and higher. Older versions can set the as the top-level property. to: ref('accounts') field: id ``` ##### Generic tests with default config values It is possible to include a `config()` block in a generic test definition. Values set there will set defaults for all specific instances of that generic test, unless overridden within the specific instance's `.yml` properties. tests/generic/warn\_if\_odd.sql ```sql {% test warn_if_odd(model, column_name) %} {{ config(severity = 'warn') }} select * from {{ model }} where ({{ column_name }} % 2) = 1 {% endtest %} ``` Any time the `warn_if_odd` test is used, it will *always* have warning-level severity, unless the specific test overrides that value: (Applies to dbt v1.9 and later) models/\.yml ```yaml models: - name: users columns: - name: favorite_number description: "Test favorite_number" data_tests: - warn_if_odd # default 'warn' - name: other_number description: "Test other_number" data_tests: - warn_if_odd: arguments: # available in v1.10.5 and higher. Older versions can set the as the top-level property. severity: error # overrides ``` ##### Customizing dbt's built-in tests To change the way a built-in generic test works—whether to add additional parameters, re-write the SQL, or for any other reason—you simply add a test block named `` to your own project. dbt will favor your version over the global implementation! tests/generic/\.sql ```sql {% test unique(model, column_name) %} -- whatever SQL you'd like! {% endtest %} ``` ##### Examples Here's some additional examples of custom generic ("schema") tests from the community: * [Creating a custom schema test with an error threshold](https://discourse.getdbt.com/t/creating-an-error-threshold-for-schema-tests/966) * [Using custom schema tests to only run tests in production](https://discourse.getdbt.com/t/conditionally-running-dbt-tests-only-running-dbt-tests-in-production/322) * [Additional examples of custom schema tests](https://discourse.getdbt.com/t/examples-of-custom-schema-tests/181) --- ## Category ### Account FAQs #### [Account-specific features](https://docs.getdbt.com/faqs/Accounts/account-specific-features.md) [Account-specific features](https://docs.getdbt.com/faqs/Accounts/account-specific-features.md) --- ### Available flags #### [Anonymous usage stats](https://docs.getdbt.com/reference/global-configs/usage-stats.md) [dbt Labs is on a mission to build the best version of dbt possible, and a crucial part of that is understanding how users work with dbt. To this end, we've added some simple event tracking (or telemetry) to dbt using Snowplow. Importantly, we do not track credentials, raw model contents, or model names: we consider these private, and frankly none of our business.](https://docs.getdbt.com/reference/global-configs/usage-stats.md) --- ### Behavior change flags #### [allow jinja file extensions](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) [| allowjinjafile\_extensions | v1 Latest | |](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) --- ### Cost Insights FAQs #### [Actual vs displayed costs]() [Explanation of why actual warehouse costs may differ from displayed costs]() --- ### dbt MCP FAQs #### [Troubleshooting](https://docs.getdbt.com/category/troubleshooting-1.md) [13 items](https://docs.getdbt.com/category/troubleshooting-1.md) --- ### dbt MCP troubleshooting FAQs #### [Generate HAR files](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md) [How to generate HAR files for debugging](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md) --- ### dbt MCP troubleshooting FAQs #### [Claude Code can't connect to the dbt MCP server](https://docs.getdbt.com/faqs/dbt-mcp/troubleshooting/mcp-troubleshoot-claude-code.md) [Troubleshooting: Claude Code can't connect to the dbt MCP server](https://docs.getdbt.com/faqs/dbt-mcp/troubleshooting/mcp-troubleshoot-claude-code.md) --- ### dbt v1 FAQs #### [Installing dbt v1 with pip](https://docs.getdbt.com/faqs/Core/install-pip-best-practices.md) [Instructions on how to install dbt v1 with pip](https://docs.getdbt.com/faqs/Core/install-pip-best-practices.md) --- ### Documentation FAQs #### [Types of columns included in doc site](https://docs.getdbt.com/faqs/Docs/document-all-columns.md) [All columns appear in your docs site](https://docs.getdbt.com/faqs/Docs/document-all-columns.md) --- ### Environments FAQs #### [Custom branch settings](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) [Use custom code from your repository](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) --- ### General configs #### [Advanced usage](https://docs.getdbt.com/reference/advanced-config-usage.md) [Alternative SQL file config syntax](https://docs.getdbt.com/reference/advanced-config-usage.md) --- ### General properties #### [anchors](https://docs.getdbt.com/reference/resource-properties/anchors.md) [Use the anchors key to hold reusable YAML config fragments so they pass file validation.](https://docs.getdbt.com/reference/resource-properties/anchors.md) --- ### Git FAQs #### [Branches when changing providers or repositories](https://docs.getdbt.com/faqs/Git/branch-migration.md) [How Git branch names and commit history carry over when you change providers, repositories, or remotes—for dbt v1 and the dbt platform, including platform-only notes for Studio IDE and repository settings.](https://docs.getdbt.com/faqs/Git/branch-migration.md) --- ### Jinja FAQs #### [Compiled sql has a lot of white space](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) [Managing whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) --- ### Jinja reference #### [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md) [Which dbt file types support Jinja expressions, built-in dbt functions, and custom macros in v1 and v2.](https://docs.getdbt.com/reference/jinja-file-support.md) --- ### List of commands #### [build](https://docs.getdbt.com/reference/commands/build.md) [The dbt build command will:](https://docs.getdbt.com/reference/commands/build.md) --- ### Models FAQs #### [Model configurations](https://docs.getdbt.com/faqs/Models/available-configurations.md) [Learning about model configurations](https://docs.getdbt.com/faqs/Models/available-configurations.md) --- ### Project configs #### [dbt\_project.yml](https://docs.getdbt.com/reference/dbt_project.yml.md) [Reference guide for configuring the dbt\_project.yml file.](https://docs.getdbt.com/reference/dbt_project.yml.md) --- ### Project FAQs #### [Add a seed file](https://docs.getdbt.com/faqs/Project/add-a-seed.md) [Learn how to add a seed file to your project](https://docs.getdbt.com/faqs/Project/add-a-seed.md) --- ### Runs FAQs #### [Reviewing SQL that dbt runs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) [Review logs to check the SQL dbt is running](https://docs.getdbt.com/faqs/Runs/checking-logs.md) --- ### Seeds FAQs #### [Build one seed at a time](https://docs.getdbt.com/faqs/Seeds/build-one-seed.md) [Use select flag to build one seed at a time](https://docs.getdbt.com/faqs/Seeds/build-one-seed.md) --- ### Setting flags #### [Command line options](https://docs.getdbt.com/reference/global-configs/command-line-options.md) [For consistency, command-line interface (CLI) flags should come right after the dbt prefix and its subcommands. This includes "global" flags (supported for all commands). For the full list of global CLI options for or (depending on your selected docs version), refer to Available flags. When set, CLI flags override environment variables and project flags.](https://docs.getdbt.com/reference/global-configs/command-line-options.md) --- ### Snapshots FAQs #### [Use hooks to run with snapshots](https://docs.getdbt.com/faqs/Snapshots/snapshot-hooks.md) [Run hooks with snapshots](https://docs.getdbt.com/faqs/Snapshots/snapshot-hooks.md) --- ### Tests FAQs #### [Available data tests to use in dbt](https://docs.getdbt.com/faqs/Tests/available-tests.md) [Types of data tests to use in dbt](https://docs.getdbt.com/faqs/Tests/available-tests.md) --- ### Warehouse FAQs #### [Setting up permissions in BigQuery"](https://docs.getdbt.com/faqs/Warehouse/bq-impersonate-service-account-setup.md) [Use service account to set up permissions in BigQuery](https://docs.getdbt.com/faqs/Warehouse/bq-impersonate-service-account-setup.md) --- ## Community ### Become a contributor #### Want to get involved? Start here The dbt Community predates dbt Labs as an organization and harkens back to the days when a scrappy analytics consultancy of a few [pissed off data analysts](https://www.hashpath.com/2020/12/an-analytics-engineer-is-really-just-a-pissed-off-data-analyst/#:~:text=Often%20times%2C%20an%20analytics%20engineer,necessity%20\(and%20genius%20branding\).) started hacking together an open source project around which gathered a community that would change how the world uses data. The dbt Community exists to allow analytics practitioners to share their knowledge, help others and collectively to drive forward the discipline of analytics engineering. This is something that can’t be done by any one individual or any one organization - to create a new discipline is necessarily a community effort. The only reason that dbt has become as widespread as it has is because people like you choose to get involved and share your knowledge. Contributing to the community can also be a great way to learn new skills, build up a public portfolio and make friends with other practitioners. There are opportunities here for everyone to get involved, whether you are just beginning your analytics engineering journey or you are a seasoned data professional. Contributing isn’t about knowing all of the answers, it’s about learning things together. Below you’ll find a sampling of the ways to get involved. There are a lot of options but these are ultimately just variations on the theme of sharing knowledge with the broader community. [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/community/contributing/contributing-writing.md) ###### [Writing contributions](https://docs.getdbt.com/community/contributing/contributing-writing.md) [Learn how to share and grow the collective knowledge of the dbt Community through blogs, guides, and documentation.](https://docs.getdbt.com/community/contributing/contributing-writing.md) [![](/img/icons/folder.svg)](https://docs.getdbt.com/community/contributing/contributing-coding.md) ###### [Coding contributions](https://docs.getdbt.com/community/contributing/contributing-coding.md) [The dbt Community supports a wide variety of open source and source-available projects, and this software is at the heart of everything we do. Learn how to get involved with projects in the dbt ecosystem.](https://docs.getdbt.com/community/contributing/contributing-coding.md) [![](/img/icons/discussions.svg)](https://docs.getdbt.com/community/contributing/contributing-online-community.md) ###### [Online community building](https://docs.getdbt.com/community/contributing/contributing-online-community.md) [Getting involved in the dbt Community Forum or Slack is one of the best entry points for contributing. Share your knowledge and learn from others.](https://docs.getdbt.com/community/contributing/contributing-online-community.md) [![](/img/icons/calendar.svg)](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md) ###### [Realtime event participation](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md) [Want to speak at a Meetup or conference? Learn how to get involved and check out best practices for crafting a talk that everyone will remember.](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md) --- ### Coding contributions ##### Contribute to dbt Packages ###### Overview [dbt Packages](https://docs.getdbt.com/docs/build/packages.md) are the easiest way for analytics engineers to get involved with contributing code to the dbt Community, because dbt Packages are just standard [dbt Projects](https://docs.getdbt.com/docs/build/projects.md). If you can create a dbt Project, write a macro, and ref a model: you can make a dbt Package. Packages function much like libraries do in other programming languages. They allow for prewritten, modularized development of code to solve common problems in analytics engineering. You can view all dbt Packages on the [dbt Package Hub](https://hub.getdbt.com/). ###### Contribution opportunities * Create a new package for the dbt Package Hub. This might be a new set of macros or tests that have been useful to you in your projects, a set of models for engaging with a commonly used datasource or anything else that can be done from within a dbt project. * Improve an existing package: Alternatively you can help improve an existing package. This can be done by creating and engaging with Issues or by functionality to address an existing issue via opening a PR. ###### Sample contributions * [dbt Expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) * [dbt Artifacts](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/) ###### Get started * Use packages in your own projects! The best way to know how to improve a package is to use it in a production environment then look for ways it can be modified or improved. * Read the following resources on package development: * [So You Want to Build a dbt Package](https://docs.getdbt.com/blog/so-you-want-to-build-a-package) * [Package Best Practices](https://github.com/dbt-labs/hubcap/blob/main/package-best-practices.md) * Need help: Visit #package-ecosystem in the dbt Slack ##### Contribute to dbt open source or source-available software ###### Overview dbt (including v1, v2, and adapters), as well as the sites powering the Package Hub and Developer Hub, are all vibrant community projects. Unlike dbt Packages, contributing code to these projects typically requires some working knowledge of programming languages outside of SQL and Jinja, but the supportive community around these repositories can help you advance those skills. Even without contributing code, there are many ways to be part of communal development in these projects, detailed below. You can find a curated list of the most active OSS/SA projects that dbt Labs supports [here](https://docs.getdbt.com/community/resources/oss-sa-projects.md). ###### Contribution opportunities There are three primary ways to contribute to the dbt projects. We’ll use dbt v1 as an example, as the "front door" to the dbt ecosystem and a great place to start for newcomers: * [Open an issue](https://github.com/dbt-labs/dbt/issues/new/choose) to suggest an improvement or give feedback. * Comment on or engage with existing [issues](https://github.com/dbt-labs/dbt/issues) or [discussions](https://github.com/dbt-labs/dbt/discussions). For example, upvote issues that would help your organization, comment to add nuance to a feature request, or share how a feature would impact your use of dbt. * Create a pull request that resolves an open Issue. This involves writing the code and tests that add the feature/resolve the bug described in an Issue, and then going through the code review process asynchronously with a dbt Labs engineer. Note that signed commits are required when contributing to dbt v1. For steps on how to sign commits, see [Signing commits](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits). ###### Sample contributions * Check out [this issue](https://github.com/dbt-labs/dbt/issues/3612) about improving error messages and [the PR that the community contributed to fix it](https://github.com/dbt-labs/dbt/pull/3703). * From the above issue [another issue was generated](https://github.com/dbt-labs/dbt-bigquery/issues/202) to change not just the error message, but improve the behavior. This is the virtuous cycle of community-driven development! Bit by bit we, the community, craft the tool to better fit our needs. ###### Get started * Read the dbt v1 [contribution guide](https://github.com/dbt-labs/dbt/blob/main/CONTRIBUTING.md) and the [Contributor Expectations](https://docs.getdbt.com/community/resources/contributor-expectations.md). * If contributing to `dbt-labs/dbt`, find an issue labeled “[good first issue](https://github.com/dbt-labs/dbt/issues?q=is%3Aopen+is%3Aissue+label%3A%22type%3Agood-first-issue%22)”, or look for similar labels on other repositories. If in doubt, also feel free to ask the maintainers for a good first issue, they’ll be excited to welcome you! ###### Need help? The following channels in the dbt Community Slack are a great place to ask questions: * \#dbt-development * \#adapter-ecosystem --- ### Community Forum guidelines #### What is the dbt Community Forum? [The dbt Community Forum](https://discourse.getdbt.com) is the platform for getting support with dbt as well as to have long-lived discussions about dbt, analytics engineering, and analytics. It's a place for us to build up a long-lasting knowledge base around the common challenges, opportunities, and patterns we work with every day. The forum is different from the dbt Community Slack in a few key ways, most importantly it is: * **Asynchronous** and **long-lived** - sometimes conversations continue over weeks, months, and beyond. * **Intentional** - we recommend taking at least 5 to 10 minutes thinking about and shaping your initial post and any comments. * **Citable** - Slack conversations tend to be great in the moment but get lost in the flow — forum posts can more easily shared and referenced. **Guidelines for engaging on the Forum** The community [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md) apply, and following them is the best way to get momentum behind your discussion or answers to your questions. The following guidelines will set you up for success: * Be respectful * Put effort into your posts * Mark replies as Solutions in the Help section * Don’t double post #### Categories The forum is broken down into three categories: * [Help](https://discourse.getdbt.com/c/help/19) * This is a Q\&A style forum where you can ask the dbt Community for help with specific questions about dbt, dbt, data modeling, or anything else you want a definitive answer on. * This category is for questions which can plausibly have a *single correct answer*. * ✅ How do I debug this Jinja error? * ✅ How do I set up CI in dbt on GitHub? * ❌ What is the best way to do marketing attribution? (More general Discussions like this are perfect for the [In-Depth Discussions](https://discourse.getdbt.com/c/discussions/21) category) * [Show and Tell](https://discourse.getdbt.com/c/show-and-tell/22) * This is the place to show off all of the cool things you are doing in dbt. Whether it’s a new macro, design pattern, or package, post here to show the community what you are up to! * [In-Depth Discussions](https://discourse.getdbt.com/c/discussions/21) * Share anything you’re thinking about that has to do with dbt or analytics engineering! This is a great place to jot down some thoughts to share with the community or spark a discussion on a topic that’s currently interesting you. #### Inclusivity on the Community Forum We are **strongly** committed to building a community where everyone can feel welcome. The dbt community started with people who were not traditionally considered “technical”, did not have ownership over technical systems, and were often left out of organizational decision-making. We came together to learn from each other, solve hard problems, and help build a new discipline where data folks have greater ownership over our own work. It really matters to us that everyone feels like they can ask questions and engage, no matter their professional or personal background. Technical forums have the potential to replicate harmful power structures, and can feel intimidating or hostile. We are working hard to create and sustain an inclusive environment through community-building, technological solutions, inclusive content, and diverse contributors. This is a long-term project, and we will continue to iterate and make improvements. If you have any ideas or feedback on how to make this space friendlier or more inclusive please let us know, either on the community Slack in the #community-strategy channel or via email at . We want to hear from you! #### Following new and ongoing Discussions The best way to stay up to date is to [browse the forum](https://discourse.getdbt.com/) directly. You can also Track or Watch specific threads or the whole category to receive updates on them without commenting. Each category also has a companion Slack channel (#advice-dbt-for-beginners, #show-and-tell and #in-depth-discussions). You can reply to the initial post in Slack and it will be added as a comment on the forum thread, allowing you to participate from inside Slack if you prefer. --- ### Contributor License Agreements #### Why we have a CLA As the sponsor of dbt, dbt Labs would like to ensure the long-term viability of dbt and its community. The Contributor License Agreement helps ensure everyone can enjoy dbt with confidence that dbt is here to stay. Specifically, our Contributor License Agreements (CLAs) grant the contributor and dbt Labs joint copyright interest in contributed code. Further, it provides assurance from the contributor that contributions are original work that does not violate any third-party license agreement. The agreement between contributors and project is explicit, so dbt users can be confident in the legal status of the source code and their right to use it. #### Our CLAs For all code contributions to dbt, we ask that contributors complete and sign a Contributor License Agreement. We have two different CLAs, depending on whether you are contributing to dbt in a personal or professional capacity: * [Individual Contributor License Agreement v1.0](https://docs.google.com/forms/d/e/1FAIpQLScfOV7K4enYRHozrDRP6BBIXjOij-JDGca6WBTHyP_ANXSqlg/viewform?usp=sf_link) * [Software Grant and Corporate Contributor License Agreement v1.0](https://docs.google.com/forms/d/e/1FAIpQLScDSTwGIlVyGWCMMvmszaXSE5IhIIRyeLQkgWf1-CSC2RnLww/viewform?usp=sf_link) * [Licenses FAQ](http://www.getdbt.com/licenses-faq) #### For Lawyers Our individual and corporate CLAs are based on the Contributor License Agreements published by the [Apache Software Foundation](http://www.apache.org/), with modifications: * [Individual Contributor License Agreement ("Agreement") V2.0](http://www.apache.org/licenses/icla.txt) * [Software Grant and Corporate Contributor License Agreement ("Agreement") v r190612](http://www.apache.org/licenses/cla-corporate.txt) * [Licenses FAQ](http://www.getdbt.com/licenses-faq) If you have questions about these CLAs, please contact us at . --- ### dbt Community Code of Conduct This Code of Conduct applies to all dbt Community spaces, both online and offline. This includes Slack, Discourse, code repositories (dbt v1, dbt v2, dbt packages, etc.), dbt Labs virtual events, and Meetups. Participants are responsible for knowing and abiding by this Code of Conduct. This Code of Conduct has two sections: * **dbt Community Values:** These values apply to all of our community spaces, and all of our guidelines are based on these values. * **Anti-harassment policy:** We are dedicated to providing a harassment-free experience for everyone in our community — here, we outline exactly what that means. We appreciate your support in continuing to build a community we’re all proud of. — The dbt Community Admin Team. #### dbt Community Values ##### Create more value than you capture. Each community member should strive to create more value in the community than they capture. This is foundational to being a community. Ways to demonstrate this value: * [Coding contributions](https://docs.getdbt.com/community/contributing/contributing-coding.md): Contribute to dbt v1, a package, or an adapter. Beyond implementing new functionality, you can also open issues or participate in discussions. * [Writing contributions](https://docs.getdbt.com/community/contributing/contributing-writing.md): You can suggest edits to every page of the dbt documentation, or suggest a topic for the dbt Developer Blog. * [Join in online](https://docs.getdbt.com/community/contributing/contributing-online-community.md): Ask and answer questions on the Discourse forum, kick off a lively discussion in Slack, or even maintain a Slack channel of your own. * [Participate in events](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md): Organise a community Meetup, speak at an event, or provide office space/sponsorship for an existing event. ##### Be you. Some developer communities allow and even encourage anonymity — we prefer it when people identify themselves clearly. It helps to build empathy, and form relationships. Ways to demonstrate this value: * Update your profile on dbt Community platforms to include your name and a clear picture of yourself. Where available, use the "what I do" section to add your role, title and current company. * Join your `#local-` channel in Slack, or if it doesn't exist then propose a new one. * Write in your own voice, and offer your own advice, rather than speaking in your company’s marketing or support voice. ##### Encourage diversity and participation. People with different mindsets and experiences, working together, create better outcomes. This includes diversity of race and gender, as well as the diversity of academic and career backgrounds, socio-economic backgrounds, geographic backgrounds, ideologies, and interests. Ways to demonstrate this value: * Make everyone in our community feel welcome, regardless of their background, and do everything possible to encourage participation in our community. * Demonstrate empathy for a community member’s experience — not everyone comes from the same career background, so adjust answers accordingly. * If you are sourcing speakers for events, put in additional effort to find speakers from underrepresented groups. ##### Be curious. Always ask yourself "why?" and strive to be continually learning. Ways to demonstrate this value: * Try solving a problem yourself before asking for help, e.g. rather than asking "what happens when I do X", experiment and observe the results! * When asking questions, explain the "why" behind your decisions, e.g. "I’m trying to solve X problem, by writing Y code. I’m getting Z problem" * When helping someone else, explain why you chose that solution, or if no solution exists, elaborate on the reason for that, e.g. "That’s not possible in dbt today — but here’s a workaround / check out this GitHub issue for a relevant discussion" #### Anti-harassment policy We are dedicated to providing a harassment-free experience for everyone. We do not tolerate harassment of participants in any form. Harassment includes: * Offensive comments related to gender, gender identity and expression, sexual orientation, disability, mental illness, neuro(a)typicality, physical appearance, body size, age, race, or religion. * Unwelcome comments regarding a person’s lifestyle choices and practices, including those related to food, health, parenting, drugs, and employment. * Deliberate misgendering or use of ‘dead’ or rejected names. * Gratuitous or off-topic sexual images or behaviour in spaces where they’re not appropriate. * Physical contact and simulated physical contact (eg, textual descriptions like "*hug*" or "*backrub*") without consent or after a request to stop. * Threats of violence. * Incitement of violence towards any individual, including encouraging a person to commit suicide or to engage in self-harm. * Deliberate intimidation. * Stalking or following. * Harassing photography or recording, including logging online activity for harassment purposes. * Sustained disruption of discussion. * Unwelcome sexual attention. * Pattern of inappropriate social contact, such as requesting/assuming inappropriate levels of intimacy with others * Continued one-on-one communication after requests to cease. * Deliberate "outing" of any aspect of a person’s identity without their consent except as necessary to protect vulnerable people from intentional abuse. * Publication of non-harassing private communication. Be mindful that others may not want their image or name on social media. Ask permission prior to posting about another person at in-person events. The dbt Community prioritizes marginalized people’s safety over privileged people’s comfort. The dbt Community Admin team reserves the right not to act on complaints regarding: * ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’ * Reasonable communication of boundaries, such as "leave me alone," "go away," or "I’m not discussing this with you." * Communicating in a ‘tone’ you don’t find congenial * Criticizing racist, sexist, cissexist, or otherwise oppressive behavior or assumptions ##### Reporting harassment If you are being harassed by a member of the dbt Community, notice that someone else is being harassed, or have any other concerns, please contact us at or use the workflows in [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) on Slack. We will respect confidentiality requests for the purpose of protecting victims of abuse. At our discretion, we may publicly name a person about whom we’ve received harassment complaints, or privately warn third parties about them, if we believe that doing so will increase the safety of dbt community members or the general public. We will not name harassment victims without their affirmative consent. ##### Consequences Participants asked to stop any harassing behavior are expected to comply immediately. If a participant engages in harassing behavior, the dbt Community Admin team may take any action they deem appropriate, up to and including expulsion from all dbt Community spaces and identification of the participant as a harasser to other dbt Community members or the general public. #### dbt Summit and other events We reserve the right to prohibit certain directly competitive companies from attending and / or sponsoring dbt Summit and other dbt Labs-hosted events. While we actively support the broader ecosystem around dbt, it is not the right business decision for us to allow directly competitive companies to market to or sell to our users and customers at events that we invest very significant company resources into hosting. Any event declines will be handled directly with the individuals / companies in question, and full refunds will be issued. Events that are hosted by other members of the dbt community, such as most dbt Community Meetups, are free to make their own guidelines about attendance. #### Credits Credit to [01.org](https://01.org/community/participation-guidelines), [Tizen.org](https://www.tizen.org/community/guidelines), and [Geek Feminism](https://geekfeminism.wikia.org/wiki/Community_anti-harassment/Policy) for some of the wording used in this Code of Conduct. --- ### dbt Community Rules of the Road The dbt Community includes over 100,000 data professionals and is still growing. People genuinely love this community. It's filled with smart, kind, and helpful people who share our commitment to elevating the analytics profession. We are committed to maintaining the spirit of this community, and have written these rules alongside its members to help everyone understand how to best participate. We appreciate your support in continuing to build a community we're all proud of. #### Expectations for all members ##### Rule 1: Be respectful We want everyone in this community to have a fulfilling and positive experience. Therefore, this first rule is serious and straightforward; we simply will not tolerate disrespectful behavior of any kind. Everyone interacting on a dbt platform – including Slack, the forum, codebase, issue trackers, and mailing lists – is expected to follow the [Community Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md). If you are unable to abide by the code of conduct set forth here, we encourage you not to participate in the community. ##### Rule 2: Keep it in public spaces Unless you have someone's express permission to contact them directly, do not directly message other community members, whether on a dbt Community platform or other spaces like LinkedIn. We highly value the time community members put into helping each other, and we have precisely zero tolerance for people who abuse their access to experienced professionals. If you are being directly messaged with requests for assistance without your consent, let us know in the [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) Slack channel. We will remove that person from the community. Your time and attention is valuable. ##### Rule 3: Follow messaging etiquette In short: put effort into your question, use threads, post in the right channel, and do not seek extra attention by tagging individuals or double-posting. For more information, see our [guide on getting help](https://docs.getdbt.com/community/resources/getting-help.md). ##### Rule 4: Do not solicit community members This community is built for data practitioners to discuss the work that they do, the ideas that they have, and the things that they are learning. It is decidedly not intended to be lead generation for vendors or recruiters. Vendors and recruiters are subject to additional rules to ensure this space remains welcoming to everyone. These requirements are detailed below and are enforced vigorously. #### Vendor expectations As a vendor/dbt partner, you are also a member of this community, and we encourage you to participate fully in the space. We have seen folks grow fantastic user relationships for their products when they come in with the mindset to share rather than pushing a pitch. At the same time, active community members have a finely honed sense of when they are being reduced to an audience or a resource to be monetized, and their response is reliably negative. Who is a vendor? Vendors are generally individuals belonging to companies that are creating products or services primarily targeted at data professionals, but this title also includes recruiters, investors, open source maintainers (with or without a paid offering), consultants and freelancers. If in doubt, err on the side of caution. ##### Rule 1: Identify yourself Include your company in your display name, e.g. "Alice (DataCo)". When joining a discussion about your product (after the waiting period below), be sure to note your business interests. ##### Rule 2: Let others speak first If a community member asks a question about your product directly, or mentions that they have a problem that your product could help with, wait 1 business day before responding to allow other members to share their experiences and recommendations. (This doesn't apply to unambiguously support-style questions from existing users, or in your `#tools-` channel if you have one). ##### Rule 3: Keep promotional content to specified spaces As a space for professional practice, the dbt Community is primarily a non-commercial space. However, as a service to community members who want to be able to keep up to date with the data industry, there are several areas available on the Community Slack for vendors to share promotional material: * [#vendor-content](https://getdbt.slack.com/archives/C03B0Q4EBL3) * [#events](https://getdbt.slack.com/archives/C80RCAZ5E) * \#tools-\* (post in [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) to request a channel for your tool/product) Recruiters may also post in [#jobs](https://getdbt.slack.com/archives/C7A7BARGT)/[#jobs-eu](https://getdbt.slack.com/archives/C04JMHHK6CD) but may not solicit applications in DMs. The definition of "vendor content" can be blurry at the edges, and we defer to members' instincts in these scenarios. As a rule, if something is hosted on a site controlled by that company or its employees (including platforms like Substack and Medium), or contains a CTA such as signing up for a mailing list or trial account, it will likely be considered promotional. ##### One more tip: Be yourself Speak in your own voice, and join in any or all of the conversations that interest you. Share your expertise as a data professional. Make a meme if you're so inclined. Get in a (friendly) debate. You are not limited to only your company's products and services, and making yourself known as a familiar face outside of commercial contexts is one of the most effective ways of building trust with the community. Put another way, [create more value than you capture](https://docs.getdbt.com/community/resources/code-of-conduct.md#create-more-value-than-you-capture). Because unaffiliated community members are able to share links in any channel, the most effective way to have your work reach a wider audience is to create things that are genuinely useful to the community. #### Handling violations The point of these rules is not to find opportunities to punish people, but to ensure the longevity of the community. Participation in this community is a privilege, and we reserve the right to remove people from it. To report an issue or appeal a judgement, email or use the workflows in [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) on Slack. Violations related to our anti-harassment policy will result in immediate removal. Other issues are handled in proportion to their impact, and may include: * a friendly, but public, reminder that the behavior is inappropriate according to our guidelines. * a private message with a warning that any additional violations will result in removal from the community. * temporary or permanent suspension of your account. --- ### dbt Labs Community #jobs Channels Terms and Conditions I agree to abide by the [dbt Community Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md) and all laws applicable to me in my use of the dbt Community's #jobs channels. I further agree: * dbt Labs is not responsible for not does it warrant or guarantee the validity, accuracy, completeness, legality, or reliability of any functionality of any #jobs channel, any posting's content, or any application and/or solicitation of any kind of employment. * dbt Labs does not review and approve job-related content. * dbt Labs disclaims liability of any kind whatsoever for any type of damage that occurs while using the community Slack for job-related reasons, and I waive any type of claim (including actual, special or consequential damages) to the maximum extent permitted by law. * Without limitation, dbt Labs disclaims liability for quality, performance, merchantability, and fitness for a particular purpose, express or implied, that may arise out of my use of the community Slack for job-related content, my reliance on such information, and/or my provision/receipt of job-related information. * I understand that no internet-based site is without risk, and my use is at my own risk. * My use of any job-posting template (or other forum for providing job-related information) confirms my consent to provide the data posted, confirms that I have permission to post such data, and is subject to the terms of the [dbt Labs privacy policy](https://www.getdbt.com/cloud/privacy-policy). For further information, please contact . --- ### Expectations for dbt contributors Whether it's `dbt-labs/dbt`, adapters, packages, or this very documentation site, contributing to the open source or source-available code that supports the dbt ecosystem is a great way to share your knowledge, level yourself up as a developer, and to give back to the community. The goal of this page is to help you understand what to expect when contributing to dbt ecosystem projects. Have you seen things in other projects that you like, and think we could learn from? [Open a discussion on the dbt Community Forum](https://discourse.getdbt.com), or start a conversation in the [dbt Community Slack](https://www.getdbt.com/community/join-the-community) (for example: `#community-strategy`, `#dbt-development`, `#package-ecosystem`, `#adapter-ecosystem`). We always appreciate hearing from you! #### Principles ##### dbt is a team sport We all build dbt together -- whether you write code or contribute your ideas. By using dbt, you're invested in the future of the tool, and have an active role in pushing forward the standard of analytics engineering. You already benefit from using code and documentation contributed by community members. Contributing to the dbt community is your way to be an active participant in the thing we're all creating together. There's a very practical reason, too: building in public prioritizes our collective knowledge and experience over any one person's. We don't have experience using every database, operating system, security environment, ... We rely on the community of users to hone our product capabilities and documentation to the wide variety of contexts in which it operates. In this way, dbt gets to be the handiwork of thousands, rather than a few dozen. ##### We take seriously our role as maintainers of a standard As a standard, dbt must be reliable and consistent. Our first priority is ensuring the continued high quality of existing dbt capabilities before we introduce net-new capabilities. We also believe dbt as a framework should be extensible enough to ["make the easy things easy, and the hard things possible"](https://en.wikipedia.org/wiki/Perl#Philosophy). To that end, we *don't* believe it's appropriate for dbt to have an out-of-the-box solution for every niche problem. Users have the flexibility to achieve many custom behaviors by defining their own macros, materializations, hooks, and more. We view it as our responsibility as maintainers to decide when something should be "possible" — via macros, packages, etc. — and when something should be "easy" — built into the dbt standard. So when will we say "yes" to new capabilities for dbt? The signals we look for include: * Upvotes on issues in our GitHub repos * Open source dbt packages trying to close a gap * Technical advancements in the ecosystem In the meantime — we'll do our best to respond to new issues with: * Clarity about whether the proposed feature falls into the intended scope of dbt's source-available components * Context (including links to related issues) * Alternatives and workarounds * When possible, pointers to code that would aid a community contributor ##### Initiative is everything Given that we, as maintainers, will not be able to resolve every bug or flesh out every feature request, we empower you, as a community member, to initiate a change. * If you open the bug report, it's more likely to be identified. * If you open the feature request, it's more likely to be discussed. * If you comment on the issue, engaging with ideas and relating it to your own experience, it's more likely to be prioritized. * If you open a PR to fix an identified bug, it's more likely to be fixed. * If you comment on an existing PR, to confirm it solves the concrete problem for your team in practice, it's more likely to be merged. Sometimes, this can feel like shouting into the void, especially if you aren't met with an immediate response. We promise that there are dozens (if not hundreds) of folks who will read your comment, including us as maintainers. It all adds up to a real difference. #### Practicalities ##### Discussions A discussion is best suited to propose a Big Idea, such as brand-new capability in the v1 framework. Anyone can open a discussion, comment on an existing one, or reply in a thread. When you open a new discussion, you might be looking for validation from other members of the community — folks who identify with your problem statement, who like your proposed idea, and who may have their own ideas for how it could be improved. The most helpful comments propose nuances or desirable user experiences to be considered in design and refinement. Unlike an **issue**, there is no specific code change that would “resolve” a discussion. If, over the course of a discussion, we reach a consensus on specific elements of a proposed design, we can open new implementation issues that reference the discussion for context. Those issues will connect desired user outcomes to specific implementation details, acceptance testing, and remaining questions that need answering. ##### Issues An issue could be a bug you've identified while using the product or reading the documentation. It could also be a specific idea you've had for a narrow extension of existing functionality. ###### Best practices for issues * Issues are **not** for support / troubleshooting / debugging help. Please see [dbt support](https://docs.getdbt.com/docs/dbt-support.md) for more details and suggestions on how to get help. * Always search existing issues first, to see if someone else had the same idea / found the same bug you did. * Many dbt repositories offer templates for creating issues, such as reporting a bug or requesting a new feature. If available, please select the relevant template and fill it out to the best of your ability. This information helps us (and others) understand your issue. ###### You've found an existing issue that interests you. What should you do? Comment on it! Explain that you've run into the same bug, or had a similar idea for a new feature. If the issue includes a detailed proposal for a change, say which parts of the proposal you find most compelling, and which parts give you pause. ###### You've opened a new issue. What can you expect to happen? In our most critical repositories (such as `dbt-labs/dbt`), our goal is to respond to new issues as soon as possible. This initial response will often be a short acknowledgement that the maintainers are aware of the issue, signalling our perception of its urgency. Depending on the nature of your issue, it might be well suited to an external contribution, from you or another community member. **What if you're opening an issue in a different repository?** We have engineering teams dedicated to active maintenance of [`dbt-labs/dbt`](https://github.com/dbt-labs/dbt), home to both v1 and v2. Some things to note about the `dbt-labs/dbt` repo when you're contributing: * v2 lives on the `main` branch * v1 lives on the `latest.1` branch, as well as the component libraries of v1 ([`dbt-common`](https://github.com/dbt-labs/dbt-common) + [`dbt-adapters`](https://github.com/dbt-labs/dbt-adapters) (which also includes the dbt Labs managed adapters). `dbt-adapters` applies to v1 only. We've open-sourced a number of other software projects over the years, and the majority of them do not have the same activity or maintenance guarantees. Check to see if other recent issues have responses, or when the last commit was added to the `main` branch. **You're not sure about the status of your issue.** If your issue is in an actively maintained repo and has a `status:triage` label attached, we're aware it's something that needs a response. If the issue has been triaged, but not prioritized, this could mean: * The intended scope or user experience of a proposed feature requires further refinement from a maintainer * We believe the required code change is too tricky for an external contributor We'll do our best to explain the open questions or complexity, and when / why we could foresee prioritizing it. **Automation that can help us:** In many repositories, we use a bot that marks issues as stale if they haven't had any activity for 180 days. This helps us keep our backlog organized and up-to-date. We encourage you to comment on older open issues that you're interested in, to keep them from being marked stale. You're also always welcome to comment on closed issues to say that you're still interested in the proposal. ###### Issue labels In all likelihood, the maintainer who responds will also add a number of labels. Not all of these labels are used in every repository. In some cases, the right resolution to an open issue might be tangential to the codebase. The right path forward might be in another codebase (we'll transfer it), a documentation update, or a change that you can make yourself in user-space code. In other cases, the issue might describe functionality that the maintainers are unwilling or unable to incorporate into the main codebase. In these cases, a maintainer will close the issue (perhaps using a `wontfix` label) and explain why. Many labels follow a namespaced `facet:value` model, where each facet answers one question about an issue — the kind of work it is, which subsystem it touches, which dbt engine it targets, and so on. Some facets take exactly one value; others can take several. You don't need to apply these yourself, but knowing what they mean helps you follow an issue's status. | Facet | Example values | What it means | | ------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | `type:` | `type:bug`, `type:feature`, `type:docs`, `type:tech-debt`, `type:epic`, `type:good-first-issue` | The kind of work. Exactly one per issue. | | `area:` | `area:static-analysis`, `area:engine`, `area:compute`, `area:adapters`, `area:vscode`, `area:wizard`, `area:autofix`, `area:cicd` | Which subsystem the issue touches. | | `adapter:` | `adapter:bigquery`, `adapter:snowflake`, `adapter:databricks`, `adapter:postgres`, … | Which adapter or SQL dialect is involved. | | `engine:` | `engine:v1`, `engine:v2` | Which dbt engine the issue concerns: `engine:v1` is the Python engine (dbt v1); `engine:v2` is the v2 Rust rewrite. | | `priority:` | `priority:p0`, `priority:p1`, `priority:p2` | Maintainer triage urgency. | | `status:` | `status:triage`, `status:needs-repro`, `status:has-repro`, `status:ready-for-review` | Where the issue sits in the workflow. | | `resolution:` | `resolution:wontfix` | Why an issue was closed without a code change. | Automation labels new issues with `status:triage` to mark them as awaiting a maintainer's first response. Older issues may also carry previous, un-namespaced labels such as `bug`, `enhancement`, `good_first_issue`, `help_wanted`, `duplicate`, `stale`, and `wontfix`. These map onto the faceted labels above (`bug` → `type:bug`, `enhancement` → `type:feature`, `good_first_issue` → `type:good-first-issue`, `wontfix` → `resolution:wontfix`, and so on) and remain in place for continuity. You may see either set while the transition settles. ##### Pull requests **Every PR should be associated with an issue.** Why? Before you spend a lot of time working on a contribution, we want to make sure that your proposal will be accepted. You should open an issue first, describing your desired outcome and outlining your planned change. If you've found an older issue that's already open, comment on it with an outline for your planned implementation *before* putting in the work to open a pull request. **PRs must include robust testing.** Comprehensive testing within pull requests is crucial for the stability of dbt. By prioritizing robust testing, we ensure the reliability of our codebase, minimize unforeseen issues, and safeguard against potential regressions. **We cannot merge changes that risk the backward incompatibility of existing documented behaviors.** We understand that creating thorough tests often requires significant effort, and your dedication to this process greatly contributes to the project's overall reliability. Thank you for your commitment to maintaining the integrity of our codebase and the experience of everyone using dbt! **PRs go through two review steps.** First, we aim to respond with feedback on whether we think the implementation is appropriate from a product & usability standpoint. At this point, we will close PRs that we believe fall outside the scope of dbt, or which might lead to an inconsistent user experience. Many of these are great ideas for add-on functionality that is better suited for a package or plugin. This is an important part of our role as maintainers; we're always open to hearing disagreement. If a PR passes this first review, we will queue it up for code review, at which point we aim to test it ourselves and provide thorough feedback. **We receive more PRs than we can thoroughly review, test, and merge.** Our teams have finite capacity, and our top priority is maintaining a well-scoped, high-quality framework for the tens of thousands of people who use it every week. To that end, we must prioritize overall stability and planned improvements over a long tail of niche potential features. For best results, say what in particular you'd like feedback on, and explain what would it mean to you, your team, and other community members to have the proposed change merged. Smaller PRs tackling well-scoped issues tend to be easier and faster for review. Two examples of community-contributed PRs: * [(dbt-labs/dbt#9347) Fix configuration of turning test warnings into failures](https://github.com/dbt-labs/dbt/pull/9347) * [(dbt-labs/dbt#9863) Better error message when trying to select a disabled model](https://github.com/dbt-labs/dbt/pull/9863) **Automation that can help us:** Many repositories have a template for pull request descriptions, which will include a checklist that must be completed before the PR can be merged. You don't have to do all of these things to get an initial PR, but they will delay our review process. Those include: * **Tests, tests, tests.** When you open a PR, some tests and code checks will run. (For security reasons, some may need to be approved by a maintainer.) We will not merge any PRs with failing tests. If you're not sure why a test is failing, please say so, and we'll do our best to get to the bottom of it together. * **Contributor License Agreement** (CLA): This ensures that we can merge your code, without worrying about unexpected implications for the copyright or license of open source or source-available dbt software. For more details, read: ["Contributor License Agreements"](https://docs.getdbt.com/community/resources/contributor-license-agreements.md) * **Changelog:** In projects that include a number of changes in each release, we need a reliable way to signal what's been included. The mechanism for this will vary by repository, so keep an eye out for notes about how to update the changelog. * **Semver label:** Some repositories require every pull request to carry exactly one `semver:patch` (bug fixes and other safe changes) or `semver:minor` (new, non-breaking features) label; `semver:major` isn't accepted. Where this applies, CI enforces it. ##### Inclusion in release versions ###### dbt v1 Both bug fixes and backwards-compatible new features will be included in the [next minor release of dbt v1](https://docs.getdbt.com/docs/dbt-versions.md#how-dbt-v1-uses-semantic-versioning). Fixes for regressions and net-new bugs that were present in the minor version's original release will be backported to versions with [active support](https://docs.getdbt.com/docs/dbt-versions.md). Other bug fixes may be backported when we have high confidence that they're narrowly scoped and won't cause unintended side effects. ###### dbt v2 dbt v2 is released continuously, and it contains a superset of the code and functionality available in the open source layer. Version numbers for dbt OSS and dbt v2 converge at v2.0.0. The major version of our [open source](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) offering is built on the v2 runtime. Contributions to `dbt-core` targeting the v2.x series follow the same principles as v1.x: bug fixes and backwards-compatible features are included in the next available release. --- ### Getting help #### Community help dbt is powered by open source and source-available software, and has a generous community behind it. Asking questions well contributes to the community by building our collective body of knowledge. By following these steps, you'll be more likely to receive help from another community member. ##### 1. Try to solve your problem first before asking for help ###### Search the existing documentation The docs site you're on is highly searchable, make sure to explore for the answer here as a first step. If you're new to dbt, try working through the [quickstart guide](https://docs.getdbt.com/guides.md) first to get a firm foundation on the essential concepts. ###### Try to debug the issue yourself We have a handy guide on [debugging errors](https://docs.getdbt.com/guides/debug-errors.md) to help out! This guide also helps explain why errors occur, and which docs you might need to search for help. ###### Search for answers using your favorite search engine We're committed to making more errors searchable, so it's worth checking if there's a solution already out there! Further, some errors related to installing dbt, the SQL in your models, or getting YAML right, are errors that are not-specific to dbt, so there may be other resources to check. ###### Experiment! If the question you have is "What happens when I do `X`", try doing `X` and see what happens! Assuming you have a solid dev environment set up, making mistakes in development won't affect your end users ##### 2. Take a few minutes to formulate your question well Explaining the problems you are facing clearly will help others help you. ###### Include relevant details in your question Include exactly what's going wrong! When asking your question, you should: * Paste the error message or relevant code inside three backticks in your question, instead of sharing a screenshot * Include the version of dbt you're on (which you can check with `dbt --version`) * Let us know which warehouse you're using ###### Avoid generalizing your code While we understand that you may wish to generalize your problem, or that you may have sensitive information you wish to anonymize, often replacing references in SQL can result in invalid code that creates an error different to the one you're hitting. This makes it harder for us to understand your problem. Wherever possible, share the exact code that you're trying to run. ###### Let us know what you've already tried In general, people are much more willing to help when they know you've already given something your best shot! ###### Share the context of the problem you're trying to solve Sometimes you might hit a boundary of dbt because you're trying to use it in a way that doesn't align with the opinions we've built into dbt. By sharing the context of the problem you're trying to solve, we might be able to share insight into whether there's an alternative way to think about it. ###### Post a single message and use threads The dbt Slack's culture revolves around threads. When posting a message, try drafting it to yourself first to make sure you have included all the context. Include big code blocks in a thread to avoid overwhelming the channel. ###### Don't tag individuals to demand help If someone feels inclined to answer your question, they will do so. We are a community of volunteers, and we're generally pretty responsive and helpful! If nobody has replied to your question, consider if you've asked a question that helps us understand your problem. If you require in-depth, ongoing assistance, we have a wonderful group of experienced dbt consultants in our ecosystem. You can find a full list [below](#receiving-dedicated-support). ##### 3. Choose the right medium for your question We use a number of different mediums to share information * If your question is roughly "I've hit this error and am stuck", please ask it on [the dbt Community Forum](https://discourse.getdbt.com). * If you think you've found a bug, please report it on the relevant GitHub repo (e.g. [dbt repo](https://github.com/dbt-labs/dbt), [dbt-utils repo](https://github.com/dbt-labs/dbt-utils)) * If you are looking for a more wide-ranging conversation (e.g. "What's the best approach to X?", "Why is Y done this way?"), join our [Slack community](https://getdbt.com/community). Channels are consistently named with prefixes to aid discoverability. #### Receiving dedicated support If you need dedicated support to build your dbt project, consider reaching out regarding [professional services](https://www.getdbt.com/contact/), or engaging one of our [consulting partners](https://www.getdbt.com/partner-directory). #### dbt Training If you want to receive dbt training, check out our [dbt Learn](https://learn.getdbt.com/) program. #### dbt support **Note:** If you are a **dbt user** and need help with one of the following issues, please reach out to us by clicking [**Create a support ticket**](https://docs.getdbt.com/docs/dbt-support.md#create-a-support-ticket) through the dbt navigation or [emailing support@getdbt.com](mailto:support@getdbt.com): * Account setup (e.g. connection issues, repo connections) * Billing * Bug reports related to the web interface As a rule of thumb, if you are using dbt, but your problem is related to code within your dbt project, then please follow the above process or checking out the [FAQs](https://docs.getdbt.com/docs/faqs.md) rather than reaching out to support. Refer to [dbt support](https://docs.getdbt.com/docs/dbt-support.md) for more information. --- ### How to deliver a fantastic meetup talk **Speaking at a dbt meetup? Here’s all the details you’ll need to know. If you’re speaking at another event, check out our additional tips at the end of the article.** #### Understanding dbt meetups dbt meetups are an opportunity for the dbt community to learn from each other. We’re typically on the lookout for talks that last for ~15 minutes, and we reserve an additional 5-10 minutes for Q\&A after your talk. We’re not *just* looking for talks that feature dbt — if your topic feels relevant to analytics engineers, we’d love to chat. In general, you can assume that around three quarters of the audience are dbt users. When shaping your talk, consider whether there’s something in there that might be new to an experienced dbt user, and, on the other end of the scale, something that feels relevant to a data practitioner that isn’t yet a dbt user. If you feel that your talk idea requires in-depth knowledge of dbt, consider speaking on Office Hours instead. Similarly, if you’re interested in giving a more introductory talk about dbt, consider reaching out to a local data meetup to see if it’s the right fit. For topic inspiration, you can find videos of past dbt meetup presentations [here](https://www.youtube.com/playlist?list=PL0QYlrC86xQn-jxWmEqtQRbZoyjq_ffq5). If you want to present at a dbt meetup, let us know [here](https://docs.google.com/forms/d/e/1FAIpQLScU4c0UvXLsasc7uwFBrzt6YzuGiMzEH_EyFfXGnIYDmTBDfQ/viewform). If we haven’t met you before, we might book-in a call to say hi and help shape your topic! We’ll also book a meeting before the event for a dry-run of the presentation to give any additional feedback. #### Recognize when you’re ready to give a talk Below, we’ve listed four signs that you’re ready to give a talk (originally based on [this article](https://thinkgrowth.org/how-to-write-about-your-work-652441747f41) from our Head of Marketing, Janessa — read that too!). We’ve also included examples for each category — where possible these are dbt meetup talks, but some of them are also links to blog posts from members in our community. ##### You recently finished a high-impact project These are a great option for first-time speakers as they mix together both big-picture thinking and tactics. For example: * "Improving data reliability" — Andrea Kopitz ([video](https://www.youtube.com/watch?v=M_cNspn2XsE), [slides](https://docs.google.com/presentation/d/1gHChax5aM3tqKkhepX7Mghmg0DTDbY5yoDBCfUR23lg/)) * "Predicting customer conversions using dbt + machine learning" — Kenny Ning ([video](https://www.youtube.com/watch?v=BF7HH8JDUS0), [slides](https://docs.google.com/presentation/d/1iqVjzxxRggMnRoI40ku88miDKw795djpKV_v4bbLpPE/)) * "Migrating 387 models from Redshift to Snowflake" — Sam Swift and Travis Dunlop ([video](https://www.youtube.com/watch?v=VhH614WVufM), [slides](https://docs.google.com/presentation/d/1wE8NSkFPLFKGQ8fvFUUKoZFVoUhws_FhFip-9mDhoPU/)) ##### You hit an inflection point in your career Have you recently changed something about your career that you think others can learn from? Started a new job, grown in your role? These topics might not mention dbt at all, but will be relevant to many people in the audience. For example: * “Getting hired as an analytics engineer: a candidate’s perspective” — Danielle Leong ([video](https://www.youtube.com/watch?v=6VCr30ZFxZ0)) * “One analyst's guide for going from good to great” — Jason Ganz ([blog post](https://blog.getdbt.com/one-analysts-guide-for-going-from-good-to-great/)) Other ideas: * You moved from a team of many to a team of one (or vice-versa), and want to share what each can learn from the other * You started to manage others and learned some things along the way ##### You’re digging deep into a topic If you’ve spent many hours going deep on a topic, it could be a good idea to share what you’ve learned. For example: * “The farm-to-table testing framework” — Andrea Fabry ([blog post](https://blog.getdbt.com/data-testing-framework/)) * “How to create a career ladder” — Caitlin Moorman ([blog post](https://locallyoptimistic.com/post/career-ladders-part-2/)) ##### You have a strong opinion about something Is there a “best practice” that you think is outdated? Want to convince others to see your point of view? In the data-space, we’ve seen this in topics like: * “Engineers shouldn’t write ETL” — Jeff Magnusson ([blog post](https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/)) * “You probably don’t need a data dictionary” — Michael Kaminsky & Alexander Jia ([blog post](https://locallyoptimistic.com/post/data_dictionaries/)) * “Write better SQL: In defense of `group by 1`” — Claire Carroll ([blog post](https://blog.getdbt.com/write-better-sql-a-defense-of-group-by-1/)) #### Checkpoint: Is someone else well-placed to give this talk? Once you have a topic idea, stop for a moment and consider whether someone else on your team might also be a great fit for delivering this talk. Individuals from underrepresented groups are far less likely to self-nominate to give a talk — sometimes a shoulder tap is the nudge that’s needed. #### Shaping your talk Now, it’s time to write! Rather than starting with a slide deck, open up a blank document (or use [our template](https://docs.google.com/document/d/16aog0VitdLSScgxSNKe36q1C92QmG2vjXmtXYcPAhfw/edit#)), and start writing some notes. This helps you clarify your thinking, and is a great way to get feedback early, rather than investing the time into creating slides that might later be reworked. Don’t get too hung up on a title at this stage — we’re happy to work with you on that later in the process. #### The basic structure Below, we’ve outlined a common structure used for meetup talks — if this is your first talk, this is a great way to get started (in fact, even experienced speakers often use a structure like this). Use this as a starting point, rather than an exact formula! 1. What is the business problem? Relating to a business problem helps audience members understand why you undertook a project. For example: * The finance team didn’t trust our numbers * We were never sure what led to an increase in customer conversion * The data team couldn’t find a balance between ad hoc requests and roadmap work * Our tracking across mobile and web was completely inconsistent 2. How did this manifest? Include evidence that this is a genuine problem — this helps create buy-in from the audience. Slack screenshots, quotes, charts, etc. are all good here! 3. What tactics were used to solve the problem? Three feels like a good number here. Make sure to emphasize people and process solutions as well as technology solutions. 4. What was the impact on the business problem? Since you set out a problem to be solved, it’s worth revisiting it. It’s okay if you found that your project didn’t go as planned — there’s a valuable lesson in there. Again, including evidence of improvement feels valuable. 5. What other things were learned, and/or what next steps are you taking? Summarize high level lessons that others can take-away, and potentially talk about what you’d do differently, or what you plan on doing next. ##### Why does this structure work? The previous structure might seem formulaic, but we’ve seen it work a number of times. In our opinion, this structure works because: * **Your presentation has the structure of a story** — problem, journey, solution. Human beings love stories, and so the flow feels natural and easy for your audience to follow. * **It increases the target audience**. Sharing a few different tactics means that it’s more likely there will be something in your talk for different audience members. Compare that to narrowly scoping a talk on “[Writing packages when a source table may or may not exist](https://discourse.getdbt.com/t/writing-packages-when-a-source-table-may-or-may-not-exist/1487)”— it’s not going to feel relevant to most people in the room. * **It covers both theory and application.** Too much theory and you’re giving a TedTalk, too much application and you’re just giving a product demo. The best Meetup talks help people understand how you thought through a problem and why you made certain decisions so they can apply your knowledge within their unique context. #### Examples that follow this structure Here's a few of our favorite talks mapped to the structure — trust us, it works! ##### Improving data reliability — Andrea Kopitz, Envoy *[Video](https://www.youtube.com/watch?v=M_cNspn2XsE), [slides](https://docs.google.com/presentation/d/1gHChax5aM3tqKkhepX7Mghmg0DTDbY5yoDBCfUR23lg/).* 1. What is the business problem? Envoy’s financial data appeared inconsistent. 2. How did this manifest? Respondents to the team’s data survey said they no longer trusted the data. 3. What tactics were used to solve the problem? 1. Determine responsibility 2. Build more specific dbt tests 3. Track progress 4. What was the impact on the business problem? In their next data survey, satisfaction rating increased, and no mention of financial data accuracy. 5. What other things were learned, and/or what next steps are you taking? Lesson: Send out a data survey to your company to inform your roadmap. ##### Predicting customer conversions with dbt + machine learning — Kenny Ning, Better.com *[Video](https://www.youtube.com/watch?v=BF7HH8JDUS0), [slides](https://docs.google.com/presentation/d/1iqVjzxxRggMnRoI40ku88miDKw795djpKV_v4bbLpPE/).* 1. What is the business problem? No one knew why conversion rates for better.com customers would improve or worsen, making it difficult to know the value of different parts of the business. 2. How did this manifest? Different parts of the business took responsibility when it improved, no one took responsibility when it worsened. 3. What tactics were used to solve the problem? 1. Use a different approach to conversion rates — kaplan-meier conversion rates 2. Sketch out an ideal ML solution and see if it theoretically solves the problem 3. Build it! (ft. demonstration of solution) 4. What was the impact on the business problem? In the end — not as valuable as originally hoped (and that’s ok!). Editor note: [this article](https://better.engineering/2020-06-24-wizard-part-ii/) was a great follow up on the initial project. 5. What other things were learned, and/or what next steps are you taking? * Focus on end-to-end solutions * Materialize your clean dataset to improve collaboration * Sell to the business ##### Migrating 387 models from Redshift to Snowflake — Bowery Farming Data Team *[Video](https://www.youtube.com/watch?v=VhH614WVufM), [slides](https://docs.google.com/presentation/d/1wE8NSkFPLFKGQ8fvFUUKoZFVoUhws_FhFip-9mDhoPU/).* 1. What is the business problem? A new Bowery Farming site had increased the amount of data the team were dealing with, which put a strain on their data stack. 2. How did this manifest? Charts show increased dbt times, and increased Redshift costs. 3. What tactics were used to solve the problem? 1. Push Redshift to its limit: Leverage Athena, Redshift configurations, separate clusters, python pre-processing 2. Trial Snowflake for cost and performance 3. Commit to a migration with strong project management 4. What was the impact on the business problem? Yet to be determined (at the time, they had just finished the project). But the team showed evidence that the project has been successfully completed! 5. What other things were learned, and/or what next steps are you taking? Other things learned: * Differences between Redshift and Snowflake SQL syntax * Teamwork and coordination are key to completing a migration #### Turn it into a presentation Now, it's time to take your idea and turn it into a presentation. ##### Structuring your slides As well as the slides that directly support your content, consider including: * At the start: * An intro slide for yourself (and teammates) * An intro slide for your company — you might also include some impressive numbers about your business, after all, your audience is full of people who love numbers! * Potentially include your tech stack for context — there’s no need to spend too much time on this, most audience members will be familiar with the tools. * Before diving into the specific tactics used: * Use a slide to list the three tactics at a high level — this signposting helps set expectations for audience members. * At the end: * A closing slide to prompt questions, and list your contact details. * If your company is hiring, mention that too! If available, use your corporate-branded slide deck. We also have dbt-branded slides if you want to use those. ##### Making your presentation shine When turning your story into a presentation, also consider doing the following: ###### Use full sentences in your slide headings When presenting (especially virtually), it’s hard to hold everyone’s focus. That’s ok! By including full sentences as your heading, people can “hook” back into the presentation. For example, rather than having a slide on "Slide headings", use a title like “Use full sentences in your slide headings” (woah — meta!) ###### Make your slides accessible This is a [great guide](https://www.smashingmagazine.com/2018/11/inclusive-design-accessible-presentations/) on making your slides accessible — read it! ###### Use evidence in your slides Evidence is a key part of getting buy-in that the story you’re telling is valuable. Consider including: * Screenshots of slack conversations * Quotes, survey results, charts * If talking about a complex transformation, include small samples of data to demonstrate the concept. You may need to generate some fake data to simplify the problem (example) * If one of your tactics is heavily code-based, consider sharing that code in a separate piece so that interested folks can refer back to it later. (Discourse is great for this) ###### (Virtual events) Create moments for interactivity For virtual events: is there a poll you can launch, or a question you can throw out to the chat? This can help create a sense of community at the event. #### Pair it with a blog post The hardest part of nailing a great talk is the content, so if you’ve made it this far, you’ve already done most of the work. Turning your content into a blog post is a great way to solidify your thinking, and get some extra exposure. If you’d like to be featured on the [dbt Blog](https://blog.getdbt.com/), please email us at . We’ll also be adding more resources on how to write about your work soon! #### Speaking at a non-dbt event Above, we’ve given specific advice for speaking at a dbt meetup. If you’re a dbt community member who wants to speak at a non-dbt meetup or conference, there’s a few extra ways you can adjust your process. ##### Questions to ask the event organizer ###### What is the technical baseline for the audience? Do they know about dbt? If not, are they familiar with SQL? You’ll likely have a range of people in the audience so there won’t be one exact answer, but gathering information about the median knowledge is useful. As a guideline, aim to teach something new to at least half of the audience. ###### What kind of talks have been the most successful? Is the event oriented around technical talks or strategic talks? Is there an expectation of demo-ing code? Do they have past examples of talks that were well-received, or any tips? ###### What are the event logistics? How long is your talk supposed to go for? Is there an opportunity to do Q\&A? If the event is virtual, what is the software setup like? How will questions be moderated? If the event is in-person, will you be able to use your own computer, or will you use someone else’s? What sort of screen is there? How do you connect to it? And do you have the right dongle for your MacBook Pro? ###### Is there an opportunity for topic feedback? Is the organizer interested in working with you to make your topic great? If not, can they point you to someone in their community who might be interested in helping out? ###### Are there any additional accessibility considerations you should be aware of? Do any audience members use a communication device? Can you share your slides ahead of time to make them easier for audience members to access? Will the event be recorded for those who can’t attend in person? ##### Responding to a conference Call for Speakers If you’re submitting a response for a Call for Speakers, and talking about dbt, we’re happy to work with you on this. You may email us at for more information. --- ### Join the Community Want to learn how organizations around the world are tackling the biggest challenges in data while making new friends from the best analytics teams? Join the dbt Community — data practitioners’ favorite place to learn new skills, keep on top of industry trends, and forge connections. [![](/img/icons/slack.svg)](https://www.getdbt.com/community/join-the-community/) ###### [Join us on Slack](https://www.getdbt.com/community/join-the-community/) [Follow the pulse of the dbt Community! Chat with other practitioners in your city, country or worldwide about data work, tech stacks, or simply share a killer meme.](https://www.getdbt.com/community/join-the-community/) [![](/img/icons/discussions.svg)](https://discourse.getdbt.com/) ###### [Community Forum](https://discourse.getdbt.com/) [Have a question about how to do something in dbt? Hop into the Community Forum and work with others to create long lived community knowledge.](https://discourse.getdbt.com/) [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/community/contribute.md) ###### [How to contribute](https://docs.getdbt.com/community/contribute.md) [Want to get involved? This is the place! Learn how to contribute to our public repositories, write for the blog, speak at a meetup and more.](https://docs.getdbt.com/community/contribute.md) [![](/img/icons/folder.svg)](https://docs.getdbt.com/community/resources/code-of-conduct.md) ###### [Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md) [We are committed to creating a space where everyone can feel welcome and safe. Our Code of Conduct reflects the agreement that all Community members make to uphold these ideals.](https://docs.getdbt.com/community/resources/code-of-conduct.md) [![](/img/icons/calendar.svg)](https://www.getdbt.com/resources/webinars) ###### [Webinars](https://www.getdbt.com/resources/webinars) [Browse upcoming and on-demand dbt Labs webinars and workshops.](https://www.getdbt.com/resources/webinars) [![](/img/icons/globe.svg)](https://www.getdbt.com/events) ###### [Events](https://www.getdbt.com/events) [Find meetups, conferences, training, and other dbt community events online and in person.](https://www.getdbt.com/events) [![](/img/icons/star.svg)](https://www.youtube.com/playlist?list=PL0QYlrC86xQl1DGKBopQZiZ6tSqrMlD2M) ###### [Watch past events](https://www.youtube.com/playlist?list=PL0QYlrC86xQl1DGKBopQZiZ6tSqrMlD2M) [Get a taste for the energy of our live events, get inspired, or prepare for an upcoming event by watching recordings from our YouTube archives.](https://www.youtube.com/playlist?list=PL0QYlrC86xQl1DGKBopQZiZ6tSqrMlD2M) --- ### Maintaining a Slack channel #### TL;DR There are three things you should do to be a good channel maintainer in the [dbt Slack community](https://community.getdbt.com/): * Once you see some folks in the channel, post initial conversation topics to help them get to know each other. * Keep an eye out in #introductions for folks who might benefit from your new channel. For example, if someone works in the space or on the same problems, then direct them to the channel. * Make sure folks follow the [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md) in the channel. If you notice someone is not following one, gently remind them of the rule in thread, and, ideally, provide an example of how they can rephrase their message or where they can redirect it. If you have a question about how to proceed, just post it in #moderation-and-administration with a screenshot or link to the thread and someone will give you advice. #### Scope of the role A maintainer can be a dbt Labs employee but does not have to be. *Slack channel maintainer* is philosophically similar to OSS maintainer. At the onset, the channel maintainer will help build up this new space in Slack and stir up conversation during the first few weeks of the channel's existence. They are someone who stays on top of feedback and encourages generative contributions. This is not necessarily someone who is the generator of content and contributions, or answers every question. #### Initial instructions 1. Review the [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md) and [Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md) and please let the folks who created the channel know that you read both documents and you agree to be mindful of them. 2. To request a new channel, go to the [#moderation-and-administration channel](https://getdbt.slack.com/archives/C02JJ8N822H), click **Workflows** on the top of the channel description. Click **Request a New Channel**. * Fill out the fields and click **Submit** to submit your request. Someone will get in touch from there. ![request-slack-chnl](https://github.com/siljamardla/docs.getdbt.com/assets/89008547/b14abc52-4164-40a8-b48a-e8061fb4b51a) 3. If you are a vendor, review the [Vendor Expectations](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md#vendor-expectations). 4. Add the Topic and Description to the channel. @Mention your name in the channel Description, identifying yourself as the maintainer. Ex: *Maintainer: First Last (pronouns).* If you are a vendor, make sure your Handle contains your affiliation. 5. Complete or update your Slack profile by making sure your Company (in the ‘What I do’ field), Pronouns, and Handle, if you’re a vendor, are up-to-date. 6. Post initial conversation topics once a few folks get in the channel to help folks get to know each other. Check out this [example introductory post](https://getdbt.slack.com/archives/C02FXAZRRDW/p1632407767005000). 7. Stir up conversation during the first few weeks of the channel's existence. As you get started, answer the questions you can or help find someone with answers, seed discussions about once a week, and making sure folks follow the Rules of the Road. #### Long-term expectations * Maintaining the channel, checking in, and being active on a regular basis by answering folks' questions, and seeding discussions. Want an example? Check out [this poll](https://getdbt.slack.com/archives/C022A67TLFL/p1628279819038800). * For guidance on how to answer a question, see [Answering Community Questions](https://www.getdbt.com/community/answering-community-questions). If you are not sure how to answer a lingering or unanswered question, you can post about it in #moderation-and-administration or direct it to another channel, if relevant. * If the channel is an industry channel, it’s helpful to monitor [#introductions](https://getdbt.slack.com/archives/CETJLH1V3) and invite people. Keep an eye out for folks who might benefit from being in the new channel if they mention they are working in the space, or are thinking about some of these problems. * Make sure folks follow the [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md). For example, if you notice someone is not following one, gently remind them of the rule in thread, and, ideally, provide an example of how they can rephrase their message or where they can redirect it. If you have a question about how to proceed, just post about it in #moderation-and-administration with a link to the thread or screenshot and someone will give you advice. * In tools channels, sharing customer stories and product updates is very okay in this channel because folks expect that when they join. However, please avoid any direct sales campaigns, pricing offers, etc. * If you have any questions/doubts about the [Rules of the Road and Vendor Expectations](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md), please post a question in #moderation-and-administration about what sort of things the community expects from interactions with vendors. * A reminder that we never DM anyone in Slack without their permission in a public channel or some prior relationship. * A reminder that @ here/all/channel are disabled. * Use and encourage the use of threads 🧵 to keep conversations tidy! --- ### Online community building ##### Maintaining a channel in the dbt Community Slack ###### Overview The dbt Slack is the place for real time conversations with the dbt Community. Slack channels exist for specific locations, tools, industries and methodologies. In order to make sure that every channel has dedicated attention from a committed community member, we have Community Maintainers who oversee the discussion in particular channels. ###### Contribution opportunities Every channel can benefit from people who are engaged and committed to making it a more interesting place to hang out! If there's a channel that you're interested in, spend time there. For new channels that you'd like to create and maintain, post a message in the #channel-requests channel. ###### Sample contributions: * Karen Hsieh's [contributions](https://getdbt.slack.com/archives/C02TU2DSKND/p1661483529756289) to the #local-taipei channel are a fantastic example to learn from. ###### Get started * Read the guide to [Maintaining a Slack Channel](https://docs.getdbt.com/community/resources/maintaining-a-channel.md) ##### Participating on the Community Forum ###### Overview [The dbt Community Forum](https://discourse.getdbt.com) is the preferred platform for support questions as well as a space for long-lived discussions about dbt, analytics engineering, and the analytics profession. It's a place for us to build up a long-lasting knowledge base around the common challenges, opportunities, and patterns we work with every day. ###### Contribution opportunities Participate in the Forum by asking and answering questions. These discussions are what allows us to find gaps in our best practices, documentation and other recommendation, as well as to get folks onboarded and understanding dbt. Remember, it’s a mitzvah to answer a question. If you see a great question or answer, be generous with your 💜 reactions. Click the Solved button when your question is answered, so others can benefit. ###### Sample contributions * An analytics engineer wrote about [how they modified dbt to automatically put models into the correct schema](https://discourse.getdbt.com/t/extracting-schema-and-model-names-from-the-filename/575) based on their filename. * Here's [an example of the supportive, thorough answers](https://discourse.getdbt.com/t/is-it-possible-to-have-multiple-files-with-the-same-name-in-dbt/647) you can receive when you take the time to ask a question well. ###### Get started * Read the [Community Forum Guidelines](https://docs.getdbt.com/community/resources/forum-guidelines.md) to understand what topics are a good fit and why this space is important in building long-term community knowledge. * Head over to the “[Help](https://discourse.getdbt.com/c/help/19)” section of the forum and look for areas to hop in! You don’t need to know the exact answer to a question to be able to provide a helpful pointer. --- ### Open source and source-available projects Looking for a good place to get involved contributing code? dbt Labs supports the following repositories, organized by the language primarily needed for contribution: #### Rust * [dbt OSS](https://github.com/dbt-labs/dbt) - the Apache 2.0 open-source distribution of dbt v2 #### Python * [dbt v1](https://github.com/dbt-labs/dbt/tree/1.latest) - the original engine powering dbt * [hubcap](https://github.com/dbt-labs/hubcap) - the code powering the dbt Package hub #### dbt * [dbt Labs' packages](https://hub.getdbt.com/dbt-labs/) - the dbt packages created and supported by dbt Labs. Packages are just dbt projects, so if you know the SQL, Jinja, and YAML necessary to work in dbt, you can contribute to packages. #### YAML and JSON Config * [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema) - powering completion and linting for YAML configuration in dbt projects. #### Shell * [dbt-completion.bash](https://github.com/dbt-labs/dbt-completion.bash) - provides shell completion of CLI commands and selectors such as models and tests for bash and zsh. --- ### Realtime event participation ##### Speak at a Meetup ###### Overview Meetups are all about knowledge sharing; they are a place to connect and learn with your fellow dbt Community members. They usually take place in person, with some happening virtually. The Meetups take place across the globe, and you can check them all out [here](https://www.meetup.com/pro/dbt/). ###### Contribution opportunities * Become a Meetup organizer * Speak at an event * Sponsor an event or provide your office space For all of these opportunities, please fill out an [interest form](https://docs.google.com/forms/d/e/1FAIpQLScdzuz9Ouo1b07BMHveEBJsJ3rJAYuFvbTKep2fXDL0iZTZUg/viewform) and we will get back to you. ###### Sample contributions * Take a look at [the slides](https://docs.google.com/presentation/d/1iqVjzxxRggMnRoI40ku88miDKw795djpKV_v4bbLpPE/edit#slide=id.g553a984de0_0_19) and [watch the video](https://www.youtube.com/watch?v=BF7HH8JDUS0) from Kenny Ning's 2020 Meetup talk on predicting customer conversions with dbt and ML for Better.com. * Dig into [the deck](https://docs.google.com/presentation/d/1wE8NSkFPLFKGQ8fvFUUKoZFVoUhws_FhFip-9mDhoPU/edit#slide=id.p) and [the video](https://www.youtube.com/watch?v=VhH614WVufM) from Bowery Farmings talk on migrating dbt models from Redshift to Snowflake. ###### Get Started * Read [How to Deliver a Fantastic Meetup Talk](https://docs.getdbt.com/community/resources/speaking-at-a-meetup.md). * Find a [Meetup near you](https://www.meetup.com/pro/dbt/), start attending and let the organizers know you are interested! ß ##### Speak at dbt Summit ###### Overview [dbt Summit](https://www.getdbt.com/dbt-summit) is the annual analytics engineering conference hosted by dbt Labs. While Meetups are focused on sharing knowledge with a specific local hub of the Community, dbt Summit is the way to share ideas with everyone. Each year we gather together, take stock of what we’ve learned and pool our best ideas about analytics. ###### Contribution opportunities * Attend dbt Summit: * dbt Summit is the once a year gathering for analytics engineers. Whether you choose to join online or at one of our in-person events, attending dbt Summit is the best way to get an immersive experience of what the dbt Community is like. * Speak at dbt Summit! * We’d love to hear what you’ve been working on, thinking about and dreaming up in the analytics engineering space. dbt Summit talks can be forward looking views on the industry, deep dives into particular technical solutions or personal stories about your journey in data. ###### Sample contributions * [Run Your Data Team as a Product Team](https://www.getdbt.com/coalesce-2020/run-your-data-team-as-a-product-team/) * [Tailoring dbt's incremental\_strategy to Artsy's data needs](https://www.getdbt.com/coalesce-2021/tailoring-dbts-incremental-strategy-to-artsys-data-needs/) ###### Get started * If registrations are open register on the [dbt Summit website](https://www.getdbt.com/dbt-summit) * Join #dbt-summit-updates on the dbt Community Slack --- ### The dbt Viewpoint Building a Mature Analytics Workflow: The dbt Viewpoint! In 2015-2016, a team of folks at RJMetrics had the opportunity to observe, and participate in, a significant evolution of the analytics ecosystem. The seeds of dbt were conceived in this environment, and the viewpoint below was written to reflect what we had learned and how we believed the world should be different. **dbt is our attempt to address the workflow challenges we observed, and as such, this viewpoint is the most foundational statement of the dbt project's goals.** The remainder of this document is largely unedited from [the original post](https://getdbt.com/blog/building-a-mature-analytics-workflow). #### Analytics today The center of gravity in mature analytics organizations has shifted away from proprietary, end-to-end tools towards more composable solutions made up of: * data integration scripts and/or tools, * high-performance analytic databases, * SQL, R, and/or Python, and * visualization tools. This change has unlocked significant possibility, but analytics teams (ours included) have still faced challenges in consistently delivering high-quality, low-latency analytics. We believe that analytics teams have a workflow problem. Too often, analysts operate in isolation, and this creates suboptimal outcomes. Knowledge is siloed. We too often rewrite analyses that a colleague had already written. We fail to grasp the nuances of datasets that we’re less familiar with. We differ in our calculations of a shared metric. We have convinced ourselves after hundreds of conversations that these conditions are by and large the status quo for even sophisticated analytics teams. As a result, organizations suffer from reduced decision speed and reduced decision quality. Analytics doesn’t have to be this way. In fact, the playbook for solving these problems already exists — on our software engineering teams. The same techniques that software engineering teams use to collaborate on the rapid creation of quality applications can apply to analytics. We believe it’s time to build an open set of tools and processes to make that happen. #### Analytics is collaborative We believe a mature analytics team’s techniques and workflow should have the following collaboration features: ##### Version Control Analytic code — whether it’s Python, SQL, Java, or anything else — should be version controlled. Analysis changes as data and businesses evolve, and it’s important to know who changed what, when. ##### Quality Assurance Bad data can lead to bad analyses, and bad analyses can lead to bad decisions. Any code that generates data or analysis should be reviewed and tested. ##### Documentation Your analysis is a software application, and, like every other software application, people are going to have questions about how to use it. Even though it might seem simple, in reality the “Revenue” line you’re showing could mean dozens of things. Your code should come packaged with a basic description of how it should be interpreted, and your team should be able to add to that documentation as additional questions arise. ##### Modularity If you build a series of analyses about your company’s revenue, and your colleague does as well, you should use the same input data. Copy-paste is not a good approach here — if the definition of the underlying set changes, it will need to be updated everywhere it was used. Instead, think of the schema of a data set as its public interface. Create tables, views, or other data sets that expose a consistent schema and can be modified if business logic changes. #### Analytic code is an asset The code, processes, and tooling required to produce that analysis are core organizational investments. We believe a mature analytics organization’s workflow should have the following characteristics so as to protect and grow that investment: ##### Environments Analytics requires multiple environments. Analysts need the freedom to work without impacting users, while users need service level guarantees so that they can trust the data they rely on to do their jobs. ##### Service level guarantees Analytics teams should stand behind the accuracy of all analysis that has been promoted to production. Errors should be treated with the same level of urgency as bugs in a production product. Any code being retired from production should go through a deprecation process. ##### Design for maintainability Most of the cost involved in software development is in the maintenance phase. Because of this, software engineers write code with an eye towards maintainability. Analytic code, however, is often fragile. Changes in underlying data break most analytic code in ways that are hard to predict and to fix. Analytic code should be written with an eye towards maintainability. Future changes to the schema and data should be anticipated and code should be written to minimize the corresponding impact. #### Analytics workflows require automated tools Frequently, much of an analytic workflow is manual. Piping data from source to destination, from stage to stage, can eat up a majority of an analyst’s time. Software engineers build extensive tooling to support the manual portions of their jobs. In order to implement the analytics workflows we are suggesting, similar tools will be required. Here’s one example of an automated workflow: * models and analysis are downloaded from multiple source control repositories, * code is configured for the given environment, * code is tested, and * code is deployed. Workflows like this should be built to execute with a single command. --- ### Tips for organizing inclusive events The dbt community is filled with dedicated community leaders who create opportunities for connection, learning and professional development within the analytics community. This guide is a resource to help organizers execute **inclusive digital events**. We understand that organizers, presenters, speakers, etc. might not be able to apply these tips to *every* event, but this guide will offer some food for thought. Additionally, this list can grow. If you would like to contribute a tip, please email . #### General logistics * Try to choose a date that does not overlap with [holidays](http://www.holidayscalendar.com/months/) or general major events. Don’t forget to check international holidays (if applicable) * Avoid really large national/local events (i.e. World Cup) #### Marketing * If you are using photos, share images that include community members with a wide range of presentations, including people from underrepresented groups. * Put event accessibility information on your event page (i.e. “closed captioning available for all video resources”) * In the registration process provide an opportunity for attendees to: * share pronouns * ask questions in advance * request specific needs or other accommodations (interpreting services, braille transcription, dietary restrictions, etc.) * If this is a paid event (e.g. a conference), create a scholarship for attendees that might need financial support * Think about how you are promoting your event — are you reaching underrepresented communities, marginalized populations and people who might not have access to the internet? #### Programming * Book diverse speakers. Include speakers that represent underrepresented and marginalized populations. * Do research on your speakers. Is there any reason that your speakers would make the audience uncomfortable? * Design an [accessible presentation](https://www.smashingmagazine.com/2018/11/inclusive-design-accessible-presentations/) * If possible, share a recording after the event for community members who are not able to make it and add closed captioning. * Ask speakers to introduce themselves before starting their presentation, so that transcription services can capture who is talking. #### Digital platforms for online events * Take a minute or two to explain the features of the platform that attendees will be using in the beginning of the event * Offer the option for attendees to dial-in by phone and participate without a computer or internet * Explore the accessibility features your platform offers and apply it where necessary (i.e. closed captioning, automatic transcripts, screen reader support, etc.) * Check if your platform is compatible with assistive technology #### Attendee communication * Make sure that attendees have any addresses, links, codes, numbers to accessing the event beforehand * Share the agenda of the event beforehand so that attendees are able to make arrangements (if necessary) * Share contact information with attendees so that they are able to reach out with questions before and after the event. * Ask attendees for feedback in a post-event survey so that you are able to improve future experiences. #### Speaker communication * Ask speakers how to pronounce their names before the event * Ask speakers for their pronouns before the event * Suggest that speakers use headphones to ensure clear audio * Ask speakers to use plain language and avoid jargon, slang, idioms, etc. --- ### Writing contributions ##### Contribute to the product documentation ###### Overview The [dbt Product Documentation](https://docs.getdbt.com/docs/introduction.md) sits at the heart of how people learn to use and engage with dbt. From explaining dbt to newcomers to providing references for advanced functionality and APIs, the product docs are a frequent resource for *every* dbt Developer. ###### Contribution opportunities We strive to create pathways that inspire you to learn more about dbt and enable you to continuously improve the way you solve data problems. We always appreciate the vigilance of the Community helping us to accurately represent the functionality and capabilities of dbt. You can participate by: * [Opening an issue](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose) when you see something that can be fixed, whether it’s large or small. * Creating a PR when you see something you want to fix, or to address an existing issue. You can do this by clicking **Edit this page** at the bottom of any page on [docs.getdbt.com](http://docs.getdbt.com). ###### Sample contributions We appreciate these contributions because they contain context in the original post (OP) that helps us understand their relevance. They also add value to the docs, even in small ways! * Larger contribution: * Smaller contribution: ###### Get started * You can contribute to [docs.getdbt.com](http://docs.getdbt.com) by looking at our repository’s [README](https://github.com/dbt-labs/docs.getdbt.com#readme) or clicking **Edit this page** at the bottom of most pages at docs.getdbt.com. * Read the [Contributor Expectations](https://docs.getdbt.com/community/resources/contributor-expectations.md). * Find an issue labeled “[good first issue](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22+).” * Need help: Visit #community-writers on the Community Slack or mention `@dbt-labs/product-docs` in a pull request or issue comment. ##### Write a Developer Blog Post ###### Overview The [dbt Developer Blog](https://docs.getdbt.com/blog) is the place for analytics practitioners to talk about *what it’s like to do data work right now.* This is the place to share tips and tricks, hard won knowledge and stories from the trenches with the dbt Community. ###### Contribution opportunities We want to hear your stories! Did you recently solve a cool problem, discover an interesting bug or lead an organizational change? Come tell the story on the dbt Developer Blog. ###### Sample contributions * [Founding an Analytics Engineering Team From Scratch](https://docs.getdbt.com/blog/founding-an-analytics-engineering-team-smartsheet#our-own-take-on-data-mesh) * [Tackling the Complexity of Joining Snapshots](https://docs.getdbt.com/blog/joining-snapshot-complexity) ###### Get started * [Read the contribution guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/developer-blog.md) * [Open up an issue with your idea for a post](https://github.com/dbt-labs/docs.getdbt.com/issues/new?assignees=\&labels=content%2Cdeveloper+blog\&template=contribute-to-developer-blog.yml) ###### Need help? Visit #community-writers in the dbt Community Slack --- ## Docs ### _wizard-cli-full-generated #### How to read this reference * **Flag** — the full flag as typed. Short flags (e.g. `-m`) are listed in the Short column. * **Type** — `boolean` (presence/absence), `string`, `path`, or an enumeration of allowed values. * **Description** — what the flag does and when to use it. * Global flags apply to the base interactive `wizard` command. Subcommands can have different flag sets; use each command's section for automation. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Global flags These flags work on the base interactive `wizard` command. | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config ` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable ` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable ` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote ` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env ` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image ...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model ` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider ` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile ` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 ` | — | path | Layer $DBT\_WIZARD\_HOME/\.config.toml on top of the base user config | | `-s, --sandbox ` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd ` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir ` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval ` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### Commands | Command | Aliases | Description | | ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------- | | [`exec`](#exec) | `e` | Run wizard non-interactively | | [`review`](#review) | — | Run a code review non-interactively | | [`login`](#login) | — | Manage login | | [`logout`](#logout) | — | Remove stored authentication credentials | | [`mcp`](#mcp) | — | Manage external MCP servers for wizard | | [`plugin`](#plugin) | — | Manage wizard plugins | | [`providers`](#providers) | — | Manage model providers | | [`mcp-server`](#mcp-server) | — | Start wizard as an MCP server (stdio) | | [`app-server`](#app-server) | — | \[experimental] Run the app server or related tooling | | [`remote-control`](#remote-control) | — | \[experimental] Manage the app-server daemon with remote control enabled | | [`completion`](#completion) | — | Generate shell completion scripts | | [`update`](#update) | — | Update wizard to the latest version | | [`doctor`](#doctor) | — | Diagnose local wizard installation, config, auth, and runtime health | | [`sandbox`](#sandbox) | — | Run commands within a wizard-provided sandbox | | [`debug`](#debug) | — | Debugging tools | | [`apply`](#apply) | `a` | Apply the latest diff produced by wizard agent as a `git apply` to your local working tree | | [`resume`](#resume) | — | Resume a previous interactive session (picker by default; use --last to continue the most recent) | | [`fork`](#fork) | — | Fork a previous interactive session (picker by default; use --last to fork the most recent) | | [`cloud`](#cloud) | — | \[EXPERIMENTAL] Browse tasks from wizard in dbt platform and apply changes locally | | [`exec-server`](#exec-server) | — | \[EXPERIMENTAL] Run the standalone exec-server service | | [`features`](#features) | — | Inspect feature flags | #### exec Run wizard non-interactively ```bash wizard exec [OPTIONS] [ARGS] ``` **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `[PROMPT]` | Initial instructions for the agent. If not provided as an argument (or if `-` is used), instructions are read from stdin. If stdin is piped and a prompt is also provided, stdin is appended as a `<stdin>` block | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config ` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable ` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable ` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image ...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model ` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider ` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile ` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 ` | — | path | Layer $DBT\_WIZARD\_HOME/\.config.toml on top of the base user config | | `-s, --sandbox ` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd ` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir ` | — | path | Additional directories that should be writable alongside the primary workspace | | `--skip-git-repo-check` | — | boolean | Allow running wizard outside a Git repository | | `--ephemeral` | — | boolean | Run without persisting session files to disk | | `--ignore-user-config` | — | boolean | Do not load `$DBT_WIZARD_HOME/config.toml`; auth still uses `DBT_WIZARD_HOME` | | `--ignore-rules` | — | boolean | Do not load user or project execpolicy `.rules` files | | `--output-schema ` | — | path | Path to a JSON Schema file describing the model's final response shape | | `--color ` | — | enum | Specifies color settings for use in the output | | `--json` | — | boolean | Print events to stdout as JSONL | | `--include-subagent-events` | — | boolean | When `--json` is set, capture and emit sub-agent events tagged with their thread ID. Each sub-agent event gets a `sub_agent_thread_id` field added to the JSONL line so consumers can distinguish it from the primary agent's events | | `--no-validation` | — | boolean | When set, the validation gate always responds with "declined" so the validation sub-agent is never spawned. Useful for benchmarking runs where sub-agent overhead is not desired | | `-o, --output-last-message ` | `-o` | path | Specifies file where the last message from the agent should be written | #### review Run a code review non-interactively **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------- | | `[PROMPT]` | Custom review instructions. If `-` is used, read from stdin | | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config ` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable ` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--uncommitted` | — | boolean | Review staged, unstaged, and untracked changes | | `--base ` | — | string | Review changes against the given base branch | | `--disable ` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--commit ` | — | string | Review the changes introduced by a commit | | `--title ` | — | string | Optional commit title to display in the review summary | #### login Manage login | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### logout Remove stored authentication credentials | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp Manage external MCP servers for dbt Wizard | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### plugin Manage dbt Wizard plugins | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### providers Manage model providers | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp-server Start dbt Wizard as an MCP server (stdio) | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### app-server \[experimental] Run the app server or related tooling | Flag | Short | Type | Description | | --------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `stdio://` (default), `unix://`, `unix://PATH`, `ws://IP:PORT`, `off` | | `--analytics-default-enabled` | — | boolean | Controls whether analytics are enabled by default. | | `--ws-auth <MODE>` | — | enum | Websocket auth mode for non-loopback listeners | | `--ws-token-file <PATH>` | — | path | Absolute path to the capability-token file | | `--ws-token-sha256 <HEX>` | — | string | Hex-encoded SHA-256 digest of the capability token | | `--ws-shared-secret-file <PATH>` | — | path | Absolute path to the shared secret file for signed JWT bearer tokens | | `--ws-issuer <ISSUER>` | — | string | Expected issuer for signed JWT bearer tokens | | `--ws-audience <AUDIENCE>` | — | string | Expected audience for signed JWT bearer tokens | | `--ws-max-clock-skew-seconds <SECONDS>` | — | string | Maximum clock skew when validating signed JWT bearer tokens | #### remote-control \[experimental] Manage the app-server daemon with remote control enabled | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit machine-readable JSON | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### completion Generate shell completion scripts **Arguments:** | Argument | Description | | --------- | --------------------------------- | | `[SHELL]` | Shell to generate completions for | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### update Update dbt Wizard to the latest version | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### doctor Diagnose local dbt Wizard installation, config, auth, and runtime health | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit a redacted machine-readable report | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--summary` | — | boolean | Only show grouped check rows and the final count summary | | `--all` | — | boolean | Expand long lists in detailed human output | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--no-color` | — | boolean | Disable ANSI color in human output | | `--ascii` | — | boolean | Use ASCII status labels and separators in human output | #### sandbox Run commands within a dbt Wizard-provided sandbox | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### debug Debugging tools | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### apply Apply the latest diff produced by dbt Wizard agent as a `git apply` to your local working tree **Arguments:** | Argument | Description | | ----------- | ----------- | | `<TASK_ID>` | | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### resume Resume a previous interactive session (picker by default; use --last to continue the most recent) **Arguments:** | Argument | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID) or thread name. UUIDs take precedence if it parses. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Continue the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--include-non-interactive` | — | boolean | Include non-interactive sessions in the resume picker and --last selection | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### fork Fork a previous interactive session (picker by default; use --last to fork the most recent) **Arguments:** | Argument | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID). When provided, forks this session. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Fork the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### cloud \[EXPERIMENTAL] Browse tasks from dbt Wizard in dbt platform and apply changes locally | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### exec-server \[EXPERIMENTAL] Run the standalone exec-server service | Flag | Short | Type | Description | | --------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `ws://IP:PORT` (default), `stdio`, `stdio://` | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--remote <URL>` | — | string | Register this exec-server as a remote environment using the given base URL | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--environment-id <ID>` | — | string | Environment id to attach to when registering remotely | | `--name <NAME>` | — | string | Human-readable environment name | | `--use-agent-identity-auth` | — | boolean | Use Agent Identity auth from DBT\_WIZARD\_ACCESS\_TOKEN for remote registration | #### features Inspect feature flags | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | --- ### _wizard-slash-commands-generated #### Slash commands Type `/` in the composer to open the command picker. Use arrow keys to navigate or keep typing to filter. Press **Tab** to queue a command while a task is running — it executes at the end of the current turn. ##### All slash commands | Command | Alias | Purpose | Available during task | | --------------------------------------------------- | -------- | ---------------------------------------------------------------------- | --------------------- | | [`/model`](#model-and-ai) | — | Choose what model and reasoning effort to use | ✗ | | [`/providers`](#model-and-ai) | — | Configure model providers and BYOK credentials | ✓ | | [`/ide`](#review-and-context) | — | Include current selection, open files, and other context from your IDE | ✓ | | [`/permissions`](#permissions-and-safety) | — | Choose what Wizard is allowed to do | ✗ | | [`/keymap`](#customization) | — | Remap TUI shortcuts | ✗ | | [`/vim`](#customization) | — | Toggle Vim mode for the composer | ✗ | | [`/setup-default-sandbox`](#permissions-and-safety) | — | Set up elevated agent sandbox | ✗ | | [`/sandbox-add-read-dir`](#permissions-and-safety) | — | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | ✗ | | [`/experimental`](#customization) | — | Toggle experimental features | ✗ | | [`/approve`](#customization) | — | Approve one retry of a recent auto-review denial | ✓ | | [`/memories`](#memory) | — | Configure memory use and generation | ✗ | | [`/skills`](#skills-and-extensions) | — | Use skills to improve how Wizard performs specific tasks | ✓ | | [`/hooks`](#skills-and-extensions) | — | View and manage lifecycle hooks | ✓ | | [`/review`](#review-and-context) | — | Review my current changes and find issues | ✗ | | [`/rename`](#session-management) | — | Rename the current thread | ✓ | | [`/new`](#session-management) | — | Start a new chat during a conversation | ✗ | | [`/resume`](#session-management) | — | Resume a saved chat | ✗ | | [`/fork`](#session-management) | — | Fork the current chat | ✗ | | [`/init`](#session-management) | — | Create an AGENTS.md file with instructions for Wizard | ✗ | | [`/compact`](#session-management) | — | Summarize conversation to prevent hitting the context limit | ✗ | | [`/plan`](#long-running-tasks) | — | Switch to Plan mode | ✗ | | [`/goal`](#long-running-tasks) | — | Set or view the goal for a long-running task | ✓ | | [`/agent`](#long-running-tasks) | — | Switch the active agent thread | ✓ | | [`/side`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/btw`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/copy`](#review-and-context) | — | Copy last response as markdown | ✓ | | [`/raw`](#review-and-context) | — | Toggle raw scrollback mode for copy-friendly terminal selection | ✓ | | [`/diff`](#review-and-context) | — | Show git diff (including untracked files) | ✓ | | [`/mention`](#review-and-context) | — | Mention a file | ✓ | | [`/status`](#session-info) | — | Show current session configuration and token usage | ✓ | | [`/config`](#session-info) | — | View and manage configuration | ✓ | | [`/debug-config`](#session-info) | — | Show config layers and requirement sources for debugging | ✓ | | [`/title`](#customization) | — | Configure which items appear in the terminal title | ✓ | | [`/statusline`](#customization) | — | Configure which items appear in the status line | ✓ | | [`/theme`](#customization) | — | Choose a syntax highlighting theme | ✗ | | [`/pets`](#customization) | `/pet` | Choose or hide the terminal pet | ✗ | | [`/overview`](#review-and-context) | — | Render the dbt project overview card | ✓ | | [`/mcp`](#skills-and-extensions) | — | List configured MCP tools; use /mcp verbose for details | ✓ | | [`/apps`](#skills-and-extensions) | — | Manage apps | ✓ | | [`/plugins`](#skills-and-extensions) | — | Browse plugins | ✓ | | [`/quit`](#exit) | — | Exit Wizard | ✓ | | [`/exit`](#exit) | — | Exit Wizard | ✓ | | [`/feedback`](#session-info) | — | Send feedback to maintainers | ✓ | | [`/ps`](#background-terminals) | — | List background terminals | ✓ | | [`/stop`](#background-terminals) | `/clean` | Stop all background terminals | ✓ | | [`/clear`](#session-management) | — | Clear the terminal and start a new chat | ✗ | | [`/personality`](#model-and-ai) | — | Choose a communication style for Wizard | ✗ | | [`/realtime`](#realtime-experimental) | — | Toggle realtime voice mode (experimental) | ✓ | | [`/settings`](#realtime-experimental) | — | Configure realtime microphone/speaker | ✓ | | [`/subagents`](#long-running-tasks) | — | Switch the active agent thread | ✓ | *** ##### Model and AI Control the AI model, provider, speed, and response style. | Command | Inline args | Description | | -------------- | ----------- | ---------------------------------------------- | | `/model` | — | Choose what model and reasoning effort to use | | `/providers` | — | Configure model providers and BYOK credentials | | `/personality` | — | Choose a communication style for Wizard | *** ##### Session management Start, resume, branch, and clean up sessions. | Command | Inline args | Description | | ---------- | ----------- | ----------------------------------------------------------- | | `/new` | — | Start a new chat during a conversation | | `/clear` | — | Clear the terminal and start a new chat | | `/resume` | Yes | Resume a saved chat | | `/fork` | — | Fork the current chat | | `/compact` | — | Summarize conversation to prevent hitting the context limit | | `/rename` | Yes | Rename the current thread | | `/init` | — | Create an AGENTS.md file with instructions for Wizard | *** ##### Review and context Pull information into the session or trigger a code review. | Command | Inline args | Description | | ----------- | ----------- | ---------------------------------------------------------------------- | | `/review` | Yes | Review my current changes and find issues | | `/diff` | — | Show git diff (including untracked files) | | `/mention` | — | Mention a file | | `/copy` | — | Copy last response as markdown | | `/raw` | Yes | Toggle raw scrollback mode for copy-friendly terminal selection | | `/overview` | — | Render the dbt project overview card | | `/ide` | Yes | Include current selection, open files, and other context from your IDE | *** ##### Permissions and safety Control what Wizard is allowed to execute. | Command | Inline args | Description | | ------------------------ | ----------- | --------------------------------------------------------------------- | | `/permissions` | — | Choose what Wizard is allowed to do | | `/setup-default-sandbox` | — | Set up elevated agent sandbox | | `/sandbox-add-read-dir` | Yes | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | *** ##### Customization Appearance, keybindings, and UI preferences. | Command | Inline args | Description | | --------------------- | ----------- | -------------------------------------------------- | | `/theme` | — | Choose a syntax highlighting theme | | `/keymap` | Yes | Remap TUI shortcuts | | `/vim` | — | Toggle Vim mode for the composer | | `/statusline` | — | Configure which items appear in the status line | | `/title` | — | Configure which items appear in the terminal title | | `/experimental` | — | Toggle experimental features | | `/approve` | — | Approve one retry of a recent auto-review denial | | `/pets` (alias: /pet) | Yes | Choose or hide the terminal pet | *** ##### Skills and extensions Manage capabilities Wizard can use during a session. | Command | Inline args | Description | | ---------- | ----------- | -------------------------------------------------------- | | `/skills` | — | Use skills to improve how Wizard performs specific tasks | | `/hooks` | — | View and manage lifecycle hooks | | `/mcp` | Yes | List configured MCP tools; use /mcp verbose for details | | `/apps` | — | Manage apps | | `/plugins` | — | Browse plugins | *** ##### Long-running tasks Manage multi-turn goals, parallel agents, and branched conversations. | Command | Inline args | Description | | ------------ | ----------- | ---------------------------------------------- | | `/plan` | Yes | Switch to Plan mode | | `/goal` | Yes | Set or view the goal for a long-running task | | `/agent` | — | Switch the active agent thread | | `/subagents` | — | Switch the active agent thread | | `/side` | Yes | Start a side conversation in an ephemeral fork | | `/btw` | Yes | Start a side conversation in an ephemeral fork | *** ##### Background terminals Inspect and control shell processes Wizard has running in the background. | Command | Inline args | Description | | ----------------------- | ----------- | ----------------------------- | | `/ps` | — | List background terminals | | `/stop` (alias: /clean) | — | Stop all background terminals | *** ##### Memory Control how Wizard stores and uses memory across sessions. | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------- | | `/memories` | — | Configure memory use and generation | *** ##### Session info Inspect the current session state without changing anything. | Command | Inline args | Description | | --------------- | ----------- | -------------------------------------------------------- | | `/status` | — | Show current session configuration and token usage | | `/config` | — | View and manage configuration | | `/debug-config` | — | Show config layers and requirement sources for debugging | | `/feedback` | — | Send feedback to maintainers | *** ##### Exit | Command | Inline args | Description | | ------- | ----------- | ----------- | | `/quit` | — | Exit Wizard | | `/exit` | — | Exit Wizard | *** ##### Realtime (experimental) | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------------- | | `/realtime` | — | Toggle realtime voice mode (experimental) | | `/settings` | — | Configure realtime microphone/speaker | *** --- ### 2022 dbt platform release notes dbt platform | Free Archived release notes for dbt from 2022 #### December 2022 ##### Threads default value changed to 4 Threads help parallelize node execution in the dbt directed acyclic graph [(DAG)](https://docs.getdbt.com/terms/dag). Previously, the thread value defaulted to 1, which can increase the runtime of your project. To help reduce the runtime of your project, the default value for threads in user profiles is now set to 4 threads. You can supply a custom thread count if you'd prefer more or less parallelization. For more information, read [Understanding threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). ##### Creating a new job no longer triggers a run by default To help save compute time, new jobs will no longer be triggered to run by default. When you create a new job in dbt, you can trigger the job to run by selecting **Run on schedule** and completing the desired schedule and timing information. For more information, refer to [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md). ##### Private packages must be cloned using access tokens provided by environment variables The supported method for cloning private GitHub packages is the [git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method), where an appropriate access token is passed into the package repository URL with an environment variable. A small number of people have been able to clone private packages using dbt's native GitHub application without explicitly providing an access token. This functionality is being deprecated as it’s limited in flexibility. If you have been using a package hosted in a private repository on GitHub, you must start passing an access token into the URL. An example of passing an access token: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" ``` #### November 2022 ##### The dbt Cloud + Databricks experience is getting even better dbt is the easiest and most reliable way to develop and deploy a dbt project. It helps remove complexity while also giving you more features and better performance. A simpler Databricks connection experience with support for Databricks’ Unity Catalog and better modeling defaults is now available for your use. For all the Databricks customers already using dbt with the dbt-spark adapter, you can now [migrate](https://docs.getdbt.com/guides/migrate-from-spark-to-databricks.md) your connection to the [dbt-databricks adapter](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) to get the benefits. [Databricks](https://www.databricks.com/blog/2022/11/17/introducing-native-high-performance-integration-dbt-cloud.html) is committed to maintaining and improving the adapter, so this integrated experience will continue to provide the best of dbt and Databricks. Check out our [live blog post](https://www.getdbt.com/blog/dbt-cloud-databricks-experience/) to learn more. ##### Extra features in new and refreshed IDE The refreshed version of the Studio IDE has launched four brand-new additional features, making it easier and faster for you to develop in the Studio IDE. The new features are: * **Formatting** — Format your dbt SQL files to a single code style with a click of a button. This uses the tool [sqlfmt](https://github.com/tconbeer/sqlfmt). * **Git diff view** — Highlights the changes in a file before opening a pull request. * **dbt autocomplete** — There are four new types of autocomplete features to help you develop faster: * Use `ref` to autocomplete your model names * Use `source` to autocomplete your source name + table name * Use `macro` to autocomplete your arguments * Use `env var` to autocomplete env var * **Dark mode** — Use dark mode in the Studio IDE for low-light environments. Read more about all the [Cloud Studio IDE features](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features). ##### Classic IDE deprecation notice In December 2022, dbt Labs will deprecate the classic Studio IDE. The [new and refreshed Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) will be available for *all* dbt users. You will no longer be able to access the classic Studio IDE and dbt Labs might introduce changes that break the classic Studio IDE. With deprecation, dbt Labs will only support the refreshed version of the Studio IDE. Virtual Private Cloud (VPC) customers with questions about when this change will affect your account can contact your account team or support contact for assistance. #### October 2022 ##### Announcing dbt Cloud’s native integration with Azure DevOps dbt now offers a native integration with Azure DevOps for dbt customers on the enterprise plan. We built this integration to remove friction, increase security, and unlock net new product experiences for our customers. [Setting up the Azure DevOps integration](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) in dbt provides: * easy dbt project set up, * an improved security posture, * repo permissions enforcement in Studio IDE, and * dbt Slim CI. Check out our [live blog post](https://www.getdbt.com/blog/dbt-cloud-integration-azure-devops/) to learn more! ##### Introducing a snappier, improved, and powerful Cloud IDE The new version of the Cloud Studio IDE makes it easy for you to build data models without thinking much about environment setup and configuration. The new Cloud Studio IDE includes performance upgrades, ergonomics improvements, and some delightful enhancements! Some of the improvements include: * Improved Cloud Studio IDE startup time (starting the Studio IDE), interaction time (saving and committing), and reliability. * Better organization and navigation with features like drag and drop of files, breadcrumb, build button drop-down, and more. * You can use new features like auto-format your file, auto-complete model names, and git diff view to see your changes before making a pull request. Read more about the new [Cloud Studio IDE features](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features) and check out [New and improved Cloud Studio IDE](https://www.getdbt.com/blog/new-improved-cloud-ide/) blog for more info! #### September 2022 ##### List Steps API endpoint deprecation warning On October 14th, 2022 dbt Labs is deprecating the List Steps API endpoint. From October 14th, any GET requests to this endpoint will fail. Please prepare to stop using the List Steps endpoint as soon as possible. dbt Labs will continue to maintain the [Retrieve Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retrieve%20Run) endpoint, which is a viable alternative depending on the use case. You can fetch run steps for an individual run with a GET request to the following URL, replacing `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/api/v2/accounts/{accountId}/runs/{runId}/?include_related=["run_steps"]` ##### Query the previous three months of data using the metadata API In order to make the metadata API more scalable and improve its latency, we’ve implemented data retention limits. The metadata API can now query data from the previous three months. For example, if today was March 1, you could query data back to January 1st. For more information, see [Metadata API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) #### August 2022 ##### Support for cross-database sources on Redshift RA3 instances Cross-database queries for RA3 instances are now supported by dbt projects using a Redshift connection. With cross-database queries, you can seamlessly query data from any database in the cluster, regardless of which database you are connected to with dbt. The [connection configuration](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) `ra3_node` has been defaulted to `true`. This allows users to: * benefit from the full RA3 nodes’ capabilities, * generate appropriate dbt documentation. #### July 2022 ##### Large DAG feature You can now select **Render Lineage** to visualize large DAGs. Large DAGs can take a long time (10 or more seconds, if not minutes) to render and can cause browsers to crash. The new button prevents large DAGs from rendering automatically. Instead, you can select **Render Lineage** to load the visualization. This should affect about 15% of the DAGs. [![Render Lineage](</img/docs/dbt-platform/dag v1.1.56 release.png?v=2> "Render Lineage")](#)Render Lineage #### May 2022 ##### Refresh expired access tokens in the IDE when using GitLab On May 22, GitLab changed how they treat [OAuth access tokens that don't expire](https://docs.gitlab.com/ee/update/deprecations.html#oauth-tokens-without-expiration). We updated our Studio IDE logic to handle OAuth token expiration more gracefully. Now, the first time your token expires after 2 hours of consecutive Studio IDE usage, you will have to re-authenticate in GitLab to refresh your expired OAuth access token. We will handle subsequent refreshes for you if you provide the authorization when you re-authenticate. This additional security layer in the Studio IDE is available only to the dbt enterprise plan. #### April 2022 ##### Audit log To review actions performed by people in your organization, dbt provides logs of audited user and system events. The dbt audit log lists events triggered in your organization within the last 90 days. The audit log includes details such as who performed the action, what the action was, and when it was performed. For more details, review [the audit log for dbt Enterprise](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md) documentation. ##### Credentials no longer accidentally wiped when editing an environment We resolved a bug where when updating unencrypted fields (for example, threads, schema name) in an environment setting would cause secret fields (for example, password, keypair, credential details) to be deleted from that environment. Now users can freely update environment settings without fear of unintentionally wiping credentials. ##### Email verification To enhance the security of user creation, dbt users created using SAML Just-in-Time (JIT) will now confirm identity via email to activate their account. Using email to confirm identity ensures the user still has access to the same email address they use to login via SAML. ##### Scheduler performance improvements We rolled out our new distributed scheduler, which has much faster prep times, especially at the top of the hour. We share more about our work and improvements in our [product news blog post](https://www.getdbt.com/blog/a-good-problem-to-have/). #### March 2022 ##### Spotty internet issues no longer cause a session time out message We fixed an issue where a spotty internet connection could cause the “Studio IDE session timed out” message to appear unexpectedly. People using a VPN were most likely to see this issue. We updated the health check logic so it now excludes client-side connectivity issues from the Studio IDE session check. If you lose your internet connection, we no longer update the health-check state. Now, losing internet connectivity will no longer cause this unexpected message. [![Fix Session Timeout](</img/docs/dbt-platform/Fix Session Timeout.png?v=2> "Fix Session Timeout")](#)Fix Session Timeout ##### Dividing queue time into waiting and prep time dbt now shows "waiting time" and "prep time" for a run, which used to be expressed in aggregate as "queue time". Waiting time captures the time dbt waits to run your job if there isn't an available run slot or if a previous run of the same job is still running. Prep time represents the time it takes dbt to ready your job to run in your cloud data warehouse. [![New prep time and waiting time](/img/docs/dbt-platform/v1.1.46releasenotes_img1.png?v=2 "New prep time and waiting time")](#)New prep time and waiting time #### February 2022 ##### DAG updates and performance improvements Love the DAG in the Studio IDE as much as we do? Now when you click on a node in the DAG, the model or config file will open as a new tab in the Studio IDE, so you can directly view or edit the code. We'll continue to ship better developer ergonomic functionality throughout the year. ###### Performance improvements and enhancements * Updated recommended dbt commands in the Studio IDE to include dbt v1 commands, such as "build" and the "--select" argument. ##### Service tokens and bug fixes Service tokens can now be assigned granular permissions to enforce least privilege access. If you're on Enterprise, you can assign any enterprise permission set to newly issued service tokens. If you're on Teams, you can assign the Job Admin permission set to newly issued service tokens. We highly recommend you re-issue service tokens with these new permissions to increase your security posture! See docs [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). ###### New products and features * We are joining the [GitHub secret scanning partner program](https://docs.github.com/en/developers/overview/secret-scanning-partner-program) to better secure your token against accidental public exposure and potential fraudulent usage. ###### Bug fixes * Credentials are no longer accidentally deleted when a user updates an environment setting. #### January 2022 ##### Autocomplete snippets for SQL and YAML files in IDE Some noteworthy improvements include autocomplete snippets for SQL and YAML files in the IDE, which are available for use now! We also added a [new metric layer page](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) to docs.getdbt.com to help you begin thinking about the metrics layer in dbt Cloud. ###### Performance improvements and enhancements * Branch names now default to "main" instead of "master" in new managed and unmanaged Git repositories. * Update IDE autocomplete snippets. ##### Model timing for Multi-tenant Team and Enterprise accounts We started the new year with a gift! Multi-tenant Team and Enterprise accounts can now use the new [Model timing](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#model-timing) tab in dbt. You can use this tab to further explore long-running models to see if they need refactoring or rescheduling. ###### Performance improvements and enhancements * We added client-side naming validation for file or folder creation. --- ### 2023 dbt platform release notes dbt platform | Free Archived release notes for dbt from 2023 #### December 2023 *  Semantic Layer updates The dbt Labs team continues to work on adding new features, fixing bugs, and increasing reliability for the dbt Semantic Layer. The following list explains the updates and fixes for December 2023 in more detail. #### Bug fixes * Tableau integration — The dbt Semantic Layer integration with Tableau now supports queries that resolve to a "NOT IN" clause. This applies to using "exclude" in the filtering user interface. Previously it wasn’t supported. * `BIGINT` support — The dbt Semantic Layer can now support `BIGINT` values with precision greater than 18. Previously it would return an error. * Memory leak — Fixed a memory leak in the JDBC API that would previously lead to intermittent errors when querying it. * Data conversion support — Added support for converting various Redshift and Postgres-specific data types. Previously, the driver would throw an error when encountering columns with those types. #### Improvements * Deprecation — We deprecated dbt Metrics and the legacy dbt Semantic Layer, both supported on dbt version 1.5 or lower. This change came into effect on December 15th, 2023. * Improved dbt converter tool — The [dbt converter tool](https://github.com/dbt-labs/dbt-converter) can now help automate some of the work in converting from LookML (Looker's modeling language) for those who are migrating. Previously this wasn’t available. *  External attributes The extended attributes feature in dbt Cloud is now GA! It allows for an environment level override on any YAML attribute that a dbt adapter accepts in its `profiles.yml`. You can provide a YAML snippet to add or replace any [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) value. To learn more, refer to [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes). The **Extended Attributes** text box is available from your environment's settings page: [![Example of the Extended attributes text box](/img/docs/dbt-platform/using-dbt-platform/extended-attributes.png?v=2 "Example of the Extended attributes text box")](#)Example of the Extended attributes text box *  Legacy semantic layer dbt Labs has deprecated dbt Metrics and the legacy dbt Semantic Layer, both supported on dbt version 1.5 or lower. This change starts on December 15th, 2023. This deprecation means dbt Metrics and the legacy Semantic Layer are no longer supported. We also removed the feature from the dbt Cloud user interface and documentation site. ##### Why this change? The [re-released dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), powered by MetricFlow, offers enhanced flexibility, performance, and user experience, marking a significant advancement for the dbt community. ##### Key changes and impact * **Deprecation date** — The legacy Semantic Layer and dbt Metrics will be officially deprecated on December 15th, 2023. * **Replacement** — [MetricFlow](https://docs.getdbt.com/docs/build/build-metrics-intro.md) replaces dbt Metrics for defining semantic logic. The `dbt_metrics` package will no longer be supported post-deprecation. * **New feature** — Exports replaces the materializing data with `metrics.calculate` functionality and will be available in dbt Cloud in December or January. ##### Breaking changes and recommendations * For users on dbt version 1.5 and lower with dbt Metrics and Snowflake proxy: * **Impact**: Post-deprecation, queries using the proxy *will not* run. * For users on dbt version 1.5 and lower using dbt Metrics without Snowflake proxy: * **Impact**: No immediate disruption, but the package will not receive updates or support after deprecation * **Recommendation**: Plan migration to the re-released Semantic Layer for compatibility with dbt version 1.6 and higher. ##### Engage and support * Feedback and community support — Engage and share feedback with the dbt Labs team and dbt Community slack using channels like [#dbt-cloud-semantic-layer](https://getdbt.slack.com/archives/C046L0VTVR6) and [#dbt-metricflow](https://getdbt.slack.com/archives/C02CCBBBR1D). Or reach out to your dbt Cloud account representative. * Resources for upgrading — Refer to some additional info and resources to help you upgrade your dbt version: * [Upgrade version in dbt Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Version migration guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) #### November 2023 *  New features and UI changes to dbt Catalog There are new quality-of-life improvements in dbt Cloud for email and Slack notifications about your jobs: * You can add external email addresses and send job notifications to them. External emails can be: * Addresses that are outside of your dbt Cloud account * Third-party integration addresses for configuring notifications to services like Microsoft Teams or PagerDuty * You can configure notifications for multiple Slack channels. Previously, you could only configure one Slack channel. * Any account admin can now edit slack notifications, not just the person who created them. To learn more, check out [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md). *  Job notifications There are new quality-of-life improvements in dbt Cloud for email and Slack notifications about your jobs: * You can add external email addresses and send job notifications to them. External emails can be: * Addresses that are outside of your dbt Cloud account * Third-party integration addresses for configuring notifications to services like Microsoft Teams or PagerDuty * You can configure notifications for multiple Slack channels. Previously, you could only configure one Slack channel. * Any account admin can now edit slack notifications, not just the person who created them. To learn more, check out [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md). *  Repo caching Now available for dbt Cloud Enterprise plans is a new option to enable Git repository caching for your job runs. When enabled, dbt Cloud caches your dbt project's Git repository and uses the cached copy instead if there's an outage with the Git provider. This feature improves the reliability and stability of your job runs. To learn more, refer to [Repo caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching). [![Example of the Repository caching option](/img/docs/deploy/account-settings-repository-caching.png?v=2 "Example of the Repository caching option")](#)Example of the Repository caching option #### October 2023 *  dbt Cloud APIs Beginning December 1, 2023, the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) v2 and v3 will expect you to limit all "list" or `GET` API methods to 100 results per API request. This limit enhances the efficiency and stability of our services. If you need to handle more than 100 results, then use the `limit` and `offset` query parameters to paginate those results; otherwise, you will receive an error. This maximum limit applies to [multi-tenant instances](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) only, and *does not* apply to single tenant instances. Refer to the [API v3 Pagination](https://docs.getdbt.com/dbt-cloud/api-v3#/) or [API v2 Pagination](https://docs.getdbt.com/dbt-cloud/api-v2#/) sections for more information on how to paginate your API responses. *  dbt CLI We are excited to announce the dbt CLI, **unified command line for dbt**, is available in public preview. It’s a local development experience, powered by dbt Cloud. It’s easy to get started: `pip3 install dbt` or `brew install dbt` and you’re ready to go. We will continue to invest in the dbt Cloud IDE as the easiest and most accessible way to get started using dbt, especially for data analysts who have never developed software using the command line before. We will keep improving the speed, stability, and feature richness of the IDE, as we have been [all year long](https://www.getdbt.com/blog/improvements-to-the-dbt-cloud-ide/). We also know that many people developing in dbt have a preference for local development, where they can use their favorite terminal, text editor, keybindings, color scheme, and so on. This includes people with data engineering backgrounds, as well as those analytics engineers who started writing code in the dbt Cloud IDE and have expanded their skills. The new dbt CLI offers the best of both worlds, including: * The power of developing against the dbt Cloud platform * The flexibility of your own local setup Run whichever community-developed plugins, pre-commit hooks, or other arbitrary scripts you like. Some of the unique capabilities of this dbt CLI include: * Automatic deferral of build artifacts to your Cloud project's production environment * Secure credential storage in the dbt Cloud platform * Support for dbt Mesh ([cross-project `ref`](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md)) * Development workflow for dbt Semantic Layer * Speedier, lower cost builds Refer to [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) to learn more. *  Custom branch fix If you don't set a [custom branch](https://docs.getdbt.com/docs/dbt-platform-environments.md#custom-branch-behavior) for your dbt Cloud environment, it now defaults to the default branch of your Git repository (for example, `main`). Previously, [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) would run for pull requests (PRs) that were opened against *any branch* or updated with new commits if the **Custom Branch** option wasn't set. #### Azure DevOps Your Git pull requests (PRs) might not trigger against your default branch if you're using Azure DevOps and the default branch isn't `main` or `master`. To resolve this, [set up a custom branch](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) with the branch you want to target. *  dbt deps auto install The dbt Cloud IDE and dbt CLI now automatically installs `dbt deps` when your environment starts or when necessary. Previously, it would prompt you to run `dbt deps` during initialization. This improved workflow is available to all multi-tenant dbt Cloud users (Single-tenant support coming next week) and applies to dbt versions. However, you should still run the `dbt deps` command in these situations: * When you make changes to the `packages.yml` or `dependencies.yml` file during a session * When you update the package version in the `packages.yml` or `dependencies.yml` file. * If you edit the `dependencies.yml` file and the number of packages remains the same, run `dbt deps`. (Note that this is a known bug dbt Labs will fix in the future.) *  Native retry support Previously in dbt Cloud, you could only rerun an errored job from start but now you can also rerun it from its point of failure. You can view which job failed to complete successfully, which command failed in the run step, and choose how to rerun it. To learn more, refer to [Retry jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md). [![Example of the Rerun options in dbt Cloud](/img/docs/deploy/native-retry.gif?v=2 "Example of the Rerun options in dbt Cloud")](#)Example of the Rerun options in dbt Cloud *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 15 new community contributors to docs.getdbt.com. We merged [107 PRs](https://github.com/dbt-labs/docs.getdbt.com/pulls?q=is%3Apr+merged%3A2023-09-01..2023-09-31) in September. Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/): * Migrated docs.getdbt.com from Netlify to Vercel. #### ☁ Cloud projects * Continuous integration jobs are now generally available and no longer in beta! * Added [Postgres PrivateLink set up page](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-postgres.md) * Published beta docs for [dbt Explorer](https://docs.getdbt.com/docs/explore/explore-projects.md). * Added a new Semantic Layer [GraphQL API doc](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) and updated the [integration docs](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) to include Hex. Responded to dbt community feedback and clarified Metricflow use cases for dbt Core and dbt Cloud. * Added an [FAQ](https://docs.getdbt.com/faqs/Git/git-migration.md) describing how to migrate from one git provider to another in dbt Cloud. * Clarified an example and added a [troubleshooting section](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md#troubleshooting) to Snowflake connection docs to address common errors and provide solutions. #### 🎯 Core projects * Deprecated dbt Core v1.0 and v1.1 from the docs. * Added configuration instructions for the [AWS Glue](https://docs.getdbt.com/docs/local/connect-data-platform/glue-setup.md) community plugin. * Revised the dbt Core quickstart, making it easier to follow. Divided this guide into steps that align with the [other guides](https://docs.getdbt.com/guides/manual-install.md?step=1). #### New 📚 Guides, ✏️ blog posts, and FAQs Added a [style guide template](https://docs.getdbt.com/best-practices/how-we-style/6-how-we-style-conclusion.md#style-guide-template) that you can copy & paste to make sure you adhere to best practices when styling dbt projects! #### Upcoming changes Stay tuned for a flurry of releases in October and a filterable guides section that will make guides easier to find! *  Semantic layer GA If you're using the legacy Semantic Layer, we *highly* recommend you [upgrade your dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to dbt v1.6 or higher and migrate to the latest Semantic Layer. dbt Labs is thrilled to announce that the [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) is now generally available. It offers consistent data organization, improved governance, reduced costs, enhanced efficiency, and accessible data for better decision-making and collaboration across organizations. It aims to bring the best of modeling and semantics to downstream applications by introducing: * Brand new [integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) such as Tableau, Google Sheets, Hex, Mode, and Lightdash. * New [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) using GraphQL and JDBC to query metrics and build integrations. * dbt Cloud [multi-tenant regional](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) support for North America, EMEA, and APAC. Single-tenant support coming soon. * Coming soon — Schedule exports (a way to build tables in your data platform) as part of your dbt Cloud job. Use the APIs to call an export, then access them in your preferred BI tool. [![Use the universal dbt Semantic Layer to define and queried metrics in integration tools.](/img/docs/dbt-platform/semantic-layer/sl-architecture.jpg?v=2 "Use the universal dbt Semantic Layer to define and queried metrics in integration tools.")](#)Use the universal dbt Semantic Layer to define and queried metrics in integration tools. The dbt Semantic Layer is available to [dbt Cloud Team or Enterprise](https://www.getdbt.com/) multi-tenant plans on dbt v1.6 or higher. * Team and Enterprise customers can use 1,000 Queried Metrics per month for no additional cost on a limited trial basis, subject to reasonable use limitations. Refer to [Billing](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#what-counts-as-a-queried-metric) for more information. * dbt Developer plans and dbt v1 users can define metrics but won't be able to query them with integrated tools. #### September 2023 *  CI updates dbt Cloud now has two distinct job types: [deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) for building production data assets, and [continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) for checking code changes. These jobs perform fundamentally different tasks so dbt Labs improved the setup experience with better defaults for each. With two types of jobs, instead of one generic type, we can better guide you through the setup flow. Best practices are built into the default settings so you can go from curious to being set up in seconds. And, we now have more efficient state comparisons on CI checks: never waste a build or test on code that hasn’t been changed. We now diff between the Git pull request (PR) code and what’s running in production more efficiently with the introduction of deferral to an environment versus a job. To learn more, refer to [Continuous integration in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md). Below is a comparison table that describes how deploy jobs and CI jobs behave differently: | | Deploy Jobs | CI Jobs | | ---------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | | Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. | | Trigger types | Triggered by a schedule or by API. | Triggered by a commit to a PR or by API. | | Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. | | Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. | | Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. | | State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | #### What you need to update * If you want to set up a CI environment for your jobs, dbt Labs recommends that you create your CI job in a dedicated [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) that's connected to a staging database. To learn more about these environment best practices, refer to the guide [Get started with continuous integration tests](https://docs.getdbt.com/guides/set-up-ci.md). * If you had set up a CI job before October 2, 2023, the job might've been misclassified as a deploy job with this update. Below describes how to fix the job type: If you used the [Create Job](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint but didn't set `"triggers":triggers.git_provider_webhook`, the job was misclassified as a deploy job and you must re-create it as described in [Trigger a CI job with the API](https://docs.getdbt.com/docs/deploy/ci-jobs.md#trigger-a-ci-job-with-the-api). If you used the dbt UI but didn't enable the **Run on Pull Requests** option that was in the **Continuous Integration** (CI) tab, the job was misclassified as a deploy job and you must re-create it as described in [Set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs). To check for the job type, review your CI jobs in dbt's [Run History](https://docs.getdbt.com/docs/deploy/run-visibility.md#run-history) and check for the **CI Job** tag below the job name. If it doesn't have this tag, it was misclassified and you need to re-create the job. **CI update phase 3 — Update: Improved automatic deletion of temporary schemas** Temporary schemas are now being automatically deleted (dropped) for all adapters (like Databricks), PrivateLink connections, and environment variables in connection strings. dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr_` that's been generated by a PR run. However, temporary schemas will not be automatically deleted if: * Your project overrides the [generate\_schema\_name macro](https://docs.getdbt.com/docs/build/custom-schemas.md) but it doesn't contain the required prefix `dbt_cloud_pr_`. For details, refer to [Troubleshooting](https://docs.getdbt.com/docs/deploy/ci-jobs.md#troubleshooting). * You're using a [non-native Git integration](https://docs.getdbt.com/docs/deploy/ci-jobs.md#trigger-a-ci-job-with-the-api). This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. *  Product docs updates Hello from dbt's Product Documentation team (the stewards of the docs.getdbt.com site): @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun. What a busy summer! We merged 256 PRs between July 1st and August 31. We'd like to recognize all of the docs and support from our partner team, Developer Experience: @jasnonaz @gwenwindflower @dbeatty10 @dataders @joellabes @Jstein77 @dave-connors-3! We'd also like to give a special thanks to the 22 community members who contributed to the [dbt Product docs](https://docs.getdbt.com) for the first time. 🙏 Based on feedback from the dbt community, we made these changes: * Added a [permissions table](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for Enterprise accounts * Added a [browser session page](https://docs.getdbt.com/docs/platform/about-platform/browsers.md#browser-sessions) that clarifies dbt Cloud’s browser session time and when it logs users off. You can provide feedback by opening a pull request or issue in [our repo](https://github.com/dbt-labs/docs.getdbt.com) or reaching out in the dbt community Slack channel [#dbt-product-docs](https://getdbt.slack.com/archives/C0441GSRU04)). #### ⚡ General docs projects * Added the ability to collapse sections you’re not currently looking at. There were quite a few people who wanted this, and it bugged us too, so we were happy to get this shipped! * Introduced the idea of [“Trusted” adapters](https://docs.getdbt.com/docs/community-adapters.md). #### ☁ Cloud projects * The **What’s new?** product update widget is back in the dbt Cloud UI! The Docs team will begin updating the content to keep you informed about new features. * Launched the re-released [Semantic Layer beta docs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), which introduces users to the new API, new guide to set up MetricFlow and the new Semantic Layer, as well as revamp the ‘Use the dbt Semantic Layer’ section for users. * Updated [Admin API v2 and v3](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to help you understand the differences between them and which version includes the endpoints you use. * To improve discoverability, the docs team made changes to the [deploy dbt sidebar](https://docs.getdbt.com/docs/deploy/deployments.md). We added cards and aligned better with the dbt Cloud UI and the way it’s used. * Deprecated legacy job schemas in the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). * Added a page to describe [experimental and beta features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) in dbt Cloud and what you need to know about them. * Added a section to introduce a new beta feature [**Extended Attributes**](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes-beta), which allows users to set a flexible `profiles.yml` snippet in their dbt Cloud Environment settings. #### 🎯 Core projects * We released [dbt 1.6](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.6.md>)! We added docs for the new commands `dbt retry` and `dbt clone` #### New 📚 Guides, ✏️ blog posts, and FAQs * Learn how to [join the dbt community](https://docs.getdbt.com/community/join.md). * Blog posts published this summer include [Optimizing Materialized Views with dbt](https://docs.getdbt.com/blog/announcing-materialized-views), [Data Vault 2.0 with dbt Cloud](https://docs.getdbt.com/blog/data-vault-with-dbt-cloud), and [Create dbt Documentation and Tests 10x faster with ChatGPT](https://docs.getdbt.com/blog/create-dbt-documentation-10x-faster-with-ChatGPT) - We now have two new best practice guides: [How we build our metrics](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md) and [Set up Continuous Integration](https://docs.getdbt.com/guides/set-up-ci.md). *  Removing prerelease versions Previously, when dbt Labs released a new [version](https://docs.getdbt.com/docs/dbt-versions.md#how-dbt-v1-uses-semantic-versioning) in dbt Cloud, the older patch *prerelease* version and the *latest* version remained as options in the dropdown menu available in the **Environment settings**. Now, when the *latest* version is released, the *prerelease* version will be removed and all customers remaining on it will be migrated seamlessly. There will be no interruptions to service when this migration occurs. To see which version you are currently using and to upgrade, select **Deploy** in the top navigation bar and select **Environments**. Choose the preferred environment and click **Settings**. Click **Edit** to make a change to the current dbt version. dbt Labs recommends always using the latest version whenever possible to take advantage of new features and functionality. #### August 2023 *  Deprecation of endpoints in the Discovery API dbt Labs has deprecated and will be deprecating certain query patterns and replacing them with new conventions to enhance the performance of the dbt Cloud [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). All these changes will be in effect on *September 7, 2023*. We understand that these changes might require adjustments to your existing integration with the Discovery API. Please [contact us](mailto:support@getdbt.com) with any questions. We're here to help you during this transition period. #### Job-based queries Job-based queries that use the data type `Int` for IDs will be deprecated. They will be marked as deprecated in the [GraphQL explorer](https://metadata.cloud.getdbt.com/graphql). The new convention will be for you to use the data type `BigInt` instead. This change will be in effect starting September 7, 2023. Example of query before deprecation: ```graphql query ($jobId: Int!) { models(jobId: $jobId){ uniqueId } } ``` Example of query after deprecation: ```graphql query ($jobId: BigInt!) { job(id: $jobId) { models { uniqueId } } } ``` #### modelByEnvironment queries The `modelByEnvironment` object has been renamed and moved into the `environment` object. This change is in effect and has been since August 15, 2023. Example of query before deprecation: ```graphql query ($environmentId: Int!, $uniqueId: String) { modelByEnvironment(environmentId: $environmentId, uniqueId: $uniqueId) { uniqueId executionTime executeCompletedAt } } ``` Example of query after deprecation: ```graphql query ($environmentId: BigInt!, $uniqueId: String) { environment(id: $environmentId) { applied { modelHistoricalRuns(uniqueId: $uniqueId) { uniqueId executionTime executeCompletedAt } } } } ``` #### Environment and account queries Environment and account queries that use `Int` as a data type for ID have been deprecated. IDs must now be in `BigInt`. This change is in effect and has been since August 15, 2023. Example of query before deprecation: ```graphql query ($environmentId: Int!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId executionInfo { lastRunId } } } } } } } ``` Example of query after deprecation: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId executionInfo { lastRunId } } } } } } } ``` *  dbt Cloud IDE v1.2 We're excited to announce that we replaced the backend service that powers the Cloud IDE with a more reliable server -- dbt-server. Because this release contains foundational changes, IDE v1.2 requires dbt v1.6 or higher. This significant update follows the rebuild of the IDE frontend last year. We're committed to improving the IDE to provide you with a better experience. Previously, the Cloud IDE used dbt-rpc, an outdated service that was unable to stay up-to-date with changes from dbt-core. The dbt-rpc integration used legacy dbt-core entry points and logging systems, causing it to be sluggish, brittle, and poorly tested. The Core team had been working around this outdated technology to avoid breaking it, which prevented them from developing with velocity and confidence. #### New features * **Better dbt-core parity:** The Cloud IDE has better command parity with dbt-core, including support for commands like `dbt list` and improved treatment of flags like `--vars`, `--fail-fast`, etc. * **Improved maintainability:** With the new dbt-server, it's easier to fix bugs and improve the overall quality of the product. With dbt-rpc, fixing bugs was a time-consuming and challenging process that required extensive testing. With the new service, we can identify and fix bugs more quickly, resulting in a more stable and reliable IDE. * **A more reliable service:** Simplified architecture that's less prone to failure. ##### Product refinements * Improved `Preview` capabilities with Core v1.6 + IDE v1.2. [This Loom](https://www.loom.com/share/12838feb77bf463c8585fc1fc6aa161b) provides more information. ##### Bug fixes * Global page can become "inert" and stop handling clicks * Switching back and forth between files in the git diff view can cause overwrite * Browser gets stuck during markdown preview for doc with large table * Editor right click menu is offset * Unable to Cancel on the Save New File component when Closing All Files in the IDE * Mouse flicker in the modal's file tree makes it difficult to select a folder where you want to save a new file * Snapshots not showing in Lineage when inside a subfolder and is mixed cased named * Tooltips do not work for Format and Save * When a dbt invocation is in progress or if parsing is ongoing, attempting to switch branches will cause the `Git Branch` dropdown to close automatically ##### Known issues * `{{this}}` function does not display properly in preview/compile with dbt-server #### July 2023 *  Faster runs and unlimited job concurrency for Enterprise account We’ve introduced significant improvements to the dbt Cloud Scheduler, offering improved performance, durability, and scalability. Read more on how you can experience faster run start execution and how enterprise users can now run as many jobs concurrently as they want to. #### Faster run starts The Scheduler takes care of preparing each dbt Cloud job to run in your cloud data platform. This [prep](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) involves readying a Kubernetes pod with the right version of dbt installed, setting environment variables, loading data platform credentials, and git provider authorization, amongst other environment-setting tasks. Only after the environment is set up, can dbt execution begin. We display this time to the user in dbt Cloud as “prep time”. [![The scheduler prepares a job for execution and displays it as 'prep time' in dbt Cloud.](/img/run-start.jpg?v=2 "The scheduler prepares a job for execution and displays it as 'prep time' in dbt Cloud.")](#)The scheduler prepares a job for execution and displays it as 'prep time' in dbt Cloud. For all its strengths, Kubernetes has challenges, especially with pod management impacting run execution time. We’ve rebuilt our scheduler by ensuring faster job execution with a ready pool of pods to execute customers’ jobs. This means you won't experience long prep times at the top of the hour, and we’re determined to keep runs starting near instantaneously. Don’t just take our word, review the data yourself. [![Job prep time data has seen a 75% speed improvement from Jan 2023 to July 2023. Prep time took 106 secs in Jan and now takes 27 secs as of July.](/img/prep-start.jpg?v=2 "Job prep time data has seen a 75% speed improvement from Jan 2023 to July 2023. Prep time took 106 secs in Jan and now takes 27 secs as of July.")](#)Job prep time data has seen a 75% speed improvement from Jan 2023 to July 2023. Prep time took 106 secs in Jan and now takes 27 secs as of July. Jobs scheduled at the top of the hour used to take over 106 seconds to prepare because of the volume of runs the scheduler has to process. Now, even with increased runs, we have reduced prep time to 27 secs (at a maximum) — a 75% speed improvement for runs at peak traffic times! #### Unlimited job concurrency for Enterprise accounts Our enhanced scheduler offers more durability and empowers users to run jobs effortlessly. This means Enterprise, multi-tenant accounts can now enjoy the advantages of unlimited job concurrency. Previously limited to a fixed number of run slots, Enterprise accounts now have the freedom to operate without constraints. Single-tenant support will be coming soon. Something to note, each running job occupies a run slot for its duration, and if all slots are occupied, jobs will queue accordingly. For more feature details, refer to the [dbt pricing page](https://www.getdbt.com/pricing/). Note, Team accounts created after July 2023 benefit from unlimited job concurrency: * Legacy Team accounts have a fixed number of run slots. * Both Team and Developer plans are limited to one project each. For larger-scale needs, our [Enterprise plan](https://www.getdbt.com/pricing/) offers features such as audit logging, unlimited job concurrency and projects, and more. #### June 2023 *  Lint format dbt Labs is excited to announce you can now lint and format your dbt code in the dbt Cloud IDE. This is an enhanced development workflow which empowers you to effortlessly prioritize code quality. You can perform linting and formatting on five different file types: SQL, YAML, Markdown, Python, and JSON. For SQL files, you can easily lint and format your code using [SQLFluff](https://sqlfluff.com/) and apply consistent formatting using [sqlfmt](http://sqlfmt.com/). Additionally, for other file types like YAML, Markdown, JSON, and Python, you can utilize the respective tools powered by [Prettier](https://prettier.io/) and [Black](https://black.readthedocs.io/en/latest/) to ensure clean and standardized code formatting. For more info, read [Lint and format your code](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md). [![Use SQLFluff to lint/format your SQL code, and view code errors in the Code Quality tab.](/img/docs/dbt-platform/platform-ide/sqlfluff.gif?v=2 "Use SQLFluff to lint/format your SQL code, and view code errors in the Code Quality tab.")](#)Use SQLFluff to lint/format your SQL code, and view code errors in the Code Quality tab. [![Use sqlfmt to format your SQL code.](/img/docs/dbt-platform/platform-ide/sqlfmt.gif?v=2 "Use sqlfmt to format your SQL code.")](#)Use sqlfmt to format your SQL code. [![Format YAML, Markdown, and JSON files using Prettier.](/img/docs/dbt-platform/platform-ide/prettier.gif?v=2 "Format YAML, Markdown, and JSON files using Prettier.")](#)Format YAML, Markdown, and JSON files using Prettier. *  CI updates dbt Cloud CI is a critical part of the analytics engineering workflow. Large teams rely on process to ensure code quality is high, and they look to dbt Cloud CI to automate testing code changes in an efficient way, enabling speed while keep the bar high. With status checks directly posted to their dbt PRs, developers gain the confidence that their code changes will work as expected in production, and once you’ve grown accustomed to seeing that green status check in your PR, you won’t be able to work any other way. [![CI checks directly from within Git](/img/docs/release-notes/ci-checks.png?v=2 "CI checks directly from within Git")](#)CI checks directly from within Git What separates dbt CI from other CI providers is its ability to keep track of state of what’s running in your production environment, so that when you run a CI job, only the modified data assets in your pull request and their downstream dependencies get built and tested in a staging schema. dbt aims to make each CI check as efficient as possible, so as to not waste any data warehouse resources. As soon as the CI run completes, its status posts directly back to the PR in GitHub, GitLab, or Azure DevOps, depending on which Git provider you’re using. Teams can set up guardrails to let only PRs with successful CI checks be approved for merging, and the peer review process is greatly streamlined because dbt does the first testing pass. We're excited to introduce a few critical capabilities to dbt CI that will improve productivity and collaboration in your team’s testing and integration workflow. As of this week, you can now: * **Run multiple CI checks in parallel**. If more than one contributor makes changes to the same dbt project in dbt Cloud in short succession, the later arriving CI check no longer has to wait for the first check to complete. Both checks will execute concurrently. * **Automatically cancel stale CI runs**. If you push multiple commits to the same PR, dbt will cancel older, now-out-of-date CI checks automatically. No resources wasted on checking stale code. * **Run CI checks without blocking production runs**. CI checks will no longer consume run slots, meaning you can have as many CI checks running as you want, without impeding your production jobs. To learn more, refer to [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) and [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md). *  Admin API dbt Labs updated the docs for the [dbt Cloud Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and they are now available for both [v2](https://docs.getdbt.com/dbt-cloud/api-v2#/) and [v3](https://docs.getdbt.com/dbt-cloud/api-v3#/). * Now using Spotlight for improved UI and UX. * All endpoints are now documented for v2 and v3. Added automation to the docs so they remain up to date. * Documented many of the request and response bodies. * You can now test endpoints directly from within the API docs. And, you can choose which [regional server](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) to use (North America, APAC, or EMEA). * With the new UI, you can more easily generate code for any endpoint. *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 17 new community contributors to docs.getdbt.com — ✨ @aaronbini, @sjaureguimodo, @aranke, @eiof, @tlochner95, @mani-dbt, @iamtodor, @monilondo, @vrfn, @raginjason, @AndrewRTsao, @MitchellBarker, @ajaythomas, @smitsrr, @leoguyaux, @GideonShils, @michaelmherrera! Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/) in June: #### ☁ Cloud projects * We clarified the nuances of [CI and CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md), updated the [Scheduler content](https://docs.getdbt.com/docs/deploy/job-scheduler.md), added two new pages for the job settings and run visibility, moved the project state page to the [Syntax page](https://docs.getdbt.com/reference/node-selection/syntax.md), and provided a landing page for [Deploying with Cloud](https://docs.getdbt.com/docs/deploy/jobs.md) to help readers navigate the content better. * We reformatted the [Supported data platforms page](https://docs.getdbt.com/docs/supported-data-platforms.md) by adding dbt Cloud to the page, splitting it into multiple pages, using cards to display verified adapters, and moving the [Warehouse setup pages](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md) to the Docs section. * We launched a new [Lint and format page](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md), which highlights the awesome new dbt Cloud IDE linting/formatting function. * We enabled a connection between [dbt Cloud release notes](https://docs.getdbt.com/docs/dbt-versions/release-notes.md) and the dbt Slack community. This means new dbt Cloud release notes are automatically sent to the slack community [#dbt-cloud channel](https://getdbt.slack.com/archives/CMZ2V0X8V) via RSS feed, keeping users up to date with changes that may affect them. * We’ve added two new docs links in the dbt Cloud Job settings user interface (UI). This will provide additional guidance and help users succeed when setting up a dbt Cloud job: [job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) and job triggers. * We added information related to the newly created [IT license](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#license-based-access-control), available for Team and Enterprise plans. * We added a new [Supported browser page](https://docs.getdbt.com/docs/platform/about-platform/browsers.md), which lists the recommended browsers for dbt Cloud. * We launched a new page informing users of [new Experimental features option](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) in dbt Cloud. * We worked with dbt Engineering to help publish new beta versions of the dbt [dbt Cloud Administrative API docs](https://docs.getdbt.com/docs/dbt-apis/admin-api.md). #### 🎯 Core projects * We launched the new [MetricFlow docs](https://docs.getdbt.com/docs/build/build-metrics-intro.md) on dbt Core v1.6 beta. * Split [Global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) into individual pages, making it easier to find, especially using search. #### New 📚 Guides, ✏️ blog posts, and FAQs * Add an Azure DevOps example in the [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md) guide. #### May 2023 *  dbt Cloud IDE To continue improving your [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) development experience, the dbt Labs team continues to work on adding new features, fixing bugs, and increasing reliability ✨. Stay up-to-date with [IDE-related changes](https://docs.getdbt.com/tags/ide.md). #### New features * Lint via SQL Fluff is now available in beta (GA over the next 2-3 weeks) * Format markdown files with prettier * Leverage developer experience shortcuts, including ``Ctrl + ` ``(toggle history drawer), `CMD + Option + /` (toggle block comment), `CMD + Shift + P` (open command palette), `Option + W` (close editor tab) * Display parent folder name for files with same name in Changes section * Navigate the new IDE features quickly using [the IDE User Interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) help page * Use `top X` in SQL when previewing in the IDE * Opt into the new IDE backend layer over the past month (still with dbt-rpc). Ready for beta later in June! #### Product refinements * Performance-related upgrades: * Reduced cold start time by 60+% * Improved render time of modals in the IDE by 98% * Improved IDE performance with dbt Core v1.5+ (faster and snappier – highly encourage you to [upgrade your dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md)!) * Upgraded sqlfmt (which powers the Format button) to 0.18.0 * Updated Build button to change menu options based on file/model type (snapshot, macro, etc.) * Display message to disable adblocker for file contents error * Moved Format button to console bar * Made many security enhancements in the IDE #### Bug fixes * File icon sizes no longer get wonky in small screen * Toast notifications no longer take over command bar menu * Hover info inside the text editor no longer gets cut off * Transition between a file and a recently modified scratchpad no longer triggers a console error * dbt v1.5+ now can access the IDE * Confirm button on the Unsaved Changes modal now closes after clicking it * Long node names no longer overflow in the parsed logs section in history drawer * Status pill in history drawer no longer scales with longer command * Tooltip for tab name with a long file name is no longer cut off * Lint button should no longer available in main branch *  Run history improvements New usability and design improvements to the **Run History** dashboard in dbt Cloud are now available. These updates allow people to discover the information they need more easily by reducing the number of clicks, surfacing more relevant information, keeping people in flow state, and designing the look and feel to be more intuitive to use. Highlights include: * Usability improvements for CI runs with hyperlinks to the branch, PR, and commit SHA, along with more discoverable temporary schema names * Preview of runs' error messages on hover * Hyperlinks to the environment * Better iconography on run status * Clearer run trigger cause (API, scheduled, pull request, triggered by user) * More details on the schedule time on hover * Run timeout visibility dbt Labs is making a change to the metadata retrieval policy for Run History in dbt Cloud. **Beginning June 1, 2023,** developers on the dbt multi-tenant application will be able to self-serve access to their account’s run history through the dbt user interface (UI) and API for only 365 days, on a rolling basis. Older run history will be available for download by reaching out to Customer Support. We're seeking to minimize the amount of metadata we store while maximizing application performance. Specifically, all `GET` requests to the dbt Cloud [Runs endpoint](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs) will return information on runs, artifacts, logs, and run steps only for the past 365 days. Additionally, the run history displayed in the dbt Cloud UI will only show runs for the past 365 days. [![The dbt Cloud UI displaying a Run History](/img/docs/dbt-platform/rn-run-history.jpg?v=2 "The dbt Cloud UI displaying a Run History")](#)The dbt Cloud UI displaying a Run History We will retain older run history in cold storage and can make it available to customers who reach out to our Support team. To request older run history info, contact the Support team at <support@getdbt.com> or use the dbt Cloud application chat by clicking the `?` icon in the dbt Cloud UI. *  Run details and log improvements New usability and design improvements to the run details and logs in dbt Cloud are now available. The ability to triage errors in logs is a big benefit of using dbt Cloud's job and scheduler functionality. The updates help make the process of finding the root cause much easier. Highlights include: * Surfacing a warn state on a run step * Search in logs * Easier discoverability of errors and warnings in logs * Lazy loading of logs, making the whole run details page load faster and feel more performant * Cleaner look and feel with iconography * Helpful tool tips *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 13 new community contributors to docs.getdbt.com! Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/) in May: #### 🔎 Discoverability * We made sure everyone knows that Cloud-users don’t need a [profiles.yml file](https://docs.getdbt.com/docs/local/profiles.yml.md) by adding a callout on several key pages. * Fleshed out the [model Jinja variable page](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md), which originally lacked conceptual info and didn’t link to the schema page. * Added a new [Quickstarts landing page](https://docs.getdbt.com/guides.md). This new format sets up for future iterations that will include filtering! But for now, we are excited you can step through quickstarts in a focused way. #### Cloud projects * We launched [dbt Cloud IDE user interface doc](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md), which provides a thorough walkthrough of the IDE UI elements and their definitions. * Launched a sparkling new [dbt Cloud Scheduler page](https://docs.getdbt.com/docs/deploy/job-scheduler.md) ✨! We went from previously having little content around the scheduler to a subsection that breaks down the awesome scheduler features and how it works. * Updated the [dbt Cloud user license page](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses) to clarify how to add or remove cloud users. * Shipped these Discovery API docs to coincide with the launch of the Discovery API: * [About the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) * [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) * [Query the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) #### 🎯 Core projects * See what’s coming up [in Core v 1.6](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aissue+label%3A%22dbt-core+v1.6%22)! * We turned the `profiles.yml` [page](https://docs.getdbt.com/docs/local/profiles.yml.md) into a landing page, added more context to profiles.yml page, and moved the ‘About CLI’ higher up in the `Set up dbt` section. #### New 📚 Guides, ✏️ blog posts, and FAQs If you want to contribute to a blog post, we’re focusing on content * Published a blog post: [Accelerate your documentation workflow: Generate docs for whole folders at once](https://docs.getdbt.com/blog/generating-dynamic-docs-dbt) * Published a blog post: [Data engineers + dbt v1.5: Evolving the craft for scale](https://docs.getdbt.com/blog/evolving-data-engineer-craft) * Added an [FAQ](https://docs.getdbt.com/faqs/Warehouse/db-connection-dbt-compile.md) to clarify the common question users have on *Why does dbt compile needs to connect to the database?* * Published a [discourse article](https://discourse.getdbt.com/t/how-to-configure-external-user-email-notifications-in-dbt-cloud/8393) about configuring job notifications for non-dbt Cloud users #### April 2023 *  dbt Cloud IDE #### New features * New warning message suggests you invoke `dbt deps` when it's needed (as informed by `dbt-score`). * New warning message appears when you select models but don't save them before clicking **Build** or invoking dbt (like, dbt build/run/test). * Previews of Markdown and CSV files are now available in the IDE console. * The file tree menu now includes a Duplicate File option. * Display loading time when previewing a model #### Product refinements * Enhance autocomplete experience which has performed slowly for people with large projects and who implement a limit to max `manifest.json` for this feature * Introduce pagination for invocation node summary view (displaying 100 nodes at a time) * Improve rendering for the Changes / Version Control section of the IDE * Update icons to be consistent in dbt Cloud * Add table support to the Markdown preview * Add the lineage tab back to seed resources in the IDE * Implement modal priority when there are multiple warning modals * Improve a complex command's description in the command palette #### Bug fixes * File tree no longer collapses on first click when there's a project subdirectory defined * **Revert all** button now works as expected * CSV preview no longer fails with only one column * Cursor and scroll bar location are now persistent with their positions * `git diff` view now shows just change diffs and no longer shows full diff (as if file is new) until page refreshes * ToggleMinimap Command no longer runs another Command at the same time * `git diff` view no longer shows infinite spins in specific scenarios (new file, etc.) * File contents no longer get mixed up when using diff view and one file has unsaved changes * YML lineage now renders model without tests (in dbt Core v1.5 and above) * Radio buttons for **Summary** and **Details** in the logs section now consistently update to show the accurate tab selection * IDE no longer throws the console error `Error: Illegal argument` and redirects to the `Something went wrong` page *  API updates Starting May 15, 2023, we will support only the following `order_by` functionality for the List Runs endpoint: * `id` and `-id` * `created_at` and `-created_at` * `finished_at` and `-finished_at` We recommend that you change your API requests to https://\<YOUR\_ACCESS\_URL>/api/v2/accounts/{accountId}/runs/ to use a supported `order_by` before this date. Access URLs dbt Cloud is hosted in multiple regions around the world, and each region has a different access URL. Users on Enterprise plans can choose to have their account hosted in any one of these regions. For a complete list of available dbt Cloud access URLs, refer to [Regions & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). For more info, refer to our [documentation](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs). *  Scheduler optimization The dbt Cloud Scheduler now prevents queue clog by canceling unnecessary runs of over-scheduled jobs. The duration of a job run tends to grow over time, usually caused by growing amounts of data in the warehouse. If the run duration becomes longer than the frequency of the job’s schedule, the queue will grow faster than the scheduler can process the job’s runs, leading to a runaway queue with runs that don’t need to be processed. Previously, when a job was in this over-scheduled state, the scheduler would stop queuing runs after 50 were already in the queue. This led to a poor user experience where the scheduler canceled runs indiscriminately. You’d have to log into dbt Cloud to manually cancel all the queued runs and change the job schedule to "unclog" the scheduler queue. Now, the dbt Cloud scheduler detects when a scheduled job is set to run too frequently and appropriately cancels runs that don’t need to be processed. Specifically, scheduled jobs can only ever have one run of the job in the queue, and if a more recent run gets queued, the early queued run will get canceled with a helpful error message. Users will still need to either refactor the job so it runs faster or change the job schedule to run less often if the job often gets into an over-scheduled state. *  Starburst adapter GA The Starburst (Trino compatible) connection is now generally available in dbt Cloud. This means you can now use dbt Cloud to connect with Starburst Galaxy, Starburst Enterprise, and self-hosted Trino. This feature is powered by the [`dbt-trino`](https://github.com/starburstdata/dbt-trino) adapter. To learn more, check out our Quickstart guide for [dbt Cloud and Starburst Galaxy](https://docs.getdbt.com/guides/starburst-galaxy.md). *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! We want to share some highlights introduced to docs.getdbt.com in the last month: #### 🔎 Discoverability * [API docs](https://docs.getdbt.com/docs/dbt-apis/overview.md) now live in the left sidebar to improve discoverability. * [The deploy dbt jobs sidebar](https://docs.getdbt.com/docs/deploy/deployments.md) has had a glow up 💅 that splits the ‘about deployment’ into two paths (deploy w dbt cloud and deploy w other tools), adds more info about the dbt cloud scheduler, its features, and how to create a job, adds ADF deployment guidance. We hope the changes improve the user experience and provide users with guidance when deploying with other tools. #### ☁ Cloud projects * Added Starburst/Trino adapter docs, including: - [dbt Cloud quickstart guide](https://docs.getdbt.com/guides/starburst-galaxy.md),  - [connection page](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-starburst-trino.md),  - [set up page](https://docs.getdbt.com/docs/local/connect-data-platform/trino-setup.md), and [config page](https://docs.getdbt.com/reference/resource-configs/trino-configs.md). * Enhanced [dbt Cloud jobs page](https://docs.getdbt.com/docs/deploy/jobs.md) and section to include conceptual info on the queue time, improvements made around it, and about failed jobs. * Check out the April dbt [Cloud release notes](https://docs.getdbt.com/docs/dbt-versions/release-notes.md) #### 🎯 Core projects * Clearer descriptions in the [Jinja functions page](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md), that improve content for each card.  * [1.5 Docs](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) have been released as a Release Candidate (RC)! * See the beautiful [work captured in Core v 1.5](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aissue+label%3A%22dbt-core+v1.5%22+is%3Aclosed). #### New 📚 Guides and ✏️ blog posts * [Use Databricks workflows to run dbt Cloud jobs](https://docs.getdbt.com/guides/databricks-workflows.md) * [Refresh Tableau workbook with extracts after a job finishes](https://docs.getdbt.com/guides/zapier-refresh-tableau-workbook.md) * [dbt Python Snowpark workshop/tutorial](https://docs.getdbt.com/guides/dbt-python-snowpark.md) * [How to optimize and troubleshoot dbt Models on Databricks](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md) * [The missing guide to debug() in dbt](https://docs.getdbt.com/blog/guide-to-jinja-debug) * [dbt Squared: Leveraging dbt Core and dbt Cloud together at scale](https://docs.getdbt.com/blog/dbt-squared) * [Audit\_helper in dbt: Bringing data auditing to a higher level](https://docs.getdbt.com/blog/audit-helper-for-migration) #### March 2023 *  dbt v1.0 deprecation dbt Cloud now requires dbt version 1.0 or later. As of March 1, 2023, we removed all instances of older dbt versions from dbt Cloud. Any environments or jobs configured with a dbt version lower than 1.0 were automatically updated to dbt v1.4, which is the latest minor version available on dbt Cloud. For more info on dbt versions, releases, and dbt Cloud support timeline, refer to [About dbt Core versions](https://docs.getdbt.com/docs/dbt-versions.md#latest-releases). Refer to some additional info and resources to help you upgrade your dbt version: * [How to upgrade dbt without fear](https://docs.getdbt.com/blog/upgrade-dbt-without-fear) * [Upgrade Q\&A on breaking changes](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#upgrading-legacy-versions-under-10) * [Version migration guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) *  dbt Cloud IDE To continue improving your [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) development experience, the dbt Labs team continue to work on adding new features, fixing bugs, and increasing reliability ✨. Read more about the [upcoming improvements to the Cloud IDE](https://www.getdbt.com/blog/improvements-to-the-dbt-cloud-ide/) and stay up-to-date with [IDE-related changes](https://docs.getdbt.com/tags/ide.md). #### New features * Commit and revert individual files under **Version Control**. * Use the [command palette](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features) to invoke common complex dbt commands, such as resuming from the last failure. * Create PRs even when there are uncommitted changes (under the **git** dropdown). * The IDE will display more autocomplete suggestions when editing a YML file, powered by [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema). * The file tree now has additional options in the right-click menu, such as Copy model as ref or Copy file path. * The DAG view has been adjusted to a default of `2+model+2`. * A lineage selector has been implemented in the DAG/lineage sub-tab. * Edit directly in the git diff view located in the right pane. * A warning message will now appear when users press Command-W/Control-W when there are unsaved changes. * A new onboarding flow guide is now available. #### Product refinements * The DAG selector now uses `name` instead of `file_uri` to build selectors. * The DAG is now vertically centered under the new Selector Input element * sqlfmt has been upgraded to v0.17.0. * When the Format button fails, a toast notification will display a syntax error. * The editor now has the option to toggle minimap/word-wrap via right-click. * The history drawer displays elapsed time in real-time and s/m/h increments. * When deleting development environments, the delete modal will now warn users that any uncommitted changes will be lost. * The context for the Git button has been adjusted to show that it will link to an external site (such as GitHub or GitLab) when users create a pull request. #### Bug fixes * The IDE now displays an error message when the git repository is not reachable. Previously, it failed silently. * The kebab menu is now visible when the invocation history drawer is open. Previously, it wasn't showing. * DAGs are now updated/populated consistently. Previously, it occasionally failed. * The purple highlight for DAG selection is now consistent across files. Previously, it was inconsistent. * Users can now rename files back to their original name. Previously, this wasn't possible. * The link to the IDE from the project setup page has been corrected. * The IDE no longer has issues with single-space file names. * Adding invalid characters in the sub-directory config no longer causes the IDE to fail. * YML autocomplete triggers consistently now. Previously, it occasionally didn't trigger. * Reverting single files now reloads the file contents in the tab. Previously, it didn't reload. * The file tree no longer collapses on the first click when there is a project subdirectory defined. *  API updates To make the API more scalable and reliable, we've implemented a maximum limit of `100` for all API requests to our `list` endpoints. If API requests exceed the maximum limit parameter of `100`, a user will receive an API error message. This maximum limit applies to [multi-tenant instances](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) only, and *does not* apply to single tenant instances. Refer to the [Pagination](https://docs.getdbt.com/dbt-cloud/api-v2#/) section of the overview for more information on this change. #### Feb 2023 *  Disable partial parsing in job commands You can now use the `--no-partial-parse` flag to disable partial parsing in your dbt Cloud job commands.  Previously, the [`--no-partial-parse` global config](https://docs.getdbt.com/reference/global-configs/parsing.md) was only available in dbt Core. For more information, refer to [partial parsing](https://docs.getdbt.com/reference/parsing.md#partial-parsing). *  dbt Cloud IDE To continue improving our [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) experience, the dbt Labs team worked on fixing bugs, increasing reliability, and adding new features ✨. Learn more about the [February changes](https://getdbt.slack.com/archives/C03SAHKKG2Z/p1677605383451109). #### New features * Support for custom node colors in the IDE DAG visualization * Ref autocomplete includes models from seeds and snapshots * Prevent menus from getting cropped (git controls dropdown, file tree dropdown, build button, editor tab options) * Additional option to access the file menu by right-clicking on the files and folders in the file tree * Rename files by double-clicking on files in the file tree and the editor tabs * Right-clicking on file tabs has new options and will now open at your cursor instead of in the middle of the tab * The git branch name above **Version Control** links to the repo for specific git providers * Currently available for all [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instances using GitHub or GitLab providers #### Product refinements * Added an error modal for RPC parsing errors when users attempt to invoke dbt commands (preview, compile, or general dbt invocations) * Enabled syntax highlighting for Jinja expression and statement delimiters * Clarified and renamed the options under the **Build** button * Changed the term for RPC status from `Compiling` to `Parsing` to match dbt-core construct * Implemented a new File Tree component to improve render time by 60% * Disabled the Local Storage of File Tree to prevent users from running into max LocalStorage issue for large projects * Changed snapshot snippet template (`__snapshot`) to a select from source #### Bug fixes * You no longer have file contents carrying over when you switch to a different project that has the same file name * The preview max limit no longer allows you to override the maximum * You no longer encounter node statuses failing to update in the history drawer for those on version 1.4 core. (This is a partial fix that may be fully addressed by core version 1.5) * You can now use the **Copy File Name** option to copy up to the last dot, rather than the first dot * You can now use the `--no-partial-parse` flag to disable partial parsing in your dbt Cloud job commands.  * Previously, the [`--no-partial-parse` global config](https://docs.getdbt.com/reference/global-configs/parsing.md) was only available in dbt Core. For more information, refer to [partial parsing](https://docs.getdbt.com/reference/parsing.md#partial-parsing). #### January 2023 *  dbt Cloud IDE In the spirit of continuing to improve our [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) experience, the dbt Labs team worked on fixing bugs, increasing reliability, and adding new features ✨. Learn more about the [January changes](https://getdbt.slack.com/archives/C03SAHKKG2Z/p1675272600286119) and what's coming soon. #### New features * Improved syntax highlighting within the IDE for better Jinja-SQL combination (double quotes now show proper syntax highlight!) * Adjusted the routing URL for the IDE page and removed the `next` from the URL * Added a *new* easter egg within the IDE 🐶🦆 #### Product refinements * Performance improvements and reduced IDE slowness. The IDE should feel faster and snappier. * Reliability improvements – Improved error handling that previously put IDE in a bad state * Corrected the list of dropdown options for the Build button * Adjusted startup page duration * Added code snippets for `unique` and `not_null` tests for YAML files * Added code snippets for metrics based on environment dbt versions * Changed “commit and push” to “commit and sync” to better reflect the action * Improved error message when saving or renaming files to duplicate names #### Bug fixes * You no longer arbitrarily encounter an `RPC server got an unknown async ID` message * You can now see the build button dropdown, which had been hidden behind the placeholder DAG screen * You can now close toast notifications for command failure when the history drawer is open * You no longer encounter a `Something went wrong` message when previewing a model * You can now see repository status in the IDE, and the IDE finds the SSH folder * Scroll bars and download CSV no longer flicker within the preview pane --- ### 2024 dbt platform release notes dbt platform | Free dbt release notes for recent and historical changes. Release notes fall into one of the following categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by month for both multi-tenant and virtual private cloud (VPC)\* environments \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. #### December 2024 * **New**: Saved queries now support [tags](https://docs.getdbt.com/reference/resource-configs/tags.md), which allow you to categorize your resources and filter them. Add tags to your [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: dbt\_project.yml ```yml saved-queries: jaffle_shop: customer_order_metrics: +tags: order_metrics ``` * **New**: [Dimensions](https://docs.getdbt.com/reference/resource-configs/meta.md) now support the `meta` config property in [dbt Cloud **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and from dbt Core 1.9. You can add metadata to your dimensions to provide additional context and information about the dimension. Refer to [meta](https://docs.getdbt.com/reference/resource-configs/meta.md) for more information. * **New**: [Downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) are now generally available to dbt Enterprise plans. Downstream exposures integrate natively with Tableau (Power BI coming soon) and auto-generate downstream lineage in dbt Explorer for a richer experience. * **New**: The Semantic Layer supports Sigma as a [partner integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), available in Preview. Refer to [Sigma](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) for more information. * **New**: The Semantic Layer now supports Azure Single-tenant deployments. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information on how to get started. * **Fix**: Resolved intermittent issues in Single-tenant environments affecting Semantic Layer and query history. * **Fix**: [The dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) now respects the BigQuery [`execution_project` attribute](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#execution-project), including for exports. * **New**: [Model notifications](https://docs.getdbt.com/docs/deploy/model-notifications.md) are now generally available in dbt. These notifications alert model owners through email about any issues encountered by models and tests as soon as they occur while running a job. * **New**: You can now use your [Azure OpenAI key](https://docs.getdbt.com/docs/platform/account-integrations.md?ai-integration=azure#ai-integrations) (available in beta) to use dbt features like [](https://docs.getdbt.com/docs/platform/wizard-platform.md). Additionally, you can use your own [OpenAI API key](https://docs.getdbt.com/docs/platform/account-integrations.md?ai-integration=openai#ai-integrations) or use [dbt Labs-managed OpenAI](https://docs.getdbt.com/docs/platform/account-integrations.md?ai-integration=dbtlabs#ai-integrations) key. Refer to [AI integrations](https://docs.getdbt.com/docs/platform/account-integrations.md#ai-integrations) for more information. * **New**: The [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table. #### November 2024 * **Enhancement**: Data health signals in dbt Explorer are now available for Exposures, providing a quick view of data health while browsing resources. To view trust signal icons, go to dbt Explorer and click **Exposures** under the **Resource** tab. Refer to [Data health signals for resources](https://docs.getdbt.com/docs/explore/data-health-signals.md) for more info. * **Bug**: Identified and fixed an error with Semantic Layer queries that take longer than 10 minutes to complete. * **Fix**: Job environment variable overrides in credentials are now respected for Exports. Previously, they were ignored. * **Behavior change**: If you use a custom microbatch macro, set a [`require_batched_execution_for_custom_microbatch_strategy` behavior flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) in your `dbt_project.yml` to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md#how-microbatch-compares-to-other-incremental-strategies). * **Enhancement**: For users that have Advanced CI's [compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) feature enabled, you can optimize performance when running comparisons by using custom dbt syntax to customize deferral usage, exclude specific large models (or groups of models with tags), and more. Refer to [Compare changes custom commands](https://docs.getdbt.com/docs/deploy/job-commands.md#compare-changes-custom-commands) for examples of how to customize the comparison command. * **New**: SQL linting in CI jobs is now generally available in dbt. You can enable SQL linting in your CI jobs, using [SQLFluff](https://sqlfluff.com/), to automatically lint all SQL files in your project as a run step before your CI job builds. SQLFluff linting is available on [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and to dbt [Team or Enterprise](https://www.getdbt.com/pricing/) accounts. Refer to [SQL linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) for more information. * **New**: Use the [`dbt_valid_to_current`](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This feature is available in [the dbt Cloud **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) (formerly called `Versionless`) and dbt Core v1.9 and later. * **New**: Use the [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) configuration to specify "at what time did the row occur." This configuration is required for [Incremental microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) and can be added to ensure you're comparing overlapping times in [Advanced CI's compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md). Available in [the dbt Cloud **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) (formerly called `Versionless`) and dbt Core v1.9 and higher. * **Fix**: This update improves [Semantic Layer Tableau integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) making query parsing more reliable. Some key fixes include: * Error messages for unsupported joins between saved queries and ALL tables. * Improved handling of queries when multiple tables are selected in a data source. * Fixed a bug when an IN filter contained a lot of values. * Better error messaging for queries that can't be parsed correctly. * **Enhancement**: The Semantic Layer supports creating new credentials for users who don't have permissions to create service tokens. In the **Credentials & service tokens** side panel, the **+Add Service Token** option is unavailable for those users who don't have permission. Instead, the side panel displays a message indicating that the user doesn't have permission to create a service token and should contact their administration. Refer to [Set up Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more details. #### October 2024  Coalesce 2024 announcements Documentation for new features and functionality announced at Coalesce 2024: * Iceberg table support for [Snowflake](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#iceberg-table-format) * [Athena](https://docs.getdbt.com/reference/resource-configs/athena-configs.md) and [Teradata](https://docs.getdbt.com/reference/resource-configs/teradata-configs.md) adapter support in dbt Cloud * dbt Cloud now hosted on [Azure](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) * Get comfortable with [dbt Cloud Release Tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) that keep your project up-to-date, automatically — on a cadence appropriate for your team * Scalable [microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) * Advanced CI [features](https://docs.getdbt.com/docs/deploy/advanced-ci.md) * [Linting with CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) * dbt Assist is now [dbt Copilot](https://docs.getdbt.com/docs/platform/wizard-platform.md) * Developer blog on [Snowflake Feature Store and dbt: A bridge between data pipelines and ML](https://docs.getdbt.com/blog/snowflake-feature-store) * [Downstream exposures with Tableau](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) * Semantic Layer integration with [Excel desktop and M365](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) * [Data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) * [Semantic Layer and Cloud IDE integration](https://docs.getdbt.com/docs/build/metricflow-commands.md#metricflow-commands) * Query history in [Explorer](https://docs.getdbt.com/docs/explore/model-query-history.md#view-query-history-in-explorer) * Semantic Layer Metricflow improvements, including [improved granularity and custom calendar](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#custom-calendar) * [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) is now generally available - **Behavior change:** [Multi-factor authentication](https://docs.getdbt.com/docs/platform/manage-access/mfa.md) is now enforced on all users who log in with username and password credentials. - **Enhancement**: The dbt Semantic Layer JDBC now allows users to paginate `semantic_layer.metrics()` and `semantic_layer.dimensions()` for metrics and dimensions using `page_size` and `page_number` parameters. Refer to [Paginate metadata calls](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) for more information. - **Enhancement**: The dbt Semantic Layer JDBC now allows you to filter your metrics to include only those that contain a specific substring, using the `search` parameter. If no substring is provided, the query returns all metrics. Refer to [Fetch metrics by substring search](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) for more information. - **Fix**: The [Semantic Layer Excel integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) now correctly surfaces errors when a query fails to execute. Previously, it was not clear why a query failed to run. - **Fix:** Previously, POST requests to the Jobs API with invalid `cron` strings would return HTTP response status code 500s but would update the underlying entity. Now, POST requests to the Jobs API with invalid `cron` strings will result in status code 400s, without the underlying entity being updated. - **Fix:** Fixed an issue where the `Source` view page in dbt Explorer did not correctly display source freshness status if older than 30 days. - **Fix:** The UI now indicates when the description of a model is inherited from a catalog comment. - **Behavior change:** User API tokens have been deprecated. Update to [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) if you have any still in use. - **New**: The Cloud IDE supports signed commits for Git, available for Enterprise plans. You can sign your Git commits when pushing them to the repository to prevent impersonation and enhance security. Supported Git providers are GitHub and GitLab. Refer to [Git commit signing](https://docs.getdbt.com/docs/platform/studio-ide/git-commit-signing.md) for more information. - **New:** With Mesh, you can now enable bidirectional dependencies across your projects. Previously, dbt enforced dependencies to only go in one direction. dbt checks for cycles across projects and raises errors if any are detected. For details, refer to [Cycle detection](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#cycle-detection). There's also the [Intro to Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) guide to help you learn more best practices. - **New**: The [Semantic Layer Python software development kit](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) is now [generally available](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). It provides users with easy access to the Semantic Layer with Python and enables developers to interact with the Semantic Layer APIs to query metrics/dimensions in downstream tools. - **Enhancement**: You can now add a description to a singular data test. Use the [`description` property](https://docs.getdbt.com/reference/resource-properties/description.md) to document [singular data tests](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests). You can also use [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to capture your test description. The enhancement is available now in [the **Latest** release track in dbt Cloud](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), and it will be included in dbt Core v1.9. - **New**: Introducing the [microbatch incremental model strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) (beta), available now in [dbt Cloud Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and will soon be supported in dbt Core v1.9. The microbatch strategy allows for efficient, batch-based processing of large time-series datasets for improved performance and resiliency, especially when you're working with data that changes over time (like new records being added daily). To enable this feature in dbt Cloud, set the `DBT_EXPERIMENTAL_MICROBATCH` environment variable to `true` in your project. - **New**: The dbt Semantic Layer supports custom calendar configurations in MetricFlow, available in [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). Custom calendar configurations allow you to query data using non-standard time periods like `fiscal_year` or `retail_month`. Refer to [custom calendar](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#custom-calendar) to learn how to define these custom granularities in your MetricFlow timespine YAML configuration. - **New**: In the **Latest** release track in dbt, [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) have been updated to use YAML configuration files instead of SQL snapshot blocks. This new feature simplifies snapshot management and improves performance, and will soon be released in dbt v1 version 1.9. * Who does this affect? Users of the **Latest** release track in dbt can define snapshots using the new YAML specification. Users upgrading to **Latest** who have existing snapshot definitions can keep their existing configurations, or they can choose to migrate their snapshot definitions to YAML. * Users on older versions: No action is needed; existing snapshots will continue to work as before. However, we recommend upgrading to the **Latest** release track to take advantage of the new snapshot features. - **Behavior change:** Set [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. - **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. - **Enhancement**: In the **Latest** release track in dbt Cloud, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This will also be released in dbt Core 1.9. - **New**: In the **Latest** release track in dbt Cloud, the `snapshot_meta_column_names` config allows for customizing the snapshot metadata columns. This feature allows an organization to align these automatically-generated column names with their conventions, and will be included in the upcoming dbt Core 1.9 release. - **Enhancement**: the **Latest** release track in dbt Cloud infers a model's `primary_key` based on configured data tests and/or constraints within `manifest.json`. The inferred `primary_key` is visible in dbt Explorer and utilized by the dbt Cloud [compare changes](https://docs.getdbt.com/docs/deploy/run-visibility.md#compare-tab) feature. This will also be released in dbt Core 1.9. Read about the [order dbt infers columns can be used as primary key of a model](https://github.com/dbt-labs/dbt/blob/7940ad5c7858ff11ef100260a372f2f06a86e71f/core/dbt/contracts/graph/nodes.py#L534-L541). - **New:** dbt Explorer now includes trust signal icons, which is currently available as a [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). Trust signals offer a quick, at-a-glance view of data health when browsing your dbt models in dbt Explorer. These icons indicate whether a model is **Healthy**, **Caution**, **Degraded**, or **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Refer to [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) for more information. - **New:** Downstream exposures are now available in Preview in dbt. Downstream exposures helps users understand how their models are used in downstream analytics tools to inform investments and reduce incidents. It imports and auto-generates exposures based on Tableau dashboards, with user-defined curation. To learn more, refer to [Downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md). #### September 2024 * **Fix**: MetricFlow updated `get_and_expire` to replace the unsupported `GETEX` command with a `GET` and conditional expiration, ensuring compatibility with Azure Redis 6.0. * **Enhancement**: The [dbt Semantic Layer Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) now supports `TimeGranularity` custom grain for metrics. This feature allows you to define custom time granularities for metrics, such as `fiscal_year` or `retail_month`, to query data using non-standard time periods. * **New**: Use the AI engine to generate semantic model for your models, now available in beta. automatically generates documentation, tests, and now semantic models based on the data in your model, . To learn more, refer to [](https://docs.getdbt.com/docs/platform/wizard-platform.md). * **New**: Use the new recommended syntax for [defining `foreign_key` constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md) using `refs`, available in the **Latest** release track in dbt Cloud. This will soon be released in dbt Core v1.9. This new syntax will capture dependencies and works across different environments. * **Enhancement**: You can now run [Semantic Layer commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) commands in the [dbt Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). The supported commands are `dbt sl list`, `dbt sl list metrics`, `dbt sl list dimension-values`, `dbt sl list saved-queries`, `dbt sl query`, `dbt sl list dimensions`, `dbt sl list entities`, and `dbt sl validate`. * **New**: Microsoft Excel, a Semantic Layer integration, is now generally available. The integration allows you to connect to Microsoft Excel to query metrics and collaborate with your team. Available for [Excel Desktop](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationId=4132ecd1-425d-982d-efb4-de94ebc83f26) or [Excel Online](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationid=4132ecd1-425d-982d-efb4-de94ebc83f26\&isWac=True). For more information, refer to [Microsoft Excel](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md). * **New**: [Data health tile](https://docs.getdbt.com/docs/explore/data-tile.md) is now generally available in dbt Explorer. Data health tiles provide a quick at-a-glance view of your data quality, highlighting potential issues in your data. You can embed these tiles in your dashboards to quickly identify and address data quality issues in your dbt project. * **New**: dbt Explorer's Model query history feature is now in Preview for dbt Enterprise customers. Model query history allows you to view the count of consumption queries for a model based on the data warehouse's query logs. This feature provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products. To learn more, refer to [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md). * **Enhancement**: You can now use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. If you're using exports, job environment variable overrides aren't supported yet, but they will be soon. * **New:** There are two new [environment variable defaults](https://docs.getdbt.com/docs/build/environment-variables.md#dbt-cloud-context) — `DBT_CLOUD_ENVIRONMENT_NAME` and `DBT_CLOUD_ENVIRONMENT_TYPE`. * **New:** The [Amazon Athena warehouse connection](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-amazon-athena.md) is available as a public preview for dbt accounts that have upgraded to [the **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### August 2024 * **Fix:** Fixed an issue in [dbt Explorer](https://docs.getdbt.com/docs/explore/explore-projects.md) where navigating to a consumer project from a public node resulted in displaying a random public model rather than the original selection. * **New**: You can now configure metrics at granularities at finer time grains, such as hour, minute, or even by the second. This is particularly useful for more detailed analysis and for datasets where high-resolution time data is required, such as minute-by-minute event tracking. Refer to [dimensions](https://docs.getdbt.com/docs/build/dimensions.md) for more information about time granularity. * **Enhancement**: Microsoft Excel now supports [saved selections](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md#using-saved-selections) and [saved queries](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md#using-saved-queries). Use Saved selections to save your query selections within the Excel application. The application also clears stale data in [trailing rows](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md#other-settings) by default. To return your results and keep any previously selected data intact, un-select the **Clear trailing rows** option. * **Behavior change:** GitHub is no longer supported for OAuth login to dbt. Use a supported [SSO or OAuth provider](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) to securely manage access to your dbt account. #### July 2024 * **Behavior change:** `target_schema` is no longer a required configuration for [snapshots](https://docs.getdbt.com/docs/build/snapshots.md). You can now target different schemas for snapshots across development and deployment environments using the [schema config](https://docs.getdbt.com/reference/resource-configs/schema.md). * **New:** [Connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management) are now available under **Account settings** as a global setting. Previously, they were found under **Project settings**. This is being rolled out in phases over the coming weeks. * **New:** Admins can now assign [environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md) to groups for specific roles. * **New:** [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) for implementing [continuous deployment (CD)](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) workflows are now GA in dbt. Previously, you had to either set up a custom GitHub action or manually build the changes every time a pull request is merged. * **New**: The ability to lint your SQL files from the dbt CLI is now available. To learn more, refer to [Lint SQL files](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md#lint-sql-files). * **Behavior change:** dbt Cloud IDE automatically adds a `--limit 100` to preview queries to avoid slow and expensive queries during development. Recently, dbt Core changed how the `limit` is applied to ensure that `order by` clauses are consistently respected. Because of this, queries that already contain a limit clause might now cause errors in the IDE previews. To address this, dbt Labs plans to provide an option soon to disable the limit from being applied. Until then, dbt Labs recommends removing the (duplicate) limit clause from your queries during previews to avoid these IDE errors. * **Enhancement**: Introducing a revamped overview page for dbt Explorer, available in beta. It includes a new design and layout for the dbt Explorer homepage. The new layout provides a more intuitive experience for users to navigate their dbt projects, as well as a new **Latest updates** section to view the latest changes or issues related to project resources. To learn more, refer to [Overview page](https://docs.getdbt.com/docs/explore/explore-projects.md#overview-page). ###### dbt Semantic Layer * **New**: Introduced the [`dbt-sl-sdk` Python software development kit (SDK)](https://github.com/dbt-labs/semantic-layer-sdk-python) Python library, which provides you with easy access to the dbt Semantic Layer with Python. It allows developers to interact with the dbt Semantic Layer APIs and query metrics and dimensions in downstream tools. Refer to the [dbt Semantic Layer Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) for more information. * **New**: Introduced Semantic validations in CI pipelines. Automatically test your semantic nodes (metrics, semantic models, and saved queries) during code reviews by adding warehouse validation checks in your CI job using the `dbt sl validate` command. You can also validate modified semantic nodes to guarantee code changes made to dbt models don't break these metrics. Refer to [Semantic validations in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to learn about the additional commands and use cases. * **New**: We now expose the `meta` field within the [config property](https://docs.getdbt.com/reference/resource-configs/meta.md) for dbt Semantic Layer metrics in the [JDBC and GraphQL APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) under the `meta` field. * **New**: Added a new command in the dbt CLI called `export-all`, which allows you to export multiple or all of your saved queries. Previously, you had to explicitly specify the [list of saved queries](https://docs.getdbt.com/docs/build/metricflow-commands.md#list-saved-queries). * **Enhancement**: The Semantic Layer now offers more granular control by supporting multiple data platform credentials, which can represent different roles or service accounts. Available for dbt Enterprise plans, you can map credentials to service tokens for secure authentication. Refer to [Set up Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer) for more details. * **Fix**: Addressed a bug where unicode query filters (such as Chinese characters) were not working correctly in the Semantic Layer Tableau integration. * **Fix**: Resolved a bug with parsing certain private keys for BigQuery when running an export. * **Fix**: Addressed a bug that caused a "closed connection" error to be returned when querying or running an Export. * **Fix**: Resolved an issue in dbt v1 where, during partial parsing, all generated metrics in a file were incorrectly deleted instead of just those related to the changed semantic model. Now, only the metrics associated with the modified model are affected. #### June 2024 * **New:** Introduced new granularity support for cumulative metrics in MetricFlow. Granularity options for cumulative metrics are slightly different than granularity for other metric types. For other metrics, we use the `date_trunc` function to implement granularity. However, because cumulative metrics are non-additive (values can't be added up), we can't use the `date_trunc` function to change their time grain granularity. Instead, we use the `first()`, `last()`, and `avg()` aggregation functions to aggregate cumulative metrics over the requested period. By default, we take the first value of the period. You can change this behavior by using the `period_agg` parameter. For more information, refer to [Granularity options for cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md#granularity-options). ###### dbt Semantic Layer * **New:** Added support for Predicate pushdown SQL optimization in MetricFlow. We will now push down categorical dimension filters to the metric source table. Previously filters were applied after we selected from the metric source table. This change helps reduce full table scans on certain query engines. * **New:** Enabled `where` filters on dimensions (included in saved queries) to use the cache during query time. This means you can now dynamically filter your dashboards without losing the performance benefits of caching. Refer to [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md#result-caching) for more information. * **Enhancement:** In [Google Sheets](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md), we added information icons and descriptions to metrics and dimensions options in the Query Builder menu. Click on the **Info** icon button to view a description of the metric or dimension. Available in the following Query Builder menu sections: metric, group by, where, saved selections, and saved queries. * **Enhancement:** In [Google Sheets](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md), you can now apply granularity to all time dimensions, not just metric time. This update uses our [APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) to support granularity selection on any chosen time dimension. * **Enhancement**: MetricFlow time spine warnings now prompt users to configure missing or small-grain-time spines. An error message is displayed for multiple time spines per granularity. * **Enhancement**: Errors now display if no time spine is configured at the requested or smaller granularity. * **Enhancement:** Improved querying error message when no semantic layer credentials were set. * **Enhancement:** Querying grains for cumulative metrics now returns multiple granularity options (day, week, month, quarter, year) like all other metric types. Previously, you could only query one grain option for cumulative metrics. * **Fix:** Removed errors that prevented querying cumulative metrics with other granularities. * **Fix:** Fixed various Tableau errors when querying certain metrics or when using calculated fields. * **Fix:** In Tableau, we relaxed naming field expectations to better identify calculated fields. * **Fix:** Fixed an error when refreshing database metadata for columns that we can't convert to Arrow. These columns will now be skipped. This mainly affected Redshift users with custom types. * **Fix:** Fixed Private Link connections for Databricks. ###### Also available this month: * **Enhancement:** Updates to the UI when [creating merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) are now available. The updates include improvements to helper text, new deferral settings, and performance improvements. * **New**: The Semantic Layer now offers a seamless integration with Microsoft Excel, available in [preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). Build semantic layer queries and return data on metrics directly within Excel, through a custom menu. To learn more and install the add-on, check out [Microsoft Excel](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md). * **New:** [Job warnings](https://docs.getdbt.com/docs/deploy/job-notifications.md) are now GA. Previously, you could receive email or Slack alerts about your jobs when they succeeded, failed, or were canceled. Now with the new **Warns** option, you can also receive alerts when jobs have encountered warnings from tests or source freshness checks during their run. This gives you more flexibility on *when* to be notified. * **New:** A [preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) of the dbt Snowflake Native App is now available. With this app, you can access dbt Explorer, the **Ask dbt** chatbot, and orchestration observability features, extending your dbt experience into the Snowflake UI. #### May 2024 * **Enhancement:** We've now introduced a new **Prune branches** [Git button](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#prune-branches-modal) in the IDE. This button allows you to delete local branches that have been deleted from the remote repository, keeping your branch management tidy. Available in all regions now and will be released to single tenant accounts during the next release cycle. ###### dbt Cloud Launch Showcase event The following features are new or enhanced as part of our [dbt Launch Showcase](https://www.getdbt.com/resources/webinars/dbt-cloud-launch-showcase) event on May 14th, 2024: * **New:** [](https://docs.getdbt.com/docs/platform/wizard-platform.md)is a powerful AI engine helping you generate documentation, tests, and semantic models, saving you time as you deliver high-quality data. Available in private beta for a subset of dbt Enterprise users and in the IDE. [Register your interest](https://docs.google.com/forms/d/e/1FAIpQLScPjRGyrtgfmdY919Pf3kgqI5E95xxPXz-8JoVruw-L9jVtxg/viewform) to join the private beta. * **New:** The new low-code editor, now in private beta, enables less SQL-savvy analysts to create or edit dbt models through a visual, drag-and-drop experience inside of dbt. These models compile directly to SQL and are indistinguishable from other dbt models in your projects: they are version-controlled, can be accessed across projects in Mesh, and integrate with dbt Explorer and the Cloud IDE. [Register your interest](https://docs.google.com/forms/d/e/1FAIpQLScPjRGyrtgfmdY919Pf3kgqI5E95xxPXz-8JoVruw-L9jVtxg/viewform) to join the private beta. * **New:** [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) is now Generally Available (GA) to all users. The dbt CLI is a command-line interface that allows you to interact with dbt, use automatic deferral, leverage Mesh, and more! * **New:** [Unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) are now GA in dbt. Unit tests enable you to test your SQL model logic against a set of static inputs. *  New: Native support for Azure Synapse Analytics[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Native support in dbt Cloud for Azure Synapse Analytics is now available as a [preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md)! To learn more, refer to [Connect Azure Synapse Analytics](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-azure-synapse-analytics.md) and [Microsoft Azure Synapse DWH configurations](https://docs.getdbt.com/reference/resource-configs/azuresynapse-configs.md). Also, check out the [Quickstart for dbt Cloud and Azure Synapse Analytics](https://docs.getdbt.com/guides/azure-synapse-analytics.md?step=1). The guide walks you through: * Loading the Jaffle Shop sample data (provided by dbt Labs) into Azure Synapse Analytics. * Connecting dbt Cloud to Azure Synapse Analytics. * Turning a sample query into a model in your dbt project. A model in dbt is a SELECT statement. * Adding tests to your models. * Documenting your models. * Scheduling a job to run. * **New:** MetricFlow enables you to now add metrics as dimensions to your metric filters to create more complex metrics and gain more insights. Available for all Semantic Layer users. * **New:** [Staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) is now GA. Use staging environments to grant developers access to deployment workflows and tools while controlling access to production data. Available to all dbt users. * **New:** Oauth login support via [Databricks](https://docs.getdbt.com/docs/platform/manage-access/set-up-databricks-oauth.md) is now GA to Enterprise customers. *  New: GA of dbt Explorer's features dbt Explorer's current capabilities — including column-level lineage, model performance analysis, and project recommendations — are now Generally Available for dbt Cloud Enterprise and Teams plans. With Explorer, you can more easily navigate your dbt Cloud project – including models, sources, and their columns – to gain a better understanding of its latest production or staging state. To learn more about its features, check out: * [Explore projects](https://docs.getdbt.com/docs/explore/explore-projects.md) * [Explore multiple projects](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) * [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) * [Model performance](https://docs.getdbt.com/docs/explore/model-performance.md) * [Project recommendations](https://docs.getdbt.com/docs/explore/project-recommendations.md) * **New:** Native support for Microsoft Fabric in dbt is now GA. This feature is powered by the [dbt-fabric](https://github.com/Microsoft/dbt-fabric) adapter. To learn more, refer to [Connect Microsoft Fabric](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-microsoft-fabric.md) and [Microsoft Fabric DWH configurations](https://docs.getdbt.com/reference/resource-configs/fabric-configs.md). There's also a [quickstart guide](https://docs.getdbt.com/guides/microsoft-fabric.md?step=1) to help you get started. * **New:** Mesh is now GA to dbt Enterprise users. Mesh is a framework that helps organizations scale their teams and data assets effectively. It promotes governance best practices and breaks large projects into manageable sections. Get started with Mesh by reading the [Mesh quickstart guide](https://docs.getdbt.com/guides/mesh-qs.md?step=1). * **New:** The Semantic Layer [Tableau Desktop, Tableau Server](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md), and [Google Sheets integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) is now GA to dbt Team or Enterprise accounts. These first-class integrations allow you to query and unlock valuable insights from your data ecosystem. * **Enhancement:** As part of our ongoing commitment to improving the [IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#considerations), the filesystem now comes with improvements to speed up dbt development, such as introducing a Git repository limit of 10GB. ###### Also available this month: * **Update**: The [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) is now available for Azure single tenant and is accessible in all [deployment regions](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for both multi-tenant and single-tenant accounts. * **New**: The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) introduces [declarative caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md), allowing you to cache common queries to speed up performance and reduce query compute costs. Available for dbt Team or Enterprise accounts. *  New: Latest Release Track The **Latest** Release Track is now Generally Available (previously Public Preview). On this release track, you get automatic upgrades of dbt, including early access to the latest features, fixes, and performance improvements for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for *a second sandbox project* just to try out new features in development. To learn more about the new setting, refer to [Release Tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for details. [![Example of the Latest setting](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Example of the Latest setting")](#)Example of the Latest setting * **Behavior change:** Introduced the `require_resource_names_without_spaces` flag, opt-in and disabled by default. If set to `True`, dbt will raise an exception if it finds a resource name containing a space in your project or an installed package. This will become the default in a future version of dbt. Read [No spaces in resource names](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_spaces.md) for more information. #### April 2024 *  New: Merge jobs[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md), which is a new [job type](https://docs.getdbt.com/docs/deploy/jobs.md), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production. [![Example of creating a merge job](/img/docs/dbt-platform/using-dbt-platform/example-create-merge-job.png?v=2 "Example of creating a merge job")](#)Example of creating a merge job * **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and dbt Cloud release tracks). Read [Package override for built-in materialization](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_explicit_package_overrides_for_builtin_materializations.md) for more information. **Semantic Layer** * **New**: Use Saved selections to [save your query selections](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md#using-saved-selections) within the [Google Sheets application](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md). They can be made private or public and refresh upon loading. * **New**: Metrics are now displayed by their labels as `metric_name`. * **Enhancement**: [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) now supports the [`meta` option](https://docs.getdbt.com/reference/resource-configs/meta.md) under the [config](https://docs.getdbt.com/reference/resource-properties/config.md) property. Previously, we only supported the now deprecated `meta` tag. * **Enhancement**: In the Google Sheets application, we added [support](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md#using-saved-queries) to allow jumping off from or exploring MetricFlow-defined saved queries directly. * **Enhancement**: In the Google Sheets application, we added support to query dimensions without metrics. Previously, you needed a dimension. * **Enhancement**: In the Google Sheets application, we added support for time presets and complex time range filters such as "between", "after", and "before". * **Enhancement**: In the Google Sheets application, we added supported to automatically populate dimension values when you select a "where" filter, removing the need to manually type them. Previously, you needed to manually type the dimension values. * **Enhancement**: In the Google Sheets application, we added support to directly query entities, expanding the flexibility of data requests. * **Enhancement**: In the Google Sheets application, we added an option to exclude column headers, which is useful for populating templates with only the required data. * **Deprecation**: For the Tableau integration, the [`METRICS_AND_DIMENSIONS` data source](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md#using-the-integration) has been deprecated for all accounts not actively using it. We encourage users to transition to the "ALL" data source for future integrations. #### March 2024 * **New:** The Semantic Layer services now support using Privatelink for customers who have it enabled. * **New:** You can now develop against and test your Semantic Layer in the dbt CLI if your user credential uses SSO. * **Enhancement:** You can select entities to Group By, Filter By, and Order By. * **Fix:** `dbt parse` no longer shows an error when you use a list of filters (instead of just a string filter) on a metric. * **Fix:** `join_to_timespine` now properly gets applied to conversion metric input measures. * **Fix:** Fixed an issue where exports in Redshift were not always committing to the DWH, which also had the side-effect of leaving table locks open. * **Behavior change:** Introduced the `source_freshness_run_project_hooks` flag, opt-in and disabled by default. If set to `True`, dbt will include `on-run-*` project hooks in the `source freshness` command. This will become the default in a future version of dbt. Read [Project hooks with source freshness](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md) for more information. #### February 2024 * **New:** [Exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#define-exports) allow you to materialize a saved query as a table or view in your data platform. By using exports, you can unify metric definitions in your data platform and query them as you would any other table or view. * **New:** You can access a list of your [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) with the new list saved-queries command by adding `--show-exports` * **New:** The Semantic Layer and [Tableau Connector](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) now supports relative date filters in Tableau. *  New: Use exports to write saved queries You can now use the [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) feature with [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), allowing you to query reliable metrics and fast data reporting. Exports enhance the saved queries feature, allowing you to write commonly used queries directly within your data platform using dbt Cloud's job scheduler. By exposing tables of metrics and dimensions, exports enable you to integrate with additional tools that don't natively connect with the dbt Semantic Layer, such as PowerBI. Exports are available for dbt Cloud multi-tenant [Team or Enterprise](https://www.getdbt.com/pricing/) plans on dbt versions 1.7 or newer. Refer to the [exports blog](https://www.getdbt.com/blog/announcing-exports-for-the-dbt-semantic-layer) for more details. [![Add an environment variable to run exports in your production run.](/img/docs/dbt-platform/semantic-layer/deploy_exports.png?v=2 "Add an environment variable to run exports in your production run.")](#)Add an environment variable to run exports in your production run. *  New: Trigger on job completion teamenterprise Now available for dbt Cloud Team and Enterprise plans is the ability to trigger deploy jobs when other deploy jobs are complete. You can enable this feature [in the UI](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) with the **Run when another job finishes** option in the **Triggers** section of your job or with the [Create Job API endpoint](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job). When enabled, your job will run after the specified upstream job completes. You can configure which run status(es) will trigger your job. It can be just on `Success` or on all statuses. If you have dependencies between your dbt projects, this allows you to *natively* orchestrate your jobs within dbt Cloud — no need to set up a third-party tool. *  New: Latest Release Track[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") *Now available in the dbt version dropdown in dbt Cloud — starting with select customers, rolling out to wider availability through February and March.* On this release track, you get automatic upgrades of dbt, including early access to the latest features, fixes, and performance improvements for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for *a second sandbox project* just to try out new features in development. To learn more about the new setting, refer to [Release Tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for details. [![Example of the Latest setting](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Example of the Latest setting")](#)Example of the Latest setting *  New: Override dbt version with new User development settings You can now [override the dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#override-dbt-version) that's configured for the development environment within your project and use a different version — affecting only your user account. This lets you test new dbt features without impacting other people working on the same project. And when you're satisfied with the test results, you can safely upgrade the dbt version for your project(s). Use the **dbt version** dropdown to specify the version to override with. It's available on your project's credentials page in the **User development settings** section. For example: [![Example of overriding the dbt version on your user account](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-override-version.png?v=2 "Example of overriding the dbt version on your user account")](#)Example of overriding the dbt version on your user account *  Enhancement: Edit in primary git branch in IDE You can now edit, format, or lint files and execute dbt commands directly in your primary git branch in the [dbt Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). This enhancement is available across various repositories, including native integrations, imported git URLs, and managed repos. This enhancement is currently available to all dbt Cloud multi-tenant regions and will soon be available to single-tenant accounts. The primary branch of the connected git repo has traditionally been *read-only* in the IDE. This update changes the branch to *protected* and allows direct edits. When a commit is made, dbt Cloud will prompt you to create a new branch. dbt Cloud will pre-populate the new branch name with the GIT\_USERNAME-patch-#; however, you can edit the field with a custom branch name. Previously, the primary branch was displayed as read-only, but now the branch is displayed with a lock icon to identify it as protected: [![Previous read-only experience](/img/docs/dbt-platform/using-dbt-platform/read-only.png?v=2 "Previous read-only experience")](#)Previous read-only experience [![New protected experience](/img/docs/dbt-platform/using-dbt-platform/protected.png?v=2 "New protected experience")](#)New protected experience When you make a commit while on the primary branch, a modal window will open prompting you to create a new branch and enter a commit message: [![Create new branch window](/img/docs/dbt-platform/using-dbt-platform/create-new-branch.png?v=2 "Create new branch window")](#)Create new branch window * **Enhancement:** The Semantic Layer [Google Sheets integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) now exposes a note on the cell where the data was requested, indicating clearer data requests. The integration also now exposes a new **Time Range** option, which allows you to quickly select date ranges. * **Enhancement:** The [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) includes a `requiresMetricTime` parameter to better handle metrics that must be grouped by time. (Certain metrics defined in MetricFlow can't be looked at without a time dimension). * **Enhancement:** Enable querying metrics with offset and cumulative metrics with the time dimension name, instead of `metric_time`. [Issue #1000](https://github.com/dbt-labs/metricflow/issues/1000) * Enable querying `metric_time` without metrics. [Issue #928](https://github.com/dbt-labs/metricflow/issues/928) * **Enhancement:** Added support for consistent SQL query generation, which enables ID generation consistency between otherwise identical MF queries. Previously, the SQL generated by `MetricFlowEngine` was not completely consistent between identical queries. [Issue 1020](https://github.com/dbt-labs/metricflow/issues/1020) * **Fix:** The Tableau Connector returns a date filter when filtering by dates. Previously it was erroneously returning a timestamp filter. * **Fix:** MetricFlow now validates if there are `metrics`, `group by`, or `saved_query` items in each query. Previously, there was no validation. [Issue 1002](https://github.com/dbt-labs/metricflow/issues/1002) * **Fix:** Measures using `join_to_timespine` in MetricFlow now have filters applied correctly after time spine join. * **Fix:** Querying multiple granularities with offset metrics: * If you query a time offset metric with multiple instances of `metric_time`/`agg_time_dimension`, only one of the instances will be offset. All of them should be. * If you query a time offset metric with one instance of `metric_time`/`agg_time_dimension` but filter by a different one, the query will fail. * **Fix:** MetricFlow prioritizes a candidate join type over the default type when evaluating nodes to join. For example, the default join type for distinct values queries is `FULL OUTER JOIN`, however, time spine joins require `CROSS JOIN`, which is more appropriate. * **Fix:** Fixed a bug that previously caused errors when entities were referenced in `where` filters. #### January 2024 *  January docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 10 new community contributors to docs.getdbt.com 🙏 What a busy start to the year! We merged 110 PRs in January. Here's how we improved the [docs.getdbt.com](http://docs.getdbt.com/) experience: * Added new hover behavior for images * Added new expandables for FAQs * Pruned outdated notices and snippets as part of the docs site maintenance January saw some great new content: * New [dbt Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md) page * Beta launch of [Explorer’s column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) feature * Developer blog posts: * [More time coding, less time waiting: Mastering defer in dbt](https://docs.getdbt.com/blog/defer-to-prod) * [Deprecation of dbt Server](https://docs.getdbt.com/blog/deprecation-of-dbt-server) * From the community: [Serverless, free-tier data stack with dlt + dbt core](https://docs.getdbt.com/blog/serverless-dlt-dbt-stack) * The Extrica team added docs for the [dbt-extrica community adapter](https://docs.getdbt.com/docs/local/connect-data-platform/extrica-setup.md) * Semantic Layer: New [conversion metrics docs](https://docs.getdbt.com/docs/build/conversion.md) and added the parameter `fill_nulls_with` to all metric types (launched the week of January 12, 2024) * New [dbt environment command](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) and its flags for the dbt CLI January also saw some refreshed content, either aligning with new product features or requests from the community: * Native support for [partial parsing in dbt Cloud](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing) * Updated guidance on using dots or underscores in the [Best practice guide for models](https://docs.getdbt.com/best-practices/how-we-style/1-how-we-style-our-dbt-models.md) * Updated [PrivateLink for VCS docs](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-self-hosted.md) * Added a new `job_runner` role in our [Enterprise project role permissions docs](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#project-role-permissions) * Added saved queries to [Metricflow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md#list-saved-queries) * Removed [as\_text docs](https://github.com/dbt-labs/docs.getdbt.com/pull/4726) that were wildly outdated * **New:** New metric type that allows you to measure conversion events. For example, users who viewed a web page and then filled out a form. For more details, refer to [Conversion metrics](https://docs.getdbt.com/docs/build/conversion.md). * **New:** Instead of specifying the fully qualified dimension name (for example, `order__user__country`) in the group by or filter expression, you now only need to provide the primary entity and dimensions name, like `user__county`. * **New:** You can now query the [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) you've defined in the Semantic Layer using [Tableau](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md), [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md), [JDBC API](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md), and the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). *  New: Native support for partial parsing By default, dbt parses all the files in your project at the beginning of every dbt invocation. Depending on the size of your project, this operation can take a long time to complete. With the new partial parsing feature in dbt Cloud, you can reduce the time it takes for dbt to parse your project. When enabled, dbt Cloud parses only the changed files in your project instead of parsing all the project files. As a result, your dbt invocations will take less time to run. To learn more, refer to [Partial parsing](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing). [![Example of the Partial parsing option](/img/docs/deploy/account-settings-partial-parsing.png?v=2 "Example of the Partial parsing option")](#)Example of the Partial parsing option * **Enhancement:** The YAML spec parameter `label` is now available for Semantic Layer metrics in [JDBC and GraphQL APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). This means you can conveniently use `label` as a display name for your metrics when exposing them. * **Enhancement:** Added support for `create_metric: true` for a measure, which is a shorthand to quickly create metrics. This is useful in cases when metrics are only used to build other metrics. * **Enhancement:** Added support for Tableau parameter filters. You can use the [Tableau connector](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) to create and use parameters with your Semantic Layer data. * **Enhancement:** Added support to expose `expr` and `agg` for [Measures](https://docs.getdbt.com/docs/build/measures.md) in the [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md). * **Enhancement:** You have improved error messages in the command line interface when querying a dimension that is not reachable for a given metric. * **Enhancement:** You can now query entities using our Tableau integration (similar to querying dimensions). * **Enhancement:** A new data source is available in our Tableau integration called "ALL", which contains all semantic objects defined. This has the same information as "METRICS\_AND\_DIMENSIONS". In the future, we will deprecate "METRICS\_AND\_DIMENSIONS" in favor of "ALL" for clarity. * **Fix:** Support for numeric types with precision greater than 38 (like `BIGDECIMAL`) in BigQuery is now available. Previously, it was unsupported so would return an error. * **Fix:** In some instances, large numeric dimensions were being interpreted by Tableau in scientific notation, making them hard to use. These should now be displayed as numbers as expected. * **Fix:** We now preserve dimension values accurately instead of being inadvertently converted into strings. * **Fix:** Resolved issues with naming collisions in queries involving multiple derived metrics using the same metric input. Previously, this could cause a naming collision. Input metrics are now deduplicated, ensuring each is referenced only once. * **Fix:** Resolved warnings related to using two duplicate input measures in a derived metric. Previously, this would trigger a warning. Input measures are now deduplicated, enhancing query processing and clarity. * **Fix:** Resolved an error where referencing an entity in a filter using the object syntax would fail. For example, `{{Entity('entity_name')}}` would fail to resolve. --- ### 2025 dbt platform release notes dbt platform | Free dbt release notes for recent and historical changes. Release notes fall into one of the following categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by month for both multi-tenant and virtual private cloud (VPC) environments. #### December 2025 * **New**: [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) is now the default experience for Catalog, providing a unified search experience that lets you find dbt resources across all your projects, as well as non-dbt resources in Snowflake. Global navigation is now generally available to all users. You can access Catalog by clicking **Catalog** in the top-level navigation. * **Enhancement**: dbt SSO slugs are now system-generated during SSO setup and aren't customizable. SSO slug configurations currently in use will remain valid; they will be read-only and cannot be changed. If you delete your existing SSO configuration and create a new one, you'll be provided with a new system-generated SSO slug. This change enhances security and prevents accounts from setting slugs that "impersonate" other organizations. * **Enhancement**: For users in the default region (`US1`) that previously created a dbt account in the past, the dbt VS Code extension now supports registering with OAuth . This makes it easier to register the extension for users who may have forgotten their password or are locked out of their account. For more information, see [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md). * **New and enhancements:** The dbt [Studio IDE user interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) has been enhanced to bring more powerful development features to your fingertips: * A newly designed toolbar that groups all of your action and project insight tabs for easy access. * A dedicated inline **Commands** tab for history and logs. * When you upgrade your development environment to dbt v2, the environment includes a new **Problems** tab that gives you live error detection on issues that could block your project from running successfully. #### November 2025 * **Behavior change**: [dbt Copilot](https://docs.getdbt.com/docs/platform/wizard-platform.md) now requires all input files to use UTF-8 encoding. Files that use other encodings will return an error. If you're working with legacy files that use a different encoding, convert them to UTF-8 before using Copilot. * **Enhancement**: dbt Copilot now has improved reliability when working with OpenAI. This includes longer timeouts, better retry behavior, and improved handling of reasoning messages for long code generations, resulting in fewer failures and more successful completions. * **New**: The Snowflake adapter now supports basic table materialization on Iceberg tables registered in a Glue catalog through a [catalog-linked database](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database#label-catalog-linked-db-create). For more information, see [Glue Data Catalog](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md#external-catalogs). * **New**: You can use the `platform_detection_timeout_seconds` parameter to control how long the Snowflake connector waits when detecting the cloud platform where the connection is being made. For more information, see [Snowflake setup](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md#platform_detection_timeout_seconds). * **New**: The `cluster_by` configuration is supported in dynamic tables. For more information, see [Dynamic table clustering](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-table-clustering). * **New**: When jobs exceed their configured timeout, the BigQuery adapter sends a cancellation request to the BigQuery job. For more information, see [Connect BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md#job-creation-timeout-seconds). #### October 2025 * **Enhancement**: dbt enforces cumulative log size limits on run endpoints. If logs exceed this limit, dbt omits them and displays a banner. For more information, see [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md#log-size-limits). * **New**: The [docs.getdbt.com](http://docs.getdbt.com/) documentation site has introduced an LLM Context menu on all product documentation and guide pages. This menu provides users with quick options to interact with the current page using LLMs. You can now: * Copy the page as raw Markdown — This makes it easier to reference or reuse documentation content. * Open the page directly in ChatGPT or Claude — This redirects you to a chat with the LLM and automatically loads a message asking it to read the page, helping you start a conversation with context from the page. [![LLM Context menu on documentation pages](/img/llm-menu.png?v=2 "LLM Context menu on documentation pages")](#)LLM Context menu on documentation pages * **Enhancement**: The CodeGenCodeLen feature has been re-introduced to the Studio IDE. This feature was [temporarily](#pre-coalesce) removed in the previous release due to compatibility issues. ##### Coalesce 2025 announcements The following features are new or enhanced as part of [dbt's Coalesce analytics engineering conference](https://coalesce.getdbt.com/event/21662b38-2c17-4c10-9dd7-964fd652ab44/summary) from October 13-16, 2025: * **New**: The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) is now generally available (GA). For more information on the dbt MCP server and dbt Agents, refer to the [Announcing dbt Agents and the remote dbt MCP Server: Trusted AI for analytics](https://www.getdbt.com/blog/dbt-agents-remote-dbt-mcp-server-trusted-ai-for-analytics) blog post. * **Private preview**: The [dbt platform (powered by Fusion)](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) is now in private preview. If you have any questions, please reach out to your account manager. * [About data platform connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) lists all available dbt platform connections on Fusion and the supported authentication methods per connection. * **New**: Fusion‑specific configuration is now available for BigQuery, Databricks, Redshift, and Snowflake. For more information, see [Connect Fusion to your data platform](https://docs.getdbt.com/docs/local/profiles.yml.md). * **Alpha**: The `dbt-salesforce` adapter is available via dbt v2 CLI. Note that this connection is in the Alpha product stage and is not production-ready. For more information, see [Salesforce Data Cloud setup](https://docs.getdbt.com/docs/local/connect-data-platform/salesforce-data-cloud-setup.md). * **Private preview**: [State-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) is now in private preview! * **New**: You can now [enable state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-setup.md) by selecting **Enable Fusion cost optimization features** in your job settings. Previously, you had to disable **Force node selection** to enable state-aware orchestration. * **Private beta**: The [Efficient Testing feature](https://docs.getdbt.com/docs/deploy/state-aware-about.md#efficient-testing-in-state-aware-orchestration) is now available in private beta. This feature reduces warehouse costs by avoiding redundant data tests and combining multiple tests in a single query. * **New**: To improve visibility into state‑aware orchestration and provide better control when you need to reset cached state, the following [UI enhancements](https://docs.getdbt.com/docs/deploy/state-aware-interface.md) are introduced: * **Models built and reused chart** on your **Account home** * New charts in the **Overview** section of your job that display **Recent runs**, **Total run duration**, **Models built**, and **Models reused** * A new structure to view logs grouped by models, with a **Reused** tab to quickly find reused models * **Reused** tag in **Latest status** lineage lens to see reused models in your DAG * **Clear cache** button on the **Environments** page to reset cached state when needed * **New**: [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) is now generally available (GA)! * **Private beta**: The [Analyst agent](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#dbt-copilot) is now available in dbt Insights. The Analyst agent is a conversational AI feature where you can ask natural language prompts and receive analysis in real-time. For more information, see [Analyze data with the Analyst agent](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md). * **Beta**: The [Semantic Layer querying](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#semantic-layer-querying) within dbt Insights is now available in beta. With this feature, you can build SQL queries against the Semantic Layer without writing SQL code. It guides you in creating queries based on available metrics, dimensions, and entities. * **Enhancement**: In [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md), projects upgraded to the [dbt v2](https://docs.getdbt.com/docs/introduction.md) get [Language Server Protocol (LSP) features](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#lsp-features) and their compilation running on dbt v2. * **New**: [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) is now developed and maintained as part of the [Open Semantic Interchange (OSI)](https://www.snowflake.com/en/blog/open-semantic-interchange-ai-standard/) initiative, and is distributed under the [Apache 2.0 license](https://github.com/dbt-labs/metricflow/blob/main/LICENSE). For more information, see the blog post about [Open sourcing MetricFlow](https://www.getdbt.com/blog/open-source-metricflow-governed-metrics). ##### Pre-Coalesce * **Behavior change**: dbt platform [access URLs](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for accounts in the US multi-tenant (US MT) region are transitioning from `cloud.getdbt.com` to dedicated domains on `dbt.com` (for example, `us1.dbt.com`). Users will be automatically redirected, which means no action is required. EMEA and APAC MT accounts are not impacted by this change and will be updated by the end of November 2025. Organizations that use network allow-listing should add `YOUR_ACCESS_URL.dbt.com` to their allow list (for example, if your access URL is `ab123.us1.dbt.com`, add the entire domain `ab123.us1.dbt.com` to your allow list). All OAuth, Git, and public API integrations will continue to work with the previous domain. View the updated access URL in dbt platform's **Account settings** page. For questions, contact <support@getdbt.com>. * **Enhancement**: * **dbt v2 MCP tools** — Added dbt v2 tools that support `compile_sql` and `get_column_lineage` (Fusion-exclusive) for both [Remote](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#fusion-tools-remote) and [Local](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#fusion-tools-local) usage. Remote dbt v2 tools defer to your prod environment by default (set with `x-dbt-prod-environment-id`); you can disable deferral with `x-dbt-fusion-disable-defer=true`. Refer to [set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md) for more info. * **Self-hosted MCP server OAuth** — You can now authenticate the self-hosted dbt MCP server to the dbt platform with OAuth (supported docs for [Claude](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md), [Cursor](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md), and [VS Code](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md)), reducing local secret management and standardizing setup. Refer to [dbt platform authentication](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#dbt-platform-authentication) for more information. * **Behavior change**: The CodeGenCodeLens feature for creating models from your sources with a click of a button has been temporarily removed from the Studio IDE due to compatibility issues. We plan to reintroduce this feature in the near future for both the IDE and the VS Code extension. #### September 2025 * **Fix**: Improved how [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) handles [offset metrics](https://docs.getdbt.com/docs/build/derived.md) for more accurate results when querying time-based data. MetricFlow now joins data *after* aggregation when the query grain matches the offset grain. Previously, when querying offset metrics, the offset join was applied *before* aggregation, which could exclude some values from the total time period. #### August 2025 * **Fix**: Resolved a bug that caused [saved query](https://docs.getdbt.com/docs/build/saved-queries.md) exports to fail during `dbt build` with `Unable to get saved_query` errors. * **New**: The Semantic Layer GraphQL API now has a [`queryRecords`](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#query-records) endpoint. With this endpoint, you can view the query history both for Insights and Semantic Layer queries. * **Fix**: Resolved a bug that caused Semantic Layer queries with a trailing whitespace to produce an error. This issue mostly affected [Push.ai](https://docs.push.ai/data-sources/semantic-layers/dbt) users and is fixed now. * **New**: You can now use [personal access tokens (PATs)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to authenticate in the Semantic Layer. This enables user-level authentication and reduces the need for sharing tokens between users. When you authenticate using PATs, queries are run using your personal user credentials. For more information, see [Set up the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). #### July 2025 * **New**: The [Tableau Cloud](https://www.tableau.com/products/cloud-bi) integration with Semantic Layer is now available. For more information, see [Tableau](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md). * **Preview**: The [Semantic Layer Power BI integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) is now available in Preview. * **Enhancement:** You can now use `limit` and `order_by` parameters when creating [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md). * **Enhancement:** Users assigned IT [licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) can now edit and manage [global connections settings](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management). * **New**: Paginated [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) endpoints for metadata queries in Semantic Layer are now available. This improves integration load times for large manifests. For more information, see [Metadata calls](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#metadata-calls). #### June 2025 * **New**: [System for Cross-Domain Identity Management](https://docs.getdbt.com/docs/platform/manage-access/scim.md#scim-configuration-for-entra-id) (SCIM) through Microsoft Entra ID is now GA. Also available on legacy Enterprise plans. * **Enhancement:** You can now set the [compilation environment](https://docs.getdbt.com/docs/explore/access-dbt-insights.md#set-jinja-environment) to control how Jinja functions are rendered in dbt Insights. * **Beta**: The dbt Fusion engine supports the BigQuery adapter in beta. * **New:** You can now view the history of settings changes for [projects](https://docs.getdbt.com/docs/platform/account-settings.md), [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md), and [jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md). * **New:** Added support for the latest version of BigQuery credentials in Semantic Layer and MetricFlow. * **New:** Snowflake External OAuth is now supported for Semantic Layer queries. Snowflake connections that use External OAuth for user credentials can now emit queries for Insights, dbt platform CLI, and Studio IDE through the Semantic Layer Gateway. This enables secure, identity-aware access via providers like Okta or Microsoft Entra ID. * **New:** You can now [download your managed Git repo](https://docs.getdbt.com/docs/platform/git/managed-repository.md#download-managed-repository) from the dbt platform. * **New**: The Semantic Layer now supports Trino as a data platform. For more details, see [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * **New**: The dbt Fusion engine supports Databricks in beta. * **Enhancement**: Group owners can now specify multiple email addresses for model-level notifications, enabling broader team alerts. Previously, only a single email address was supported. Check out the [Configure groups](https://docs.getdbt.com/docs/deploy/model-notifications.md#configure-groups) section to learn more. * **New**: The Semantic Layer GraphQL API now has a [`List a saved query`](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#list-a-saved-query) endpoint. #### May 2025 ##### 2025 dbt Launch Showcase The following features are new or enhanced as part of our [dbt Launch Showcase](https://www.getdbt.com/resources/webinars/2025-dbt-cloud-launch-showcase) on May 28th, 2025: * **New**: The dbt Fusion engine is the brand new dbt engine re-written from the ground up to provide incredible speed, cost-savings tools, and comprehensive SQL language tools. The dbt Fusion engine is now available in beta for Snowflake users. * Read more [about Fusion](https://docs.getdbt.com/docs/introduction.md). * Understand what actions you need to take to get your projects Fusion-ready with the [upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). * Begin testing today with the [quickstart guide](https://docs.getdbt.com/guides/dbt.md). * Know [where we're headed with the dbt Fusion engine](https://getdbt.com/blog/where-we-re-headed-with-the-dbt-fusion-engine). * **New**: The dbt VS Code extension is a powerful new tool that brings the speed and productivity of the dbt Fusion engine into your Visual Studio Code editor. This is a free download that will forever change your dbt development workflows. The dbt VS Code extension is now available as beta [alongside Fusion](https://getdbt.com/blog/get-to-know-the-new-dbt-fusion-engine-and-vs-code-extension). Check out the [installation instructions](https://docs.getdbt.com/docs/install-dbt-extension.md) and read more [about the features](https://docs.getdbt.com/docs/about-dbt-extension.md) to get started enhancing your dbt workflows today! * **New**: dbt Explorer is now Catalog! Learn more about the change [here](https://getdbt.com/blog/updated-names-for-dbt-platform-and-features). * dbt's Catalog, global navigation provides a search experience that lets you find dbt resources across all your projects, as well as non-dbt resources in Snowflake. * External metadata ingestion allows you to connect directly to your data warehouse, giving you visibility into tables, views, and other resources that aren't defined in dbt. * **New**: [dbt Canvas is now generally available](https://getdbt.com/blog/dbt-canvas-is-ga) (GA). Canvas is the intuitive visual editing tool that enables anyone to create dbt models with an easy to understand drag-and-drop interface. Read more [about Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to begin empowering your teams to build more, faster! * **New**: [State-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) is now in beta! Every time a new job in Fusion runs, state-aware orchestration automatically determines which models to build by detecting changes in code or data. * **New**: With Hybrid Projects, your organization can adopt complementary dbt Core and dbt Cloud workflows and seamlessly integrate these workflows by automatically uploading dbt Core artifacts into dbt Cloud. [Hybrid Projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) is now available as a preview to [dbt Enterprise accounts](https://www.getdbt.com/pricing). * **New**: [System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim.md) through Okta is now GA. * **New**: dbt now acts as a [Model Context Protocol](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) (MCP) server, allowing seamless integration of AI tools with data warehouses through a standardized framework. * **New**: The [quickstart guide for data analysts](https://docs.getdbt.com/guides/analyze-your-data.md) is now available. With dbt, data analysts can use built-in, AI-powered tools to build governed data models, explore how they’re built, and run their own analysis. * **New**: You can view your [usage metering and limiting in dbt Copilot](https://docs.getdbt.com/docs/platform/billing/dbt-ai-usage.md) on the billing page of your dbt Cloud account. * **New**: You can use Copilot to create a `dbt-styleguide.md` for dbt projects. The generated style guide template includes SQL style guidelines, model organization and naming conventions, model configurations and testing practices, and recommendations to enforce style rules. For more information, see [dbt Wizard skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md). * **New**: Copilot chat is an interactive interface within the Studio IDE where you can generate SQL code from natural language prompts and ask analytics-related questions. It integrates contextual understanding of your dbt project and assists in streamlining SQL development. For more information, see [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). * **New**: Leverage dbt Copilot to generate SQL queries in [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) from natural language prompts, enabling efficient data exploration within a context-aware interface. * **New**: The dbt platform Cost management dashboard was available as a preview for Snowflake users on Enterprise and Enterprise Plus plans. On November 25, 2025, we retired the cost management dashboard to focus on building a more scalable and integrated cost-insights experience, expected in early 2026. * **New**: Apache Iceberg catalog integration support is now available on Snowflake and BigQuery! This is essential to making your dbt Mesh interoperable across platforms, built on Iceberg. Read more about [Iceberg](https://docs.getdbt.com/docs/build/iceberg/apache-iceberg-support.md) to begin creating Iceberg tables. * **Update**: Product renaming and other changes. For more information, refer to [Updated names for dbt platform and features](https://getdbt.com/blog/updated-names-for-dbt-platform-and-features).  Product names key * Canvas (previously Visual Editor) * Catalog (previously Explorer) * Copilot * Cost Management * dbt Fusion engine * Insights * Mesh * Orchestrator * Studio IDE (previously Cloud IDE) * Semantic Layer * Pricing plan changes. For more information, refer to [One dbt](https://www.getdbt.com/product/one-dbt). #### April 2025 * **Enhancement**: The [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) now supports lazy loading for large fields for `dimensions`, `entities`, and `measures` on `Metric` objects. For more information, see [Lazy loading for large fields](https://docs.getdbt.com/docs/dbt-apis/sl-python.md#lazy-loading-for-large-fields). * **Enhancement**: The Semantic Layer now supports SSH tunneling for [Postgres](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md#connecting-using-an-ssh-tunnel) or [Redshift](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md#connecting-using-an-ssh-tunnel) connections. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information. * **Behavior change**: Users assigned the [`job admin` permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-admin) now have access to set up integrations for projects, including the [Tableau](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) integration to populate downstream exposures. #### March 2025 * **Behavior change**: As of March 31st, 2025, dbt v1 versions 1.0, 1.1, and 1.2 have been deprecated from dbt. They are no longer available to select as versions for dbt projects. Workloads currently on these versions will be automatically upgraded to v1.3, which may cause new failures. * **Enhancement**: [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) users on single-tenant configurations no longer need to contact their account representative to enable this feature. Setup is now self-service and available across all tenant configurations. * **New**: The Semantic Layer now supports Postgres as a data platform. For more details on how to set up the Semantic Layer for Postgres, see [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * **New**: New [environment variable default](https://docs.getdbt.com/docs/build/environment-variables.md#dbt-cloud-context) `DBT_CLOUD_INVOCATION_CONTEXT`. * **Enhancement**: Users assigned [read-only licenses](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#licenses) are now able to view the [Deploy](https://docs.getdbt.com/docs/deploy/deployments.md) section of their dbt account and click into the individual sections but not edit or otherwise make any changes. ###### dbt Developer day The following features are new or enhanced as part of our [dbt Developer day](https://www.getdbt.com/resources/webinars/dbt-developer-day) on March 19th and 20th, 2025: * **New**: The [`--sample` flag](https://docs.getdbt.com/docs/build/sample-flag.md), now available for the `run` and `build` commands, helps reduce build times and warehouse costs by running dbt in sample mode. It generates filtered refs and sources using time-based sampling, allowing developers to validate outputs without building entire models. * **New**: dbt Wizard, an AI-powered assistant, is now generally available in the Cloud IDE for all dbt Enterprise accounts. Check out [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) for more information. ###### Also available this month * **New**: Bringing your own [Azure OpenAI key](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#manage-ai-features) for [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is now generally available. Your organization can configure dbt Wizard to use your own Azure OpenAI keys, giving you more control over data governance and billing. * **New**: The Semantic Layer supports Power BI as a [partner integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), available in private beta. To join the private beta, please reach out to your account representative. Check out the [Power BI](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) integration for more information. * **New**: [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) are Generally Available. Depending on their plan, customers may select among the Latest, Compatible, or Extended tracks to manage the update cadences for development and deployment environments. * **New:** The dbt-native integration with Azure DevOps now supports [Entra ID service principals](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md). Unlike a services user, which represents a real user object in Entra ID, the service principal is a secure identity associated with your dbt app to access resources in Azure unattended. Please [migrate your service user](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md#migrate-to-service-principal) to a service principal for Azure DevOps as soon as possible. #### February 2025 * **Enhancement**: The [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) added a new timeout parameter to Semantic Layer client and to underlying GraphQL clients to specify timeouts. Set a timeout number or use the `total_timeout` parameter in the global `TimeoutOptions` to control connect, execute, and close timeouts granularly. `ExponentialBackoff.timeout_ms` is now deprecated. * **New**: The [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) integration for Git now supports [Entra service principal apps](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md) on dbt Enterprise accounts. Microsoft is enforcing MFA across user accounts, including service users, which will impact existing app integrations. This is a phased rollout, and dbt Labs recommends [migrating to a service principal](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md#migrate-to-service-principal) on existing integrations once the option becomes available in your account. * **New**: Added the `dbt invocation` command to the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). This command allows you to view and manage active invocations, which are long-running sessions in the dbt CLI. For more information, see [dbt invocation](https://docs.getdbt.com/reference/commands/invocation.md?version=2.0). * **New**: Users can now switch themes directly from the user menu, available [in Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). We have added support for **Light mode** (default), **Dark mode**, and automatic theme switching based on system preferences. The selected theme is stored in the user profile and will follow users across all devices. * Dark mode is currently available on the Developer plan and will be available for all [plans](https://www.getdbt.com/pricing) in the future. We’ll be rolling it out gradually, so stay tuned for updates. For more information, refer to [Change your dbt theme](https://docs.getdbt.com/docs/platform/about-platform/change-your-dbt-theme.md). * **Fix**: Semantic Layer errors in the Cloud IDE are now displayed with proper formatting, fixing an issue where newlines appeared broken or difficult to read. This fix ensures error messages are more user-friendly and easier to parse. * **Fix**: Fixed an issue where [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) with no [exports](https://docs.getdbt.com/docs/build/saved-queries.md#configure-exports) would fail with an `UnboundLocalError`. Previously, attempting to process a saved query without any exports would cause an error due to an undefined relation variable. Exports are optional, and this fix ensures saved queries without exports don't fail. * **New**: You can now query metric alias in Semantic Layer [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) and [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) APIs. * For the JDBC API, refer to [Query metric alias](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#query-metric-alias) for more information. * For the GraphQL API, refer to [Query metric alias](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#query-metric-alias) for more information. * **Enhancement**: Added support to automatically refresh access tokens when Snowflake's SSO connection expires. Previously, users would get the following error: `Connection is not available, request timed out after 30000ms` and would have to wait 10 minutes to try again. * **Enhancement**: The [`dbt_version` format](https://docs.getdbt.com/reference/commands/version.md#versioning) in dbt Cloud now better aligns with [semantic versioning rules](https://semver.org/). Leading zeroes have been removed from the month and day (`YYYY.M.D+<suffix>`). For example: * New format: `2024.10.8+996c6a8` * Previous format: `2024.10.08+996c6a8` --- ### About AI and dbt Analytics engineering requires more than code generation — it requires understanding your data, how it connects, and what breaks when something changes. dbt brings purpose-built AI to that workflow, grounded in your project's lineage, tests, contracts, and metric definitions. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. #### Explore dbt Wizard and AI [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/platform/wizard-overview.md) ###### [About dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) [Purpose-built AI for governed data development in dbt platform, Desktop, and CLI.](https://docs.getdbt.com/docs/platform/wizard-overview.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) ###### [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) [Install dbt Wizard locally and start a terminal session.](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/platform/wizard-platform.md) ###### [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) [AI is on by default in your dbt platform account. Use dbt Wizard for governed dbt data development and dbt Copilot for inline AI assistance.](https://docs.getdbt.com/docs/platform/wizard-platform.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) ###### [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) [A native app for data authors and analysts. View rich data visualizations, data diffs, lineage, and more to get to your results faster than ever before.](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) ###### [Move to dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) [Move from Claude Code, Cursor, or another AI agent to dbt Wizard CLI with native project context.](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) #### Explore dbt Copilot [![](/img/icons/dbt-copilot.svg)](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) ###### [dbt Copilot](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) [dbt's inline AI assistance for single-click generation of SQL, docs, tests, and semantic models.](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) [![](/img/icons/dbt-copilot.svg)](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md) ###### [dbt Copilot in Insights](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md) [Chat with your data using dbt Copilot in Insights, powered by the Semantic Layer.](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md) #### Explore the dbt MCP server [![](/img/icons/integration.svg)](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) ###### [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) [Bring dbt structured context into any MCP-compatible AI tool.](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) --- ### About continuous integration (CI) in dbt dbt platform Use [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) in dbt to set up automation for testing code changes before merging to production. Additionally, [enable Advanced CI features](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-ci-features) for these jobs to evaluate whether the code changes are producing the appropriate data changes you want by reviewing the comparison differences dbt provides. Refer to the guide [Get started with continuous integration tests](https://docs.getdbt.com/guides/set-up-ci.md?step=1) for more information. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) ###### [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [Set up CI checks to test every single change prior to deploying the code to production.](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/advanced-ci.md) ###### [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md) [Compare the differences between what's in the production environment and the pull request before merging those changes, ensuring that you're always shipping trusted data products.](https://docs.getdbt.com/docs/deploy/advanced-ci.md) <br /> --- ### About dbt Insights Beta ### About dbt Insights [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Learn how to query data with Insights and view documentation in Catalog. Insights in dbt empowers users to seamlessly explore and query data with an intuitive, context-rich interface. It bridges technical and business users by combining metadata, documentation, AI-assisted tools, and powerful querying capabilities into one unified experience. Insights in dbt integrates with [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), [Canvas](https://docs.getdbt.com/docs/platform/canvas.md), [dbt Copilot in Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md), and [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) to make it easier for you to perform exploratory data analysis, leverage AI-assisted tools, make faster decisions, and collaborate across teams. [![Overview of the dbt Insights and its features](/img/docs/dbt-insights/insights-main.gif?v=2 "Overview of the dbt Insights and its features")](#)Overview of the dbt Insights and its features #### Prerequisites * Be on a dbt [Enterprise or Enterprise+](https://www.getdbt.com/pricing) plan — [book a demo](https://www.getdbt.com/contact) to learn more about Insights. * Available on all [tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) configurations. * Have a dbt [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) with access to Insights. * Configured [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide). * Your production and development [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) are on dbt’s **v1 Latest** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or a supported dbt version. * Use a supported data platform: Snowflake, BigQuery, Databricks, Redshift, or Postgres. * Single sign-on (SSO) for development user accounts is supported. Deployment environments will be queried leveraging the user's user credentials configured in **Account settings**. * (Optional) — To query [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) metrics from the Insights, you must also: * [Configure](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) the Semantic Layer for your dbt project. * Have a successful job run in the environment where you configured the Semantic Layer. * (Optional) To enable [Language Server Protocol (LSP) features](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#lsp-features-in-dbt-insights) in Insights and run your compilations on dbt v2, set your development environment to use the **v2 Stable** dbt version. #### Key benefits Key benefits include: * Quickly write, run, and iterate on SQL queries with tools like syntax highlighting, tabbed editors, and query history. * Leverage dbt metadata, trust signals, and lineage from Catalog for informed query construction. * Make data accessible to users of varied technical skill levels with SQL, Semantic Layer queries, and visual tools. * Use dbt Copilot in Insights to generate or edit SQL queries, descriptions, and more. Some example use cases include: * Analysts can quickly construct queries to analyze sales performance metrics across regions and view results. * All users have a rich development experience powered by Catalog's end-to-end exploration experience. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. --- ### About dbt integrations Many data applications integrate with dbt, enabling you to leverage the power of dbt for a variety of use cases and workflows. #### Integrations with dbt [![](/img/icons/vsce.svg)](https://docs.getdbt.com/docs/about-dbt-extension.md) ###### [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) [The dbt extension brings a hyper-fast, intelligent, and cost-efficient dbt development experience to VS Code. The best way to experience all the power of the new dbt v2 while developing locally.](https://docs.getdbt.com/docs/about-dbt-extension.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) ###### [Visualize and orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) [Configure downstream exposures automatically from dashboards and understand how models are used in downstream tools. Proactively refresh the underlying data sources during scheduled dbt jobs.](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) ###### [dbt Semantic layer integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) [Review a wide range of partners you can integrate and query with the dbt Semantic Layer.](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) --- ### About dbt LSP dbt v2 offers benefits beyond the speed and power of the framework. The dbt VS Code extension, Studio IDE, and Insights all contain a powerful set of features backed by our Language Server Protocol (LSP) that enable fast, efficient development workflows. The following features are supported across these tools: | | VS Code extension | Studio IDE | Insights | | ---------------------------- | ----------------- | ---------- | -------- | | Autocomplete function names | ✅ | ✅ | ❌ | | Autocomplete ref/source args | ✅ | ✅ | ✅ | | CTE Preview | ✅ | ✅ | ✅ | | Column-level lineage | ✅ | ❌ | ❌ | | Compare changes locally | ✅ | ❌ | ❌ | | Command palette | ✅ | N/A | ❌ | | Error detection | ✅ | ✅ | ✅ | | Go-to definition | ✅ | ✅ | ❌ | | Go-to reference | ✅ | ✅ | ❌ | | Incremental compilation | ✅ | ✅ | ❌ | | Lazy compilation | ✅ | ✅ | ❌ | | Preview query results | ✅ | N/A | ❌ | | Problems tab | ✅ | ✅ | ❌ | | Propagate column renames | ✅ | ❌ | ❌ | | Propagate model renames | ✅ | ❌ | ❌ | | Show column type on hover | ✅ | ✅ | ✅ | | Show compiled SQL | ✅ | ✅ | ❌ | | View table lineage | ✅ | N/A | ❌ | | Warning detection | ✅ | ✅ | ❌ | dbt v1 and v2 both support the dbt VS Code extension. Features that depend on built-in SQL parsing, such as column-level lineage, column go-to definition and find references, and `SELECT *` expansion, require [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2). #### Lazy compilation The dbt language server uses on-demand compilation, also called lazy compilation. Lazy compilation starts automatically when you open a model file, you don't need to run `dbt compile` to trigger it. It compiles only the nodes it needs to answer questions about the file you are working in, instead of blocking on a full project compile first. That improves performance because you get editor features for your active file much sooner. ##### What compiles first When you open or focus on a model, the server determines a minimal set of nodes to compile so it can produce up-to-date LSP results for that model. That set includes the current model and its upstream dependencies (ancestors in the DAG), because rendered SQL and analysis depend on `ref`, sources, and inherited context from parents. Nodes you are not actively working on remain `not compiled` until the background compilation pass reaches them. How long that takes depends on the size of your project. Until a node is compiled, LSP results for that node are not available. When you switch to another file, the server reuses results from any compilations that already finished. If a compilation was still in progress when you switched files, it is cancelled and that partial work is discarded; the server then schedules a fresh compile for the newly focused model and its dependencies. ##### Background compilation After the minimal compile for your active file, the server continues with a background compile of the rest of the project. That pass fills in project-wide state without preventing you from using tooling on models that already finished compiling. Background compilation enables full project analysis once it completes. Until then, some features that need the full graph may be limited. You can monitor compilation progress in your editor's status bar. When the progress notifications clear, the background compile is complete. dbt and the language server run independently. Running a command like `dbt run` or `dbt compile` from the terminal does not interrupt or affect LSP compilation. #### Versioning The v2 CLI and the language server ship in a single binary, so they always share the same version and can't be mismatched. The dbt VS Code extension declares which dbt versions it supports and checks the installed binary at startup. For the compatibility matrix and how to verify binaries for manual or air-gapped installs, refer to [Version compatibility](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md). --- ### About dbt Mesh Organizations of all sizes rely upon dbt to manage their data transformations, from small startups to large enterprises. At scale, it can be challenging to coordinate all the organizational and technical requirements demanded by your stakeholders within the scope of a single dbt project. To date, there also hasn't been a first-class way to effectively manage the dependencies, governance, and workflows between multiple dbt projects. That's where **Mesh** comes in - empowering data teams to work *independently and collaboratively*; sharing data, code, and best practices without sacrificing security or autonomy. Mesh is not a single product - it is a pattern enabled by a convergence of several features in dbt: * **[Cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref)** - this is the foundational feature that enables the multi-project deployments. `{{ ref() }}`s now work across dbt projects on Enterprise and Enterprise+ plans. * **[Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md)** - dbt's metadata-powered documentation platform, complete with full, cross-project lineage. * **Governance** - dbt's governance features allow you to manage access to your dbt models both within and across projects. * **[Groups](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups)** - With groups, you can organize nodes in your dbt DAG that share a logical connection (for example, by functional area) and assign an owner to the entire group. * **[Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers)** - access configs allow you to control who can reference models. * **[Model Versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md)** - when coordinating across projects and teams, we recommend treating your data models as stable APIs. Model versioning is the mechanism to allow graceful adoption and deprecation of models as they evolve. * **[Model Contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md)** - data contracts set explicit expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. #### When is the right time to use dbt Mesh? The multi-project architecture helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. If you're already using dbt and your project has started to experience any of the following, you're likely ready to start exploring this paradigm: * The **number of models** in your project is degrading performance and slowing down development. * Teams have developed **separate workflows** and need to decouple development from each other. * Teams are experiencing **communication challenges**, and the reliability of some of your data products has started to deteriorate. * **Security and governance** requirements are increasing and would benefit from increased isolation. dbt is designed to coordinate the features above and simplify the complexity to solve for these problems. If you're just starting your dbt journey, don't worry about building a multi-project architecture right away. You can *incrementally* adopt the features as you scale. The collection of features work effectively as independent tools. Familiarizing yourself with the tooling and features that make up a multi-project architecture, and how they can apply to your organization will help you make better decisions as you grow. For additional information, refer to the [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md). --- ### About dbt Model Context Protocol (MCP) server The [dbt MCP server](https://github.com/dbt-labs/dbt-mcp) provides a standardized framework that lets you integrate AI applications with dbt‑managed data assets across different data platforms. This ensures consistent, governed access to models, metrics, lineage, and freshness across your AI tools. To help with dbt, assistants need your project metadata and, when you allow it, supported actions such as CLI runs, platform APIs, and Semantic Layer queries. The dbt MCP server exposes those to MCP clients and supports use cases such as conversational access to data, agentic automation for dbt workflows, and AI-assisted development. This page covers local and remote setups, available tools, and how to get started. The MCP server provides access to [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md), dbt platform CLI, [API](https://docs.getdbt.com/docs/dbt-apis/overview.md), the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md), and [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). It provides access to private APIs, text-to-SQL, and SQL execution. For more information on MCP, have a look at [Get started with the Model Context Protocol (MCP)](https://modelcontextprotocol.io/introduction). The dbt MCP server comes in two flavors: self-hosted and remote. * [Self-hosted MCP server](#local-mcp-server): runs on your machine and requires installing `uvx` (which installs dbt-mcp locally). * [Remote MCP server](#remote-mcp-server): uses an HTTP connection and makes calls to dbt-mcp remotely on the managed dbt platform. This is a dbt-hosted solution and is ideal for data consumption use cases. For more details on the server types, refer to [Server access](#server-access). #### Get started To get started, choose the quickstart that matches your setup: | I want to... | Quickstart | Tool access | | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | Query data and run dbt platform CLI commands locally while connected to my dbt platform account (Semantic Layer, Discovery API, Admin API, SQL, Codegen). | [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) | Uses [self-hosted MCP server](#local-mcp-server). | | Run dbt platform CLI commands locally, with or without a dbt platform account; with an account, also query data and explore metadata through the same server. | [Run dbt locally](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) | Uses [self-hosted MCP server](#local-mcp-server). | | Use MCP with zero installation (query data only through dbt-hosted tools; no dbt platform CLI commands). | [Connect to the remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md) | Uses [remote MCP server](#remote-mcp-server). | To configure or disable specific tools (self-hosted MCP), see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Server access You can use the dbt MCP server in the following ways: * [Self-hosted MCP server](#local-mcp-server) — runs on your machine and requires installing `uvx` (which installs dbt-mcp locally) and then running `uvx dbt-mcp` to start the server. No need to clone the repo unless you want to contribute to [dbt MCP server](https://github.com/dbt-labs/dbt-mcp). * [Remote MCP server](#remote-mcp-server) — uses an HTTP connection and makes calls to dbt-mcp hosted on the managed dbt platform. This setup requires no self-hosted installation and is ideal for data consumption use cases. ##### Self-hosted MCP server The self-hosted MCP server provides the best experience for development workflows, like authoring dbt models, tests, and documentation. The [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) runs on your machine and requires installing `uvx` (which installs dbt-mcp locally) and then running `uvx dbt-mcp` to start the server. You don't need to clone the repository unless you want to contribute to dbt MCP. The self-hosted MCP server provides: * Full access to dbt commands (`dbt run`, `dbt build`, `dbt test`, and more) * Support for dbt v1, dbt platform CLI, and dbt v2 * Ability to work with local dbt projects with or without a dbt platform account * Optional integration with dbt platform APIs for metadata discovery and Semantic Layer access ##### Remote MCP server The remote MCP server from dbt offers data consumption use cases without a self-hosted setup. It doesn't support local development or dbt platform CLI commands; use the [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) for those workflows. The [remote MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md) connects to the dbt platform via HTTP and requires no installation. This option is useful when: * You either don’t want to install, or are restricted from installing, additional software on your system. * Your use case is primarily consumption-based (for example, querying metrics, exploring metadata, viewing lineage). The remote MCP server is available on all dbt platform [plans](https://www.getdbt.com/pricing). However, the underlying [dbt APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) that the server's tools rely on vary by plan type. For example, the Discovery API and Semantic Layer APIs. As a result, the tools available to you through the remote MCP server depend on your plan. Remote MCP uses the same default global API rate limit as other dbt APIs (5,000 requests per minute per IP). Self-hosted MCP is subject to the public Administrative and Discovery API limits. Refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md) for details. Remote MCP doesn't require AI features You don't need [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled to use remote MCP. If an admin turns AI features off, [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features) is the only tool hidden and every other tool keeps working. `text_to_sql` is also the only tool that consumes your dbt Copilot action allotment. Other MCP tools don't. If your account runs out of dbt Copilot actions, `text_to_sql` will not work until the limit resets. Other remote MCP tools keep working. If you need help, contact your account manager. ##### Supported tools by MCP server type The dbt MCP server has access to many parts of the dbt experience related to development, deployment, and discovery. Here are the categories of tools supported based on what form of the MCP server you connect to as well as detailed information on exact commands or queries available to the LLM. Self-hosted MCP is required for dbt platform CLI commands, Codegen, and Administrative API; remote MCP supports Semantic Layer, SQL, Discovery, Administrative API, and dbt v2 tools only. Note that access to the [dbt APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) is limited depending on your [plan type](https://www.getdbt.com/pricing). | Tools | Self-hosted | Remote | | ------------------------- | ----------- | ------ | | dbt platform CLI commands | ✅ | ❌ | | Semantic Layer | ✅ | ✅ | | SQL | ✅ | ✅ | | Metadata Discovery | ✅ | ✅ | | Administrative API | ✅ | ✅ | | Codegen Tools | ✅ | ❌ | | dbt v2 Tools | ✅ | ✅ | | Product Docs Tools | ✅ | ✅ | | MCP Server Metadata Tools | ✅ | ✅ | #### Available tools The dbt MCP server has access to many tools related to development, deployment, and discovery — like CLI A full list of tools is available for your MCP server and is auto-fetched from the [dbt MCP server README on GitHub](https://github.com/dbt-labs/dbt-mcp#tools) when the docs are built, so it stays in sync with each release. To view the full list of tools, see [Available tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md). #### MCP integrations The dbt MCP server integrates with any [MCP client](https://modelcontextprotocol.io/clients) that supports OAuth or token authentication and tool use capabilities, depending on your setup. We have created integration guides for the following clients: * [Claude](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md) * [Cursor](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md) * [VS Code](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md) * [Snowflake Cortex](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-snowflake-cortex.md) #### Data retention The dbt MCP server doesn't store or retain any production data or job run results. It's a read-only access layer that reads metadata, job results, and Semantic Layer data from the dbt platform in real time when a tool is called. Your [dbt platform data retention policy](https://www.getdbt.com/security) determines how long job runs and artifacts are available, not the MCP server. As long as a job or artifact exists in dbt platform, the MCP server can read it. #### Resources * [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) — full list of variables and tool configuration for self-hosted MCP * For more information, refer to our blog on [Introducing the dbt MCP Server](https://docs.getdbt.com/blog/introducing-dbt-mcp-server#getting-started). --- ### About dbt models Self-hosted and cloud-hosted dbt are composed of different moving parts working harmoniously. All of them are important to what dbt does — transforming data—the 'T' in ELT. When you execute `dbt run`, you are running a model that will transform your data without that data ever leaving your warehouse. Models are where your developers spend most of their time within a dbt environment. Models are primarily written as a `select` statement and saved as a `.sql` file. While the definition is straightforward, the complexity of the execution will vary from environment to environment. Models will be written and rewritten as needs evolve and your organization finds new ways to maximize efficiency. SQL is the language most dbt users will utilize, but it is not the only one for building models. Starting in version 1.3, dbt v1 and dbt support Python models. Python models are useful for training or deploying data science models, complex transformations, or where a specific Python package meets a need — such as using the `dateutil` library to parse dates. ##### Models and modern workflows The top level of a dbt workflow is the project. A project is a directory of a `.yml` file (the project configuration) and either `.sql` or `.py` files (the models). The project file tells dbt the project context, and the models let dbt know how to build a specific data set. For more details on projects, refer to [About dbt projects](https://docs.getdbt.com/docs/build/projects.md). Your organization may need only a few models, but more likely you’ll need a complex structure of nested models to transform the required data. A model is a single file containing a final `select` statement, and a project can have multiple models, and models can even reference each other. Add to that, numerous projects and the level of effort required for transforming complex data sets can improve drastically compared to older methods. Learn more about models in [SQL models](https://docs.getdbt.com/docs/build/sql-models.md) and [Python models](https://docs.getdbt.com/docs/build/python-models.md) pages. If you'd like to begin with a bit of practice, visit our [Getting Started Guide](https://docs.getdbt.com/guides.md) for instructions on setting up the Jaffle\_Shop sample data so you can get hands-on with the power of dbt. --- ### About dbt projects A dbt project informs dbt about the context of your project and how to transform your data (build your data sets). By design, dbt enforces the top-level structure of a dbt project such as the `dbt_project.yml` file, the `models` directory, the `snapshots` directory, and so on. Within the directories of the top-level, you can organize your project in any way that meets the needs of your organization and data pipeline. At a minimum, all a project needs is the `dbt_project.yml` project configuration file. dbt supports a number of different resources, so a project may also include: | Resource | Description | | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [models](https://docs.getdbt.com/docs/build/models.md) | Each model lives in a single file and contains logic that either transforms raw data into a dataset that is ready for analytics or, more often, is an intermediate step in such a transformation. | | [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) | A way to capture the state of your mutable tables so you can refer to it later. | | [seeds](https://docs.getdbt.com/docs/build/seeds.md) | CSV files with static data that you can load into your data platform with dbt. | | [data tests](https://docs.getdbt.com/docs/build/data-tests.md) | SQL queries that you can write to test the models and resources in your project. | | [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) | Blocks of code that you can reuse multiple times. | | [docs](https://docs.getdbt.com/docs/build/documentation.md) | Docs for your project that you can build. | | [sources](https://docs.getdbt.com/docs/build/sources.md) | A way to name and describe the data loaded into your warehouse by your Extract and Load tools. | | [exposures](https://docs.getdbt.com/docs/build/exposures.md) | A way to define and describe a downstream use of your project. | | [metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) | A way for you to define metrics for your project. | | [groups](https://docs.getdbt.com/docs/build/groups.md) | Groups enable collaborative node organization in restricted collections. | | [analysis](https://docs.getdbt.com/docs/build/analyses.md) | A way to organize analytical SQL queries in your project such as the general ledger from your QuickBooks. | | [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) | Semantic models define the foundational data relationships in [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) and the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), enabling you to query metrics using a semantic graph. | | [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) | Saved queries organize reusable queries by grouping metrics, dimensions, and filters into nodes visible in the dbt DAG. | | [user-defined functions](https://docs.getdbt.com/docs/build/udfs.md) | User-defined functions (UDFs) let you create reusable custom functions in your warehouse, shareable across dbt, BI tools, data science workflows, and more. | When building out the structure of your project, you should consider these impacts on your organization's workflow: * **How would people run dbt commands** — Selecting a path * **How would people navigate within the project** — Whether as developers in the Studio IDE or stakeholders from the docs * **How would people configure the models** — Some bulk configurations are easier done at the directory level so people don't have to remember to do everything in a config block with each new model #### Project configuration Every dbt project includes a project configuration file called `dbt_project.yml`. It defines the directory of the dbt project and other project configurations. Edit `dbt_project.yml` to set up common project configurations such as: | YAML key | Value description | | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | [name](https://docs.getdbt.com/reference/project-configs/name.md) | Your project’s name in [snake case](https://en.wikipedia.org/wiki/Snake_case) | | [version](https://docs.getdbt.com/reference/project-configs/version.md) | Version of your project | | [require-dbt-version](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) | Restrict your project to only work with a range of [dbt v1 versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [profile](https://docs.getdbt.com/reference/project-configs/profile.md) | The profile dbt uses to connect to your data platform | | [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) | Directories to where your model and source files live | | [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) | Directories to where your seed files live | | [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) | Directories to where your test files live | | [analysis-paths](https://docs.getdbt.com/reference/project-configs/analysis-paths.md) | Directories to where your analyses live | | [macro-paths](https://docs.getdbt.com/reference/project-configs/macro-paths.md) | Directories to where your macros live | | [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md) | Directories to where your snapshots live | | [docs-paths](https://docs.getdbt.com/reference/project-configs/docs-paths.md) | Directories to where your docs blocks live | | [vars](https://docs.getdbt.com/docs/build/project-variables.md) | Project variables you want to use for data compilation | For complete details on project configurations, see [dbt\_project.yml](https://docs.getdbt.com/reference/dbt_project.yml.md). #### Project subdirectories You can use the Project subdirectory option in dbt to specify a subdirectory in your git repository that dbt should use as the root directory for your project. This is helpful when you have multiple dbt projects in one repository or when you want to organize your dbt project files into subdirectories for easier management. To use the Project subdirectory option in dbt, follow these steps: 1. Click your account name in the bottom left and select **Your profile**. 2. Under **Projects**, select the project you want to configure as a project subdirectory. 3. Select **Edit** on the lower right-hand corner of the page. 4. In the **Project subdirectory** field, add the name of the subdirectory. For example, if your project YAML files are located in a subdirectory called `<repository>/finance`, you would enter `finance` as the subdirectory. * You can also reference nested subdirectories. For example, if your project YAML files are located in `<repository>/teams/finance`, you would enter `teams/finance` as the subdirectory. **Note**: You do not need a leading or trailing `/` in the Project subdirectory field. 5. Click **Save** when you've finished. After configuring the Project subdirectory option, dbt will use it as the root directory for your dbt project. This means that dbt commands, such as `dbt run` or `dbt test`, will operate on files within the specified subdirectory. If there is no `dbt_project.yml` file in the Project subdirectory, you will be prompted to initialize the dbt project. Project support in dbt plans Some [plans](https://www.getdbt.com/pricing) support only one dbt project, while [Enterprise-tier plans](https://www.getdbt.com/contact) allow multiple projects and [cross-project references](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with Mesh. #### New projects You can create new projects and [share them](https://docs.getdbt.com/docs/platform/git/git-version-control.md) with other people by making them available on a hosted git repository like GitHub, GitLab, and BitBucket. After you set up a connection with your data platform, you can [initialize your new project in dbt](https://docs.getdbt.com/guides.md) and start developing. Or, run [dbt init from the command line](https://docs.getdbt.com/reference/commands/init.md) to set up your new project. During project initialization, dbt creates sample model files in your project directory to help you start developing quickly. #### Sample projects If you want to explore dbt projects more in-depth, refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md). It's a runnable project that contains sample configurations and helpful notes. If you want to see what a mature, production project looks like, check out the [GitLab Data Team public repo](https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt). #### Related docs * [Best practices: How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [Quickstarts for dbt](https://docs.getdbt.com/guides.md) * [Quickstart for dbt v1](https://docs.getdbt.com/guides/manual-install.md) --- ### About dbt self-hosted installations Local development You can run dbt locally from your terminal with the dbt CLI, or from your code editor with the dbt VS Code extension. Local development lets you build, test, and run dbt projects from your own machine while connecting to your data platform. (Applies to dbt v2.0 and later) #### Install dbt Install dbt locally to run dbt commands from your terminal. You can install it with `pip`: ```shell python -m pip install dbt ``` Or with `curl`: ```shell curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh ``` For full installation instructions, including adapter-specific setup, refer to [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). #### dbt VS Code extension The [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) lets you develop dbt projects from VS Code, Cursor, or Windsurf. Use the extension if you want an editor-based local development experience. For installation and setup, refer to the [extension docs](https://docs.getdbt.com/docs/about-dbt-extension.md). #### dbt Wizard [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) is a natural next step for local dbt development. It works with dbt and adds an AI agent that understands your full project through the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md), a structured view of your [lineage](https://docs.getdbt.com/docs/explore/explore-projects.md), model health, test coverage, and semantic definitions. * **Build and refactor from natural language:** Describe the change, get a reviewable diff, approve before anything is written. * **Validate in a tight loop:** Every proposed change compiles and runs against your warehouse, catching issues before production. * **Navigate with full project context:** Traverse the [DAG](https://docs.getdbt.com/docs/explore/explore-projects.md), surface downstream impact, and keep tests and YAML in sync as models evolve. For data practitioners, dbt Wizard adds an AI layer that knows your project, not just your code. Refer to the [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) to get started. #### dbt MCP server The dbt MCP server connects your local dbt project to AI assistants using the [Model Context Protocol](https://modelcontextprotocol.io/). It works with dbt and requires no repository clone. * **dbt platform CLI tools:** Run `dbt run`, `build`, `test`, `compile`, `list`, `parse`, and `show` directly from your AI assistant's chat interface. * **Local project context:** Surface model lineage, node details, and dependency graphs from your local `manifest.json` without leaving your editor. * **Code generation:** Auto-generate model YAML, source definitions, and staging SQL from your warehouse schema (requires the codegen toolset to be enabled). * **Zero-clone install:** Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp`. No repository clone needed! [Connect dbt MCP server to your local project](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md). #### Licensing info dbt framework has two distributions which can both be installed locally for free, powered by a single engine: * The Apache 2.0 licensed open-source distribution. Both v1 and v2 are available as open source installations. * dbt v2 extends the dbt OSS offering with additional advanced capabilities — all free to use! Refer to [licensing](https://docs.getdbt.com/docs/dbt-licensing.md) for more info. --- ### About dbt State Login required | Usage-based dbt State makes dbt smarter about what to build. Instead of rebuilding every node on every run, dbt reuses nodes by cloning from another location or skipping a rebuild when the logic and data haven't changed. With dbt State, dbt first compares the logic and data of each node to previous builds across multiple environments on every run — whether orchestrated in the dbt platform, through your own orchestrator, or in development. If the logic is the same and the data is still fresh, dbt reuses an existing object. It will either clone an existing node from elsewhere, or skip executing a model that already exists, rather than building it anew. Additionally, it will automatically defer to production state without the need to manually set the `--defer` or `--state` flags. dbt State can reuse all node types that create relations in the database (such as SQL models, snapshots, seeds) and data tests. Note that the following models are not eligible for reuse: * **Python models**: dbt State builds Python models on every run, even if their code and upstream data have not changed. * **Models with custom materializations**: dbt State builds these models on every run because custom materializations may have side effects (for example, modifying table properties or writing to other schemas), and dbt State cannot safely determine whether skipping the run would produce the same result. dbt State works with dbt (v1 and v2) and the dbt platform, across all environments and orchestrators, making it a flexible approach regardless of how you run dbt. It requires authentication through a dbt platform account. For pricing details, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md). #### Benefits dbt State reduces warehouse compute costs and simplifies how you develop, orchestrate, and execute dbt: * **Fresher data, lower costs**: Nodes only rebuild when the result would be different (new data or code changes), reducing warehouse compute while keeping production data fresh. * **Faster, simpler development**: Without dbt State, developing in a fresh schema means you need to build upstream dependencies, maintain development copies, or configure `--defer` with a state manifest. With dbt State, none of that is needed since dbt finds the freshest available state across environments, clones fresh upstream assets, and rebuilds only what changed. * **Simpler orchestration**: Orchestration decisions typically happen at the job level — which nodes belong in which job, and how often that job should run. dbt State adds node-level decision-making within each run: instead of rebuilding every selected node, dbt evaluates each one individually and decides whether to rebuild, clone, or skip based on its logic and data freshness. Unlike standard deferral, which always builds selected nodes and only defers unselected upstream references, dbt State can skip or clone selected nodes, too. * **Model-level freshness threshold**: The [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) config sets how long dbt State waits before rebuilding a node once its upstream data changes. A node rebuilds only when its last build is older than the tolerance window and its upstream data has changed. It decouples downstream models from high-frequency upstream changes, and prevents costly rebuilds on stagnant data when an upstream dependency misses its freshness [Service Level Agreement (SLA)](https://www.getdbt.com/blog/data-slas-best-practices). #### How dbt State works When you run a command like `dbt build --select +my_model`, dbt State evaluates each selected node and applies the most efficient approach it can: * **Reuse node from same schema (skip)** — dbt checks whether the object already exists in the target schema, its logic hasn't changed, and it is not yet due for rebuilding based on its [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) (either because the last build is still within the tolerance window, or its upstream data hasn't changed). If all conditions are met, dbt skips the node entirely, as if it was never selected. For data tests, if the nodes being tested haven't changed since the last run, the previous test result is reused without re-executing the test query. For views, if the view's logic is unchanged, dbt State reuses it even if new data has arrived upstream. Because views don't store data, new upstream data is automatically reflected when the view is queried, even without a rebuild. Note that views using `select *` on an upstream node may behave differently — refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select) for more information. * **Reuse node from different schema (clone)** — dbt State looks across all environments and jobs for a matching object with identical logic and fresh data. This includes schemas where a model was built before it ever ran in production. When multiple candidates exist, dbt State clones from the one with the freshest data, regardless of which environment it came from. For example, if a CI schema has fresher data than production and identical logic, dbt State clones from there. The node is marked as **Reused** at a fraction of the compute cost. If you want to prevent cloning into a specific target (for example, in regulated environments), set [`allow_clones: false`](https://docs.getdbt.com/reference/resource-configs/allow-clones.md) on that target in `profiles.yml` or as an [extended attribute](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) in the dbt platform. * **Normal build** — If reuse is not possible, dbt builds the node as normal, automatically deferring any unselected upstream nodes. dbt State fetches table metadata (for example, last-modified timestamps) in the background at the start of each run. Any node ready to skip, clone, or execute proceeds immediately; nodes with an undetermined action wait for the fetch to complete. Without dbt State, every selected node rebuilds on every run regardless of whether anything has changed. To see which decision dbt State made for each node after a run and why, you can run the (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) command. If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. For the full list of available configs, see [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md).  How dbt State decides whether to rebuild, clone, or reuse The following decision tree shows how dbt State chooses the most efficient valid action for each node. [![Decision tree showing how dbt State chooses whether to rebuild, clone, or reuse a node based on state bypasses, volatile SQL handling, execution hashes, freshness, schema matches, clone eligibility, and whether fresh upstream data can still be cloned from time travel or another schema](/img/docs/deploy/run-cache-decision-tree.png?v=2 "dbt State decision tree for rebuild, clone, and reuse")](#)dbt State decision tree for rebuild, clone, and reuse The key idea is that dbt State only skips work when it can prove the existing object is sufficiently equivalent for the current run. If the SQL logic, relevant config, schema, or upstream freshness means the result might be different, dbt rebuilds instead. ##### Use dbt State with state:\* selectors [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") In self-managed deployments, you can also use dbt State with `state:*` selectors. Instead of comparing against a single `manifest.json`, these selectors use dbt State as the comparison source, with state tracked for each individual node. Refer to [dbt State-powered `state:*` selectors](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md#dbt-state-powered-state-selectors) for more information. #### Signing up for dbt State dbt State requires a dbt platform account for authentication, even if you run dbt locally or in a non-interactive environment. * **Existing dbt platform users**: Your dbt State credentials are the same as your platform credentials, and dbt State has access to your platform environments and jobs. * **New to dbt platform**: Create an account from the [dbt platform sign-up page](http://us1.dbt.com/register?_dbtsrc=dbt-state). For dbt v2 and v1 users, you can start the account creation workflow from the CLI by running `dbt login` (v1.13/v2) or `dbt run`/`dbt build` (dbt v1.7–1.12), which opens the sign-up page where you can create a free account. For setup steps, refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). The dbt State app is deprecated The standalone dbt State app (`app.state.dbt.com`) is no longer accepting new users. If you have an existing dbt State app account, [create a free dbt platform account](http://us1.dbt.com/register?_dbtsrc=dbt-state) to continue using dbt State — your free trial will be extended upon account creation. #### FAQs What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)** as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). How is dbt State different from using state:modified? On its own, `state:modified` requires manual management of `manifest.json`, which is cumbersome and error-prone. dbt State is completely managed with almost zero setup and no workflow changes. In self-managed deployments, dbt State can also power `state:*` selectors directly — see the last item below. `state:modified` only checks if a file has changed. dbt State has semantic understanding of SQL, so meaningless changes like whitespace or table aliases are not counted as a change — making dbt State smarter about what actually needs to rebuild. `state:modified` does not consider upstream data changes. dbt State checks all sources to see if there is any new data or if the schema has been modified. This enables dbt State to skip running models if the result of the run would be the same as before. For example, if you run `dbt run` with `state:modified` twice, it runs all modified models both times. dbt State only reruns models the second time if upstream sources have changed. dbt State also has the ability to auto-defer refs and automatically clone tables when the result of the clone would have been the same as a full model run. `state:modified` has a limitation on seed files over 1MB, while dbt State does not. When dbt State is enabled in a self-managed deployment, all [`state:*` selectors](https://docs.getdbt.com/reference/node-selection/methods.md#state) can use dbt State as their comparison source. This provides per-node granularity — each node is compared against its own last execution in the [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) environment, rather than comparing all nodes against a single `manifest.json` from the most recent job. This applies automatically when no explicit [`--state`](https://docs.getdbt.com/reference/node-selection/configure-state.md) manifest has been provided. For more details, refer to [dbt State-powered `state:*` selectors](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md#dbt-state-powered-state-selectors). Does dbt State support incremental models? dbt State works with incremental models. When you make a change to an incremental model and run it in development, dbt State automatically clones the model from production if it exists, then runs the new model logic on top of the clone. If you want to revert to the original dbt behavior and fully refresh the incremental model, pass the [`--full-refresh` flag](https://docs.getdbt.com/reference/commands/run.md#refresh-incremental-models). Does dbt State support Python models? dbt State builds Python models but does not reuse them. It executes Python models on every run, even when their code and upstream data have not changed. How is data stored in dbt State? dbt State sends the following metadata to dbt Labs servers: * **Last-modified timestamps**: Used to determine whether upstream data has changed since the last run * **SQL statement hashes**: SQL statements are processed to detect and classify changes, then hashed. Only the hash is persisted for future comparisons. No actual data from your warehouse is transmitted. The dbt State service runs in a single US multi-tenant (MT) instance. The service never connects to your data warehouse. No actual data from your warehouse is transmitted. The only connection is to your running dbt process (CLI or platform) in order to exchange the metadata described above. For data retention details, refer to the [dbt Labs privacy policy](https://www.getdbt.com/cloud/privacy-policy). Where does the metadata about last updated timestamp come from? Last updated timestamps come directly from the data warehouse, for example from `INFORMATION_SCHEMA` tables. How does dbt State calculate that a model has changed? dbt State only considers substantial changes to a model. Because dbt State understands the entire lineage of your models, it can see through things like whitespace and aliases to determine whether a model is the same or different across environments. By default, dbt State compares rendered SQL to detect code changes. Any change to the rendered SQL — including from non-deterministic macros or environment variables — triggers a rebuild. You can enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md) to also check the Jinja template (unrendered code). When enabled, a rebuild only occurs when both the template *and* the rendered SQL have changed. Non-deterministic values (for example, `{{ env_var('AIRFLOW_RUN_ID') }}` or a macro that calls `uuid()`) don't trigger rebuilds as long as the template itself is unchanged, which helps avoid unnecessary warehouse compute costs. Why is my model being rebuilt instead of reused? dbt State decides whether to reuse a model by parsing the rendered SQL into a syntax tree and comparing the hash. If the hash has changed (implying the model's logic has changed), dbt State rebuilds the model. dbt State prioritizes safety and precision; if it can't guarantee skipping a node is safe, then it rebuilds the node to be sure. A few patterns that commonly cause overeager rebuilds are listed on this page, along with recommendations to increase reuse rate. The following patterns commonly cause unexpected rebuilds: * [Views with `select *`](#views-with-select) * [Non-deterministic Jinja templating](#non-deterministic-jinja-templating) * [Models with external sources in BigQuery](#models-with-external-sources-in-bigquery) * [Models with custom materializations](#models-with-custom-materializations) #### Views with `select *` dbt State reuses a model when its compiled SQL matches the stored hash. When a view uses `select *` directly on a `ref()` or `source()`, dbt can't determine the column list at parse time — the upstream model or source table might have gained or lost columns since the last run. To be safe, dbt State forces a rebuild. For example, this view will be rebuilt even if `stg_orders` hasn't changed because dbt can't know at parse time whether `stg_orders` has the same columns as before: ```sql -- stg_orders_view.sql (materialized: view) select * from {{ ref('stg_orders') }} ``` However, if you use `select *` on a CTE, dbt can resolve the columns from the CTE definition and safely reuse the view: ```sql with renamed as ( select order_id, customer_id, order_total from {{ ref('stg_orders') }} ) select * from renamed ``` If a CTE explicitly names its columns, a `select *` that reads from that CTE won't force a rebuild even if an earlier CTE used `select *` on a `ref()` or `source()`. The typical staging pattern is reused: ```sql with source as ( select * from {{ source('jaffle_shop', 'orders') }} ), renamed as ( select id as order_id, user_id as customer_id, amount as order_total from source ) select * from renamed ``` tip To avoid forced rebuilds, use explicit column names when selecting directly from a `ref()` or `source()`. You can also exclude views from execution using `--exclude config.materialized:view`. #### Non-deterministic Jinja templating Some macros and environment variables can cause unexpected rebuilds. For example, `dbt_utils.get_relations_by_pattern` (an introspective macro) combined with `dbt_utils.union_relations` can return relations in a different order on each run, producing different rendered SQL even when your project logic hasn't changed. Similarly, environment variables that change between runs produce different rendered SQL on every run: ```sql select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` Because the query result order or the environment variable's value changes, the rendered SQL differs from the stored hash on every run. dbt State treats this as a code change and rebuilds the model, even though the underlying project logic hasn't changed. This pattern can affect any model type, not just views; if a base or staging model rebuilds on every run, all of its downstream models rebuild, too. To avoid these unnecessary rebuilds, enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md). When enabled, dbt State checks both the Jinja template and rendered SQL; non-deterministic values that don't change the template don't trigger a rebuild. For example: ```sql {{ config(state={"compare_unrendered_code": true}) }} select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` #### Models with external sources on BigQuery On BigQuery, models that use external sources (such as Google Sheets) always rebuild because BigQuery doesn't expose modification timestamps for external sources, so dbt State can't determine freshness. tip To prevent external sources from always being considered stale, configure [`loaded_at_field`](https://docs.getdbt.com/reference/resource-configs/freshness.md) or [`loaded_at_query`](https://docs.getdbt.com/reference/resource-configs/freshness.md) in your source definition to point to a timestamp field. This lets dbt State query a timestamp field directly to determine freshness, rather than relying on warehouse metadata. #### Models with custom materializations Models using custom materializations are always built and are never reused. Custom materializations may have side effects (for example, modifying table properties or writing to other schemas), and dbt State cannot safely determine whether skipping the run would produce the same result. #### How to diagnose After a run, use (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) to see why dbt State rebuilt, reused, or cloned a specific model. For a detailed breakdown, use the `--verbose` flag with `-s` to select your model: note The command name differs by version: dbt v2 uses `dbt state explain` (with a space), while dbt v1 uses `dbt-state explain` (with a hyphen). (Applies to dbt v2.0 and later) ```bash dbt state explain --verbose -s my_model_name ``` If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. What happens if dbt State servers fail? If dbt State servers are unavailable, dbt gracefully falls back to normal dbt behavior. What if I work on multiple projects that each use their own dbt State? You can specify your org ID in `dbt_project.yml`: ```yaml dbt-cloud: state-org-id: <your-org-id> ``` What if my prod environment isn't named prod? You can specify the defer-to environment using the [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) config in `profiles.yml`: ```yaml my_project: outputs: prod: type: snowflake defer_to_target: production ``` `defer_to_target` only applies to self-managed deployments. If you're using the dbt platform, deferral is configured through your environment settings in the UI. For more details, refer to [Configuring deferral](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md). #### Related docs * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) * [Set up authentication for non-interactive environments](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) * [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md) * [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md) --- ### About dbt versions dbt v2 is the current generation of dbt and uses the 2.x release series. dbt v1 is the Python-based generation and stays on the 1.x series. Both follow semantic versioning. This page covers versioning for local dbt installations. v2 ships in two distributions. Install dbt v2 for the full, recommended experience. If your organization has a strict requirement for an Apache 2.0 licensed runtime, install dbt OSS instead. For more information, refer to [dbt licensing](https://docs.getdbt.com/docs/dbt-licensing.md). If you're using the dbt platform (including the dbt platform CLI), you don't need to manage dbt versions yourself. [Release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) automatically keep you up to date and provide early access to new features. #### dbt v2 versioning dbt v2 uses semantic versioning starting with version 2.0. To install or update dbt v2, see [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md?version=2). ##### Latest release | dbt v2 | Initial release | Support level and end date | | ------------------------------------------------------------------------------------ | --------------- | ---------------------------------- | | [**v2.0**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) | Sept 14, 2026 | **Active support — Sept 13, 2027** | ##### Semantic versioning dbt v2 follows [semantic versioning](https://semver.org/): * **Major versions** (for example, v2 to v3) may include breaking changes. Deprecated functionality will stop working. * **Minor versions** (for example, v2.0 to v2.1) add features and are backwards compatible. They will not break project code that relies on documented functionality. * **Patch versions** (for example, v2.0.0 to v2.0.1) include fixes only: bug fixes, security fixes, or installation fixes. ##### Release channels dbt v2 is distributed through release channels: | Channel | Description | Stability | | -------- | -------------------------------------- | ------------------------------------------------------------------- | | `latest` | The stable, "known good" version | ✅ Recommended for most users | | `canary` | The latest officially released version | ⚠️ Most recent stable version but still undergoing thorough testing | | `dev` | The latest development build | ❌ May not have passed all tests | Run `dbt system update` to get the latest stable release, or specify a channel with `dbt system update --version canary`. For current versions and release history, see [dbt v2 releases](https://docs.getdbt.com/docs/dbt/dbt-releases.md). ##### Checking your version Run `dbt --version` to check your installed version: ```text $ dbt --version dbt v2 2.0.0 ``` ##### Further reading * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md?version=2): Install or update dbt v2. * [dbt v2 releases](https://docs.getdbt.com/docs/dbt/dbt-releases.md): View current versions and release history. * [Get started with dbt v2](https://docs.getdbt.com/docs/dbt/get-started-dbt.md): Learn about dbt v2 features and migration. #### dbt v1 versioning The dbt v1 engine uses semantic versioning for the 1.x release series. To install or update dbt v1, see [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md?version=1). * **[Active](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: In the first few months after a minor version's initial release, we patch it with bugfix releases. These include fixes for regressions, new bugs, and older bugs / quality-of-life improvements. We implement these changes when we have high confidence that they're narrowly scoped and won't cause unintended side effects. * **[Critical](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: When a newer minor version ships, the previous one transitions to "Critical Support" for the remainder of its one-year window. Patches during this period are limited to critical security and installation fixes. After the one-year window ends, the version reaches end of life. * **[End of Life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions)**: Minor versions that have reached EOL no longer receive new patch releases. * **Deprecated**: dbt v1 versions that are no longer maintained by dbt Labs, nor supported in the dbt platform. ##### Latest releases | dbt v1 | Initial release | Support level and end date | | ------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------ | | [**v1.13**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.13.md) | (coming soon) | Beta | | [**v1.12**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.12.md) | Jul 16, 2026 | **Active support — July 15, 2027** | | [**v1.11**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.11.md) | Dec 19, 2025 | **Critical support — Dec 18, 2026** | | [**v1.10**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md) | Jun 16, 2025 | Deprecated ⛔️ | | [**v1.9**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md) | Dec 9, 2024 | Deprecated ⛔️ | | [**v1.8**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.8.md) | May 9, 2024 | Deprecated ⛔️ | | [**v1.7**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.7.md) | Nov 2, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.6**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.6.md>) | Jul 31, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.5**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) | Apr 27, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.4**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.4.md>) | Jan 25, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.3**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.3.md>) | Oct 12, 2022 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.2**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.2.md>) | Jul 26, 2022 | Deprecated ⛔️ | | [**v1.1**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.1.md>) | Apr 28, 2022 | Deprecated ⛔️ | | [**v1.0**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md>) | Dec 3, 2021 | Deprecated ⛔️ | | **v0.X** ⛔️ | (Various dates) | Deprecated ⛔️ | All functionality in v1 since the v1.7 release is available in [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), which provide automated upgrades at a cadence appropriate for your team. 1 Release tracks are required for the Developer and Starter plans on dbt. Accounts using older dbt versions will be migrated to the **v1 Latest** release track. For customers of dbt: dbt Labs strongly recommends migrating environments on older and unsupported versions to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or a supported version. On January 31, 2027, dbt Labs will deprecate v1 versions v1.3 through v1.7, removing them from availability in dbt platform. ##### How dbt v1 uses semantic versioning dbt follows [semantic versioning](https://semver.org/): * **Major versions** (for example, v1 to v2) may include breaking changes. Deprecated functionality will stop working. * **Minor versions** (for example, v1.8 to v1.9) add features and are backwards compatible. They will not break project code that relies on documented functionality. * **Patch versions** (for example, v1.8.0 to v1.8.1) include fixes only: bug fixes, security fixes, or installation fixes. We are committed to avoiding breaking changes in minor versions for end users of dbt. There are two types of breaking changes that may be included in minor versions: * Changes to the Python interface for adapter plugins. These changes are relevant only to adapter maintainers, and they will be clearly communicated in documentation and release notes. For more information, refer to [Build, test, document, and promote adapters guide](https://docs.getdbt.com/guides/adapter-creation.md). * Changes to metadata interfaces, including [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) and [logging](https://docs.getdbt.com/reference/events-logging.md), signalled by a version bump. Those version upgrades may require you to update external code that depends on these interfaces, or to coordinate upgrades between dbt orchestrations that share metadata, such as [state-powered selection](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). ##### Adapter plugin versions dbt releases `dbt-core` and adapter plugins (such as `dbt-snowflake`) independently. Their minor and patch version numbers may not match, but they coordinate through the `dbt-adapters` interface so you won't get a broken experience. For example, `dbt-core==1.8.0` can work with `dbt-snowflake==1.9.0`. If you're building or maintaining an adapter, refer to the [adapter creation guide](https://docs.getdbt.com/guides/adapter-creation.md) for details on the `dbt-adapters` interface. Run `dbt --version` to check your installed versions: ```text $ dbt --version Core: - installed: 1.8.0 - latest: 1.8.0 - Up to date! Plugins: - snowflake: 1.9.0 - Up to date! ``` You can also find the registered adapter version in [logs](https://docs.getdbt.com/reference/global-configs/logs.md). For example, in `logs/dbt.log`: ```text [0m13:13:48.572182 [info ] [MainThread]: Registered adapter: snowflake=1.9.0 ``` Refer to [Supported data platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) for the full list of adapters. ##### Further reading * [Choosing a dbt v1 version in dbt](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md): Learn how to use dbt v1 versions in dbt. * [Install dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md?version=1): Install or update dbt v1. * [`require-dbt-version`](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) and [`dbt_version`](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt_version.md): Restrict your project to work with a specific range of versions. #### End-of-life versions info On January 31, 2027, dbt versions v1.3-v1.7 will be deprecated and removed from availability in dbt platform. Upgrade any environments still on these versions to a supported version or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then. Once a dbt version reaches end-of-life (EOL), it no longer receives patches, including for known bugs. We recommend upgrading to a newer version in [dbt](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md), [dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md?version=2), or [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). All versions prior to v1.0 have been deprecated. #### Current version support dbt supports each minor version (for example, v1.8) for *one year* from its initial release. During that window, we release patches with bug fixes and security updates. When we refer to a minor version, we mean its latest available patch (v1.8.x). After a newer minor version ships, the previous one transitions to **critical support** (security and installation fixes only) for the remainder of its one-year window. After the one-year window ends, the version reaches **end of life** and no longer receives patches. While a minor version is officially supported: * You can use it in dbt. For more on dbt versioning, see [Choosing a dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). * You can select it from the version dropdown on this website to see documentation that is accurate for use with that minor version. #### Upgrading Upgrade to new patch versions as soon as they're available. Upgrade to new minor versions when you're ready because you can only get some features and fixes on the latest minor version. dbt makes all versions available as prereleases before the final release. For minor versions, we aim to release one or more betas 4+ weeks before the final release so you can try new features and share feedback. Release candidates are available about two weeks before the final release for testing in production-like environments. Refer to the [`dbt-labs/dbt` milestones](https://github.com/dbt-labs/dbt/milestones) for details. --- ### About documentation Good documentation for your dbt models will help downstream consumers discover and understand the datasets you curate for them. dbt provides a way to generate documentation for your dbt project and render it as a website. Tip Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to generate documentation in the dbt platform or locally in the CLI. #### Related documentation * [Declaring properties](https://docs.getdbt.com/reference/configs-and-properties.md) * [`dbt docs` command](https://docs.getdbt.com/reference/commands/cmd-docs.md) * [`doc` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md) * If you're new to dbt, we recommend that you check out our [quickstart guide](https://docs.getdbt.com/guides.md) to build your first dbt project, complete with documentation. #### Assumed knowledge * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) #### Overview dbt provides a scalable way to [generate](#generating-documentation) documentation for your dbt project using descriptions and commands. The documentation for your project includes: * **Information about your project**: including model code, a DAG of your project, any tests you've added to a column, and more. * **Information about your data warehouse**: including column data types, and table sizes. This information is generated by running queries against the information schema. * Importantly, dbt also provides a way to add **descriptions** to models, columns, sources, and more, to further enhance your documentation. The following sections describe how to [add descriptions](#adding-descriptions-to-your-project) to your project, [generate documentation](#generating-documentation), how to use [docs blocks](#using-docs-blocks), and set a [custom overview](#setting-a-custom-overview) for your documentation. #### Adding descriptions to your project Before generating documentation, add [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) to your project resources. Add the `description:` key to the same YAML files where you declare [data tests](https://docs.getdbt.com/docs/build/data-tests.md). For example: models/\<filename>.yml ```yaml models: - name: events description: This table contains clickstream events from the marketing website columns: - name: event_id description: This is a unique identifier for the event data_tests: - unique - not_null - name: user-id quote: true description: The user who performed the event data_tests: - not_null ``` ##### FAQs Are there any example dbt documentation sites? Yes! * **Quickstart Tutorial:** You can build your own example dbt project in the [quickstart guide](https://docs.getdbt.com/docs/get-started-dbt.md) * **Jaffle Shop:** A demonstration project (closely related to the tutorial) for a fictional e-commerce store. Refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) for clone instructions, or view the [main source code](https://github.com/dbt-labs/jaffle-shop) and [source code using DuckDB](https://github.com/dbt-labs/jaffle_shop_duckdb) on GitHub. * **GitLab:** Gitlab's internal dbt project is open source and is a great example of how to use dbt at scale ([source code](https://gitlab.com/gitlab-com/content-sites/handbook/blob/main/content/handbook/enterprise-data/platform/dbt-guide.md)) * **dummy-dbt:** A containerized dbt project that populates the Sakila database in Postgres and populates dbt seeds, models, snapshots, and tests. The project can be used for testing and experimentation purposes ([source code](https://github.com/gmyrianthous/dbt-dummy)) * **Google Analytics 4:** A demonstration project that transforms the Google Analytics 4 BigQuery exports to various models ([source code](https://github.com/stacktonic-com/stacktonic-dbt-example-project), [docs](https://stacktonic.com/article/google-analytics-big-query-and-dbt-a-dbt-example-project)) * **Make Open Data:** A production-grade ELT with tests, documentation, and CI/CD (GHA) about French open data (housing, demography, geography, etc). It can be used to learn with voluminous and ambiguous data. Contributions are welcome ([source code](https://github.com/make-open-data/make-open-data). If you have an example project to add to this list, suggest an edit by clicking **Edit this page** below. Do I need to add a YAML entry for column for it to appear in the docs site? Fortunately, no! dbt will introspect your warehouse to generate a list of columns in each relation, and match it with the list of columns in your `.yml` files. As such, any undocumented columns will still appear in your documentation! How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. Can I document things other than models, like sources, seeds, and snapshots? Yes! You can document almost everything in your project using the `description:` key. Check out the reference docs on [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) for more info! #### Generating documentation dbt provides both self-hosted and cloud-hosted solutions to [view documentation](https://docs.getdbt.com/docs/build/view-documentation.md) for your project. Which one you use depends on your dbt version and where you run dbt. (Applies to dbt v2.0 and later) | Option | What it is | Where you use it | How to generate it | | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | [**dbt Docs v2**](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2) | A redesigned, open-source documentation site with Semantic Layer metadata and [column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) that you can host anywhere | Locally with dbt v2. Not available in dbt platform | Run `dbt docs generate` locally. Refer to [generate docs locally](#generate-docs-locally) for steps | | [**dbt Catalog**](https://docs.getdbt.com/docs/explore/explore-projects.md) | A dynamic, real-time interface with enhanced metadata, customizable views, deeper project insights, and collaboration tools. | dbt platform [Starter, Enterprise, or Enterprise+ plans](https://www.getdbt.com/pricing) | Populated automatically when your jobs run with dbt v2. No extra step required! | Sharing docs with stakeholders? Anyone with a developer or read-only seat can explore your project(s) in Catalog. Add as many read-only seats as you need to share docs with stakeholders, no separate docs site required. ##### Generate docs locally (Applies to dbt v2.0 and later) Using dbt v2, dbt Docs v2 enhances the original v1 static site with a modern, performant catalog. dbt docs generate compiles your project, produces the v2 Parquet artifacts, and writes a static site that the browser queries directly with DuckDB-WASM (WebAssembly), so you don't need a server to view it. To generate and serve documentation locally: * Run `dbt docs generate` to compile your project, write the index, and export the documentation site in a single command. * Run `dbt docs serve` to preview the site locally. dbt Docs v2 availability dbt Docs v2 only works self-hosted installations of dbt v2. If you're on the dbt platform, use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) which is populated automatically when your jobs run with v2. Adding a `dbt docs generate` step to a job won't produce a static site in dbt platform. Refer to [dbt docs commands](https://docs.getdbt.com/reference/commands/cmd-docs.md) for full usage details, and [View documentation](https://docs.getdbt.com/docs/build/view-documentation.md) to get the most out of your project's documentation. #### Using docs blocks Docs blocks provide a robust method for documenting models and other resources using Jinja and markdown. Docs block files can contain arbitrary markdown, but they must be uniquely named. ##### Syntax To declare a docs block, use the Jinja `docs` tag. The name of a docs block can't start with a digit and may contain: * Uppercase and lowercase letters (A-Z, a-z) * Digits (0-9) * Underscores (\_) events.md ```markdown {% docs table_events %} This table contains clickstream events from the marketing website. The events in this table are recorded by Snowplow and piped into the warehouse on an hourly basis. The following pages of the marketing site are tracked: - / - /about - /team - /contact-us {% enddocs %} ``` In this example, a docs block named `table_events` is defined with some descriptive markdown contents. There is nothing significant about the name `table_events` — docs blocks can be named however you like, as long as the name only contains alphanumeric and underscore characters and doesn't start with a numeric character. ##### Placement (Applies to dbt v1.12 and later) Place docs blocks in `.md` files. You can also use Jinja-style extensions (`.md.j2`, `.md.jinja`, `.md.jinja2`), however these require setting [`allow_jinja_file_extensions: true`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) in your `dbt_project.yml`. This enables Jinja-aware syntax highlighting in IDEs that associate these suffixes with Jinja templating. By default, dbt searches in all resource paths for docs blocks (for example, the combined list of [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md), [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md), [analysis-paths](https://docs.getdbt.com/reference/project-configs/analysis-paths.md), [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md), [macro-paths](https://docs.getdbt.com/reference/project-configs/macro-paths.md), and [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md)). You can adjust this behavior using the [docs-paths](https://docs.getdbt.com/reference/project-configs/docs-paths.md) config. ##### Usage To use a docs block, reference it from your `schema.yml` file with the [doc()](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md) function in place of a markdown string. Using the examples above, the `table_events` docs can be included in the `schema.yml` file as shown here: schema.yml ```yaml models: - name: events description: '{{ doc("table_events") }}' columns: - name: event_id description: This is a unique identifier for the event data_tests: - unique - not_null ``` In the resulting documentation, `'{{ doc("table_events") }}'` will be expanded to the markdown defined in the `table_events` docs block. #### Setting a custom overview Availability This feature is available only in [dbt Docs](https://docs.getdbt.com/docs/build/documentation.md), the generated documentation site for your dbt project. The "overview" shown in the dbt Docs website can be overridden by supplying your own docs block called `__overview__`. * By default, dbt supplies an overview with helpful information about the docs site itself. * Depending on your needs, it may be a good idea to override this docs block with specific information about your company style guide, links to reports, or information about who to contact for help. * To override the default overview, create a docs block that looks like this: models/overview.md ```markdown {% docs __overview__ %} # Monthly Recurring Revenue (MRR) playbook. This dbt project is a worked example to demonstrate how to model subscription revenue. **Check out the full write-up [here](https://blog.getdbt.com/modeling-subscription-revenue/), as well as the repo for this project [here](https://github.com/dbt-labs/mrr-playbook/).** ... {% enddocs %} ``` ##### Custom project-level overviews You can set different overviews for each dbt project/package included in your documentation site by creating a docs block named `__[project_name]__`. For example, in order to define custom overview pages that appear when a viewer navigates inside the `dbt_utils` or `snowplow` package: models/overview.md ```markdown {% docs __dbt_utils__ %} # Utility macros Our dbt project heavily uses this suite of utility macros, especially: - `surrogate_key` - `test_equality` - `pivot` {% enddocs %} {% docs __snowplow__ %} # Snowplow sessionization Our organization uses this package of transformations to roll Snowplow events up to page views and sessions. {% enddocs %} ``` --- ### About Hybrid projects dbt platform | Enterprise+ With Hybrid projects, your organization can adopt complementary dbt v1 and dbt workflows (where some teams deploy projects in dbt v1 and others in dbt) and seamlessly integrate these workflows by automatically uploading dbt v1 [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) into dbt. Available in public preview Hybrid projects is available in public preview to [dbt Enterprise accounts](https://www.getdbt.com/pricing). dbt v1 users can seamlessly upload [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) like [run results.json](https://docs.getdbt.com/reference/artifacts/run-results-json.md), [manifest.json](https://docs.getdbt.com/reference/artifacts/manifest-json.md), [catalog.json](https://docs.getdbt.com/reference/artifacts/catalog-json.md), [sources.json](https://docs.getdbt.com/reference/artifacts/sources-json.md), and so on — into dbt after executing a run in the dbt v1 command line interface (CLI), which helps: * Collaborate with dbt + dbt v1 users by enabling them to visualize and perform [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) to dbt models that live in dbt v1 projects. * (Coming soon) New users interested in the [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) can build off of dbt models already created by a central data team in dbt v1 rather than having to start from scratch. * dbt v1 and dbt users can navigate to [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and view their models and assets. To view Catalog, you must have a [read-only seat](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). #### Prerequisites To upload artifacts, make sure you meet these prerequisites: * Your organization is on a [dbt Enterprise+ plan](https://www.getdbt.com/pricing) * Your dbt platform account is on [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and your project is configured for the [latest supported version](https://docs.getdbt.com/docs/dbt-versions.md) dbt. * [Configured](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#connect-project-in-dbt-cloud) a hybrid project in dbt. * Updated your existing dbt v1 project with latest changes and [configured it with model access](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#make-dbt-models-public): * Ensure models that you want to share with other dbt projects use `access: public` in their model configuration. This makes the models more discoverable and shareable * Learn more about [access modifier](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers) and how to set the [`access` config](https://docs.getdbt.com/reference/resource-configs/access.md) * Update [dbt permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to create a new project in dbt **Note:** Uploading artifacts doesn't count against dbt run slots. --- ### About Iceberg catalogs A technical data catalog is a metadata management layer that enables users and tools to programmatically discover, understand, and govern data assets that are available across multiple data platforms and query engines. #### Background In the early 2010s, with the introduction of [Hadoop](https://hadoop.apache.org/) and data lakes, [Hive Metastore](https://hive.apache.org/) became the standard for managing schema metadata in Hadoop ecosystems. While effective in supporting early distributed query engines (separated compute and storage), such as Apache Spark and Trino, the Hive Metastore is limited in the kinds of structural metadata it can support, and it's generally slower and costlier than modern technical catalogs. In recent years, the emergence and consolidation around the [Iceberg standard](https://iceberg.apache.org/terms/) has led to the evolution of existing data catalogs (such as [Unity Catalog](https://www.unitycatalog.io/)) or the creation of new open source catalogs (including [Polaris](https://polaris.apache.org/) and [Lakekeeper](https://docs.lakekeeper.io/)). It's important to note that **technical data catalogs** serve a different purpose from **business catalogs**, although both benefit data teams: * **Technical data catalogs:** Focus on structural metadata, including information about data like table and column names, data types, storage locations (particularly important for open table formats), and access controls. They can be built into an existing data platform (no setup needed) — such as Databricks Unity Catalog or Snowflake Horizon Catalog — or externally managed and integrated with each query engine. Those engines use the catalog to locate, read, and write data. * **Business data catalogs:** Serve broader organizational users (such as BI analysts and product managers). They enrich technical metadata with business context in the form of metrics, business definitions, data quality indicators, usage patterns, and ownership. #### Why data catalogs are important to dbt For dbt users working in a lakehouse or multi-engine architecture, data catalogs can serve two purposes: * **Table discovery:** dbt models are materialized in an open table format (often Iceberg) and registered in a catalog. Understanding the catalog structure is critical for managing datasets and informing dbt about what has already been built and where it resides. * **Cross-engine interoperability:** Iceberg catalogs allow datasets created by one compute engine to be read by another, without replication. This is the foundation for [cross-platform dbt Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh.md). Without a catalog, each Iceberg table's metadata needs to be registered with the query engine individually, requiring many additional `create iceberg table` and `alter ... refresh` queries. With a catalog, each table's metadata is registered and refreshed automatically, and there's a single endpoint to ask about every table. Over the past year, data platform vendors have been adding support for catalog "linking" or "federation," which is when the data platform manages the synchronization of metadata between external data catalogs and its managed data catalog. This means that an Iceberg table written by one query engine into one catalog is automatically available for reading by another query engine. See: [Snowflake catalog-linked databases](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked), [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation), [AWS Glue catalog federation](https://docs.aws.amazon.com/lake-formation/latest/dg/federated-catalog-data-connection.html), [BigQuery catalog federation](https://docs.cloud.google.com/lakehouse/docs/use-catalog-federation). #### How dbt works with Iceberg catalogs dbt uses [Iceberg catalogs defined in `catalogs.yml`](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) in order to: * **Materialize models:** When dbt materializes a model as a table or view, if the catalog integration is declared, the underlying adapter (such as Spark, Trino, or Snowflake) creates an Iceberg table entry in the specified catalog, both built-in or external. * **Resolve references:** When dbt reads a model materialized to a catalog, it uses the "physical" location defined by the active adapter (new spec) or write integration (old spec). This enables referencing (reading) a model (Iceberg table) in one query engine that was originally materialized (written) by another one, so long as both engines are integrated to the same catalog. #### Limitations Many data platforms, query engines, and data catalogs have added substantially more support for Iceberg standards over the past few years — but it's still not a guarantee that any given data warehouse can interoperate with any given data catalog. When it does exist, that support can be limited. For example, most Iceberg catalogs only support tables (`table` + `incremental` materialization in dbt), not views. They may not support atomic `create table as` statements, meaning that dbt must update tables using multiple statements without ACID guarantees. Benoit (DX advocate at dbt Labs) created a [useful visualization](https://b-per.github.io/iceberg-roulette/#matrix) (also available as a [fun interactive game](https://b-per.github.io/iceberg-roulette/)) to show the current state of warehouse / catalog interoperability. (See something that looks wrong or out-of-date? Open an issue or PR in [the repo](https://github.com/b-per/iceberg-roulette).) --- ### About incremental models This is an introduction on incremental models, when to use them, and how they work in dbt. Incremental models in dbt is a [materialization](https://docs.getdbt.com/docs/build/materializations.md) strategy designed to efficiently update your data warehouse tables by only transforming and loading new or changed data since the last run. Instead of processing your entire dataset every time, incremental models append or update only the new rows, significantly reducing the time and resources required for your data transformations. This page will provide you with a brief overview of incremental models, their importance in data transformations, and the core concepts of incremental materializations in dbt. [![A visual representation of how incremental models work. Source: Materialization best practices guide (/best-practices/materializations/1-guide-overview)](/img/docs/building-a-dbt-project/incremental-diagram.jpg?v=2 "A visual representation of how incremental models work. Source: Materialization best practices guide (/best-practices/materializations/1-guide-overview)")](#)A visual representation of how incremental models work. Source: Materialization best practices guide (/best-practices/materializations/1-guide-overview) Learn by video! For video tutorials on Incremental models, go to dbt Learn and check out the [Incremental models course](https://learn.getdbt.com/courses/incremental-models). Build idempotent incremental models Incremental models are stateful, so they're the easiest place to accidentally break [idempotence](https://docs.getdbt.com/best-practices/idempotence.md) — the expectation that re-running a model produces the same result. Before you configure one, review [Idempotence in dbt](https://docs.getdbt.com/best-practices/idempotence.md#idempotence-and-incremental-models). #### Understand incremental models Incremental models enable you to significantly reduce the build time by just transforming new records. This is particularly useful for large datasets, where the cost of processing the entire dataset is high. Incremental models [require extra configuration](https://docs.getdbt.com/docs/build/incremental-models.md) and are an advanced usage of dbt. We recommend using them when your dbt runs are becoming too slow. ##### When to use an incremental model Building models as tables in your data warehouse is often preferred for better query performance. However, using `table` materialization can be computationally intensive, especially when: * Source data has millions or billions of rows. * Data transformations on the source data are computationally expensive (take a long time to execute) and complex, like when using Regex or UDFs. Incremental models offer a balance between complexity and improved performance compared to `view` and `table` materializations and offer better performance of your dbt runs. In addition to these considerations for incremental models, it's important to understand their limitations and challenges, particularly with large datasets. For more insights into efficient strategies, performance considerations, and the handling of late-arriving data in incremental models, refer to the [On the Limits of Incrementality](https://discourse.getdbt.com/t/on-the-limits-of-incrementality/303) discourse discussion or to our [Materialization best practices](https://docs.getdbt.com/best-practices/materializations/2-available-materializations.md) page. ##### How incremental models work in dbt dbt's [incremental materialization strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) works differently on different databases. Where supported, a `merge` statement is used to insert new records and update existing records. On warehouses that do not support `merge` statements, a merge is implemented by first using a `delete` statement to delete records in the target table that are to be updated, and then an `insert` statement. Transaction management, a process used in certain data platforms, ensures that a set of actions is treated as a single unit of work (or task). If any part of the unit of work fails, dbt will roll back open transactions and restore the database to a good state. #### Related docs * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) to learn how to configure incremental models in dbt. * [Incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) to understand how dbt implements incremental models on different databases. * [Microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) to understand a new incremental strategy intended for efficient and resilient processing of very large time-series datasets. * [Materializations best practices](https://docs.getdbt.com/best-practices/materializations/1-guide-overview.md) to learn about the best practices for using materializations in dbt. * [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md) to decide when an incremental model is the right CDC pattern. --- ### About incremental strategy Incremental strategies for materializations optimize performance by defining how to handle new and changed data. There are various strategies to implement the concept of incremental materializations. The value of each strategy depends on: * The volume of data. * The reliability of your `unique_key`. * The support of certain features in your data platform. An optional `incremental_strategy` config is provided in some adapters that controls the code that dbt uses to build incremental models. Microbatch The [`microbatch` incremental strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) is intended for large time-series datasets. dbt will process the incremental model in multiple queries (or "batches") based on a configured `event_time` column. Depending on the volume and nature of your data, this can be more efficient and resilient than using a single query for adding new data. ##### Supported incremental strategies by adapter This table shows the support of each incremental strategy across adapters available on dbt v2 or dbt's [**v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Some strategies may be unavailable if you're not on **v1 Latest** and the feature hasn't been released to the **v1 Compatible** track. If you're interested in incremental strategies for additional adapters, check out the [adapter's individual configuration page](https://docs.getdbt.com/reference/resource-configs.md) for more details. Click the name of the adapter in the following table for more information about supported incremental strategies: | Data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | `microbatch` | | ----------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | --------------- | ------------------ | ------------ | | [dbt-postgres](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#incremental-materialization-strategies) | ✅ | ✅ | ✅ | | ✅ | | [dbt-redshift](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#incremental-materialization-strategies) | ✅ | ✅ | ✅ | | ✅ | | [dbt-bigquery](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#merge-behavior-incremental-models) | | ✅ | | ✅ | ✅ | | [dbt-spark](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#incremental-models) | ✅ | ✅ | | ✅ | ✅ | | [dbt-databricks](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#incremental-models) | ✅ | ✅ | ✅ | ✅ | ✅ | | [dbt-snowflake](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#merge-behavior-incremental-models) | ✅ | ✅ | ✅ | ✅ | ✅ | | [dbt-trino](https://docs.getdbt.com/reference/resource-configs/trino-configs.md#incremental) | ✅ | ✅ | ✅ | | ✅ | | [dbt-fabric](https://docs.getdbt.com/reference/resource-configs/fabric-configs.md#incremental) | ✅ | ✅ | ✅ | | ✅ | | [dbt-athena](https://docs.getdbt.com/reference/resource-configs/athena-configs.md#incremental-models) | ✅ | ✅ | | ✅ | ✅ | | [dbt-teradata](https://docs.getdbt.com/reference/resource-configs/teradata-configs.md#valid_history-incremental-materialization-strategy) | ✅ | ✅ | ✅ | | ✅ | | [dbt-duckdb](https://docs.getdbt.com/reference/resource-configs/duckdb-configs.md#incremental) | ✅ | ✅ | ✅ | | ✅ | | [dbt-clickhouse](https://docs.getdbt.com/reference/resource-configs/clickhouse-configs.md#incremental-models) | ✅ | | ✅ | ✅ | | ##### Configuring incremental strategy The `incremental_strategy` config can either be defined in specific models or for all models in your `dbt_project.yml` file: dbt\_project.yml ```yaml models: +incremental_strategy: "insert_overwrite" ``` or: models/my\_model.sql ```sql {{ config( materialized='incremental', unique_key='date_day', incremental_strategy='delete+insert', ... ) }} select ... ``` ##### Strategy-specific configs If you use the `merge` strategy and specify a `unique_key`, by default, dbt will entirely overwrite matched rows with new values. On adapters which support the `merge` strategy, you may optionally pass a list of column names to a `merge_update_columns` config. In that case, dbt will update *only* the columns specified by the config, and keep the previous values of other columns. models/my\_model.sql ```sql {{ config( materialized = 'incremental', unique_key = 'id', merge_update_columns = ['email', 'ip_address'], ... ) }} select ... ``` Alternatively, you can specify a list of columns to exclude from being updated by passing a list of column names to a `merge_exclude_columns` config. models/my\_model.sql ```sql {{ config( materialized = 'incremental', unique_key = 'id', merge_exclude_columns = ['created_at'], ... ) }} select ... ``` ##### About incremental\_predicates `incremental_predicates` is an advanced use of incremental models, where data volume is large enough to justify additional investments in performance. This config accepts a list of any valid SQL expression(s). dbt does not check the syntax of the SQL statements. This an example of a model configuration in a `yml` file you might expect to see on Snowflake: ```yml models: - name: my_incremental_model config: materialized: incremental unique_key: id # this will affect how the data is stored on disk, and indexed to limit scans cluster_by: ['session_start'] incremental_strategy: merge # this limits the scan of the existing table to the last 7 days of data incremental_predicates: ["DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)"] # `incremental_predicates` accepts a list of SQL statements. # `DBT_INTERNAL_DEST` and `DBT_INTERNAL_SOURCE` are the standard aliases for the target table and temporary table, respectively, during an incremental run using the merge strategy. ``` Alternatively, here are the same configurations configured within a model file: ```sql -- in models/my_incremental_model.sql {{ config( materialized = 'incremental', unique_key = 'id', cluster_by = ['session_start'], incremental_strategy = 'merge', incremental_predicates = [ "DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)" ] ) }} ... ``` This will template (in the `dbt.log` file) a `merge` statement like: ```sql merge into <existing_table> DBT_INTERNAL_DEST from <temp_table_with_new_records> DBT_INTERNAL_SOURCE on -- unique key DBT_INTERNAL_DEST.id = DBT_INTERNAL_SOURCE.id and -- custom predicate: limits data scan in the "old" data / existing table DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date) when matched then update ... when not matched then insert ... ``` Limit the data scan of *upstream* tables within the body of their incremental model SQL, which will limit the amount of "new" data processed/transformed. ```sql with large_source_table as ( select * from {{ ref('large_source_table') }} {% if is_incremental() %} where session_start >= dateadd(day, -3, current_date) {% endif %} ), ... ``` info The syntax depends on how you configure your `incremental_strategy`: * If using the `merge` strategy, you may need to explicitly alias any columns with either `DBT_INTERNAL_DEST` ("old" data) or `DBT_INTERNAL_SOURCE` ("new" data). * There's a decent amount of conceptual overlap with the `insert_overwrite` incremental strategy. ##### Built-in strategies Before diving into [custom strategies](#custom-strategies), it's important to understand the built-in incremental strategies in dbt and their corresponding macros: | `incremental_strategy` | Corresponding macro | | ------------------------------------------------------------------------------------------------- | -------------------------------------- | | [`append`](https://docs.getdbt.com/docs/build/incremental-strategy.md#append) | `get_incremental_append_sql` | | [`delete+insert`](https://docs.getdbt.com/docs/build/incremental-strategy.md#deleteinsert) | `get_incremental_delete_insert_sql` | | [`merge`](https://docs.getdbt.com/docs/build/incremental-strategy.md#merge) | `get_incremental_merge_sql` | | [`insert_overwrite`](https://docs.getdbt.com/docs/build/incremental-strategy.md#insert_overwrite) | `get_incremental_insert_overwrite_sql` | | [`microbatch`](https://docs.getdbt.com/docs/build/incremental-strategy.md#microbatch) | `get_incremental_microbatch_sql` | For example, a built-in strategy for the `append` can be defined and used with the following files: macros/append.sql ```sql {% macro get_incremental_append_sql(arg_dict) %} {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %} {% endmacro %} {% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %} {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%} insert into {{ target_relation }} ({{ dest_cols_csv }}) ( select {{ dest_cols_csv }} from {{ temp_relation }} ) {% endmacro %} ``` Define a model models/my\_model.sql: ```sql {{ config( materialized="incremental", incremental_strategy="append", ) }} select * from {{ ref("some_model") }} ``` ###### About built-in incremental strategies ###### `append` The `append` strategy is simple to implement and has low processing costs. It inserts selected records into the destination table without updating or deleting existing data. This strategy doesn’t align directly with type 1 or type 2 [slowly changing dimensions](https://www.thoughtspot.com/data-trends/data-modeling/slowly-changing-dimensions-in-data-warehouse) (SCD). It differs from SCD1, which overwrites existing records, and only loosely resembles SCD2. While it adds new rows (like SCD2), it doesn’t manage versioning or track historical changes explicitly. Importantly, `append` doesn't check for duplicates or verify whether a record already exists in the destination. If the same record appears multiple times in the source, it will be inserted again, potentially resulting in duplicate rows. This may not be an issue depending on your use case and data quality requirements. ###### `delete+insert` The `delete+insert` strategy deletes the data for the `unique_key` from the target table and then inserts the data for those with a `unique_key`, which may be less efficient for larger datasets. It ensures updated records are fully replaced, avoiding partial updates and can be useful when a `unique_key` isn't truly unique or when `merge` is unsupported. `delete+insert` doesn't map directly to SCD logic (type 1 or 2) because it overwrites data at the row level and while it can add new rows, it does not track history. For SCD2, use [dbt snapshots](https://docs.getdbt.com/docs/build/snapshots.md#what-are-snapshots), not `delete+insert`. ###### `merge` `merge` inserts records with a `unique_key` that don’t exist yet in the destination table and updates records with keys that do exist — mirroring the logic of SCD1, where changes are overwritten rather than historically tracked. This strategy shouldn't be confused with `delete+insert` which deletes matching records before inserting new ones. By specifying a `unique_key` (which can be composed of one or more columns), `merge` can also help resolve duplicates. If the `unique_key` already exists in the destination table, `merge` will update the record, so you won't have duplicates. If the records don’t exist, `merge` will insert them. Note, if you use `merge` without specifying a `unique_key`, it behaves like the `append` strategy. While the `merge` strategy is useful for keeping tables current, it's best suited for smaller tables or incremental datasets. It can be expensive for large tables because it scans the entire destination table to determine what to update or insert. ###### `insert_overwrite` The [`insert_overwrite`](https://downloads.apache.org/spark/docs/3.1.1/sql-ref-syntax-dml-insert-overwrite-table.html) strategy is used to efficiently update partitioned tables by replacing entire partitions with new data, rather than merging or updating individual rows. It overwrites only the affected partitions, not the whole table. Because it is designed for partitioned data and replaces entire partitions wholesale, it does not align with typical SCD logic, which tracks row-level history or changes. It's ideal for tables partitioned by date or another key and useful for refreshing recent or corrected data without full table rebuilds. ###### `microbatch` [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md#what-is-microbatch-in-dbt) is an incremental strategy designed for processing large time-series datasets by splitting the data into time-based batches (for example, daily or hourly). It supports [parallel batch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md#how-parallel-batch-execution-works) for faster runs. For details on which incremental strategies are supported by each adapter, refer to the section [Supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter). ##### Custom strategies limited support Custom strategies are not currently supported on the BigQuery and Spark adapters. From dbt v1.2 and onwards, users have an easier alternative to [creating an entirely new materialization](https://docs.getdbt.com/guides/create-new-materializations.md). They define and use their own "custom" incremental strategies by: 1. Defining a macro named `get_incremental_STRATEGY_sql`. Note that `STRATEGY` is a placeholder and you should replace it with the name of your custom incremental strategy. 2. Configuring `incremental_strategy: STRATEGY` within an incremental model. dbt won't validate user-defined strategies, it will just look for the macro by that name, and raise an error if it can't find one. For example, a user-defined strategy named `insert_only` can be defined and used with the following files: macros/my\_custom\_strategies.sql ```sql {% macro get_incremental_insert_only_sql(arg_dict) %} {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %} {% endmacro %} {% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %} {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%} insert into {{ target_relation }} ({{ dest_cols_csv }}) ( select {{ dest_cols_csv }} from {{ temp_relation }} ) {% endmacro %} ``` models/my\_model.sql ```sql {{ config( materialized="incremental", incremental_strategy="insert_only", ... ) }} ... ``` If you use a custom microbatch macro, use the [`require_batched_execution_for_custom_microbatch_strategy` behavior flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) in your `dbt_project.yml` to control batched execution. Set it to `true` to opt in before the flag matures. After the flag matures (default: `true`), set it to `false` to revert to single-invocation behavior. ##### Custom strategies from a package To use the `merge_null_safe` custom incremental strategy from the `example` package: * [Install the package](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) * Add the following macro to your project: macros/my\_custom\_strategies.sql ```sql {% macro get_incremental_merge_null_safe_sql(arg_dict) %} {% do return(example.get_incremental_merge_null_safe_sql(arg_dict)) %} {% endmacro %} ``` --- ### About Lake Compute Private beta ### About Lake Compute [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Lake Compute is in private beta To request access or region support, fill out the [Lake Compute signup form](https://docs.google.com/forms/d/e/1FAIpQLSdWXHU3VFqq1t8GVAA2BMfY_AfeVJH26OXJFlNwXiVv4pcuXQ/viewform). Lake Compute is in active development and likely to change. Breaking changes may occur, documentation may be incomplete, technical support may be limited, and no service-level agreement covers it during beta. Refer to [Private beta limitations](https://docs.getdbt.com/docs/lake-compute.md#private-beta-limitations) before you plan real work around it. Lake Compute is a hosted, serverless compute engine for running dbt models. It is powered by DuckDB and Apache Iceberg, and it is intended to support transformation workloads that do not need the full scale or feature set of a distributed cloud data warehouse. Lake Compute supports two execution models: 1. As a "sidecar" alongside your existing data warehouse, such as Snowflake or Databricks. You can choose the execution engine model by model while keeping one dbt project, one DAG, and one `ref` graph. 2. As the primary compute adapter for new dbt projects, along with an Iceberg catalog (such as Fivetran MDLS). Lake Compute doesn't intend to fully replace your cloud data warehouse, and it isn't a general-purpose query engine. Use Lake Compute for dbt transformations only. Keep workloads that need high concurrency, BI connectivity, advanced data warehouse features (such as ML or AI functions), or distributed compute on your primary data warehouse. #### Why use Lake Compute Optionality and flexibility. Simpler and smaller transformations can run on Lake Compute, while your warehouse remains available for workloads that need distributed processing, high concurrency, proprietary functions, or advanced analytics. Lake Compute always reads and writes data in the Iceberg table format, so it’s easy to add in other compute engines. #### Prerequisites To set up Lake Compute, you need: * Enrollment in the Lake Compute private beta. Request access with the [Lake Compute signup form](https://docs.google.com/forms/d/e/1FAIpQLSdWXHU3VFqq1t8GVAA2BMfY_AfeVJH26OXJFlNwXiVv4pcuXQ/viewform). The **Compute** tab doesn't appear on your MDLS destination until your account is enrolled. * A Fivetran account. You can create a new Fivetran account, and use it indefinitely, since there is not yet billing for MDLS or Lake Compute; or trial full features (including connectors) for 14 days. * An existing Fivetran Managed Data Lake Service (MDLS) destination — which includes an MDLS Iceberg catalog (Polaris) and an Amazon S3 storage bucket — already in a supported AWS region, or create a new one with a storage bucket in a supported AWS region. Lake Compute, your storage bucket, and (if using Lake Compute as "sidecar") your primary data must all be in the same region. Lake Compute currently supports AWS regions `us-east-1` and `us-west-2`. If you need a region that isn’t listed, complete the [sign-up form](https://docs.google.com/forms/d/e/1FAIpQLSdWXHU3VFqq1t8GVAA2BMfY_AfeVJH26OXJFlNwXiVv4pcuXQ/viewform) to request it. * Permission to configure the required cloud storage, catalog integration, warehouse access, authentication, and—where applicable—network policy allowlisting. After you complete setup, you need a dbt project running on **dbt v2**. You also need time to identify suitable models, adapt SQL where necessary, and validate results. ##### Supported warehouses Snowflake is the only warehouse supported during private beta. Early Databricks support is available to a limited number of design partners. To evaluate Lake Compute with Databricks, fill out the [Lake Compute signup form](https://docs.google.com/forms/d/e/1FAIpQLSdWXHU3VFqq1t8GVAA2BMfY_AfeVJH26OXJFlNwXiVv4pcuXQ/viewform) and name Databricks as your platform. In "sidecar" execution, Lake Compute runs alongside your warehouse, reading from and writing/propagating back to the warehouse’s managed Iceberg catalog (Snowflake Horizon or Databricks Unity). ##### Required permissions Setup requires permissions to: * Create a cloud storage bucket * Create or configure the MDLS destination in your Fivetran account Depending on your data warehouse, you may also need to: * Establish the warehouse catalog integration, and grant catalog integration access to the user or role running dbt * Configure key-pair or other required programmatic authentication * Allow Lake Compute egress if your warehouse account enforces network policies For Snowflake, some setup steps require `ACCOUNTADMIN` or equivalent delegated privileges, such as `NETWORK_ADMIN` for network policy changes. Always refer to the [latest setup guide](https://docs.getdbt.com/docs/lake-compute/compute-onboarding.md), which may have changed since you initially set up Lake Compute. ##### Required environment variable dbt gates Lake Compute behind an experimental opt-in. dbt refuses to run any Lake Compute configuration unless you set: ```bash export DBT_ENGINE_EXPERIMENTAL_MULTI_ADAPTER=true ``` One variable gates every surface that selects an adapter: | Gated | Notes | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `type: lakecompute` in `profiles.yml` | Required for **both** the "sidecar" pattern and Lake Compute as your default adapter | | The `adapter` config on a node | Applies to models, seeds, snapshots, analyses, functions, and data tests. dbt refuses **any** value while the gate is off, not just `lakecompute` | | The `--adapter` command-line flag | — | For the dbt platform, set this as an environment variable on the environment. Refer to [Onboarding to Lake Compute](https://docs.getdbt.com/docs/lake-compute/compute-onboarding.md#dbt-platform). #### Private beta limitations The following are not supported or are intentionally limited during private beta: * Projects running on dbt v1 * JDBC or ODBC access and direct BI tool connections. * Role-based access control (RBAC) on Iceberg tables within MDLS written by Lake Compute. * Warehouse-native AI SQL functions and other proprietary warehouse features. * Automatic SQL dialect translation. * Distributed or multi-node execution. * Large two-sided joins and other workloads that require distributed processing. * GCP and Azure storage. * BigQuery integration. * AWS regions other than the currently supported beta regions. Additional capabilities and regions may be introduced over time, but roadmap items and dates are not commitments. ##### Which dbt framework features does Lake Compute support during private beta? You can: * Build dbt models with simple materializations: * `table`, which is the default * `incremental`, with `incremental_strategy: append` or `insert_overwrite` (along with `partition_by`) * Use `ephemeral` models * Execute data tests, using Lake Compute, against models built with Lake Compute * Preview with `dbt show` * `dbt show --select MODEL_NAME --adapter lakecompute` * `dbt show --inline SQL_QUERY --adapter lakecompute` Not yet supported: * `view` materialization * Grants * `persist_docs` * Pre-hook / post-hook * Unit tests #### Related docs * [Lake Compute setup](https://docs.getdbt.com/docs/lake-compute/compute-onboarding.md) * [Lake Compute FAQs](https://docs.getdbt.com/docs/lake-compute/compute-faq.md) --- ### About MetricFlow This guide introduces MetricFlow's fundamental ideas for people new to this feature. MetricFlow, which powers the Semantic Layer, helps you define and manage the logic for your company's metrics. It's an opinionated set of abstractions and helps data consumers retrieve metric datasets from a data platform quickly and efficiently. MetricFlow handles SQL query construction and defines the specification for dbt semantic models and metrics. It allows you to define metrics in your dbt project and query them with [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) whether in dbt or dbt v1. #### Prerequisites Before you start, consider the following guidelines: (Applies to dbt v1.12 and later) * Define metrics in YAML and query them using the [latest metric specifications](https://docs.getdbt.com/docs/build/semantic-models.md). * Available on the [dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) or [dbt Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform. * Use MetricFlow with Snowflake, BigQuery, Databricks, Postgres (dbt v1 only), or Redshift. * Discover insights and query your metrics using the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and its diverse range of [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). #### MetricFlow MetricFlow is a SQL query generation tool designed to streamline metric creation across different data dimensions for diverse business needs. * It operates through YAML files, where a semantic graph links language to data. This graph comprises [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) (data entry points) and [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) (functions for creating quantitative indicators). * MetricFlow is developed and maintained by dbt Labs and works with the [Apache Ossie](https://github.com/apache/ossie) format. Starting in dbt v1.12, you can also define semantic models using [Ossie documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md) as an alternative to dbt's native YAML configuration. * MetricFlow is compatible with dbt version 1.6 and higher. * MetricFlow is distributed under the [Apache 2.0 license](https://github.com/dbt-labs/metricflow/blob/main/LICENSE). Data practitioners and enthusiasts are highly encouraged to contribute. Read more about [MetricFlow's license history](https://github.com/dbt-labs/metricflow?tab=readme-ov-file#license-history). * As a part of the Semantic Layer, MetricFlow empowers organizations to define metrics using YAML abstractions. * To query metric dimensions, dimension values, and validate configurations, use [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md). note MetricFlow doesn't support dbt [builtin functions or packages](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md) at this time, however, support is planned for the future. MetricFlow abides by these principles: * **Flexibility with completeness**: Define metric logic using flexible abstractions on any data model. * **DRY (Don't Repeat Yourself)**: Minimize redundancy by enabling metric definitions whenever possible. * **Simplicity with gradual complexity:** Approach MetricFlow using familiar data modeling concepts. * **Performance and efficiency**: Optimize performance while supporting centralized data engineering and distributed logic ownership. ##### Semantic graph We're introducing a new concept: a "semantic graph". It's the relationship between semantic models and YAML configurations that creates a data landscape for building metrics. You can think of it like a map, where tables are like locations, and the connections between them (edges) are like roads. Although it's under the hood, the semantic graph is a subset of the DAG, and you can see the semantic models as nodes on the DAG. The semantic graph helps us decide which information is available to use for consumption and which is not. The connections between tables in the semantic graph are more about relationships between the information. This is different from the DAG, where the connections show dependencies between tasks. When MetricFlow generates a metric, it uses its SQL engine to figure out the best path between tables using the framework defined in YAML files for semantic models and metrics. When these models and metrics are correctly defined, they can be used downstream with Semantic Layer's integrations. ##### Semantic models Semantic models are the starting points of your data and correspond to models in your dbt project. You can create multiple semantic models from each model. Semantic models have metadata, like a data table, that define important information such as the table name and primary keys for the graph to be navigated correctly. For a semantic model, there are three main pieces of metadata: * [Entities](https://docs.getdbt.com/docs/build/entities.md): The join keys of your semantic model (think of these as the traversal paths, or edges between semantic models). * [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md): These are the ways you want to group or slice/dice your metrics. (Applies to dbt v1.12 and later) * [Simple metrics](https://docs.getdbt.com/docs/build/simple.md): Metrics that directly reference a single column expression within a semantic model, without any additional columns involved. ##### Metrics (Applies to dbt v1.12 and later) Metrics, which is a key concept, are functions that combine simple metrics, constraints, or other mathematical functions to define new quantitative indicators. MetricFlow uses various aggregation types, such as average, sum, and count distinct, to create metrics. Dimensions add context to metrics and without them, a metric is simply a number for all time. You can define metrics in the same YAML files as your semantic models, or create a new file. MetricFlow supports different metric types: (Applies to dbt v1.12 and later) * [Conversion](https://docs.getdbt.com/docs/build/conversion.md): Tracks when a base event and a subsequent conversion event occurs for an entity within a set time period. * [Cumulative](https://docs.getdbt.com/docs/build/cumulative.md): Aggregates a simple metric over a given window. * [Derived](https://docs.getdbt.com/docs/build/derived.md): Defines a metric as an expression of other metrics, which allows you to do calculations on top of metrics. * [Ratio](https://docs.getdbt.com/docs/build/ratio.md): Defines a metric as the ratio of two simple metrics, such as revenue per customer. * [Simple](https://docs.getdbt.com/docs/build/simple.md): Defines a metric that directly references a single column expression within a semantic model. #### Use case In the upcoming sections, we'll show how data practitioners currently calculate metrics and compare it to how MetricFlow makes defining metrics easier and more flexible. The following example data is based on the Jaffle Shop repo. You can view the complete [dbt project](https://github.com/dbt-labs/jaffle-sl-template). The tables we're using in our example model are: * `orders` is a production data platform export that has been cleaned up and organized for analytical consumption * `customers` is a partially denormalized table in this case with a column derived from the orders table through some upstream process To make this more concrete, consider the metric `order_total`, which is defined using the SQL expression: `select sum(order_total) as order_total from orders` This expression calculates the total revenue for all orders by summing the `order_total` column in the orders table. In a business setting, the metric `order_total` is often calculated according to different categories, such as: * Time, for example `date_trunc(ordered_at, 'day')` * Order Type, using `is_food_order` dimension from the `orders` table ##### Calculate metrics Next, we'll compare how data practitioners currently calculate metrics with multiple queries versus how MetricFlow simplifies and streamlines the process. ##### Calculate with multiple queries The following example displays how data practitioners typically would calculate the `order_total` metric aggregated. It's also likely that analysts are asked for more details on a metric, like how much revenue came from new customers. Using the following query creates a situation where multiple analysts working on the same data, each using their own query method — this can lead to confusion, inconsistencies, and a headache for data management. ```sql select date_trunc('day',orders.ordered_at) as day, case when customers.first_ordered_at is not null then true else false end as is_new_customer, sum(orders.order_total) as order_total from orders left join customers on orders.customer_id = customers.customer_id group by 1, 2 ``` ##### Calculate with MetricFlow In the following three example tabs, use MetricFlow to define a semantic model that uses `order_total` as a metric and a sample schema to create consistent and accurate results — eliminating confusion, code duplication, and streamlining your workflow. ##### Revenue example (Applies to dbt v1.12 and later) In this example, a simple metric named `order_total` is defined on the `orders` model and semantic model. The metric sums the `order_total` column. The time dimension `metric_time` provides daily granularity and can be rolled up to weekly or monthly periods. Additionally, the `customers` semantic model defines a derived categorical dimension `is_new_customer`, which returns `true` when `first_ordered_at` is `not null` and `false` otherwise. ```yaml models: - name: orders # The name of the model semantic_model: enabled: true name: orders_semantic_model agg_time_dimension: metric_time # Default aggregation time dimension columns: # Primary entity - order_id - name: order_id description: "Primary key for orders table" entity: type: primary name: order_id label: "Order ID" # Foreign entity - customer - name: customer_id description: "Foreign key linking to customers" entity: type: foreign name: customer label: "Customer" # Time dimension - metric_time - name: ordered_at granularity: day dimension: type: time label: "Order Date" description: "Date when the order was placed" metrics: # Simple metric for order total revenue - name: order_total description: "Total revenue from orders" label: "Order Total Revenue" type: simple agg: sum expr: order_total - name: customers # The customers model with semantic layer constructs defined semantic_model: enabled: true name: customers_semantic_model agg_time_dimension: first_ordered_at columns: # Primary entity - customer - name: customer_id description: "Primary key for customers table" entity: type: primary name: customer label: "Customer" # Time dimension - first_ordered_at - name: first_ordered_at description: "Date of customer's first order" granularity: day dimension: type: time name: first_ordered_at label: "First Order Date" ``` ##### More dimensions example Similarly, you can add additional dimensions like `is_food_order` to your semantic models to incorporate even more dimensions to slice and dice your revenue `order_total`. (Applies to dbt v1.12 and later) ```yaml models: - name: orders # The name of the semantic model semantic_model: enabled: true name: orders_semantic_model agg_time_dimension: metric_time # Default aggregation time dimension columns: # Primary entity - order_id - name: order_id description: "Primary key for orders table" entity: type: primary name: order_id label: "Order ID" # Foreign entity - customer - name: customer_id description: "Foreign key linking to customers" entity: type: foreign name: customer label: "Customer" # Time dimension - metric_time - name: ordered_at description: "Date when the order was placed" granularity: day dimension: type: time name: metric_time label: "Order Date" # Categorical dimension - is_food_order - name: is_food_order description: "Indicates if this is a food order" dimension: type: categorical name: is_food_order label: "Is Food Order" metrics: # Simple metric for order total revenue - name: order_total description: "Total revenue from orders" label: "Order Total Revenue" type: simple agg: sum expr: order_total ``` ##### Advanced example Imagine an even more complex metric is needed, such as the amount of money earned each day from food orders from returning customers. Without MetricFlow, the data practitioner's original SQL might look like this: ```sql select date_trunc('day',orders.ordered_at) as day, sum(case when is_food_order = true then order_total else null end) as food_order, sum(orders.order_total) as sum_order_total, food_order/sum_order_total from orders left join customers on orders.customer_id = customers.customer_id where case when customers.first_ordered_at is not null then true else false end = true group by 1 ``` MetricFlow simplifies the SQL process through metric YAML configurations as shown below. You can also commit them to your git repository to ensure everyone on the data and business teams can see and approve them as the true and only source of information. (Applies to dbt v1.12 and later) ```yaml models: - name: orders semantic_model: enabled: true name: orders_semantic_model agg_time_dimension: ordered_at # Default aggregation time dimension columns: # Primary entity - order_id - name: order_id description: "Primary key for orders table" entity: type: primary name: order_id label: "Order ID" # Foreign entity - customer - name: customer_id description: "Foreign key linking to customers" entity: type: foreign name: customer label: "Customer" # Time dimension - ordered_at - name: ordered_at description: "Date when the order was placed" granularity: day dimension: type: time label: "Order Date" # Categorical dimension - is_food_order - name: is_food_order description: "Indicates if this is a food order" dimension: type: categorical name: is_food_order label: "Is Food Order" metrics: # Simple metric for total order revenue - name: order_total description: "Total revenue from orders" label: "Order Total Revenue" type: simple agg: sum expr: order_total # Simple metric for food order revenue - name: food_revenue description: "Revenue from food orders only" label: "Food order revenue" type: simple agg: sum expr: "case when is_food_order = true then order_total else 0 end" # Simple metric for count of distinct customers in orders - name: total_customers description: "Count of unique customers with orders" label: "Total customers" type: simple agg: count_distinct expr: customer_id - name: customers # The name of the second semantic model semantic_model: enabled: true name: customers_semantic_model agg_time_dimension: first_ordered_at columns: # Primary entity - customer - name: customer_id description: "Primary key for customers table" entity: type: primary name: customer label: "Customer" # Time dimension - first_ordered_at - name: first_ordered_at description: "Date of customer's first order" granularity: day dimension: type: time name: first_ordered_at label: "First Order Date" metrics: - name: food_revenue_per_customer description: "Revenue from food orders from returning customers" label: "Food % of order total" type: ratio numerator: name: food_revenue denominator: name: total_customers ``` #### FAQs  Do my datasets need to be normalized? Not at all! While a cleaned and well-modeled dataset can be extraordinarily powerful and is the ideal input, you can use any dataset from raw to fully denormalized datasets. It's recommended that you apply quality data consistency, such as filtering bad data, normalizing common objects, and data modeling of keys and tables, in upstream applications. The Semantic Layer is more efficient at doing data denormalization instead of normalization. If you have not invested in data consistency, that is okay. The Semantic Layer can take SQL queries or expressions to define consistent datasets.  Why is normalized data the ideal input? MetricFlow is built to do denormalization efficiently. There are better tools to take raw datasets and accomplish the various tasks required to build data consistency and organized data models. On the other end, by putting in denormalized data you are potentially creating redundancy which is technically challenging to manage, and you are reducing the potential granularity that MetricFlow can use to aggregate metrics.  How does the dbt Semantic Layer handle joins? The dbt Semantic Layer, powered by MetricFlow, builds joins based on the types of keys and parameters that are passed to entities. To better understand how joins are constructed, see the documentation on [join types](https://docs.getdbt.com/docs/build/join-logic.md#types-of-joins). Rather than capturing arbitrary join logic, MetricFlow captures the types of each identifier and then helps users navigate to appropriate joins. This allows us to avoid the construction of fan out and chasm joins as well as generate legible SQL.  Are entities and join keys the same thing? If it helps you to think of entities as join keys, that is very reasonable. Entities in MetricFlow have applications beyond joining two tables, such as acting as a dimension.  Can a table without a primary or unique entities have dimensions? Yes, but because a dimension is considered an attribute of the primary or unique entity of the table, they are only usable by the metrics that are defined in that table. They cannot be joined to metrics from other tables. This is common in event logs. #### Related docs * [Joins](https://docs.getdbt.com/docs/build/join-logic.md) * [Validations](https://docs.getdbt.com/docs/build/validation.md) --- ### About microbatch incremental models Use microbatch incremental models to process large time-series datasets efficiently. info Available for [dbt **v1 Latest**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and dbt v1.9 or higher. If you use a custom microbatch macro, set a [distinct behavior flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) in your `dbt_project.yml` to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the [microbatch strategy](#how-microbatch-compares-to-other-incremental-strategies). Read and participate in the discussion: [Core#10672](https://github.com/dbt-labs/dbt/discussions/10672). Refer to [Supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for a list of supported adapters. #### What is "microbatch" in dbt? Incremental models in dbt are a [materialization](https://docs.getdbt.com/docs/build/materializations.md) designed to efficiently update your data warehouse tables by only transforming and loading *new or changed data* since the last run. Instead of reprocessing an entire dataset every time, incremental models process a smaller number of rows, and then append, update, or replace those rows in the existing table. This can significantly reduce the time and resources required for your data transformations. Microbatch is an incremental strategy designed for large time-series datasets: * It relies solely on a time column ([`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md)) to define time-based ranges for filtering. * Set the `event_time` column for your microbatch model and its direct parents (upstream models). Note that this differs from `partition_by`, which groups rows into partitions. Required For incremental microbatch models, if your upstream models don't have `event_time` configured, dbt *cannot* automatically filter them during batch processing and will perform full table scans on every batch run. To avoid this, configure `event_time` on every upstream model that should be filtered. Learn how to exclude a model from auto-filtering by [opting out of auto-filtering](https://docs.getdbt.com/docs/build/incremental-microbatch.md#opting-out-of-auto-filtering). * It complements, rather than replaces, existing incremental strategies by focusing on efficiency and simplicity in batch processing. * Unlike traditional incremental strategies, microbatch enables you to [reprocess failed batches](https://docs.getdbt.com/docs/build/incremental-microbatch.md#retry), auto-detect [parallel batch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md), and eliminate the need to implement complex conditional logic for [backfilling](#backfills). * Note that microbatch might not be the best [strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) for all use cases. Consider other strategies for use cases such as not having a reliable `event_time` column or if you want more control over the incremental logic. Read more in [How `microbatch` compares to other incremental strategies](#how-microbatch-compares-to-other-incremental-strategies). #### How microbatch works When dbt runs a microbatch model — whether for the first time, during incremental runs, or in specified backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` and `batch_size` you configure. Each "batch" corresponds to a single bounded time period (by default, a single day of data). Where other incremental strategies operate only on "old" and "new" data, microbatch models treat every batch as an atomic unit that can be built or replaced on its own. Each batch is independent and idempotent. This is a powerful abstraction that makes it possible for dbt to run batches [separately](#backfills), concurrently, and [retry](#retry) them independently. ##### Adapter-specific behavior dbt's microbatch strategy uses the most efficient mechanism available for "full batch" replacement on each adapter. This can vary depending on the adapter: * `dbt-postgres`: Uses the `merge` strategy, which performs "update" or "insert" operations. * `dbt-redshift`: Uses the `delete+insert` strategy, which "inserts" or "replaces." * `dbt-snowflake`: Uses the `delete+insert` strategy, which "inserts" or "replaces." * `dbt-bigquery`: Uses the `insert_overwrite` strategy, which "inserts" or "replaces." * `dbt-spark`: Uses the `insert_overwrite` strategy, which "inserts" or "replaces." * `dbt-databricks`: Uses the `replace_where` strategy, which "inserts" or "replaces." * `dbt-athena` (Hive): Uses the `insert_overwrite` strategy, which "inserts" or "replaces." * `dbt-athena` (Iceberg): Uses the `merge` strategy, which performs "update" or "insert" operations. Check out the [supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for more info. #### Example A `sessions` model aggregates and enriches data that comes from two other models: * `page_views` is a large, time-series table. It contains many rows, new records almost always arrive after existing ones, and existing records rarely update. It uses the `page_view_start` column as its `event_time`. * `customers` is a relatively small dimensional table. Customer attributes update often, and not in a time-based manner — that is, older customers are just as likely to change column values as newer customers. The customers model doesn't configure an `event_time` column. As a result: * Each batch of `sessions` will filter `page_views` to the equivalent time-bounded batch. * The `customers` table isn't filtered, resulting in a full scan for every batch. tip In addition to configuring `event_time` for the target table, you should also specify it for any upstream models that you want to filter, even if they have different time columns. models/staging/page\_views.yml ```yaml models: - name: page_views config: event_time: page_view_start ``` We run the `sessions` model for October 1, 2024, and then again for October 2. It produces the following queries: ##### Model definition The [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) for the `sessions` model is set to `session_start`, which marks the beginning of a user’s session on the website. This setting allows dbt to combine multiple page views (each tracked by their own `page_view_start` timestamps) into a single session. This way, `session_start` differentiates the timing of individual page views from the broader timeframe of the entire user session. models/sessions.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', event_time='session_start', begin='2020-01-01', batch_size='day' ) }} with page_views as ( -- this ref will be auto-filtered select * from {{ ref('page_views') }} ), customers as ( -- this ref won't select * from {{ ref('customers') }} ) select page_views.id as session_id, page_views.page_view_start as session_start, customers.* from page_views left join customers on page_views.customer_id = customers.id ``` ##### Compiled (Oct 1, 2024) target/compiled/sessions.sql ```sql with page_views as ( select * from ( -- filtered on configured event_time select * from "analytics"."page_views" where page_view_start >= '2024-10-01 00:00:00' -- Oct 1 and page_view_start < '2024-10-02 00:00:00' ) ), customers as ( select * from "analytics"."customers" ), ... ``` ##### Compiled (Oct 2, 2024) target/compiled/sessions.sql ```sql with page_views as ( select * from ( -- filtered on configured event_time select * from "analytics"."page_views" where page_view_start >= '2024-10-02 00:00:00' -- Oct 2 and page_view_start < '2024-10-03 00:00:00' ) ), customers as ( select * from "analytics"."customers" ), ... ``` dbt will instruct the data platform to take the result of each batch query and [insert, update, or replace](#adapter-specific-behavior) the contents of the `analytics.sessions` table for the same day of data. To perform this operation, dbt will use the most efficient atomic mechanism for "full batch" replacement that is available on each data platform. For details, see [How microbatch works](#how-microbatch-works). It does not matter whether the table already contains data for that day. Given the same input data, the resulting table is the same no matter how many times a batch is reprocessed. [![Each batch of sessions filters page\_views to the matching time-bound batch, but doesn't filter customers, performing a full scan for each batch.](/img/docs/building-a-dbt-project/microbatch/microbatch_filters.png?v=2 "Each batch of sessions filters page_views to the matching time-bound batch, but doesn't filter customers, performing a full scan for each batch.")](#)Each batch of sessions filters page\_views to the matching time-bound batch, but doesn't filter customers, performing a full scan for each batch. #### Relevant configs Several configurations are relevant to microbatch models, and some are required: | Config | Description | Default | Type | Required | | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- | -------- | | [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) | The column indicating "at what time did the row occur." Required for your microbatch model and any direct parents that should be filtered. | N/A | Column | Required | | [`begin`](https://docs.getdbt.com/reference/resource-configs/begin.md) | The "beginning of time" for the microbatch model. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches (it's a leap year!) plus the batch for "today." | N/A | Date | Required | | [`batch_size`](https://docs.getdbt.com/reference/resource-configs/batch-size.md) | The granularity of your batches. Supported values are `hour`, `day`, `month`, and `year` | N/A | String | Required | | [`lookback`](https://docs.getdbt.com/reference/resource-configs/lookback.md) | Process X batches prior to the latest bookmark to capture late-arriving records. | `1` | Integer | Optional | | [`concurrent_batches`](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md) | Overrides dbt's auto detect for running batches concurrently (at the same time). Read more about [configuring concurrent batches](https://docs.getdbt.com/docs/build/parallel-batch-execution.md#configure-concurrent_batches). Setting to<br />\* `true` runs batches concurrently (in parallel).<br />\* `false` runs batches sequentially (one after the other). | `None` | Boolean | Optional | [![The event\_time column configures the real-world time of this record](/img/docs/building-a-dbt-project/microbatch/event_time.png?v=2 "The event_time column configures the real-world time of this record")](#)The event\_time column configures the real-world time of this record ##### Required configs for specific adapters Some adapters require additional configurations for the microbatch strategy. This is because each adapter implements the microbatch strategy differently. The following table lists the required configurations for the specific adapters, in addition to the standard microbatch configs: | Adapter | `unique_key` config | `partition_by` config | | ------------------------------------------------------------------------------------------------------------------------------- | ------------------- | --------------------- | | [`dbt-postgres`](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#incremental-materialization-strategies) | ✅ Required | N/A | | [`dbt-spark`](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#incremental-models) | N/A | ✅ Required | | [`dbt-bigquery`](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#merge-behavior-incremental-models) | N/A | ✅ Required | For example, if you're using `dbt-postgres`, configure `unique_key` as follows: models/sessions.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', unique_key='sales_id', ## required for dbt-postgres event_time='transaction_date', begin='2023-01-01', batch_size='day' ) }} select sales_id, transaction_date, customer_id, product_id, total_amount from {{ source('sales', 'transactions') }} ``` In this example, `unique_key` is required because `dbt-postgres` microbatch uses the `merge` strategy, which needs a `unique_key` to identify which rows dbt should merge in the data warehouse. Without a `unique_key`, dbt can't match rows between the incoming batch and the existing table. ##### Full refresh As a best practice, we recommend [configuring `full_refresh: false`](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) on microbatch models so that they ignore invocations with the `--full-refresh` flag. Note that running `dbt run --full-refresh` on a microbatch model by itself won't reset or reload data unless you have a `begin` datetime config for the model. If you need to reprocess historical data, we recommend using a targeted backfill with `--event-time-start` and `--event-time-end`. You must configure both for the full refresh to successfully run. ```bash dbt run --full-refresh --event-time-start "2024-01-01" --event-time-end "2024-02-01" ``` #### Usage **You must write your model query to process (read and return) exactly one "batch" of data**. This is a simplifying assumption and a powerful one: * You don’t need to think about `is_incremental` filtering * You don't need to pick among DML strategies (upserting/merging/replacing) * You can preview your model, and see the exact records for a given batch that will appear when that batch is processed and written to the table When you run a microbatch model, dbt will evaluate which batches need to be loaded, break them up into a SQL query per batch, and load each one independently. dbt will automatically filter upstream inputs (`source` or `ref`) that define `event_time`, based on the `lookback` and `batch_size` configs for this model. Note that dbt doesn't know the minimum `event_time` in your data — it only uses the configs you provide (like `begin`, `lookback`) to decide which batches to run. If you want to process data from the actual start of your dataset, you *must* explicitly define it using the `begin` config or the `--event-time-start` flag. During standard incremental runs, dbt will process batches according to the current timestamp and the configured `lookback`, with one query per batch. [![Configure a lookback to reprocess additional batches during standard incremental runs](/img/docs/building-a-dbt-project/microbatch/microbatch_lookback.png?v=2 "Configure a lookback to reprocess additional batches during standard incremental runs")](#)Configure a lookback to reprocess additional batches during standard incremental runs ###### Opting out of auto-filtering If there's an upstream model that configures `event_time`, but you *don't* want the reference to it to be filtered, you can specify `ref('upstream_model').render()` to opt out of auto-filtering. This isn't generally recommended — most models that configure `event_time` are fairly large, and if you don't filter the reference, each batch performs a full scan of this input table. #### Backfills Whether to fix erroneous source data or retroactively apply a change in business logic, you may need to reprocess a large amount of historical data. Backfilling a microbatch model is as simple as selecting it to run or build, and specifying a "start" and "end" for `event_time`. Note that `--event-time-start` and `--event-time-end` are mutually necessary, meaning that if you specify one, you must specify the other. As always, dbt will process the batches between the start and end as independent queries. ```bash dbt run --event-time-start "2024-09-01" --event-time-end "2024-09-04" ``` [![Configure a lookback to reprocess additional batches during standard incremental runs](/img/docs/building-a-dbt-project/microbatch/microbatch_backfill.png?v=2 "Configure a lookback to reprocess additional batches during standard incremental runs")](#)Configure a lookback to reprocess additional batches during standard incremental runs #### Retry If one or more of your batches fail, you can use `dbt retry` to reprocess *only* the failed batches. ![Partial retry](https://github.com/user-attachments/assets/f94c4797-dcc7-4875-9623-639f70c97b8f) #### Timezones For now, dbt assumes that all values supplied are in UTC: * `event_time` * `begin` * `--event-time-start` * `--event-time-end` While we may consider adding support for custom time zones in the future, we also believe that defining these values in UTC makes everyone's lives easier. #### How microbatch compares to other incremental strategies As data warehouses roll out new operations for concurrently replacing/upserting data partitions, we may find that the new operation for the data warehouse is more efficient than what the adapter uses for microbatch. In such instances, we reserve the right to update the default operation for microbatch, so long as it works as intended/documented for models that fit the microbatch paradigm. Most incremental models rely on the end user (you) to explicitly tell dbt what "new" means, in the context of each model, by writing a filter in an `{% if is_incremental() %}` conditional block. You are responsible for crafting this SQL in a way that queries [`{{ this }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md) to check when the most recent record was last loaded, with an optional look-back window for late-arriving records. Other incremental strategies will control *how* the data is being added into the table — whether append-only `insert`, `delete` + `insert`, `merge`, `insert overwrite`, etc — but they all have this in common. As an example: ```sql {{ config( materialized='incremental', incremental_strategy='delete+insert', unique_key='date_day' ) }} select * from {{ ref('stg_events') }} {% if is_incremental() %} -- this filter will only be applied on an incremental run -- add a lookback window of 3 days to account for late-arriving records where date_day >= (select {{ dbt.dateadd("day", -3, "max(date_day)") }} from {{ this }}) {% endif %} ``` For this incremental model: * "New" records are those with a `date_day` greater than the maximum `date_day` that has previously been loaded * The lookback window is 3 days * When there are new records for a given `date_day`, the existing data for `date_day` is deleted and the new data is inserted Let’s take our same example from before, and instead use the new `microbatch` incremental strategy: models/staging/stg\_events.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', event_time='event_occurred_at', batch_size='day', lookback=3, begin='2020-01-01', full_refresh=false ) }} select * from {{ ref('stg_events') }} -- this ref will be auto-filtered ``` Where you’ve also set an `event_time` for the model’s direct parents - in this case, `stg_events`: models/staging/stg\_events.yml ```yaml models: - name: stg_events config: event_time: my_time_field ``` And that’s it! When you run the model, each batch templates a separate query. For example, if you were running the model on October 1, dbt would template separate queries for each day between September 28 and October 1, inclusive — four batches in total. The query for `2024-10-01` would look like: target/compiled/staging/stg\_events.sql ```sql select * from ( select * from "analytics"."stg_events" where my_time_field >= '2024-10-01 00:00:00' and my_time_field < '2024-10-02 00:00:00' ) ``` Based on your data platform, dbt will choose the most efficient atomic mechanism to insert, update, or replace these four batches (`2024-09-28`, `2024-09-29`, `2024-09-30`, and `2024-10-01`) in the existing table. --- ### About model governance dbt supports model governance to help you control who can access models, what data they contain, how they change over time, and reference them across projects. dbt supports model governance in dbt v1 and the dbt platform, with some differences in the features available across environments/plans. * Use model governance to define model structure and visibility in dbt v1 and the dbt platform. * dbt builds on this with features like [cross-project ref](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) that enable collaboration at scale across multiple projects, powered by its metadata service and [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). Available in dbt Enterprise or Enterprise+ plans. All of the following features are available in dbt v1 and the dbt platform, *except* project dependencies, which is only available to [dbt Enterprise-tier plans](https://www.getdbt.com/pricing). * [**Model access**](https://docs.getdbt.com/docs/mesh/govern/model-access.md) — Mark models as "public" or "private" to distinguish between mature data products and implementation details — and to control who can `ref` each. * [**Model contracts**](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) —Guarantee the shape of a model (column names, data types, constraints) before it builds, to prevent surprises for downstream data consumers. * [**Model versions**](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) — When a breaking change is unavoidable, provide a smoother upgrade pathway and deprecation window for downstream data consumers. * [**Model namespaces**](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) — Organize models into [groups](https://docs.getdbt.com/docs/build/groups.md) and [packages](https://docs.getdbt.com/docs/build/packages.md) to delineate ownership boundaries. Models in different packages can share the same name, and the `ref` function can take the project/package namespace as its first argument. * [**Project dependencies**](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) — Resolve references to public models in other projects ("cross-project ref") using an always-on stateful metadata service, instead of importing all models from those projects as packages. Each project serves data products (public model references) while managing its own implementation details, enabling an [enterprise data mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md). [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Supporting data freshness SLAs Use [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) and the [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) config to govern data freshness at the model level, helping your team align with freshness Service Level Agreements (SLAs) without unnecessary rebuilds. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. --- ### About profiles.yml Local development If you're using dbt from the command line, you need a `profiles.yml` file that contains the connection details for your data platform. dbt platform accounts dbt platform projects don't require a profiles.yml file unless you're developing from your local machine instead of the cloud-based UI. #### About profiles.yml The `profiles.yml` file stores database connection credentials and configuration for dbt projects, including: * **Connection details** — Account identifiers, hosts, ports, and authentication credentials. * **Target definitions** — Define different environments (dev, staging, prod) within a single profile. * **Default target** — Set which environment to use by default. * **Execution parameters** — Thread count, timeouts, and retry settings. * **Credential separation** — Keep sensitive information out of version control. The `profile` field in [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md) references a profile name defined in `profiles.yml`. #### Location of profiles.yml Only one `profiles.yml` file is required and it can manage multiple projects and connections. ##### dbt v2 dbt v2 searches for the parent directory of `profiles.yml` in the following order and uses the first location it finds: 1. `--profiles-dir` flag — Override for CI/CD or testing. 2. Project root directory — Project-specific credentials. 3. `~/.dbt/` directory (Recommended location) — Shared across all projects. ##### dbt v1 dbt v1 searches for the parent directory of `profiles.yml` in the following order and uses the first location it finds: (Applies to dbt v1.11 and later) 1. `--profiles-dir` flag 2. `DBT_ENGINE_PROFILES_DIR` environment variable 3. `DBT_PROFILES_DIR` environment variable (legacy variable but supported for backward compatibility) 4. Current working directory 5. `~/.dbt/` directory (Recommended location) **Note:** dbt v1 prefers `DBT_ENGINE_PROFILES_DIR` for the profiles directory, which aligns with the `DBT_ENGINE_*` env var naming in v1.11. Use `DBT_ENGINE_PROFILES_DIR` going forward; `DBT_PROFILES_DIR` remains supported for compatibility. Note: dbt v1 supports using the (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable or a `profiles.yml` file in the current working directory. These options aren't currently supported in dbt v2. `~/.dbt/profiles.yml` is the recommended location for the following reasons: * **Security** — Keeps credentials out of project directories and version control. * **Reusability** — A single file for all dbt projects on the machine. * **Separation** — Connection details don't travel with project code. ###### When should I use project root? Place your `profiles.yml` file in the project root directory for: * Self-contained demo or tutorial projects. * Docker containers with baked-in credentials. * CI/CD pipelines with environment-specific configs. #### Create and configure the `profiles.yml` file The easiest way to create and configure a `profiles.yml` file is to execute `dbt init` after you've installed dbt on your machine. This takes you through the process of configuring an adapter and places the file into the recommended `~/.dbt/` location. If your project has an existing `profiles.yml` file, running `dbt init` will prompt you to amend or overwrite it. If you select the existing adapter for configuration, dbt will automatically populate the existing values. You can also manually create the file and add it to the proper location. To configure an adapter manually, copy and paste the fields from the adapter setup instructions for [dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md) or [dbt v2](https://docs.getdbt.com/docs/local/profiles.yml.md) along with the appropriate values for each. ##### Example configuration To set up your profile, copy the correct sample profile for your warehouse into your `profiles.yml` file and update the details as follows: * Profile name: Replace the name of the profile with a sensible name – it’s often a good idea to use the name of your organization. Make sure that this is the same name as the `profile` indicated in your `dbt_project.yml` file. * `target`: This is the default target your dbt project will use. It must be one of the targets you define in your profile. Commonly it is set to `dev`. * Populating your `outputs`: * `type`: The type of data warehouse you are connecting to * Warehouse credentials: Get these from your database administrator if you don’t already have them. Remember that user credentials are very sensitive information that should not be shared. May include fields like `account`, `username`, and `password`. * `schema`: The default schema that dbt will build objects in. * `threads`: The number of threads the dbt project will run on. The following example highlights the format of the `profiles.yml` file. Note that many of the configs are adapter-specific and their syntax varies. \~/.dbt/profiles.yml ```yml my_project_profile: # Profile name (matches dbt_project.yml) target: dev # Default target to use outputs: dev: # Development environment type: adapter_type # Required: snowflake, bigquery, databricks, redshift, postgres, etc # Connection identifiers (placeholder examples, see adapter-specific pages for supported configs) account: abc123 database: docs_team schema: dev_schema # Authentication (adapter-specific) auth_method: username_password username: username password_credentials: password # Execution settings (common across adapters) threads: 4 # Number of parallel threads # Multiple profiles (for multiple projects) my_second_project_profile: target: dev outputs: dev: type: snowflake # Example adapter account: account user: user password: password database: database schema: schema warehouse: warehouse threads: 4 ``` ##### Environment variables Use environment variables to keep sensitive credentials out of your `profiles.yml` file. Check out the [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) reference for more information. Example: \~/.dbt/profiles.yml ```yml my_profile: target: dev outputs: dev: type: ADAPTER_NAME account: "{{ env_var("ADAPTER_ACCOUNT") }}" user: "{{ env_var("ADAPTER_USER") }}" password: "{{ env_var("ADAPTER_PASSWORD") }}" database: "{{ env_var("ADAPTER_DATABASE") }}" schema: "{{ env_var("ADAPTER_SCHEMA") }}" warehouse: "{{ env_var("ADAPTER_WAREHOUSE") }}" role: "{{ env_var("ADAPTER_ROLE") }}" threads: 4 ``` When using dbt locally, you can also store environment variables in a `.env` file in your project root instead of setting them directly in your shell. dbt, the dbt VS Code extension, and dbt v1.12+ automatically load the `.env` file from your current working directory. Environment variables set in your shell take precedence over values in the `.env` file. For more information, refer to [About env\_var function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md#using-the-env-file). To keep credentials out of version control, add `.env` to your `.gitignore` file — new projects on v1.12 and higher created with `dbt init` include this by default. #### User config You can set default values of global configs for all projects that you run using your local machine. Refer to [About global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) for details. #### Understanding targets in profiles dbt supports multiple targets within one profile to encourage the use of separate development and production environments as discussed in [dbt environments](https://docs.getdbt.com/docs/local/dbt-environments.md). A typical profile for an analyst using dbt locally will have a target named `dev`, and have this set as the default. You may also have a `prod` target within your profile, which creates the objects in your production schema. However, since it's often desirable to perform production runs on a schedule, we recommend deploying your dbt project to a separate machine other than your local machine. Most dbt users only have a `dev` target in their profile on their local machine. If you do have multiple targets in your profile, and want to use a target other than the default, you can do this using the `--target` flag when running a dbt command. For example, to run against your `prod` target instead of the default `dev` target: ```bash dbt run --target prod ``` You can use the `--target` flag with any dbt command, such as: ```bash dbt build --target prod dbt test --target dev dbt compile --target qa ``` ##### Overriding profiles and targets When running dbt commands, you can specify which profile and target to use from the CLI using the `--profile` and `--target` [flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). These flags override what’s defined in your `dbt_project.yml` as long as the specified profile and target are already defined in your `profiles.yml` file. To run your dbt project with a different profile or target than the default, you can do so using the followingCLI flags: * `--profile` flag — Overrides the profile set in `dbt_project.yml` by pointing to another profile defined in `profiles.yml`. * `--target` flag — Specifies the target within that profile to use (as defined in `profiles.yml`). These flags help when you're working with multiple profiles and targets and want to override defaults without changing your files. ```bash dbt run --profile my-profile-name --target dev ``` In this example, the `dbt run` command will use the `my-profile-name` profile and the `dev` target. #### Understanding warehouse credentials We recommend that each dbt user has their own set of database credentials, including a separate user for production runs of dbt – this helps debug rogue queries, simplifies ownerships of schemas, and improves security. To ensure the user credentials you use in your target allow dbt to run, you will need to ensure the user has appropriate privileges. While the exact privileges needed varies between data warehouses, at a minimum your user must be able to: * Read source data * Create schemas¹ * Read system tables Running dbt without create schema privileges If your user is unable to be granted the privilege to create schemas, your dbt runs should instead target an existing schema that your user has permission to create relations within. #### Understanding target schemas The target schema represents the default schema that dbt will build objects into, and is often used as the differentiator between separate environments within a warehouse. Schemas in BigQuery dbt uses the term "schema" in a target across all supported warehouses for consistency. Note that in the case of BigQuery, a schema is actually a dataset. The schema used for production should be named in a way that makes it clear that it is ready for end-users to use for analysis – we often name this `analytics`. In development, a pattern we’ve found to work well is to name the schema in your `dev` target `dbt_<username>`. Suffixing your name to the schema enables multiple users to develop in dbt, since each user will have their own separate schema for development, so that users will not build over the top of each other, and ensuring that object ownership and permissions are consistent across an entire schema. Note that there’s no need to create your target schema beforehand – dbt will check if the schema already exists when it runs, and create it if it doesn’t. While the target schema represents the default schema that dbt will use, it may make sense to split your models into separate schemas, which can be done by using [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). #### Understanding threads When dbt runs, it creates a directed acyclic graph (DAG) of links between models. The number of threads represents the maximum number of paths through the graph dbt may work on at once – increasing the number of threads can minimize the run time of your project. The default value for threads in user profiles is 4 threads. For more information, check out [using threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). #### Related docs * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) * [Connection profiles](https://docs.getdbt.com/docs/local/connection-profiles.md) --- ### About state-aware orchestration Private preview ### About state-aware orchestration [Private preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Every time a job runs, state-aware orchestration automatically determines which models to build by detecting changes in code or data. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). important dbt v2 is available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our dbt Community Slack channel [`#dbt-v2`](https://getdbt.slack.com/archives/C088YCAB6GH). State-aware orchestration saves you compute costs and reduces runtime because when a job runs, it checks for new records and only builds the models that will change. [![dbt v2 powered state-aware orchestration](/img/docs/deploy/sao.gif?v=2 "dbt v2 powered state-aware orchestration")](#)dbt v2 powered state-aware orchestration We built dbt's state-aware orchestration on these four core principles: * **Real-time shared state:** All jobs write to a real-time shared model-level state, allowing dbt to rebuild only changed models regardless of which jobs the model is built in. * **Model-level queueing:** Jobs queue up at the model-level so you can avoid any 'collisions' and prevent rebuilding models that were just updated by another job. * **State-aware and state agnostic support:** You can build jobs dynamically (state-aware) or explicitly (state-agnostic). Both approaches update shared state so everything is kept in sync. * **Sensible defaults:** State-aware orchestration works out-of-the-box (natively), with an optional configuration setting for more advanced controls. For more information, refer to [state-aware advanced configurations](https://docs.getdbt.com/docs/deploy/state-aware-setup.md#advanced-configurations). note State-aware orchestration does not depend on [static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md#principles-of-static-analysis) and works even when `static_analysis` is disabled. #### Optimizing builds with state-aware orchestration State-aware orchestration uses shared state tracking to determine which models need to be built by detecting changes in code or data every time a job runs. It also supports custom refresh intervals and custom source freshness configurations, so dbt only rebuilds models when they're actually needed. For example, you can configure your project so that dbt skips rebuilding the `dim_wizards` model (and its parents) if they’ve already been refreshed within the last 4 hours, even if the job itself runs more frequently. Without configuring anything, dbt's state-aware orchestration automatically knows to build your models either when the code has changed or if there’s any new data in a source (or upstream model in the case of [dbt Mesh](https://docs.getdbt.com/docs/mesh/about-mesh.md)). **Note:** When a model fails a [data test](https://docs.getdbt.com/docs/build/data-tests.md), state-aware orchestration rebuilds it on subsequent runs instead of reusing it from prior state. This ensures dbt reevaluates models with unresolved data quality issues. ##### Handling concurrent jobs Not supported in dbt State Concurrent job handling works differently in dbt State. For a full list of behavioral differences, refer to [Known differences from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md#known-differences-from-state-aware-orchestration). If two separate jobs both depend on the same downstream model (for example, `model_ab`) and both detect upstream changes (`updates_on = any`), `model_ab` could run twice — once for each job. However, if `model_ab` was already built and nothing has changed since that build, neither job will rebuild it. Instead, both jobs will reuse the existing version instead of rebuilding. Under state-aware orchestration, all jobs read and write from the same shared state and build a model only when either the code or data state has changed. This means that each job individually evaluates whether a model needs rebuilding based on the model’s compiled code and upstream data state. What happens when jobs overlap: * If both jobs reach the same model at exactly the same time, one job waits until the other finishes. This is to prevent collisions in the data warehouse when two jobs try to build the same model at the same time. * After the first job finishes building the model, the second job still checks whether a rebuild is needed. If there are new data or code changes to incorporate, the second job builds the model again. If there are no changes and building the model would produce the same result, the second job reuses the model. To prevent a job from being built too frequently even when the code or data state has changed, you can reduce build frequency by using the `build_after` config. For information on how to use `build_after`, refer to [Model freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md) and [Advanced configurations](https://docs.getdbt.com/docs/deploy/state-aware-setup.md#advanced-configurations). ##### Handling deleted tables State-aware orchestration detects and rebuilds models when their tables are deleted in the warehouse, even if there are no code or data changes. When a table is deleted in the warehouse: * dbt raises a warning that the expected table is missing. * The affected model is queued for rebuild during the current run, even if there are no code or data changes. This behavior ensures consistency between the dbt state and the actual warehouse state. It also reduces the need to manually clear cache or disable state-aware orchestration when models are modified outside of dbt. #### Efficient testing in state-aware orchestration [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Private beta feature State-aware orchestration features in the dbt platform are only available in dbt v2. Contact your account manager to enable dbt v2 in your account. Data quality can get degraded in two ways: * New code changes definitions or introduces edge cases. * New data, like duplicates or unexpected values, invalidates downstream metrics. Running dbt’s out-of-the-box [data tests](https://docs.getdbt.com/docs/build/data-tests.md) (`unique`, `not_null`, `accepted_values`, `relationships`) on every build helps catch data errors before they impact business decisions. Catching these errors often requires having multiple tests on every model and running tests even when not necessary. If nothing relevant has changed, repeated test executions don’t improve coverage and only increase cost. With dbt v2, dbt gains an understanding of the SQL code based on the logical plan for the compiled code. dbt then can determine when a test must run again, or when a prior upstream test result can be reused. Efficient testing in state-aware orchestration reduces warehouse costs by avoiding redundant data tests and combining multiple tests into one run. This feature includes two optimizations: * **Test reuse** — Tests are reused in cases where no logic in the code or no new data could have changed the test's outcome. * **Test aggregation** — When there are multiple tests on a model, dbt combines tests to run as a single query against the warehouse, rather than running separate queries for each test. Currently, Efficient testing is only available in deploy jobs, not in continuous integration (CI) or merge jobs. ##### Supported data tests The following tests can be reused when Efficient testing is enabled: * [`unique`](https://docs.getdbt.com/reference/resource-properties/data-tests.md#unique) * [`not_null`](https://docs.getdbt.com/reference/resource-properties/data-tests.md#not_null) * [`accepted_values`](https://docs.getdbt.com/reference/resource-properties/data-tests.md#accepted_values) ##### Enabling Efficient testing Before enabling Efficient testing, make sure you have configured [`static_analysis`](https://docs.getdbt.com/docs/build/about-static-analysis.md#configuring-static_analysis). To enable Efficient testing: 1. From the main menu, go to **Orchestration** > **Jobs**. 2. Select your deploy job. Go to your job settings and click **Edit**. 3. Under **Enable dbt v2 cost optimization features**, expand **More options**. 4. Select **Efficient testing**. This feature is disabled by default. 5. Click **Save**. ##### Example In the following query, you’re joining an `orders` and a `customers` table: ```sql with orders as ( select * from {{ ref('orders') }} ), customers as ( select * from {{ ref('customers') }} ), joined as ( select customers.customer_id as customer_id, orders.order_id as order_id from customers left join orders on orders.customer_id = customers.customer_id ) select * from joined ``` * `not_null` test: A `left join` can introduce null values for customers without orders. Even if upstream tests verified `not_null(order_id)` in orders, the join can create null values downstream. dbt must always run a `not_null` test on `order_id` in this joined result. * `unique` test: If `orders.order_id` and `customers.customer_id` are unique upstream, uniqueness of `order_id` is preserved and the upstream result can be reused. ##### Limitations The following section lists some considerations when using Efficient testing in state-aware-orchestration: * **Aggregated tests do not support custom configs**. Tests that include the following [custom config options](https://docs.getdbt.com/reference/data-test-configs.md) will run individually rather than as part of the aggregated batch: ```yaml config: fail_calc: <string> limit: <integer> severity: error | warn error_if: <string> warn_if: <string> store_failures: true | false where: <string> ``` * **Efficient testing is available only in deploy jobs**. CI and merge jobs currently do not have the option to enable this feature. #### Related FAQs What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)** as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). ### How is state-aware orchestration different from using selectors in dbt? dbt platform | Enterprise, Enterprise+ In dbt v1, running with the selectors `state:modified+` and `source_status:fresher+` builds models that either: * Have changed since the prior run (`state:modified+`) * Have upstream sources that are fresher than in the prior run (`source_status:fresher+`) Instead of relying only on these selectors and prior-run artifacts, state-aware orchestration decides whether to rebuild a model based on: * Compiled SQL diffs that ignore non-meaningful changes like whitespace and comments * Upstream data changes at runtime and model-level freshness settings * Shared state across jobs While dbt v1 uses selectors like `state:modified+` and `source_status:fresher+` to decide what to build *only for a single run in a single job*, state-aware orchestration with dbt v2 maintains a *shared, real-time model state across every job in the environment* and uses that state to determine whether a model’s code or upstream data have actually changed before rebuilding. This ensures dbt only rebuilds models when something has changed, no matter which job runs them. #### Related docs * [State-aware orchestration configuration](https://docs.getdbt.com/docs/deploy/state-aware-setup.md) * [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [`freshness`](https://docs.getdbt.com/reference/resource-configs/freshness.md) * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### About static analysis dbt v2 [fully comprehends your project's SQL](https://docs.getdbt.com/blog/the-levels-of-sql-comprehension), enabling advanced capabilities like dialect-aware validation and precise column-level lineage. It can do this because its compilation step is more comprehensive than that of the dbt v1.x engine. When dbt v1.x referred to *compilation*, it only meant *rendering* — converting Jinja-templated strings into a SQL query to send to a database. dbt v2 can also render Jinja, but then it completes a second phase: *static analysis*, producing and validating a logical plan for every rendered query in the project. This step is the cornerstone of dbt v2's new capabilities. #### Principles of static analysis The software engineering concept of [static analysis](https://en.wikipedia.org/wiki/Static_program_analysis) describes checks that can be done on code before it runs (static == not running). The most rigorous static analysis means you can trust that if the analysis succeeds, the code will run in production without compilation errors. Less strict static analysis also surfaces helpful information to developers as they work. There's no free lunch—what you gain in responsiveness you lose in correctness guarantees. dbt v2 uses the [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) config to help you control how it performs static analysis for your models. dbt v2 is unique in that it can statically analyze not just a single model in isolation, but every query from one end of your DAG to the other. Even your database can only validate the query in front of it! Concepts like [information flow theory](https://roundup.getdbt.com/i/156064124/beyond-cll-information-flow-theory-and-metadata-propagation) — although not incorporated into the dbt platform [yet](https://www.getdbt.com/blog/where-we-re-headed-with-the-dbt-fusion-engine) — rely on stable inputs and the ability to trace columns DAG-wide. ##### Baseline mode: A smooth transition from dbt v1 dbt v2 defaults to `static_analysis: baseline` mode, inspired by similar type-checking and linting tools like [TypeScript's migration approach](https://www.typescriptlang.org/docs/handbook/migrating-from-javascript.html), [basedpyright's baseline feature](https://docs.basedpyright.com/latest/benefits-over-pyright/baseline/), and [Pydantic's strict/lax modes](https://docs.pydantic.dev/latest/why/#strict-lax). The philosophy behind the above-mentioned tools and dbt v2's baseline mode is: * **Smooth transition**: Provide a familiar first-time experience for users coming from dbt v1. * **Incremental opt-in**: Offer a clear pathway to adopt more dbt v2 features over time. * **Pragmatic validation**: Catch most SQL errors without requiring a complete project overhaul. Use this style of gradual typing to start with lightweight validation, then incrementally adopt strict guarantees as your project is ready. ###### What baseline mode changes Baseline mode introduces several fundamental behavior changes compared to the previous binary (off/on) approach: * **No downloading of remote schemas** — Baseline mode does not fetch schemas from the warehouse. * **Unit tests work without strict mode** — Previously, unit tests required static analysis to be fully on. In baseline mode, they work out of the box. * **No unsafe introspection warnings** — We no longer warn about unsafe introspection, though we'd still love to help you assess it in the future. The following table shows how baseline mode expands what's available without requiring strict mode. ##### LSP feature comparison Baseline mode unlocks a meaningful set of features without requiring strict mode. We're also investing in moving more features into baseline over time. VS Code extension features by static analysis configuration: ✅ = Available | ❌ = Not available | Feature | off | baseline | strict | | ---------------------------------------------- | --- | -------- | ------ | | Go-to-definition/reference (except columns) | ✅ | ✅ | ✅ | | Table lineage | ✅ | ✅ | ✅ | | YAML validation | ✅ | ✅ | ✅ | | Render + preview SQL | ✅ | ✅ | ✅ | | Unit tests | ✅ | ✅ | ✅ | | Detect syntax errors | ❌ | ✅ | ✅ | | Preview CTE results | ❌ | ✅ | ✅ | | Go-to-definition/reference (columns) | ❌ | ❌ | ✅ | | Automatic refactor column names | ❌ | ❌ | ✅ | | Column-level lineage | ❌ | ❌ | ✅ | | Detect data type and function signature errors | ❌ | ❌ | ✅ | * dbt VS Code extension features in this table are available to all users — what you get depends on your static analysis mode, not your account. Supported Snowflake functions To check out which Snowflake functions are supported in dbt v2 in `strict` mode, refer to [Snowflake function support](https://docs.getdbt.com/reference/resource-configs/snowflake-function-support.md) CodeLens visibility The VS Code extension and Studio IDE provide CodeLens even when static analysis is off, giving you visibility into which models have static analysis disabled and why. Ultimately, we want everyone developing in strict mode for maximum guarantees. We acknowledge this isn't a change that can happen overnight — baseline exists to smooth the transition. Many planned features (like local compute) require strict mode. We're also exploring inferring column types on your behalf, which would enable more functionality in baseline mode without requiring you to manually provide type information. ##### Introspection handling in baseline mode In `baseline` mode, all static analysis findings are warnings, not errors — your project can continue running even when the compiler flags invalid or problematic SQL. This section is a good example of why that design exists. Previously, with `strict` mode, the system assumed local schemas of your compiled models would be available. In `baseline` mode, we can no longer assume the full local schema is available and complete, so `baseline` uses the remote database as the source of truth — similar to dbt v1. The practical result is that the dbt v2 compiler may sometimes flag incorrect queries that result from introspective queries that come back empty. If you encounter this, you can: 1. Ignore the warning 2. Build the model locally 3. (Coming soon) Use `warn_error_options` to disable the warning For example, consider this query using the `dbt_utils.unpivot` macro: ```sql select * from ( {{ dbt_utils.unpivot( relation=ref('example_model'), cast_to='integer', exclude=['order_id', 'customer_id'], field_name='product_type', value_name='quantity' ) }} ) ``` If the introspection query fails or returns no results, this renders to: ```sql select * from ( ) ``` This is invalid SQL. In `baseline` mode, dbt v2 displays a warning so your project can continue running while still alerting you to the issue: ```bash dbt0101: no viable alternative at input '( )' --> models/example_model.sql:17:1 ``` ###### Migration scenarios Migrating to dbt v2 can involve more than moving YAML around. Some scenarios that can make migration more involved include: 1. **Limited access to sources**: You don't have access to all the sources and models of a large dbt project. 2. **Intricate Jinja workflows**: Your project uses post-hooks and introspection extensively. 3. **Package compatibility**: Your project depends on packages that aren't yet dbt v2-compatible. 4. **Unsupported SQL features**: Your models or sources use advanced data types (`STRUCT`, `ARRAY`, `GEOGRAPHY`) or built-in functions (`AI.PREDICT`, `JSON_FLATTEN`, `st_pointfromgeohash`) not yet supported by dbt v2. Setting `static_analysis` to `baseline` mode lets you start using dbt v2 immediately while you address these scenarios incrementally. As you resolve compatibility issues, you can opt specific models or your entire project into `strict` mode for maximum validation guarantees. #### Recapping the differences between engines dbt v1: * Renders and runs models one at a time. * Never runs static analysis. dbt v2 (baseline mode — default): * Statically analyzes all models, catching most SQL errors while providing a familiar migration experience. dbt v2 (strict mode): * Renders and statically analyzes all models before execution begins. * Guarantees nothing runs until the entire project is proven valid. * Parses `CREATE FUNCTION` in [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) and in [`on-run-start`](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) hooks, then registers those UDFs so strict compilation can resolve calls to them. `baseline` and `off` don't register UDFs this way. See [User-defined functions (UDFs) in `strict` mode](https://docs.getdbt.com/reference/resource-configs/static-analysis.md#user-defined-functions-udfs-in-strict-mode). #### Configuring `static_analysis` You can modify the way static analysis is applied for specific models in your project. The static analysis configuration cascades from most strict to least strict. Going downstream in your lineage, a model can keep the same mode or relax it — it can't be stricter than its parent. Setting `static_analysis: strict` on a model does not automatically set `strict` for downstream models; they keep the project default unless you set them explicitly. For rules and examples, refer to [How modes cascade in your lineage](#how-modes-cascade-in-your-lineage) and [strict mode inheritance](#strict-mode-inheritance). Models that use custom materializations follow these same rules, with one exception for `strict`. Refer to [Custom materializations](#custom-materializations). The [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) config options are: * `baseline` (default): Statically analyze SQL. This is the recommended starting point for users transitioning from dbt v1, providing a smooth migration experience while still catching most SQL errors. * `strict`: Statically analyze all SQL before execution begins. Use this for maximum validation guarantees — nothing runs until the entire project is proven valid. * `off`: Skip SQL analysis on this model and its descendants. When you disable static analysis, features of the VS Code extension which depend on SQL comprehension will be unavailable. The best place to configure `static_analysis` is as a config on an individual model or group of models. As a debugging aid, you can also use the [`--static-analysis strict`](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) or `--static-analysis off` CLI flags to override all model-level configuration. ##### Incrementally adopting strict mode Once you're comfortable with dbt v2 in baseline mode, you can incrementally opt models or directories into `strict` mode: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: # Start with strict analysis on your cleanest models staging: +static_analysis: strict # Keep baseline for models that need more work marts: +static_analysis: baseline ``` ###### How modes cascade in your lineage Two rules determine which mode a model can use: * **Eligibility rule:** A model is eligible for static analysis only if all of its upstream dependencies are eligible. * **Strictness rule:** A model can't be stricter than its parents. The strictness hierarchy is `strict` → `baseline` → `off`. Going downstream, a model can keep its parent's mode or relax it, but it can't tighten it: | Upstream model | Downstream can be | | -------------- | ---------------------------------- | | `strict` | `strict`, `baseline`, or `off` | | `baseline` | `baseline` or `off` (not `strict`) | | `off` | `off` only | The strictness rule exists because `baseline` doesn't produce the full analyzed schema that `strict` needs from its upstream models. Without that information, a downstream model can't run strict-level type checking. For the complete reference, refer to [How static analysis modes cascade](https://docs.getdbt.com/reference/resource-configs/static-analysis.md#how-static-analysis-modes-cascade). ###### Strict mode inheritance Unlike `baseline` or `off`, `strict` mode doesn't propagate to downstream models. If you configure a model as `strict`, its downstream models won't inherit `strict` mode unless you set them explicitly. To make all models `strict`, you must set `+static_analysis: strict` on root models first, or use the project-wide config in the next section at the project level. For example, in A → B → C with a default of `baseline`, configuring A (a root node) as `strict` makes only A `strict` — B and C remain baseline unless configured. To make the full chain `strict`, set `+static_analysis: strict` on each relevant model or group, or use a project-wide setting. This approach lets you gain the benefits of strict validation where possible while keeping the flexibility of baseline analysis for models that aren't yet compatible. Refer to [CLI options](https://docs.getdbt.com/reference/global-configs/command-line-options.md) and [Configurations and properties](https://docs.getdbt.com/reference/configs-and-properties.md) to learn more about configs. ##### Custom materializations Models that use a [custom materialization](https://docs.getdbt.com/guides/create-new-materializations.md) follow the same `static_analysis` rules as any other model, including the default `baseline` mode and the usual [cascading rules](#how-modes-cascade-in-your-lineage). This applies to both kinds of custom materializations: * **A name you invented:** such as `materialized='my_custom_load'`. Find these in your model configs. * **A built-in name:** such as your own macro named `materialization table, default`. These are harder to spot, because models that say `materialized='table'` look standard but run your code instead of dbt's. ###### Schema-changing custom materializations dbt v2 infers your model’s columns from its SQL, without looking inside your materialization code. Custom `create` statements, merge logic, or grants are fine *if* they only change how a model is built and they leave the columns unchanged. But if your materialization adds or renames columns, or changes their data types, dbt’s inferred schema won’t match the table in your warehouse. This mainly affects `strict`, which needs accurate upstream schemas for type checking, unlike `baseline`. If you use `strict`, set `static_analysis: off` on models that use a schema-changing materialization: models/my\_schema\_changing\_model.sql ```sql {{ config( materialized='my_custom_load', static_analysis='off' ) }} select ... ``` Because `off` cascades, every downstream model also loses static analysis. To keep coverage across your DAG, use schema-changing materializations on models that nothing else depends on. ##### Identify a model's mode The mode you configure isn't always the mode in effect, because a model's effective mode also depends on its parents. The [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) and the Studio IDE both show a CodeLens above each model that tells you when static analysis is off and why. Keep in mind that `dbt ls --output json --output-keys config.static_analysis` reports the mode you *configured* for each model, not the mode v2 resolves after applying the cascading rules. ##### Example configurations ###### Configure strict for the entire project Many teams want to enable `strict` mode for the whole project and all packages. You can do this by setting `+static_analysis: strict` under each resource type in `dbt_project.yml` for your project name (and for any package names if you want those to be strict too): dbt\_project.yml ```yaml models: my_project: +static_analysis: strict seeds: my_project: +static_analysis: strict snapshots: my_project: +static_analysis: strict tests: my_project: +static_analysis: strict unit_tests: my_project: +static_analysis: strict sources: my_project: +static_analysis: strict analyses: my_project: +static_analysis: strict ``` Use your project name in place of `my_project` — that's the same value as the `name:` key at the top of `dbt_project.yml` (for example, `jaffle_shop`). To apply `strict` to a package as well, add another entry under each resource type using the package name as the key; for example, under `models:` add `your_package_name:` with `+static_analysis: strict` beneath it. ###### Disable static analysis for all models in a package: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: marts: +materialized: table a_package_with_introspective_queries: +static_analysis: off ``` ###### Disable static analysis in YAML: models/my\_udf\_using\_model.yml ```yml models: - name: model_with_static_analysis_off config: static_analysis: off ``` ###### Disable static analysis for a model using a custom UDF: models/my\_udf\_using\_model.sql ```sql {{ config(static_analysis='off') }} select user_id, my_cool_udf(ip_address) as cleaned_ip from {{ ref('my_model') }} ``` ##### Can I use strict mode in development and baseline in deployment? Yes. This pattern is valid and recommended: use `strict` while you develop for stronger validation, and keep `baseline` in deployment for faster runs that are less likely to stop on analysis findings. For more information, including CLI examples and an optional environment variable pattern, refer to [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md). ##### When should I turn static analysis `off`? With baseline mode enabled by default, static analysis is less likely to block your runs. You should only disable it if dbt v2 cannot parse SQL that is valid for your database of choice. This is a very rare occurrence. If you encounter this situation, please [open an issue](https://github.com/dbt-labs/dbt/issues) with an example of the failing SQL so we can update our parsers. #### More information about dbt v2 * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ### About the --empty flag ### About the `--empty` flag note The `--empty` flag is not currently available for Python models. If the flag is used with a Python model, it will be ignored. During dbt development, you might want to validate that your models are semantically correct without the time-consuming cost of building the entire model in the data warehouse. The [`run`](https://docs.getdbt.com/reference/commands/run.md), [`build`](https://docs.getdbt.com/reference/commands/build.md), [`snapshot`](https://docs.getdbt.com/reference/commands/snapshot.md), and [`compile`](https://docs.getdbt.com/reference/commands/compile.md) commands support the `--empty` flag. Starting in dbt v1.12, [`seed`](https://docs.getdbt.com/reference/commands/seed.md) also supports the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. ##### Examples Run all models in a project while building only the schemas in your development environment: ```text dbt run --empty ``` Run a specific model: ```text dbt run --select path/to/your_model --empty ``` dbt will build and execute the SQL, resulting in an empty schema in the data warehouse. --- ### About the --sample flag ### About the `--sample` flag note The `--sample` flag is not currently available for Python models. If the flag is used with a Python model, it will be ignored. Seeds will be created normally, but are sampled when referenced by downstream nodes. Large data sets can drastically increase build times and reduce how quickly dbt developers can build and test new code. The dbt `--sample` flag can help to reduce build times and warehouse spend by running dbt in sample mode. Sample mode enables you to address cases where you don't need to build the entire model during the development or CI cycle but include enough data to validate the outputs. Sample mode takes the [`--empty` flag's](https://docs.getdbt.com/docs/build/empty-flag.md) validation of semantic results a step further by including a sampling of data from the model(s) in your development schema. It won't solve every scenario; for example, there are cases where not all joins will be populated. However, it presents a viable solution for faster building, testing, and validating many strategies. The `--sample` flag will become more robust over time, but it only supports time-based sampling for now. #### Using the `--sample` flag The `--sample` flag is available for the [`run`](https://docs.getdbt.com/reference/commands/run.md) and [`build`](https://docs.getdbt.com/reference/commands/build.md) commands. When used, sample mode generates filtered refs and sources. Since it's using time-based sampling, if you have refs like `{{ ref('some_model') }}` being sampled, you need to set [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) for `some_model` to the field that will be used as the timestamp. There are two time-based sample specifications supported for sample mode: * **Relative time specs:** Filters sampled data from the time the command is run back to a specified integer and granularity. Supported granularities are: * Hours * Days * Months * Years * **Static time specs:** Filters your data between a defined start and end period using date and/or timestamp. ##### Examples Let's say you want to run your `stg_customers` model and build the table in your development schema with a relative time spec sample size of three days. Your command in the IDE would look something like this: ```text dbt run --select path/to/stg_customers --sample="3 days" ``` If you have an even larger model, for example, `stg_orders` you can set sample mode to hours: ```text dbt run --select path/to/stg_customers --sample="6 hours" ``` Next, let's say you want to validate data for your entire business from a sample size further in the past - your busiest week in July, from the first until closing time on the eighth. You can run the following: ```text dbt run --sample="{'start': '2024-07-01', 'end': '2024-07-08 18:00:00'}" ``` To prevent a `ref` from being sampled, append `.render()` to it: ```sql with source as ( select * from {{ ref('stg_customers').render() }} ), ... ``` dbt will then execute the model SQL against the target data warehouse and build the tables with data from the sample sizes. --- ### About the dbt VS Code extension Local development The dbt VS Code extension brings a hyper-fast, intelligent, and cost-efficient dbt development experience to VS Code. This is the only way to enjoy all the power of dbt v2 while developing with a self-hosted installation. * *Save time and resources* with near-instant parsing, live error detection, powerful IntelliSense capabilities, and more. * *Stay in flow* with a seamless, end-to-end dbt development experience designed from scratch for local dbt development. The dbt VS Code extension is available in the [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt). The dbt VS Code extension works with v2, the default free-to-use product you get when you install dbt. #### Version compatibility dbt v1 and v2 both support the dbt VS Code extension. You get editor features out of the box as long as your project complies with the v2 language spec: * Autocomplete for `ref`, sources, macros, and doc blocks * Go to definition and find references for models, sources, macros, and docs * Jinja syntax error detection * Model renames that update `ref` calls, and source or table reference renames * View compiled SQL, and preview a model, selected SQL, or an individual CTE * Run `dbt run`, `build`, `test`, and selectors * Table descriptions, metadata, and metadata on hover * Production column types, descriptions, and tests * Visual and textual table lineage in the extension pane If your project isn't compliant with the v2 language spec yet, the extension has [dbt-autofix](https://github.com/dbt-labs/dbt-autofix) built in. Opt in to fix deprecations across your project in a single pull request. Each issue in the **Problems** pane also has an autofix action, available from the lightbulb or right-click menu. To unlock the full language server, including column-level lineage, column go-to definition and find references, nonexistent-column and type detection, warehouse-specific syntax checks, column renames that propagate downstream, and `SELECT *` expansion, enable [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2) and resolve your parse errors. Refer to [About dbt LSP](https://docs.getdbt.com/docs/about-dbt-lsp.md) for the full feature list. Try out the dbt v2 quickstart guide Check out the [v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) to try the dbt VS Code extension in action. #### Navigating the dbt extension Once the dbt VS Code extension has been installed, several visual enhancements will be added to your IDE to help you navigate the features and functionality. To read more about the features and functionality, see the [dbt extension features](https://docs.getdbt.com/docs/dbt-extension-features.md). Check out the following video to see the features and functionality of the dbt VS Code extension: [dbt v2 + VS Code extension walkthrough](https://app.storylane.io/share/a1rkqx0mbd7a) ##### Sign in and feature access After you install the dbt VS Code extension, its [features](https://docs.getdbt.com/docs/dbt-extension-features.md) are available to you. What's available depends on your project's [static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) mode. For example, column-level lineage and column go-to definition need `static_analysis: strict`. A few capabilities read data from your dbt platform account, such as the **Catalog** tab and [compare changes](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md) with platform deferral. [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to a dbt platform account so the extension can reach your account for those. Refer to [VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability) for the full list of features and their availability. Signing in is handled by [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0). Your login state is shared across the CLI, dbt VS Code extension, and dbt State (if you log in using dbt platform). You can use the [get started wizard](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started) or run `dbt login` from your terminal, then restart or reload VS Code. The extension detects your login automatically. ##### The dbt extension menu The dbt logo on the sidebar (or the **dbt Extension** text on the bottom tray) launches the main menu for the extension. This menu contains helpful information and actions you can take: * **Get started button:** Launches the [v2 upgrade](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) workflow. * **Extension info:** Information about the extension, v2, and your dbt project. Includes configuration options and actions. * **Help:** Quick links to support, bug submissions, and documentation. [![dbt VS Code extension welcome screen.](/img/docs/extension/sidebar-menu.png?v=2 "dbt VS Code extension welcome screen.")](#)dbt VS Code extension welcome screen. ##### Caching The dbt extension caches important schema information from your data warehouse to improve speed and performance. This will automatically update over time, but if recent changes have been made that aren't reflected in your project, you can manually update the schema information: 1. Click the **dbt logo** on the sidebar to open the menu. 2. Expand the **Extension info** section and location the **Actions** subsection. 3. Click **Clear Cache** to update. #### Using the extension Once installed, the dbt extension automatically activates when you open any `.sql` or `.yml` file inside of a dbt project directory. #### Configuration After installation, you may want to configure the extension to better fit your development workflow: 1. Open the VS Code settings by pressing `Ctrl+,` (Windows/Linux) or `Cmd+,` (Mac). 2. Search for `dbt`. On this page, you can adjust the extension’s configuration options to fit your needs. [![dbt extension settings within the VS Code settings.](/img/docs/extension/dbt-extension-settings.png?v=2 "dbt extension settings within the VS Code settings.")](#)dbt extension settings within the VS Code settings. #### Known limitations The following are currently known limitations of the dbt extension: * **Remote development:** The dbt extension does not yet support remote development sessions over SSH. Support will be added in a future release. For more information on remote development, refer to [Supporting Remote Development and GitHub Codespaces](https://code.visualstudio.com/api/advanced-topics/remote-extensions) and [Visual Studio Code Server](https://code.visualstudio.com/docs/remote/vscode-server). * **Working with YAML files:** Today, the dbt extension has the following limitations with operating on YAML files: * Go-to-definition is not supported for nodes defined in YAML files (like snapshots). * Renaming models and columns will not update references in YAML files. * Future releases of the dbt extension will address these limitations. * **Renaming models:** When you rename a model file, the dbt extension applies edits to update all `ref()` calls that reference the renamed model. Due to limitations of VS Code's Language Server Client, the extension can't auto-save these edited files. As a result, renaming a model file may cause compiler errors in your project. To fix these errors, either manually save each file that the dbt extension edited, or click **File** --> **Save All** to save all edited files. * **Using Cursor's Agent mode:** When using the dbt extension in Cursor, lineage visualization works best in Editor mode and doesn't render in Agent mode. If you're working in Agent mode and need to view lineage, switch to Editor mode to access the full lineage tab functionality. ##### Extension conflicts The extension may occasionally conflict with other VS Code extensions that provide similar services (such as code validation). You may need to disable these third-party extensions while working with the dbt extension. **YAML by Red Hat:** The YAML extension by Red Hat may erroneously flag some keys (such as `static_analysis`) in dbt YAML files as invalid in the IDE. [![Static analysis erroneously tagged as invalid](/img/docs/extension/false-yaml-error.png?v=2 "Static analysis erroneously tagged as invalid")](#)Static analysis erroneously tagged as invalid To solve this issue, do one of the following: * (Recommended) Disable the Red Hat YAML extension while working with the dbt extension. * Add the following configuration to your VS Code `settings.json` file: ```json "yaml.schemas": { "Core/dbtschema.json": "data/dbt/models/**/schema.yml", "": "data/dbt/dbt_project.yml" }, ``` This could disable *all* use of the schema store, resulting in unintended consequences. #### dbt Wizard The dbt VS Code extension and [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) are designed to work together. The extension gives you fast parsing, inline error detection, and IntelliSense. [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) adds an AI layer on top — one that works with a live understanding of your project through the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md), a structured view of your [lineage](https://docs.getdbt.com/docs/explore/explore-projects.md), model health, test coverage, and semantic definitions. * **Build and refactor from natural language**: Describe what you want, review the diff, approve or redirect before anything is saved. * **Validate changes before they land**: The agent compiles and runs against your warehouse in a tight loop — not just editor linting. * **Work with your full project map**: Traverse the [DAG](https://docs.getdbt.com/docs/explore/explore-projects.md), understand downstream impact, and keep tests and YAML in sync as models change. For data practitioners, combining the extension's dbt-powered editor experience with dbt Wizard's project-aware agent means less manual YAML, fewer context switches, and faster iteration on complex modeling work. See [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) to get started. #### Support dbt platform customers can contact dbt Labs support at <support@getdbt.com>. You can also get in touch with us by reaching out to your Account Manager directly. For organizations that aren't customers of the dbt platform, the best place for questions and discussion is the [dbt Community Slack](https://www.getdbt.com/community/join-the-community). We welcome feedback as we work to continuously improve the extension, and would love to hear from you! The extension is available to all users and for more information, refer to the [dbt Product License Agreement](https://www.getdbt.com/dbt-product-license-agreement). Developing locally as a dbt platform user? Refer to the [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) guide for how to keep credentials, environment variables, and v2 versions in sync between your local extension and dbt platform. #### More information about dbt v2 * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ### Access Catalog from dbt platform features dbt platform Access Catalog from jobs, Insights, and other areas of dbt to move quickly between run context, lineage, and resource metadata in your project. This page describes how to open Catalog from orchestration and exploration workflows in dbt. The primary way to open Catalog is **Catalog** in the navigation; you can also open it from jobs and Insights as described in the sections below. ##### Lineage tab in jobs The **Lineage tab** in dbt jobs displays the lineage associated with the [job run](https://docs.getdbt.com/docs/deploy/jobs.md). You can open Catalog directly from this tab to understand the dependencies and relationships of resources in your project. ###### Access Catalog from the lineage tab * From a job, select the **Lineage tab**. * Double-click a node in the lineage graph to open a new tab and view its metadata in Catalog. [![Access dbt Catalog from the lineage tab by double-clicking on the lineage node.](/img/docs/collaborate/dbt-explorer/explorer-from-lineage.gif?v=2 "Access dbt Catalog from the lineage tab by double-clicking on the lineage node.")](#)Access dbt Catalog from the lineage tab by double-clicking on the lineage node. ##### Model timing tab in jobs [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") The [model timing tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing) in dbt jobs displays the composition, order, and time taken by each model in a job run. You can open Catalog from the **model timing tab** to investigate resources, diagnose performance bottlenecks, understand dependencies and relationships of slow-running models, and make changes to improve their performance. ###### Access Catalog from the model timing tab * From a job, select the **model timing tab**. * Hover over a resource and select **View in Catalog** to open its metadata in Catalog. [![Access dbt Catalog from the model timing tab by hovering over the resource and clicking 'View in Explorer'.](/img/docs/collaborate/dbt-explorer/explorer-from-model-timing.png?v=2 "Access dbt Catalog from the model timing tab by hovering over the resource and clicking 'View in Explorer'.")](#)Access dbt Catalog from the model timing tab by hovering over the resource and clicking 'View in Explorer'. ##### dbt Insights [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Open Catalog from [Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md) to view project lineage and resources such as tables, columns, metrics, dimensions, and more. To open Catalog from Insights, select the **Catalog** icon in the Query console sidebar menu, then search for the resource you want. [![dbt Insights integrated with dbt Catalog](/img/docs/dbt-insights/insights-explorer.png?v=2 "dbt Insights integrated with dbt Catalog")](#)dbt Insights integrated with dbt Catalog --- ### Access the dbt Insights interface dbt platform | Enterprise, Enterprise+ Learn how to access Insights, run queries, and view results. Insights provides a rich console experience with editor navigation. You can expect Insights to: * Enable you to write SQL queries, with the option to open multiple tabs * Have SQL + dbt autocomplete suggestions and syntax highlighting * Save SQL queries * View the results of the query and its details using the **Data** or **Details** tabs * Create a visualization of your query results using the **Chart** tab * View the history of queries and their statuses (like Success, Error, Pending) using the **Query history** tab * Use dbt Copilot in Insights to generate or edit SQL queries using natural language prompts * Integrate with [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), and [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to provide a seamless experience for data exploration, AI-assisted writing, and collaboration #### Access the dbt Insights interface Before accessing Insights, ensure that the [prerequisites](https://docs.getdbt.com/docs/explore/dbt-insights.md#prerequisites) are met. 1. To access Insights, select the **Insights** option in the navigation sidebar. 2. If your [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) aren’t set up, Insights will prompt you to set them up. The ability to query data is subject to warehouse provider permissions according to your user credentials. 3. Once your credentials are set up, you can write, run, and edit SQL queries in the Insights editor for existing models in your project. #### Run queries To run queries in Insights, you can use: * Standard SQL * Jinja ([`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) functions, and other Jinja functions) * Links from SQL code `ref` to the corresponding Explorer page * CTEs and subqueries * Basic aggregations and joins * Semantic Layer queries using Semantic Layer Jinja functions #### Example Let's use an example to illustrate how to run queries in Insights: * A [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) location wants to count unique orders and unique customers to understand whether they can expand their awesome Jaffle shop business to other parts of the world. * To express this logic in SQL, you (an analyst assigned to this project) want to understand yearly trends to help guide expansion decisions. Write the following SQL query to calculate the number of unique customers, cities, and total order revenue: <br /> <br /> ```sql with orders as ( select * from {{ ref('orders') }} ), customers as ( select * from {{ ref('customers') }} ) select date_trunc('year', ordered_at) as order_year, count(distinct orders.customer_id) as unique_customers, count(distinct orders.location_id) as unique_cities, to_char(sum(orders.order_total), '999,999,999.00') as total_order_revenue from orders join customers on orders.customer_id = customers.customer_id group by 1 order by 1 ``` ##### Use dbt Copilot To make things easier, [use dbt Copilot in Insights](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md) to save time and explore other ways to analyze the data. dbt Copilot can help you quickly update the query or generate a new one based on your prompt. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. 1. Click the **dbt Copilot** icon in the Query console sidebar. 2. In the dropdown menu above the dbt Copilot prompt box, select **Generate SQL**. 3. Enter your prompt in natural language and ask for a yearly breakdown of unique customers and total revenue. 4. Click **↑** to submit your prompt. 5. dbt Copilot responds with: * A summary of the query * An explanation of the logic * The SQL it generated * Options to **Add** or **Replace** the existing query with the generated SQL 6. Review the output and click **Replace** to use the dbt Copilot-generated SQL in your editor. 7. Click **Run** to preview the results. [![dbt Insights with dbt Copilot](/img/docs/dbt-insights/insights-copilot.png?v=2 "dbt Insights with dbt Copilot")](#)dbt Insights with dbt Copilot From here, you can: * Continue building or modifying the query using dbt Copilot. * Explore the [results](#view-results) in the **Data** tab. * [View metadata and query details](#view-details) in the **Details** tab. * [Visualize results](#chart-results) in the **Chart** tab. * Check the [**Query history**](#query-history) for status and past runs. * Use [**Catalog**](#use-dbt-explorer) to explore model lineage and context. * If you want to save the query, you can click **Save Insight** in the [query console menu](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#query-console-menu) to save it for future reference. Want to turn a query into a model? You can access the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) from the [Query console menu](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#query-console-menu) to promote your SQL into a reusable dbt model — all within dbt! ##### View results Using the same example, you can perform some exploratory data analysis by running the query and: * Viewing results in **Data** tab — View the paginated results of the query. * Sorting results — Click on the column header to sort the results by that column. * Exporting to CSV — On the top right of the table, click the download button to export the dataset. [![dbt Insights Export to CSV](/img/docs/dbt-insights/insights-export-csv.png?v=2 "dbt Insights Export to CSV")](#)dbt Insights Export to CSV ##### View details View the details of the query by clicking on the **Details** tab: * **Query metadata** — dbt Copilot-generated title and description, the supplied SQL, and corresponding compiled SQL. * **Connection details** — Relevant data platform connection information. * **Query details** — Query duration, status, column count, row count. [![dbt Insights Details tab](/img/docs/dbt-insights/insights-details.png?v=2 "dbt Insights Details tab")](#)dbt Insights Details tab ##### Chart results Visualize the chart results of the query by clicking on the **Chart** tab to: * Select the chart type using the chart icon. * Choose from **line chart, bar chart, or scatterplot**. * Select the axis and columns to visualize using the **Chart settings** icon. [![dbt Insights Chart tab](/img/docs/dbt-insights/insights-chart.png?v=2 "dbt Insights Chart tab")](#)dbt Insights Chart tab ##### Query history View the history of queries and their statuses (All, Success, Error, or Pending) using the **Query history** icon: * Select a query to re-run to view the results. * Search for past queries and filter by status. * Hover over the query to view the SQL code or copy it. The query history is stored indefinitely. [![dbt Insights Query history icon](/img/docs/dbt-insights/insights-query-history.png?v=2 "dbt Insights Query history icon")](#)dbt Insights Query history icon ##### Use dbt Catalog Access [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) directly in Insights to view project resources such as models, columns, metrics, and dimensions, and more — all integrated in the Insights interface. This integrated view allows you and your users to maintain your query workflow, while getting more context on models, semantic models, metrics, macros, and more. The integrated Catalog view comes with: * Same search capabilities as Catalog * Allows users to narrow down displayed objects by type * Hyperlink from SQL code `ref` to the corresponding Catalog page * View assets in more detail by opening with the full Catalog experience or open them in dbt Copilot. To access Catalog, click on the **Catalog** icon in the [Query console sidebar menu](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#query-console-sidebar-menu). [![dbt Insights integrated with dbt Catalog](/img/docs/dbt-insights/insights-explorer.png?v=2 "dbt Insights integrated with dbt Catalog")](#)dbt Insights integrated with dbt Catalog ##### Set Jinja environment Set the compilation environment to control how Jinja functions are rendered. This feature: * Supports "typed" environments marked as `Production`, `Staging`, and/or `Development`. * Enables you to run Semantic Layer. queries against staging environments (development environments not supported). * Still uses the individual user credentials, so users must have appropriate access to query `PROD` and `STG`. * Changing the environment changes context for the Catalog view in Insights, as well as the environment context during the handoff to Catalog and Canvas. For example, switching to `Staging` in Insights and selecting **View in Catalog** will open the `Staging` view in Catalog. [![Set the environment for your Jinja context](/img/docs/dbt-insights/insights-jinja-environment.png?v=2 "Set the environment for your Jinja context")](#)Set the environment for your Jinja context #### Save your Insights Insights offers a robust save feature for quickly finding the queries you use most. There's also an option to share saved Insights with other dbt users (and have them share with you). Click the **bookmark icon** in a query to add it to your list! * Click the **bookmark icon** on the right menu to manage your saved Insights. You can view your personal and shared queries [![Manage your saved Insights](/img/docs/dbt-insights/saved-insights.png?v=2 "Manage your saved Insights")](#)Manage your saved Insights * View saved Insight details including description and creation date in the **Overview** tab. * View the Insight history in the **Version history** tab. Click a version to compare it the current and view changes. #### Considerations * Insights uses your user credentials to query. You have the ability to query against any object in your data warehouse that is accessible using your user credentials configured in **Account settings**. * Every Jinja function uses [`defer --favor-state`](https://docs.getdbt.com/reference/node-selection/defer.md) to resolve Jinja. #### FAQs * What’s the difference between Insights and Catalog? * That’s a great question! Catalog helps you understand your dbt project's structure, resources, lineage, and metrics, offering context for your data. * Insights builds on that context, allowing you to write, run, and iterate on SQL queries directly in dbt. It’s designed for ad-hoc or exploratory analysis and empowers business users and analysts to explore data, ask questions, and collaborate seamlessly. * Catalog provides the context, while Insights enables action. --- ### Add data tests to your DAG Tip Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to generate data tests in the dbt platform or locally in the CLI. #### Related reference docs * [Test command](https://docs.getdbt.com/reference/commands/test.md) * [Data test properties](https://docs.getdbt.com/reference/resource-properties/data-tests.md) * [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md) * [Test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) * [Batch tests](https://docs.getdbt.com/reference/global-configs/batch-tests.md) important Tests are now called data tests to disambiguate from [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md). The YAML key `tests:` is still supported as an alias for `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. #### Overview Data tests are assertions you make about your models and other resources in your dbt project (for example, sources, seeds, and snapshots). When you run `dbt test`, dbt will tell you if each test in your project passes or fails. You can use data tests to improve the integrity of the SQL in each model by making assertions about the results generated. Out of the box, you can test whether a specified column in a model only contains non-null values, unique values, or values that have a corresponding value in another model (for example, a `customer_id` for an `order` corresponds to an `id` in the `customers` model), and values from a specified list. You can extend data tests to suit business logic specific to your organization – any assertion that you can make about your model in the form of a select query can be turned into a data test. Data tests return a set of failing records. Generic data tests (also known as schema tests) are defined using `test` blocks. Like almost everything in dbt, data tests are SQL queries. In particular, they are `select` statements that seek to grab "failing" records, ones that disprove your assertion. If you assert that a column is unique in a model, the test query selects for duplicates; if you assert that a column is never null, the test seeks nulls. If the data test returns zero failing rows, it passes, and your assertion has been validated. There are two ways of defining data tests in dbt: * A **singular** data test, in its simplest form, is when you write a SQL query that returns failing rows, you can save that query in a `.sql` file within your [test directory](https://docs.getdbt.com/reference/project-configs/test-paths.md). It's now a data test, and it will be executed by the `dbt test` command. * A **generic** data test is a parameterized query that accepts arguments. The test query is defined in a special `test` block (like a [macro](https://docs.getdbt.com/docs/build/jinja-macros.md)). Once defined, you can reference the generic test by name throughout your `.yml` files—define it on models, columns, sources, snapshots, and seeds. dbt ships with four generic data tests built in, and we think you should use them! Defining data tests is a great way to confirm that your outputs and inputs are as expected, and helps prevent regressions when your code changes. Because you can use them over and over again, making similar assertions with minor variations, generic data tests tend to be much more common—they should make up the bulk of your dbt data testing suite. That said, both ways of defining data tests have their time and place. Creating your first data tests If you're new to dbt, we recommend that you check out our [online dbt Fundamentals course](https://learn.getdbt.com/learn/course/dbt-fundamentals/data-tests-30min/building-tests?page=1) or [quickstart guide](https://docs.getdbt.com/guides.md) to build your first dbt project with models and tests. #### Singular data tests The simplest way to define a data test is by writing the exact SQL that will return failing records. We call these "singular" data tests, because they're one-off assertions usable for a single purpose. These tests are defined in `.sql` files, typically in your `tests` directory (as defined by your `test-paths` config). **Note:** The `tests/` directory (`test-paths`) is reserved for singular and generic data tests (SQL). Unit test YAML definitions must live under your project’s `model-paths` (for example, in the `models/` directory), not in `tests/`. You can use Jinja (including `ref` and `source`) in the test definition, just like you can when creating models. Each `.sql` file contains one `select` statement, and it defines one data test: tests/assert\_total\_payment\_amount\_is\_positive.sql ```sql -- Refunds have a negative amount, so the total amount should always be >= 0. -- Therefore return records where total_amount < 0 to make the test fail. select order_id, sum(amount) as total_amount from {{ ref('fct_payments') }} group by 1 having total_amount < 0 ``` The test name is the file name: `assert_total_payment_amount_is_positive`. Note: * Omit semicolons (;) at the end of the SQL statement in your singular test files, as they can cause your data test to fail. * Singular data tests placed in the tests directory are automatically executed when running `dbt test`. Don't reference singular tests in `model_name.yml`, as they are not treated as generic tests or macros, and doing so will result in an error. To add a description to a singular data test in your project, add a `.yml` file to your `tests` directory, for example, `tests/schema.yml` with the following content: tests/schema.yml ```yaml data_tests: - name: assert_total_payment_amount_is_positive description: > Refunds have a negative amount, so the total amount should always be >= 0. Therefore return records where total amount < 0 to make the test fail. ``` Singular data tests are so easy that you may find yourself writing the same basic structure repeatedly, only changing the name of a column or model. By that point, the test isn't so singular! In that case, we recommend generic data tests. #### Generic data tests Certain data tests are generic: they can be reused over and over again. A generic data test is defined in a `test` block, which contains a parameterized query and accepts arguments. It might look like: ```sql {% test not_null(model, column_name) %} select * from {{ model }} where {{ column_name }} is null {% endtest %} ``` You'll notice that there are two arguments, `model` and `column_name`, which are then templated into the query. This is what makes the data test "generic": it can be defined on as many columns as you like, across as many models as you like, and dbt will pass the values of `model` and `column_name` accordingly. Once that generic test has been defined, it can be added as a *property* on any existing model (or source, seed, or snapshot). These properties are added in `.yml` files in the same directory as your resource. info If this is your first time working with adding properties to a resource, check out the docs on [declaring properties](https://docs.getdbt.com/reference/configs-and-properties.md). Out of the box, dbt ships with four generic data tests already defined: `unique`, `not_null`, `accepted_values`, and `relationships`. Here's a full example using those tests on an `orders` model: ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'returned'] - name: customer_id data_tests: - relationships: arguments: to: ref('customers') field: id ``` In plain English, these data tests translate to: * `unique`: the `order_id` column in the `orders` model should be unique * `not_null`: the `order_id` column in the `orders` model should not contain null values * `accepted_values`: the `status` column in the `orders` model should be one of `'placed'`, `'shipped'`, `'completed'`, or `'returned'` * `relationships`: each `customer_id` in the `orders` model exists as an `id` in the `customers` table (also known as referential integrity) Behind the scenes, dbt constructs a `select` query for each data test, using the parameterized query from the generic test block. These queries return the rows where your assertion is *not* true; if the test returns zero rows, your assertion passes. You can find more information about these data tests, and additional configurations (including [`severity`](https://docs.getdbt.com/reference/resource-configs/severity.md) and [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md)) in the [reference section](https://docs.getdbt.com/reference/resource-properties/data-tests.md). You can also add descriptions to the Jinja macro that provides the core logic of a generic data test. Refer to the [Add description to generic data test logic](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md#add-description-to-generic-data-test-logic) for more information. ##### More generic data tests Those four tests are enough to get you started. You'll quickly find you want to use a wider variety of data tests — a good thing! You can also install generic data tests from a package, or write your own, to use (and reuse) across your dbt project. Check out the [guide on custom generic data tests](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) for more information. info There are generic data tests defined in some open-source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn more! ##### Example To add a generic (or "schema") data test to your project: 1. Add a `.yml` file to your `models` directory, for example, `models/schema.yml`, with the following content (you may need to adjust the `name:` values for an existing model) models/schema.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` 2. Run the [`dbt test` command](https://docs.getdbt.com/reference/commands/test.md): ```text $ dbt test Found 3 models, 2 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 0 seed files, 0 sources 17:31:05 | Concurrency: 1 threads (target='learn') 17:31:05 | 17:31:05 | 1 of 2 START test not_null_order_order_id..................... [RUN] 17:31:06 | 1 of 2 PASS not_null_order_order_id........................... [PASS in 0.99s] 17:31:06 | 2 of 2 START test unique_order_order_id....................... [RUN] 17:31:07 | 2 of 2 PASS unique_order_order_id............................. [PASS in 0.79s] 17:31:07 | 17:31:07 | Finished running 2 tests in 7.17s. Completed successfully Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2 ``` 3. Check out the SQL dbt is running by either: * **dbt:** checking the Details tab. * **dbt v1:** checking the `target/compiled` directory **Unique test** ##### Compiled SQL ```sql select * from ( select order_id from analytics.orders where order_id is not null group by order_id having count(*) > 1 ) validation_errors ``` ##### Templated SQL ```sql select * from ( select {{ column_name }} from {{ model }} where {{ column_name }} is not null group by {{ column_name }} having count(*) > 1 ) validation_errors ``` **Not null test** ##### Compiled SQL ```sql select * from analytics.orders where order_id is null ``` ##### Templated SQL ```sql select * from {{ model }} where {{ column_name }} is null ``` #### Running only data tests To run data tests while excluding unit tests, use the `test_type` selector — this works across all engines (dbt v1 and dbt v2): ```bash dbt test --select "test_type:data" ``` In dbt (v1.9+), you can also use `dbt test --resource-type test`. For more options, refer to [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md). #### Batch your data tests In dbt v2, you can reduce the number of queries dbt issues for tests by enabling [batch tests](https://docs.getdbt.com/reference/global-configs/batch-tests.md). Batching groups `unique` and `not_null` tests that are attached to the same model into one query per test type, and each test still reports its own pass or fail result. #### Storing data test failures Normally, a data test query will calculate failures as part of its execution. If you set the optional `--store-failures` flag, the [`store_failures`](https://docs.getdbt.com/reference/resource-configs/store_failures.md), or the [`store_failures_as`](https://docs.getdbt.com/reference/resource-configs/store_failures_as.md) configs, dbt will first save the results of a test query to a table in the database, and then query that table to calculate the number of failures. This workflow allows you to query and examine failing records much more quickly in development: [![Store test failures in the database for faster development-time debugging.](/img/docs/building-a-dbt-project/test-store-failures.gif?v=2 "Store test failures in the database for faster development-time debugging.")](#)Store test failures in the database for faster development-time debugging. Note that, if you choose to store data test failures: * Test result tables are created in a schema suffixed or named `dbt_test__audit`, by default. It is possible to change this value by setting a `schema` config. (For more details on schema naming, see [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md).) * A test's results will always **replace** previous failures for the same test. #### New `data_tests:` syntax Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests, the key was renamed from `tests:` to `data_tests:`. dbt still supports `tests:` in your YAML configuration files for backward-compatibility purposes, and you might see it used throughout our documentation. However, you can't have a `tests` and a `data_tests` key associated with the same resource (for example, a single model) at the same time. models/schema.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` dbt\_project.yml ```yaml data_tests: +store_failures: true ``` #### Tests with and without `arguments` Some generic data tests need only the test name. Others need extra inputs, which you nest under an `arguments` property (available in v1.10.5 and higher). In v2, nesting test inputs under `arguments` is required. You will get a validation error if you set test inputs as top-level properties next to the test name. ##### Without `arguments` Tests like `unique` and `not_null` don't need extra inputs. For example, `unique` asserts that a column has no duplicate values, and `not_null` asserts that a column has no nulls. dbt already knows the model and column from the YAML context, so you can list the test by name: models/schema.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` ##### With `arguments` Tests like `accepted_values` and `relationships` need extra inputs for the test macro. For example, `accepted_values` asserts that a column's values are in a supplied list, and `relationships` asserts that each value exists in another model (referential integrity). Nest those inputs under `arguments:`: models/schema.yml ```yaml models: - name: orders columns: - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. values: ['placed', 'shipped', 'completed', 'returned'] - name: customer_id data_tests: - relationships: arguments: to: ref('customers') field: id ``` Use `arguments:` for inputs to the test macro (for example, `values`, `to`, and `field`). Use [`config`](https://docs.getdbt.com/reference/data-test-configs.md) for framework options such as `severity`, `where`, and `store_failures`. If you previously set test inputs as top-level properties next to the test name, nest them under `arguments:` instead. For details, refer to [`require_generic_test_arguments_property`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_generic_test_arguments_property.md) and [MissingArgumentsPropertyInGenericTestDeprecation](https://docs.getdbt.com/reference/deprecations.md#missingargumentspropertyingenerictestdeprecation). #### Add context to failing rows Data tests return one row for each failure. The columns in your test's SQL select statement are the columns you see when you debug failures, including when you [store test failures](#storing-data-test-failures). Built-in tests often return only the failing value. To include more context per failing row, write a [singular data test](#singular-data-tests) (or a [custom generic data test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md)) that selects the extra columns you want. Building on the `accepted_values` test above, here's a singular test that also captures `customer_id` for context. tests/assert\_unexpected\_order\_status.sql ```sql select order_id, status, customer_id from {{ ref('orders') }} where status not in ('placed', 'shipped', 'completed', 'returned') ``` When you run `dbt test --store-failures` (or set the [`store_failures`](https://docs.getdbt.com/reference/resource-configs/store_failures.md) config), dbt saves those rows so you can query them and inspect every selected column. #### FAQs What data tests are available for me to use in dbt? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). Can I store my data tests in a directory other than the \`tests\` directory in my project? By default, dbt expects your singular data test files to be located in the `tests` subdirectory of your project, and generic data test definitions to be located in `tests/generic` or `macros`. To change this, update the [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml test-paths: ["my_cool_tests"] ``` Then, you can define generic data tests in `my_cool_tests/generic/`, and singular data tests everywhere else in `my_cool_tests/`. How do I run data tests on just my sources? To run data tests on all sources, use the following command: ```shell dbt test --select "source:*" ``` (You can also use the `-s` shorthand here instead of `--select`) To run data tests on one source (and all of its tables): ```shell $ dbt test --select source:jaffle_shop ``` And, to run data tests on one source table only: ```shell $ dbt test --select source:jaffle_shop.orders ``` Can I set test failure thresholds? You can use the `error_if` and `warn_if` configs to set custom failure thresholds in your tests. For more details, see [reference](https://docs.getdbt.com/reference/resource-configs/severity.md) for more information. You can also try the following solutions: * Setting the [severity](https://docs.getdbt.com/reference/resource-configs/severity.md) to `warn` or `error` * Writing a [custom generic test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) that accepts a threshold argument ([example](https://discourse.getdbt.com/t/creating-an-error-threshold-for-schema-tests/966)) Can I test the uniqueness of two columns? Yes, there's a few different options for testing the uniqueness of two columns. Consider an orders table that contains records from multiple countries, and the combination of ID and country code is unique: | order\_id | country\_code | | --------- | ------------- | | 1 | AU | | 2 | AU | | ... | ... | | 1 | US | | 2 | US | | ... | ... | Here are some approaches: ###### 1. Create a unique key in the model and test that models/orders.sql ```sql select country_code || '-' || order_id as surrogate_key, ... ``` models/orders.yml ```yml models: - name: orders columns: - name: surrogate_key data_tests: - unique ``` ###### 2. Test an expression models/orders.yml ```yml models: - name: orders data_tests: - unique: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. column_name: "(country_code || '-' || order_id)" ``` ###### 3. Use the `dbt_utils.unique_combination_of_columns` test This is especially useful for large datasets since it is more performant. Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) for more information. models/orders.yml ```yml models: - name: orders data_tests: - dbt_utils.unique_combination_of_columns: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. combination_of_columns: - country_code - order_id ``` --- ### Add Exposures to your DAG Exposures make it possible to define and describe a downstream use of your dbt project, such as in a dashboard, application, or data science pipeline. By defining exposures, you can then: * run, test, and list resources that feed into your exposure * populate a dedicated page in the auto-generated [documentation](https://docs.getdbt.com/docs/build/documentation.md) site with context relevant to data consumers Exposures can be defined in two ways: * Manual — Declared [explicitly](https://docs.getdbt.com/docs/build/exposures.md#declaring-an-exposure) in your project’s YAML files. * Automatic — dbt [creates and visualizes downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) automatically for supported integrations, removing the need for manual YAML definitions. These downstream exposures are stored in dbt’s metadata system, appear in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), and behave like manual exposures. However, they don’t exist in YAML files. ##### Declaring an exposure Exposures are defined in `.yml` files nested under an `exposures:` key. The following example shows an exposure definition in a `models/<filename>.yml` file: models/\<filename>.yml ```yaml exposures: - name: weekly_jaffle_metrics label: Jaffles by the Week type: dashboard maturity: high url: https://bi.tool/dashboards/1 description: > Did someone say "exponential growth"? depends_on: - ref('fct_orders') - ref('dim_customers') - source('gsheets', 'goals') - metric('count_orders') owner: name: Callum McData email: data@jaffleshop.com ``` ##### Available properties *Required:* * **name**: a unique exposure name written in [snake case](https://en.wikipedia.org/wiki/Snake_case) * **type**: one of `dashboard`, `notebook`, `analysis`, `ml`, `application` (used to organize in docs site) * **owner**: `name` or `email` required; additional properties allowed *Expected:* * **depends\_on**: list of refable nodes, including `metric`, `ref`, and `source`. While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly. tip Do not confuse this `depends_on` YAML property with the [`-- depends_on`](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks.md#example-using-depends_on) a SQL comment directive defined at the top of a model SQL file, which explicitly adds dependencies to the DAG and ensures the referenced resources are built before the model executes. *Optional:* * **label**: May contain spaces, capital letters, or special characters. * **url**: Activates and populates the link to **View this exposure** in the upper right corner of the generated documentation site * **maturity**: Indicates the level of confidence or stability in the exposure. One of `high`, `medium`, or `low`. For example, you could use `high` maturity for a well-established dashboard, widely used and trusted within your organization. Use `low` maturity for a new or experimental analysis. *General properties (optional)* * [**description**](https://docs.getdbt.com/reference/resource-properties/description.md) * [**tags**](https://docs.getdbt.com/reference/resource-configs/tags.md) * [**meta**](https://docs.getdbt.com/reference/resource-configs/meta.md) * [**enabled**](https://docs.getdbt.com/reference/resource-configs/enabled.md) — You can set this property at the exposure level or at the project level in the [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md) file. ##### Referencing exposures Once an exposure is defined, you can run commands that reference it: ```text dbt run -s +exposure:weekly_jaffle_report dbt test -s +exposure:weekly_jaffle_report ``` When we generate the [Catalog site](https://docs.getdbt.com/docs/explore/explore-projects.md), you'll see the exposure appear: [![Exposures has a dedicated section, under the project name in dbt Catalog, which lists each exposure in your project.](/img/docs/building-a-dbt-project/dbt-explorer-exposures.png?v=2 "Exposures has a dedicated section, under the project name in dbt Catalog, which lists each exposure in your project.")](#)Exposures has a dedicated section, under the project name in dbt Catalog, which lists each exposure in your project. [![Exposures appear as nodes in the dbt Catalog DAG. It displays an orange 'EXP' indicator within the node. ](/img/docs/building-a-dbt-project/dag-exposures.png?v=2 "Exposures appear as nodes in the dbt Catalog DAG. It displays an orange 'EXP' indicator within the node. ")](#)Exposures appear as nodes in the dbt Catalog DAG. It displays an orange 'EXP' indicator within the node. #### Related docs * [Exposure properties](https://docs.getdbt.com/reference/exposure-properties.md) * [`exposure:` selection method](https://docs.getdbt.com/reference/node-selection/methods.md#exposure) * [Data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) --- ### Add groups to your DAG A group is a collection of nodes within a dbt DAG. Groups are named, and every group has an `owner`. They enable intentional collaboration within and across teams by restricting [access to private](https://docs.getdbt.com/reference/resource-configs/access.md) models. Group members may include models, tests, seeds, snapshots, analyses, and metrics. (Not included: sources and exposures.) Each node may belong to only one group. ##### Declaring a group Groups are defined in `.yml` files, nested under a `groups:` key. In version 1.10 and higher, you can add a `description` and a `meta` config to add more information about the group. (Applies to dbt v1.10 and later) models/marts/finance/finance.yml ```yaml groups: - name: finance description: "All finance-related models owned by the Finance team." # optional owner: # 'name' or 'email' is required; additional properties will no longer be allowed in a future release email: finance@jaffleshop.com config: meta: # optional data_owner: Finance team cost_center: finance data_classification: sensitive ``` ###### Centrally defining a group To centrally define a group in your project, there are two options: * Create one `_groups.yml` file in the root of the `models` directory. * Create one `_groups.yml` file in the root of a `groups` directory. For this option, you also need to configure [`model-paths`](https://docs.getdbt.com/reference/project-configs/model-paths.md) in the `dbt_project.yml` file: ```yml model-paths: ["models", "groups"] ``` ##### Group properties The following properties are available when defining a group: | Property | Required | Description | | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Required | A unique name for the group within the project. Used to assign resources via the [`group` config](https://docs.getdbt.com/reference/resource-configs/group.md). | | `owner` | Required | Identifies who is responsible for the group. Must include either `name:` or `email:`. | | `description` | Optional | A human readable description of the group's purpose. Supports markdown and the [`doc` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md). Supported in v1.10 and later. | | `config.meta` | Optional | A dictionary of arbitrary key/value metadata about the group. Useful for storing information such as cost centers, data classifications, or team contact details. | models/marts/finance/finance.yml ```yaml groups: - name: finance description: "All models owned by the Finance team." owner: email: finance@jaffleshop.com config: meta: data_owner: Finance team cost_center: finance data_classification: sensitive ``` ##### Adding a model to a group Use the `group` configuration to add one or more models to a group. ##### Project-level dbt\_project.yml ```yml models: marts: finance: +group: finance ``` ##### Model-level models/schema.yml ```yml models: - name: model_name config: group: finance ``` ##### In-file models/model\_name.sql ```sql {{ config(group = 'finance') }} select ... ``` ##### Referencing a model in a group By default, all models within a group have the `protected` [access modifier](https://docs.getdbt.com/reference/resource-configs/access.md). This means they can be referenced by downstream resources in *any* group in the same project, using the [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function. If a grouped model's `access` property is set to `private`, only resources within its group can reference it. models/schema.yml ```yml models: - name: finance_private_model config: access: private # changed to config in v1.10 group: finance # in a different group! - name: marketing_model config: group: marketing ``` models/marketing\_model.sql ```sql select * from {{ ref('finance_private_model') }} ``` ```shell $ dbt run -s marketing_model ... dbt.exceptions.DbtReferenceError: Parsing Error Node model.jaffle_shop.marketing_model attempted to reference node model.jaffle_shop.finance_private_model, which is not allowed because the referenced node is private to the finance group. ``` #### Related docs * [Model Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) * [Group configuration](https://docs.getdbt.com/reference/resource-configs/group.md) * [Group selection](https://docs.getdbt.com/reference/node-selection/methods.md#group) --- ### Add Seeds to your DAG #### Related reference docs * [Seed configurations](https://docs.getdbt.com/reference/seed-configs.md) * [Seed properties](https://docs.getdbt.com/reference/seed-properties.md) * [`seed` command](https://docs.getdbt.com/reference/commands/seed.md) #### Overview Seeds are CSV files in your dbt project (typically in your `seeds` directory), that dbt can load into your data warehouse using the `dbt seed` command. Seeds can be referenced in downstream models the same way as referencing models — by using the [`ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md). Because these CSV files are located in your dbt repository, they are version controlled and code reviewable. Seeds are best suited to static data which changes infrequently. Good use-cases for seeds: * A list of mappings of country codes to country names * A list of test emails to exclude from analysis * A list of employee account IDs Poor use-cases of dbt seeds: * Loading raw data that has been exported to CSVs * Any kind of production data containing sensitive information. For example personal identifiable information (PII) and passwords. #### Example To load a seed file in your dbt project: 1. Add the file to your `seeds` directory, with a `.csv` file extension, for example, `seeds/country_codes.csv` seeds/country\_codes.csv ```text country_code,country_name US,United States CA,Canada GB,United Kingdom ... ``` 2. Run the `dbt seed` [command](https://docs.getdbt.com/reference/commands/seed.md) — a new table will be created in your warehouse in your target schema, named `country_codes` ```text $ dbt seed Found 2 models, 3 tests, 0 archives, 0 analyses, 53 macros, 0 operations, 1 seed file 14:46:15 | Concurrency: 1 threads (target='dev') 14:46:15 | 14:46:15 | 1 of 1 START seed file analytics.country_codes........................... [RUN] 14:46:15 | 1 of 1 OK loaded seed file analytics.country_codes....................... [INSERT 3 in 0.01s] 14:46:16 | 14:46:16 | Finished running 1 seed in 0.14s. Completed successfully Done. PASS=1 ERROR=0 SKIP=0 TOTAL=1 ``` 3. Refer to seeds in downstream models using the `ref` function. models/orders.sql ```sql -- This refers to the table created from seeds/country_codes.csv select * from {{ ref('country_codes') }} ``` #### Configuring seeds Seeds are configured in your `dbt_project.yml`, check out the [seed configurations](https://docs.getdbt.com/reference/seed-configs.md) docs for a full list of available configurations. #### Documenting and testing seeds You can document and test seeds in YAML by declaring properties — check out the docs on [seed properties](https://docs.getdbt.com/reference/seed-properties.md) for more information. #### FAQs Can I use seeds to load raw data? Seeds should **not** be used to load raw data (for example, large CSV exports from a production database). Since seeds are version controlled, they are best suited to files that contain business-specific logic, for example a list of country codes or user IDs of employees. Loading CSVs using dbt's seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your data warehouse. Can I store my seeds in a directory other than the \`seeds\` directory in my project? By default, dbt expects your seed files to be located in the `seeds` subdirectory of your project. To change this, update the [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml seed-paths: ["custom_seeds"] ``` The columns of my seed changed, and now I get an error when running the \`seed\` command, what should I do? If you changed the columns of your seed, you may get a `Database Error`: ##### Snowflake ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 1 seed file, 0 sources 12:12:27 | Concurrency: 8 threads (target='dev_snowflake') 12:12:27 | 12:12:27 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:12:30 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 2.78s] 12:12:31 | 12:12:31 | Finished running 1 seed in 10.05s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) 000904 (42000): SQL compilation error: error line 1 at position 62 invalid identifier 'COUNTRY_NAME' Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` ##### Redshift ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 149 macros, 0 operations, 1 seed file, 0 sources 12:14:46 | Concurrency: 1 threads (target='dev_redshift') 12:14:46 | 12:14:46 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:14:46 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 0.23s] 12:14:46 | 12:14:46 | Finished running 1 seed in 1.75s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) column "country_name" of relation "country_codes" does not exist Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` In this case, you should rerun the command with a `--full-refresh` flag, like so: ```text dbt seed --full-refresh ``` **Why is this the case?** When you typically run dbt seed, dbt truncates the existing table and reinserts the data. This pattern avoids a `drop cascade` command, which may cause downstream objects (that your BI users might be querying!) to get dropped. However, when column names are changed, or new columns are added, these statements will fail as the table structure has changed. The `--full-refresh` flag will force dbt to `drop cascade` the existing table before rebuilding it. How do I test and document seeds? To test and document seeds, use a [properties file](https://docs.getdbt.com/reference/configs-and-properties.md) and nest the configurations under a `seeds:` key #### Example seeds/properties.yml ```yml seeds: - name: country_codes description: A mapping of two letter country codes to country names columns: - name: country_code data_tests: - unique - not_null - name: country_name data_tests: - unique - not_null ``` How do I set a datatype for a column in my seed? dbt will infer the datatype for each column based on the data in your CSV. You can also explicitly set a datatype using the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) like so: dbt\_project.yml ```yml seeds: jaffle_shop: # you must include the project name warehouse_locations: +column_types: zipcode: varchar(5) ``` How do I run models downstream of a seed? You can run models downstream of a seed using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md), and treating the seed like a model. For example, the following would run all models downstream of a seed named `country_codes`: ```shell $ dbt run --select country_codes+ ``` How do I preserve leading zeros in a seed? If you need to preserve leading zeros (for example in a zipcode or mobile number), include leading zeros in your seed file, and use the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) with a varchar datatype of the correct length. How do I build one seed at a time? You can use a `--select` option with the `dbt seed` command, like so: ```shell $ dbt seed --select country_codes ``` There is also an `--exclude` option. Check out more in the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) documentation. Do hooks run with seeds? Yes! The following hooks are available: * [pre-hooks & post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start & on-run-end hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) Configure these in your `dbt_project.yml` file. --- ### Add snapshots to your DAG #### Related documentation * [Snapshot configurations](https://docs.getdbt.com/reference/snapshot-configs.md) * [Snapshot properties](https://docs.getdbt.com/reference/snapshot-properties.md) * [`snapshot` command](https://docs.getdbt.com/reference/commands/snapshot.md) * [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md) for guidance on using snapshots for CDC Learn by video! For video tutorials on Snapshots, go to dbt Learn and check out the [Snapshots course](https://learn.getdbt.com/courses/snapshots). #### What are snapshots? Analysts often need to "look back in time" at previous data states in their mutable tables. While some source data systems are built in a way that makes accessing historical data possible, this is not always the case. dbt provides a mechanism, **snapshots**, which records changes to a mutable table over time. Snapshots implement [type-2 Slowly Changing Dimensions](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) over mutable source tables. These Slowly Changing Dimensions (or SCDs) identify how a row in a table changes over time. Imagine you have an `orders` table where the `status` field can be overwritten as the order is processed. | id | status | updated\_at | | -- | ------- | ----------- | | 1 | pending | 2024-01-01 | Now, imagine that the order goes from "pending" to "shipped". That same record will now look like: | id | status | updated\_at | | -- | ------- | ----------- | | 1 | shipped | 2024-01-02 | This order is now in the "shipped" state, but we've lost the information about when the order was last in the "pending" state. This makes it difficult (or impossible) to analyze how long it took for an order to ship. dbt can "snapshot" these changes to help you understand how values in a row change over time. Here's an example of a snapshot table for the previous example: | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | | -- | ------- | ----------- | ---------------- | -------------- | | 1 | pending | 2024-01-01 | 2024-01-01 | 2024-01-02 | | 1 | shipped | 2024-01-02 | 2024-01-02 | `null` | #### Configuring snapshots (Applies to dbt v1.9 and later) Configure your snapshots in YAML files to tell dbt how to detect record changes. Define snapshots configurations in YAML files, alongside your models, for a cleaner, faster, and more consistent set up. Place snapshot YAML files in the models directory or in a snapshots directory. snapshots/orders\_snapshot.yml ```yaml snapshots: - name: string relation: relation # source('my_source', 'my_table') or ref('my_model') description: markdown_string config: database: string schema: string alias: string strategy: timestamp | check unique_key: column_name_or_expression check_cols: [column_name] | all updated_at: column_name snapshot_meta_column_names: dictionary dbt_valid_to_current: string hard_deletes: ignore | invalidate | new_record ``` The following table outlines the configurations available for snapshots: | Config | Description | Required? | Example | | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | -------------------------------- | | [database](https://docs.getdbt.com/reference/resource-configs/database.md) | Specify a custom database for the snapshot | No | analytics | | [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) | Specify a custom schema for the snapshot | No | snapshots | | [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) | Specify an alias for the snapshot | No | your\_custom\_snapshot | | [strategy](https://docs.getdbt.com/reference/resource-configs/strategy.md) | The snapshot strategy to use. Valid values: `timestamp` or `check` | Yes | timestamp | | [unique\_key](https://docs.getdbt.com/reference/resource-configs/unique_key.md) | A primary key column(s) (string or array) or expression for the record | Yes | `id` or `[order_id, product_id]` | | [check\_cols](https://docs.getdbt.com/reference/resource-configs/check_cols.md) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | \["status"] | | [updated\_at](https://docs.getdbt.com/reference/resource-configs/updated_at.md) | A column in your snapshot query results that indicates when each record was last updated, used in the `timestamp` strategy. May support ISO date strings and unix epoch integers, depending on the data platform you use. | Only if using the `timestamp` strategy | updated\_at | | [dbt\_valid\_to\_current](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. | No | string | | [snapshot\_meta\_column\_names](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md) | Customize the names of the snapshot meta fields | No | dictionary | | [hard\_deletes](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) | Specify how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. | No | string | * In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. * Developers can still set a custom location with [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) and [`database`](https://docs.getdbt.com/reference/resource-configs/database.md) configs, consistent with other resource types. * A number of other configurations are also supported (for example, `tags` and `post-hook`). For the complete list, refer to [Snapshot configurations](https://docs.getdbt.com/reference/snapshot-configs.md). * You can configure snapshots from both the `dbt_project.yml` file and a `config` block. For more information, refer to the [configuration docs](https://docs.getdbt.com/reference/snapshot-configs.md). * Starting dbt v1.12, you can inspect the SQL generated for snapshots by running [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md). You can find compiled SQL files in the `target/compiled/` directory of your dbt project. ##### Add a snapshot to your project To add a snapshot to your project follow these steps. For users on versions 1.8 and earlier, refer to [Legacy snapshot configurations](https://docs.getdbt.com/reference/resource-configs/snapshots-jinja-legacy.md). 1. Create a properties YAML file in your `snapshots` directory: `snapshots/orders_snapshot.yml` and add your configuration details. You can also configure your snapshot from your project YAML file (`dbt_project.yml`) ([docs](https://docs.getdbt.com/reference/snapshot-configs.md)). snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: source('jaffle_shop', 'orders') config: schema: snapshots database: analytics unique_key: id strategy: timestamp updated_at: updated_at dbt_valid_to_current: "to_date('9999-12-31')" # Specifies that current records should have `dbt_valid_to` set to `'9999-12-31'` instead of `NULL`. ``` 2. (Optional) Apply transformations using an ephemeral model. By default, snapshots reference a source directly (as shown in the YAML in the previous step). If you need to apply transformations (such as filtering or deduplication), define an ephemeral model first to apply those transformations, and reference it in the snapshot relation field instead of calling `source()` directly. For example, here's an ephemeral model: models/ephemeral\_orders.sql ```sql {{ config(materialized='ephemeral') }} select * from {{ source('jaffle_shop', 'orders') }} ``` This is how to reference the ephemeral model in the `relation` field: snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: ref('ephemeral_orders') ... rest of config... ``` 3. Check whether the result set of your query includes a reliable timestamp column that indicates when a record was last updated. For our example, the `updated_at` column reliably indicates record changes, so we can use the `timestamp` strategy. If your query result set does not have a reliable timestamp, you'll need to instead use the `check` strategy — more details on this below. 4. Run the `dbt snapshot` [command](https://docs.getdbt.com/reference/commands/snapshot.md) — for our example, a new table will be created at `analytics.snapshots.orders_snapshot`. The [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) config will utilize the `generate_schema_name` macro. ```text $ dbt snapshot Running with dbt=1.9.0 15:07:36 | Concurrency: 8 threads (target='dev') 15:07:36 | 15:07:36 | 1 of 1 START snapshot snapshots.orders_snapshot...... [RUN] 15:07:36 | 1 of 1 OK snapshot snapshots.orders_snapshot..........[SELECT 3 in 1.82s] 15:07:36 | 15:07:36 | Finished running 1 snapshots in 0.68s. Completed successfully Done. PASS=2 ERROR=0 SKIP=0 TOTAL=1 ``` Compiled SQL for snapshots Starting dbt v1.12, you can inspect the SQL generated for this snapshot by running [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) or `dbt compile --select orders_snapshot`. Open the compiled SQL in `target/compiled/` to inspect or debug the generated queries. Each snapshot is compiled into its own SQL file, even if multiple snapshots are defined in the same source file. 5. Inspect the results by selecting from the table dbt created (`analytics.snapshots.orders_snapshot`). After the first run, you should see the results of your query, plus the [snapshot meta fields](#snapshot-meta-fields) as described later on. 6. Run the `dbt snapshot` command again and inspect the results. If any records have been updated, the snapshot should reflect this. 7. Select from the `snapshot` in downstream models using the `ref` function. models/changed\_orders.sql ```sql select * from {{ ref('orders_snapshot') }} ``` 8. Snapshots are only useful if you run them frequently — schedule the `dbt snapshot` command to run regularly. ##### Configuration best practices  Use the timestamp strategy where possible The timestamp strategy is recommended because it handles column additions and deletions more efficiently than the `check` strategy. This is because it's more robust to schema changes, especially when columns are added or removed over time. The timestamp strategy relies on a single `updated_at` field, which means it avoids the need to constantly update your snapshot configuration as your source table evolves. Why timestamp is the preferred strategy: * Requires tracking only one column (`updated_at`) * Automatically handles new or removed columns in the source table * Less prone to errors when the table schema evolves over time (for example, if using the `check` strategy, you might need to update the `check_cols` configuration)  Use dbt\_valid\_to\_current for easier date range queries By default, `dbt_valid_to` is `NULL` for current records. However, if you set the [`dbt_valid_to_current` configuration](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) (available in dbt v1.9+), `dbt_valid_to` will be set to your specified value (such as `9999-12-31`) for current records. This allows for straightforward date range filtering.  Ensure your unique key is really unique The unique key is used by dbt to match rows up, so it's extremely important to make sure this key is actually unique! If you're snapshotting a source, I'd recommend adding a uniqueness test to your source ([example](https://github.com/dbt-labs/jaffle_shop/blob/8e7c853c858018180bef1756ec93e193d9958c5b/models/staging/schema.yml#L26)). (Applies to dbt v1.9 and later)  Use a schema that is separate to your models' schema Snapshots can't be rebuilt. Because of this, it's a good idea to put snapshots in a separate schema so end users know they're special. From there, you may want to set different privileges on your snapshots compared to your models, and even run them as a different user (or role, depending on your warehouse) to make it very difficult to drop a snapshot unless you really want to.  Use ephemeral model to clean or transform data before snapshotting If you need to clean or transform your data before snapshotting, create an ephemeral model or a staging model that applies the necessary transformations. Then, reference this model in your snapshot configuration. This approach keeps your snapshot definitions clean and allows you to test and run transformations separately. ##### How snapshots work When you run the [`dbt snapshot` command](https://docs.getdbt.com/reference/commands/snapshot.md): * **On the first run:** dbt will create the initial snapshot table — this will be the result set of your `select` statement, with additional columns including `dbt_valid_from` and `dbt_valid_to`. All records will have a `dbt_valid_to = null` or the value specified in [`dbt_valid_to_current`](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) (available in dbt 1.9+) if configured. * **On subsequent runs:** dbt will check which records have changed or if any new records have been created: * The `dbt_valid_to` column will be updated for any existing records that have changed. * The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` or the value configured in `dbt_valid_to_current` (available in dbt v1.9+). Snapshots ignore full refresh Snapshots ignore both the `full_refresh` config and the `--full-refresh` flag. A command such as `dbt build --full-refresh` or `dbt snapshot --full-refresh` that includes a snapshot node runs the snapshot as normal — it won't drop or recreate the snapshot table, so existing snapshot history is preserved. (Applies to dbt v1.9 and later) ###### Note * These column names can be customized to your team or organizational conventions using the [snapshot\_meta\_column\_names](#snapshot-meta-fields) config. * Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. * Use the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config to track hard deletes by adding a new record when row become "deleted" in source. Supported options are `ignore`, `invalidate`, and `new_record`. Snapshots can be referenced in downstream models the same way as referencing models — by using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function. #### Detecting row changes Snapshot "strategies" define how dbt knows if a row has changed. There are two strategies built-in to dbt: * [Timestamp](#timestamp-strategy-recommended) — Uses an `updated_at` column to determine if a row has changed. * [Check](#check-strategy) — Compares a list of columns between their current and historical values to determine if a row has changed. ##### Timestamp strategy (recommended) The `timestamp` strategy uses an `updated_at` field to determine if a row has changed. If the configured `updated_at` column for a row is more recent than the last time the snapshot ran, then dbt will invalidate the old record and record the new one. If the timestamps are unchanged, then dbt will not take any action. Why timestamp is recommended? * Requires tracking only one column (`updated_at`) * Automatically handles new or removed columns in the source table * Less prone to errors when the table schema evolves over time (for example, if using the `check` strategy, you might need to update the `check_cols` configuration) The `timestamp` strategy requires the following configurations: | Config | Description | Example | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | updated\_at | A column which represents when the source row was last updated. May support ISO date strings and unix epoch integers, depending on the data platform you use. | `updated_at` | **Example usage:** (Applies to dbt v1.9 and later) snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot_timestamp relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: timestamp updated_at: updated_at ``` ##### Check strategy The `check` strategy is useful for tables which do not have a reliable `updated_at` column. This strategy works by comparing a list of columns between their current and historical values. If any of these columns have changed, then dbt will invalidate the old record and record the new one. If the column values are identical, then dbt will not take any action. The `check` strategy requires the following configurations: | Config | Description | Example | | ----------- | --------------------------------------------------------------------- | ------------------- | | check\_cols | A list of columns to check for changes, or `all` to check all columns | `["name", "email"]` | check\_cols = 'all' The `check` snapshot strategy can be configured to track changes to *all* columns by supplying `check_cols = 'all'`. It is better to explicitly enumerate the columns that you want to check. Consider using a surrogate key to condense many columns into a single column. ###### Example usage (Applies to dbt v1.9 and later) snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot_check relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: check check_cols: - status - is_cancelled ``` ###### Example usage with `updated_at` When using the `check` strategy, dbt tracks changes by comparing values in `check_cols`. By default, dbt uses the timestamp to update `dbt_updated_at`, `dbt_valid_from` and `dbt_valid_to` fields. Optionally you can set an `updated_at` column: * If `updated_at` is configured, the `check` strategy uses this column instead, as with the timestamp strategy. * If `updated_at` value is null, dbt defaults to using the current timestamp. Check out the following example, which shows how to use the `check` strategy with `updated_at`: ```yaml snapshots: - name: orders_snapshot relation: ref('stg_orders') config: schema: snapshots unique_key: order_id strategy: check check_cols: - status - is_cancelled updated_at: updated_at ``` In this example: * If at least one of the specified `check_cols `changes, the snapshot creates a new row. If the `updated_at` column has a value (is not null), the snapshot uses it; otherwise, it defaults to the timestamp. * If `updated_at` isn’t set, then dbt automatically falls back to [using the current timestamp](#sample-results-for-the-check-strategy) to track changes. * Use this approach when your `updated_at` column isn't reliable for tracking record updates, but you still want to use it — rather than the snapshot's execution time — whenever row changes are detected. ##### Hard deletes (opt-in) (Applies to dbt v1.9 and later) In dbt v1.9 and higher, the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config replaces the `invalidate_hard_deletes` config to give you more control on how to handle deleted rows from the source. The `hard_deletes` config is not a separate strategy but an additional opt-in feature that can be used with any snapshot strategy. The `hard_deletes` config has three options/fields: | Field | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | `ignore` (default) | No action for deleted records. | | `invalidate` | Behaves the same as the existing `invalidate_hard_deletes=true`, where deleted records are invalidated by setting `dbt_valid_to`. | | `new_record` | Tracks deleted records as new rows using the `dbt_is_deleted` [meta field](#snapshot-meta-fields) when records are deleted. |  When to use the hard\_deletes and invalidate\_hard\_deletes config? **Use `invalidate_hard_deletes` (v1.8 and earlier) if:** * Gaps in the snapshot history (missing records for deleted rows) are acceptable. * You want to invalidate deleted rows by setting their `dbt_valid_to` timestamp to the current time (implicit delete). * You are working with smaller datasets where tracking deletions as a separate state is unnecessary. **Use `hard_deletes: new_record` (v1.9 and higher) if:** * You want to maintain continuous snapshot history without gaps. * You want to explicitly track deletions by adding new rows with a `dbt_is_deleted` column (explicit delete). * You are working with larger datasets where explicitly tracking deleted records improves data lineage clarity. ###### Example usage snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot_hard_delete relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: timestamp updated_at: updated_at hard_deletes: new_record # options are: 'ignore', 'invalidate', or 'new_record' ``` In this example, the `hard_deletes: new_record` config will add a new row for deleted records with the `dbt_is_deleted` column set to `True`. Any restored records are added as new rows with the `dbt_is_deleted` field set to `False`. The resulting table will look like this: | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_is\_deleted | | -- | -------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | False | | 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | False | | 1 | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 | 2024-01-01 12:00 | True | | 1 | restored | 2024-01-01 12:00 | 2024-01-01 12:00 | | False | #### Snapshot meta-fields Snapshot tables will be created as a clone of your source dataset, plus some additional meta-fields. In dbt v1.9+ (or available sooner in [the **v1 Latest** release track in the dbt platform](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md)): * These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md) config. * Use the [`dbt_valid_to_current` config](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. * Use the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config to track deleted records as new rows with the `dbt_is_deleted` meta field when using the `hard_deletes='new_record'` field. | Field | Meaning | Notes | Example | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `dbt_valid_from` | The timestamp when this snapshot row became valid. | Use this to order versions of a record. For a given row, this often matches `dbt_updated_at`, but they mean different things. Refer to [Strategy and meta-field timestamps](#strategy-and-meta-field-timestamps) for examples by strategy. | `snapshot_meta_column_names: {dbt_valid_from: start_date}` | | `dbt_valid_to` | The timestamp when this row became invalidated. For current records, this is `NULL` by default or the value specified in `dbt_valid_to_current`. | The most recent snapshot record will have `dbt_valid_to` set to `NULL` or the specified value. | `snapshot_meta_column_names: {dbt_valid_to: end_date}` | | `dbt_scd_id` | A unique key generated for each snapshot row. | Used internally by dbt to identify each SCD version. Refer to [How dbt\_scd\_id is calculated](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md#how-dbt_scd_id-is-calculated) to learn how dbt builds this key. | `snapshot_meta_column_names: {dbt_scd_id: scd_id}` | | `dbt_updated_at` | The source record's change timestamp when this snapshot row was inserted. | Used internally by dbt. [Strategy and meta-field timestamps](#strategy-and-meta-field-timestamps) describes which value populates this column for each strategy. | `snapshot_meta_column_names: {dbt_updated_at: modified_date}` | | `dbt_is_deleted` | A string value indicating if the record has been deleted. (`True` if deleted, `False` if not deleted). | Added when `hard_deletes='new_record'` is configured. | `snapshot_meta_column_names: {dbt_is_deleted: is_deleted}` | All of these column names can be customized using the `snapshot_meta_column_names` config. Refer to this [example](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md#example) for more details. On insert, `dbt_valid_from` and `dbt_updated_at` are set from the same value. They represent validity start and recorded change time, respectively. ##### Strategy and meta-field timestamps The timestamps used for each column depend on the strategy you use: * For the `timestamp` strategy, the configured `updated_at` column is used to populate the `dbt_valid_from`, `dbt_valid_to` and `dbt_updated_at` columns.  Sample results for the timestamp strategy Snapshot query results at `2024-01-01 11:00` | id | status | updated\_at | | -- | ------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | Snapshot results (note that `11:00` is not used anywhere): | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | ---------------- | -------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | | 2024-01-01 10:47 | Query results at `2024-01-01 11:30`: | id | status | updated\_at | | -- | ------- | ---------------- | | 1 | shipped | 2024-01-01 11:05 | Snapshot results (note that `11:30` is not used anywhere): | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 | | 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | | 2024-01-01 11:05 | Snapshot results with `hard_deletes='new_record'`: | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | dbt\_is\_deleted | | -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 | False | | 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | 2024-01-01 11:05 | False | | 1 | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 | | 2024-01-01 11:20 | True | * For the `check` strategy, the current timestamp is used to populate each column. If configured, the `check` strategy uses the `updated_at` column instead, as with the timestamp strategy.  Sample results for the check strategy Snapshot query results at `2024-01-01 11:00` | id | status | | -- | ------- | | 1 | pending | Snapshot results: | id | status | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | -------------- | ---------------- | | 1 | pending | 2024-01-01 11:00 | | 2024-01-01 11:00 | Query results at `2024-01-01 11:30`: | id | status | | -- | ------- | | 1 | shipped | Snapshot results: | id | status | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 | | 1 | shipped | 2024-01-01 11:30 | | 2024-01-01 11:30 | Snapshot results with `hard_deletes='new_record'`: | id | status | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | dbt\_is\_deleted | | -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 | False | | 1 | shipped | 2024-01-01 11:30 | 2024-01-01 11:40 | 2024-01-01 11:30 | False | | 1 | deleted | 2024-01-01 11:40 | | 2024-01-01 11:40 | True | #### FAQs How do I run one snapshot at a time? To run one snapshot, use the `--select` flag, followed by the name of the snapshot: ```shell $ dbt snapshot --select order_snapshot ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. How often should I run the snapshot command? Snapshots are a batch-based approach to [change data capture](https://en.wikipedia.org/wiki/Change_data_capture). The `dbt snapshot` command must be run on a schedule to ensure that changes to tables are actually recorded! While individual use-cases may vary, snapshots are intended to be run between hourly and daily. If you find yourself snapshotting more frequently than that, consider if there isn't a more appropriate way to capture changes in your source data tables. For best practices on when to use snapshots vs incremental models, refer to [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md). What happens if I add new columns to my snapshot query? When the columns of your source query changes, dbt will attempt to reconcile this change in the destination snapshot table. dbt does this by: 1. Creating new columns from the source query in the destination table 2. Expanding the size of string types where necessary (eg. `varchar`s on Redshift) dbt *will not* delete columns in the destination snapshot table if they are removed from the source query. It will also not change the type of a column beyond expanding the size of varchar columns. That is, if a `string` column is changed to a `date` column in the snapshot source query, dbt will not attempt to change the type of the column in the destination table. Do hooks run with snapshots? Yes! The following hooks are available for snapshots: * [pre-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) * [on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) Can I store my snapshots in a directory other than the \`snapshot\` directory in my project? By default, dbt expects your snapshot files to be located in the `snapshots` subdirectory of your project. To change this, update the [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml snapshot-paths: ["snapshots"] ``` Note that you cannot co-locate snapshots and models in the same directory. Debug Snapshot target is not a snapshot table errors If you see the following error when you try executing the snapshot command: > Snapshot target is not a snapshot table (missing `dbt_scd_id`, `dbt_valid_from`, `dbt_valid_to`) Double check that you haven't inadvertently caused your snapshot to behave like table materializations by setting its `materialized` config to be `table`. Prior to dbt version 1.4, it was possible to have a snapshot like this: ```sql {% snapshot snappy %} {{ config(materialized = 'table', ...) }} ... {% endsnapshot %} ``` dbt is treating snapshots like tables (issuing `create or replace table ...` statements) **silently** instead of actually snapshotting data (SCD2 via `insert` / `merge` statements). When upgrading to dbt versions 1.4 and higher, dbt now raises a Parsing Error (instead of silently treating snapshots like tables) that reads: ```text A snapshot must have a materialized value of 'snapshot' ``` This tells you to change your `materialized` config to `snapshot`. But when you make that change, you might encounter an error message saying that certain fields like `dbt_scd_id` are missing. This error happens because, previously, when dbt treated snapshots as tables, it didn't include the necessary [snapshot meta-fields](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields) in your target table. Since those meta-fields don't exist, dbt correctly identifies that you're trying to create a snapshot in a table that isn't actually a snapshot. When this happens, you have to start from scratch — re-snapshotting your source data as if it was the first time by dropping your "snapshot" which isn't a real snapshot table. Then dbt snapshot will create a new snapshot and insert the snapshot meta-fields as expected. --- ### Add sources to your DAG #### Related reference docs * [Source properties](https://docs.getdbt.com/reference/source-properties.md) * [Source configurations](https://docs.getdbt.com/reference/source-configs.md) * [`{{ source() }}` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) (Applies to dbt v2.0 and later) * [`dbt freshness` command](https://docs.getdbt.com/reference/commands/freshness.md) #### Using sources Sources make it possible to name and describe the data loaded into your warehouse by your Extract and Load tools. By declaring these tables as sources in dbt, you can then * select from source tables in your models using the [`{{ source() }}` function,](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) helping define the lineage of your data * test your assumptions about your source data * calculate the freshness of your source data ##### Declaring a source Sources are defined in `.yml` files nested under a `sources:` key. models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop tables: - name: orders - name: customers - name: stripe tables: - name: payments ``` \*By default, `schema` will be the same as `name`. Add `schema` only if you want to use a source name that differs from the existing schema. If you're not already familiar with these files, be sure to check out [the documentation on properties.yml files](https://docs.getdbt.com/reference/configs-and-properties.md) before proceeding. ##### Selecting from a source Once a source has been defined, it can be referenced from a model using the [`{{ source()}}` function](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md). models/orders.sql ```sql select ... from {{ source('jaffle_shop', 'orders') }} left join {{ source('jaffle_shop', 'customers') }} using (customer_id) ``` dbt will compile this to the full table name: target/compiled/jaffle\_shop/models/my\_model.sql ```sql select ... from raw.jaffle_shop.orders left join raw.jaffle_shop.customers using (customer_id) ``` Using the `{{ source () }}` function also creates a dependency between the model and the source table. [![The source function tells dbt a model is dependent on a source ](/img/docs/building-a-dbt-project/sources-dag.png?v=2 "The source function tells dbt a model is dependent on a source ")](#)The source function tells dbt a model is dependent on a source ##### Testing and documenting sources You can also: * Add data tests to sources * Add descriptions to sources, that get rendered as part of your documentation site These should be familiar concepts if you've already added data tests and descriptions to your models (if not check out the guides on [testing](https://docs.getdbt.com/docs/build/data-tests.md) and [documentation](https://docs.getdbt.com/docs/build/documentation.md)). models/\<filename>.yml ```yaml sources: - name: jaffle_shop description: This is a replica of the Postgres database used by our app tables: - name: orders database: raw description: > One record per order. Includes cancelled and deleted orders. columns: - name: id description: Primary key of the orders table data_tests: - unique - not_null - name: status description: Note that the status can change over time - name: ... - name: ... ``` You can find more details on the available properties for sources in the [reference section](https://docs.getdbt.com/reference/source-properties.md). ##### FAQs What if my source is in a poorly named schema or table? By default, dbt will use the `name:` parameters to construct the source reference. If these names are a little less-than-perfect, use the [schema](https://docs.getdbt.com/reference/resource-properties/schema.md) and [identifier](https://docs.getdbt.com/reference/resource-properties/identifier.md) properties to define the names as per the database, and use your `name:` property for the name that makes sense! models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: postgres_backend_public_schema tables: - name: orders identifier: api_orders ``` In a downstream model: ```sql select * from {{ source('jaffle_shop', 'orders') }} ``` Will get compiled to: ```sql select * from raw.postgres_backend_public_schema.api_orders ``` What if my source is in a different database to my target database? Use the [`database` property](https://docs.getdbt.com/reference/resource-properties/database.md) to define the database that the source is in. models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: jaffle_shop tables: - name: orders - name: customers ``` I need to use quotes to select from my source, what should I do? This is reasonably common on Snowflake in particular. By default, dbt will not quote the database, schema, or identifier for the source tables that you've specified. To force dbt to quote one of these values, use the [`quoting` property](https://docs.getdbt.com/reference/resource-properties/quoting.md): models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop quoting: database: true schema: true identifier: true tables: - name: order_items - name: orders # This overrides the `jaffle_shop` quoting config quoting: identifier: false ``` How do I run data tests on just my sources? To run data tests on all sources, use the following command: ```shell dbt test --select "source:*" ``` (You can also use the `-s` shorthand here instead of `--select`) To run data tests on one source (and all of its tables): ```shell $ dbt test --select source:jaffle_shop ``` And, to run data tests on one source table only: ```shell $ dbt test --select source:jaffle_shop.orders ``` How do I run models downstream of one source? To run models downstream of a source, use the `source:` selector: ```shell $ dbt run --select source:jaffle_shop+ ``` (You can also use the `-s` shorthand here instead of `--select`) To run models downstream of one source table: ```shell $ dbt run --select source:jaffle_shop.orders+ ``` Check out the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) for more examples! #### Source data freshness With a couple of extra configs, dbt can optionally capture the "freshness" of the data in your source tables. This is useful for understanding if your data pipelines are in a healthy state, and is a critical component of defining Service Level Agreements (SLAs) for your warehouse. ##### dbt v2 and dbt State State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). If you're using dbt v2 with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), dbt automatically tracks source freshness using warehouse metadata. You don't need to configure `warn_after` or `error_after` for dbt to detect when source data changes. If you're using [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), use [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) to control how frequently models rebuild based on upstream data changes. You can also configure `loaded_at_field` or `loaded_at_query` on your source for more accurate freshness detection (for example, for streaming data or late-arriving records). However, you should still configure source freshness if you want to: * Receive SLA alerts when sources don't update within expected timeframes. * Define custom freshness logic using `loaded_at_field` or `loaded_at_query` (for example, for streaming data or partial loads). * Track freshness for source views. dbt v2 treats views as "always fresh" since it can't determine freshness from view metadata. ##### Declaring source freshness To configure source freshness information, add a `freshness` block to your source and `loaded_at_field` to your table declaration: models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw config: freshness: # default freshness # changed to config in v1.9 warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at # changed to config in v1.10 tables: - name: orders config: freshness: # make this a little more strict warn_after: {count: 6, period: hour} error_after: {count: 12, period: hour} - name: customers # this inherits the default freshness defined in the jaffle_shop source block at the beginning - name: product_skus config: freshness: null # do not check freshness for this table ``` In the `freshness` block, one or both of `warn_after` and `error_after` can be provided. If neither is provided, then dbt will not calculate freshness for the tables in this source. Additionally, the `loaded_at_field` is required to calculate freshness for a table (except for cases where dbt can leverage warehouse metadata to calculate freshness). If a `loaded_at_field`, or viable alternative, is not provided, then dbt will not calculate freshness for the table. These configs are applied hierarchically, so `freshness` and `loaded_at_field` values specified for a `source` will flow through to all of the `tables` defined in that source. This is useful when all of the tables in a source have the same `loaded_at_field`, as the config can just be specified once in the top-level source definition. ##### Evaluate source freshness (Applies to dbt v2.0 and later) To evaluate freshness for your sources, use [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) with `--resource-type source`: ```shell dbt freshness --resource-type source ``` Running `dbt freshness` without a resource type evaluates every source *and* model that has a `freshness` config. Behind the scenes, dbt uses the freshness properties to construct a `select` query, shown below. You can find this query in the [query logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md). ```sql select max(_etl_loaded_at) as max_loaded_at, convert_timezone('UTC', current_timestamp()) as calculated_at from raw.jaffle_shop.orders ``` The results of this query are used to determine whether the source is fresh or not: (Applies to dbt v2.0 and later) ```text $ dbt freshness --resource-type source Passed source jaffle_shop.orders (last updated 32m 14s ago) [0.42s] Stale source jaffle_shop.customers (last updated 2days 6h 18m ago) [0.39s] [error] [StaleSource (dbt1063)]: Stale source source.jaffle_shop.jaffle_shop.customers --> models/<filename>.yml ``` ##### Build models based on source freshness Our best practice recommendation is to use [data source freshness](https://docs.getdbt.com/docs/build/sources.md#declaring-source-freshness), configured in your `.yml` files using the [`freshness`](https://docs.getdbt.com/reference/resource-configs/freshness.md) config. To build models based on source freshness in dbt: 1. (Applies to dbt v2.0 and later) Run [`dbt freshness --resource-type source`](https://docs.getdbt.com/reference/commands/freshness.md) to check the freshness of your sources. 2. Use the `dbt build --select source_status:fresher+` command to build and test models downstream of fresher sources. Using these commands in order makes sure models update with the latest data. This eliminates wasted compute cycles on unchanged data and builds models *only* when necessary. Set [source freshness checks](https://docs.getdbt.com/docs/deploy/source-freshness.md#enabling-source-freshness-checks) to 30 minutes, then run a job which rebuilds every hour. This setup retrieves all the models and rebuilds them in one attempt if their source freshness has expired. For more information, refer to [Source freshness check frequency](https://docs.getdbt.com/docs/deploy/source-freshness.md#source-freshness-check-frequency). tip You can also use [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) for this use case. Set the [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) config to control how much time must pass since the last upstream data change before dbt triggers a rebuild, without chaining commands manually. ##### Filter Some databases can have tables where a filter over certain columns are required, in order prevent a full scan of the table, which could be costly. In order to do a freshness check on such tables a `filter` argument can be added to the configuration, for example, `filter: _etl_loaded_at >= date_sub(current_date(), interval 1 day)`. For the example above, the resulting query would look like ```sql select max(_etl_loaded_at) as max_loaded_at, convert_timezone('UTC', current_timestamp()) as calculated_at from raw.jaffle_shop.orders where _etl_loaded_at >= date_sub(current_date(), interval 1 day) ``` ##### FAQs How do I exclude a table from a freshness snapshot? Some tables in a data source may be updated infrequently. If you've set a `freshness` property at the source level, this table is likely to fail checks. To work around this, you can set the table's freshness to null (`freshness: null`) to "unset" the freshness for a particular table: models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop config: freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at tables: - name: orders - name: product_skus config: freshness: null # do not check freshness for this table ``` How do I snapshot freshness for one source only? Use the `--select` flag to snapshot freshness for specific sources. Eg: ```shell # Snapshot freshness for all Jaffle Shop tables: $ dbt source freshness --select source:jaffle_shop # Snapshot freshness for a particular source table: $ dbt source freshness --select source:jaffle_shop.orders # Snapshot freshness for multiple particular source tables: $ dbt source freshness --select source:jaffle_shop.orders source:jaffle_shop.customers ``` See the [`source freshness` command reference](https://docs.getdbt.com/reference/commands/source.md) for more information. Are the results of freshness stored anywhere? Yes! (Applies to dbt v2.0 and later) The [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) command measures how recently data was loaded and reports a pass/warning/error for each source or model based on your `warn_after` and `error_after` thresholds. dbt writes the model and source freshness results to `target/freshness.json`. When sources are included, it also writes `target/sources.json` for backward compatibility. After enabling source freshness within a job, configure [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) in your **Project Details** page, which you can find by selecting your account name on the left side menu in dbt and clicking **Account settings**. You can see the current status for source freshness by clicking **View Sources** in the job page. --- ### Administer the Semantic Layer dbt platform | Starter, Enterprise, Enterprise+ With the dbt Semantic Layer, you can centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. This topic shows you how to set up credentials and tokens so that other tools can query the Semantic Layer. #### Prerequisites * Have a dbt Starter, Enterprise, or Enterprise+ account. Available on all [tenant configurations](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). * Ensure your production and development environments are on a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). * Use Snowflake, BigQuery, Databricks, Redshift, Postgres, or Trino. * Create a successful run in the environment where you configure the Semantic Layer. * **Note:** Semantic Layer supports querying in Deployment environments; development querying is coming soon. * Understand [MetricFlow's](https://docs.getdbt.com/docs/build/about-metricflow.md) key concepts powering the Semantic Layer. * Note that the Semantic Layer doesn't support using [Single sign-on (SSO)](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for [production credentials](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens), though SSO is supported for development user accounts. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Administer the Semantic Layer You must be part of the Owner group and have the correct [license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to administer the Semantic Layer at the environment and project level. * Enterprise+ and Enterprise plan: * Developer license with Account Admin permissions, or * Owner with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. * Starter plan: Owner with a Developer license. * Free trial: You are on a free trial of the Starter plan as an Owner, which means you have access to the dbt Semantic Layer. ##### 1. Select environment Select the environment where you want to enable the Semantic Layer: 1. Navigate to **Account settings** in the navigation menu. 2. Under **Settings**, click **Projects** and select the specific project you want to enable the Semantic Layer for. 3. In the **Project details** page, navigate to the **Semantic Layer** section. Select **Configure Semantic Layer**. [![Semantic Layer section in the 'Project details' page](/img/docs/dbt-platform/semantic-layer/new-sl-configure.png?v=2 "Semantic Layer section in the 'Project details' page")](#)Semantic Layer section in the 'Project details' page 4. In the **Set Up Semantic Layer Configuration** page, select the deployment environment you want for the Semantic Layer and click **Save**. This provides administrators with the flexibility to choose the environment where the Semantic Layer will be enabled. [![Select the deployment environment to run your Semantic Layer against.](/img/docs/dbt-platform/semantic-layer/sl-select-env.png?v=2 "Select the deployment environment to run your Semantic Layer against.")](#)Select the deployment environment to run your Semantic Layer against. ##### 2. Configure credentials and create tokens There are two options for setting up Semantic Layer using API tokens: * [Add a credential and create service tokens](#add-a-credential-and-create-service-tokens) * [Configure user credentials and create personal tokens](#configure-development-credentials-and-create-a-personal-token) ###### Add a credential and create service tokens The first option is to use [service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) for authentication which are tied to an underlying data platform credential that you configure. The credential configured is used to execute queries that the Semantic Layer issues against your data platform. This credential controls the physical access to underlying data accessed by the Semantic Layer, and all access policies set in the data platform for this credential will be respected. | Feature | Starter plan | Enterprise+ and Enterprise plan | | --------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Service tokens | Can create multiple service tokens linked to one credential. | Can use multiple credentials and link multiple service tokens to each credential. Note that you cannot link a single service token to more than one credential. | | Credentials per project | One credential per project. | Can [add multiple](#4-add-more-credentials) credentials per project. | | Link multiple service tokens to a single credential | ✅ | ✅ | *If you're on a Starter plan and need to add more credentials, consider upgrading to our [Enterprise+ or Enterprise plan](https://www.getdbt.com/contact). All Enterprise users can refer to [Add more credentials](#4-add-more-credentials) for detailed steps on adding multiple credentials.* ###### 1. Select deployment environment * After selecting the deployment environment, you should see the **Credentials & service tokens** page. * Click the **Add Semantic Layer credential** button. ###### 2. Configure credential * In the **1. Add credentials** section, enter the credentials specific to your data platform that you want the Semantic Layer to use. * Use credentials with minimal privileges. The Semantic Layer requires read access to the schema(s) containing the dbt models used in your semantic models for downstream applications * Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. [![Add credentials and map them to a service token. ](/img/docs/dbt-platform/semantic-layer/sl-add-credential.png?v=2 "Add credentials and map them to a service token. ")](#)Add credentials and map them to a service token. ###### 3. Create or link service tokens * If you have permission to create service tokens, you’ll see the [**Map new service token** option](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#map-service-tokens-to-credentials) after adding the credential. Name the token, set permissions to 'Semantic Layer Only' and 'Metadata Only', and click **Save**. * Once the token is generated, you won't be able to view this token again, so make sure to record it somewhere safe. * If you don’t have access to create service tokens, you’ll see a message prompting you to contact your admin to create one for you. Admins can create and link tokens as needed. [![If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you.](/img/docs/dbt-platform/semantic-layer/sl-credential-no-service-token.png?v=2 "If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you.")](#)If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you. info * Starter plans can create multiple service tokens that link to a single underlying credential, but each project can only have one credential. * All Enterprise plans can [add multiple credentials](#4-add-more-credentials) and map those to service tokens for tailored access. [Book a free live demo](https://www.getdbt.com/contact) to discover the full potential of dbt Enterprise and higher plans. ###### Configure development credentials and create a personal token Using [personal access tokens (PATs)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) is also a supported authentication method for the dbt Semantic Layer. This enables user-level authentication, reducing the need for sharing tokens between users. When you authenticate using PATs, queries are run using your personal user credentials. To use PATs in Semantic Layer: 1. Configure your user credentials. 1. Click your account name at the bottom left-hand menu and go to **Account settings** > **Credentials**. 2. Select your project. 3. Click **Edit**. 4. Go to **User credentials** and enter your details. 5. Click **Save**. 2. [Create a personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Make sure to copy the token. You can use the generated PAT as the authentication method for Semantic Layer [APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) and [integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). ##### 3. View connection detail 1. Go back to the **Project details** page for connection details to connect to downstream tools. 2. Copy and share the Environment ID, service or personal token, Host, as well as the service or personal token name to the relevant teams for BI connection setup. If your tool uses the GraphQL API, save the GraphQL API host information instead of the JDBC URL. For info on how to connect to other integrations, refer to [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). [![After configuring, you'll be provided with the connection details to connect to you downstream tools.](/img/docs/dbt-platform/semantic-layer/sl-configure-example.png?v=2 "After configuring, you'll be provided with the connection details to connect to you downstream tools.")](#)After configuring, you'll be provided with the connection details to connect to you downstream tools. ##### 4. Add more credentials [Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") All dbt Enterprise plans can optionally add multiple credentials and map them to service tokens, offering more granular control and tailored access for different teams, which can then be shared to relevant teams for BI connection setup. These credentials control the physical access to underlying data accessed by the Semantic Layer. We recommend configuring credentials and service tokens to reflect your teams and their roles. For example, create tokens or credentials that align with your team's needs, such as providing access to finance-related schemas to the Finance team.  Considerations for linking credentials * Admins can link multiple service tokens to a single credential within a project, but each service token can only be linked to one credential per project. * When you send a request through the APIs, the service token of the linked credential will follow access policies of the underlying view and tables used to build your semantic layer requests. * Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. ###### 1. Add more credentials * After configuring your environment, on the **Credentials & service tokens** page, click the **Add Semantic Layer credential** button to create multiple credentials and map them to a service token. <br /> * In the **1. Add credentials** section, fill in the data platform's credential fields. We recommend using “read-only” credentials. [![Add credentials and map them to a service token. ](/img/docs/dbt-platform/semantic-layer/sl-add-credential.png?v=2 "Add credentials and map them to a service token. ")](#)Add credentials and map them to a service token. ###### 2. Map service tokens to credentials * In the **2. Map new service token** section, [map a service token to the credential](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#map-service-tokens-to-credentials) you configured in the previous step. dbt automatically selects the service token permission set you need (Semantic Layer Only and Metadata Only). * To add another service token during configuration, click **Add Service Token**. * You can link more service tokens to the same credential later on in the **Semantic Layer Configuration Details** page. To add another service token to an existing Semantic Layer configuration, click **Add service token** under the **Linked service tokens** section. * Click **Save** to link the service token to the credential. Remember to copy and save the service token securely, as it won't be viewable again after generation. [![Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control.](/img/docs/dbt-platform/semantic-layer/sl-credentials-service-token.png?v=2 "Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control.")](#)Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control. ###### 3. Delete credentials * To delete a credential, go back to the **Credentials & service tokens** page. * Under **Linked Service Tokens**, click **Edit** and, select **Delete Credential** to remove a credential. When you delete a credential, any service tokens mapped to that credential in the project will no longer work and will break for any end users. ##### Delete configuration You can delete the entire Semantic Layer configuration for a project. Note that deleting the Semantic Layer configuration will remove all credentials and unlink all service tokens to the project. It will also cause all queries to the Semantic Layer to fail. Follow these steps to delete the Semantic Layer configuration for a project: 1. Navigate to the **Project details** page. 2. In the **Semantic Layer** section, select **Delete Semantic Layer**. 3. Confirm the deletion by clicking **Yes, delete semantic layer** in the confirmation pop up. To re-enable the dbt Semantic Layer setup in the future, you will need to recreate your setup configurations by following the [previous steps](#set-up-dbt-semantic-layer). If your semantic models and metrics are still in your project, no changes are needed. If you've removed them, you'll need to set up the YAML configs again. [![Delete the Semantic Layer configuration for a project.](/img/docs/dbt-platform/semantic-layer/sl-delete-config.png?v=2 "Delete the Semantic Layer configuration for a project.")](#)Delete the Semantic Layer configuration for a project. #### Additional configuration The following are the additional flexible configurations for Semantic Layer credentials. ##### Map service tokens to credentials * After configuring your environment, you can map additional service tokens to the same credential if you have the required [permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#permission-sets). * Go to the **Credentials & service tokens** page and click the **+Add Service Token** button in the **Linked Service Tokens** section. * Type the service token name and select the permission set you need (Semantic Layer Only and Metadata Only). * Click **Save** to link the service token to the credential. * Remember to copy and save the service token securely, as it won't be viewable again after generation. [![Map additional service tokens to a credential.](/img/docs/dbt-platform/semantic-layer/sl-add-service-token.gif?v=2 "Map additional service tokens to a credential.")](#)Map additional service tokens to a credential. ##### Unlink service tokens * Unlink a service token from the credential by clicking **Unlink** under the **Linked service tokens** section. If you try to query the Semantic Layer with an unlinked credential, you'll experience an error in your BI tool because no valid token is mapped. ##### Manage from service token page **View credential from service token** * View your Semantic Layer credential directly by navigating to the **API tokens** and then **Service tokens** page. * Select the service token to view the credential it's linked to. This is useful if you want to know which service tokens are mapped to credentials in your project. ###### Create a new service token * From the **Service tokens** page, create a new service token and map it to the credential(s) (assuming the semantic layer permission exists). This is useful if you want to create a new service token and directly map it to a credential in your project. * Make sure to select the correct permission set for the service token (Semantic Layer Only and Metadata Only). [![Create a new service token and map credentials directly on the separate 'Service tokens page'.](/img/docs/dbt-platform/semantic-layer/sl-create-service-token-page.png?v=2 "Create a new service token and map credentials directly on the separate 'Service tokens page'.")](#)Create a new service token and map credentials directly on the separate 'Service tokens page'. #### Next steps * If you haven't created metrics yet, follow the [Quickstart for the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md). * Now that you've set up your credentials and tokens, start querying your metrics with the [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). * [Optimize querying performance](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) using declarative caching. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. * If you haven't already, learn how to [build you metrics and semantic models](https://docs.getdbt.com/docs/build/build-metrics-intro.md) in your development tool of choice. * Learn about commonly asked [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md). #### FAQs  How does caching interact with access controls? Cached data is stored separately from the underlying models. If metrics are pulled from the cache, we don’t have the security context applied to those tables at query time. In the future, we plan to clone credentials, identify the minimum access level needed, and apply those permissions to cached tables. --- ### Advanced CI dbt platform | Enterprise, Enterprise+ [Continuous integration workflows](https://docs.getdbt.com/docs/deploy/continuous-integration.md) help increase the governance and improve the quality of the data. Additionally for these CI jobs, you can use Advanced CI features, such as [compare changes](#compare-changes), that provide details about the changes between what's currently in your production environment and the pull request's latest commit, giving you observability into how data changes are affected by your code changes. By analyzing the data changes that code changes produce, you can ensure you're always shipping trustworthy data products as you're developing. How to enable this feature You can opt into Advanced CI in dbt. Please refer to [Account access to Advanced CI features](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-ci-features) to learn how enable it in your dbt account. #### Prerequisites * You have a dbt Enterprise or Enterprise+ account. * You have [Advanced CI features](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-features) enabled. * You use a supported data platform: BigQuery, Databricks, Postgres, Redshift, or Snowflake. Support for additional data platforms coming soon. #### Compare changes feature For [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) that have the [**dbt compare** option enabled](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs), dbt compares the changes between the last applied state of the production environment (defaulting to deferral for lower compute costs) and the latest changes from the pull request, whenever a pull request is opened or new commits are pushed. You can also compare changes in development. For more details, see [Compare changes in development](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes-in-development). dbt reports the comparison differences in: * **dbt** — Shows the changes (if any) to the data's primary keys, rows, and columns in the [Compare tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#compare-tab) from the [Job run details](https://docs.getdbt.com/docs/deploy/run-visibility.md#job-run-details) page. * **The pull request from your Git provider** — Shows a summary of the changes as a Git comment. [![Example of the Compare tab](/img/docs/dbt-platform/example-ci-compare-changes-tab.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab [![Example of Modified rows tab which shows you details of the modified rows.](/img/docs/dbt-platform/advanced-ci-modified-rows.png?v=2 "Example of Modified rows tab which shows you details of the modified rows.")](#)Example of Modified rows tab which shows you details of the modified rows. ###### Compare changes in development You can compare changes locally in development to preview data changes caused by your local edits (for example, added/removed rows or joins) directly in your editor, and without waiting on CI. Compare changes in development is available through the dbt VS Code extension, which is powered by dbt v2. For more details on how to use this feature, see [Compare changes in local development](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md).  Differences between compare changes in development and Advanced CI compare changes | Aspect | In development (compare changes) | In deployment (Advanced CI) | | ------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Affects** | Development for one modified model at a time | Deployment for all modified models in a project | | **Trigger** | On-demand in editor | PR open/update and CI job | | **Scope** | Your working copy and local target | Branch head versus prod state in CI | | **Output** location | Compare panel in VS Code/Cursor. Does not create a PR comment in Git provider | Deployment job compare tab and PR summary comment in Git provider | | **Data caching** | Editor-side | dbt platform [caches](https://docs.getdbt.com/docs/deploy/advanced-ci.md#about-the-cached-data) limited samples | | **Governance** | Local development credentials | Production credentials | ##### Optimizing comparisons When an [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) column is specified on your model, compare changes can optimize comparisons by using only the overlapping timeframe (meaning the timeframe exists in both the CI and production environment), helping you avoid incorrect row-count changes and return results faster. This is useful in scenarios like: * **Subset of data in CI** — When CI builds only a [subset of data](https://docs.getdbt.com/best-practices/best-practice-workflows.md#limit-the-data-processed-when-in-development) (like the most recent 7 days), compare changes would interpret the excluded data as "deleted rows." Configuring `event_time` allows you to avoid this issue by limiting comparisons to the overlapping timeframe, preventing false alerts about data deletions that are just filtered out in CI. * **Fresher data in CI than in production** — When your CI job includes fresher data than production (because it has run more recently), compare changes would flag the additional rows as "new" data, even though they’re just fresher data in CI. With `event_time` configured, the comparison only includes the shared timeframe and correctly reflects actual changes in the data. [![event\_time ensures the same time-slice of data is accurately compared between your CI and production environments.](/img/docs/deploy/apples_to_apples.png?v=2 "event_time ensures the same time-slice of data is accurately compared between your CI and production environments.")](#)event\_time ensures the same time-slice of data is accurately compared between your CI and production environments. #### About the cached data After [comparing changes](#compare-changes), dbt stores a cache of no more than 100 records for each modified model for preview purposes. By caching this data, you can view the examples of changed data without rerunning the comparison against the data warehouse every time (optimizing for lower compute costs). To display the changes, dbt uses a cached version of a sample of the data records. These data records are queried from the database using the connection configuration (such as user, role, service account, and so on) that's set in the CI job's environment. You control what data to use. This may include synthetic data if pre-production or development data is heavily regulated or sensitive. * The selected data is cached on dbt Labs' systems for up to 30 days. No data is retained on dbt Labs' systems beyond this period. * The cache is encrypted and stored in an Amazon S3 or Azure blob storage in your account’s region. * dbt Labs will not access cached data from Advanced CI for its benefit and the data is only used to provide services as directed by you. * Third-party subcontractors, other than storage subcontractors, will not have access to the cached data. If you access a CI job run that's more than 30 days old, you will not be able to see the comparison results. Instead, a message will appear indicating that the data has expired. [![Example of message about expired data in the Compare tab](/img/docs/deploy/compare-expired.png?v=2 "Example of message about expired data in the Compare tab")](#)Example of message about expired data in the Compare tab #### Connection permissions The compare changes feature uses the same credentials as the CI job, as defined in the CI job’s environment. The dbt administrator must ensure that client CI credentials are appropriately restricted since all customer's account users will be able to view the comparison results and the cached data. If using dynamic data masking in the data warehouse, the cached data will no longer be dynamically masked in the Advanced CI output, depending on the permissions of the users who view it. dbt Labs recommends limiting user access to unmasked data or considering using synthetic data for the Advanced CI testing functionality. [![Example of credentials in the user settings](/img/docs/deploy/compare-credentials.png?v=2 "Example of credentials in the user settings")](#)Example of credentials in the user settings #### Troubleshooting  Compare changes CI models need to be on same database host/connection Compare Changes only works if both CI and production models live on the same database host/connection. Compare Changes runs SQL queries in the current CI job’s environment to compare the CI model (like `ci.dbt_cloud_123.foo`) to the production model (`prod.analytics.foo`). If the CI job defers to a production job that's on a different database connection or host, then the compare changes feature will not work as expected. This is because the CI environment can't access or query production objects on another host. In the following example, the CI job can’t access the production model to compare them because they’re on different database hosts: * The dbt CI job in environment `ci.dbt_cloud_123.foo` that connects to host `abc123.rds.amazonaws.com` * The dbt production job in environment `prod.analytics.foo` that connects to host `def456.rds.amazonaws.com` --- ### Advanced data modeling This section covers advanced topics for the Semantic Layer and MetricFlow, such as data modeling workflows, and more. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) ###### [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) [Use fill\_nulls\_with to set null metric values to zero, ensuring numeric values for every data row.](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) ###### [Metrics as dimensions with metric filters](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) [Add metrics as dimensions to your metric filters to create more complex metrics and gain more insights.](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) --- ### Amazon Web Services and Apache Iceberg dbt materializes Iceberg tables on AWS through the [AWS Glue Data Catalog](#aws-glue-data-catalog), the catalog that AWS analytics engines share. There are two ways to configure it: * **Simplest:** Set `table_type = 'iceberg'` in your model config to instruct dbt to materialize this model as an Iceberg table in the AWS Glue Data Catalog through Amazon Athena. * **Extensible:** Define an Iceberg catalog in `catalogs.yml` and configure this model with `catalog_name`. This is required to target [Amazon S3 Tables](#amazon-s3-tables). dbt supports creating Iceberg tables for two Athena materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental), with the `merge` strategy #### AWS Glue Data Catalog On AWS, the [AWS Glue Data Catalog](https://docs.aws.amazon.com/athena/latest/ug/glue-athena.html) is the metadata layer that registers Iceberg tables. It is the shared source of truth that lets one engine write a table and another read it: Amazon Athena, Amazon Redshift, Amazon EMR, and Apache Spark all resolve Iceberg tables through Glue. To create these tables with dbt through Amazon Athena, use the [dbt-athena](https://docs.getdbt.com/docs/local/connect-data-platform/athena-setup.md). Amazon S3 Tables extends this model with a managed Iceberg catalog that owns its own storage, surfaced to Glue through a federated catalog. Both paths register governed Iceberg tables that any Glue-connected engine can read. #### Amazon Athena and Iceberg Athena queries Iceberg tables registered in the AWS Glue Data Catalog. ##### Prerequisites * Iceberg tables require [Athena engine version 3](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html). * Each table also needs a unique storage location, unless you use Amazon S3 Tables, which manages storage for you. 1. To connect dbt to Athena locally, define a dbt-athena target in `profiles.yml`: profiles.yml ```yaml my_profile: target: dev outputs: dev: type: athena s3_staging_dir: s3://my-athena-results-bucket/staging/ region_name: <region> database: awsdatacatalog schema: my_schema threads: 4 ``` For the full set of connection options, see the [dbt-athena setup page](https://docs.getdbt.com/docs/local/connect-data-platform/athena-setup.md). 2. To create an Iceberg table without `catalogs.yml`, set `table_type='iceberg'` in your model config: models/my\_iceberg\_model.sql ```sql {{ config( materialized = 'table', table_type = 'iceberg', format = 'parquet' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` #### Amazon S3 Tables [Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html) is AWS's managed Apache Iceberg storage. S3 Tables owns its own storage location and Iceberg snapshots, so it isn't addressed the same way as a regular Glue-backed Iceberg table — it needs the dedicated `type: s3_tables` catalog integration. Adapter availability S3 Tables support is available on dbt platform and in `dbt-athena` version `1.11.1` or later. ##### Enable the S3 Tables integration with Glue Athena accesses an S3 Tables bucket through a federated Glue catalog named `s3tablescatalog`. Each table bucket appears under this catalog. Enable the integration once per AWS account and Region either from the Amazon S3 console by selecting **Enable integration** when you create a table bucket or using the AWS CLI: ```shell aws glue create-catalog \ --name "s3tablescatalog" \ --catalog-input '{ "Description": "Federated catalog for S3 Tables", "FederatedCatalog": { "Identifier": "arn:aws:s3tables:<region>:<account-id>:bucket/*", "ConnectionName": "aws:s3tables" }, "CreateDatabaseDefaultPermissions": [{ "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }], "CreateTableDefaultPermissions": [{ "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }] }' ``` This example uses `IAM_ALLOWED_PRINCIPALS` to configure IAM-only access. Access is controlled by the query role's IAM permissions, without requiring AWS Lake Formation setup. You can optionally use Lake Formation if you need fine-grained permissions or centrally managed access. For both options, refer to [Enabling S3 Tables integration with the Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/enable-s3-tables-catalog-integration.html). ##### Configure your dbt project 1. Enable the `use_catalogs_v2` behavior flag in `dbt_project.yml`: dbt\_project.yml ```yaml flags: use_catalogs_v2: true ``` 2. Create a `catalogs.yml` at the top level of your dbt project: catalogs.yml ```yaml catalogs: - name: s3_tables_catalog type: s3_tables table_format: iceberg config: athena: catalog_database: s3tablescatalog/<your-table-bucket-name> file_format: parquet ``` Set `catalog_database` to the federated Glue catalog name for your S3 table bucket (the same name Athena uses to address it). dbt uses this value as the model's `database`. Omitting `catalog_database` falls back to the profile's `database`. The model's resolved `schema` identifies the S3 Tables namespace. 3. Add the `catalog_name` config parameter in either a config block (inside the `.sql` model file), properties YAML file (model folder), or your project YAML file (`dbt_project.yml`): models/my\_s3\_tables\_model.sql ```sql {{ config( materialized = 'table', catalog_name = 's3_tables_catalog', schema = 'my_namespace' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` 4. Run your dbt model: ```shell dbt run -s my_s3_tables_model ``` ##### Create an incremental model Full refresh is not supported The implementation added in [dbt-adapters#2047](https://github.com/dbt-labs/dbt-adapters/pull/2047) supports initial and repeated incremental `merge` runs, but not `--full-refresh` for incremental S3 Tables models. Refer to [Full-refresh limitation](#full-refresh-limitation) before rebuilding an incremental table. To update an existing table with new or changed rows, use the `incremental` materialization with the `merge` strategy. Set a `unique_key` so dbt can match incoming records to existing rows: models/my\_incremental\_s3\_tables\_model.sql ```sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', unique_key = 'id', catalog_name = 's3_tables_catalog', schema = 'my_namespace' ) }} select * from {{ ref('my_s3_tables_model') }} ``` On the first run, when the target table does not yet exist, dbt creates the incremental model using `CREATE TABLE AS` (the same drop-and-recreate-aware path the `table` materialization uses), then switches to `merge` on subsequent runs. No `--full-refresh` is needed to bootstrap the table. ##### How dbt writes to S3 Tables The `s3_tables` catalog integration accounts for storage and DDL differences between S3 Tables and a standard Glue-backed Iceberg table: * **Storage locations:** dbt omits `LOCATION` from `CREATE TABLE AS` because S3 Tables manages storage and rejects an explicit location. * **Table replacement:** dbt drops and recreates tables instead of renaming them. S3 Tables doesn't support `ALTER TABLE ... RENAME`, which the standard Glue table materialization uses to swap tables with near-zero downtime. dbt removes the table's metadata through the Glue Data Catalog rather than issuing SQL `DROP TABLE` or deleting S3 files directly. * **Table recognition:** dbt recognizes S3 Tables as Iceberg tables even though Glue reports their `TableType` as `customer`, or `aws` for service-managed tables. ##### Full-refresh limitation Running an incremental S3 Tables model with `--full-refresh` fails with this error: ```text An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: Unsupported DDL query for S3 table buckets ``` This happens because the incremental materialization's full-refresh path still swaps tables with `ALTER TABLE ... RENAME TO ...__bkp`, which S3 Tables doesn't support. The `table` materialization's drop-and-recreate fix doesn't currently extend to this path. Until this is addressed upstream, avoid `--full-refresh` on incremental S3 Tables models. To rebuild a table: * Drop the table through the S3 Tables API or console, then rerun the model. * Rebuild it using the `table` materialization. #### AWS-specific configs for Iceberg catalogs Supply these configs nested under `config.athena` in a `catalogs.yml` catalog entry: | Field | Required | Description | | ------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `catalog_database` | Optional | Federated Glue catalog name for the target bucket, for example `s3tablescatalog/<bucket-name>`. Routes the model's `database`. Falls back to the profile's `database` if omitted. | | `file_format` | Optional | File format for written data files. Defaults to `parquet`. | #### Other AWS engines This page covers Iceberg catalog integration through `dbt-athena`. For support for additional AWS engines, refer to [Apache Iceberg support adapters page](https://docs.getdbt.com/docs/build/iceberg/apache-iceberg-support.md). Today, engines such as dbt-spark (on Amazon EMR), dbt-trino, and dbt-redshift can read and write Glue-registered Iceberg tables by configuring the engine's own Iceberg catalog against the AWS Glue Data Catalog or the S3 Tables Iceberg REST endpoint, but they do not yet expose a native AWS `catalog_type` in `catalogs.yml`. --- ### Analyses #### Overview dbt's notion of `models` makes it easy for data teams to version control and collaborate on data transformations. Sometimes though, a certain SQL statement doesn't quite fit into the mold of a dbt model. These more "analytical" SQL files can be versioned inside of your dbt project using the `analysis` functionality of dbt. Any `.sql` files found in the `analyses/` directory of a dbt project will be compiled, but not executed. This means that analysts can use dbt functionality like `{{ ref(...) }}` to select from models in an environment-agnostic way. In practice, an analysis file might look like this (via the [open source Quickbooks models](https://github.com/dbt-labs/quickbooks)): analyses/running\_total\_by\_account.sql ```sql -- analyses/running_total_by_account.sql with journal_entries as ( select * from {{ ref('quickbooks_adjusted_journal_entries') }} ), accounts as ( select * from {{ ref('quickbooks_accounts_transformed') }} ) select txn_date, account_id, adjusted_amount, description, account_name, sum(adjusted_amount) over (partition by account_id order by id rows unbounded preceding) from journal_entries order by account_id, id ``` You can configure individual analyses in their property file. For example: analyses/\<filename>.yml ```yaml analyses: - name: running_total_by_account description: Calculates the running total of adjusted amounts per account over time. config: enabled: true docs: show: true node_color: purple tags: ['finance', 'accounting'] columns: - name: account_id description: The unique identifier for the account. - name: adjusted_amount description: The adjusted transaction amount for the account. ``` For more information, refer to [Analysis properties](https://docs.getdbt.com/reference/analysis-properties.md). (Applies to dbt v1.12 and later) #### Project-level configuration You can also configure analyses at the project level in `dbt_project.yml`. To configure analyses at the project level, set the [`require_corrected_analysis_fqns`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md) flag to `true` and add an `analyses` block in your `dbt_project.yml`. The project-level configuration applies to existing analyses in the `analyses/` folder — for example, setting `+enabled: false` disables them all. dbt\_project.yml ```yaml flags: require_corrected_analysis_fqns: true analyses: +enabled: true | false ``` #### Compiling analyses To compile this analysis into runnable sql, run: ```text dbt compile ``` Then, look for the compiled SQL file in `target/compiled/{project name}/analyses/running_total_by_account.sql`. This SQL can then be pasted into a data visualization tool, for instance. Note that no `running_total_by_account` relation will be materialized in the database as this is an `analysis`, not a `model`. --- ### Apache Iceberg support Apache Iceberg is an open table format that brings greater portability and interoperability to the data ecosystem. By standardizing how data is stored and accessed, Iceberg enables teams to use the same data across multiple engines and platforms, without replication. There are multiple layers of Iceberg support, in data platform and in dbt: * **Iceberg table format** — An open-source table format. Iceberg tables are a combination of data files in object storage (such as Parquet files in an S3 bucket), as well as metadata files (recording the table's schema, versioning, and more) that also live in object storage. * **Iceberg data catalog** — An open-source specification for a metadata system that tracks the schema, partition, and versions of multiple Iceberg tables. * **Iceberg REST protocol** (also referred to as the Iceberg REST API) — Defines standard endpoints for interacting with Iceberg-compatible catalogs. In theory, the Iceberg standard enables one query engine to read from or write to Iceberg tables in an external catalog, managed by another engine or platform. In practice, different Iceberg catalogs work differently, and different query engines support the Iceberg spec to varying degrees. To the extent possible, dbt tries to abstract away the complexity of table formats, and the divergence among vendor-specific Iceberg implementations, so teams can focus on delivering reliable, well-modeled data. To learn more, select one of the following tiles: [![](/img/icons/dbt-icon.svg)](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) ###### [Iceberg catalogs](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) [About Iceberg catalogs](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) [![](/img/icons/dbt-icon.svg)](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) ###### [Using table\_format + catalogs.yml](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) [dbt support for Iceberg + catalogs](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) [![](/img/icons/snowflake.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) ###### [Snowflake + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) [Snowflake Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) [![](/img/icons/bigquery.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md) ###### [BigQuery + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md) [BigQuery Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md) [![](/img/icons/databricks.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md) ###### [Databricks + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md) [Databricks Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md) [![](/img/icons/duckdb-seeklogo.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/duckdb-iceberg-support.md) ###### [DuckDB + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/duckdb-iceberg-support.md) [DuckDB Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/duckdb-iceberg-support.md) [![](/img/icons/aws.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/aws-iceberg-support.md) ###### [AWS + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/aws-iceberg-support.md) [AWS Iceberg configurations, including Amazon S3 Tables](https://docs.getdbt.com/docs/build/iceberg/adapters/aws-iceberg-support.md) --- ### Apache Ossie semantic layer documents Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). dbt v1.12 supports the [Apache Ossie](https://github.com/apache/ossie) standard for defining semantic models and metrics. You can place Ossie-format `.json` files in an `osi/` directory at the root of your project, or configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml` to use one or more custom directories relative to your project root. dbt parses them into the manifest alongside any native dbt semantic models. Ossie-sourced definitions and native dbt semantic models can coexist in the same project. #### Prerequisites * You must be on dbt v1.12. v2 support is coming soon. * Ossie documents must use version `0.1.0` or `0.1.1`. Any other version string raises a parse error. #### Defining semantic models using Ossie documents To define semantic models with Ossie documents: 1. Create an `osi/` directory at the root of your dbt project, at the same level as `dbt_project.yml`. To use one or more custom directories instead, configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml` with paths relative to your project root. 2. Add one or more Ossie `.json` files to the directory. You can organize files into subdirectories; dbt scans the entire directory tree. The `source` field must be the fully qualified warehouse location of a dbt model in this project, in the form `database.schema.alias` (for example, `my_database.my_schema.fct_orders`). dbt matches each dataset on database, schema, and model alias. Each dataset `source` must resolve to a dbt model. For restrictions on dataset sources, refer to [Limitations](#limitations). The following is an example Ossie document that defines a semantic model on a dbt model called `fct_orders`: ```json { "version": "0.1.1", "semantic_model": [ { "name": "orders", "datasets": [ { "name": "orders", "source": "my_database.my_schema.fct_orders" } ] } ] } ``` This example defines a semantic model only. To add metrics, include a `metrics` array on the semantic model per the [Ossie specification](https://github.com/apache/ossie). 3. Run any command that triggers compilation, such as `dbt compile` or `dbt run`. dbt automatically discovers and parses Ossie files. The resulting semantic models (and metrics, when defined in your Ossie documents) appear in [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) in your `target/` directory, including [`manifest.json`](https://docs.getdbt.com/reference/artifacts/manifest-json.md), [`semantic_manifest.json`](https://docs.getdbt.com/reference/artifacts/sl-manifest.md), and [`osi_document.json`](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#osi-document). #### Limitations * dbt scans only the root project's Ossie directories (configured through [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md), default `osi/`). Ossie files in installed dependency packages are ignored. * Each Ossie dataset source must resolve to a dbt model. Ossie documents that reference sources, seeds, snapshots, or external tables are not supported. * If the Ossie converter encounters unsupported metric types or other constructs, those elements are dropped and dbt emits a warning (event code `I078`), but parsing continues. Warnings appear in the CLI and in `logs/dbt.log`; for more information, refer to [Events and logs](https://docs.getdbt.com/reference/events-logging.md). --- ### Artifacts dbt platform When running dbt jobs, dbt generates and saves *artifacts*. You can use these artifacts, like `manifest.json`, `catalog.json`, and `sources.json` to power different aspects of the dbt platform, namely: [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), and [source freshness reporting](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness). #### Create dbt Artifacts [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) uses the metadata provided by the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) to display the details about [the state of your project](https://docs.getdbt.com/docs/dbt-apis/project-state.md). It uses metadata from your staging and production [deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md). Catalog automatically retrieves the metadata updates after each job run in the production or staging deployment environment so it always has the latest results for your project — meaning it's always automatically updated after each job run. To view a resource, its metadata, and what commands are needed, refer to [generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details.  For dbt Docs The following steps are for legacy dbt Docs only. For the current documentation experience, see [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). While running any job can produce artifacts, you should only associate one production job with a given project to produce the project's artifacts. You can designate this connection on the **Project details** page. To access this page: 1. From the dbt platform, click on your account name in the left side menu and select **Account settings**. 2. Select your project, and click **Edit** in the lower right. 3. Under **Artifacts**, select the jobs you want to produce documentation and source freshness artifacts for. [![Configuring Artifacts](/img/docs/dbt-platform/using-dbt-platform/project-level-artifact-updated.png?v=2 "Configuring Artifacts")](#)Configuring Artifacts If you don't see your job listed, you might need to edit the job and select **Run source freshness** When you add a production job to a project, dbt updates the content and provides links to the production documentation and source freshness artifacts it generated for that project. You can see these links by clicking **Deploy** in the upper left, selecting **Jobs**, and then selecting the production job. From the job page, you can select a specific run to see how artifacts were updated for that run only. ##### Documentation Navigate to [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) through the **Explore** link to view your project's resources and lineage to gain a better understanding of its latest production state. To view a resource, its metadata, and what commands are needed, refer to [generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details.  For dbt Docs When set up, dbt updates the Documentation link in the header tab so it links to documentation for this job. This link always directs you to the latest version of the documentation for your project. ##### Source Freshness To view the latest source freshness result, refer to [generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more detail. Then navigate to Catalog through the **Explore** link.  For dbt Docs Configuring a job for the Source Freshness artifact setting also updates the data source link under **Orchestration** > **Data sources**. The link points to the latest Source Freshness report for the selected job. --- ### Available dbt versions Whether you're using the CLI or working within the dbt platform, your environments are aligned with a versioned release of dbt. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions.md) ###### [About dbt versions](https://docs.getdbt.com/docs/dbt-versions.md) [Learn about semantic versioning for dbt engines, and how long those versions are supported.](https://docs.getdbt.com/docs/dbt-versions.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) ###### [dbt v2 version compatibility](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) [Learn about version compatibility between v2, LSP, and VS Code extension.](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) ###### [About release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) [Learn about how versions of dbt align with the release tracks available on the dbt platform.](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) ###### [Upgrade versions in dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) [Instructions for upgrading your dbt platform projects to the latest version of dbt, including dbt v2.](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) ###### [Product lifecycles](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) [Learn about the dbt product lifecycles from beta through end of life.](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) ###### [Preview new dbt platform features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) [Learn how to enable self-service beta and preview features for your dbt platform account.](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) ###### [dbt version upgrade guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) [All the information you need to prepare your projects for the next version of dbt, including dbt v2. Includes guidance on new features, behavior changes, deprecations, and much more.](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) <br /> --- ### Available integrations dbt platform | Starter, Enterprise, Enterprise+ There are a number of data applications that seamlessly integrate with the Semantic Layer, powered by MetricFlow, from business intelligence tools (like [dbt Charts](https://dbtcharts.com/)) to notebooks, spreadsheets, data catalogs, and more. These integrations allow you to query and unlock valuable insights from your data ecosystem. Use the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) to simplify metric queries, optimize your development workflow, and reduce coding. This approach also ensures data governance and consistency for data consumers. The following tools integrate with the dbt Semantic Layer: [![](/img/icons/pbi.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) ###### [Power BI](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) [Use reports to query the dbt Semantic Layer with Power BI and produce dashboards with trusted data.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) [![](/img/icons/tableau-software.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) ###### [Tableau](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) [Learn how to connect to Tableau for querying metrics and collaborating with your team.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) [![](/img/icons/google-sheets-logo-icon.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) ###### [Google Sheets](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) [Discover how to connect to Google Sheets for querying metrics and collaborating with your team.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) [![](/img/icons/excel.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) ###### [Microsoft Excel](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) [Connect to Microsoft Excel to query metrics and collaborate with your team. Available for Excel Desktop or Excel Online.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) [![](/img/icons/omni.svg)](https://docs.omni.co/integrations/dbt/semantic-layer) ###### [Omni](https://docs.omni.co/integrations/dbt/semantic-layer) [Connect Omni to the dbt Semantic Layer to query trusted metrics directly within your Omni data model.](https://docs.omni.co/integrations/dbt/semantic-layer) [](https://docs.omni.co/integrations/dbt/semantic-layer) [![](/img/icons/dot-ai.svg)](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) ###### [Dot](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) [Connect Dot to the dbt Semantic Layer to ask questions in Slack, Microsoft Teams, or the web and get answers built on your metrics.](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) [](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) [![](/img/icons/hex.svg)](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) ###### [Hex](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) [Check out how to connect, analyze metrics, collaborate, and discover more data possibilities.](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) [](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) [![](/img/icons/klipfolio.svg)](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) ###### [Klipfolio PowerMetrics](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) [Learn how to connect to a streamlined metrics catalog and deliver metric-centric analytics to business users.](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) [](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) [![](/img/icons/mode.svg)](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) ###### [Mode](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) [Discover how to connect, access, and get trustworthy metrics and insights.](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) [](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) [![](/img/icons/push.svg)](https://docs.push.ai/data-sources/semantic-layers/dbt) ###### [Push.ai](https://docs.push.ai/data-sources/semantic-layers/dbt) [Explore how to connect and use metrics to power reports and insights that drive change.](https://docs.push.ai/data-sources/semantic-layers/dbt) [](https://docs.push.ai/data-sources/semantic-layers/dbt) [![](/img/icons/sigma.svg)](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) ###### [Sigma (Preview)](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) [Connect Sigma to the dbt Semantic Layer to allow you to leverage your predefined dbt metrics in Sigma workbooks.](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) [](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) [![](/img/icons/steep.svg)](https://help.steep.app/integrations/dbt-cloud) ###### [Steep](https://help.steep.app/integrations/dbt-cloud) [Connect Steep to the dbt Semantic Layer for centralized, scalable analytics.](https://help.steep.app/integrations/dbt-cloud) [](https://help.steep.app/integrations/dbt-cloud) <br /> Before you connect to these tools, you'll need to first [set up the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and [generate a service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) to create **Semantic Layer Only** and **Metadata Only** permissions. ##### Custom integration * All BI tools can use [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) with the Semantic Layer, even if they don’t have a native integration. * [Consume metrics](https://docs.getdbt.com/docs/use-dbt-semantic-layer/consume-metrics.md) and develop custom integrations using different languages and tools, supported through [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md), ADBC, and [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) APIs, and [Python SDK library](https://docs.getdbt.com/docs/dbt-apis/sl-python.md). For more info, check out [our examples on GitHub](https://github.com/dbt-labs/example-semantic-layer-clients/). * Connect to any tool that supports SQL queries. These tools must meet one of the two criteria: * Offers a generic JDBC driver option (such as DataGrip) or * Is compatible Arrow Flight SQL JDBC driver version 12.0.0 or higher. #### Related docs * [Quickstart for the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) * [dbt Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview) to learn how to integrate and query your metrics in downstream tools. * [Semantic Layer API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) * [Hex Semantic Layer cells](https://learn.hex.tech/docs/explore-data/cells/data-cells/dbt-metrics-cells) to set up SQL cells in Hex. * [Resolve 'Failed APN'](https://docs.getdbt.com/faqs/Troubleshooting/sl-alpn-error.md) error when connecting to the Semantic Layer. * [Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### Available tools The dbt MCP server has access to many parts of the dbt experience related to development, deployment, and discovery. This page lists the tools available in the dbt MCP server and how to configure them. #### Supported tools by MCP server type The following sections list every tool by category, along with which server type (local or remote) supports it. | Tools | Local | Remote | | ------------------------------------------------------ | ----- | ------ | | [dbt commands](#dbt-commands) | ✅ | ❌ | | [Semantic Layer (Semantic Layer API)](#semantic-layer) | ✅ | ✅ | | [SQL](#sql) | ✅ | ✅ | | [Metadata Discovery (Discovery API)](#discovery) | ✅ | ✅ | | [Administrative API (Admin API)](#admin-api) | ✅ | ✅ | | [Codegen Tools](#dbt-codegen) | ✅ | ❌ | | [dbt v2 Tools](#dbt-lsp) | ✅ | ✅ | | [Product Docs Tools](#product-docs) | ✅ | ✅ | | [MCP Server Metadata Tools](#mcp-server-metadata) | ✅ | ✅ | Note that access to the Discovery API and the Semantic Layer API is limited depending on your [plan type](https://www.getdbt.com/pricing). ##### Tools that require AI features `text_to_sql` is the only tool that depends on [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) being enabled for your account. If an admin turns AI features off, `text_to_sql` doesn't appear in the tool list for the remote MCP server. Every other tool keeps working. Turning off AI features doesn't disable remote MCP. #### dbt MCP server tool list The following tool list is available for your MCP server and is auto-fetched from the [dbt MCP server README on GitHub](https://github.com/dbt-labs/dbt-mcp#tools) when the docs are built, so it stays in sync with each release. ##### SQL Tools for executing and generating SQL on dbt Platform infrastructure. * `execute_sql`: Executes SQL on dbt Platform infrastructure with Semantic Layer support. * `text_to_sql`: Generates SQL from natural language using project context. ##### Semantic Layer To learn more about the dbt Semantic Layer, click [here](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl). * `get_dimension_values`: Gets distinct values for a dimension; option to scope to specific metrics. * `get_dimensions`: Gets dimensions for specified metrics. * `get_entities`: Gets entities for specified metrics. * `get_metrics_compiled_sql`: Returns compiled SQL for metrics without executing the query. * `list_metrics`: Retrieves all defined metrics. * `list_saved_queries`: Retrieves all saved queries. * `query_metrics`: Executes metric queries with filtering and grouping options. ##### Discovery To learn more about the dbt Discovery API, click [here](https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-api). * `get_all_macros`: Retrieves macros; option to filter by package or return package names only. * `get_all_models`: Retrieves name and description of all models. * `get_all_sources`: Gets all sources with freshness status; option to filter by source name. * `get_exposure_details`: *(deprecated — use `get_node_details` instead)* * `get_exposures`: Gets all exposures (downstream dashboards, apps, or analyses). * `get_lineage`: Gets a bounded lineage graph with type, depth, and direction filtering. * `get_macro_details`: *(deprecated — use `get_node_details` instead)* * `get_mart_models`: Retrieves all mart models. * `get_model_children`: *(deprecated — use `get_lineage` instead)* * `get_model_details`: *(deprecated — use `get_node_details` instead)* * `get_model_health`: Gets health signals: run status, test results, and upstream source freshness. * `get_model_parents`: *(deprecated — use `get_lineage` instead)* * `get_model_performance`: Gets execution history for a model; option to include test results. * `get_node_details`: Gets full details for any dbt resource type (model, source, exposure, test, seed, snapshot, macro, semantic\_model). * `get_related_models`: Finds similar models using semantic search. * `get_seed_details`: *(deprecated — use `get_node_details` instead)* * `get_semantic_model_details`: *(deprecated — use `get_node_details` instead)* * `get_snapshot_details`: *(deprecated — use `get_node_details` instead)* * `get_source_details`: *(deprecated — use `get_node_details` instead)* * `get_test_details`: *(deprecated — use `get_node_details` instead)* * `search`: \[Alpha] Searches for resources across the dbt project (not generally available). * `search_public_mesh_models`: Finds public dbt Mesh models across projects and confirms producer identity so you can build a verified cross-project ref (multi-project configurations only). ##### dbt CLI Allowing your client to utilize dbt commands through the MCP tooling could modify your data models, sources, and warehouse objects. Proceed only if you trust the client and understand the potential impact. * `build`: Executes models, tests, snapshots, and seeds in DAG order. * `clone`: Clones selected nodes from the specified state to the target schema(s). * `compile`: Generates executable SQL from models/tests/analyses; useful for validating Jinja logic. * `docs`: Generates documentation for the dbt project. * `get_lineage_dev`: Retrieves lineage from local manifest.json with type and depth filtering. * `get_node_details_dev`: Retrieves node details from local manifest.json (models, seeds, snapshots, sources). * `list`: Lists resources in the dbt project by type with selector support. * `parse`: Parses and validates project files for syntax correctness. * `run`: Executes models to materialize them in the database. * `show`: Executes SQL against the database and returns results. * `test`: Runs tests to validate data and model integrity. ##### Admin API To learn more about the dbt Administrative API, click [here](https://docs.getdbt.com/docs/dbt-cloud-apis/admin-cloud-api). * `cancel_job_run`: Cancels a running job. * `get_job_details`: Gets job configuration including triggers, schedule, and dbt commands. * `get_job_run_artifacts`: Downloads a specific artifact from a job run; large artifacts return a guidance message to retry with a jq\_filter; supports jq filtering for targeted extraction. * `get_job_run_details`: Gets run details including status, timing, steps, and artifacts. * `get_job_run_error`: Gets error and/or warning details for a job run; option to include or show warnings only. * `list_job_run_artifacts`: Lists available artifacts from a job run. * `list_jobs`: Lists jobs in a dbt Platform account; option to filter by project or environment. * `list_jobs_runs`: Lists job runs; option to filter by job, status, or order by field. * `list_projects`: Lists all projects in the dbt Platform account. * `retry_job_run`: Retries a failed job run. * `trigger_job_run`: Triggers a job run; option to override git branch, schema, or other settings. ##### dbt Codegen These tools help automate boilerplate code generation for dbt project files. * `generate_model_yaml`: Generates model YAML with columns; option to inherit upstream descriptions. * `generate_source`: Generates source YAML by introspecting database schemas; option to include columns. * `generate_staging_model`: Generates staging model SQL from a source table. ##### dbt LSP A set of tools that leverage the Fusion engine for advanced SQL compilation and column-level lineage analysis. * `fusion.compile_sql`: Compiles SQL in project context via dbt Platform. * `fusion.get_column_lineage`: Traces column-level lineage via dbt Platform. * `get_column_lineage`: Traces column-level lineage locally (requires dbt-lsp via dbt Labs VSCE). ##### Product Docs Tools for searching and fetching content from the official dbt documentation at docs.getdbt.com. * `get_product_doc_pages`: Fetches the full Markdown content of one or more docs.getdbt.com pages by path or URL. * `search_product_docs`: Searches docs.getdbt.com for pages matching a query; returns titles, URLs, and descriptions ranked by relevance. Use get\_product\_doc\_pages to fetch full content. ##### MCP Server Metadata These tools provide information about the MCP server itself. * `get_mcp_server_branch`: Returns the current git branch of the running dbt MCP server. * `get_mcp_server_version`: Returns the current version of the dbt MCP server. ##### Find jobs across project environments Use the `list_jobs` [tool](#admin-api) with a `project_id` to list jobs across every environment in a dbt project. This is useful when you need to inspect jobs outside the configured production environment. Without `project_id`, `list_jobs` continues to use the configured production environment, or lists jobs across the account when no production environment is configured. #### FAQs  What does execute\_sql do? `execute_sql` lets your AI assistant run a SQL query against your data warehouse and read the results. It's part of the SQL toolset and runs on dbt platform, so you don't need dbt installed locally. Here's what happens when your assistant calls it: 1. The assistant writes a SQL query and calls `execute_sql`. 2. The dbt MCP server forwards the request to dbt platform, along with your token, development environment ID, and user ID. 3. dbt platform compiles the SQL (including any Jinja, such as `ref()`) and runs it in your warehouse using your development credentials. 4. The results come back to your assistant as a table. [![The AI assistant calls execute\_sql. The dbt MCP server forwards the request to dbt platform, which compiles the SQL, runs it in the warehouse with your development credentials, and returns the results to the assistant.](/img/docs/dbt-ai/mcp-execute-sql-flow.svg?v=2 "The AI assistant calls execute_sql. The dbt MCP server forwards the request to dbt platform, which compiles the SQL, runs it in the warehouse with your development credentials, and returns the results to the assistant.")](#)The AI assistant calls execute\_sql. The dbt MCP server forwards the request to dbt platform, which compiles the SQL, runs it in the warehouse with your development credentials, and returns the results to the assistant.  Is execute\_sql read-only? No. `execute_sql` doesn't block write statements. What a query can do depends on the permissions of the warehouse credentials it runs with. Use least-privilege warehouse roles for each user's development credentials.  Whose warehouse credentials does execute\_sql use? Your personal development credentials for the project's development environment, not your production deployment credentials. For more answers about `execute_sql` or troubleshooting help, refer to [dbt MCP FAQs and troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). --- ### BigQuery and Apache Iceberg dbt supports materializing models in the Iceberg table format in two ways: * **Simplest:** The model config `table_format = 'iceberg'` instructs dbt to materialize this model as an Iceberg table in BigLake Metastore (managed catalog) * **Extensible:** Define an Iceberg catalog in `catalogs.yml` and configure this model with `catalog_name` #### Creating Iceberg tables dbt supports creating Iceberg tables for two of the BigQuery materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) #### Iceberg catalogs BigQuery supports Iceberg tables through its built-in catalog [BigLake Metastore](https://cloud.google.com/bigquery/docs/iceberg-tables#architecture) today. No setup is needed to access the BigLake Metastore. However, you need to have a [storage bucket](https://docs.cloud.google.com/storage/docs/buckets#buckets) and [the required BigQuery roles](https://cloud.google.com/bigquery/docs/iceberg-tables#required-roles) configured prior to creating an Iceberg table. #### BigQuery-specific configs for `biglake_metastore` catalogs The following table outlines the configuration fields required to set up a catalog integration for [BigLake Iceberg tables in BigQuery](https://docs.cloud.google.com/bigquery/docs/iceberg-tables). Supply and nest these additional configurations, unique to BigQuery, under `config.bigquery` (in the new catalog spec) or `adapter_properties` (in the old catalog spec). | Field | Type | Required | Description | Note | | ----------------------- | ------ | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | `file_format` | String | Yes, except for [Lakehouse Runtime Catalog (LRC)](#lakehouse-runtime-catalog-lrc) catalogs | The file format for the Iceberg table. | `parquet` is the only accepted value. | | `external_volume` | String | Yes, except for LRC catalogs | The Cloud Storage bucket where Iceberg table data is written. | For example, `gs://BUCKET_NAME`. | | `lakehouse_catalog` | String | No | The name of the LRC that holds this catalog's tables. | New spec only, and requires dbt v2 (v2). Set it in `catalogs.yml`, not per model. | | `base_location_root` | String | No | If provided, the input overrides the default dbt `base_location` value of `_dbt`. | Can be set in `catalogs.yml`. | | `base_location_subpath` | String | No | An optional suffix to add to the `base_location` path that dbt automatically specifies. | Only configurable per-model. | | `storage_uri` | String | No | If provided, the input overrides the dbt `storage_uri` value. | Only configurable per-model. | * `lakehouse_catalog`: Tells dbt that this catalog's tables live in a [Lakehouse Runtime Catalog](#lakehouse-runtime-catalog-lrc), so dbt addresses them with BigQuery's four-part name. LRC catalogs don't need `external_volume` or `file_format` because the LRC derives the storage location from the namespace. * `base_location_root`: Specifies the prefix of the base location path within the storage bucket where Iceberg table data is written. * `base_location_subpath`: Specifies the suffix of the base location path within the storage bucket where Iceberg table data is written. This property can only be set in model configurations, not in `catalogs.yml`. * `storage_uri`: Completely overrides the storage\_uri, allowing you to specify the full path directly instead of using the catalog integration's external volume and base\_location components. ##### Example 1. Create a `catalogs.yml` at the top level of your dbt project. ##### (Applies to dbt v2.0 and later) New spec catalogs.yml ```yaml catalogs: - name: my_biglake_catalog type: biglake_metastore table_format: iceberg config: bigquery: external_volume: 'gs://mydbtbucket' file_format: parquet ``` ##### Old spec catalogs.yml ```yaml catalogs: - name: my_biglake_catalog active_write_integration: biglake_metastore write_integrations: - name: biglake_metastore external_volume: 'gs://mydbtbucket' table_format: iceberg file_format: parquet catalog_type: biglake_metastore ``` 2. Apply the catalog configuration at either the model, folder, or project level: iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_biglake_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` 3. Finally, run the model: `dbt run -s my_iceberg_model`. ##### Limitations BigQuery today doesn't support connecting to external Iceberg catalogs. In terms of SQL operations and table management features, refer to the [BigQuery docs](https://cloud.google.com/bigquery/docs/iceberg-tables#limitations) for more information. ##### Base location BigQuery's DDL for creating Iceberg tables requires that a fully qualified `storage_uri` be provided, including the object path. Once the user has provided the bucket name as the `external_volume` in the catalog integration, dbt manages the `storage_uri` input. The default behavior in dbt is to provide an object path, referred to in dbt as the `base_location`, in the form: `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`. We recommend using the default behavior, but if you need to customize the resulting `base_location`, you can configure `base_location` with the model configuration fields `base_location_root` and `base_location_subpath`. * If no inputs are provided, dbt outputs for base\_location `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}` * If base\_location\_root = `foo`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}` * If base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}/bar` * If base\_location\_root = `foo` and base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}/bar` note While you can customize paths with `base_location_root` and `base_location_subpath`, we don't recommend relying on them for environment isolation (such as separating development and production environments). Anyone with repository access can easily modify these configuration values. For true environment isolation, use separate `external_volume` values with infrastructure-level access controls. You can also completely override the `storage_uri` with the model configuration field `storage_uri`. This overrides both the catalog integration path and the other model configuration fields to supply the entire `storage_uri` path directly. An example model with a customized `base_location`: iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_bigquery_iceberg_catalog', base_location_root='foo', base_location_subpath='bar', ) }} select * from {{ ref('jaffle_shop_customers') }} ``` ##### (Applies to dbt v2.0 and later) New spec catalogs.yml ```yaml catalogs: - name: my_bigquery_iceberg_catalog type: biglake_metastore table_format: iceberg config: bigquery: external_volume: 'gs://mydbtbucket' file_format: parquet base_location_root: foo ``` ##### Old spec catalogs.yml ```yaml catalogs: - name: my_bigquery_iceberg_catalog active_write_integration: biglake_metastore write_integrations: - name: biglake_metastore external_volume: 'gs://mydbtbucket' table_format: iceberg file_format: parquet catalog_type: biglake_metastore adapter_properties: base_location_root: foo ``` ###### Rationale By default, dbt manages the full `storage_uri` on behalf of users for ease of use. The `base_location` parameter specifies the location within the storage bucket where the data is written. Without guardrails (for example, if the user forgets to provide a base location root), it's possible for BigQuery to reuse the same path across multiple tables. This behavior could result in future technical debt because it limits the ability to: * Navigate the underlying object store * Read Iceberg tables through an object-store integration * Grant schema-specific access to tables through object store * Use a crawler pointed at the tables within the external storage to build a new catalog with another tool To maintain best practices, dbt enforces an input and, by default, writes your tables within a `_dbt/{SCHEMA_NAME}/{TABLE_NAME}` prefix to ensure easier object-store observability and auditability. (Applies to dbt v2.0 and later) #### Lakehouse Runtime Catalog (LRC) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2 only `lakehouse_catalog` requires the [dbt v2](https://docs.getdbt.com/docs/introduction.md) (v2) with the `use_catalogs_v2` behavior flag enabled. dbt\_project.yml ```yaml flags: use_catalogs_v2: true ``` BigQuery's [Lakehouse Runtime Catalog](https://cloud.google.com/bigquery/docs/blms-rest-catalog) (LRC) addresses a table with four parts — project, catalog, namespace, and table — but BigQuery SQL accepts only three quoted segments. dbt handles this by quoting the catalog and namespace together as the middle segment: ```sql `{project}`.`{catalog}.{namespace}`.`{table}` ``` Set `lakehouse_catalog` on a `biglake_metastore` catalog to tell dbt that its tables live in an LRC. dbt then uses the four-part name and omits the connection clause and the `table_format` option, neither of which BigQuery accepts for LRC tables. ##### Prerequisites Create the LRC catalog and namespace before you run dbt. BigQuery has no SQL statement that creates them, so dbt can create only the table. Because of this, dbt attempts to create the namespace on every run against an LRC catalog, and BigQuery rejects it with the following warning: ```shell [FailedToCreateDatabase (dbt1051)]: Failed to create schema 'sales_catalog.analytics' in database 'my_project' in remote for model.my_project.my_lrc_model: [BigQuery] googleapi: Error 400: Invalid project ID 'my_project.sales_catalog'. ``` Expect this warning — it doesn't fail the run. As long as the catalog and namespace exist, dbt creates the table. ##### LRC example 1. Add a catalog with `lakehouse_catalog` set. An LRC catalog doesn't need `external_volume` or `file_format`. catalogs.yml ```yaml catalogs: - name: my_lrc_catalog type: biglake_metastore table_format: iceberg config: bigquery: lakehouse_catalog: sales_catalog ``` 2. Configure a model with `catalog_name`: my\_lrc\_model.sql ```sql {{ config( materialized='table', catalog_name='my_lrc_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` 3. Run the model: `dbt run -s my_lrc_model`. dbt generates the following DDL: ```sql create or replace table `my_project`.`sales_catalog.analytics`.`my_lrc_model` OPTIONS() as ( select * from `my_project`.`analytics`.`jaffle_shop_customers` ) ``` ##### LRC limitations BigQuery doesn't expose LRC tables through `INFORMATION_SCHEMA`, so your queries against those views don't return them. This doesn't affect `dbt docs generate`. --- ### Build and view your docs with dbt dbt platform dbt enables you to generate documentation for your project and data platform. The documentation is automatically updated with new information after a fully successful job run, ensuring accuracy and relevance. The default documentation experience in dbt is [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), available on [Starter, Enterprise, or Enterprise+ plans](https://www.getdbt.com/pricing/). Use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's resources (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Refer to [documentation](https://docs.getdbt.com/docs/build/documentation.md) for more configuration details. (Applies to dbt v2.0 and later) To build a self-hosted static documentation site, use [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2) locally. dbt Docs v2 isn't available in the dbt platform. #### Set up a documentation job (Applies to dbt v2.0 and later) Catalog uses the [metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) from each job run in your production or staging environment. Jobs running v2 generate this metadata automatically on every job run, so you don't need a separate documentation job, a `dbt docs generate` step, or a docs checkbox. To keep Catalog up to date: 1. In the top left, click **Deploy** and select **Jobs**. 2. Create a new job or select an existing job in a production or staging environment and click **Settings**. 3. Under **Execution settings**, add the commands you want to run, such as `dbt build`, and click **Save**. 4. After the job runs, click **Catalog** in the navigation to explore your project. If you add `dbt docs generate` as a run step, dbt runs `dbt compile --write-catalog` instead and displays a banner directing you to Catalog. For more info, refer to [platform behavior](https://docs.getdbt.com/reference/commands/cmd-docs.md?version=2#platform-behavior). Metadata-only jobs To refresh Catalog metadata without building models, add `dbt compile --write-catalog` in the **Commands** section. #### View your project documentation [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) is where you view your project's documentation in the dbt platform. It always shows your project's latest production state, so there's nothing to deploy or host. In Catalog, you can: * Search and filter your project's resources, such as models, sources, and metrics. * Explore the [lineage graph](https://docs.getdbt.com/docs/explore/explore-projects.md#project-lineage) to see how your resources connect. * Open a [resource's details](https://docs.getdbt.com/docs/explore/explore-projects.md#view-resource-details) to see its description, columns, tests, and recent run results. To open it, click **Catalog** in the navigation. To explore a model in the lineage graph, select your project in the left sidebar, click **View lineage**, and then click a model to view its description. [![Example of the full lineage graph in Catalog](/img/docs/collaborate/dbt-explorer/example-project-lineage-graph.png?v=2 "Example of the full lineage graph in Catalog")](#)Example of the full lineage graph in Catalog Catalog is available to Developer and read-only users. To share your project with stakeholders who don't develop in dbt, give all your applicable users [read-only access](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) to Catalog without restrictions. #### Related docs * [Documentation](https://docs.getdbt.com/docs/build/documentation.md) * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) --- ### Build your metrics Use MetricFlow in dbt to centrally define your metrics. As a key component of the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), MetricFlow is responsible for SQL query construction and defining specifications for dbt semantic models and metrics. It uses familiar constructs like semantic models and metrics to avoid duplicative coding, optimize your development workflow, ensure data governance for company metrics, and guarantee consistency for data consumers. For a complete list of configuration options and property specs, see the [Semantic Layer reference](https://docs.getdbt.com/reference/semantic-layer-reference.md). [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack. MetricFlow allows you to: * Intuitively define metrics in your dbt project * Develop from your preferred environment, whether that's the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), or [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) * Use [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) to query and test those metrics in your development environment * Harness the true magic of the universal Semantic Layer and dynamically query these metrics in downstream tools (Available for dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing/) accounts only). [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/reference/semantic-layer-reference.md) ###### [Semantic Layer reference](https://docs.getdbt.com/reference/semantic-layer-reference.md) [Complete, exhaustive configuration reference for semantic models, metrics, and dimensions.](https://docs.getdbt.com/reference/semantic-layer-reference.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) ###### [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) [Learn how to migrate from the legacy metrics YAML spec to the latest metrics YAML spec.](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/sl-qs.md) ###### [Quickstart for the dbt Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) [Use this guide to build and define metrics, set up the dbt Semantic Layer, and query them using downstream tools.](https://docs.getdbt.com/guides/sl-qs.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/about-metricflow.md) ###### [About MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) [Understand MetricFlow's core concepts, how to use joins, how to save commonly used queries, and what commands are available.](https://docs.getdbt.com/docs/build/about-metricflow.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/semantic-models.md) ###### [Semantic model](https://docs.getdbt.com/docs/build/semantic-models.md) [Use semantic models as the basis for defining data. They act as nodes in the semantic graph, with entities connecting them.](https://docs.getdbt.com/docs/build/semantic-models.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/metrics-overview.md) ###### [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) [Define metrics in your dbt project using different metric types in YAML files.](https://docs.getdbt.com/docs/build/metrics-overview.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/advanced-topics.md) ###### [Advanced topics](https://docs.getdbt.com/docs/build/advanced-topics.md) [Learn about advanced topics for dbt Semantic Layer and MetricFlow, such as data modeling workflows, and more.](https://docs.getdbt.com/docs/build/advanced-topics.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) ###### [About the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) [Introducing the dbt Semantic Layer, the universal process that allows data teams to centrally define and query metrics](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) ###### [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) [Discover the diverse range of partners that seamlessly integrate with the powerful dbt Semantic Layer, allowing you to query and unlock valuable insights from your data ecosystem.](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) <br /> #### Related docs * [Quickstart guide with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) * [The Semantic Layer: what's next](https://www.getdbt.com/blog/dbt-semantic-layer-whats-next/) blog * [Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### Cache common queries dbt platform | Enterprise, Enterprise+ The Semantic Layer allows you to cache common queries in order to speed up performance and reduce compute on expensive queries. There are two different types of caching: * [Result caching](#result-caching) leverages your data platform's built-in caching layer. * [Declarative caching](#declarative-caching) allows you to pre-warm the cache using saved queries configuration. While you can use caching to speed up your queries and reduce compute time, knowing the difference between the two depends on your use case: * Result caching happens automatically by leveraging your data platform's cache. * Declarative caching allows you to 'declare' the queries you specifically want to cache. With declarative caching, you need to anticipate which queries you want to cache. * Declarative caching also allows you to dynamically filter your dashboards without losing the performance benefits of caching. This works because filters on dimensions (that are already in a saved query config) will use the cache. #### Prerequisites * dbt [Enterprise or Enterprise+](https://www.getdbt.com/) plans. * dbt environments must be on [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and not legacy dbt v1 versions. * A successful job run and [production environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment). * For declarative caching, you need to have [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) defined in your [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) YAML configuration file. #### Result caching Result caching leverages your data platform’s built-in caching layer and features. [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) generates the same SQL for multiple query requests, this means it can take advantage of your data platform’s cache. Double-check your data platform's specifications. Here's how caching works, using Snowflake as an example, and should be similar across other data platforms: 1. **Run from cold cache** — When you run a semantic layer query from your BI tool that hasn't been executed in the past 24 hours, the query scans the entire dataset and doesn't use the cache. 2. **Run from warm cache** — If you rerun the same query after 1 hour, the SQL generated and executed on Snowflake remains the same. On Snowflake, the result cache is set per user for 24 hours, which allows the repeated query to use the cache and return results faster. Different data platforms might have different caching layers and cache invalidation rules. Here's a list of resources on how caching works on some common data platforms: * [BigQuery](https://cloud.google.com/bigquery/docs/cached-results) * [DataBricks](https://docs.databricks.com/en/optimizations/disk-cache.html) * [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/data-warehouse/caching) * [Redshift](https://docs.aws.amazon.com/redshift/latest/dg/c_challenges_achieving_high_performance_queries.html#result-caching) * [Snowflake](https://community.snowflake.com/s/article/Caching-in-the-Snowflake-Cloud-Data-Platform) * [Starburst Galaxy](https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/warp-speed-enabled.html) #### Declarative caching Declarative caching enables you to pre-warm the cache using [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) by setting the cache config to `true` in your `saved_queries` settings. This is useful for optimizing performance for key dashboards or common ad-hoc query requests. tip Declarative caching also allows you to dynamically filter your dashboards without losing the performance benefits of caching. This works because filters on dimensions (that are already in a saved query config) will use the cache. For example, if you filter a metric by geographical region on a dashboard, the query will hit the cache, ensuring faster results. This also removes the need to create separate saved queries with static filters. For configuration details, refer to [Declarative caching setup](#declarative-caching-setup). How declarative caching works: * Make sure your saved queries YAML configuration file has [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) defined. * Running a saved query triggers the Semantic Layer to: * Build a cached table from a saved query, with exports defined, into your data platform. * Make sure any query requests that match the saved query's inputs use the cache, returning data more quickly. * Automatically invalidates the cache when it detects new and fresh data in any upstream models related to the metrics in your cached table. * Refreshes (or rebuilds) the cache the next time you run the saved query. 📹 Check out this video demo to see how declarative caching works! This video demonstrates the concept of declarative caching, how to run it using the dbt scheduler, and how fast your dashboards load as a result. Refer to the following diagram, which illustrates what happens when the Semantic Layer receives a query request: [![Overview of the declarative cache query flow](/img/docs/dbt-platform/semantic-layer/declarative-cache-query-flow.jpg?v=2 "Overview of the declarative cache query flow")](#)Overview of the declarative cache query flow ##### Declarative caching setup To populate the cache, you need to configure an export in your saved query YAML file configuration *and* set the `cache config` to `true`. You can't cache a saved query without an export defined. semantic\_model.yml ```yaml saved_queries: - name: my_saved_query ... # Rest of the saved queries configuration. config: cache: enabled: true # Set to true to enable, defaults to false. exports: - name: order_data_key_metrics config: export_as: table ``` To enable saved queries at the project level, you can set the `saved-queries` configuration in the [`dbt_project.yml` file](https://docs.getdbt.com/reference/dbt_project.yml.md). This saves you time in configuring saved queries in each file: dbt\_project.yml ```yaml saved-queries: my_saved_query: config: +cache: enabled: true ``` ##### Run your declarative cache After setting up declarative caching in your YAML configuration, you can now run [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) with the dbt job scheduler to build a cached table from a saved query into your data platform. * Use [exports to set up a job](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to run a saved query dbt. * The dbt Semantic Layer builds a cache table in your data platform in a dedicated `dbt_sl_cache` schema. * The cache schema and tables are created using your deployment credentials. You need to grant read access to this schema for your Semantic Layer user. * The cache refreshes (or rebuilds) on the same schedule as the saved query job. [![Overview of the cache creation flow.](/img/docs/dbt-platform/semantic-layer/cache-creation-flow.jpg?v=2 "Overview of the cache creation flow.")](#)Overview of the cache creation flow. After a successful job run, you can go back to your dashboard to experience the speed and benefits of declarative caching. #### Cache management dbt uses the metadata from your dbt model runs to intelligently manage cache invalidation. When you start a dbt job, it keeps track of the last model runtime and checks the freshness of the metrics upstream of your cache. If an upstream model has data in it that was created after the cache was created, dbt invalidates the cache. This means queries won't use outdated cases and will instead query directly from the source data. Stale, outdated cache tables are periodically dropped and dbt will write a new cache the next time your saved query runs. You can manually invalidate the cache through the [dbt Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) using the `InvalidateCacheResult` field. #### FAQs  How does caching interact with access controls? Cached data is stored separately from the underlying models. If metrics are pulled from the cache, we don’t have the security context applied to those tables at query time. In the future, we plan to clone credentials, identify the minimum access level needed, and apply those permissions to cached tables. #### Related docs * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) * [Saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### Changelog 2019 and 2020 note This changelog references dbt versions that are no longer supported and have been removed from the docs. For more information about upgrading to a supported version of dbt in your dbt Cloud environment, read [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). Welcome to the 2019 and 2020 changelog for the dbt application! You can use this changelog to see the highlights of what was new, fixed, and enhanced during this time period. #### dbt Cloud v1.1.16 (December 23, 2020) This release adds preview support for Databricks Spark in dbt and adds two new permission sets for Enterprise accounts. ###### Enhancements * Added preview support for Databricks Spark support * Added two new Enterprise permission sets: Account Viewer and Project Creator ###### Fixed * Improve logging infrastructure for dbt run logs * Fix for SSH tunnel logging errors #### dbt Cloud v1.1.15 (December 10, 2020) Lots of great stuff to confer about this go-round: things really coalesced this week! Lots of excitement around adding Spark to the connection family, as well as knocking out some longstanding bugs. ###### Enhancements * Add Spark as an option for database setup ###### Fixed * Fix this one hairy bug where one email could have multiple user accounts * Fix setup-connection react-page routing * Break out group selection logic from license types and group names * Handle JSON errors in v1/v2 body parsing * Handle AuthForbidden and AuthCancelled graciously - ie, not throw 500s * Fix regression with Studio IDE loading spinner #### dbt Cloud v1.1.14 (November 25, 2020) This release adds a few new pieces of connective tissue, notably OAuth for BigQuery and SparkAdapter work. There are also some quality of life improvements and investments for the future, focused on our beloved Studio IDE users, and some improved piping for observability into log management and API usage. ###### Enhancements * Update IP allowlist * User can OAuth for BigQuery in profile credentials * Adding SparkAdapter backend models, mappers, and services * Added BigQuery OAuth integration * Adding db index for owner\_thread\_id ###### Fixed * Fix post /run error rate * Fix bug where bad argument was passed to dbt runs * Log out unhandled error in environment variable context manager * Remove account settings permissions for user integrations #### dbt Cloud v1.1.13 (November 12, 2020) This release adds support for triggering runs with overridden attributes via the [triggerRun](https://docs.getdbt.com/dbt-cloud/api-v2) API endpoint. Additionally, a number of bugs have been squashed and performance improvements have been made. ###### Enhancements * Improve error handling for long-running queries in the Studio IDE * Use S3 client caching to improve log download speed for scheduled runs * Support triggering jobs [with overridden attributes from the API](https://docs.getdbt.com/dbt-cloud/api-v2) * Clarify "upgrade" copy on the billing page ###### Fixed * GitLab groups endpoint now returns all groups and subgroups * Support BigQuery retry configs with value 0 * Prevent web IDE from crashing after running an invalid dbt command * Apply additional log scrubbing to filter short-lived git credentials * Fix older migration to make auth\_url field nullable * Support paths in GitLab instance URL * Fix for auth token request url in GitLab oauth flow #### dbt Cloud v1.1.12 (October 30, 2020) This release adds dbt v.18.1 and 0.19.0b1 to dbt Cloud. Additionally, a number of bugs have been fixed. ###### Enhancements * Update copy on billing page for picking a plan at the end of a trial * Improved authorization for metadata API * Add dbt 0.19.0b1 * Add dbt 0.18.1 ###### Fixed * Fixed an issue where groups from other logged-in accounts appeared in the RBAC UI * Fixed requested GitLab scopes and an issue when encrypting deploy tokens for GitLab auth * Fixed an issue where null characters in logs threw errors in scheduled runs #### dbt Cloud v1.1.11 (October 15, 2020) Release v1.1.11 includes some quality-of-life enhancements, copy tweaks, and error resolutions. It also marks the last time we'll have the same digit four times in a row in a release until v2.2.22. ###### Enhancements * Add InterfaceError exception handling for commands * Rename My Account --> Profile * Add project and connection to admin backend ###### Fixed * Resolve errors from presence of null-characters in logs * Email verifications backend * Undo run.serialize * Fix error while serialized run * Fix logic error in connection setup * Fix a bug with GitLab auth flow for unauthenticated users * Fix bug where Native Okta SSO uses the wrong port #### dbt Cloud v1.1.10 (October 8, 2020) This release adds support for repositories imported via GitLab (Enterprise) and contains a number of bugfixes and improvements in the Studio IDE. ###### Enhancements * Add Gitlab integration (Enterprise) * Add GitLab repository setup to project setup flow (Enterprise) * Add GitLab automated Deploy Token installation (Enterprise) * Add dbt 0.18.1rc1 ###### Fixed * Fix bug where Studio IDE gets stuck after changing project repository * Fix race condition where connections can be added to the wrong project * Fix revoking email invites * Fix a bug in slim CI deferring run search where missing previous run caused the scheduler to raise an error * Fix a source of Studio IDE instability * Gracefully clean up Studio IDE backend on shutdown * Always show SSO mappings on Group Details page #### dbt Cloud v1.1.9 (October 1, 2020) This release adds the ability for admins on the Enterprise plan to configure the Role Based Access Control permissions applied to Projects in their account. Additionally, job execution deferral is now available behind a feature flag, and a number of fixes and improvements were released as well. ###### Enhancements * Add dbt version in the navigation sidebar * Add RBAC Group Permission view, create, and modify UIs * Add personal git auth for Studio IDE error handling modals * Add Develop Requests to backend views * Implemented job execution deferral * Add support for dbt v0.18.1b2 ###### Fixed * Fixed the scenario where interacting with the Refresh Studio IDE button causes an index.lock file to remain in the Studio IDE file system * Validate PR URL for XSS attempts * Address RBAC inconsistencies * Fixed users not being able to update their dbt password in-app * Fix for applying user permissions across multiple accounts after SSO auth * Google API: default to common api endpoint but allow override * Fix for missing email variable in GSuite debug logging * Destroy Studio IDE session when switching projects #### dbt Cloud v1.1.8 (September 17, 2020) This release adds native support for Okta SSO and dbt v0.18.0. It also adds initial support for a GitLab integration and self-service RBAC configuration. ###### Enhancements * Add dbt 0.18.0 * Add native Okta SSO support * Add additional logging for Gsuite and Azure SSO * Add git cloning support via GitLab deploy tokens for scheduled runs (coming soon) * add RBAC Groups Detail Page and Groups List UIs ###### Fixed * Allow `*_proxy` env vars in scheduled runs #### dbt Cloud v1.1.7 \[September 3, 2020] This release adds a Release Candidate for [dbt v0.18.0](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) and includes bugfixes and improvements to the Cloud IDE and job scheduler. ###### Enhancements * Improve scheduler backoff behavior * Add dbt 0.18.0rc1 * Add support for non-standard ssh ports in connection tunnels * Add support for closing the Studio IDE filesystem context menu by clicking outside the menu ###### Fixed * Fix for joining threads in run triggers * Fix thread caching for s3 uploads #### dbt Cloud v1.1.6 (August 20, 2020) This release includes security enhancements and improvements across the entire dbt application. ###### Enhancements * Support for viewing development docs inside of the Studio IDE ([docs](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) * Change CI temporary schema names to be prefixed with `dbt_cloud` instead of `sinter` * Change coloring and iconography to improve accessibility and UX across the application * \[Enterprise] Support the specification of multiple authorized domains in SSO configuration * \[On-premises] Upgrade boto3 to support KIAM authentication ###### Fixed * \[Enterprise] Fix for missing IdP group membership mappings when users belong to >100 Azure AD groups * Disallow the creation of symlinks in the Studio IDE * Improve reliability of background cleanup processes * Improve performance and reliability of artifact management and PR webhook processing #### dbt Cloud v1.1.5 (August 4, 2020) This release adds a major new feature to the Studio IDE: merge conflict resolution! It also includes changes to the job scheduler that cut the time and resource utilization significantly. ###### Enhancements * Add dbt 0.17.2 * Add dbt 0.18.0 beta 2 * Add merge conflict resolution, a merge commit workflow, and merge abort workflow to the IDE * Deprecate dbt versions prior to 0.13.0 * Refactor to cut job scheduler loop time * Reduce extra database calls to account table in job scheduler loop * \[On-premises] Allow clients to disable authentication for SMTP * \[On-premises] Allow disabling of TLS for SMTP * \[On-premises] Making k8s access mode for Studio IDE pods an environment variable * \[Security] Force session cookie to be secure * Make api and admin modules flake8 complaint ###### Fixed * Fix incorrect usage of `region_name` in KMS client * Fix a call to a deprecated Github API * Remove extraneous billing API calls during job scheduler loop * Fix error where refreshing the IDE would leave running dbt processes in a bad state #### dbt Cloud v1.1.4 (July 21, 2020) This release dramatically speeds up the job scheduler. It adds a new stable dbt version (0.17.1) and a new prerelease (0.17.2b1), and it includes a number of bugfixes. ###### Enhancements * Add dbt 0.17.2b1 * Add dbt 0.17.1 and set as default version * Speed up job scheduler by 50% * Added generate docs to rpc service and new view docs route * Queue limiting by account for scheduled jobs ###### Fixed * Fix enterprise SSO configuration when old Auth0 Azure AD is configured * Do not schedule jobs for deleted job definitions or environments * Fix permissions issues * Fix a bug with metadata set in azure storage provider * Fixed error when switching to developer plan from trial * Fix authentication bug where we setup all accounts with same domain * \[Security] Add security check to prevent potentially malicious html files in dbt docs #### dbt Cloud v1.1.3 (July 7, 2020) This release contains a number of IDE features and bugfixes, a new release candidate of dbt, and a brand new Enterprise Single-Sign On method: Azure Active Directory! ###### Enhancements * Add dbt 0.17.1rc3 * Snowflake: Add support for `client_session_keep_alive` config * Enterprise: Native Azure Oauth2 for Enterprise accounts * Studio IDE: Add custom command palette for finding files ###### Fixed * Do not run CI builds for draft PRs in GitHub * Remove race condition when syncing account with stripe billing events * Enterprise: Fixed JIT provisioning bug impacting accounts with shared IdP domains * Studio IDE: Fix a regression with Github git clone method * Studio IDE: Fix a race condition where git clone didn't complete before user entered Studio IDE * Studio IDE: Fix bug with checking out an environment custom branch on Studio IDE refresh * Bigquery: Fix PR schema dropping #### dbt Cloud v1.1.2 (June 23, 2020) This branch includes an important security fix, two new versions of dbt, and some miscellaneous fixes. ###### Enhancements * Add project names to the account settings notifications section * Add dbt 0.17.1 release candidate * Update development dbt version to Marian Anderson * Add remember me to login page and expire user sessions at browser close * Adding Auth Provider and enabling Gsuite SSO for enterprise customers ###### Fixed * \[Security] Fix intra-account API key leakage * Support queries containing unicode characters in the Studio IDE #### dbt Cloud v1.1.1 (June 9, 2020) This release includes dbt 0.17.0 and a number of IDE quality of life improvements. ###### Enhancements * Added dbt 0.17.0 * Added the ability to create a new folder in the IDE * Added gitignore status to file system and display dbt artifacts, including directories dbt\_modules, logs, and target * (Cloud only) Added rollbar and update some various error handling clean up * (On-premises only) Admin site: allow Repository's Pull Request Template field to be blank * (On-premises only) Added AWS KMS support ###### Fixed * Expires old pending password reset codes when a new password reset is requested #### dbt Cloud v1.1.0 (June 2, 2020) This release adds some new admin backend functionality, as well as automatic seat usage reporting. ##### On-Premises Only ###### Added * Added automatic reporting of seat usage. ###### Changed * Admins can now edit remote URLs for repository in the admin backend. * Admins can now edit credentials in the admin backend. *** #### dbt Cloud v1.0.12 (May 27, 2020) This release contains a few bugfixes for the Studio IDE and email notifications, as well as the latest release candidate of 0.17.0. ##### All versions ###### Added * Use the correct starter project tag, based on dbt version, when initializing a new project in the IDE * Added branch filtering to IDE git checkout UI. * Added dbt 0.17.0-rc3. ###### Fixed * Fixed source freshness report for dbt version v0.17.0 * Fixed issue with checking-out git branches * Fixed issue of logs being omitted on long running queries in the Studio IDE * Fixed slack notifications failing to send if email notifications fail ##### On-Premises Only ###### Added * Added an Admin page for deleting credentials. *** #### dbt Cloud v1.0.11 (May 19, 2020) This version adds some new permission sets, and a new release candidate of dbt. ##### All versions ###### Added * Added permission sets for Job Viewer, Job Admin and Analyst. * Added dbt 0.17.0-rc1 *** #### dbt Cloud v1.0.10 (May 11, 2020) ##### All versions ###### Added * Added dbt 0.17.0-b1. * PR Url is now self serve configurable. * Added more granular permissions around creating and deleting permissions. (Account Admin can create new projects by default while both Account Admin and Project Admin can delete the projects they have permissions for by default) * Added an error message to display to users that do not have permissions set up for any projects on an account. ###### Fixed * Removed .sql from CSV download filename * Fixed breaking JobDefinition API with new param custom\_branch\_only * Fixed Studio IDE query table column heading casing *** #### dbt Cloud v1.0.9 (May 5, 2020) This release includes bugfixes around how permissions are applied to runs and run steps, fixes a bug where the scheduler would hang up, and improves performance of the Studio IDE. ##### All versions ###### Fixed * Fixed permission checks around Runs and Run Steps, this should only affect Enterprise accounts with per-project permissions. * Fixed receiving arbitrary remote\_url when creating a git url repository. * Fixed issue when handling non-resource specific errors from RPC server in Studio IDE. * Fixed a bug where the scheduler would stop if the database went away. * Fixed IDE query results table not supporting horizontal scrolling. ###### Changed * Improve Studio IDE query results performance. * Allow configuration on jobs to only run builds when environment target branch is env's custom branch. * Allow configuration of GitHub installation IDs in the admin backend. ##### On-Premises Only ###### Fixed * Fixed logic error for installations with user/password auth enabled in an on-premises context *** #### dbt Cloud v1.0.8 (April 28, 2020) This release adds a new version of dbt (0.16.1), fixes a number of IDE bugs, and fixes some dbt Cloud on-premises bugs. ##### All versions ###### Added * Add dbt 0.16.1 ###### Fixed * Fixed Studio IDE filesystem loading to check for directories to ensure that load and write methods are only performed on files. * Fixed a bug with generating private keys for connection SSH tunnels. * Fixed issue preventing temporary PR schemas from being dropped when PR is closed. * Fix issues with Studio IDE tabs not updating query compile and run results. * Fix issues with query runtime timer in Studio IDE for compile and run query functions. * Fixed what settings are displayed on the account settings page to align with the user's permissions. * Fixed bug with checking user's permissions in frontend when user belonged to more than one project. * Fixed bug with access control around environments and file system/git interactions that occurred when using Studio IDE. * Fixed a bug with Environments too generously matching repository. ###### Changed * Make the configured base branch in the Studio IDE read-only. * Support configuring groups using an account ID in the admin backend. * Use gunicorn webserver in Studio IDE. * Allow any repository with a Github installation ID to use build-on-PR. * Member and Owner Groups are now editable from admin UI. ##### On-Premises Only ###### Fixed * Fixed an issue where account license counts were not set correctly from onprem license file. * Fixed an issue where docs would sometimes fail to load due to a server error. *** #### dbt Cloud v1.0.7 (April 13, 2020) This release rolls out a major change to how permissions are applied in dbt's API. It also adds some minor bugfixes, and some tooling for improved future QA. ##### All versions ###### Added * Added support to permission connections on a per project basis. * Added support to permission credentials on a per project basis. * Added support to permission repositories on a per project basis. * Smoke tests for account signup, user login and basic project setup * Add dbt 0.16.1rc1 * Non-enterprise users can now add new accounts from the Accounts dropdown. ###### Fixed * Fix missing migration for credentials. * Fixed issue with testing connections with a non-default target name specified in the credentials. * Fix issue where Bigquery connections could be created with invalid values for `location`. *** #### dbt Cloud v1.0.6 (March 30, 2020) This release adds UIs to select group permissions in the project settings UI. It also contains bugfixes for the Studio IDE, PR build schema dropping, and adds support for dissociating Github and Slack integrations via the Admin backend. ##### All versions ###### Added * (Enterprise only) Added ability to create group permissions for specific projects in the project settings UI. ###### Fixed * Fix empty state for selecting github repositories * Fixed an issue with the IDE failing to report an invalid project subdirectory for a dbt project * Fix blank loading screen displayed when switching accounts while on account/profile settings page * Fix issue preventing schemas from dropping during PR builds * Fix issue where whitespace in user's name breaks default schema name * Added webhook processing for when a user disassociates github access to their account. * Added slack disassociation capability on user integrations page and on backend admin panel (for notifications). ###### Changed * Declare application store using configureStore from redux-toolkit *** #### dbt Cloud v1.0.5 (March 23, 2020) ##### All versions ###### Added * Add support for authenticating Development and Deployment Snowflake credentials using keypair auth * Add support for checking out tags, render git output in "clone" run step * Add dbt 0.15.3 * Add dbt 0.16.0 ###### Fixed * Git provider urls now built with correct github account and repository directories. * Invalid DateTime Start time in Studio IDE Results Panel KPIs. * Fix a race condition causing the Invite User UI to not work properly. * Incorrect model build times in Studio IDE. ###### Changed * Git: ignore `logs/` and `target/` directories in the IDE. *** #### 1.0.4 (March 16, 2020) This release adds two new versions of dbt, adds Snowflake SSO support for Enterprise accounts, and fixes a number of bugs. ##### All versions ###### Added * Added dbt 0.15.3rc1 * Added dbt 0.16.0rc2 * Add support for cloning private deps in the IDE when using deploy key auth. * Log user that kicked off manual runs. * Enterprise support for authenticating user Snowflake connections using Snowflake single sign-on ###### Fixed * Fixed issue loading accounts for a user if they lack permissions for any subset of accounts they have a user license for. * Fixed issue with showing blank page for user who is not associated with any accounts. * Fixed issue where runs would continue to kick off on a deleted project. * Fixed issue where accounts connected to GitHub integrations with SAML protection could not import repositories * Improved error messages shown to the user if repos are unauthorized in a GitHub integration when importing a repo * Fix colors of buttons in generated emails ##### On-Premises ###### Added * Added Admin backend UIs for managing user permissions. *** #### 1.0.3 (March 1, 2020) This release contains the building blocks for RBAC, and a number of bugfixes and upgrades. ##### All versions ###### Added * Add support for a read replica for reading runs from the API. * Added groups, group permissions, and user groups. * Add email address to email verification screen. * Add Enterprise Permissions. * Allow account-level access to resources for groups with a permission statement of "all resources" for api backwards compatibility. * Add dbt 0.16.0b3 ###### Fixed * Fix issue with loading projects after switching accounts. * Fix broken links to connections from deployment environment settings. * Fix a bug with inviting readonly users. * Fix a bug where permissions were removed from Enterprise users upon login. ###### Changed * Update Django version: 2.2.10 * Update Django admin panel version * Update Social Auth version and the related Django component * Update jobs from using account-based resource permissions to project-based resource permissions * Update modal that shows when trials are expired; fix copy for past-due accounts in modal * Replace formatted string logging with structured logging * Move connection and repository settings from account settings to project settings * Update project setup flow to be used for creating projects * Update develop requests to have a foreign key on projects ##### On-Premises ###### Added * Accounts created from admin backend will come with a default set of groups ###### Changed * Rename "Fishtown Analytics User" to "Superuser" *** #### dbt Cloud v1.0.2 (February 20, 2020) This release contains a number of package upgrades, and a number of bugfixes. ##### All versions ###### Added * Add request context data to logs * Comprehensive logging for git subprocesses ###### Fixed * Fix an issue where the "Cancel Run" button does not work * Fix warnings regarding mutable resource model defaults for jobs and job notifications * Fix bug where users can create multiple connection user credentials through the project setup workflow * Update auth for requests against Github's api from using query parameters to using an Authorization header * Remove unused threads input from deployment environments * Fix issue that prevented user from viewing documentation and data sources * Fix issue rendering code editor panel in the IDE when using Safari * Fix issue with log levels that caused dbt logs to be too chatty ###### Changed * Update Django version: 2.2.10 * Update Django admin panel version * Update Social Auth version and the related Django component * Update jobs from using account-based resource permissions to project-based resource permissions * Update modal that shows when trials are expired; fix copy for past-due accounts in modal * Replace formatted string logging with structured logging * Move connection and repository settings from account settings to project settings * Update project setup flow to be used for creating projects ###### Removed None. *** #### dbt Cloud v1.0.1 (February 4, 2020) This release makes the IDE generally available, and adds two new versions of dbt (0.15.1, 0.15.2). For on-premises customers, there is a new set of configurations in the configuration console: SMTP: You can now configure dbt to send email notifications through your own SMTP server. RSA Encryption: You can now provide your own RSA keypair for dbt to use for encryption. These fields need to be specified for your instance of dbt to function properly. ##### All versions ###### Added * New Team List page * New Team User Detail page * New Invite User page * New dashboard for Read Only users * New dbt version: 0.15.1 * New dbt version: 0.15.2 * Ability to rename files in Studio IDE * New backend service for project-based resource permissions ###### Fixed * Fix an issue where the user has to repeat steps in the onboarding flow * Fix issue where user can get stuck in the onboarding flow * Fix bug where email notifications could be sent to deleted users * Fix UI bug not allowing user to check "Build on pull request?" when creating a job * Fix UI bug in header of the Edit User page * Fix issue that did not take into account pending invites and license seats when re-sending a user invite. * Fix an issue when processing Github webhooks with unconfigured environments * Fix console warning presented when updating React state from unmounted component * Fix issue where closed tabs would continue to be shown, though the content was removed correctly * Fix issue that prevented opening an adjacent tab when a tab was closed * Fix issue creating BigQuery connections causing the account connections list to not load correctly. * Fix for locked accounts that have downgraded to the developer plan at trial end * Fix for not properly showing server error messages on the user invite page ###### Changed * Deployed a number of Studio IDE visual improvements * Batch logs up every 5 seconds instead of every second to improve database performance * Make `retries` profile configuration for BigQuery connections optional * Support `retries` profile configuration for BigQuery connections (new in dbt v0.15.1) * Replace Gravatar images with generic person icons in the top navbar * Remove deprecated account subscription models * Remove external JS dependencies ###### Removed * Remove the "read only" role (this is now a "read only" license type) * Remove the "standard" license type * Remove "beta" tag from Studio IDE * Remove unused frontend code (team page/create repository page and related services) ##### Self-Service ###### Fixed * Fix for locked accounts that have downgraded to the developer plan at trial end ###### Added * New Plans page * Add a 14 day free trial * Add the ability to provision a new repository via dbt * New Invite Team step for project setup process for trial accounts ###### Changed * The "Basic" and "Pro" plans are no longer available. The new "Developer" and "Team" plans are available. * Prorations are now charged immediately, instead of applied to the next billing cycle. * It is no longer possible to downgrade to a plan that does not support the current number of allocated seats. * A "Team" plan that has been cancelled will be locked (closed) at the end of the subscription's period ##### On-Premises ###### Added * Support custom SMTP settings * Support Azure Blob Storage for run logs + artifacts * Optionally disable anonymous usage tracking *** #### dbt Cloud v0.5.0 (December 19, 2019) This release preps dbt for the general Studio IDE release in January. Beta Studio IDE functionality can be turned on by checking "Develop file system" in the Accounts page in the dbt backend. ##### All versions ###### Added * New dbt version: 0.14.2 * New dbt version: 0.14.3 * New dbt version: 0.14.4 * New dbt version: 0.15.0 * New API endpoint: v3/projects * New API endpoint: v3/credentials * New API endpoint: v3/environments * New API endpoint: v3/events * Studio IDE: Add git workflow UI * Studio IDE: Add filesystem management * Studio IDE: Hup the server when files change * Studio IDE: Display server status and task history * Added development and deployment environments and credentials * Support `--warn-error` flag in dbt runs ###### Fixed * Fixed an issue where the run scheduler would hang up when deleting PR schemas * Fixed an issue where the webhook processor would mark a webhook as processed without queuing a run * Fix a bug where SSH tunnels were not created for the Develop Studio IDE * Fix Develop Studio IDE scrolling in Firefox * Fix a bug where requests were timed out too aggressively * Require company name at signup * Fix security issue where IP blacklist could be bypassed using shorthand * Do a better job of handling git errors * Allow users to delete projects ###### Changed * Move account picker to sidebar * Increase require.js timeout from 7s to 30s * Migrate environments to projects * Move some UIs into Account Settings * Make cron scheduling available on the free tier * Apply new styles to Studio IDE * Speed up develop --- ### Changelog 2021 note This changelog references dbt versions that are no longer supported and have been removed from the docs. For more information about upgrading to a supported version of dbt in your dbt environment, read [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). Welcome to the 2021 changelog for the dbt application! You can use this changelog to see highlights of what was new, fixed, and enhanced. #### dbt Cloud v1.1.41 (December 8, 2021) It's one of the best weeks of the year - it's [Coalesce](https://coalesce.getdbt.com/)! We'll have some exciting product announcements to share! Did somebody say [metrics](https://coalesce.getdbt.com/talks/keynote-metric-system/) and [dbt v1](https://coalesce.getdbt.com/talks/dbt-v10-reveal/)?! ###### New products and features * dbt v1.0 is now available in dbt... nbd. ###### Performance improvements and enhancements * Now whenever you log back into dbt, you'll return to the account and project that you most recently were working in! #### dbt Cloud v1.1.39 (November 10, 2021) We shipped environment variables in dbt. Environment variables create a way to separate code from configuration - allowing you to set config based on context and keep secrets like git tokens securely stored. ###### New products and features * You can now add environment variables to your dbt project. Why does this matter? Environment variables are a fundamental building block of a dbt project, which until now, we only enabled in dbt v1. They power many use cases such as cloning private packages, limiting the amount of data that is processed in development environments, changing your data sources depending on the environment, and more. Read about environment variables in our [blog post](https://blog.getdbt.com/introducing-environment-variables-in-dbt-cloud/) or [docs](https://docs.getdbt.com/docs/build/environment-variables.md). #### dbt Cloud v1.1.38 (October 27, 2021) Have you used the [Metadata API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) yet? The Metadata API is available to customers on the Team and Enterprise plans, and with it, you can learn tons about your dbt project, if it's running dbt v0.19.0 or later. You can now query information about *any* run, not just the last run of a job. Mo' data, mo' fun! #### dbt Cloud v1.1.37 (October 13, 2021) dbt v0.21 is now available in dbt Cloud. The big change with this release is it introduces the `dbt build` command. `dbt build` logically does everything you'd want to do in your DAG. It runs your models, tests your tests, snapshots your snapshots, and seeds your seeds. It does this, resource by resource, from left to right across your DAG. dbt build is an opinionated task. It’s the culmination of all we’ve built- running models with resilient materializations, prioritizing data quality with tests, updating fixtures with seeds, capturing slowly changing dimensions with snapshot. Give it a try! ###### New products and features * We have a new beta feature, which we're calling Model Bottlenecks. It allows you to visually see how long it takes to build models in each run, so you can see clearly which models are taking the longest. If you're interested in learning more, check out #beta-feedback-model-bottlenecks in the dbt community Slack, and we can add you to the beta. #### dbt Cloud v1.1.36 (September 29, 2021) Check out the release candidate for `dbt v0.21.0`! Also tab switching in the dbt Cloud IDE now keeps track of your scroll position - at last! ###### Bug fixes * Some Redshift customers were experiencing timeouts on runs. We've since fixed this bug by keeping the session alive longer. ###### Performance improvements and enhancements * You won't lose track of the code snippets you were looking at when you switch back and forth between tabs in the dbt Cloud IDE, as we now keep track of your scroll position. #### dbt Cloud v1.1.35 (September 15, 2021) Have you ever been working in the Studio IDE, taken a several hour break from developing, and when you returned to your work, the Studio IDE started behaving in unexpected ways? Your develop session became inactive, without any notification. Well, that silent failure won’t happen anymore! dbt now will let you know when you have to refresh your Studio IDE so you can continue to pick up work where you last left off. ###### New products and features * dbt v0.20.2 is released in dbt. ###### Performance improvements and enhancements * Set default threads to 4 for new jobs and in development creds. ###### Bug fixes * The user is now prompted to refresh the page when in a disconnected Studio IDE state. * dbt tasks that fail or error are now correctly ordered in the run drawer history. #### dbt Cloud v1.1.34 (September 1, 2021) We just launched our beta for supporting environment variables in dbt. Environment variables are exciting because they allow you to clone private packages. If you’re interested in joining the beta, check out the #beta-feedback-for-env-vars channel in dbt Slack for more information. ###### Performance improvements and enhancements Our Studio IDE SQL drawer got a fresh new look, and it now has improved accessibility. #### dbt Cloud v1.1.33 (August 18, 2021) We added a DAG in the Studio IDE, so that you can see your model dependencies as you develop! If you haven’t seen the DAG visualization yet, take a moment to spin up the Studio IDE, navigate to the Lineage tab, and click-click-click around in there — it is legitimately a brand new modality for developing dbt projects, and it’s something worth being excited about! ###### New products and features * [Dashboard Status Tiles](https://docs.getdbt.com/docs/explore/data-tile.md) can now be embedded on dashboards (or anywhere you can embed an iFrame) to give immediate insight into data freshness and quality. This helps dbt project maintainers build trust internally about the data that end users are seeing. * We shipped DAG in the Studio IDE to GA! * Support for `dbt v0.20.1` in Cloud. ###### Bug fixes * Databricks users will now be able to see and update the token/schema for deployment environments. * Some Github users were experiencing a broken profile image in dbt. This should be fixed if users disconnect and reconnect their Github accounts. #### dbt Cloud v1.1.32 (August 4, 2021) The Metadata API is now in GA! When dbt invokes certain commands like run, test, seed, etc, dbt generates metadata in the form of [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md). These artifacts give you tons of information about project set up, run times, test details, compiled SQL, and so much more. Now dbt serves a GraphQL API which supports arbitrary queries over these artifacts, so you can retrieve the metadata you want almost instantaneously. ###### New products and features * The Metadata API is the start of our metadata product suite. Learn more about how to use the Metadata API [here](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). * dbt Enterprise customers using GitHub now get better fine-grained access control in their dbt projects. dbt will enforce git permissions for every developer to ensure that read / write policies in GitHub carry through to the IDE. #### dbt Cloud v1.1.31 (July 21, 2021) We’ve improved the tabbing experience in the Studio IDE. Tabs now work much more intuitively, and you don’t have to worry about losing your work anymore! ###### New products and features * We are working to release a DAG directly in the IDE, so that when you’re developing, you have a clear idea of where the model you’re working on sits in the dependency graph. If you’re interested in testing out the feature early, head over to the `#beta-feedback-for-ide-dag` channel in the dbt Slack, and we’ll get the new product feature-flagged on your account! * Added dbt 0.20.0 to Cloud ###### Bug fixes * Users will now be able to initialize any project that doesn't contain a `dbt_project.yml` file, regardless of whether or not there are pre-existing files and/or commits to that repo. ###### Performance improvements and enhancements * We've been working on some nice improvements to tabs in our Studio IDE. We’ve fixed deficiencies with tabs that caused users to lose work if they didn’t hit save regularly enough. Additionally, opening, closing, and the order of the tabs work much more smoothly. * You may have noticed that there is now a source freshness checkbox in your execution settings when you configure a job on dbt Cloud. Selecting this checkbox will run `dbt source freshness` as the first step in your job, but it will not break subsequent steps if it fails. Updated source freshness documentation available [here](https://docs.getdbt.com/docs/deploy/source-freshness.md). * Added a new endpoint to allow API key rotation via `POST https://cloud.getdbt.com/api/v2/users/{user-id}/apikey` #### dbt Cloud v1.1.30 (July 7, 2021) We shipped a resizable folder pane in the Studio IDE, and we're hearing great things! "My quality of life has greatly increased with this little update!" Hope this helps everyone else enjoy the Studio IDE a little more too. ###### New products and features * Resizable folder pane in the Studio IDE: Have you ever developed in the Studio IDE and not been able to see the full name of your model because you couldn’t adjust the width of the file pane? Yeah, us too. Now you’ll be able to adjust your project’s file tree width to be as wide or as narrow as you’d like. It’s these small things that make developing in the Studio IDE so much easier. ###### Bug fixes * Made some changes to GitLab webhooks so that the status of the dbt run gets properly updated in GitLab. * Resolved an issue where users saw a blank screen rather than the SSO reauthentication page. ###### Performance improvements and enhancements * Refreshed the design of the repository import page. #### dbt Cloud v1.1.29 (June 23, 2021) We're heads down working on a handful of new features that we're going to share at the end of this month. The finish line is in sight. In the meantime, check out our latest release candidates for dbt v1. The biggest changes are better tests, providing consistency, configurability, and persistence. ###### New products and features * Add support for latest Core release candidates to dbt: v0.19.2-rc2 and v0.20.0-rc1 ###### Bug fixes * Add a safeguard for the SSO reauth page to avoid 401 interceptors ###### Performance improvements and enhancements * Ensure navigation bar is in dark mode when Studio IDE is set to dark mode #### dbt Cloud v1.1.28 (June 9, 2021) We shipped a far better experience for GitLab users. Be sure to check out new CI features that are now available for customers using GitLab. Additionally, all developers should test out Slim CI which will speed up their model builds. ###### New products and features * `Slim CI`: We’ve made Slim CI available for all our cloud customers! With Slim CI, you don't have to rebuild and test all your models; you can instruct dbt Cloud to run jobs on only modified or new resources. If you are a GitHub or GitLab user, try creating a new job that runs on pull requests and you can signal to dbt to run only on these modified resources by including the `state:modified+` argument. Read more about Slim CI [here](https://docs.getdbt.com/docs/deploy/continuous-integration.md). * Native GitLab authentication for dbt Developer and Team Tiers: We’ve shipped native GitLab auth into GA. You can now import new GitLab repos with a couple clicks, trigger CI builds when Merge Requests are opened in GitLab, and carry GitLab permissions through to Studio IDE's git actions. Read how to set up native GitLab auth [here](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md). ###### Bug fixes * Allow users to select artifacts from a job that runs source freshness on jobs with the source freshness execution settings set to `ON`. * Resolve `RUN ONLY ON CUSTOM BRANCH?` button to toggle on and off properly. * Retain information in a `Statement` tab when the page is refreshed. * Unsaved changes in the Studio IDE are now saved when committing work. * Drop temporary schemas in the data warehouse for closed or merged GitLab merge requests. ###### Performance improvements and enhancements * Behind the scenes, we’ve been moving off of Angular and onto React. We’ve started the process of migrating the central pieces of our UI over - the first of which is the main navigation. We think this will have a big impact on our ability to reduce UI bugs and improve user experience. * Added support for dbt 0.19.2rc2 + 0.20.0rc1 in dbt. #### dbt Cloud v1.1.27 (May 26, 2021) A lot of improvements coming for GitLab webhooks and native auth. We also fixed a number of bugs in the Studio IDE. Our goal is for you to never see an infinite spinner again! ###### Enhancements * Add dbt v0.19.2rc1 and v0.20.0b1 * Add an open/closable overlay for the DAG * Disable department dropdown * Add DAG flags, button, and tab context * Add run source freshness option to jobs * Implement conditional redirecting after GitLab app integration * Add Develop Pod Support for Rook and Ceph file storage * Show all common actions for valid top level commands ###### Fixed * Fix link to documentation * Disable the "Restart Studio IDE" Button while the Studio IDE is loading * Continue canceling runs when we run into deleted accounts * Fix SSO re-auth page * Fix blank verify email page * Resolve git refresh regression * Fix missing "Run on Merge" button in Job creation/edit form- * Warn users they have unsaved changes * Updates test command suggestions and regex for common action suggestions * Updates order of stylesheet import to fix missing border bug * Fix GitLab PR link for Run Page * Fix infinite spinner for missing environment or user credentials * Fix infinite spinner when user is missing dev credentials * Do not try to push if awaiting a merge * Fix deleting schemas * Fix favicon reference #### dbt Cloud v1.1.26 (May 12, 2021) If you haven't seen it yet, spin up the Studio IDE: the command bar now has recent actions (you can up-arrow like on the command line) as well as some hardcoded suggestions that will auto-populate your active model, if there is one. Check it out! Other fixes and adjustments as well, as we all get ready for Staging this Thursday - exciting week for the Product org over at ol' Fishtown! ###### Enhancements * Made dbt default version on env 0.19.1 * Rolled out new command line experience to all customers * Post webhook triggered run status back to gitlab * Temporary tabs can also populate the model from manifest * Check command line content is minimally valid * Allow user to restart server when develop pod crashes * Prevent overflow of menu items ###### Fixed * Handle validation error for improper remote URLs in the Scheduler * Refactor exception logging out of GitRepo and into exception handlers * Required tags returning null from core no longer causing infinite spinner * Removed deleted repos while fetching repository for sending commit statuses * Refactor git provider service * Resolve files with special characters becoming forever dirty * Disable input when RPC command running & add button when command bar is empty * Updating button for the Cancel/Enter button on commandline * Fix connection setup to always use the project referenced in the route * Fix "View data sources" URL in environment page * Add support for clicking on previously run commands and updating the text inside of the commandline * Fix sources URL in environments page * Fix metadata token not allowed API response #### dbt Cloud v1.1.25 (April 28, 2021) Exciting things coming down the pipe - ongoing enhancements to the command bar experience in the Studio IDE, doing some work to ensure that more git providers are presented with a first class experience in Cloud, as well as assorted bug fixes - "I must have bug fixes, always and always" - that was Monet I think ###### Enhancements * Made a grip of visual updates to the new command bar work * Moved to using the active model name instead of a placeholder in command bar work * Added user ability to delete connections, remove association from a given project. * Added verification of dbt version for command bar beta feature flag ###### Fixed * Removed testing prop that keeps drawer open * Added double encoding to handle Snowflake roles with spaces * Fixed account switching in user notifications * Handled invalid Azure SSO group responses * Fixed error which only showed common actions when run drawer was closed * Allowed unencrypted adapter fields to be edited * Fixed bugs with file and folder renaming, alongside associated tab state #### dbt Cloud v1.1.24 (April 14, 2021) Phew! As our company grows, so too does our changelog! Look at all these! The big chunks you'll see here are related to some ongoing in-Studio IDE work, focused on the command bar experience, as well as some partner & connection work (see the Gits, Databricks, and so forth), and of course ongoing longer-term bets around metadata! ###### Enhancements * Added onFocus and onBlur properties to populate and remove "dbt" in command bar * Enabled executing command on enter if user's cursor is in the command bar * Added Metadata API access button to account settings * Added feature flag for displaying only recent actions * Added dbt 0.19.1 * Added regex validation to Databrick's hostname web-form field * Updated Connection Edit to allow adapter editing * Enabled self-service Github and GitLab integration disconnection * Added link to docs for license map & handle duplicate error gracefully * Moved deferred job execution to execution settings. * Recorded user command history * Enabled new file creation flow ###### Fixed * Added styling class to popup to ensure text is readable * Fixed sourcemaps syntax for dev commands * Added timeout and retry to dbt deps * Updated databricks schema field type and add error handling to ConnectionSetup * Fixed Bigquery private keys & convert text to textarea * Fixed last used datetime in the service token UI * Added missing token URI to Bigquery connection edit * Prevent multiple develop sessions for one user * Fixed SchemaForm validating non-displayed fields * Fixed required fields for Bigquery connection JSON uploads * Fixed self selection as deferred job * Always create a Monaco model on tab open if no matching model exists * Fixed tab dirty indicator on open tab * Fixed password reset flow * Fixed docs and sources links in dashboard page for read only users * Fixed truncating first\_name to 30 characters #### dbt Cloud v1.1.23 (March 31, 2021) Some backend work, some frontend work, some bug fixes: a nice mix for this release. A few user facing changes you may have noticed already are the persistence of dark/light mode settings across refresh (no more blinding Studio IDE!), branches in the Studio IDE being categorized by Active vs. Removed from Remote, and a tidier new file creation flow, with the file tree expanding to show the new file and opening a new tab to populate the said file! ###### Enhancements * Splitting Local-only and Remote branches into different sections of the dropdown selector * Update Profile Integrations to include SSO info * Upgrade to Tailwind 2.0 and FUI 0.0.5 * Allow users to create metadata tokens from the UI * Support manually-managed group memberships * SSO: resolve bug w/ first & last names acting up * Integrate Delighted for NPS surveys * Add dbt 0.19.1rc1 to Cloud * Add an account-level setting to require users to re-authenticate via SSO * Read-only metadata ServiceToken for Cloud * Persist Studio IDE light mode / dark mode across refresh * Categorize & order git branches * Improve new file creation flow ###### Fixed * Check for an empty repository before checking matching remote * Increase wait if run was finished recently * Support default branches through git when a custom branch is not specified * Don't download logs for skipped steps * API Gateway is no longer flooded with errors due to Studio IDE blindly polling dead Develop pod * Fix user license creation via admin interface * Adjusted addition of global .gitignore #### dbt Cloud v1.1.22 (March 17, 2021) Rolling out a few long-term bets to ensure that our beloved dbt does not fall over for want of memory, as well as a grip of bug fixes and error messaging improvements (error messages should be helpful, not scolding or baffling, after all!) ###### Enhancements * Release Scribe to 100% of multi-tenant accounts * Update language for SQL drawer empty state * Reduce Scribe memory usage ###### Fixed * Fix NoSuchKey error * Guarantee unique notification settings per account, user, and type * Fix for account notification settings * Don't show deleted projects on notifications page * Fix unicode error while decoding last\_chunk * Show more relevant errors to customers * Groups are now editable by non-sudo requests * Normalize domain names across inputs/outputs * Redirect auth failed errors back to appropriate page with error description #### dbt Cloud v1.1.21 (March 3, 2021) This changelog wraps up work on what we've been calling the SQL Drawer in the Studio IDE - some design nudges, some interface adjustments, overall a cleaner and snappier experience. If you haven't dipped into the Studio IDE in a while it's worth taking a look! Some back-end work as well, making SSO and role based admin easier and more broadly available for Enterprise level folks, along with your usual assortment of bug squashes and iterations. ###### Enhancements * Styling and copy adjustments in the Cloud Studio IDE * Open self-service role based access control to all Enterprise customers * Update AuthProvider UI to enable SAML and Okta * Add a SAML auth redirect URL ###### Fixed * Add param to admin project mapper to included soft deleted projects * Fix delaying logs when we are waiting for a model to finish executing * Saving GSuite auth provider form triggers an authorize * Scribe populates truncated debug logs when runs are executing * Delay attempts for non-200 status codes * Add logic to support select fields in adapter UI * Undo clobbering groups #### dbt Cloud v1.1.20 (February 17, 2021) Continued stability and quality of life improvements for folks with multiple accounts and projects - no longer will you have to remember the chronological order of birth of your accounts and projects, as they'll be ordered by the much easier to parse (for human brains anyway) alphabetical order. We're also shipping some experience improvements in the SQL Drawer at the bottom half of the Studio IDE. ###### Enhancements * Deleted Info and Logs Studio IDE Tabs, logs will now be displayed in Results Tab * Removed service token feature flag * List Jobs dropdown in alphabetical order * List Account and Project dropdowns in alphabetical order * Pre-join Job Definition results to speed up scheduler * Combine scheduler queries to speedup runtime by about 30% ###### Fixed * Fix issue with source freshness for 0.19.0 #### dbt Cloud v1.1.19 (February 3, 2021) The latest release of dbt (Oh Nineteen Oh) is now available for your enjoyment on dbt! We're also releasing some service token pieces here, though they're not quite ready for wide release yet. Moving forward, Oh Nineteen Oh will probably end up being the minimum version required to run the Metadata API & Metadata Toolkit, so, this is a big release! ###### Enhancements * Added dbt 0.19.0 😻 * Allowed account-wide service tokens to create connections * Added integration for service token UI and API * Authorized requests that supply a service token ###### Fixed * Added logic to show the entered service token name prior to the request completing * Fixed endlessly running rpc queries with non-working cancel button on Studio IDE refresh #### dbt Cloud v1.1.18 (January 20, 2021) Most notable things here are around foundational work toward future feature releases, as well as strong assurances of future stability for dbt, and ensuring future sales tax compliance (which we understand turns out to be quite important!) - turns out to be a quite future-looking release! ###### Enhancements * Add service tokens UI (stubbed) behind a feature flag * Fixing and Upgrading social-auth * Add dbt Spark 0.19.0rc1 * Adds the reconciliation of persisted file content and tab state when navigating into the Studio IDE * Adds the reconciliation of persisted file content and tab state between Studio IDE sessions * Read logs from scribe and stop logging to db * Upgrade social auth 3.3.3 * Add warning logs for social auth failures * Add dbt 0.19.0rc1 ###### Fixed * Prevent social-auth from updating first or last name * Page through Stripe results when listing subscriptions * Prevent enqueueing runs in deleted projects * Fix Studio IDE git actions causing open tab contents to be lost on Studio IDE re-entry * Add DBT\_CLOUD\_CONTEXT environment variable * Add logic to hide IP whitelist message for on-prem customers * fix 0.19.0rc1 run image dependencies --- ### Checks Available in v2 As dbt projects grow and more contributors add models, maintaining consistent standards becomes harder: a model ships without a description, a public model has no owner, a new model ignores naming conventions. None of this breaks anything, so it's not caught, but quality erodes silently. Checks are SQL queries that assert rules and standards about your project metadata. Define rules (such as every model must have a description, required tags are set, and so on) and dbt enforces them before any warehouse work runs. Write your checks in DuckDB SQL, using the [`{{ info_schema() }}` macro](https://docs.getdbt.com/reference/dbt-jinja-functions/info-schema-macro.md) to reference the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) in your query. Checks run locally in DuckDB, so they don't require a connection to your warehouse. If the project violates a rule, `dbt build` stops before compiling or materializing a single model. Similar to [data tests](https://docs.getdbt.com/docs/build/data-tests.md), a check finds any instances in your project that do not meet your expectations. A check passes when the query returns zero rows, and fails when it returns one or more. Checks run automatically with every `dbt build`; you can use `--skip-checks` to skip them. You can also run them on demand with `dbt check`. #### Guidelines for writing SQL check files This section covers the rules and constraints for writing check SQL files and configuring check behavior. * A check is a SQL file in your `checks/` directory. You can optionally pair it with a properties YAML file in the same directory to configure it. To use a different directory, set [`check-paths`](https://docs.getdbt.com/reference/project-configs/check-paths.md) in `dbt_project.yml`. * The filename without the `.sql` extension becomes the check name (for example, `all_models_have_descriptions` is the check name for `checks/all_models_have_descriptions.sql`). * Jinja in check files renders at parse time. You can use Jinja, but the result must be valid DuckDB SQL at that point; checks do not go through a separate compile step the way models do. * Checks are dbt resources; each check appears in `manifest.json`, supports `tags` and `meta`, and `dbt ls` lists them. ##### The `info_schema()` macro [`{{ info_schema() }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/info-schema-macro.md) is the supported way to reference the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) in a check. Pass the name of the table you want to query (for example, `{{ info_schema('models') }}` for models or `{{ info_schema('edges') }}` for DAG edges). No materialized [Information Schema](https://docs.getdbt.com/reference/info-schema.md) files are required; checks run against an intermediate representation built at parse time. For the full list of available views and columns, refer to [Columns available for checks](https://docs.getdbt.com/reference/info-schema.md#columns-available-for-checks). #### Writing your first check The following steps walk you through creating your first check. 1. Declare the `info_schema` version in `dbt_project.yml`: The `info_schema.version` pins which version of the info schema the [`{{ info_schema() }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/info-schema-macro.md) macro resolves to. Currently, `1` is the only available version. dbt\_project.yml ```yaml info_schema: version: 1 ``` 2. Write a check under `checks/`: Use [`{{ info_schema() }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/info-schema-macro.md) to query project metadata. A check passes if the query returns zero rows. For available views and columns, refer to [Columns available for checks](https://docs.getdbt.com/reference/info-schema.md#columns-available-for-checks). checks/all\_models\_have\_descriptions.sql ```sql select unique_id from {{ info_schema('models') }} where description = '' ``` 3. Configure the check in a properties YAML file in your `checks/` directory. checks/\_checks.yml ```yaml version: 2 checks: - name: all_models_have_descriptions description: "Fails if any model is missing a description." config: severity: warn # default is error; 'warn' logs issues but does not fail the execution ``` 4. Run your checks: ```shell dbt check ``` #### Example checks The following examples show common project quality rules, each defined as a SQL query against the `info_schema` macros and saved as a `.sql` file under `checks/`. * Enforce that all `public` models have a description: checks/public\_models\_have\_descriptions.sql ```sql select unique_id from {{ info_schema('models') }} where access = 'public' and description = '' ``` * Flag sources that aren't referenced by models. An unreferenced source has no downstream models, and is either unused or missing a model that should reference it: checks/unused\_sources.sql ```sql select s.unique_id, s.name from {{ info_schema('sources') }} as s left join {{ info_schema('edges') }} as e on e.parent_unique_id = s.unique_id where e.child_unique_id is null ``` * Flag `public` models with no group owner: checks/public\_models\_have\_owners.sql ```sql select m.unique_id from {{ info_schema('models') }} as m left join {{ info_schema('groups') }} as g on g.name = m."group" where m.access = 'public' and ( g.unique_id is null or ( (g.owner_name is null or trim(g.owner_name) = '') and (g.owner_email is null or trim(g.owner_email) = '') ) ) ``` #### Commands Checks run with `dbt check` and `dbt build`. | Command | Behavior | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt check` | Runs all checks. | | `dbt check <name1> <name2> …` | Runs only the named checks. An unknown check name is an error; a disabled check name is accepted and skipped. | | `dbt build` | Runs all enabled checks before models compile. A failing check stops the run before any model is compiled or executed. Warn failures are reported and the build continues. Use `--skip-checks` to bypass. | #### Skipping checks on build To skip all checks, pass the [`--skip-checks` flag](https://docs.getdbt.com/reference/commands/build.md?version=2#the---skip-checks-flag) to `dbt build`. ```shell dbt build --skip-checks ``` To skip a specific check, set `enabled: false` in its config block in the YAML file. The check still appears in the manifest but does not run. ```yaml checks: - name: all_models_have_descriptions config: enabled: false ``` Disabling at the project level Setting `+enabled: false` in `dbt_project.yml` disables the check silently — nothing in the output shows checks were skipped. A successful `dbt build` in CI doesn't tell you whether the project has no checks or all checks are disabled. Running `dbt check <name>` for a disabled check also succeeds without running the check or returning an error. #### Using selectors with checks With checks, [`--select` and other selector methods](https://docs.getdbt.com/reference/node-selection/syntax.md) choose which project resources to evaluate, not which checks run. Why checks work this way: * You generally don't need to exclude checks or run only a subset of them. Checks are fast. Error-severity checks should block execution if violated; if a rule is informational, set it to `warn`. If a check is no longer relevant, disable or delete it. * You may want to limit which resources are checked. This lets you incrementally introduce checks in an existing project. In development, run `dbt build --select <the part of your DAG you're working on>` to check only those resources. In CI, your checks run only against modified resources. * When developing a new check, you can run one check at a time or optionally select resources: * `dbt check name_of_check` * `dbt check name_of_check --select <resources to check>` You can also preview any `info_schema` query directly: `dbt show --inline "select * from {{ info_schema('...') }}"`. For example, to inspect your checks' own metadata, run `dbt show --inline "select * from {{ info_schema('checks') }}"`. ##### How `selection_filter_on` works By default, dbt filters check results to selected resources by matching the `unique_id` column in the output. If the check returns no `unique_id` column, it runs against the whole project. Use [`selection_filter_on`](https://docs.getdbt.com/reference/resource-configs/selection-filter-on.md) when your check returns rows with different ID columns that you want to filter on. It accepts the following values: * **Default (not set):** If the query returns `unique_id`, dbt keeps only rows whose `unique_id` is in the selection. If the query does not return `unique_id`, the check runs against the whole project. Useful for aggregate checks like "the project has at least one model". * **`none`:** The check always runs against the whole project, ignoring any selector. Use this to make whole-project behavior explicit. * **A column name or list of column names:** dbt keeps a row if the ID in any of the named columns is in the selection. Each named column must exist in the result, or the check raises an error. Use this for checks that return relationships between resources (edges). For example, the following check returns `child_unique_id` instead of `unique_id`, so you must set `selection_filter_on`: checks/multiple\_sources\_joined.sql ```sql select child_unique_id, count(*) as source_parents from {{ info_schema('edges') }} where parent_unique_id like 'source.%' group by child_unique_id having count(*) > 1 ``` checks/\_checks.yml ```yaml checks: - name: multiple_sources_joined description: "Fails if any model reads directly from more than one source." config: selection_filter_on: child_unique_id ``` #### Related documentation * [Check properties](https://docs.getdbt.com/reference/check-properties.md) * [Check configurations](https://docs.getdbt.com/reference/check-configs.md) * [`dbt check` command](https://docs.getdbt.com/reference/commands/check.md) * [`info_schema`](https://docs.getdbt.com/reference/dbt-jinja-functions/info-schema-macro.md) * [`check-paths` project config](https://docs.getdbt.com/reference/project-configs/check-paths.md) --- ### Column-level lineage Local development | dbt platform Column-level lineage (CLL) gives you insight into the provenance of your data products at a more granular level. For each column in a resource (model, source, or snapshot) in a dbt project, dbt provides end-to-end lineage for the data in that column given how it's used. [![Overview of column level lineage](/img/docs/collaborate/dbt-explorer/example-overview-cll.png?v=2 "Overview of column level lineage")](#)Overview of column level lineage #### Prerequisites You can use CLL in two places: * **[Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) in the dbt platform**: Requires an Enterprise or Enterprise+ plan with Catalog access. * **Locally with dbt v2**: Requires [strict static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md#configuring-static_analysis). Available in the [dbt VS Code extension](https://docs.getdbt.com/docs/dbt-extension-features.md#rich-lineage-in-context), [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2), or from the command line. For more information, refer to [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md?version=2). On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### Access the column-level lineage in Catalog There is no additional setup required for CLL in Catalog if your account is an Enterprise or Enterprise+ plan with Catalog access. You can access CLL from the column card in the **Columns** tab in the Catalog [resource details page](https://docs.getdbt.com/docs/explore/explore-projects.md#view-resource-details) for a model, source, or snapshot. dbt updates the lineage in Catalog after each run that's executed in the production or staging environment. (Applies to dbt v2.0 and later) Jobs running v2 refresh this metadata automatically on every job run. Refer to [Generating metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details. [![Example of the Columns tab and where to open the CLL](/img/docs/collaborate/dbt-explorer/example-cll.png?v=2 "Example of the Columns tab and where to open the CLL")](#)Example of the Columns tab and where to open the CLL #### Access column-level lineage locally When you develop with dbt v2, you can see column-level lineage in these ways: * **dbt VS Code extension**: Right-click a filename or a model's SQL, then select **dbt: View Lineage** → **Show column lineage**. Refer to [Rich lineage in context](https://docs.getdbt.com/docs/dbt-extension-features.md#rich-lineage-in-context) for the full workflow. * **dbt Docs v2**: Run `dbt compile --generate-info-schema --static-analysis strict`, then `dbt docs generate --no-compile`. `dbt docs generate` has no `--static-analysis` flag, so the strict compile must come first or the site won't include column lineage. Refer to [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2). * **Command line or artifact**: Run `dbt compile --generate-info-schema --static-analysis strict`, then [`dbt show --info column_lineage`](https://docs.getdbt.com/docs/build/dbt-information-schema.md#querying-with-dbt-show). You can find these files in `target/info_schema/` and read the [`dbt.column_lineage`](https://docs.getdbt.com/reference/info-schema.md) Parquet directly. #### Column evolution lens You can use the column evolution lineage lens to determine when a column is transformed vs. reused (passthrough or rename). The lens helps you distinguish when and how a column is actually changed as it flows through your dbt lineage, informing debugging workflows in particular. [![Example of the Column evolution lens](/img/docs/collaborate/dbt-explorer/example-evolution-lens.png?v=2 "Example of the Column evolution lens")](#)Example of the Column evolution lens The **Transformation** column previews the formula used to produce a column’s values. To view the full formula, select the column in the full lineage view. The transformation type and formula appear after you update your project and a job runs in your production or staging environment. ##### Inherited column descriptions A reused column, labeled as **Passthrough** or **Rename** in the lineage, automatically inherits its description from the source and upstream model columns. The inheritance goes as far back as possible. As long as the column isn't transformed, you don't need to manually define the description; it'll automatically propagate downstream. Passthrough and rename columns are clearly labeled and color-coded in the lineage. In the following `dim_salesforce_accounts` model example (located at the end of the lineage), the description for a column inherited from the `stg_salesforce__accounts` model (located second to the left) indicates its origin. This helps developers quickly identify the original source of the column, making it easier to know where to make documentation changes. [![Example of lineage with propagated and inherited column descriptions.](/img/docs/collaborate/dbt-explorer/example-prop-inherit.png?v=2 "Example of lineage with propagated and inherited column descriptions.")](#)Example of lineage with propagated and inherited column descriptions. #### Column-level lineage use cases Learn more about why and how you can use CLL in the following sections. ##### Root cause analysis When there is an unexpected breakage in a data pipeline, column-level lineage can be a valuable tool to understand the exact point where the error occurred in the pipeline. For example, a failing data test on a particular column in your dbt model might've stemmed from an untested column upstream. Using CLL can help quickly identify and fix breakages when they happen. ##### Impact analysis During development, analytics engineers can use column-level lineage to understand the full scope of the impact of their proposed changes. This knowledge empowers them to create higher-quality pull requests that require fewer edits, as they can anticipate and preempt issues that would've been unchecked without column-level insights. ##### Collaboration and efficiency When exploring your data products, navigating column lineage allows analytics engineers and data analysts to more easily navigate and understand the origin and usage of their data, enabling them to make better decisions with higher confidence. #### Caveats in Catalog Refer to the following CLL caveats or limitations as you navigate Catalog. ##### Column usage Column-level lineage reflects the lineage from `select` statements in your models' SQL code. It doesn't reflect other usage like joins and filters. ##### SQL parsing Column-level lineage relies on SQL parsing. Errors can occur when parsing fails or a column's origin is unknown (like with JSON unpacking, lateral joins, and so on). In these cases, lineage may be incomplete and dbt will provide a warning about it in the column lineage. [![Example of warning in the full lineage graph](/img/docs/collaborate/dbt-explorer/example-parsing-error-pill.png?v=2 "Example of warning in the full lineage graph")](#)Example of warning in the full lineage graph To review the error details: 1. Click the **Expand** icon in the upper right corner to open the column's lineage graph 2. Select the node to open the column’s details panel Possible error cases are: * **Parsing error** — Error occurs when the SQL is ambiguous or too complex for parsing. An example of ambiguous parsing scenarios are *complex* lateral joins. * **Python error** — Error occurs when a Python model is used within the lineage. Due to the nature of Python models, it's not possible to parse and determine the lineage. * **Unknown error** — Error occurs when the lineage can't be determined for an unknown reason. An example of this would be if a dbt best practice is not being followed, like using hardcoded table names instead of `ref` statements. --- ### Community adapters Adapters connect dbt to data platforms. They are maintained by dbt Labs, partners, and community members. We welcome [contributions to an existing adapter](https://docs.getdbt.com/docs/contribute-dbt-adapters.md#contribute-to-a-pre-existing-adapter). Many adapters are maintained by volunteers, so be kind and help where you can. (Applies to dbt v2.0 and later) To contribute a v2 adapter, refer to [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md?step=1). ![](/img/icons/apache-spark.svg) ###### Apache Spark * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-apache-spark.md) * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md) dbt platformdbt v2 ![](/img/icons/bigquery.svg) ###### BigQuery * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md) * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md) dbt platformdbt v2 ![](/img/icons/clickhouse.svg) ###### ClickHouse * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-clickhouse.md) * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/clickhouse-setup.md) dbt platformdbt v2 ![](/img/icons/databricks.svg) ###### Databricks * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-databricks.md) * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) dbt platformdbt v2 ![](/img/icons/duckdb-seeklogo.svg) ###### DuckDB * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md?version=2) dbt v2 ![](/img/icons/redshift.svg) ###### Redshift * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md) * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) dbt platformdbt v2 ![](/img/icons/snowflake.svg) ###### Snowflake * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md) * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md) dbt platformdbt v2 --- ### Configure BYOK for dbt Wizard Beta ### Configure BYOK for dbt Wizard [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development You can use the dbt Wizard CLI with bring-your-own-key (BYOK), which means you supply your own credentials from a supported AI provider instead of using dbt Labs' infrastructure. The following BYOK instructions on this page apply to the CLI only. For dbt platform BYOK setup, refer to [Configure BYOK for dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md). If you'd rather skip that upkeep, the [dbt Labs-managed option](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider) is ready to use with no setup, and comes with [trial and monthly usage credits](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). The "key" in BYOK is whatever credential your chosen provider uses to authenticate API requests — an API key for OpenAI or Anthropic, a bearer token for AWS Bedrock, or a token/PAT for Snowflake Cortex. When you configure a provider with that credential, dbt Wizard calls the provider's API directly using it, so: * Usage costs appear on your provider account, not your dbt Labs account. * Token costs are billed by whichever provider you choose. dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Configure a provider You can configure a provider in one of the following ways: * [**Terminal commands**:](#configure-in-the-terminal) Best for users who want to configure providers from the shell. * [**Interactive session**:](#configure-in-the-tui) Best for most users working in the dbt Wizard text based user interface (TUI). Use the `/providers` slash command. * [**Environment variables**:](#set-your-api-key) Best for headless runs, such as `wizard exec`, automation, or temporary local sessions. ##### Configure in the terminal In the terminal, use the `providers` subcommand to list, configure, and enable providers: ```bash wizard providers list wizard providers configure PROVIDER_NAME wizard providers enable PROVIDER_NAME ``` Replace `PROVIDER_NAME` with the name of a supported provider, such as `openai`, `anthropic`, `bedrock`, `azure`, `gemini`, `snowflake`, or `databricks`. Then, follow the prompts to enter your credentials. The `wizard providers list` command shows you the currently configured providers and their status: ```bash ➜ jaffle-shop git:(mwong-v2) wizard providers list provider enabled route auth models dbt true remote dbt 3 openai false local missing 3 openai_subscription false local missing 1 anthropic false local missing 3 bedrock true local configured 15 azure false local missing 3 snowflake false local missing 3 gemini false local missing 3 ``` The `configure` command prompts you to enter credentials for the selected provider. To use environment variables or set a key without echoing it in your shell history, refer to [Set your API key](#set-your-api-key). So for example, if you're using OpenAI, you would run and follow the prompts to configure it: ```bash wizard providers configure openai wizard providers enable openai ``` To store an API key without echoing it in your shell history: ```bash printf '%s' 'sk-...' | wizard providers set-key PROVIDER_NAME ``` Credentials are stored in `~/.dbt/wizard/provider-auth.json`. Provider settings are stored in `~/.dbt/wizard/providers.json`. ##### Configure in the TUI You can configure providers from an active CLI TUI session with the `/providers` slash command: ```bash /providers ``` From the provider menu, you can: * Enable or disable a provider. * Add or update credentials. * Select available models. * Check whether a provider is authenticated and active. Example provider menu: ```bash /providers Model Providers Select a provider to inspect or update it. › 1. dbt enabled; 3/3 models selected; uses dbt login; active 2. openai disabled; 3/3 models selected; missing credentials; needs setup 3. openai_subscription disabled; 1/1 models selected; not connected; needs setup 4. anthropic disabled; 3/3 models selected; authenticated; needs setup 5. bedrock enabled; 15/15 models selected; authenticated; active 6. azure disabled; 3/3 models selected; missing credentials; needs setup 7. snowflake disabled; 3/3 models selected; missing credentials; needs setup 8. gemini disabled; 3/3 models selected; missing credentials; needs setup ``` #### Set your API key ##### Interactive session The first time you start dbt Wizard in a project, onboarding prompts you to choose a provider. 1. At the **Configure a Provider** prompt, select your provider. 2. Paste your API key or provider credentials when prompted. 3. Choose an AI model to finish setup. To add or switch providers later from an active session, type `/providers` in the TUI. When you configure a provider through dbt Wizard, credentials are stored in `~/.dbt/wizard/provider-auth.json`. ##### Environment variable Set the key as an environment variable if you want to: * Run dbt Wizard in headless mode, such as with `wizard exec`. * Use a key for the current terminal [session](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#sessions) only. * Reuse the same key across different terminal sessions. * Avoid storing credentials in the dbt Wizard config directory. ##### OpenAI ```bash export OPENAI_API_KEY="sk-..." ``` ##### Anthropic ```bash export ANTHROPIC_API_KEY="sk-ant-..." ``` ##### Amazon Bedrock ```bash export AWS_BEARER_TOKEN_BEDROCK="ABSK..." ``` ##### Azure AI Foundry ```bash export AZURE_API_KEY="..." ``` ##### Google Gemini ```bash export GOOGLE_API_KEY="..." ``` ##### Snowflake Cortex ```bash export SNOWFLAKE_API_KEY="..." ``` ##### Databricks ```bash export DATABRICKS_API_KEY="dapi..." export DATABRICKS_API_BASE="https://adb-1234567890.azuredatabricks.net" ``` To make an environment variable available across terminal sessions, add it to your shell profile, such as `.zshrc`, `.bashrc`, or equivalent. Avoid committing API keys to version control, project files, or shared configuration. To persist provider credentials, use one of the following options: * `wizard providers configure PROVIDER_NAME` * `wizard providers set-key PROVIDER_NAME` * The provider's environment variable #### Choose an AI model dbt Wizard CLI uses dbt Labs-managed models out of the box. You can switch to a BYOK model in the following ways: | Method | Description | Example | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Interactive session | Change the AI model interactively in the TUI. | `/model` | | Invocation flag | Change the AI model at invocation with the `-m` flag. | `wizard -m gpt-4o "refactor stg_orders to use incremental materialization"` | | Config file | Set a default AI model in the config file (`~/.dbt/wizard/config.toml`); applies to all future sessions. Run `wizard debug models` to list available model IDs. | `model = "gpt-4o"` | Restart the dbt Wizard CLI after changing the model. #### Examples The following examples use the same provider configuration flow described earlier, with provider-specific credential requirements. ##### AWS Bedrock AWS Bedrock is supported in the CLI only. dbt Wizard currently supports Bedrock through an Amazon Bedrock API key, not the full AWS credential chain. Ensure your AWS account has access to the Bedrock models you plan to use and that your Bedrock API key has permission to invoke them. ```bash export AWS_BEARER_TOKEN_BEDROCK="ABSK..." wizard providers enable bedrock wizard providers bedrock set-region us-east-1 wizard providers list wizard debug models ``` To set a default Bedrock model, add the model ID to `~/.dbt/wizard/config.toml`: ```toml model = "BEDROCK_MODEL_ID" ``` ##### Snowflake Cortex [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Snowflake Cortex BYOK support in the CLI is in preview. Availability and setup steps may change. Ensure your Snowflake account has the privileges required for Cortex large language model (LLM) functions. Refer to the [Snowflake Cortex documentation](https://docs.snowflake.com/en/user-guide/snowflake-cortex/overview). You can [authenticate](#authentication-options) with Snowflake Cortex using an API token or a Programmatic Access Token (PAT) (for SSO and Okta users). ```bash wizard providers list wizard providers enable snowflake wizard providers configure snowflake wizard providers list wizard debug models ``` The `wizard providers configure snowflake` command walks you through the following prompts: ```text Enable this provider? [Y/n]: Models to enable [1]: Snowflake account ID: Snowflake API base override (optional): Paste API key/token, or press enter to configure it later: ``` | Prompt | What to enter | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | | **Snowflake account ID** | Your Snowflake account identifier (for example, `myorg-myaccount`) | | **Snowflake API base override (optional)** | Leave blank — this is only needed for custom or private Snowflake endpoints | | **Paste API key/token** | Your authentication token — refer to [Authentication options](#authentication-options) in the next section | ###### Authentication options The key/token field accepts a regular API token or a Programmatic Access Token (PAT), depending on how your Snowflake account is configured. Both are entered in the same place — the **Paste API key/token** prompt in the terminal, or **Set key/token** (option 3) in the TUI. 1. In your Snowflake account, [generate your API or PAT token](https://docs.snowflake.com/en/user-guide/programmatic-access-tokens#label-pat-generate). 2. Select **Generate token** and copy the token value. 3. Go back to dbt Wizard and paste the PAT at the key/token prompt. To set a default Snowflake Cortex model, add the model ID to `~/.dbt/wizard/config.toml`: ```toml model = "SNOWFLAKE_CORTEX_MODEL_ID" ``` ##### Databricks Unity AI Gateway [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt Wizard connects to Databricks through the [Unity Catalog AI Gateway](https://docs.databricks.com/aws/en/ai-gateway/), so you bring your own models served from your Databricks workspace. Make sure the [serving endpoints](https://docs.databricks.com/en/machine-learning/model-serving/index.html) you plan to use are deployed and that your Databricks token has permission to query them. ```bash export DATABRICKS_API_KEY="dapi..." wizard providers configure databricks wizard providers enable databricks wizard providers list wizard debug models ``` The `wizard providers configure databricks` command first asks for your workspace URL, then for the serving endpoint name behind each model you enable: ```text Enable this provider? [Y/n]: Models to enable [1]: Databricks workspace URL (e.g. https://adb-1234567890.azuredatabricks.net): Endpoint name for databricks/claude-sonnet-4-6: Paste API key/token, or press enter to configure it later: ``` | Prompt | What to enter | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | **Databricks workspace URL** | Your Databricks workspace URL (for example, `https://adb-1234567890.azuredatabricks.net`) | | **Endpoint name** | The serving endpoint name for each model (for example, `databricks-claude-sonnet-4-6`). dbt Wizard suggests a default endpoint name for each model | | **Paste API key/token** | Your Databricks personal access token (PAT) | By default, dbt Wizard maps the following models to Databricks serving endpoints. You can override any endpoint name during configuration: | Model | Default endpoint name | | ------------------- | ------------------------------ | | `claude-sonnet-4-6` | `databricks-claude-sonnet-4-6` | | `claude-opus-4-7` | `databricks-claude-opus-4-7` | | `claude-haiku-4-5` | `databricks-claude-haiku-4-5` | | `gpt-5.5` | `databricks-gpt-5-5` | | `gpt-5.4` | `databricks-gpt-5-4` | | `gpt-5.4-mini` | `databricks-gpt-5-4-mini` | To set a default Databricks model, add the model ID to `~/.dbt/wizard/config.toml`: ```toml model = "databricks/claude-sonnet-4-6" ``` #### Related docs * [Install dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) * [Configuration reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) --- ### Configure environment variables Local development dbt v2 automatically loads environment variables from a `.env` file in your current working directory (the folder you `cd` into and run dbt commands from in your terminal). This helps you manage credentials and settings without hardcoding them in your `profiles.yml` or exposing them in your shell history. #### Using a `.env` file 1. Create a `.env` file in your current working directory (typically at the root of your dbt project): ```env DBT_MY_DATABASE=my_database DBT_MY_SCHEMA=my_schema DBT_SECRET_KEY=my_secret_value ``` 2. Reference these variables in your `profiles.yml` using the [`env_var` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md): ```yaml my_profile: target: dev outputs: dev: type: snowflake account: my_account database: "{{ env_var('DBT_MY_DATABASE') }}" schema: "{{ env_var('DBT_MY_SCHEMA') }}" ``` 3. Run dbt commands normally. dbt v2 will automatically load the variables from the `.env` file. For example, running `dbtf debug` will show your connection using the values from `.env`: ```shell dbtf debug ... Debugging connection: "authenticator": "my_authenticator", "account": "my_account", "user": "my_user", "database": "my_database", # Loaded from DBT_MY_DATABASE in .env "schema": "my_schema", # Loaded from DBT_MY_SCHEMA in .env ``` note We recommend placing your `.env` file in the project root and running dbt commands from that location because the file is loaded *only* from your current working directory. It doesn't support the `--project-dir` flag or (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` environment variable, and dbt won't search your project root if you're running commands from a different directory location. ##### Precedence order When the same environment variable is defined in multiple places, dbt v2 uses the following precedence order (highest to lowest): 1. Shell environment — Variables set directly in your shell (for example, `export DBT_MY_VAR=value`) 2. `.env` file — Variables defined in the `.env` file in your current working directory This means environment variables set in your shell always override values from the `.env` file. tip Add `.env` to your `.gitignore` file to prevent sensitive credentials from being committed to version control. The `dbtf init` command automatically includes `.env` in the generated `.gitignore` file. For more details on managing environment variables locally, refer to [Configure your local environment](https://docs.getdbt.com/docs/configure-dbt-extension.md#set-environment-variables-locally). --- ### Configure incremental models Learn how to configure and optimize incremental models when developing in dbt. Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. Incremental models are built as tables in your data warehouse. The first time a model is run, the table is built by transforming *all* rows of source data. On subsequent runs, dbt transforms *only* the rows in your source data that you tell dbt to filter for, inserting them into the target table which is the table that has already been built. Often, the rows you filter for on an incremental run will be the rows in your source data that have been created or updated since the last time dbt ran. As such, on each dbt run, your model gets built incrementally. Using an incremental model limits the amount of data that needs to be transformed, vastly reducing the runtime of your transformations. This improves warehouse performance and reduces compute costs. #### Configure incremental materializations Like the other materializations built into dbt, incremental models are defined with `select` statements, with the materialization defined in a config block. ```sql {{ config( materialized='incremental' ) }} select ... ``` To use incremental models, you also need to tell dbt: * How to filter the rows on an incremental run * The unique key of the model (if any) ##### Understand the is\_incremental() macro The `is_incremental()` macro powers incremental materializations. It will return `True` if *all* of the following conditions are met: * The model must already exist as a table in the database * The `full-refresh` flag *is not* passed * The running model is configured with `materialized='incremental'` Note that the SQL in your model needs to be valid whether `is_incremental()` evaluates to `True` or `False`. ##### Filtering rows on an incremental run To tell dbt which rows it should transform on an incremental run, wrap valid SQL that filters for these rows in the `is_incremental()` macro. Often, you'll want to filter for "new" rows, as in, rows that have been created since the last time dbt ran this model. The best way to find the timestamp of the most recent run of this model is by checking the most recent timestamp in your target table. dbt makes it easy to query your target table by using the "[{{ this }}](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md)" variable. Also common is wanting to capture both new and updated records. For updated records, you'll need to [define a unique key](#defining-a-unique-key-optional) to ensure you don't bring in modified records as duplicates. Your `is_incremental()` code will check for rows created *or modified* since the last time dbt ran this model. For example, a model that includes a computationally slow transformation on a column can be built incrementally, as follows: models/stg\_events.sql ```sql {{ config( materialized='incremental' ) }} select *, my_slow_function(my_column) from {{ ref('app_data_events') }} {% if is_incremental() %} -- this filter will only be applied on an incremental run -- (uses >= to include records whose timestamp occurred since the last run of this model) -- (If event_time is NULL or the table is truncated, the condition will always be true and load all records) where event_time >= (select coalesce(max(event_time),'1900-01-01') from {{ this }} ) {% endif %} ``` Optimizing your incremental model For more complex incremental models that make use of Common Table Expressions (CTEs), you should consider the impact of the position of the `is_incremental()` macro on query performance. In some warehouses, filtering your records early can vastly improve the run time of your query! ##### About incremental\_predicates `incremental_predicates` is an advanced use of incremental models, where data volume is large enough to justify additional investments in performance. This config accepts a list of any valid SQL expression(s). dbt does not check the syntax of the SQL statements. This an example of a model configuration in a `yml` file you might expect to see on Snowflake: ```yml models: - name: my_incremental_model config: materialized: incremental unique_key: id # this will affect how the data is stored on disk, and indexed to limit scans cluster_by: ['session_start'] incremental_strategy: merge # this limits the scan of the existing table to the last 7 days of data incremental_predicates: ["DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)"] # `incremental_predicates` accepts a list of SQL statements. # `DBT_INTERNAL_DEST` and `DBT_INTERNAL_SOURCE` are the standard aliases for the target table and temporary table, respectively, during an incremental run using the merge strategy. ``` Alternatively, here are the same configurations configured within a model file: ```sql -- in models/my_incremental_model.sql {{ config( materialized = 'incremental', unique_key = 'id', cluster_by = ['session_start'], incremental_strategy = 'merge', incremental_predicates = [ "DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)" ] ) }} ... ``` This will template (in the `dbt.log` file) a `merge` statement like: ```sql merge into <existing_table> DBT_INTERNAL_DEST from <temp_table_with_new_records> DBT_INTERNAL_SOURCE on -- unique key DBT_INTERNAL_DEST.id = DBT_INTERNAL_SOURCE.id and -- custom predicate: limits data scan in the "old" data / existing table DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date) when matched then update ... when not matched then insert ... ``` Limit the data scan of *upstream* tables within the body of their incremental model SQL, which will limit the amount of "new" data processed/transformed. ```sql with large_source_table as ( select * from {{ ref('large_source_table') }} {% if is_incremental() %} where session_start >= dateadd(day, -3, current_date) {% endif %} ), ... ``` ##### Defining a unique key Defining the optional [`unique_key` parameter](https://docs.getdbt.com/reference/resource-configs/unique_key.md) enables updating existing rows instead of just appending new rows. If new information arrives for an existing `unique_key`, that new information can replace the current information instead of being appended to the table. If a duplicate row arrives, it can be ignored. Refer to [strategy specific configs](https://docs.getdbt.com/docs/build/incremental-strategy.md#strategy-specific-configs) for more options on managing this update behavior, like choosing only specific columns to update. If you don't specify a `unique_key`, most adapters will result in `append`-only behavior, which means dbt inserts all rows returned by the model's SQL into the preexisting target table without regard for whether the rows represent duplicates. The optional `unique_key` parameter specifies a field (or combination of fields) that defines the grain of your model. That is, the field(s) identify a single unique row. You can define `unique_key` in a configuration block at the top of your model, and it can be a single column name or a list of column names. The `unique_key` should be supplied in your model definition as a string representing a single column or a list of single-quoted column names that can be used together, for example, `['col1', 'col2', …])`. Columns used in this way should not contain any nulls, or the incremental model may fail to match rows and generate duplicate rows. Either ensure that each column has no nulls (for example with `coalesce(COLUMN_NAME, 'VALUE_IF_NULL')`) or define a single-column [surrogate key](https://www.getdbt.com/blog/guide-to-surrogate-key) (for example with [`dbt_utils.generate_surrogate_key`](https://github.com/dbt-labs/dbt-utils#generate_surrogate_key-source)). tip In cases where you need multiple columns in combination to uniquely identify each row, we recommend you pass these columns as a list (`unique_key = ['user_id', 'session_number']`), rather than a string expression (`unique_key = 'concat(user_id, session_number)'`). By using the first syntax, which is more universal, dbt can ensure that the columns will be templated into your incremental model materialization in a way that's appropriate to your database. When you pass a list in this way, please ensure that each column does not contain any nulls, or the incremental model run may fail. Alternatively, you can define a single-column [surrogate key](https://www.getdbt.com/blog/guide-to-surrogate-key), for example with [`dbt_utils.generate_surrogate_key`](https://github.com/dbt-labs/dbt-utils#generate_surrogate_key-source). When you define a `unique_key`, you'll see this behavior for each row of "new" data returned by your dbt model: * If the same `unique_key` is present in the "new" and "old" model data, dbt will update/replace the old row with the new row of data. The exact mechanics of how that update/replace takes place will vary depending on your database, [incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md), and [strategy specific configs](https://docs.getdbt.com/docs/build/incremental-strategy.md#strategy-specific-configs). * If the `unique_key` is *not* present in the "old" data, dbt will insert the entire row into the table. Please note that if there's a unique\_key with more than one row in either the existing target table or the new incremental rows, the incremental model may fail depending on your database and [incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md). If you're having issues running an incremental model, it's a good idea to double check that the unique key is truly unique in both your existing database table and your new incremental rows. You can [learn more about surrogate keys here](https://www.getdbt.com/blog/guide-to-surrogate-key). info While common incremental strategies, such as `delete+insert` + `merge`, might use `unique_key`, others don't. For example, the `insert_overwrite` strategy does not use `unique_key`, because it operates on partitions of data rather than individual rows. For more information, see [About incremental\_strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md). ###### `unique_key` example Consider a model that calculates the number of daily active users (DAUs), based on an event stream. As source data arrives, you will want to recalculate the number of DAUs for both the day that dbt last ran, and any days since then. The model would look as follows: models/staging/fct\_daily\_active\_users.sql ```sql {{ config( materialized='incremental', unique_key='date_day' ) }} select date_trunc('day', event_at) as date_day, count(distinct user_id) as daily_active_users from {{ ref('app_data_events') }} {% if is_incremental() %} -- this filter will only be applied on an incremental run -- (uses >= to include records arriving later on the same day as the last run of this model) where date_day >= (select coalesce(max(date_day), '1900-01-01') from {{ this }}) {% endif %} group by 1 ``` Building this model incrementally without the `unique_key` parameter would result in multiple rows in the target table for a single day – one row for each time dbt runs on that day. Instead, the inclusion of the `unique_key` parameter ensures the existing row is updated instead. #### How do I rebuild an incremental model? If your incremental model logic has changed, the transformations on your new rows of data may diverge from the historical transformations, which are stored in your target table. In this case, you should rebuild your incremental model. To force dbt to rebuild the entire incremental model from scratch, use the `--full-refresh` flag on the command line. This flag will cause dbt to drop the existing target table in the database before rebuilding it for all-time. ```bash $ dbt run --full-refresh --select my_incremental_model+ ``` The trailing `+` in the command above will also run all downstream models that depend on `my_incremental_model`. If any of those downstream dependencies are also incremental models, they will be fully refreshed as well. You can optionally use the [`full_refresh config`](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) to set a resource to always or never full-refresh at the project or resource level. If specified as true or false, the `full_refresh` config will take precedence over the presence or absence of the `--full-refresh` flag. For detailed usage instructions, check out the [dbt run](https://docs.getdbt.com/reference/commands/run.md) documentation. #### What if the columns of my incremental model change? Incremental models can be configured to include an optional `on_schema_change` parameter to enable additional control when incremental model columns change. These options enable dbt to continue running incremental models in the presence of schema changes, resulting in fewer `--full-refresh` scenarios and saving query costs. You can configure the `on_schema_change` setting as follows. dbt\_project.yml ```yaml models: +on_schema_change: "sync_all_columns" ``` models/staging/fct\_daily\_active\_users.sql ```sql {{ config( materialized='incremental', unique_key='date_day', on_schema_change='fail' ) }} ``` The possible values for `on_schema_change` are: * `ignore`: Default behavior (see below). * `fail`: Triggers an error message when the source and target schemas diverge * `append_new_columns`: Append new columns to the existing table. Note that this setting does *not* remove columns from the existing table that are not present in the new data. * `sync_all_columns`: Adds any new columns to the existing table, and removes any columns that are now missing. Note that this is *inclusive* of data type changes. On BigQuery, changing column types requires a full table scan; be mindful of the trade-offs when implementing. **Note**: None of the `on_schema_change` behaviors backfill values in old records for newly added columns. If you need to populate those values, we recommend running manual updates, or triggering a `--full-refresh`. `on_schema_change` tracks top-level changes Currently, `on_schema_change` only tracks top-level column changes. It does not track nested column changes. For example, on BigQuery, adding, removing, or modifying a nested column will not trigger a schema change, even if `on_schema_change` is set appropriately. ##### Default behavior This is the behavior of `on_schema_change: ignore`, which is set by default. If you add a column to your incremental model, and execute a `dbt run`, this column will *not* appear in your target table. If you remove a column from your incremental model and execute a `dbt run`, `dbt run` will fail. Instead, whenever the logic of your incremental changes, execute a full-refresh run of both your incremental model and any downstream models. --- ### Configure your local environment Local development Whether you currently use dbt platform or self-host dbt, follow the instructions on this page to: * [Prepare your local setup](#prepare-your-local-setup) * [Set environment variables locally](#set-environment-variables-locally) * [Configure the dbt extension](#configure-the-dbt-extension) If you're new to dbt or getting started with a new project, you can skip this page and check out our [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md?step=1) to get started with the dbt extension. The steps differ slightly depending on whether you use dbt platform or self host dbt. * dbt platform — You'll mirror your dbt platform environment locally to unlock powerful features like Mesh, deferral, and so on. If your project has environment variables, you'll also set them locally to leverage the VS Code extension's features. * Self-hosted — When you self-host dbt, you'll most likely already have a local setup and environment variables. Use this page to confirm that your existing local setup and environment variables work seamlessly with the VS Code extension. #### Prerequisites * Downloaded and installed the dbt VS Code extension * Basic understanding of [Git workflows](https://docs.getdbt.com/docs/platform/git/version-control-basics.md) and [dbt project structure](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [Developer or analyst license](https://www.getdbt.com/pricing)\* if you're using dbt platform. Note, the [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. #### Prepare your local setup In this section, we'll walk you through the steps to prepare your local setup for the dbt VS Code extension. If you're a dbt platform user that installed the VS Code extension, follow these steps. If you're a self-hosted user, you most likely already have a local setup and environment variables but can confirm using these steps. 1. [Clone](https://code.visualstudio.com/docs/sourcecontrol/overview#_cloning-a-repository) your dbt project repository from your Git provider to your local machine. If you use dbt platform, clone the same repo connected to your project. 2. Ensure you have a dbt [`profiles.yml` file](https://docs.getdbt.com/docs/local/profiles.yml.md). This file defines your data warehouse connection. If you don't have one, run `dbt init` in the terminal to configure your adapter. 3. Validate your `profiles.yml` and project configuration by running `dbt debug`. 4. Add a `dbt_cloud.yml` file from the dbt platform Account settings: * Navigate to **Your profile** -> **VS Code Extension** -> **Download credentials**. * Place the downloaded file in your `~/.dbt/` directory. This registers and connects the extension to dbt platform and enables platform features such as Mesh and deferral. Refer to [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) for the file's structure and required fields. * Check the `project-id` in your [`dbt_project.yml` `dbt-cloud` block](https://docs.getdbt.com/reference/dbt_cloud.yml.md#the-dbt-cloud-block-in-dbt_projectyml) matches the project you're working on. 5. Confirm connection from your workstation (like running `dbt debug` in the terminal). Your local computer connects directly to your data warehouse and Git. * dbt platform users: Ensure your laptop/VPN is allowed; dbt platform IPs no longer apply. Check with your admin if you have any issues. * dbt v1 users: This has likely already been configured. 6. (Optional) If your project uses environment variables, [find them](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables) in the dbt platform and [set them](#set-environment-variables-locally) in VS Code or Cursor. * dbt platform users: Copy any environment variables from **Deploy → Environments → Environment variables** tab in dbt platform. Masked secrets are hidden. Work with your admin to get those values. [![Environment variables tab](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/navigate-to-env-vars.png?v=2> "Environment variables tab")](#)Environment variables tab #### Set environment variables locally Environment variables are used for authentication and configuration. This section is most relevant for [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) and dbt platform users who have environment variables configured as part of their workspace setup. If you're using dbt locally, you can also install the VS Code extension and use its features and actions — you just may not need to configure these variables unless your setup specifically requires them. The following table shows the different options and when to use them: | Location | Affects | Session state | When to use | | --------------------------------------------------------------------------- | --------------------- | ----------------------------- | ------------------------------------------------------------------------ | | [**Shell profile**](#configure-at-the-os-or-shell-level) | Terminal | ✅ Permanent | Variables remain active globally and available across terminal sessions. | | [**VS Code/Cursor settings**](#configure-in-the-vs-code-extension-settings) | Extension menus + LSP | ✅ Per VS Code/Cursor profile | Editor-only workflows using the extension menu actions. | | [**Terminal session**](#configure-in-the-terminal-session) | Current terminal only | ❌ Temporary | One off testing. | tip If you want to use both the VS Code extension menus and terminal to run dbt commands, define your variables in the `shell` profile and VS Code/Cursor settings so they remain active in the terminal globally and in VS Code/Cursor. ##### Configure at the OS or shell level Define variables once at the OS or shell level to ensure they're available to all terminal sessions. Even if you close a terminal window, the variables will remain available to you. ##### Mac / Linux 1. Open your shell configuration file in a text editor using the following commands (If the file does not exist, create it using a text editor using `vi ~/.zshrc` or `vi ~/.bashrc`): ```bash open -e ~/.zshrc ## for zsh (macOS) nano ~/.bashrc ## for bash (Linux or older macOS) ``` 2. A file will open up and you can add your environment variables to the file. For example: * For zsh (macOS): ```bash ## ~/.zshrc export DBT_ENV_VAR1="my_value" export DBT_ENV_VAR2="another_value" ``` * For bash (Linux or older macOS): ```bash ## ~/.bashrc or ~/.bash_profile export DBT_ENV_VAR1="my_value" export DBT_ENV_VAR2="another_value" ``` 3. Save the file. 4. Start a new shell session by closing and reopening the terminal or running `source ~/.zshrc` or `source ~/.bashrc` in the terminal. 5. Verify the variables by running `echo $DBT_ENV_VAR1` and `echo $DBT_ENV_VAR2` in the terminal. <br /> If you see the value printed back in the terminal, you're all set! These variables will now be available: * In all future terminal sessions * For all dbt commands run in the terminal ##### Windows There are two ways to create persistent environment variables on Windows: through PowerShell or the System Properties. The following steps will explain how to configure environment variables using PowerShell. **PowerShell** 1. Run the following commands in PowerShell: ```powershell [Environment]::SetEnvironmentVariable("DBT_ENV_VAR1","my_value","User") [Environment]::SetEnvironmentVariable("DBT_ENV_VAR2","another_value","User") ``` 1. This saves the variables permanently for your user account. To make them available system-wide for all users, replace "User" with "Machine" (requires admin rights). 2. Then, restart VS Code or select **Developer: Reload Window** for changes to take effect. 3. Verify the changes by running `echo $DBT_ENV_VAR1` and `echo $DBT_ENV_VAR2` in the terminal. **System properties (Environment Variables)** 1. Press **Start** → search for **Environment Variables** → open **Edit the system environment variables**. 2. From the **Advanced** tab of the System Properties, click **Environment Variables…**. 3. Under **User variables**, click **New…**. 4. Add the variables and values. For example: * Variable name: `DBT_ENV_VAR1` * Variable value: `my_value` 5. Repeat for any others, then click **OK**. 6. Restart VS Code or Cursor. 7. Verify the changes by running `echo $DBT_ENV_VAR1` and `echo $DBT_ENV_VAR2` in the terminal. ###### About `.env` file support [dbt](https://docs.getdbt.com/docs/local/install-dbt.md) and the dbt VS Code extension can automatically read environment variables from a `.env` file in your current working directory (the folder you `cd` into and run dbt commands from in your terminal), if one exists. The environment variables you define in the `.env` file are available when running dbt commands in the terminal and when using the extension's menu actions. Here are some considerations when defining environment variables in the `.env` file: * The `.env` file provides a convenient way to set environment variables that work across both the CLI and the VS Code extension. * We recommend placing your `.env` file in the project root and running dbt commands from that location because the file is loaded *only* from your current working directory. It doesn't support the `--project-dir` flag or (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` environment variable, and dbt won't search your project root if you're running commands from a different directory location. * Add `.env` to your `.gitignore` file to prevent sensitive credentials from being committed to your repository. * Environment variables set directly in your shell (such as `export DBT_ENV_VAR=value`) take precedence over values defined in the `.env` file. ##### Configure in the VS Code extension settings To use the dbt extension menu actions/buttons, you can configure environment variables directly in the [VS Code User Settings](vscode://settings/dbt.environmentVariables) interface or in a `.env` file in your current working directory. This includes both your custom variables and any automatic [dbt platform variables](https://docs.getdbt.com/docs/build/environment-variables.md) (like `DBT_CLOUD_ENVIRONMENT_NAME`) that your project depends on. * Configure variables in the VS Code **User Settings** or in a `.env` file to have them recognized by the extension. For example, when using LSP -powered features, "Show build menu," and more. * VS Code does not inherit variables set by the VS Code terminal or external shells. * The terminal uses system environmental variables, and does not inherit variables set in the dbt VS Code extension config. For example, running a dbt command in the terminal won't fetch or use the dbt VS Code extension variables. To configure environment variables in VS Code/Cursor: ##### Open User Settings 1. Open the [Command Palette](https://code.visualstudio.com/docs/configure/settings#_user-settings) (Cmd + Shift + P for Mac, Ctrl + Shift + P for Windows/Linux). 2. Then select either **Preferences: Open User Settings** in the dropdown menu. 3. Open the [VS Code user settings page](vscode://settings/dbt.environmentVariables). 4. Search for `dbt.environmentVariables`. 5. In the **dbt:Environment Variables** section, add your item and value for the environment variables. 6. Click **Ok** to save the changes. 7. Reload the VS Code extension to apply the changes. Open the Command Palette and select **Developer: Reload Window**. 8. Verify the changes by running a dbt command and checking the output. ##### Open .env file 1. Create a `.env` file in your current working directory (typically at the root level of your dbt project, same level as your `dbt_project.yml` file). 2. Add your environment variables to the file. For example: ```env DBT_ENV_VAR1=my_value DBT_ENV_VAR2=another_value ``` 3. Save the file. 4. Reload the VS Code extension to apply the changes. 5. Verify the changes by running a dbt command using the extension menu button on the top right corner and checking the output. For example, running `dbtf debug` will show your connection using the values from `.env`: ```shell dbtf debug ... Debugging connection: "authenticator": "my_authenticator", "account": "my_account", "user": "my_user", "database": "my_database", # Loaded from DBT_MY_DATABASE in .env "schema": "my_schema", # Loaded from DBT_MY_SCHEMA in .env ``` ##### Configure in the terminal session Configure environment variables in the terminal session using the `export` command. Something to keep in mind: * Doing so will make variables visible to commands that run in that terminal session only. * It lasts only for the current session and opening a new terminal will lose the values. * The built-in dbt VS Code extension buttons and menus will not pick these up. To configure environment variables in the terminal session: 1. Run the following command in the terminal, replacing `DBT_ENV_VAR1` and `test1` with your own variable and value. ##### Mac / Linux ```bash export DBT_ENV_VAR1=test1 ``` ##### Windows Cmd Refer to [Microsoft's documentation](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/set_1) for more information on the `set` command. ```bash set DBT_ENV_VAR1=test1 ``` ##### Windows PowerShell Refer to [Microsoft's documentation](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-7.5#use-the-variable-syntax) for more information on the `$env:` syntax. ```bash $env:DBT_ENV_VAR1 = "test1" ``` 2. Verify the changes by running a dbt command and checking the output. #### dbt extension settings After installing the dbt extension and configuring your local setup, you may want to configure it to better fit your development workflow: 1. Open the VS Code settings by pressing `Ctrl+,` (Windows/Linux) or `Cmd+,` (Mac). 2. Search for `dbt`. On this page, you can adjust the extension's configuration options to fit your needs. [![dbt extension settings within the VS Code settings.](/img/docs/extension/dbt-extension-settings.png?v=2 "dbt extension settings within the VS Code settings.")](#)dbt extension settings within the VS Code settings. The following settings are the most relevant when you install or manage dbt yourself. Most users never need to set these because the extension can download and manage v2 automatically. | Setting | Description | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt.fusionPath` | Path to the v2 binary. The extension invokes the language server through this binary (`dbt-fusion lsp`). Set this when you install v2 manually — for example, in an [air-gapped environment](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#verify-binaries-for-manual-and-air-gapped-installs) — instead of letting the extension download it. | | `dbt.badReleasesManifestPath` | Path to a local copy of the [known-bad-releases manifest](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases). Use this if you don't have outbound network access and distribute the manifest alongside your binary bundle (for example, air-gapped installations). | | `dbt.environmentVariables` | Environment variables the extension passes to dbt. Refer to [Configure in the VS Code extension settings](#configure-in-the-vs-code-extension-settings). | Upgrading from a separate language server binary Earlier extension versions used separate `dbt.cliPath` and `dbt.lspPath` settings for two distinct binaries. Current versions use a single `dbt.fusionPath` because the CLI and language server ship as one binary in dbt v2. If you previously configured `dbt.lspPath`, migrate that path to `dbt.fusionPath`. Refer to [Version compatibility](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) for details. #### File associations and other extensions The dbt extension doesn't depend on your `.sql` [file associations](https://code.visualstudio.com/docs/languages/identifiers). It activates on your `dbt_project.yml`, so LSP features — like autocomplete, go-to-definition, and database-aware IntelliSense — work whether your files are set to `sql` or `jinja-sql`. The only difference between those two is syntax highlighting and the file icon. Where the association does matter is for *other* extensions. Any extension that claims the `sql` file type — whether that's Snowflake, SQL Server, or another database tool — can intercept your dbt files and override the IntelliSense you'd otherwise get from the dbt extension. For example, the Snowflake extension only picks up files associated with `snowflake-sql`. That makes `snowflake-sql` the wrong association for your dbt files — it hands them to the Snowflake extension instead of treating them as dbt files. If you have multiple SQL-aware extensions installed, it's worth checking which one is handling your `.sql` files to avoid unexpected behavior. If you want to use both extensions, keep your ad hoc query files in a separate, gitignored directory and scope the Snowflake association to just that path: ```json { "files.associations": { "**/snowflake_sandbox/*.sql": "snowflake-sql", "*.sql": "sql" } } ``` #### Next steps Now that you've configured your local environment, you can start using the dbt extension to streamline your dbt development workflows. Check out the following resources to get started: * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [dbt extension features](https://docs.getdbt.com/docs/dbt-extension-features.md) * [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) --- ### Configuring deferral in dbt State Login required | Usage-based By default, dbt State defers to your production environment. You only need to configure this if you want to change that behavior: * **dbt platform**: To defer to an environment other than the default (for example, staging), add `defer-env-id` to the `dbt-cloud` block in `dbt_project.yml`. Refer to [Configure the dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) for more information. dbt\_project.yml ```yaml dbt-cloud: project-id: <your-project-id> defer-env-id: <your-environment-id> ``` note `defer-env-id` is manifest-based. If you set it, [dbt State-powered `state:*` selectors](#dbt-state-powered-state-selectors) are disabled. Refer to that section's requirements for more information. * **Self-managed deployments**: If you can't access a production or deployment manifest, you can set [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) in `profiles.yml` for best-effort auto-deferral. Note that this approach has known limitations; refer to [Caveats to dbt State without a manifest](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md#caveats-to-dbt-state-without-a-manifest). profiles.yml ```yaml my_project: outputs: uat: type: snowflake # ... connection settings defer_to_target: staging ``` In self-managed deployments, you can use [dbt State-powered `state:*` selectors](#dbt-state-powered-state-selectors), which compare each node against its own last execution rather than a single job's `manifest.json`. To use this feature, connect your project to the dbt platform. You can also pass `--state` or `--defer-state` to explicitly point dbt State to a specific `manifest.json`. note If you've overridden `generate_*_name()` macros with runtime values (such as environment variables, file paths, or dates), provide a `manifest.json` file so dbt State can locate objects correctly. Without one, it infers object locations from your macros and profile target, which may be incorrect in these cases. Refer to [Caveats to dbt State without a manifest](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md#caveats-to-dbt-state-without-a-manifest). #### dbt State-powered `state:*` selectors [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") When dbt State is enabled, [`state:*` selectors](https://docs.getdbt.com/reference/node-selection/methods.md#state) can use dbt State as their comparison source instead of a single `manifest.json` from the most recent job run. Each node (models, snapshots, seeds, and tests) is compared against its own last execution in the [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) environment (default: `prod`), so a node is only selected as modified if its own logic changed, no matter which job last ran it. This feature is for **self-managed deployments**, meaning you orchestrate dbt runs yourself (for example, with Airflow or GitHub Actions) rather than with dbt platform jobs. You still authenticate to the dbt platform for dbt State using `project-id`. ##### Prerequisites * dbt State is enabled for your project. * `project-id` is set in the `dbt-cloud` block in `dbt_project.yml`: dbt\_project.yml ```yaml dbt-cloud: project-id: <your-project-id> ``` note When orchestrating with the dbt platform, each job produces a `manifest.json`, so dbt State uses that for `state:*` comparisons instead of per-node execution history. #### Specify your project or org If you have multiple projects or orgs that use dbt State, configure the `dbt-cloud` block in `dbt_project.yml` so dbt State knows which one to use: * **For dbt platform users with multiple projects**: Add `project-id` to identify which project dbt State should use. dbt\_project.yml ```yaml dbt-cloud: project-id: <your-project-id> ``` * **For self-managed deployments with multiple dbt State orgs**: Add `state-org-id` to identify which org dbt State should use. dbt\_project.yml ```yaml dbt-cloud: state-org-id: <your-org-id> ``` #### Related docs * [Defer in dbt](https://docs.getdbt.com/docs/platform/about-defer.md) * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### Connect dbt MCP server to dbt platform dbt platform This quickstart uses the local MCP server: it runs on your machine using `uvx dbt-mcp`, connects to your dbt platform for Semantic Layer, Discovery, and SQL, and optionally runs local dbt. For self-hosted CLI only (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) or [Run self-hosted dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). To configure or disable specific tools, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) * A [dbt platform account](https://www.getdbt.com/signup) * For OAuth connections: * MCP OAuth is available for Starter, Enterprise, and Enterprise+ plans. #### Step 1: Choose your auth method and configure ##### OAuth *MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* OAuth is the fastest setup for dbt platform accounts, no tokens to copy or manage. A browser window opens to authenticate the first time you connect. For OAuth *without* a self-hosted installation, use the [remote MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md). Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. If your client does not support OAuth or you need token-based access, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication). Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. ###### Find your Access URL 1. Log in to your dbt platform account. 2. Go to **Account settings** and copy your **Access URL** (for example, `abc123.us1.dbt.com`). Multi-cell and DBT\_HOST format * The `DBT_HOST` field accepts both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com`. * If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ###### Add the config to your MCP client ##### Claude Desktop **Option A: Quick install (recommended)** 1. Go to the [latest dbt MCP release](https://github.com/dbt-labs/dbt-mcp/releases/latest) and download `dbt-mcp.mcpb`. 2. Double-click the file to open it in Claude Desktop. 3. Enter your **Access URL** as the dbt platform Host. 4. Enable the server. **Option B: Manual config** 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing `YOUR-ACCESS-URL` with your Access URL: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing `YOUR-ACCESS-URL` with your Access URL: ```shell claude mcp add dbt \ -e DBT_HOST=YOUR-ACCESS-URL \ -- uvx dbt-mcp ``` For example, if your Access URL is `abc123.us1.dbt.com`: ```shell claude mcp add dbt \ -e DBT_HOST=abc123.us1.dbt.com \ -- uvx dbt-mcp ``` ##### Cursor Click a link below with Cursor open to auto-configure, then replace the placeholder with your Access URL: * [dbt platform only (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJESVNBQkxFX0RCVF9DTEkiOiJ0cnVlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features only, no CLI * [dbt platform + CLI (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJEQlRfUFJPSkVDVF9ESVIiOiIvcGF0aC90by9wcm9qZWN0IiwiREJUX1BBVEgiOiJwYXRoL3RvL2RidC9leGVjdXRhYmxlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features + self-hosted dbt CLI commands After clicking, replace `YOUR-ACCESS-URL` with your actual Access URL (for example, `abc123.us1.dbt.com`) and save. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` Replace `YOUR-ACCESS-URL` with your Access URL (for example, `abc123.us1.dbt.com`) and save. ###### Optional: Add self-hosted dbt CLI commands To also run dbt platform CLI commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. ##### Tokens Token-based auth gives you more control and is better for shared or team setups. You'll need a service token or Personal Access Token (PAT). Which token should I use? * **PAT (Personal Access Token):** Required if you want to use `execute_sql`. Tied to your user account. * **Service token:** Works for all other platform toolsets. Better for shared or team setups. See [Choosing an auth method](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#choose-your-auth-method) for full guidance. ##### Find your paths and IDs You need the following values. See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. | Variable | Where to find it | | ----------------- | -------------------------------------------------------------------------- | | `DBT_HOST` | Your dbt platform hostname, found in **Account settings** → **Access URL** | | `DBT_TOKEN` | A service token or PAT from **Account settings** → **API tokens** | | `DBT_PROD_ENV_ID` | Your production environment ID, found in **Deploy** → **Environments** | | `DBT_DEV_ENV_ID` | Your development environment ID (required for `execute_sql`) | | `DBT_USER_ID` | Your numeric user ID (required for `execute_sql`) | | `DBT_ACCOUNT_ID` | Your account ID (required for Admin API tools) | Use values only, not full URLs These variables expect hostnames or numeric IDs — not full URLs: ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` Multi-cell accounts If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ##### Add the config to your MCP client Use the configuration below, replacing the placeholder values with your IDs from above. Include only the variables you need: ##### Claude Desktop 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing the placeholders with your actual values: ```bash claude mcp add dbt \ -e DBT_HOST=cloud.getdbt.com \ -e DBT_TOKEN=your-token-here \ -e DBT_PROD_ENV_ID=12345 \ -- uvx dbt-mcp ``` Add `-e DBT_DEV_ENV_ID=...` and `-e DBT_USER_ID=...` if you use `execute_sql`; add `-e DBT_ACCOUNT_ID=...` for Admin API. ##### Cursor 1. In Cursor, open **Settings** → **MCP** → **Edit config** (or your config file). 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save the configuration. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 4. Save `mcp.json` and restart VS Code.  Optional: add self-hosted dbt CLI commands To also run dbt commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. #### Step 2: Authenticate ##### OAuth The first time you connect, dbt MCP opens a browser window to complete OAuth. After signing in, your session is saved and future connections are automatic. If authentication doesn't start, close your client and run: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` Then restart your client. ##### Tokens No additional authentication step is needed — your token is already in the configuration. The server connects automatically when your MCP client starts. #### Step 3: Test your setup Ask your AI assistant a data-related question (for example, *"What models are in my dbt project?"* or *"What metrics are defined in my Semantic Layer?"*). If dbt MCP is working, the response will use your dbt metadata. #### What's available With the platform setup, your AI assistant can use: * Semantic Layer queries * Metadata Discovery (model lineage, test results, source freshness) * Admin API (trigger jobs, list runs, get artifacts) * SQL execution and text-to-SQL (requires a [PAT](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md)) * All dbt commands if you added `DBT_PROJECT_DIR` and `DBT_PATH` For the complete tool list, see [Available tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md). Looking for self-hosted dbt CLI only? If you only need to run dbt commands locally (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md). #### Troubleshooting Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). execute\_sql tool not working **Symptoms:** The `execute_sql` tool returns an authentication error or is unavailable. **Cause:** How you fix this depends on how you connected: * **Remote MCP with OAuth:** Sign in through your MCP client when prompted. You don't need a personal access token. * **Remote MCP with token-based auth:** You need a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. You also need `x-dbt-dev-environment-id` and `x-dbt-user-id` headers. The user ID must match the user who created the PAT. * **Local MCP:** Set `DBT_TOKEN` to a personal access token (PAT). Service tokens won't work. **Solution:** 1. If you use OAuth for remote MCP, confirm you completed sign-in and consent in your MCP client. 2. If you use token-based auth or local MCP, create a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) in **Account settings** → **API tokens** → **Personal tokens**. 3. For token-based remote MCP, add the PAT to the `Authorization` header as `Token YOUR_PAT`. For local MCP, set the PAT as your `DBT_TOKEN` value. 4. Also ensure `DBT_DEV_ENV_ID` and `DBT_USER_ID` are set (local MCP) or `x-dbt-dev-environment-id` and `x-dbt-user-id` headers are set (remote MCP). Refer to [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. Toolset unavailable or showing as disabled **Symptoms:** A toolset (Semantic Layer, Discovery, Admin API) is not available in your AI client even though you've configured credentials. **Cause:** Either the required variables are missing, or the toolset has been explicitly disabled. **Solution:** 1. Check that all required variables for the toolset are set — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). 2. Check whether you have any `DISABLE_*` variables set to `true` that might be turning off the toolset. 3. If you're using enable mode (`DBT_MCP_ENABLE_*`), make sure the toolset you need is listed. 4. Set `DBT_MCP_LOG_LEVEL=DEBUG` to see which toolsets are active at startup. Pasting full URLs instead of IDs **Symptoms:** Authentication errors, unexpected behavior, or the server failing to connect to the right environment. **Cause:** Environment variables like `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_ACCOUNT_ID` expect numeric integers, not full browser URLs. **Solution:** ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. Multi-cell or static subdomain account connection issues **Symptoms:** Connection errors when your account URL includes a prefix (for example, `abc123.us1.dbt.com`). **Solution (as of v1.14.0):** Set `DBT_HOST` to the full hostname including the prefix. If you're using PAT-based auth, also set `DBT_ACCOUNT_ID`. ```bash # ✅ Correct DBT_HOST=abc123.us1.dbt.com DBT_ACCOUNT_ID=12345 # required for PAT-based auth ``` You no longer need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. If you have these set from an older configuration, remove them. For all troubleshooting topics, see [dbt MCP FAQs and troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). #### Next steps * Run dbt commands locally: see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) * Configure specific toolsets: see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) * Understand toolset requirements: see [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance) --- ### Connect to the remote dbt MCP server dbt platform The remote MCP server connects to dbt platform using HTTP. No self-hosted installation is required — you configure your MCP client with a URL and headers instead of running `uvx dbt-mcp`. [![Remote dbt MCP server architecture](/img/mcp/remote-dbt-mcp.jpg?v=2 "Remote dbt MCP server architecture")](#)Remote dbt MCP server architecture #### When to use remote MCP Remote MCP is a good fit when: * You don't want to or can't install software (`uvx`, dbt-mcp) on your machine. * Your use case is *consumption-based*: querying metrics, exploring metadata, viewing lineage, or running SQL on the platform. * You need Semantic Layer, Administrative, and Discovery APIs access without a local dbt project. Self-hosted development requires self-hosted MCP Self-hosted development and agentic workflows (for example, running dbt commands like `dbt run` or `dbt build` from your AI assistant) require the **self-hosted** MCP server. Remote MCP does not support the self-hosted dbt v1 or dbt v2 CLI or local project access. Use [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) or [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) for those workflows. #### Set up remote MCP Follow these steps to set up the remote MCP server. ##### 1. Manage AI features (optional) AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Remote MCP works without AI features enabled. Only `text_to_sql` requires them; when disabled, this tool is hidden and all other tools work as usual. ##### 2. Get your credentials Obtain the following information from dbt platform: * **dbt platform host**: Form the URL as `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/` (for example, `https://cloud.getdbt.com/api/ai/v1/mcp/`). For multi-cell accounts, the host is in the format `ACCOUNT_PREFIX.us1.dbt.com`. Refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). * **Production environment ID**: From **Orchestration** → **Environments** in dbt platform. You will use it as the `x-dbt-prod-environment-id` header (token-based setup only). Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **Token** — PAT or service token with Semantic Layer and Developer permissions (token-based setup only). * **If you use `execute_sql` with token-based auth:** You must use a PAT, plus your development environment ID and user ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. With OAuth, you only need your MCP URL. ##### 3. Choose authentication: OAuth or tokens | Type | Info | | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **[OAuth (remote)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp)** | No API tokens in your client config. Requires an OAuth-capable MCP client. Supports `execute_sql`.<br /><br />Available in public beta for Starter, Enterprise, and Enterprise+ accounts. | | **Token-based** | PAT or service token in the `Authorization` header. Works with any client and is required for shared/CI setups. For `execute_sql`, you must use a PAT (service tokens do not work). | info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. ##### 4. Get your MCP URL and IDs You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Depending on your auth method, you may also need: * **Production environment ID**: From **Orchestration** → **Environments** in dbt platform. Used as the `x-dbt-prod-environment-id` header for token-based setup. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **Token** — PAT or service token with Semantic Layer and Developer permissions (token-based setup only). * **If you use `execute_sql` with token-based auth:** You must use a PAT, plus your development environment ID and user ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. With OAuth, you only need your MCP URL. Remote MCP doesn't require AI features You don't need [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled to use remote MCP. If an admin turns AI features off, [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features) is the only tool hidden and every other tool keeps working. `text_to_sql` is also the only tool that consumes your dbt Copilot action allotment. Other MCP tools don't. If your account runs out of dbt Copilot actions, `text_to_sql` will not work until the limit resets. Other remote MCP tools keep working. If you need help, contact your account manager. ##### 5. Configure your MCP client Configure your MCP client with the MCP URL from the previous step. If you use token-based authentication, also add the required headers. ##### OAuth *Available for Starter, Enterprise, and Enterprise+ accounts* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). Configure your client with the MCP URL from the previous step. On first connect, your client opens a browser for sign-in and consent. You don't need a personal access token or extra headers to use `execute_sql`. Some tools (like Claude Desktop) let you add dbt as a custom connector through their UI instead of editing a config file: The following steps show how to connect dbt as a custom connector in Claude Desktop. The exact UI varies by tool, but the flow is the same: add a custom connector with your MCP URL, complete the OAuth consent flow, then connect. 1. In your AI tool, go to its connector settings and choose to add a custom connector (in Claude Desktop, go to **Chat → Customize → Connectors**, then click **Add custom connector**). 2. Enter a name (for example, `dbt`) and paste your dbt platform MCP URL (for example, `https://abc123.us1.dbt.com/api/ai/v1/mcp`), then click **Add**. [![Custom connector dialog showing the dbt MCP URL](/img/docs/dbt-cloud/oauth-add-custom-connector.png?v=2 "Custom connector dialog showing the dbt MCP URL")](#)Custom connector dialog showing the dbt MCP URL 3. Click **Connect**. The tool redirects you to dbt to complete the OAuth consent flow, where you can approve or deny individual [scopes](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent). [![OAuth consent screen showing requested scopes and project access](/img/docs/dbt-cloud/oauth-consent-screen.png?v=2 "OAuth consent screen showing requested scopes and project access")](#)OAuth consent screen showing requested scopes and project access 4. After you approve, the connector is added to the **Custom connectors** table and shows as connected. [![Adding a custom dbt connector in an AI tool's connector settings](/img/docs/dbt-cloud/oauth-connectors-page.png?v=2 "Adding a custom dbt connector in an AI tool's connector settings")](#)Adding a custom dbt connector in an AI tool's connector settings 5. That's it 🎉! Ask your tool a data question like *"What is the total revenue for the last 30 days?"* to confirm the connection. For tools configured with a JSON file, use the tab that matches your client: ##### Claude Code Add this to `.mcp.json` at your project root: ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Cursor Add this to `.cursor/mcp.json` (or use the Cursor deeplink in [Integrate Cursor with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md#set-up-with-remote-dbt-mcp-server)): ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### VS Code Add this to `mcp.json` (run **MCP: Open Workspace Folder MCP Configuration** from the command palette). VS Code uses the `servers` key, not `mcpServers`: ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Token-based Set the server `url` to `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/` and add the required headers: * **Required:** `Authorization` (value `Token YOUR_TOKEN` or `Bearer YOUR_TOKEN`), `x-dbt-prod-environment-id` * **For `execute_sql` with token-based auth or dbt v2 tools:** Also add `x-dbt-dev-environment-id` and `x-dbt-user-id` * Use numeric IDs in headers, not full URLs copied from your browser. ##### Claude Code ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Cursor ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### VS Code VS Code uses the `servers` key, not `mcpServers`: ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Gemini Gemini uses the `httpUrl` key instead of `url`: ```json { "mcpServers": { "dbt": { "httpUrl": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` * For the complete list of headers, Cursor and other client examples, and optional headers, refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md). * For self-hosted MCP, configuration uses environment variables; check out the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) for more information. Once you have configured your MCP client, you can test your setup by asking your AI assistant a data-related question (for example, *"What models are in my dbt project?"* or *"What metrics are defined in my Semantic Layer?"*). If dbt MCP is working, the response will use your dbt metadata. --- ### Connection profiles Local development When you invoke dbt from the command line, dbt parses your `dbt_project.yml` and obtains the `profile` name, which dbt needs to connect to your data warehouse. dbt\_project.yml ```yaml # Example dbt_project.yml file name: 'jaffle_shop' profile: 'jaffle_shop' ... ``` dbt then checks your `profiles.yml` file for a profile with the same name. A profile contains all the details required to connect to your data warehouse. dbt will search the current working directory for the `profiles.yml` file and will default to the `~/.dbt/` directory if not found. This file generally lives outside of your dbt project to avoid sensitive credentials being checked in to version control, but `profiles.yml` can be safely checked in when [using environment variables](#advanced-using-environment-variables) to load sensitive credentials. \~/.dbt/profiles.yml ```yaml # example profiles.yml file jaffle_shop: target: dev outputs: dev: type: postgres host: localhost user: alice password: <password> port: 5432 dbname: jaffle_shop schema: dbt_alice threads: 4 prod: # additional prod target type: postgres host: prod.db.example.com user: alice password: <prod_password> port: 5432 dbname: jaffle_shop schema: analytics threads: 8 ``` To add an additional target (like `prod`) to your existing `profiles.yml`, you can add another entry under the `outputs` key. #### The `env_var` function The `env_var` function can be used to incorporate environment variables from the system into your dbt project. You can use the `env_var` function in your `profiles.yml` file, the `dbt_project.yml` file, the `sources.yml` file, your `schema.yml` files, and in model `.sql` files. Essentially, `env_var` is available anywhere dbt processes Jinja code. When used in a `profiles.yml` file (to avoid putting credentials on a server), it can be used like this: profiles.yml ```yaml profile: target: prod outputs: prod: type: postgres host: 127.0.0.1 # IMPORTANT: Make sure to quote the entire Jinja string here user: "{{ env_var('DBT_USER') }}" password: "{{ env_var('DBT_PASSWORD') }}" .... ``` #### About the `profiles.yml` file In your `profiles.yml` file, you can store as many profiles as you need. Typically, you would have one profile for each warehouse you use. Most organizations only have one profile. #### About profiles A profile consists of *targets*, and a specified *default target*. Each *target* specifies the type of warehouse you are connecting to, the credentials to connect to the warehouse, and some dbt-specific configurations. The credentials you need to provide in your target varies across warehouses — sample profiles for each supported warehouse are available in the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) section. **Pro Tip:** You may need to surround your password in quotes if it contains special characters. More details [here](https://stackoverflow.com/a/37015689/10415173). #### Setting up your profile To set up your profile, copy the correct sample profile for your warehouse into your `profiles.yml` file and update the details as follows: * Profile name: Replace the name of the profile with a sensible name – it’s often a good idea to use the name of your organization. Make sure that this is the same name as the `profile` indicated in your `dbt_project.yml` file. * `target`: This is the default target your dbt project will use. It must be one of the targets you define in your profile. Commonly it is set to `dev`. * Populating your target: * `type`: The type of data warehouse you are connecting to * Warehouse credentials: Get these from your database administrator if you don’t already have them. Remember that user credentials are very sensitive information that should not be shared. * `schema`: The default schema that dbt will build objects in. * `threads`: The number of threads the dbt project will run on. You can find more information on which values to use in your targets below. Use the [debug](https://docs.getdbt.com/reference/dbt-jinja-functions/debug-method.md) command to validate your warehouse connection. Run `dbt debug` from within a dbt project to test your connection. #### Understanding targets in profiles dbt supports multiple targets within one profile to encourage the use of separate development and production environments as discussed in [dbt environments](https://docs.getdbt.com/docs/local/dbt-environments.md). A typical profile for an analyst using dbt locally will have a target named `dev`, and have this set as the default. You may also have a `prod` target within your profile, which creates the objects in your production schema. However, since it's often desirable to perform production runs on a schedule, we recommend deploying your dbt project to a separate machine other than your local machine. Most dbt users only have a `dev` target in their profile on their local machine. If you do have multiple targets in your profile, and want to use a target other than the default, you can do this using the `--target` flag when running a dbt command. For example, to run against your `prod` target instead of the default `dev` target: ```bash dbt run --target prod ``` You can use the `--target` flag with any dbt command, such as: ```bash dbt build --target prod dbt test --target dev dbt compile --target qa ``` ##### Overriding profiles and targets When running dbt commands, you can specify which profile and target to use from the CLI using the `--profile` and `--target` [flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). These flags override what’s defined in your `dbt_project.yml` as long as the specified profile and target are already defined in your `profiles.yml` file. To run your dbt project with a different profile or target than the default, you can do so using the following CLI flags: * `--profile` flag — Overrides the profile set in `dbt_project.yml` by pointing to another profile defined in `profiles.yml`. * `--target` flag — Specifies the target within that profile to use (as defined in `profiles.yml`). These flags help when you're working with multiple profiles and targets and want to override defaults without changing your files. ```bash dbt run --profile my-profile-name --target dev ``` In this example, the `dbt run` command will use the `my-profile-name` profile and the `dev` target. #### Understanding warehouse credentials We recommend that each dbt user has their own set of database credentials, including a separate user for production runs of dbt – this helps debug rogue queries, simplifies ownerships of schemas, and improves security. To ensure the user credentials you use in your target allow dbt to run, you will need to ensure the user has appropriate privileges. While the exact privileges needed varies between data warehouses, at a minimum your user must be able to: * read source data * create schemas¹ * read system tables Running dbt without create schema privileges If your user is unable to be granted the privilege to create schemas, your dbt runs should instead target an existing schema that your user has permission to create relations within. #### Understanding target schemas The target schema represents the default schema that dbt will build objects into, and is often used as the differentiator between separate environments within a warehouse. Schemas in BigQuery dbt uses the term "schema" in a target across all supported warehouses for consistency. Note that in the case of BigQuery, a schema is actually a dataset. The schema used for production should be named in a way that makes it clear that it is ready for end-users to use for analysis – we often name this `analytics`. In development, a pattern we’ve found to work well is to name the schema in your `dev` target `dbt_<username>`. Suffixing your name to the schema enables multiple users to develop in dbt, since each user will have their own separate schema for development, so that users will not build over the top of each other, and ensuring that object ownership and permissions are consistent across an entire schema. Note that there’s no need to create your target schema beforehand – dbt will check if the schema already exists when it runs, and create it if it doesn’t. While the target schema represents the default schema that dbt will use, it may make sense to split your models into separate schemas, which can be done by using [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). #### Understanding threads When dbt runs, it creates a directed acyclic graph (DAG) of links between models. The number of threads represents the maximum number of paths through the graph dbt may work on at once – increasing the number of threads can minimize the run time of your project. The default value for threads in user profiles is 4 threads. For more information, check out [using threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). #### Advanced: Customizing a profile directory ##### dbt v2 dbt v2 determines the parent directory for `profiles.yml` using the following precedence: 1. `--profiles-dir` option 2. Project root directory 3. `~/.dbt/` directory Note that dbt v2 doesn't currently support the (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable or setting the `profiles.yml` in the current working directory. ##### dbt v1 dbt v1 determines the parent directory for `profiles.yml` using the following precedence: 1. `--profiles-dir` option 2. (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable 3. current working directory 4. `~/.dbt/` directory To check the expected location of your `profiles.yml` file for your installation of dbt, you can run the following: ```bash $ dbt debug --config-dir To view your profiles.yml file, run: open /Users/alice/.dbt ``` You may want to have your `profiles.yml` file stored in a different directory than `~/.dbt/` – for example, if you are [using environment variables](#advanced-using-environment-variables) to load your credentials, you might choose to include this file in the root directory of your dbt project. Note that the file always needs to be called `profiles.yml`, regardless of which directory it is in. There are multiple ways to direct dbt to a different location for your `profiles.yml` file: ##### 1. Use the `--profiles-dir` option when executing a dbt command This option can be used as follows: ```text $ dbt run --profiles-dir path/to/directory ``` If using this method, the `--profiles-dir` option needs to be provided every time you run a dbt command. ##### 2. Use the (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable to change the default location (dbt v1 only) Setting this environment variable tells dbt v1 to look for your `profiles.yml` file in the specified directory instead of the default location. You can specify this by running: (Applies to dbt v1.11 and later) ```text $ export DBT_ENGINE_PROFILES_DIR=path/to/directory ``` Note: This environment variable isn't supported in dbt v2. #### Advanced: Using environment variables Credentials can be placed directly into the `profiles.yml` file or loaded from environment variables. Using environment variables is especially useful for production deployments of dbt. You can find more information about environment variables [here](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md). #### Related docs * [About `profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) --- ### Consume metrics from your Semantic Layer dbt platform | Starter, Enterprise, Enterprise+ After [deploying](https://docs.getdbt.com/docs/use-dbt-semantic-layer/deploy-sl.md) your Semantic Layer, the next important (and fun!) step is querying and consuming the metrics you’ve defined. This page links to key resources that guide you through the process of consuming metrics across different integrations, APIs, and tools, using various different [query syntaxes](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata). Connect your AI to your metrics AI tools like Claude and ChatGPT can connect to the Semantic Layer through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=sl-connect-ai), so their answers use your governed metrics instead of guessing at raw tables. Once your Semantic Layer is deployed, you can start querying your metrics using a variety of tools and APIs. Here are the main resources to get you started: ##### Available integrations Integrate the Semantic Layer with a variety of business intelligence (BI) tools and data platforms, enabling seamless metric queries within your existing workflows. Explore the following integrations: * [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) — Review a wide range of partners such as Tableau, Google Sheets, Microsoft Excel, and more, where you can query your metrics directly from the Semantic Layer. ##### Query with APIs To leverage the full power of the Semantic Layer, you can use the Semantic Layer APIs for querying metrics programmatically: * [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) — Learn how to use the Semantic Layer APIs to query metrics in downstream tools, ensuring consistent and reliable data metrics. * [JDBC API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) — Dive into the syntax for querying metrics with the JDBC API, with examples and detailed instructions. * [GraphQL API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#querying) — Learn the syntax for querying metrics via the GraphQL API, including examples and detailed instructions. * [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md#usage-examples) — Use the Python SDK library to query metrics programmatically with Python. ##### Query during development For developers working within the dbt ecosystem, it’s essential to understand how to query metrics during the development phase using MetricFlow commands: * [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) — Learn how to use MetricFlow commands to query metrics directly during the development process, ensuring your metrics are correctly defined and working as expected. #### Next steps After understanding the basics of querying metrics, consider optimizing your setup and ensuring the integrity of your metric definitions: * [Optimize querying performance](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) — Improve query speed and efficiency by using declarative caching techniques. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) — Ensure that any changes to dbt models don’t break your metrics by validating semantic nodes in Continuous Integration (CI) jobs. * [Build your metrics and semantic models](https://docs.getdbt.com/docs/build/build-metrics-intro.md) — If you haven’t already, learn how to define and build your metrics and semantic models using your preferred development tool. --- ### Continuous deployment in dbt dbt platform To help you improve data transformations and ship data products faster, you can run [merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) to implement a continuous deployment (CD) workflow in dbt. Merge jobs can automatically build modified models whenever a pull request (PR) merges, making sure the latest code changes are in production. You don't have to wait for the next scheduled job to run to get the latest updates. [![Workflow of continuous deployment in dbt](/img/docs/dbt-platform/using-dbt-platform/cd-workflow.png?v=2 "Workflow of continuous deployment in dbt")](#)Workflow of continuous deployment in dbt You can also implement continuous integration (CI) in dbt, which can further reduce the time it takes to push changes to production and improve code quality. To learn more, refer to [Continuous integration in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md). #### Trigger jobs with automation or APIs [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) and [deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) start from your [Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md) or from other triggers you configure inside dbt. Merge jobs that react to merges use the webhook flow summarized in [How merge jobs work](#how-merge-jobs-work) below and detailed under [Set up job trigger on Git merge](https://docs.getdbt.com/docs/deploy/merge-jobs.md#set-up-merge-jobs). To start the same merge or deployment job from your own tooling, use the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Trigger Job Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run). Configure and validate the job in dbt first, note the account and job identifiers from the deployment URL or job settings, authenticate with [service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md), assemble the HTTPS request headers and JSON body from [Trigger Job Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run), then monitor the resulting run alongside any webhook-triggered executions in job history. * **CI triggered through Administrative API payloads** behaves differently than rerunning deployment or merge jobs. Follow [Trigger a CI job with the API](https://docs.getdbt.com/docs/deploy/ci-jobs.md#trigger-a-ci-job-with-the-api) on the [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) page. * **Integrations** with schedulers and platforms (for example [Airflow and dbt](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md), Prefect, Databricks) are collected on [Deployment tools](https://docs.getdbt.com/docs/deploy/deployment-tools.md). #### How merge jobs work When you set up merge jobs, dbt listens for notifications from your [Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md) indicating that a PR has been merged. When dbt receives one of these notifications, it enqueues a new run of the merge job. You can set up merge jobs to perform one of the following when a PR merges: | Command to run | Usage description | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `dbt build --select state:modified+` | (Default) Build the modified data with every merge.<br /><br />dbt builds only the changed data models and anything downstream of it, similar to CI jobs. This helps reduce computing costs and ensures that the latest code changes are always pushed to production. | | `dbt compile` | Refresh the applied state for performant (the slimmest) CI job runs.<br /><br />dbt generates the executable SQL (from the source model, test, and analysis files) but does not run it. This ensures the changes are reflected in the manifest for the next time a CI job is run and keeps track of only the relevant changes. | --- ### Continuous integration in dbt dbt platform To implement a continuous integration (CI) workflow in dbt, you can set up automation that tests code changes by running [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) before merging to production. dbt tracks the state of what’s running in your production environment. When you run a CI job, only the modified data assets in your pull request (PR) and their downstream dependencies are built and tested in a staging schema. You can also view the status of the CI checks (tests) directly from within the PR; this information is posted to your Git provider as soon as a CI job completes. Additionally, you can enable settings in your Git provider that allow PRs only with successful CI checks to be approved for merging. For workflows that promote changes after a PR merges (merge jobs or other deployment triggers, including triggering jobs manually or through APIs), refer to [Continuous deployment in dbt](https://docs.getdbt.com/docs/deploy/continuous-deployment.md), including [merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md). [![Workflow of continuous integration in dbt](/img/docs/dbt-platform/using-dbt-platform/ci-workflow.png?v=2 "Workflow of continuous integration in dbt")](#)Workflow of continuous integration in dbt Using CI helps: * Provide increased confidence and assurances that project changes will work as expected in production. * Reduce the time it takes to push code changes to production, through build and test automation, leading to better business outcomes. * Allow organizations to make code changes in a standardized and governed way that ensures code quality without sacrificing speed. #### How CI works When you [set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs), dbt listens for a notification from your Git provider indicating that a new PR has been opened or updated with new commits. When dbt receives one of these notifications, it enqueues a new run of the CI job. dbt builds and tests models, semantic models, metrics, and saved queries affected by the code change in a temporary schema, unique to the PR. This process ensures that the code builds without error and that it matches the expectations as defined by the project's dbt tests. The unique schema name follows the naming convention `dbt_cloud_pr_<job_id>_<pr_id>` (for example, `dbt_cloud_pr_1862_1704`) and can be found in the run details for the given run, as shown in the following image: [![Viewing the temporary schema name for a run triggered by a PR](/img/docs/dbt-platform/using-dbt-platform/using_ci_dbt_cloud.png?v=2 "Viewing the temporary schema name for a run triggered by a PR")](#)Viewing the temporary schema name for a run triggered by a PR When the CI run completes, you can view the run status directly from within the pull request. dbt updates the pull request in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run. The status message states whether the models and tests ran successfully or not. dbt deletes the temporary schema from your data warehouse when you close or merge the pull request. If your project has schema customization using the [generate\_schema\_name](https://docs.getdbt.com/docs/build/custom-schemas.md#how-does-dbt-generate-a-models-schema-name) macro, dbt might not drop the temporary schema from your data warehouse. For more information, refer to [Troubleshooting](https://docs.getdbt.com/docs/deploy/ci-jobs.md#troubleshooting). #### Availability of features by Git provider * If your git provider has a [native dbt integration](https://docs.getdbt.com/docs/platform/git/configure-git.md), you can seamlessly set up [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/ci-jobs.md) jobs directly within dbt. * For providers without native integration, you can still use the [Git clone method](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to import your git URL and leverage the [dbt Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to trigger a CI job to run. The following table outlines the available integration options and their corresponding capabilities. | **Git provider** | **Native dbt integration** | **Automated CI job** | **Git clone** | **Information** | **Supported plans** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | -------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md)<br /> | ✅ | ✅ | ✅ | Organizations on the Starter and Developer plans can connect to Azure DevOps using a deploy key. Note, you won’t be able to configure automated CI jobs but you can still develop. | Enterprise, Enterprise+ | | [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | All other git providers using [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) ([BitBucket](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#bitbucket), [AWS CodeCommit](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#aws-codecommit), and others) | ❌ | ❌ | ✅ | Refer to the [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=1) guide to set up continuous integration and continuous deployment (CI/CD). | | #### Differences between CI jobs and other deployment jobs The [dbt scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) executes CI jobs differently from other deployment jobs in these important ways: * [**Concurrent CI checks**](#concurrent-ci-checks) — The scheduler can execute multiple CI runs from the same dbt CI job concurrently (in parallel) when appropriate. * [**Smart cancellation of stale builds**](#smart-cancellation-of-stale-builds) — The scheduler cancels stale in-flight CI runs when you push new commits to the PR. * [**Run slot treatment**](#run-slot-treatment) — CI runs don't consume a run slot. * [**SQL linting**](#sql-linting) — When enabled, linting runs on all SQL files in your project as a step before your CI job builds. ##### Concurrent CI checks [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") When you collaborate on a dbt project with your team and open pull requests in the same dbt repository, the same CI job can run for each qualifying event. Each CI run writes to a dedicated, temporary schema that is tied to its pull request. That isolation lets dbt execute CI runs *concurrently* instead of *sequentially*. This differs from deployment dbt jobs. You don't have to wait for someone else’s CI run to finish before your own check can start. Concurrent CI checks help your whole team test and integrate dbt code faster. The following describes the conditions when CI checks are run concurrently and when they’re not: * CI runs with different PR numbers execute concurrently. * CI runs with the *same* PR number and *different* commit SHAs execute serially because they’re building into the same schema. dbt will run the latest commit and cancel any older, stale commits. For details, refer to [Smart cancellation of stale builds](#smart-cancellation). * CI runs with the same PR number and same commit SHA can still execute concurrently when they come from different dbt projects. This can happen when two CI jobs are set up in different dbt projects that share the same dbt repository. ##### Smart cancellation of stale builds [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") When you push a new commit to a PR, dbt enqueues a new CI run for the latest commit and cancels any CI run that is (now) stale and still in flight. This can happen when you’re pushing new commits while a CI build is still in process and not yet done. By canceling runs in a safe and deliberate way, dbt helps improve productivity and reduce data platform spend on wasteful CI runs. [![Example of an automatically canceled run](/img/docs/dbt-platform/using-dbt-platform/example-smart-cancel-job.png?v=2 "Example of an automatically canceled run")](#)Example of an automatically canceled run ##### Run slot treatment [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") CI runs don't consume run slots. This guarantees a CI check will never block a production run. ##### SQL linting [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Available on [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and dbt Starter or Enterprise-tier accounts. When [enabled for your CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs), dbt lints the changed SQL files in your project. On v1 runs, dbt invokes [SQLFluff](https://sqlfluff.com/), a modular, configurable SQL linter. On v2 runs, dbt uses the built-in `dbt lint` command instead (see below). Linting warns you about complex functions, syntax, formatting, and compilation errors. Linting on dbt v2 CI jobs that run on v2 automatically use the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. `dbt lint` is SQLFluff-compatible and it reads your existing `.sqlfluff` config, uses the same rule codes, and respects `-- noqa` suppression comments. For parity expectations between `dbt lint` and SQLFluff, refer to [Rule parity with SQLFluff](https://docs.getdbt.com/reference/commands/lint.md?version=2.0#rule-parity-with-sqlfluff). By default, SQL linting lints all the changed SQL files in your project, compared to the last deferred production state. If dbt can't find the commit for that deferred state, linting doesn't fail the run. dbt shows a warning with the unresolved commit SHA and lints the entire project instead. To refresh the baseline, rerun the job you defer to. Note that [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) can be defined in YAML *and* `.sql` files. Their SQL isn't lintable and can cause errors during linting. To prevent SQLFluff from linting snapshot files, add the snapshots directory to your `.sqlfluffignore` file (for example `snapshots/`). Refer to [snapshot linting](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#snapshot-linting) for more information. If the linter runs into errors, you can specify whether dbt should stop running the job on error or continue running it on error. When failing jobs, it helps reduce compute costs by avoiding builds for pull requests that don't meet your SQL code quality CI check. ###### To configure SQLFluff linting You can optionally configure SQLFluff linting rules to override default linting behavior. * Use [SQLFluff Configuration Files](https://docs.sqlfluff.com/en/stable/configuration/setting_configuration.html#configuration-files) to override the default linting behavior in dbt. * Create a `.sqlfluff` configuration file in your project, add your linting rules to it, and dbt will use them when linting. * When configuring, you can use `dbt` as the templater (for example, `templater = dbt`) * If you’re using the Studio IDE, dbt CLI, or any other editor, refer to [Customize linting](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#customize-linting) for guidance on how to add the dbt-specific (or dbtonic) linting rules we use for our own project. * For complete details, refer to [Custom Usage](https://docs.sqlfluff.com/en/stable/gettingstarted.html#custom-usage) in the SQLFluff documentation. --- ### Continuous integration jobs in dbt dbt platform You can set up [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) (CI) jobs to run when someone opens a new pull request (PR) in your Git repository. By running and testing only *modified* models, dbt keeps these jobs as resource-conscious as possible on your data platform. Triggering CI jobs in monorepos If you have a monorepo with several dbt projects, opening a single pull request in one of your projects will trigger jobs for all projects connected to the monorepo. To address this, you can use separate target branches per project (for example, `main-project-a`, `main-project-b`) to separate CI triggers. #### Prerequisites * You have a dbt account. * CI features: * For both the [concurrent CI checks](https://docs.getdbt.com/docs/deploy/continuous-integration.md#concurrent-ci-checks) and [smart cancellation of stale builds](https://docs.getdbt.com/docs/deploy/continuous-integration.md#smart-cancellation) features, your dbt account must be on the [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * [SQL linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) is available on [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and to dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing/) accounts. Refer to [Configure SQLFluff linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#to-configure-sqlfluff-linting) when you add SQLFluff to your project. Linting on dbt v2 CI jobs that run on v2 automatically use the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. `dbt lint` is SQLFluff-compatible and it reads your existing `.sqlfluff` config, uses the same rule codes, and respects `-- noqa` suppression comments. * [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md) features: * For the [compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) feature, your dbt account must be on an [Enterprise-tier plan](https://www.getdbt.com/pricing/) and have enabled Advanced CI features. Please ask your [dbt administrator to enable](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-ci-features) this feature for you. After enablement, the **dbt compare** option becomes available in the CI job settings. * Set up a [connection with your Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md). This integration lets dbt run jobs on your behalf for job triggering. * If you're using a native [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) integration, you need a paid or self-hosted account that includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). If you're using GitLab Free, merge requests will trigger CI jobs but CI job status updates (success or failure of the job) will not be reported back to GitLab. #### Availability of features by Git provider * If your git provider has a [native dbt integration](https://docs.getdbt.com/docs/platform/git/configure-git.md), you can seamlessly set up [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/ci-jobs.md) jobs directly within dbt. * For providers without native integration, you can still use the [Git clone method](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to import your git URL and leverage the [dbt Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to trigger a CI job to run. The following table outlines the available integration options and their corresponding capabilities. | **Git provider** | **Native dbt integration** | **Automated CI job** | **Git clone** | **Information** | **Supported plans** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | -------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md)<br /> | ✅ | ✅ | ✅ | Organizations on the Starter and Developer plans can connect to Azure DevOps using a deploy key. Note, you won’t be able to configure automated CI jobs but you can still develop. | Enterprise, Enterprise+ | | [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | All other git providers using [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) ([BitBucket](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#bitbucket), [AWS CodeCommit](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#aws-codecommit), and others) | ❌ | ❌ | ✅ | Refer to the [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=1) guide to set up continuous integration and continuous deployment (CI/CD). | | #### Set up CI jobs dbt Labs recommends that you create your CI job in a dedicated dbt [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) that's connected to a staging database. A separate CI environment improves isolation between your temporary CI schemas and production builds. You can trigger CI jobs for pull requests targeting a branch other than `main`. For example, if you use a long-lived staging branch, you can associate a CI environment with that branch via [custom branch](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) settings. Then the job in that environment runs only when someone opens PRs against that branch. To learn more, refer to [Get started with CI tests](https://docs.getdbt.com/guides/set-up-ci.md). To make CI job creation easier, many options on the **CI job** page are set to default values that dbt Labs recommends that you use. If you don't want to use the defaults, you can change them. 1. On your deployment environment page, click **Create job** > **Continuous integration job** to create a new CI job. 2. Options in the **Job settings** section: * **Job name** — Specify the name for this CI job. * **Description** — Provide a description about the CI job. * **Environment** — By default, this will be set to the environment you created the CI job from. Use the dropdown to change the default setting. 3. Options in the **Git trigger** section: * **Triggered by pull requests** — By default, it’s enabled. Every time a developer opens up a pull request or pushes a commit to an existing pull request, this job will get triggered to run. * **Run on draft pull request** — Enable this option if you want to also trigger the job to run every time a developer opens up a draft pull request or pushes a commit to that draft pull request. 4. Options in the **Execution settings** section: * **Commands** — By default, this includes the `dbt build --select state:modified+` command. This informs dbt to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when this job runs. * **Linting** — Enable this option for dbt to [lint the SQL files](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) in your project as the first step in `dbt run`. If this check runs into an error, dbt can either **Stop running on error** or **Continue running on error**. * **dbt compare**[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") — Enable this option to compare the last applied state of the production environment (if one exists) with the latest changes from the pull request, and identify what those differences are. To enable record-level comparison and primary key analysis, you must add a [primary key constraint](https://docs.getdbt.com/reference/resource-properties/constraints.md) or [uniqueness test](https://docs.getdbt.com/reference/resource-properties/data-tests.md#unique). Otherwise, you'll receive a "Primary key missing" error message in dbt. To review the comparison report, navigate to the [Compare tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#compare-tab) in the job run's details. A summary of the report is also available from the pull request in your Git provider (refer to the [CI report example](#example-ci-report)). Optimization tip When you enable the **dbt compare** checkbox, you can customize the comparison command to optimize your CI job. For example, if you have large models that take a long time to compare, you can exclude them to speed up the process using the [`--exclude` flag](https://docs.getdbt.com/reference/node-selection/exclude.md). Refer to [compare changes custom commands](https://docs.getdbt.com/docs/deploy/job-commands.md#compare-changes-custom-commands) for more details. Additionally, if you set [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) in your models/seeds/snapshots/sources, it allows you to compare matching date ranges between tables by filtering to overlapping date ranges. This is useful for faster CI workflows or custom sampling setups. * **Compare changes against an environment (Deferral)** — By default, it’s set to the **Production** environment if you created one. This option allows dbt to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG. Comparison manifests The latest successful run in the environment can come from *any* job that updates artifacts there. If many jobs run with different settings, your CI comparison state can change in ways that are hard to predict. Even when you defer to Production, a merge or deploy that refreshes the manifest while other pull requests are open can make those runs pick up unrelated `state:modified` nodes until branches are updated. * **Run timeout** — Cancel the CI job if the run time exceeds the timeout value. You can use this option to help ensure that a CI check doesn't consume too much of your warehouse resources. If you enable the **dbt compare** option, the timeout value defaults to `3600` (one hour) to prevent long-running comparisons. * **dbt State** — [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) reuses nodes when their logic and data haven’t changed, avoiding unnecessary rebuilds. Select **On**, **Off**, or **Inherited from environment** to use the environment’s setting. This option appears only when dbt State is [enabled](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) on your account. Learn more about [enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md#enabling-dbt-state-on-individual-jobs). 5. (optional) Options in the **Advanced settings** section: * **Environment variables** — Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when this CI job runs. You can specify that a CI job is running in a *Staging* or *CI* environment by setting an environment variable and modifying your project code to behave differently, depending on the context. It's common for teams to process only a subset of data for CI runs, using environment variables to branch logic in their dbt project code. * **Target name** — Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md). Similar to **Environment Variables**, this option lets you customize the behavior of the project. You can use this option to specify that a CI job is running in a *Staging* or *CI* environment by setting the target name and modifying your project code to behave differently, depending on the context. * **dbt version** — By default, it’s set to inherit the [dbt version](https://docs.getdbt.com/docs/dbt-versions.md) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior. * **Threads** — By default, it’s set to 4 [threads](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads). Increase the thread count to increase model execution concurrency. - **Run source freshness** — Enable this option to invoke the `dbt source freshness` command before running this CI job. Refer to [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) for more details. [![Example of CI Job page in the dbt UI](/img/docs/dbt-platform/using-dbt-platform/create-ci-job.png?v=2 "Example of CI Job page in the dbt UI")](#)Example of CI Job page in the dbt UI ##### Example of CI check in pull request The following is an example of a CI check in a GitHub pull request. The green checkmark means the dbt build and tests were successful. Clicking on the dbt section takes you to the relevant CI run in dbt. [![Example of CI check in GitHub pull request](/img/docs/dbt-platform/using-dbt-platform/example-github-pr.png?v=2 "Example of CI check in GitHub pull request")](#)Example of CI check in GitHub pull request ##### Example of CI report in pull request [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") The following is an example of a CI report in a GitHub pull request, which is shown when the **dbt compare** option is enabled for the CI job. It displays a high-level summary of the models that changed from the pull request. [![Example of CI report comment in GitHub pull request](/img/docs/dbt-platform/using-dbt-platform/example-github-ci-report.png?v=2 "Example of CI report comment in GitHub pull request")](#)Example of CI report comment in GitHub pull request #### Trigger a CI job with the API [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") If you're not using dbt’s native Git integration with [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md), [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md), or [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md), you can use the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to trigger a CI job to run. However, dbt will not automatically delete the temporary schema for you. This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. If you instead need workflows that run after a merge (not CI checks on an open pull request), refer to [Continuous deployment in dbt](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) and [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md). ##### Prerequisites * You have a dbt account. * You have a dbt [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/). Legacy Team plans also retain access. * For the [Concurrent CI checks](https://docs.getdbt.com/docs/deploy/continuous-integration.md#concurrent-ci-checks) and [Smart cancellation of stale builds](https://docs.getdbt.com/docs/deploy/continuous-integration.md#smart-cancellation) features, your dbt account must be on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/), and legacy Team plans. Starter plans do not have access to these features when triggering a CI job with the API. 1. Set up a CI job with the [Create Job](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint using `"job_type": ci` or from the [dbt UI](#set-up-ci-jobs). 2. Call the [Trigger Job Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run) API endpoint to trigger the CI job. You must include both of these fields to the payload: * Provide the pull request (PR) ID using one of these fields: * `github_pull_request_id` * `gitlab_merge_request_id` * `azure_devops_pull_request_id` * `non_native_pull_request_id` (for example, Bitbucket) * Provide the `git_sha` or `git_branch` to target the correct commit or branch to run the job against. #### Semantic validations in CI [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Automatically test your semantic nodes (metrics, semantic models, and saved queries) during code reviews by adding warehouse validation checks in your CI job, guaranteeing that any code changes made to dbt models don't break these metrics. To do this, add the command `dbt sl validate --select state:modified+` in the CI job. This ensures the validation of modified semantic nodes and their downstream dependencies. [![Semantic validations in CI workflow](/img/docs/dbt-platform/deployment/sl-ci-job.png?v=2 "Semantic validations in CI workflow")](#)Semantic validations in CI workflow ###### Benefits * Testing semantic nodes in a CI job supports deferral and selection of semantic nodes. * It allows you to catch issues early in the development process and deliver high-quality data to your end users. * Semantic validation executes an explain query in the data warehouse for semantic nodes to ensure the generated SQL will execute. * For semantic nodes and models that aren't downstream of modified models, dbt defers to the production models. ##### Set up semantic validations in your CI job To learn how to set this up, refer to the following steps: 1. Click **Orchestration**. 2. Click **Jobs**. 3. Select the relevant job. 4. Click **Settings**. 5. Click **Edit**. 6. Add the `dbt sl validate --select state:modified+` command to the **Commands** field in the **Execution settings** section. The command uses state selection and deferral to run validation on any semantic nodes downstream of model changes. To reduce job times, we recommend only running CI on modified semantic models. 7. Click **Save** to save your changes. There are additional commands and use cases described in the [next section](#use-cases), such as validating all semantic nodes, validating specific semantic nodes, and so on. [![Validate semantic nodes downstream of model changes in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-downstream.png?v=2 "Validate semantic nodes downstream of model changes in your CI job.")](#)Validate semantic nodes downstream of model changes in your CI job. ##### Use cases Use or combine different selectors or commands to validate semantic nodes in your CI job. Semantic validations in CI support the following use cases:  Semantic nodes downstream of model changes (recommended) To validate semantic nodes that are downstream of a model change, add the two commands in your job **Execution settings** section: ```bash dbt build --select state:modified+ dbt sl validate --select state:modified+ ``` * The first command builds the modified models. * The second command validates the semantic nodes downstream of the modified models. Before running semantic validations, dbt must build the modified models. This process ensures that downstream semantic nodes are validated using the CI schema through the dbt Semantic Layer API. For semantic nodes and models that aren't downstream of modified models, dbt defers to the production models. [![Validate semantic nodes downstream of model changes in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-downstream.png?v=2 "Validate semantic nodes downstream of model changes in your CI job.")](#)Validate semantic nodes downstream of model changes in your CI job.  Semantic nodes that are modified or affected by downstream modified nodes. To only validate modified semantic nodes, use the following command (with [state selection](https://docs.getdbt.com/reference/node-selection/state-selection.md)): ```bash dbt sl validate --select state:modified+ ``` [![Use state selection to validate modified metric definition models in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-modified.png?v=2 "Use state selection to validate modified metric definition models in your CI job.")](#)Use state selection to validate modified metric definition models in your CI job. This will only validate semantic nodes. It will use the defer state set configured in your orchestration job, deferring to your production models.  Select specific semantic nodes Use the selector syntax to select the *specific* semantic node(s) you want to validate: ```bash dbt sl validate --select metric:revenue ``` [![Use state selection to validate modified metric definition models in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-select.png?v=2 "Use state selection to validate modified metric definition models in your CI job.")](#)Use state selection to validate modified metric definition models in your CI job. In this example, the CI job will validate the selected `metric:revenue` semantic node. To select multiple semantic nodes, use the selector syntax: `dbt sl validate --select metric:revenue metric:customers`. If you don't specify a selector, dbt will validate all semantic nodes in your project.  Select all semantic nodes To validate *all* semantic nodes in your project, add the following command to defer to your production schema when generating the warehouse validation queries: ```bash dbt sl validate ``` [![Validate all semantic nodes in your CI job by adding the command: 'dbt sl validate' in your job execution settings.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-all.png?v=2 "Validate all semantic nodes in your CI job by adding the command: 'dbt sl validate' in your job execution settings.")](#)Validate all semantic nodes in your CI job by adding the command: 'dbt sl validate' in your job execution settings. #### Troubleshooting Unable to trigger a CI job with GitLab When you connect dbt to a GitLab repository, GitLab automatically registers a webhook in the background, viewable under the repository settings. This webhook is also used to trigger [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) when you push to the repository. If you're unable to trigger a CI job, this usually indicates that the webhook registration is missing or incorrect. To resolve this issue, navigate to the repository settings in GitLab and view the webhook registrations by navigating to GitLab --> **Settings** --> **Webhooks**. Some things to check: * The webhook registration is enabled in GitLab. * The webhook registration is configured with the correct URL and secret. If you're still experiencing this issue, reach out to the Support team at <support@getdbt.com> and we'll be happy to help!  CI selects or fails on models that are not in my pull request This usually means the comparison manifest does not line up with your branch’s commits. Another job overwrote `manifest.json` with different settings, Production (or another deferred environment) advanced after someone else merged while your pull request stayed open, or the manifest is stale right after a merge. Built-in deferral targets an environment, not a hand-picked job ID. To resolve this, merge or rebase the latest base branch into your pull request so your branch includes recent merges. To refresh the comparison manifest without waiting on a long deploy, use a [merge job](https://docs.getdbt.com/docs/deploy/merge-jobs.md) in the same environment your CI job defers to. For example, one that runs `dbt parse --no-partial-parse` (or [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md)) immediately after merges.  CI jobs aren't triggering occasionally when opening a PR using the Azure DevOps (ADO) integration dbt won't trigger a CI job run if the latest commit in a pull or merge request has already triggered a run for that job. However, some providers (like GitHub) will enforce the result of the existing run on multiple pull/merge requests. Scenarios where dbt does not trigger a CI job with Azure DevOps: 1. Reusing a branch in a new PR * If you abandon a previous PR (PR 1) that triggered a CI job for the same branch (`feature-123`) merging into `main`, and then open a new PR (PR 2) with the same branch merging into`main` — dbt won't trigger a new CI job for PR 2. 2. Reusing the same commit * If you create a new PR (PR 2) on the same commit (`#4818ceb`) as a previous PR (PR 1) that triggered a CI job — dbt won't trigger a new CI job for PR 2.  Temporary schemas aren't dropping If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates one of these issues: * You have overridden the `generate_schema_name` macro and it isn't using `dbt_cloud_pr_` as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: * ✅ Temporary PR schema name contains the prefix `dbt_cloud_pr_` (like `dbt_cloud_pr_123_456_marketing`). * ❌ Temporary PR schema name doesn't contain the prefix `dbt_cloud_pr_` (like `marketing`). A macro is creating a schema but there are no dbt models writing to that schema. dbt doesn't drop temporary schemas that weren't written to as a result of running a dbt model.  Error messages that refer to schemas from previous PRs If you receive a schema-related error message referencing a *previous* PR, this is usually an indicator that you are not using a production job for your deferral and are instead using *self*. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the CI job for the current PR is kicked off. To fix this issue, select a production job run to defer to instead of self.  Production job runs failing at the 'Clone Git Repository step' dbt can only check out commits that belong to the original repository. dbt *cannot* checkout commits that belong to a fork of that repository. If you receive the following error message at the **Clone Git Repository** step of your job run: ```text Error message: Cloning into '/tmp/jobs/123456/target'... Successfully cloned repository. Checking out to e845be54e6dc72342d5a8f814c8b3316ee220312...> Failed to checkout to specified revision. git checkout e845be54e6dc72342d5a8f814c8b3316ee220312 fatal: reference is not a tree: e845be54e6dc72342d5a8f814c8b3316ee220312 ``` Double-check that your PR isn't trying to merge using a commit that belongs to a fork of the repository attached to your dbt project.  CI job not triggering for Virtual Private dbt users To trigger jobs on dbt using the [API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md), your Git provider needs to connect to your dbt account. If you're on a Virtual Private dbt Enterprise plan using security features like ingress PrivateLink or IP Allowlisting, registering CI hooks may not be available and can cause the job to fail silently.  PR status for CI job stays in 'pending' in Azure DevOps after job run finishes When you start a CI job, the pull request status should show as `pending` while it waits for an update from dbt. Once the CI job finishes, dbt sends the status to Azure DevOps (ADO), and the status will change to either `succeeded` or `failed`. If the status doesn't get updated after the job runs, check if there are any git branch policies in place blocking ADO from receiving these updates. One potential issue is the **Reset conditions** under **Status checks** in the ADO repository branch policy. If you enable the **Reset status whenever there are new changes** checkbox (under **Reset conditions**), it can prevent dbt from updating ADO about your CI job run status. You can find relevant information here: * [Azure DevOps Services Status checks](https://learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops\&tabs=browser#status-checks) * [Azure DevOps Services Pull Request Stuck Waiting on Status Update](https://support.hashicorp.com/hc/en-us/articles/18670331556627-Azure-DevOps-Services-Pull-Request-Stuck-Waiting-on-Status-Update-from-Terraform-Cloud-Enterprise-Run) * [Pull request status](https://learn.microsoft.com/en-us/azure/devops/repos/git/pull-request-status?view=azure-devops#pull-request-status) --- ### Contribute a dbt v2 adapter Available in v2 | Local development Work in progress This guide is a work in progress — synthesized from internal v2 adapter team docs, DuckDB/ClickHouse reference implementations, the Exasol community adapter, and the v1 adapter creation guide. **Community contribution scope:** You are responsible for **Phase 1 (foundation)** only — connecting, running macros, and basic materializations. Phase 2 (SQL static analysis) is led by dbt Labs after your Phase 1 PR is merged. #### Step 1: Introduction dbt v2 adapters work very differently from dbt v1 adapters. In v1, each adapter was a standalone Python package that implemented a fragmented Python interface. In v2, adapters live **inside a monorepo written in Rust**, connected to warehouses through ADBC (Arrow Database Connectivity) drivers — and the community contribution model has changed accordingly. This guide walks you through contributing a new v2 adapter to `dbt-labs/dbt` as a community member. The file breakdown in the [reference section](#reference-file-by-file-implementation-guide) shows the ~13 files a complete community v2 adapter touches. Exasol, a community contributed v2 adapter, is the example used throughout this guide. What is dbt v2? dbt v2 is the new Rust-based dbt engine. Adapters in v2 are written in Rust and live inside the `dbt-labs/dbt` monorepo, rather than as standalone Python packages. ##### How adapters are different now In dbt v1, every adapter is: * A **separate Python package** (for example, `dbt-snowflake` and `dbt-bigquery`) * **Community-owned** and maintained independently * **Dynamically loaded** at runtime through Python's plugin system * Built upon vendor-maintained Python SDK connectors/drivers In dbt v2, adapters are: * **Part of the `dbt-labs/dbt` monorepo**, contributed as PRs * **Written in Rust** (with Jinja SQL macros still used for SQL logic) * Connected to warehouses through **ADBC drivers** — a unified driver interface that abstracts away connection management * Organized by **vertical** (feature area) rather than by warehouse — meaning a bug fix in one crate often benefits all adapters The practical upside: **adding a new adapter requires far less code than it did in v1**, and the cost of development decreases as more adapters are added because shared logic increases. ##### What you'll be building As a community contributor, you're building the **foundation**: connect your warehouse, run dbt macros, and support basic materializations (`dbt run`, `dbt build`). The ~13 files in the reference section define the complete scope. *** #### Step 2: Prerequisites ##### Background knowledge Before starting, you should: * **Have some Rust familiarity** — you do not need to be a Rust expert (an LLM will likely do most of the code writing). You need to be able to read Rust, understand enum match expressions, and make sense of compiler errors. If Rust is new to you, skim [The Rust Book](https://doc.rust-lang.org/book/) Chapter 6 (enums and pattern matching) — those are the concepts you'll encounter most. * **Understand dbt fundamentals** — how [`profiles.yml`](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) works, what [materializations](https://docs.getdbt.com/docs/build/materializations) are, and what [adapter dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter) macros do. * **Understand your query engine** and how it differs from others. The following questions cover the key considerations for building an adapter: * What character is used for quoting identifiers? * Are three-part names supported, or only two? * Are transactions supported? * What's the most performant way to get all the columns in a schema? * What datatype represents a string? * What connection fields does your warehouse require? * **Have an LLM coding assistant available** (optional) — the adapter development workflow is designed with AI-assisted coding in mind, and this guide calls out where to use it. ##### Your ADBC driver The ADBC driver for your warehouse must already exist before you start dbt Labs does not write drivers. If your warehouse doesn't have an ADBC driver yet, that work comes first — it's a prerequisite you can't skip. **What "having an ADBC driver" actually means** For certain adapters (Snowflake, BigQuery, Databricks, Redshift, DuckDB, ClickHouse, Spark, SQL Server), v2 automatically downloads the correct driver binary from the dbt Labs CDN on first use. Users never have to think about it. Community adapters don't have CDN support. Instead, dbt v2 looks for a shared library by name on the user's system — for example, `libadbc_driver_exasol.dylib` on macOS and `libadbc_driver_exasol.so` on Linux. If the file isn't present, the connection fails at runtime. This means two things for you as a contributor: 1. **The driver binary must exist somewhere.** It's a compiled shared library (`.dylib`/`.so`/`.dll`) that implements the ADBC C ABI for your warehouse. Exasol's driver is `exarrow-rs`, a separate Rust crate maintained by Exasol — not part of `dbt` and not in `apache/arrow-adbc`. Your warehouse's driver will likely live in a similar separate repo. 2. **Your users will need to install it manually.** Unlike first-party adapters, there's no auto-download. Your documentation (Step 6) needs to tell users where to get the driver and how to install it. Check whether a driver already exists for your warehouse: * [dbt-labs/arrow-adbc](https://github.com/dbt-labs/arrow-adbc) — the dbt Labs fork * [apache/arrow-adbc](https://github.com/apache/arrow-adbc) — the upstream community repo * [adbc-drivers](https://github.com/orgs/adbc-drivers/repositories?type=all) — the [ADBC driver foundry](https://adbc-drivers.org/2025/10/29/announcing-adbc-driver-foundry.html), the primary community home for ADBC drivers, with active drivers for Trino, MySQL, BigQuery, Databricks, Snowflake, and more; also includes Rust/Go frameworks (`driverbase-rs`, `template-rs`) for building new drivers from scratch * The warehouse vendor's own GitHub org (many vendors publish ADBC drivers independently) Driver source is flexible — with a security caveat dbt v2 loads drivers by shared library name from the system path (for example, `libadbc_driver_exasol.dylib` on macOS) — the source repository is flexible, as long as the binary implements the ADBC C ABI. dbt Labs is working on a driver signing and verification mechanism. Until that ships, users are responsible for trusting the driver binary they install. If no driver exists yet, building one is a separate project that comes before the adapter contribution. This is outside the scope of what dbt Labs can help with. [Columnar](https://columnar.tech/) specializes in building ADBC drivers and may be a useful resource if you need help getting a driver built. ##### Are you porting an existing v1 adapter? Building from scratch with no v1 adapter? Skip to the checklist below, then go straight to Step 4. You won't have macro SQL or connection logic to port, but the rest of the guide (crate structure, Rust patterns, AI workflow) still applies. Most community contributors aren't building from scratch — they're porting an adapter they already maintain or use in Python for dbt v1. If that's you, **you have a significant head start**, and this is the most realistic path for the vast majority of contributors. Before writing any Rust, check a few things: **1. Does a v1 adapter already exist for your warehouse?** Refer to the [community adapters](https://docs.getdbt.com/docs/community-adapters.md) list. If one exists, find its GitHub repo. The macro SQL and connection logic are almost directly reusable. **2. Did dbt Labs already add a placeholder for your warehouse?** Some warehouses already appear in v2's `AdapterType` enum but aren't fully implemented yet — the enum variant exists, which means the boilerplate is partially in place. When you add the remaining code, the compiler shows you exactly what's still missing. The warehouses in this state are: | Warehouse | v1 adapter | dbt v2 status | | --------- | ------------------------------------------------------- | ------------------------------------------------------------------ | | Athena | [dbt-athena](https://github.com/dbt-athena/dbt-athena) | `AdapterType::Athena` exists — needs auth, macros, adapter arms | | Trino | [dbt-trino](https://github.com/starburstdata/dbt-trino) | `AdapterType::Trino` exists — needs auth, macros, adapter arms | | Starburst | [dbt-trino](https://github.com/starburstdata/dbt-trino) | `AdapterType::Starburst` exists — needs auth, macros, adapter arms | | Dremio | [dbt-dremio](https://github.com/dremio/dbt-dremio) | `AdapterType::Dremio` exists — needs auth, macros, adapter arms | | Oracle | [dbt-oracle](https://github.com/oracle/dbt-oracle) | `AdapterType::Oracle` exists — needs auth, macros, adapter arms | For warehouses not yet in `AdapterType` at all (MySQL, Hive, Vertica, SQL Server, Teradata, and more), you start from Step 4.1 by adding the `AdapterType` variant. **What transfers from v1 to v2** | v1 component | Where it goes in v2 | Notes | | ----------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `macros/adapters.sql` | `dbt-loader/.../dbt-<wh>/macros/adapters.sql` | Mostly a direct port — same `<wh>__` dispatch prefix, same macro names, same Jinja patterns | | `macros/catalog.sql` | Same location | The catalog SQL (`list_relations_without_caching`, `get_catalog`) transfers almost verbatim | | Custom materializations and adapter-overrides | Same location | Look out for Jinja that might not yet be supported in v2; that will need to be addressed separately | | Profile fields in `credentials.py` / `profile_template.yml` | `DbConfig` struct in `dbt-schemas` | Each profile field becomes a struct field — optional fields use `Option<T>` | | Connection URI / DSN construction in `connections.py` | `dbt-auth/src/<wh>/mod.rs` | The URI building logic maps cleanly to the auth module pattern | | `BaseRelation.quote_policy` / identifier casing behavior | `Policy::new(...)` in `relation_object.rs` | The 3-part vs. 2-part name structure and quote flags map 1:1 to the v2 `Policy` struct | | Catalog introspection SQL in macros and `adapter.py` | `get_relation.rs` and Jinja macros | The system catalog table names and queries you already know transfer directly | **What doesn't transfer by design** A few components won't transfer 1:1 from v1 to v2 — meaning less code and maintenance for you: * `ConnectionManager` methods (`open`, `cancel`, `get_response`, `execute`) — In v1, these were your responsibility to implement against the Python DB API 2.0 spec. In v2, connection management is owned entirely by the ADBC driver. * Python adapter class hierarchy and execution-wrapping methods — now handled by `match adapter_type()` expressions in the shared `adapter_impl.rs` (Step 4.5 explains how these work). * `setup.py` / package dependencies on `dbt-core` — dbt Labs handles packaging and distribution once your PR is merged. No `setup.py`, no PyPI release, and no version pinning required. **How to use the v1 adapter with AI** The Jinja macros are the biggest asset. Feed your LLM your v1 `macros/adapters.sql` alongside the reference `adapters.sql` (in the file breakdown at the bottom of this guide) and ask it to port the macros into the v2 structure. The SQL logic is the same; you're confirming the dialect is right and the system catalog table names match. For profile config and auth, feed your LLM your v1 `credentials.py` or `profile_template.yml` alongside the reference `DbConfig` struct and auth module (see the file breakdown below). Field names and connection parameters usually map directly. The most common mismatch: v1 `connections.py` has a lot of Python connection-management code that has no equivalent in v2 — the driver handles all of that. Focus only on URI/DSN construction and credential fields — skip anything related to cursors, retries, or connection pooling. ##### Dev machine setup ```bash # Rust rustup # install from https://rustup.rs rustup show # verify # Go (needed for driver builds) go version # verify # Clone the repo git clone https://github.com/dbt-labs/dbt cd dbt # Verify you can build cargo build --bin dbt ``` If you hit Z3 errors: ```bash brew install pkg-config z3 ``` If disk fills during build: ```bash cargo clean # frees old build artifacts — you'll do this often ``` ##### Development workflow The core development loop is the same whether you write code manually or with an AI assistant: 1. Add `AdapterType::MyWarehouse` to the enum (Step 4.1) 2. Run `cargo check -p <crate>` — the compiler lists every match arm missing for your new variant 3. Use the compiler output + the file you're editing + the equivalent reference file (from the breakdown below) to write each arm. If using an AI assistant, paste all three in as context. * *Example:* to fill in the `quote_char` arm, paste the compiler error listing `AdapterType::MyWarehouse` as missing, the `quote_char` match block from `dbt-adapter-core/src/lib.rs`, and the Exasol line (`Exasol => '"'`) as the pattern to follow. 4. Fill in the arm; verify with `cargo check` again 5. Repeat for each crate until error-free Every missing case is a compile error, so the AI always has a precise specification to work from. **What context helps for each arm:** * **The file you're editing** — paste the relevant `match` block or function. For example: the `match self.adapter_type()` block from `adapter_impl.rs` that needs a new arm. * **The equivalent reference file** from the breakdown below. For example: when writing your auth module, paste `crates/dbt-auth/src/exasol/mod.rs` as the pattern. * **The compiler error output.** For example: `error[E0004]: non-exhaustive patterns: AdapterType::MyWarehouse not covered` * **Your warehouse's specifics** — system catalog table names and the connection fields from your `profiles.yml`. **Watch out for:** * **Hallucinated file paths** — AI often invents v2 paths. Use the file breakdown below as ground truth. * **Always verify with the type checker** — run `cargo check -p <crate>` after any AI-generated changes. * **SQL macro patterns from v1** may not apply cleanly in dbt v2. Compare against the reference `adapters.sql` in `crates/dbt-loader/src/dbt_macro_assets/dbt-exasol/macros/adapters.sql` in [dbt-labs/dbt](https://github.com/dbt-labs/dbt). *** #### Step 3: Understand the architecture Before writing code, it helps to understand the layers you'll be working in. ##### Why ADBC? A key architectural decision in dbt v2 is the use of **ADBC (Arrow Database Connectivity)** as the unified driver interface. In dbt v1, adapters connected through Python drivers — often wrapping `pyodbc` or proprietary connection mechanisms. Each adapter owned its connection logic entirely. In v2, each warehouse connects through an **ADBC driver** — a pre-compiled binary that handles the wire protocol, authentication handshakes, and connection pooling. Your adapter code never touches any of that. For CDN-supported first-party adapters, dbt v2 downloads this driver automatically on first use. For community adapters, users install it manually — which is why Step 2 covers finding or building a driver. ADBC is column-native end-to-end — if your warehouse supports columnar output (for example, Arrow IPC or Arrow Flight SQL), data flows through with zero conversion. This means **you do not need to write connection management code** — that lives in the driver. What you write is the warehouse-specific configuration, authentication, relation naming, macro logic, and catalog introspection that sits above the driver. ##### The vertical model Unlike v1 where *each warehouse* had its own monolithic adapter class (`SnowflakeAdapter`, `BigqueryAdapter`), v2 organizes adapter logic by **vertical** — feature areas that *span all warehouses*: * **`dbt-xdbc`** — driver loading, ADBC connections (all warehouses) * **`dbt-auth`** — credential resolution (all warehouses, per-warehouse modules) * **`dbt-adapter`** — a single `ConcreteAdapter` with warehouse-specific behavior driven by `match adapter_type()`, not per-warehouse structs * **`dbt-loader`** — Jinja SQL macros per warehouse When you add a new warehouse, you add a variant to each vertical, not a new top-level package. The compiler enforces this: if you add `AdapterType::MyWarehouse` and any `match adapter_type()` block doesn't handle it, Rust won't compile. For example, if `adapter_impl.rs` has a `match self.adapter_type()` that covers Postgres, Snowflake, Exasol, and more, but not `MyWarehouse`, the compiler reports an error at that exact line — so you always know what's left. This is a meaningful improvement over v1's class inheritance model. In v1, a new adapter subclass would silently inherit parent behavior for any method you didn't explicitly override — which could mask bugs where a warehouse quietly used the wrong default. In v2 there's no inheritance and no implicit behavior. Every function that varies by warehouse requires an explicit match arm, so nothing falls through silently. For example: ```text error[E0004]: non-exhaustive patterns: `AdapterType::MyWarehouse` not covered --> crates/dbt-adapter/src/adapter/adapter_impl.rs:142:18 | | match self.adapter_type() { | ^^^^^^^^^^^^^^^^^^^ pattern `AdapterType::MyWarehouse` not covered ``` ##### Layer stack The diagram below shows how a dbt project request flows through the crates at runtime — from configuration to SQL execution. Your adapter work lives in the middle layers: profile config, credential resolution, driver loading, relation logic, and macros. The bottom layer (SQL execution against the warehouse) is owned by dbt Labs. ```text profiles.yml / dbt_project.yml │ ▼ dbt-schemas ← profile config struct for your warehouse │ ▼ dbt-auth ← credential resolution │ ▼ dbt-xdbc ← ADBC driver loading │ ▼ dbt-adapter ← relation types, metadata adapter, adapter methods │ dbt-loader ← Jinja SQL macros (materializations, catalog, utils) │ ▼ [dbt internal] ← SQL execution against warehouse (dbt Labs) ``` ##### Crate map In Rust, a **crate** is a package — the unit of compilation, roughly equivalent to a "library" or "module" in other languages. The `dbt-labs/dbt` monorepo has multiple crates, each responsible for one vertical slice of functionality across all warehouses. This is a quick-reference map of the six crates you'll touch to build your adapter, in the order you'll work through them in Step 4. | Crate | Location | What you touch | | ------------------ | -------------------------- | ------------------------------------------------------------------------------------------ | | `dbt-adapter-core` | `crates/dbt-adapter-core/` | `AdapterType` enum variant, `quote_char` arm | | `dbt-xdbc` | `crates/dbt-xdbc/` | ADBC driver registration (`Backend` enum, library name) | | `dbt-schemas` | `crates/dbt-schemas/` | `profiles.yml` config struct (`DbConfig` variant) | | `dbt-auth` | `crates/dbt-auth/` | Credential resolution, connection URI construction | | `dbt-adapter` | `crates/dbt-adapter/` | Relation quoting, metadata catalog queries, adapter match arms, column builder, sql\_types | | `dbt-loader` | `crates/dbt-loader/` | Jinja SQL macros (`dbt_macro_assets/dbt-<adapter>/`) | *** #### Step 4: Build a new adapter This step walks you through each crate you need to touch. Work through them in order — each builds on the last. After each sub-step, run the type checker to catch missed match arms and type errors: ```bash cargo check -p <crate-name> ``` Replace `<crate-name>` with the crate you just edited — for example, `dbt-adapter-core`, `dbt-xdbc`, `dbt-schemas`, `dbt-auth`, `dbt-adapter`, or `dbt-loader`. These match the names in the crate map. ##### 4.1 — Register the adapter type **Crate:** `crates/dbt-adapter-core/` `AdapterType` is the central enum that identifies your warehouse throughout the entire codebase. Add your variant here first. Rust's `match` expressions must handle every possible variant — there's no catch-all default case. So when you add `AdapterType::MyWarehouse`, every `match adapter_type()` block in the codebase that doesn't handle your variant becomes a **compile error**. Run `cargo check -p <crate>` and the compiler hands you a complete list of exactly what still needs to be written. ```rust // crates/dbt-adapter-core/src/lib.rs pub enum AdapterType { Postgres, Snowflake, Bigquery, Databricks, Redshift, Spark, DuckDB, Fabric, ClickHouse, Athena, Starburst, Trino, Datafusion, Dremio, Oracle, Exasol, // ← already added MyWarehouse, // ← add yours here } ``` Also add a `quote_char` arm in the same file. Use double quotes `'"'` for most warehouses; BigQuery and Databricks use a backtick instead: ```rust // crates/dbt-adapter-core/src/lib.rs fn quote_char(&self) -> char { match self { // ... existing adapters ... Exasol => '"', // double-quote (most warehouses) MyWarehouse => '"', // ← add yours here BigQuery | Databricks => '`', } } ``` ##### 4.2 — Register the ADBC driver **Crate:** `crates/dbt-xdbc/` This is where v2 learns how to find and load your warehouse's ADBC driver at runtime. | File | What to do | Required? | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `src/driver.rs` | Add a variant to the `Backend` enum; add the ADBC library name (for example, `"adbc_driver_exasol"`) and FFI protocol; also define a `LoadStrategy` | **Yes** | | `src/install.rs` | Add CDN download URL and platform strings | No — only if the driver will be distributed through the dbt Labs CDN, which requires separate coordination with dbt Labs. Community adapter drivers are not on the CDN by default. | The `Backend` enum maps to the ADBC shared library name (`lib<name>.so` / `<name>.dll` / `lib<name>.dylib`). You're registering its identity so v2 knows what to load — you're not writing the driver here. For **custom Arrow type mappings** (for example, DuckDB needed this for `HUGEINT` and `UTINYINT`): only add warehouse-specific type handling if your driver returns types that Arrow's standard schema doesn't cover. Most warehouses don't need this. ##### 4.3 — Add your connection profile **Crate:** `crates/dbt-schemas/` | File | What to do | | ------------------------- | ----------------------------------------------------------------------- | | `src/schemas/profiles.rs` | Add a `DbConfig` variant and a config struct with all connection fields | The config struct should include everything a user would put in their `profiles.yml` for your warehouse. Example from Exasol: ```rust #[derive(Debug, Clone, Serialize, Deserialize, Default, DbtSchema, Merge)] pub struct ExasolDbConfig { pub user: Option<String>, #[serde(skip_serializing_if = "Option::is_none", alias = "pass")] pub password: Option<String>, pub host: Option<String>, pub port: Option<StringOrInteger>, pub database: Option<String>, pub schema: Option<String>, pub encryption: Option<bool>, pub certificate_validation: Option<bool>, pub certificate_fingerprint: Option<String>, pub connection_timeout: Option<StringOrInteger>, pub threads: Option<StringOrInteger>, } ``` After adding the config struct, also add `DbConfig::MyWarehouse(Box<MyWarehouseDbConfig>)` as a new variant to the `DbConfig` enum. This registers your new config type so the rest of the codebase knows it exists. Once you do, the compiler will point you at every place that reads from `DbConfig` and needs a new case for your warehouse — follow those errors to wire it in. ##### 4.4 — Add authentication **Crate:** `crates/dbt-auth/` | File | What to do | Required? | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `src/<warehouse>/mod.rs` | Credential resolution — reads config fields, env vars, key files, tokens; builds the ADBC connection URI and credentials | **Yes** | | `src/<warehouse>/init.rs` | Init SQL generation — SQL that must run when the connection opens (for example, `USE SCHEMA`, `SET` statements, and extension loading) | Optional — only for warehouses that need to run SQL on connection open. Most warehouses don't need this. | | `src/lib.rs` | Register the new module with `mod <warehouse>;` and wire it into the auth dispatch match | **Yes** | The auth module turns a `DbConfig` into a live, authenticated ADBC connection. At minimum you need basic credential handling. More sophisticated auth (OAuth, SSO, key-pair) can be added incrementally. The pattern is: read config fields → construct URI → call `builder.with_parse_uri(uri)`, `builder.with_username(user)`, `builder.with_password(password)`. ##### 4.5 — Build the adapter layer **Crate:** `crates/dbt-adapter/` This is the largest step. You're adding warehouse-specific behavior to the shared adapter layer using `match adapter_type()` arms. Simple vs. complex adapters Most adapters — including Exasol, Athena, Trino, Starburst, Dremio, Oracle, and ClickHouse — add match arms directly to the shared files in `src/relation/`. Only adapters with highly custom relation logic (Snowflake's multi-part names and case rules; BigQuery's project/dataset structure) have their own subdirectory under `src/relation/`. If your warehouse uses standard `schema.table` or `database.schema.table` naming with straightforward quoting, you're likely a simple adapter. Start simple and only add complexity if the compiler forces it. ###### Relation type and quoting note The relation crate is currently being verticalized — `Policy` is moving from `relation_object.rs` to `relation_impl.rs`. Verify file paths against the current repo before using them. **File:** `src/relation/relation_object.rs` Add a match arm that constructs a `Relation` with your quoting policy. The policy controls three things: whether the database is included in fully-qualified names, whether the schema is quoted, and whether the identifier is quoted. For example, Exasol uses a 2-part name (`schema.table`, no database prefix) and quotes both: ```rust // src/relation/relation_object.rs Exasol => Box::new(Relation::new_with_policy( Exasol, RelationPath { database: Some(database).filter(|s| !s.is_empty()), schema: Some(schema), identifier, }, relation_type, Policy::new(false, true, true), // ↑ ↑ ↑ // database schema identifier // disabled quoted quoted custom_quoting, None, false, false, )) as Box<dyn BaseRelation>, ``` Decide up front whether your warehouse uses 2-part or 3-part names, and whether identifiers are case-sensitive. For example, Exasol uppercases unquoted identifiers by default — so all catalog lookup SQL uses `upper()` comparisons. **File:** `src/relation/factory.rs` Add your `AdapterType` to the `create_static_relation` match, wiring it to `RelationStatic` (the generic static relation used for Jinja's `api.Relation`): ```rust Databricks | Spark | Fabric | DuckDB | Exasol | Postgres | Redshift | Bigquery => { let relation_type = RelationStatic { adapter_type, quoting }; StaticBaseRelationObject::new(Arc::new(relation_type)) } ``` ###### Catalog introspection **File:** `src/metadata/get_relation.rs` Add a match arm and a function that queries your warehouse's system catalog to look up a single relation by name: ```rust AdapterType::Exasol => exasol_get_relation( adapter, state, ctx, conn, database, schema, identifier, token, ), ``` ```rust fn exasol_get_relation( adapter: &AdapterImpl, state: &State, ctx: &QueryCtx, conn: &mut dyn Connection, database: &str, schema: &str, identifier: &str, token: CancellationToken, ) -> AdapterResult<Option<Box<dyn BaseRelation>>> { let q_schema = schema.to_uppercase(); let q_ident = identifier.to_uppercase(); let sql = format!( "select 'table' as \"type\" from sys.exa_all_tables \ where table_schema = '{q_schema}' and table_name = '{q_ident}' \ union all \ select 'view' from sys.exa_all_views \ where view_schema = '{q_schema}' and view_name = '{q_ident}'" ); // execute, read result, return relation } ``` Use `information_schema` if your warehouse supports standard SQL, or system catalog tables (`sys.*`, `information_schema.*`) as appropriate. ###### Adapter match arms **File:** `src/adapter/adapter_impl.rs` After adding `AdapterType::MyWarehouse`, the compiler will enumerate every `match self.adapter_type()` block that needs a new arm. Most arms simply delegate to an existing pattern. For example: ```rust // Column builder — delegate to postgres-like builder Exasol => Ok(Self::build_postgres_like(field, type_ops)), // Schema column name for listing relations Exasol => "name", // DATA_TYPE column name in information schema AdapterType::Exasol => "DATA_TYPE", // in src/sql_types.rs ``` For capabilities your adapter doesn't support yet (for example, `valid_incremental_strategies`), return `unimplemented!()` — that's fine for an initial community adapter contribution. The reference PR has several of these. ###### Column builder **File:** `src/column/column_builder.rs` Add a match arm for how your warehouse's Arrow record batches map to dbt column objects. Most adapters can delegate to `build_postgres_like`: ```rust Exasol => Ok(Self::build_postgres_like(field, type_ops)), ``` Only implement custom logic if your warehouse has unusual type handling. ##### 4.6 — Write your SQL macros **Crate:** `crates/dbt-loader/` Create a new directory at `src/dbt_macro_assets/dbt-<adapter_type>/`. Registration is automatic The loader discovers adapter packages by scanning `src/dbt_macro_assets/`. You don't need to add any Rust code to register your directory — creating it and the `dbt_project.yml` is sufficient. | File | What to do | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Adapter plugin project definition: `name: dbt_<warehouse>`, `macro-paths: ["macros"]` | | `macros/adapters.sql` | dbt v1 adapter macros: `create_schema`, `drop_schema`, `drop_relation`, `rename_relation`, `truncate_relation`, `create_table_as`, `create_view_as`, `list_schemas`, `check_schema_exists`, `information_schema_name`, `current_timestamp`, `get_columns_in_relation`, `list_relations_without_caching` | v2 uses the same `adapter.dispatch()` pattern as v1. Your macros use the `<warehouse>__` prefix to override defaults: ```sql {% macro exasol__create_table_as(temporary, relation, sql) -%} create or replace table {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__create_view_as(relation, sql) -%} create or replace view {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__drop_relation(relation) -%} {% call statement('drop_relation', auto_begin=False) -%} drop {{ relation.type }} if exists {{ relation }} cascade {%- endcall %} {% endmacro %} {% macro exasol__rename_relation(from_relation, to_relation) -%} {% call statement('rename_relation') -%} rename {{ from_relation.type }} {{ from_relation }} to {{ to_relation.identifier }} {%- endcall %} {% endmacro %} ``` Note: `rename_relation` uses only `to_relation.identifier`, not the full relation — Exasol's `RENAME` syntax doesn't take a fully qualified target. For catalog introspection, use your warehouse's system catalog. For example, Exasol queries `sys.*` instead of `information_schema`: ```sql {% macro exasol__list_relations_without_caching(schema_relation) -%} {% call statement('list_relations_without_caching', fetch_result=True) -%} select '{{ schema_relation.database }}' as "database", table_name as "name", table_schema as "schema", 'table' as "type" from sys.exa_all_tables where upper(table_schema) = upper('{{ schema_relation.schema }}') union all select '{{ schema_relation.database }}' as "database", view_name as "name", view_schema as "schema", 'view' as "type" from sys.exa_all_views where upper(view_schema) = upper('{{ schema_relation.schema }}') {%- endcall %} {{ return(load_result('list_relations_without_caching').table) }} {%- endmacro %} {% macro exasol__information_schema_name(database) -%} sys {%- endmacro %} ``` If your warehouse is similar to an existing one (for example, Postgres-compatible), start by delegating to that dialect's macros and only override where behavior differs: ```sql {% macro mywarehouse__create_table_as(temporary, relation, sql) -%} {{ return(postgres__create_table_as(temporary, relation, sql)) }} {%- endmacro %} ``` *** #### Step 5: Test your adapter ##### Type check after each crate Run the type checker after completing work in each crate to catch missed match arms and type errors: ```bash cargo build -p dbt-adapter-core cargo build -p dbt-xdbc cargo build -p dbt-schemas cargo build -p dbt-auth cargo build -p dbt-adapter cargo build -p dbt-loader ``` ##### End-to-end test Run a real `dbt build` against your warehouse. At minimum, exercise table, view, incremental, and snapshot materializations. A clean `dbt build` on `jaffle-shop-classic` is the standard acceptance bar for a community adapter. ```bash # Build the CLI cargo build --bin dbt # Create a test project (or clone jaffle-shop-classic) ./target/debug/dbt init # Run against your warehouse ./target/debug/dbt build --project-dir <your-project> ``` ##### CI testing CI testing for community adapter PRs is coordinated with the dbt Labs adapters team — the test infrastructure is not publicly distributed. When your PR is ready, reach out in `#adapter-ecosystem` and the adapters team will work with you on warehouse validation. Known gap: CI requires coordination Community contributors cannot run CI independently. dbt Labs' CI pipeline requires warehouse credentials for the target warehouse — which means dbt Labs needs access configs for whatever warehouse you're contributing. The process for this handoff is still being defined. Expect to coordinate closely with the adapters team. If you hit friction, flag it in `#adapter-ecosystem`. *** #### Step 6: Document your adapter Once your adapter is merged and available in a release, document it so users can find and configure it. ##### Write a setup guide Document the `profiles.yml` configuration for your warehouse — what fields are required, what's optional, and example values. Follow the format of existing adapter setup guides on [docs.getdbt.com](http://docs.getdbt.com). Driver installation is critical to document Unlike first-party adapters, your users won't get the driver automatically — dbt v2 won't download it for them. Your setup guide must explain where to get the driver binary and how to install it so dbt v2 can find it at runtime. Without this, users will configure a valid profile and still get a connection error. Include the exact library name dbt v2 looks for (for example, `libadbc_driver_<yourwarehouse>.dylib`) and where to put it. ##### General documentation guidelines * Assume the reader knows dbt fundamentals but is not an expert on your warehouse inner workings. * Include a complete working `profiles.yml` example. * Document any warehouse-specific quirks (for example, 2-part versus 3-part naming and identifier case sensitivity). * Link to the warehouse vendor's ADBC driver documentation. *** #### Step 7: Promote your adapter Your PR must be merged first dbt Labs reviews and merges community adapter PRs into `dbt-labs/dbt`. Wait until the PR is merged and the adapter ships in a published release before directing users to it. ##### Community channels Join the dbt Community Slack and find: * **`#adapter-ecosystem`** — the main channel for adapter developers * **`#db-<yourwarehouse>`** — if a channel exists for your warehouse, let users know v2 support is available. Note: v1 adapter users will still be on the Python-based adapter and will need to migrate. * **`#proj-isv-adapters-in-fusion`** — ping the dbt Labs adapters team here once your PR is up for review. ##### Before you announce Align with the adapters team on: which materializations you're targeting in the initial implementation, any known gaps in your ADBC driver, and timeline. This prevents surprises during review and sets accurate expectations for users. *** #### Reference: File-by-file implementation guide A community contributed v2 adapter touches roughly 13 files, all in the public [dbt-labs/dbt](https://github.com/dbt-labs/dbt) repo. The "Exasol example" column shows what it looks like in practice — substitute your warehouse name and system catalog throughout. | Generic path | What it does | Exasol example | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `.changes/unreleased/Features-*.yaml` | Changelog entry | Same for all adapters | | `crates/dbt-adapter-core/src/lib.rs` | Add `AdapterType::<Warehouse>` variant and `quote_char` arm | `AdapterType::Exasol` | | `crates/dbt-xdbc/src/driver.rs` | Add `Backend::<Warehouse>` variant — the ADBC library name and FFI protocol | `Backend::Exasol`, library name `"adbc_driver_exasol"` | | `crates/dbt-schemas/src/schemas/profiles.rs` | `<Warehouse>DbConfig` struct + `DbConfig::<Warehouse>` variant, wired into all config match arms | `ExasolDbConfig` struct + `DbConfig::Exasol` variant | | `crates/dbt-auth/src/<warehouse>/mod.rs` | Auth module — reads config fields, resolves credentials, constructs connection URI | `src/exasol/mod.rs` — builds URI from host, port, user/pass, TLS options | | `crates/dbt-adapter/src/adapter/adapter_impl.rs` | All exhaustive `match adapter_type()` arms — most delegate to existing patterns; use `unimplemented!()` for unimplemented features | Exasol adds ~10 arms, most are 1–2 lines | | `crates/dbt-adapter/src/column/column_builder.rs` | Column builder match arm — maps Arrow record batches to dbt column objects | `Exasol => Ok(Self::build_postgres_like(...))` | | `crates/dbt-adapter/src/metadata/get_relation.rs` | Catalog lookup function — queries system tables to find a relation by name | `exasol_get_relation()` — queries `sys.exa_all_tables` / `sys.exa_all_views` | | `crates/dbt-adapter/src/relation/relation_object.rs` | Match arm — quoting policy and whether database prefix is included in fully-qualified names | `Policy::new(false, true, true)` — database disabled, schema + identifier quoted | | `crates/dbt-adapter/src/relation/factory.rs` | Add warehouse to `RelationStatic` arm in `create_static_relation` | Exasol added alongside Postgres, Redshift, DuckDB, and more. | | `crates/dbt-adapter/src/sql_types.rs` | Metadata column name arms — `DATA_TYPE` and schema listing column names | Exasol uses standard `DATA_TYPE` column name | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Macro plugin project definition | `dbt-exasol/dbt_project.yml` | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/macros/adapters.sql` | All required adapter macros | `dbt-exasol/macros/adapters.sql` — uses `sys.*` instead of `information_schema` | #### Reference: Useful commands ```bash # Type check a specific crate cargo build -p <crate-name> # Build the CLI binary cargo build --bin dbt # Run a full dbt build against your warehouse ./target/debug/dbt build --project-dir <your-project> # Free disk space cargo clean ``` --- ### Contribute to dbt adapters The dbt Community helps analytics practitioners share their knowledge, help others, and collectively drive forward the discipline of analytics engineering. There are opportunities here for everyone to contribute, whether you're at the beginning of your analytics engineering journey or you are a seasoned data professional. This section explains how you can contribute to existing adapters or create a new adapter. ##### Contribute to a pre-existing adapter Community-supported plugins are works in progress, and you can contribute by testing and writing code. If you're interested in contributing: * Join both the dedicated channel, [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM), in [dbt Slack](https://community.getdbt.com/) and the channel for your adapter's data store. See the **Slack Channel** link in the [dbt v1 platform](https://docs.getdbt.com/docs/local/profiles.yml.md) pages. * Review open issues in the plugin's source repository. Use the relevant **GitHub repo** link in the [dbt v1 platform](https://docs.getdbt.com/docs/local/profiles.yml.md) pages. ##### Create a new v2 adapter v2 is a Rust-based rewrite of the dbt engine built around a single monorepo. Instead of maintaining a separate Python package per warehouse, all adapters live together, organized by feature area rather than warehouse, inside `dbt-labs/dbt`. A bug fix in authentication or relation logic benefits every adapter at once, and your contribution makes the entire ecosystem stronger. ADBC drivers handle connection management—pre-compiled binaries you register, not write. You build the warehouse-specific logic: credentials, relation naming, SQL macros, and catalog queries across roughly 13 files in total. Contributing a v2 adapter means opening a pull request directly against the monorepo. You'll register your warehouse in the `AdapterType` enum, implement credential and relation types in Rust, write Jinja macros for your SQL dialect, and wire up basic integration tests against a real warehouse. The Rust compiler's match exhaustiveness enforcement acts as your to-do list — every unhandled variant is a compile error until your adapter is complete. For a step-by-step walkthrough, see [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md). --- ### Conversion metrics Conversion metrics let you measure how often one event leads to another for a specific entity within a defined time window. For example, you can track how often a user (entity) who visits your site (base event) makes a purchase (conversion event) within 7 days (time window). To set this up, you’ll specify both the time range and the entity that links/joins the two events. Conversion metrics are different from [ratio metrics](https://docs.getdbt.com/docs/build/ratio.md) because you need to include an entity in the pre-aggregated join. #### Parameters The specification for conversion metrics is as follows: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------- | | `name` | The name of the metric. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of metric. Set as `conversion` for conversion metrics. | Required | String | | `label` | The display label for the metric. Accepts plain text, spaces, and quotes. | Optional | String | | `config` | Configuration settings for the metric. | Optional | Dict | | `config.group` | The group the metric belongs to. | Optional | String | | `config.tags` | Tags associated with the metric. | Optional | List | | `config.meta` | Metadata for the metric. | Optional | Dict | | `entity` | The entity for each conversion event. | Required | String | | `calculation` | Method of calculation. Either `conversion_rate` or `conversions`. Defaults to `conversion_rate`. | Optional | String | | `base_metric` | The base metric name or configuration for the conversion event. Can be a string (metric name) or a dict (for additional customization). | Required | String or Dict | | `base_metric.name` | The name of the base metric (when using dict format). | Required | String | | `base_metric.filter` | Filter to apply to the base metric (when using dict format). | Optional | String | | `base_metric.alias` | Alias for the base metric (when using dict format). | Optional | String | | `conversion_metric` | The conversion metric name or configuration. Can be a string (metric name) or a dict (for additional customization). | Required | String or Dict | | `conversion_metric.name` | The name of the conversion metric (when using dict format). | Required | String | | `conversion_metric.filter` | Filter to apply to the conversion metric (when using dict format). | Optional | String | | `conversion_metric.alias` | Alias for the conversion metric (when using dict format). | Optional | String | | `window` | The time window for the conversion event (such as `7 days`, `1 week`, `3 months`). Defaults to infinity. | Optional | String | | `constant_properties` | List of properties to hold constant between base and conversion events. Can be a dimension or entity. | Optional | List | | `constant_properties.base_property` | The dimension or entity of the semantic model linked to the `base_metric`. | Required | String | | `constant_properties.conversion_property` | The dimension or entity of the semantic model linked to the `conversion_metric`. | Required | String | Refer to [additional settings](#additional-settings) to learn how to customize conversion metrics with settings for null values, calculation type, and constant properties. The following code example displays the complete specification for conversion metrics and details how they're applied: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: your_model_name semantic_model: enabled: true .... rest of configs.... metrics: - name: my_conversion_metric description: "Tracks how often a base event leads to a conversion event for an entity" label: "My conversion metric" type: conversion entity: my_primary_entity # Required; the entity the conversion is tracked for calculation: conversion_rate # Optional; conversion_rate | conversions base_metric: my_base_event_metric # Required; metrics defined in another semantic model conversion_metric: my_conversion_event_metric # Required; metrics defined in another semantic model window: 7 days # Optional; defines the time window for conversion constant_properties: # Optional; list of constant properties - base_property: my_dimension_or_entity conversion_property: my_dimension_or_entity ``` #### Conversion metric example The following example will measure conversions from website visits (`VISITS` table) to order completions (`BUYS` table) and calculate a conversion metric for this scenario step by step. Suppose you have two semantic models, `VISITS` and `BUYS`: * The `VISITS` table represents visits to an e-commerce site. * The `BUYS` table represents someone completing an order on that site. The underlying tables look like the following: `VISITS`<br />Contains user visits with `USER_ID` and `REFERRER_ID`. | DS | USER\_ID | REFERRER\_ID | | ---------- | -------- | ------------ | | 2020-01-01 | bob | facebook | | 2020-01-04 | bob | google | | 2020-01-07 | bob | amazon | `BUYS`<br />Records completed orders with `USER_ID` and `REFERRER_ID`. | DS | USER\_ID | REFERRER\_ID | | ---------- | -------- | ------------ | | 2020-01-02 | bob | facebook | | 2020-01-07 | bob | amazon | Next, define a conversion metric as follows: (Applies to dbt v1.12 and later) ```yaml models: - name: your_model_name semantic_model: enabled: true .... rest of configs.... metrics: - name: visit_to_buy_conversion_rate_7d description: "Conversion rate from visiting to transaction in 7 days" type: conversion label: Visit to buy conversion rate (7-day window) entity: user calculation: conversion_rate base_metric: name: visits filter: {{ Dimension('visits__referrer_id') }} = 'facebook' conversion_metric: buys window: 7 days ``` To calculate the conversion, link the `BUYS` event to the nearest `VISITS` event (or closest base event). The following steps explain this process in more detail: ##### Step 1: Join `VISITS` and `BUYS` This step joins the `BUYS` table to the `VISITS` table and gets all combinations of visits-buys events that match the join condition where buys occur within 7 days of the visit (any rows that have the same user and a buy happened at most 7 days after the visit). The SQL generated in these steps looks like the following: ```sql select v.ds, v.user_id, v.referrer_id, b.ds, b.uuid, 1 as buys from visits v inner join ( select *, uuid_string() as uuid from buys -- Adds a uuid column to uniquely identify the different rows ) b on v.user_id = b.user_id and v.ds <= b.ds and v.ds > b.ds - interval '7 days' ``` The dataset returns the following (note that there are two potential conversion events for the first visit): | V.DS | V.USER\_ID | V.REFERRER\_ID | B.DS | UUID | BUYS | | ---------- | ---------- | -------------- | ---------- | ----- | ---- | | 2020-01-01 | bob | facebook | 2020-01-02 | uuid1 | 1 | | 2020-01-01 | bob | facebook | 2020-01-07 | uuid2 | 1 | | 2020-01-04 | bob | google | 2020-01-07 | uuid2 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | ##### Step 2: Refine with window function Instead of returning the raw visit values, use window functions to link conversions to the closest base event. You can partition by the conversion source and get the `first_value` ordered by `visit ds`, descending to get the closest base event from the conversion event: ```sql select first_value(v.ds) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as v_ds, first_value(v.user_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as user_id, first_value(v.referrer_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as referrer_id, b.ds, b.uuid, 1 as buys from visits v inner join ( select *, uuid_string() as uuid from buys ) b on v.user_id = b.user_id and v.ds <= b.ds and v.ds > b.ds - interval '7 day' ``` The dataset returns the following: | V.DS | V.USER\_ID | V.REFERRER\_ID | B.DS | UUID | BUYS | | ---------- | ---------- | -------------- | ---------- | ----- | ---- | | 2020-01-01 | bob | facebook | 2020-01-02 | uuid1 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | This workflow links the two conversions to the correct visit events. Due to the join, you end up with multiple combinations, leading to fanout results. After applying the window function, duplicates appear. To resolve this and eliminate duplicates, use a distinct select. The UUID also helps identify which conversion is unique. The next steps provide more detail on how to do this. ##### Step 3: Remove duplicates Instead of regular select used in the [Step 2](#step-2-refine-with-window-function), use a distinct select to remove the duplicates: ```sql select distinct first_value(v.ds) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as v_ds, first_value(v.user_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as user_id, first_value(v.referrer_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as referrer_id, b.ds, b.uuid, 1 as buys from visits v inner join ( select *, uuid_string() as uuid from buys ) b on v.user_id = b.user_id and v.ds <= b.ds and v.ds > b.ds - interval '7 day'; ``` The dataset returns the following: | V.DS | V.USER\_ID | V.REFERRER\_ID | B.DS | UUID | BUYS | | ---------- | ---------- | -------------- | ---------- | ----- | ---- | | 2020-01-01 | bob | facebook | 2020-01-02 | uuid1 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | You now have a dataset where every conversion is connected to a visit event. To proceed: 1. Sum up the total conversions in the "conversions" table. 2. Combine this table with the "opportunities" table, matching them based on group keys. 3. Calculate the conversion rate. ##### Step 4: Aggregate and calculate Now that you’ve tied each conversion event to a visit, you can calculate the aggregated conversions and opportunities (Applies to dbt v1.12 and later) simple metric. Then, you can join them to calculate the actual conversion rate. The SQL to calculate the conversion rate is as follows: ```sql select coalesce(subq_3.metric_time__day, subq_13.metric_time__day) as metric_time__day, cast(max(subq_13.buys) as double) / cast(nullif(max(subq_3.visits), 0) as double) as visit_to_buy_conversion_rate_7d from ( -- base select metric_time__day, sum(visits) as visits from ( select date_trunc('day', first_contact_date) as metric_time__day, 1 as visits from visits ) subq_2 group by metric_time__day ) subq_3 full outer join ( -- conversion select metric_time__day, sum(buys) as buys from ( -- ... -- The output of this subquery is the table produced in Step 3. The SQL is hidden for legibility. -- To see the full SQL output, add --explain to your conversion metric query. ) subq_10 group by metric_time__day ) subq_13 on subq_3.metric_time__day = subq_13.metric_time__day group by metric_time__day ``` ##### Additional settings Use the following additional settings to customize your conversion metrics: * **Null conversion values:** Set null conversions to zero using `fill_nulls_with`. Refer to [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) for more info. * **Calculation type:** Choose between showing raw conversions or conversion rate. * **Constant property:** Add conditions for specific scenarios to join conversions on constant properties. ##### Set null conversion events to zero To return zero in the final data set, you can set the value of a null conversion event to zero instead of null. You can add the `fill_nulls_with` parameter to your conversion metric definition like this: (Applies to dbt v1.12 and later) ```yaml metrics: - name: visits type: simple agg: count expr: visit_id fill_nulls_with: 0 # set null conversion values to zero in a simple metric - name: buys type: simple agg: count expr: purchase_id fill_nulls_with: 0 - name: visit_to_buy_conversion_rate_7_day_window description: "Conversion rate from viewing a page to making a purchase" type: conversion label: Visit to buy conversion rate (7 day window) entity: user calculation: conversions base_metric: visits conversion_metric: buys window: 7 days ``` This will return the following results: [![Conversion metric with fill nulls with parameter](/img/docs/dbt-platform/semantic-layer/conversion-metrics-fill-null.png?v=2 "Conversion metric with fill nulls with parameter")](#)Conversion metric with fill nulls with parameter Refer to [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) for more info. ##### Set calculation type parameter Use the conversion calculation parameter to either show the raw number of conversions or the conversion rate. The default value is the conversion rate. You can change the default to display the number of conversions by setting the `calculation: conversion` parameter: (Applies to dbt v1.12 and later) ```yaml metrics: - name: visit_to_buy_conversions_1_week_window description: "Visit to buy conversions" type: conversion label: Visit to buy conversions (1 week window) entity: user calculation: conversions base_metric: visits conversion_metric: buys window: 1 week fill_nulls_with: 0 ``` ##### Set constant property *Refer to [Amplitude's blog posts on constant properties](https://amplitude.com/blog/holding-constant) to learn about this concept.* You can add a constant property to a conversion metric to count only those conversions where a specific dimension or entity matches in both the base and conversion events. For example, if you're at an e-commerce company and want to answer the following question: * *How often did visitors convert from `View Item Details` to `Complete Purchase` with the same product in each step?* <br /> * This question is tricky to answer because users could have completed these two conversion milestones across many products. For example, they may have viewed a pair of shoes, then a T-shirt, and eventually checked out with a bow tie. This would still count as a conversion, even though the conversion event only happened for the bow tie. Back to the initial questions, you want to see how many customers viewed an item detail page and then completed a purchase for the *same* product. In this case, you want to set `product_id` as the constant property. You can specify this in the configs as follows: (Applies to dbt v1.12 and later) ```yaml metrics: - name: view_item_detail_to_purchase_with_same_item description: "Conversion rate for users who viewed the item detail page and purchased the item" type: conversion label: View item detail > Purchase entity: user calculation: conversions base_metric: view_item_detail conversion_metric: purchase window: 1 week constant_properties: - base_property: product conversion_property: product ``` You will add an additional condition to the join to make sure the constant property is the same across conversions. ```sql select distinct first_value(v.ds) over (partition by buy_source.ds, buy_source.user_id, buy_source.session_id order by v.ds desc rows between unbounded preceding and unbounded following) as ds, first_value(v.user_id) over (partition by buy_source.ds, buy_source.user_id, buy_source.session_id order by v.ds desc rows between unbounded preceding and unbounded following) as user_id, first_value(v.referrer_id) over (partition by buy_source.ds, buy_source.user_id, buy_source.session_id order by v.ds desc rows between unbounded preceding and unbounded following) as referrer_id, buy_source.uuid, 1 as buys from {{ source_schema }}.fct_view_item_details v inner join ( select *, {{ generate_random_uuid() }} as uuid from {{ source_schema }}.fct_purchases ) buy_source on v.user_id = buy_source.user_id and v.ds <= buy_source.ds and v.ds > buy_source.ds - interval '7 day' and buy_source.product_id = v.product_id --Joining on the constant property product_id ``` #### Related docs * [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Cost Insights dbt platform | Enterprise, Enterprise+ Cost Insights shows estimated costs and compute time for your dbt projects and models directly in the dbt platform, so you can measure and share the impact of optimizations like [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) and [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md). State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) and [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) make your dbt workflows more efficient by reusing models and tests instead of running full rebuilds. When either is enabled, Cost Insights helps you demonstrate the resulting cost reductions and efficiency gains. These cost and cost reduction estimates are based on a retroactive analysis of runs after you enable dbt State or state-aware orchestration. They reflect actual historical usage, *not* forecasts of future costs or cost reductions. With Cost Insights, you can see: * **How much your dbt models cost to run**: See the compute cost and times for each model and job in your warehouse's native units. * **The cost reductions from using dbt State or state-aware orchestration**: Understand the cost reduction when dbt State or state-aware orchestration reuses unchanged models. * **Cost trends over time**: Track your warehouse spend and optimization impact across your dbt projects. * **Asset type filtering**: On Cost Insights charts (**Cost**, **Usage**, **Query run time**, **Builds**), use the **Assets** dropdown menu to filter data by **Models**, **Tests**, or **All**. Each tab keeps its own selection. * **Per-job cost breakdown**: In the Cost Insights table view, use the **All** and **Jobs** buttons to switch between an aggregated view and a per-job cost breakdown. The Cost Insights section is available in different dbt platform areas and lets you view your cost data and the impact of dbt State and state-aware orchestration optimizations across various dimensions: * [Project dashboard](https://docs.getdbt.com/docs/explore/explore-cost-data.md#project-dashboard) * [Catalog on Model page](https://docs.getdbt.com/docs/explore/explore-cost-data.md#model-performance-in-catalog) * [Job details page](https://docs.getdbt.com/docs/explore/explore-cost-data.md#job-details) [![Cost Insights in the project dashboard](/img/docs/dbt-platform/cost-insights/cost-insights-project.png?v=2 "Cost Insights in the project dashboard")](#)Cost Insights in the project dashboard [![Cost Insights in Catalog](/img/docs/dbt-platform/cost-insights/cost-insights-model.png?v=2 "Cost Insights in Catalog")](#)Cost Insights in Catalog [![Cost Insights in job details](/img/docs/dbt-platform/cost-insights/cost-insights-job.png?v=2 "Cost Insights in job details")](#)Cost Insights in job details #### Prerequisites To view cost data, ensure you have: * One of the roles listed in [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * A supported data warehouse: * Snowflake * BigQuery * Databricks * Amazon Redshift [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") For setup instructions, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md). #### Understanding cost and reduction estimates note Cost estimates are intended for visibility and optimization, not billing reconciliation. dbt calculates the cost of running your dbt models using your data warehouse's usage metadata and billing context. dbt computes costs daily using up to the *last seven days of available data*. ##### Warehouse-specific logic The following sections explain how costs are calculated for each supported warehouse. Expand each section to view the details.  Snowflake dbt computes Snowflake query costs using Snowflake's query attribution data and your credit price (`price_per_credit`). dbt pulls the `price_per_credit` value directly from Snowflake when available; otherwise, dbt uses the configured or default value in the dbt platform. For more information about configuring or viewing these values, see [Configure Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional). Formula: ```text credits_per_query * price_per_credit ``` Where: * `credits_per_query` — Cloud services, compute, and query acceleration credits attributed to the query. * For standard and Generation 2 warehouses, dbt sources this value from [`QUERY_ATTRIBUTION_HISTORY`](https://docs.snowflake.com/en/sql-reference/account-usage/query_attribution_history). * For [Adaptive Warehouses](https://docs.snowflake.com/en/user-guide/warehouses-adaptive), dbt sources this value from [`QUERY_METERING_HISTORY`](https://docs.snowflake.com/en/sql-reference/account-usage/query_metering_history) — refer to [Configure platform metadata credentials](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#snowflake) for required permissions. * `price_per_credit` — Your Snowflake credit price (from Snowflake system tables when available, otherwise from your configured input or the default rate). Snowflake attribution limitation Snowflake doesn't attribute cost to queries that run in roughly 100 milliseconds or less. As a result, Cost Insights totals may be *lower* than the compute spend shown in your Snowflake billing dashboards. For more information, see the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/account-usage/query_attribution_history#usage-notes).  BigQuery BigQuery does not expose per-query cost directly in system tables. Instead, dbt estimates cost by combining *query usage* with a *pricing input* (either from your configuration or the default rate). * **On-demand pricing** The cost is determined by how much data each query processes. The usage shows the amount of that data billed for the query. Formula: ```text data_processed_per_query * price_per_tib ``` Where: * `data_processed_per_query` - Total data billed for the query (normalized to TiB). dbt sources this value from `information_schema.jobs.total_bytes_billed`. For more information, see the [BigQuery documentation](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs). * `price_per_tib` - BigQuery on-demand price per TiB (from your configuration or the default rate). * **Capacity pricing (reservations)** The cost is determined by how long each query runs on reserved compute. The usage shows the amount of that reserved compute time consumed by a query. Formula: ```text compute_time_per_query * price_per_slot_hour ``` Where: * `compute_time_per_query` - Total slot time used by the query (in hours). dbt sources this value from `information_schema.jobs.total_slot_ms`. For more information, see the [BigQuery documentation](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs). * `price_per_slot_hour` - BigQuery capacity price per slot-hour (from your configuration or the default rate) * **Cached queries** Queries served from cache do not consume compute and are counted as $0.  Databricks Databricks does not directly attribute usage to individual queries. Instead, dbt estimates per-query cost by proportionally allocating Databricks Units (DBUs) based on how long each query ran during a billing period. * Queries that run longer receive a larger share of usage. * Usage is converted to dollars using your list price. Formula: ```text usage_per_query * cost_per_dbu ``` Where: * `usage_per_query` - DBUs attributed to the query. * `cost_per_dbu` - Dollar cost per DBU for the relevant stock-keeping unit. For information about the pricing system table, see the [Databricks documentation](https://docs.databricks.com/aws/en/admin/system-tables/pricing). Databricks reports usage in billing windows. These windows are periods of time where a compute resource consumed a known number of DBUs. Queries have their own start and end times. For information about the billing usage system table, see the [Databricks documentation](https://docs.databricks.com/aws/en/admin/system-tables/billing). To attribute usage to queries: 1. dbt identifies which billing windows each query overlaps. 2. dbt calculates how long the query ran during each window. 3. dbt allocates DBUs *proportionally* based on the query’s share of total execution time on that compute resource during the same window. Conceptually: ```text DBUs_in_window * (query_runtime / total_query_runtime_in_window) ``` dbt sums this across all overlapping windows to get `usage_per_query`.  Amazon Redshift[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Cost Insights supports both Amazon Redshift Serverless and provisioned cluster deployments. dbt detects your deployment type automatically when testing the connection. note On Redshift, dbt attributes query costs using the comments it automatically injects into each query. If another process removes or replaces these comments, dbt can't tie the query back to its model, and its cost won't be attributed. Make sure nothing in your Redshift environment removes or replaces dbt's query comments. * **Redshift Serverless** dbt estimates cost by identifying which Redshift Processing Unit (RPU) billing periods overlap with each query's execution time and attributing the proportional share of RPU-hours to the query. Formula: ```text rpu_hours_per_query * rpu_price_per_hour ``` Where: * `rpu_hours_per_query` - RPU-hours attributed to the query based on its proportional overlap with each billing period. dbt sources billing period data from `SYS_SERVERLESS_USAGE`. For more information, see the [Amazon Redshift documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-billing.html). * `rpu_price_per_hour` - Your RPU price per hour (from your configured value in [Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional)). * **Redshift Provisioned** dbt estimates cost based on how long each query ran and how many nodes your cluster has. Formula: ```text elapsed_time_hours * node_count * node_price_per_hour ``` Where: * `elapsed_time_hours` - Query execution time in hours. dbt sources this from `SYS_QUERY_HISTORY`. * `node_count` - Number of nodes in your cluster. dbt sources this from `STV_SLICES`. * `node_price_per_hour` - Your price per node per hour (from your configured value in [Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional)). **Additional considerations:** * **Data retention**: Redshift system tables (`SYS_QUERY_HISTORY`, `SYS_SERVERLESS_USAGE`) retain only seven days of history. Cost data for Redshift may cover a shorter window than other warehouses. dbt calculates costs for whatever data is available within that window. * **Pricing required**: There are no default price values for Redshift. Costs will appear as $0 until you configure `rpu_price_per_hour` (serverless) or `node_price_per_hour` (provisioned) in [Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional). Concurrent query costs Redshift attributes cost to each query as if it ran alone on the cluster. If queries run concurrently, the sum of individual query costs may exceed your actual bill. ##### Cost reduction calculation dbt calculates cost reductions by comparing actual costs to what costs would have been *without model reuse*. To do this, dbt uses data from the last seven days (where available) and performs the following steps: 1. Calculates the average cost per model build. 2. Counts how many times a model was reused instead of rebuilt. 3. Multiplies the reused model count by the average cost per build to determine total cost reduction. Formula: ```text average_cost_per_build * reuse_count ``` dbt calculates reductions per model and per deployment environment (production and staging), based on recent historical runs. Additional notes: * dbt calculates estimated costs and savings daily. * Pricing inputs come from warehouse system tables (where available), connection-level configuration, or default list prices. ###### Example The following example shows how dbt calculates cost reductions. Looking back seven days, assuming a model runs on two distinct days: | Day | Total cost | Total executions | | --------- | ---------- | ---------------- | | Day 1 | $5 | 5 | | Day 2 | $10 | 10 | | **Total** | **$15** | **15** | <br /> The average cost per execution: $15 ÷ 15 runs = $1 per run If the model was *reused* eight times instead of rebuilt during this same period, the estimated cost reduction is: $1 average cost per run \* 8 reuses = $8 #### Considerations Keep the following in mind when using Cost Insights: **Data collection and refresh** * Cost Insights uses your platform metadata credentials to access warehouse system tables. No separate credentials are needed beyond the platform metadata setup. * You need sufficient [permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials) to query warehouse metadata tables. * Cost data is calculated *once per day* by a scheduled job that runs at approximately 17:00 UTC. * The data collection job processes completed calendar days only. It does not include the current day because warehouse usage data may still be incomplete. * Jobs that ran yesterday (or earlier) will have cost data available after the next daily refresh. * Jobs that ran today will not have cost data until the following day’s refresh, regardless of what time they ran. * If you don’t see cost data for a recent job, make sure at least one full calendar day has passed since it ran. The **Updated** badge in the **Cost Insights** section shows when the last refresh occurred. **Cost accuracy** * dbt calculates costs using warehouse-reported usage data and applies default credit or compute costs based on standard warehouse pricing. * If you have custom pricing agreements with your warehouse provider, override the default values in your account settings to ensure accurate cost reporting. For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional). * Update your cost variables whenever your warehouse pricing contracts change to maintain accurate tracking. * Changes to cost variables only apply to future calculations — historical cost data remains unchanged. **Optimization data** * Optimization and usage reduction data is available once dbt State or state-aware orchestration is enabled and begins reusing models across runs. * For accounts already using dbt State or state-aware orchestration, run at least one full model build within the last 10 days before enabling Cost Insights to establish a baseline for cost reduction calculations. If you don't see cost reduction data, run a full build to establish the baseline. * Cost Insights currently calculates estimated reductions in warehouse compute usage at the model level and will expand to include tests and seeds in the future. **Exporting data** * You can export cost data as a CSV file for further analysis and reporting. For more information, see [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md). #### Related FAQs Why might my actual warehouse costs differ from displayed costs? Cost Insights shows estimates based on warehouse-reported usage and your configured pricing variables. These estimates are based on a retroactive analysis of historical runs and reflect actual usage, *not* forecasts of future costs. Adjustments and differences may occur if: * Your warehouse has custom pricing that differs from the default compute credit unit. * There are discounts or credits applied at the billing level that aren't reflected in usage tables. * Costs include other charges beyond compute. Costs Insights in the dbt platform is designed to be directionally accurate, showing you dbt-specific components rather than matching your billing exactly. How often is cost data refreshed? Cost data refreshes daily and reflects the previous day's usage. This means there is a lag of up to one day between when a job runs and when its cost data appears in Cost Insights. How do I troubleshoot if cost data isn't appearing? If cost data isn't appearing in Cost Insights, check the following: * Verify that platform metadata credentials are configured in your account settings and that the credential test is passing. For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials). * Ensure you have one of the required permissions to view cost data. For more information, see [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * Confirm that at least one job is running in a production environment. Cost data only appears after jobs have executed. * Cost data refreshes daily and reflects the previous day's usage, which means there is a lag of up to one day between when a job runs and when its cost data appears. If you just ran a job, wait until the next day to see the data. * After enabling Cost Insights, dbt looks back 10 days to build baselines for cost reduction calculations. If you don't see cost reduction data, ensure you have sufficient job history within the last 10 days. Does the Cost Insights feature incur warehouse costs? dbt issues lightweight, read-only queries against your warehouse to retrieve metadata and to power features such as Cost Insights. dbt scopes and filters these queries to minimize impact, and most customers see negligible costs (typically on the order of cents). How does increasing job frequency affect cost reduction estimates? Cost reduction metrics reflect how dbt optimizes compute costs by reusing existing results instead of running the same model again. When you increase your job run frequency (for example, because performance improvements make it easier to schedule jobs more often), dbt has more opportunities to reuse models. As reuse increases, dbt optimizes more compute, which means your reported cost reductions may also increase. This metric shows the efficiency impact of reuse within your current workload. It reflects the compute costs that dbt reduces by reusing models instead of rebuilding them, rather than showing your total warehouse spend reduction. What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)** as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). --- ### Creating metrics After building [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md), it's time to start adding metrics. This page explains the different supported metric types you can add to your dbt project. (Applies to dbt v1.12 and later) You can define metrics in two locations within your `models` YAML files: * **Within semantic models** (under `models` -> `semantic_model` -> `metrics`) for metrics that use dimensions from a single semantic model. Recommended for simple metrics. * **At the top level** (under a separate `metrics` key) for advanced metrics (cumulative, ratio, derived, conversion) that reference metrics from different semantic models. Simple metrics can't be defined at the top level. Define metrics in YAML files within your dbt project, and not in a `config` block on a model. #### Parameters The keys for metrics parameters are: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------ | | `name` | Provide the reference name for the metric. This name must be a unique metric name and can consist of lowercase letters, numbers, and underscores. | Required | String | | `description` | Describe your metric. | Optional | String | | `type` | Define the type of metric, which can be `conversion`, `cumulative`, `derived`, `ratio`, or `simple`. | Required | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `config` | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your metric. Supports [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md), and [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) configurations. | Optional | Dict | | `filter` | You can optionally add a [filter](#filters) string to any metric type, applying filters to dimensions, entities, time dimensions, or other metrics during metric computation. Consider it as your WHERE clause. | Optional | String | ##### Type-specific parameters Each metric type has additional specific parameters: * **Simple metrics**: `agg` (required), `expr`, `percentile`, `percentile_type`, `non_additive_dimension`, `agg_time_dimension`, `join_to_timespine`, `fill_nulls_with` * **Cumulative metrics**: `input_metric` (required), `window`, `grain_to_date`, `period_agg` * **Derived metrics**: `expr` (required), `input_metrics` (required) * **Ratio metrics**: `numerator` (required), `denominator` (required) * **Conversion metrics**: `entity` (required), `calculation` (required), `base_metric` (required), `conversion_metric` (required), `window`, `constant_properties` Refer to the following sections about each metric type for detailed information on type-specific parameters. ##### Example Here's a complete example of the metrics spec configuration: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true name: my_semantic_model # Define columns, dimensions, entities... columns: - name: my_column # column config... # Metrics defined within the semantic model metrics: - name: my_simple_metric description: A metric description type: simple label: My Simple Metric agg: sum expr: revenue_column # Optional, defaults to metric name config: meta: owner: "@analytics_team" # Advanced metrics that reference metrics from different semantic models metrics: - name: my_advanced_cumulative_metric type: cumulative window: 1 week input_metric: name: my_metric_name_from_another_semantic_model filter: my_filter alias: my_metric_name_a_week_ago_in_another_semantic_model ``` 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Default granularity for metrics (Applies to dbt v1.12 and later) Set the native grain on the time dimension column and choose the rollup grain at query time. * Define the data’s grain on the time dimension column with granularity (for example, hour, day, month). * Metrics inherit the model’s default aggregation time dimension (set at the model level), and you can optionally override it per metric with `agg_time_dimension`. * When querying, pick the rollup grain (day/week/month/year) in your BI tool or with the [Semantic Layer API](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). You can only roll up to grains coarser than or equal to the column’s native grain. ##### Example * `order_time` is stored at hourly grain. * The `orders` metric uses that time dimension; you roll it up to month at query time (not in the metric spec). * A second metric shows how to point at a different time dimension using `agg_time_dimension`. models/file\_name.yml ```yaml models: - name: orders semantic_model: enabled: true # Model-level default aggregation time dimension (optional) agg_time_dimension: order_time columns: - name: order_time granularity: hour # Native grain of the column dimension: type: time - name: created_at granularity: day dimension: type: time metrics: - name: orders type: simple agg: count expr: 1 # No time_granularity here. Roll up to month at query time. - name: orders_by_created_day type: simple agg: count expr: 1 agg_time_dimension: created_at # Use an alternate time dimension for this metric ``` #### Conversion metrics [Conversion metrics](https://docs.getdbt.com/docs/build/conversion.md) help you track when a base event and a subsequent conversion event occur for an entity within a set time period. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: my_conversion_metric description: Conversion metric description type: conversion label: My conversion metric entity: my_primary_entity # Required calculation: conversion_rate # conversion_rate | conversions base_metric: simple_metric # Required, must refer to metric name or metric dict conversion_metric: my_simple_metric_that_uses_the_other_time_dimension # Required, must refer to metric name or metric dict window: 1 week # Optional time window constant_properties: # Optional - base_property: dimension_or_entity_name # Dimension or entity from base metric's semantic model conversion_property: dimension_or_entity_name # Dimension or entity from conversion metric's semantic model # For conversion metrics referencing metrics from different semantic models metrics: - name: my_advanced_conversion_metric config: group: example_group tags: - example_tag - another_tag meta: owner: "@docs-team" type: conversion entity: my_primary_entity # required for conversion metrics calculation: conversion_rate # conversion_rate | conversions # required for conversion metrics base_metric: my_simple_metric # str | metric dict (if filters are needed) conversion_metric: my_other_metric_from_another_semantic_model # str | metric dict (if filters are needed) window: 1 week # optional, as above constant_properties: # optional - DIMENSION OR ENTITY - base_property: my_primary_entity # Required. A reference to a dimension/entity of the semantic model linked to the base_metric conversion_property: another_semantic_model_categorical_dimension ``` #### Cumulative metrics (Applies to dbt v1.12 and later) [Cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md) aggregate a simple metric over a given period. If no period is specified, the window will accumulate the simple metric over all of the recorded time period. Note that you will need to create the [time spine model](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) before you add cumulative metrics. models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: active_users type: simple agg: count_distinct expr: user_id - name: wau_rolling_7 type: cumulative label: Weekly active users description: Rolling 7 day window of active users input_metric: active_users # Required - must refer to metric name or dict window: 7 days # Optional - omit for infinite window (all time) join_to_timespine: true # Optional fill_nulls_with: 0 # Optional # For cumulative metrics using metrics from different semantic models metrics: - name: my_advanced_cumulative_metric type: cumulative label: Advanced cumulative metric input_metric: name: metric_from_another_model filter: "{{ Dimension('entity__dimension') }} > 0" alias: filtered_metric window: 1 week # Optional: can specify window OR grain_to_date, not both # grain_to_date: month # Alternative to window # period_agg: first # Optional: first | last | average ``` #### Derived metrics [Derived metrics](https://docs.getdbt.com/docs/build/derived.md) allow you to perform calculations using other metrics. For example, you can calculate `gross_profit` by subtracting a `cost` metric from a `revenue` metric, or calculate growth by comparing a metric to its value from a previous time period. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: order_total type: simple agg: sum expr: order_amount - name: order_cost type: simple agg: sum expr: cost_amount - name: order_gross_profit description: Gross profit from each order type: derived label: Order gross profit expr: revenue - cost # Required for derived metrics input_metrics: # Required for derived metrics - name: order_total alias: revenue - name: order_cost alias: cost # For derived metrics using metrics from different semantic models metrics: - name: my_advanced_derived_metric type: derived label: Advanced derived metric expr: metric_a - metric_b_offset + metric_c # Required input_metrics: # Required - list of metric names or dicts - name: metric_a filter: "{{ Dimension('entity__dimension') }} > 10" - name: metric_b alias: metric_b_offset offset_window: 1 week # Only allowed for derived metrics - name: metric_c offset_to_grain: month # Only allowed for derived metrics ``` #### Ratio metrics [Ratio metrics](https://docs.getdbt.com/docs/build/ratio.md) involve a numerator metric and a denominator metric. A `filter` string can be applied to both the numerator and denominator or separately to the numerator or denominator. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: cancellations type: simple agg: sum - name: transaction_amount description: The amount of transactions label: Transaction Amount type: simple agg: sum - name: cancellation_rate type: ratio label: Cancellation rate numerator: cancellations # Can be string or dict denominator: transaction_amount # Can be string or dict filter: | # Optional - applies to both numerator and denominator "{{ Dimension('customer__country') }} = 'MX'" - name: enterprise_cancellation_rate type: ratio label: Enterprise cancellation rate numerator: name: cancellations filter: "{{ Dimension('company__tier') }} = 'enterprise'" # Optional alias: enterprise_cancellations # Optional denominator: transaction_amount filter: | # Optional - applies to both {{ Dimension('customer__country') }} = 'MX' # For ratio metrics using metrics from different semantic models metrics: - name: my_advanced_ratio_metric type: ratio label: Advanced ratio metric numerator: name: metric_from_model_a filter: "{{ Dimension('entity__dimension') }} > 10" alias: filtered_numerator denominator: name: metric_from_model_b filter: "{{ Dimension('entity__dimension') }} < 100" ``` #### Simple metrics (Applies to dbt v1.12 and later) [Simple metrics](https://docs.getdbt.com/docs/build/simple.md) point directly to a single column expression within a semantic model. You can think of a simple metric as the foundational building block for other metrics. It performs an aggregation (like `sum`, `count`, or `average`, and so on) on a single field in your model. models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: cancellations description: The number of cancellations type: simple label: Cancellations agg: sum # Required: sum | max | min | average | median | count_distinct | percentile | count | sum_boolean expr: cancellations_usd # Optional, defaults to metric name join_to_timespine: true # Optional fill_nulls_with: 0 # Optional filter: | # Optional {{ Dimension('order__value') }} > 100 and "{{ Dimension('user__acquisition') }} is not null # Example with percentile aggregation - name: order_percentile type: simple label: Order 95th Percentile agg: percentile expr: order_value percentile: 95.0 # Required if agg is percentile percentile_type: discrete # Required: discrete | continuous # Example with non-additive dimension - name: account_balance type: simple label: Account balance agg: sum expr: balance_amount non_additive_dimension: name: snapshot_date # Must be a dimension window_agg: max # min | max group_by: # Optional - list of entity names - account_id # Example with config - name: revenue type: simple label: Total revenue agg: sum expr: revenue_amount config: meta: owner: "@finance_team" team: "Revenue analytics" ``` #### Filters Configure a filter using Jinja templating and the following syntax to reference entities, dimensions, time dimensions, or metrics in filters. Refer to [Metrics as dimensions](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) for details on how to use metrics as dimensions with metric filters: models/metrics/file\_name.yml ```yaml filter: | {{ Entity('entity_name') }} filter: | {{ Dimension('primary_entity__dimension_name') }} filter: | {{ TimeDimension('time_dimension', 'granularity') }} filter: | {{ Metric('metric_name', group_by=['entity_name']) }} ``` For example, if you want to filter for the order date dimension grouped by month, use the following syntax: ```yaml filter: | {{ TimeDimension('order_date', 'month') }} ``` #### Related docs * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) * [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) * [Metrics as dimensions with metric filters](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) --- ### Cross-platform Mesh using Iceberg catalogs Beta ### Cross-platform Mesh using Iceberg catalogs [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") If a model is configured with `catalog_name`, dbt uses the current project's [catalog definition](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) (in `catalogs.yml`), for the current active adapter, to resolve the top-level namespace of that model. This means that you can materialize a dbt model in your `databricks_project` to an Iceberg table in Unity catalog, and select from it in another model in your `snowflake_project`. You can even materialize that model back to Unity catalog, and then use it for other models in `databricks_project`. The requirement is an automated service for linking or syncing metadata across your Iceberg catalog and connected data platforms. These include [Snowflake catalog-linked databases](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked), [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation), [AWS Glue catalog federation](https://docs.aws.amazon.com/lake-formation/latest/dg/catalog-federation.html), and [GCP Lakehouse runtime catalog](https://docs.cloud.google.com/lakehouse/docs/about-lakehouse-catalogs). Supported cross-platform combinations include: * Snowflake ↔ Databricks using Unity catalog ([Snowflake docs](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-unity), [tutorial](https://docs.snowflake.com/en/user-guide/tutorials/tables-iceberg-set-up-bidirectional-access-to-unity-catalog)) * Snowflake ↔ Athena using Glue catalog ([Snowflake docs](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-glue)) * Snowflake ↔ BigQuery using BigLake catalog ([Snowflake docs](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-biglake)) * Snowflake ↔ DuckDB using Horizon catalog ([example](https://github.com/dataders/dbt_aws_cloud_cost)) * Databricks ↔ DuckDB using Unity catalog ([example](https://github.com/dataders/dbt_aws_cloud_cost)) #### Example: Snowflake ↔ Databricks using Unity catalog Let's imagine two "mesh" projects, [`jaffle_finance`](https://github.com/dbt-labs/jaffle-shop-mesh-finance) and [`jaffle_marketing`](https://github.com/dbt-labs/jaffle-shop-mesh-marketing), with a cross-project dependency `jaffle_finance -> jaffle_marketing`. So long as the `jaffle_finance` project writes its public models to an Iceberg catalog that the `jaffle_marketing` project can read from, and both projects configure that Iceberg catalog in `catalogs.yml` by the same `name` — these projects can now run on **different data platforms.** ```yaml # jaffle_finance/dbt_project.yml flags: use_catalogs_v2: true models: jaffle_finance: marts: access: public catalog_name: finance_db ``` ```yaml # jaffle_finance/catalogs.yml catalogs: - name: finance_db type: unity config: databricks: # where jaffle_finance project will write its public models to catalog_database: finance_db ``` ```yaml # jaffle_marketing/dbt_project.yml flags: use_catalogs_v2: true ``` ```yaml # jaffle_marketing/catalogs.yml catalogs: - name: finance_db type: unity config: snowflake: # catalog-linked database pointing to same Unity catalog # where jaffle_marketing project will read jaffle_finance public models from catalog_database: snowflake_cld__finance_db ``` note If you create the [Snowflake catalog-linked database](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked) with `CATALOG_CASE_SENSITIVITY = CASE_INSENSITIVE`, then you don't need to quote all column and identifier names that Snowflake is reading from or writing to Unity. What's going on here? * The `jaffle_finance` project, running on Databricks, materializes "mart" models into the `finance_db` database in Unity catalog. * This is actually an Iceberg catalog, which we've made accessible to Snowflake through a catalog-linked database. In Snowflake, the catalog-linked database is named `snowflake_cld__finance_db`. * When running the `jaffle_marketing` project, dbt sees a cross-project reference to a public model in the `jaffle_finance` project: ```sql -- marketing/models/marts/roi_by_channel.sql with monthly_revenue as ( select * from {{ ref('jaffle_finance', 'monthly_revenue') }} ), ... ``` ![Flow chart: The marketing project's ROI by channel model depends on the finance project's monthly revenue model](https://github.com/user-attachments/assets/e0a07911-b9d5-4422-9a81-6287e1946ca2) This works for [both ways of resolving cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md): 1. `package` dependencies (supported in dbt v1 + dbt platform) 2. `project` dependencies (dbt platform Enterprise; refer to the [advantages of this approach](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#advantages)) **For `package` dependencies:** ```yaml # dependencies.yml packages: - git: https://github.com/dbt-labs/jaffle-shop-mesh-finance ``` The upstream package model `jaffle_finance.monthly_revenue` is configured with `catalog_name: finance_db`. dbt uses the currently running (root) project's `catalogs.yml` to resolve its three-part relation name. In this project (`jaffle_marketing`) + this platform (Snowflake), that catalog is configured with `database: snowflake_cld__finance_db`. Therefore, dbt resolves the reference to: **For `project` dependencies:** ```yaml # dependencies.yml projects: - name: jaffle_finance ``` Previously, if we only used the upstream model's `database` config, then dbt would resolve this to `finance_db.jaffle_finance.monthly_revenue` — which doesn't exist in Snowflake. Instead, dbt now **matches up the `catalog_name`** for the referenced model with the entries in `catalogs.yml` for the currently running (root) project. dbt sees that `jaffle_finance.monthly_revenue` has `catalog_name: finance_db`, which for this project (`jaffle_marketing`) + this platform (Snowflake) is configured with `database: snowflake_cld__finance_db`. Therefore, dbt resolves the reference to: ```sql -- marketing/target/compiled/models/marts/roi_by_channel.sql with monthly_revenue as ( select * from snowflake_cld__finance_db."jaffle_finance"."monthly_revenue" ), ... ``` *And it just works!* Behind the scenes, Snowflake is syncing Databricks' `finance_db` ↔ Snowflake's `snowflake_cld__finance_db`, with eventual consistency. This means that `finance_db.jaffle_finance.monthly_revenue` in Databricks and `snowflake_cld__finance_db."jaffle_finance"."monthly_revenue"` in Snowflake are pointers to *the exact same Iceberg table* in the Databricks-managed Unity catalog. ##### Alternative approach: catalog federation Writes to external Iceberg catalogs are generally slower than writes to managed Iceberg tables, and they can also run into reliability issues at scale. (See our [blog](https://docs.getdbt.com/blog/catalog-linked-databases) and [benchmark](https://github.com/dbt-labs/snow-dbx-iceberg-benchmark).) Instead of having Snowflake write directly to Unity catalog through a catalog-linked database, you can have Snowflake write to its (managed) Horizon catalog, and use [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation) to synchronize the Iceberg metadata for subsequent reads. [![Multi-catalog pattern for Databricks + Snowflake cross-platform Mesh](/img/docs/building-a-dbt-project/snow-dbx-xplat-mesh.png?v=2 "Multi-catalog pattern for Databricks + Snowflake cross-platform Mesh")](#)Multi-catalog pattern for Databricks + Snowflake cross-platform Mesh catalogs.yml ```yaml catalogs: - name: finance_unity type: unity config: databricks: # catalog where jaffle_finance project will write its public models to catalog_database: finance_db snowflake: # name of catalog-linked database # where jaffle_marketing will read jaffle_finance's public models catalog_database: snowflake_cld__finance_db - name: marketing_horizon type: horizon config: snowflake: # database where jaffle_marketing will write its public models to catalog_database: horizon_db databricks: # name of federated catalog in Unity # where jaffle_finance will read jaffle_marketing's public models catalog_database: dbx_federation__horizon_db ``` --- ### Cumulative metrics (Applies to dbt v1.12 and later) Cumulative metrics aggregate values from other metrics across a defined accumulation period. If you don’t specify a period, the metric accumulates values over the entire available time range. Use cumulative metrics when you want to calculate rolling or period-to-date values, such as weekly active users or month-to-date revenue. * You must create a [time spine model](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) before you define cumulative metrics so that MetricFlow can join time-based aggregations to the time spine. * If a cumulative metric depends on metrics or dimensions defined in a different semantic model, set cumulative metrics under the top level `metrics` key. #### Parameters (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | | `name` | The name of the metric. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of the metric (cumulative, derived, ratio, or simple). | Required | String | | `label` | Optional string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `input_metric` | The name of the metric being referenced. Supports the following nested parameters: `name`, `filter`, and `alias`. | Required | Dict | | `input_metric.name` | The name of the metric being referenced. | Required | String | | `input_metric.filter` | The [filter](https://docs.getdbt.com/docs/build/metrics-overview.md#filters) to apply to the metric. | Optional | String | | `input_metric.alias` | The alias to apply to the metric. | Optional | String | | `join_to_timespine` | Boolean indicating if the aggregated metric should be joined to the time spine table to fill in missing dates. Default is `false`. | Optional | Boolean | | `window` | Specifies the accumulation window, such as `1 month`, `7 days`, or `1 year`. Cannot be used with `grain_to_date`. | Optional | String | | `grain_to_date` | Sets the accumulation grain, such as `hour`, `day`, `week`, `month`, `year`, restarting accumulation at the beginning of each specified grain period. For example, selecting `month` will aggregate the `month to date` aggregation of the metric. Cannot be used with `window`. | Optional | String | | `period_agg` | Defines how to re-aggregate the cumulative metric when querying with a non-default granularity: `first`, `last`, or `average`. Defaults to `first` if `period_agg` isn't specified. | Optional | String | ##### Complete specification The following displays the complete specification for cumulative metrics, along with an example: (Applies to dbt v1.12 and later) ```yaml metrics: - name: my_advanced_cumulative_metric # required description: my_description # optional label: my_label # optional type: cumulative # required -- cumulative | ratio | derived | conversion # if cumulative, optionally can supply window or grain_to_date, but not both window: 1 week # (interval) # grain_to_date: hour # hour | day | week | month | year | ... input_metric: # required, must refer to metric name or metric dict name: my_metric_name_from_another_semantic_model filter: "{{ Dimension('entity__dimension_name') }} > 10" alias: my_metric_name_a_week_ago_in_another_semantic_model ``` #### Cumulative metrics example Cumulative metrics measure data over a given window and consider the window infinite when no window parameter is passed, accumulating the data over all time. The following example shows how to define cumulative metrics in a YAML file: (Applies to dbt v1.12 and later) * `cumulative_order_total`: Calculates the cumulative order total over all time. Uses `input_metric` to specify the simple metric `order_total` to be aggregated. * `cumulative_order_total_l1m`: Calculates the trailing 1-month cumulative order total. Uses `window` parameter to specify a window of 1 month. * `cumulative_order_total_mtd`: Calculates the month-to-date cumulative order total. Uses `grain_to_date` parameter to specify a grain of `month`. models/marts/fct\_orders.yml ```yaml metrics: - name: cumulative_order_total label: "Cumulative order total (All-Time)" description: "The cumulative value of all orders" type: cumulative input_metric: order_total - name: cumulative_order_total_l1m label: "Cumulative order total (L1M)" description: "Trailing 1-month cumulative order total" type: cumulative window: 1 month input_metric: order_total - name: cumulative_order_total_mtd label: "Cumulative order total (MTD)" description: "The month-to-date value of all orders" type: cumulative grain_to_date: month input_metric: order_total ``` ##### Granularity options (Applies to dbt v1.12 and later) Use the `period_agg` parameter with `first`, `last`, and `average` functions to aggregate cumulative metrics over the requested period. This is because granularity options for cumulative metrics are different than the options for other metric types. * For other metrics, we use the `date_trunc` function to implement granularity. * However, cumulative metrics are non-additive (values can't be added up), so we can't use the `date_trunc` function to change their time grain granularity. * By default, we take the first value of the period. You can change this by specifying a different function using the `period_agg` parameter. In the following example, we define a cumulative metric, `cumulative_revenue`, that calculates the cumulative revenue for all orders: models/marts/fct\_orders.yml ```yaml metrics: - name: cumulative_revenue description: "The cumulative revenue for all orders." label: "Cumulative revenue (all-time)" type: cumulative input_metric: revenue period_agg: first # Optional. Defaults to first. Accepted values: first|last|average ``` In this example, `period_agg` is set to `first`, which chooses the first value for the selected granularity window. To query `cumulative_revenue` by week, use the following query syntax: * `dbt sl query --metrics cumulative_revenue --group-by metric_time__week`  Expand toggle to view how the SQL compiles Note the use of the `window` function to select the `first` value. For `last` and `average`, we would replace the `first_value()` function in the generated SQL with `last_value()` and `average` respectively. ```sql -- re-aggregate metric via the group by select metric_time__week, metric_time__quarter, revenue_all_time from ( -- window function for metric re-aggregation select metric_time__week, metric_time__quarter, first_value(revenue_all_time) over ( partition by metric_time__week, metric_time__quarter order by metric_time__day rows between unbounded preceding and unbounded following ) as revenue_all_time from ( -- join self over time range -- pass only elements: ['txn_revenue', 'metric_time__week', 'metric_time__quarter', 'metric_time__day'] -- aggregate measures -- compute metrics via expressions select subq_11.metric_time__day as metric_time__day, subq_11.metric_time__week as metric_time__week, subq_11.metric_time__quarter as metric_time__quarter, sum(revenue_src_28000.revenue) as revenue_all_time from ( -- time spine select ds as metric_time__day, date_trunc('week', ds) as metric_time__week, date_trunc('quarter', ds) as metric_time__quarter from mf_time_spine subq_12 group by ds, date_trunc('week', ds), date_trunc('quarter', ds) ) subq_11 inner join fct_revenue revenue_src_28000 on ( date_trunc('day', revenue_src_28000.created_at) <= subq_11.metric_time__day ) group by subq_11.metric_time__day, subq_11.metric_time__week, subq_11.metric_time__quarter ) subq_16 ) subq_17 group by metric_time__week, metric_time__quarter, revenue_all_time ``` ##### Window options This section details examples of when to specify and not to specify window options. (Applies to dbt v1.12 and later) * When a period is specified, MetricFlow applies a sliding window to the underlying simple metric, such as tracking weekly active users with a 7-day window. * Without specifying a period, cumulative metrics accumulate values over all time, useful for running totals like current revenue and active subscriptions.  Example of window specified If a window option is specified, MetricFlow applies a sliding window to the underlying metric. Suppose the underlying metrics, `customers`, is configured to count the unique customers making orders at the Jaffle shop: models/marts/customers.yml ```yaml models: - name: customers semantic_model: enabled: true ... metrics: - name: customers description: "Unique customers making orders at the Jaffle shop" label: "Customers" type: simple agg: count_distinct expr: customer_id ``` We can write a cumulative metric `weekly_customers` as such: models/marts/customers.yml ```yaml metrics: - name: weekly_customers description: "Weekly active customers with a 7-day sliding window" label: "Weekly Active Customers" type: cumulative input_metric: customers window: 7 days period_agg: first # When using non-default granularity with cumulative metrics, re-aggregation is required. period_agg: first selects the first value in each granularity window during re-aggregation. ``` From the sample YAML example, note the following: * `type`: Specify cumulative to indicate the type of metric. * `input_metric`: Specify the metric to be aggregated (in this case, `customers`). * `window`: Specify the accumulation window (in this case, 7 days). * `period_agg`: Specify the re-aggregation function (in this case, `first`). * `grain_to_date`: Specify the grain to date (in this case, `week`). For example, in the `weekly_customers` cumulative metric, MetricFlow takes a sliding 7-day window of relevant customers and applies a count distinct function. If you remove `window`, the metric will accumulate over all time.  Example of window not specified Suppose you (a subscription-based company for the sake of this example) have an event-based log table with the following columns: * `date`: a date column * `user_id`: (integer) an ID specified for each user that is responsible for the event * `subscription_plan`: (integer) a column that indicates a particular subscription plan associated with the user. * `subscription_revenue`: (integer) a column that indicates the value associated with the subscription plan. * `event_type`: (integer) a column that populates with +1 to indicate an added subscription, or -1 to indicate a deleted subscription. * `revenue`: (integer) a column that multiplies `event_type` and `subscription_revenue` to depict the amount of revenue added or lost for a specific date. Using cumulative metrics without specifying a window, you can calculate running totals for metrics like the count of active subscriptions and revenue at any point in time. The following YAML file shows creating a cumulative metrics to obtain current revenue and the total number of active subscriptions as a cumulative sum using the `revenue` and `subscription_count` metrics: models/marts/fct\_orders.yml ```yaml metrics: - name: current_revenue description: "Current revenue" label: "Current Revenue" type: cumulative input_metric: revenue - name: active_subscriptions description: "Count of active subscriptions" label: "Active Subscriptions" type: cumulative input_metric: subscription_count ``` ##### Grain to date You can choose to specify a grain to date in your cumulative metric configuration to accumulate a metric from the start of a grain (such as week, month, or year). When using a window, such as a month, MetricFlow will go back one full calendar month. However, grain to date will always start accumulating from the beginning of the grain, regardless of the latest date of data. (Applies to dbt v1.12 and later) For example, let's consider an underlying simple metric `order_total` defined within a semantic model: ```yaml models: - name: fct_orders semantic_model: enabled: true # Simple metrics defined within semantic model metrics: - name: order_total description: "Sum of order total amounts" type: simple agg: sum expr: order_total ``` We can compare the difference between a 1-month window and a monthly grain to date. * The cumulative metric in a window approach applies a sliding window of 1 month * The grain to date by month resets at the beginning of each month. (Applies to dbt v1.12 and later) models/marts/fct\_orders.yml ```yaml # Top-level metrics key for cumulative metrics metrics: - name: cumulative_order_total_mtd label: "Cumulative order total (MTD)" description: "The month-to-date value of all orders" type: cumulative grain_to_date: month input_metric: order_total ``` Cumulative metric with grain to date: (Applies to dbt v1.12 and later) models/marts/fct\_orders.yml ```yaml # Top-level metrics key for cumulative metrics metrics: - name: orders_last_month_to_date label: "Orders month to date" description: "Month-to-date count of orders" type: cumulative grain_to_date: month input_metric: order_count ```  Expand toggle to view how the SQL compiles ```sql with staging as ( select subq_3.date_day as metric_time__day, date_trunc('week', subq_3.date_day) as metric_time__week, sum(subq_1.order_count) as orders_last_month_to_date from dbt_jstein.metricflow_time_spine subq_3 inner join ( select date_trunc('day', ordered_at) as metric_time__day, 1 as order_count from analytics.dbt_jstein.orders orders_src_10000 ) subq_1 on ( subq_1.metric_time__day <= subq_3.date_day ) and ( subq_1.metric_time__day >= date_trunc('month', subq_3.date_day) ) group by subq_3.date_day, date_trunc('week', subq_3.date_day) ) select * from ( select metric_time__week, first_value(orders_last_month_to_date) over (partition by date_trunc('week', metric_time__day) order by metric_time__day) as cumulative_revenue from staging ) group by metric_time__week, cumulative_revenue order by metric_time__week 1 ``` #### SQL implementation example To calculate the cumulative value of the metric over a given window we do a time range join to a timespine table using the primary time dimension as the join key. We use the accumulation window in the join to decide whether a record should be included on a particular day. The following SQL code produced from an example cumulative metric is provided for reference: To implement cumulative metrics, refer to the SQL code example: ```sql select count(distinct distinct_users) as weekly_active_users, metric_time from ( select subq_3.distinct_users as distinct_users, subq_3.metric_time as metric_time from ( select subq_2.distinct_users as distinct_users, subq_1.metric_time as metric_time from ( select metric_time from transform_prod_schema.mf_time_spine subq_1356 where ( metric_time >= cast('2000-01-01' as timestamp) ) and ( metric_time <= cast('2040-12-31' as timestamp) ) ) subq_1 inner join ( select distinct_users as distinct_users, date_trunc('day', ds) as metric_time from demo_schema.transactions transactions_src_426 where ( (date_trunc('day', ds)) >= cast('1999-12-26' as timestamp) ) AND ( (date_trunc('day', ds)) <= cast('2040-12-31' as timestamp) ) ) subq_2 on ( subq_2.metric_time <= subq_1.metric_time ) and ( subq_2.metric_time > dateadd(day, -7, subq_1.metric_time) ) ) subq_3 ) group by metric_time, limit 100; ``` #### Limitations If you specify a `window` in your cumulative metric definition, you must include `metric_time` as a dimension in the SQL query. This is because the accumulation window is based on metric time. For example, ```sql select count(distinct subq_3.distinct_users) as weekly_active_users, subq_3.metric_time from ( select subq_2.distinct_users as distinct_users, subq_1.metric_time as metric_time group by subq_3.metric_time ``` #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Custom aliases #### Overview When dbt runs a model, it will generally create a relation (either a table or a view ) in the database, except in the case of an [ephemeral model](https://docs.getdbt.com/docs/build/materializations.md), when it will create a CTE for use in another model. By default, dbt uses the model's filename as the identifier for the relation or CTE it creates. This identifier can be overridden using the [`alias`](https://docs.getdbt.com/reference/resource-configs/alias.md) model configuration. ##### Why alias model names? The names of schemas and tables are effectively the "user interface" of your data warehouse. Well-named schemas and tables can help provide clarity and direction for consumers of this data. In combination with [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md), model aliasing is a powerful mechanism for designing your warehouse. The file naming scheme that you use to organize your models may also interfere with your data platform's requirements for identifiers. For example, you might wish to namespace your files using a period (`.`), but your data platform's SQL dialect may interpret periods to indicate a separation between schema names and table names in identifiers, or it may forbid periods from being used at all in CTE identifiers. In cases like these, model aliasing can allow you to retain flexibility in the way you name your model files without violating your data platform's identifier requirements. ##### Usage The `alias` config can be used to change the name of a model's identifier in the database. The following table shows examples of database identifiers for models both with and without a supplied `alias`, and with different materializations. | Model | Config | Relation Type | Database Identifier | | ---------------- | ----------------------------------------------------------- | ------------- | -------------------------------- | | ga\_sessions.sql | {{ config(materialization='view') }} | view | "analytics"."ga\_sessions" | | ga\_sessions.sql | {{ config(materialization='view', alias='sessions') }} | view | "analytics"."sessions" | | ga\_sessions.sql | {{ config(materialization='ephemeral') }} | CTE | "\_\_dbt\_\_cte\_\_ga\_sessions" | | ga\_sessions.sql | {{ config(materialization='ephemeral', alias='sessions') }} | CTE | "\_\_dbt\_\_cte\_\_sessions" | To configure an alias for a model, supply a value for the model's `alias` configuration parameter. For example: models/google\_analytics/ga\_sessions.sql ```sql -- This model will be created in the database with the identifier `sessions` -- Note that in this example, `alias` is used along with a custom schema {{ config(alias='sessions', schema='google_analytics') }} select * from ... ``` Or in a `schema.yml` file. models/google\_analytics/schema.yml ```yaml models: - name: ga_sessions config: alias: sessions ``` When referencing the `ga_sessions` model above from a different model, use the `ref()` function with the model's *filename* as usual. For example: models/combined\_sessions.sql ```sql -- Use the model's filename in ref's, regardless of any aliasing configs select * from {{ ref('ga_sessions') }} union all select * from {{ ref('snowplow_sessions') }} ``` ##### generate\_alias\_name The alias generated for a model is controlled by a macro called `generate_alias_name`. This macro can be overridden in a dbt project to change how dbt aliases models. This macro works similarly to the [generate\_schema\_name](https://docs.getdbt.com/docs/build/custom-schemas.md#advanced-custom-schema-configuration) macro. To override dbt's alias name generation, create a macro named `generate_alias_name` in your own dbt project. The `generate_alias_name` macro accepts two arguments: 1. The custom alias supplied in the model config 2. The node that a custom alias is being generated for The default implementation of `generate_alias_name` simply uses the supplied `alias` config (if present) as the model alias, otherwise falling back to the model name. This implementation looks like this: get\_custom\_alias.sql ```jinja2 {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- elif node.version -%} {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} {%- else -%} {{ node.name }} {%- endif -%} {%- endmacro %} ``` 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). (Applies to dbt v1.12 and later) ##### generate\_latest\_version\_pointer\_alias When the [`latest_version_pointer`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) config is enabled, dbt uses the `generate_latest_version_pointer_alias` macro to determine the name of the pointer view it creates for the latest version of a versioned model. This macro follows the same pattern as [`generate_alias_name`](#generate_alias_name). The default implementation uses the model's base name (for example, `dim_customers`) unless a custom alias is set using `latest_version_pointer.alias`: macros/generate\_latest\_version\_pointer\_alias.sql ```jinja2 {% macro generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {{ return(adapter.dispatch('generate_latest_version_pointer_alias', 'dbt')(custom_alias_name, node)) }} {%- endmacro %} {% macro default__generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- else -%} {{ node.name }} {%- endif -%} {%- endmacro %} ``` To override the default, create a macro named `generate_latest_version_pointer_alias` in your project. For example, to use a `_latest` suffix instead of the base name: macros/get\_latest\_version\_pointer\_alias.sql ```jinja2 {% macro generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- else -%} {{ node.name ~ "_latest" }} {%- endif -%} {%- endmacro %} ``` ##### Dispatch macro - SQL alias management for databases and dbt packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md#managing-different-global-overrides-across-packages) ##### Caveats ###### Ambiguous database identifiers Using aliases, it's possible to accidentally create models with ambiguous identifiers. Given the following two models, dbt would attempt to create two views with *exactly* the same names in the database (ie. `sessions`): models/snowplow\_sessions.sql ```sql {{ config(alias='sessions') }} select * from ... ``` models/sessions.sql ```sql select * from ... ``` Whichever one of these models runs second would "win", and generally, the output of dbt would not be what you would expect. To avoid this failure mode, dbt will check if your model names and aliases are ambiguous in nature. If they are, you will be presented with an error message like this: ```text $ dbt compile Encountered an error: Compilation Error dbt found two resources with the database representation "analytics.sessions". dbt cannot create two resources with identical database representations. To fix this, change the "schema" or "alias" configuration of one of these resources: - model.my_project.snowplow_sessions (models/snowplow_sessions.sql) - model.my_project.sessions (models/sessions.sql) ``` If these models should indeed have the same database identifier, you can work around this error by configuring a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) for one of the models. ###### Model versions **Related documentation:** * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) * [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md#alias) By default, dbt will create versioned models with the alias `<model_name>_v<v>`, where `<v>` is that version's unique identifier. You can customize this behavior just like for non-versioned models by configuring a custom `alias` or re-implementing the `generate_alias_name` macro. #### Related docs * [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md?step=1) to learn how to customize dbt models database, schema, and alias * [Custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) to learn how to customize dbt schema * [Custom database](https://docs.getdbt.com/docs/build/custom-databases.md) to learn how to customize dbt database --- ### Custom databases A word on naming Different warehouses have different names for *logical databases*. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. The values `project` and `database` are interchangeable in BigQuery project configurations. #### Configuring custom databases The logical database that dbt models are built into can be configured using the `database` model configuration. If this configuration is not supplied to a model, then dbt will use the database configured in the active target from your `profiles.yml` file. If the `database` configuration *is* supplied for a model, then dbt will build the model into the configured database. The `database` configuration can be supplied for groups of models in the `dbt_project.yml` file, or for individual models in model SQL files. ##### Configuring database overrides in `dbt_project.yml`: This config changes all models in the `jaffle_shop` project to be built into a database called `jaffle_shop`. dbt\_project.yml ```yaml name: jaffle_shop models: jaffle_shop: +database: jaffle_shop # For BigQuery users: # project: jaffle_shop ``` ##### Configuring database overrides in a model file This config changes a specific model to be built into a database called `jaffle_shop`. models/my\_model.sql ```sql {{ config(database="jaffle_shop") }} select * from ... ``` ##### generate\_database\_name The database name generated for a model is controlled by a macro called `generate_database_name`. This macro can be overridden in a dbt project to change how dbt generates model database names. This macro works similarly to the [generate\_schema\_name](https://docs.getdbt.com/docs/build/custom-schemas.md#advanced-custom-schema-configuration) macro. To override dbt's database name generation, create a macro named `generate_database_name` in your own dbt project. The `generate_database_name` macro accepts two arguments: 1. The custom database supplied in the model config 2. The node that a custom database is being generated for The default implementation of `generate_database_name` simply uses the supplied `database` config if one is present, otherwise the database configured in the active `target` is used. This implementation looks like this: get\_custom\_database.sql ```jinja2 {% macro generate_database_name(custom_database_name=none, node=none) -%} {%- set default_database = target.database -%} {%- if custom_database_name is none -%} {{ default_database }} {%- else -%} {{ custom_database_name | trim }} {%- endif -%} {%- endmacro %} ``` 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). ##### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) #### Considerations ##### BigQuery When dbt opens a BigQuery connection, it will do so using the `project_id` defined in your active `profiles.yml` target. This `project_id` will be billed for the queries that are executed in the dbt run, even if some models are configured to be built in other projects. #### Related docs * [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md?step=1) to learn how to customize dbt models database, schema, and alias * [Custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) to learn how to customize dbt model schema * [Custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) to learn how to customize dbt model alias name --- ### Custom schemas By default, all dbt models are built in the schema specified in your [environment](https://docs.getdbt.com/docs/dbt-platform-environments.md) (dbt platform) or [profile's target](https://docs.getdbt.com/docs/local/dbt-environments.md) (dbt v1). This default schema is called your *target schema*. For projects with many models, it's common to organize them across multiple schemas. For example, you might want to: * Group models based on the business unit using the model, creating schemas such as `core`, `marketing`, `finance` and `support`. * Hide intermediate models in a `staging` schema, and only present models that should be queried by an end user in an `analytics` schema. To do this, specify a custom schema. dbt generates the schema name for a model by appending the custom schema to the target schema. For example, `<target_schema>_<custom_schema>`. | Target schema | Custom schema | Resulting schema | | ------------------------ | ------------- | ----------------------------------- | | analytics\_prod | None | analytics\_prod | | alice\_dev | None | alice\_dev | | dbt\_cloud\_pr\_123\_456 | None | dbt\_cloud\_pr\_123\_456 | | analytics\_prod | marketing | analytics\_prod\_marketing | | alice\_dev | marketing | alice\_dev\_marketing | | dbt\_cloud\_pr\_123\_456 | marketing | dbt\_cloud\_pr\_123\_456\_marketing | #### How do I use custom schemas? To specify a custom schema for a model, use the `schema` configuration key. As with any configuration, you can do one of the following: * apply this configuration to a specific model by using a config block within a model * apply it to a subdirectory of models by specifying it in your `dbt_project.yml` file orders.sql ```sql {{ config(schema='marketing') }} select ... ``` dbt\_project.yml ```yaml # models in `models/marketing/ will be built in the "*_marketing" schema models: my_project: marketing: +schema: marketing ``` #### Understanding custom schemas When first using custom schemas, it's a common misunderstanding to assume that a model *only* uses the new `schema` configuration; for example, a model that has the configuration `schema: marketing` would be built in the `marketing` schema. However, dbt puts it in a schema like `<target_schema>_marketing`. There's a good reason for this deviation. Each dbt user has their own target schema for development (refer to [Managing Environments](#managing-environments)). If dbt ignored the target schema and only used the model's custom schema, every dbt user would create models in the same schema and would overwrite each other's work. By combining the target schema and the custom schema, dbt ensures that objects it creates in your data warehouse don't collide with one another. If you prefer to use different logic for generating a schema name, you can change the way dbt generates a schema name (see below). ##### How does dbt generate a model's schema name? dbt uses a default macro called `generate_schema_name` to determine the name of the schema that a model should be built in. The following code represents the default macro's logic: ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` <br /> 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). #### Changing the way dbt generates a schema name If your dbt project has a custom macro called `generate_schema_name`, dbt will use it instead of the default macro. This allows you to customize the name generation according to your needs. To customize this macro, copy the example code in the section [How does dbt generate a model's schema name](#how-does-dbt-generate-a-models-schema-name) into a file named `macros/generate_schema_name.sql` and make changes as necessary. Be careful. dbt will ignore any custom `generate_schema_name` macros included in installed packages.  Warning: Don't replace \`default\_schema\` in the macro If you're modifying how dbt generates schema names, don't just replace `{{ default_schema }}_{{ custom_schema_name | trim }}` with `{{ custom_schema_name | trim }}` in the `generate_schema_name` macro. If you remove `{{ default_schema }}`, it causes developers to override each other's models if they create their own custom schemas. This can also cause issues during development and continuous integration (CI). ❌ The following code block is an example of what your code *should not* look like: ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} # The following is incorrect as it omits {{ default_schema }} before {{ custom_schema_name | trim }}. {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` ##### generate\_schema\_name arguments | Argument | Description | Example | | -------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | custom\_schema\_name | The configured value of `schema` in the specified node, or `none` if a value is not supplied | `marketing` | | node | The `node` that is currently being processed by dbt | `{"name": "my_model", "resource_type": "model",...}` | ##### Jinja context available in generate\_schema\_name If you choose to write custom logic to generate a schema name, it's worth noting that not all variables and methods are available to you when defining this logic. In other words: the `generate_schema_name` macro is compiled with a limited Jinja context. The following context methods *are* available in the `generate_schema_name` macro: | Jinja context | Type | Available | | --------------------------------------------------------------------------------- | -------- | ------------------ | | [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) | Variable | ✅ | | [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) | Variable | ✅ | | [var](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) | Variable | Limited, see below | | [exceptions](https://docs.getdbt.com/reference/dbt-jinja-functions/exceptions.md) | Macro | ✅ | | [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) | Macro | ✅ | | Other macros in your project | Macro | ✅ | | Other macros in your packages | Macro | ✅ | ##### Which vars are available in generate\_schema\_name? Globally-scoped variables and variables defined on the command line with [--vars](https://docs.getdbt.com/docs/build/project-variables.md) are accessible in the `generate_schema_name` context. ##### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) #### A built-in alternative pattern for generating schema names A common customization is to use the custom schema in production when provided, with the target schema serving only as a fallback if no custom schema is specified. In other environments, such as development and CI, custom schema configurations are ignored, defaulting to the target schema instead. Production Environment (`target.name == 'prod'`) | Target schema | Custom schema | Resulting schema | | --------------- | ------------- | ---------------- | | analytics\_prod | None | analytics\_prod | | analytics\_prod | marketing | marketing | Development/CI Environment (`target.name != 'prod'`) | Target schema | Custom schema | Resulting schema | | ------------------------ | ------------- | ------------------------ | | alice\_dev | None | alice\_dev | | alice\_dev | marketing | alice\_dev | | dbt\_cloud\_pr\_123\_456 | None | dbt\_cloud\_pr\_123\_456 | | dbt\_cloud\_pr\_123\_456 | marketing | dbt\_cloud\_pr\_123\_456 | Similar to the regular macro, this approach guarantees that schemas from different environments will not collide. dbt ships with a macro for this use case — called `generate_schema_name_for_env` — which is disabled by default. To enable it, add a custom `generate_schema_name` macro to your project that contains the following code: macros/generate\_schema\_name.sql ```sql -- put this in macros/generate_schema_name.sql {% macro generate_schema_name(custom_schema_name, node) -%} {{ generate_schema_name_for_env(custom_schema_name, node) }} {%- endmacro %} ``` When using this macro, you'll need to set the target name in your production job to `prod`. #### Managing environments In the `generate_schema_name` macro examples shown in the [built-in alternative pattern](#a-built-in-alternative-pattern-for-generating-schema-names) section, the `target.name` context variable is used to change the schema name that dbt generates for models. If the `generate_schema_name` macro in your project uses the `target.name` context variable, you must ensure that your different dbt environments are configured accordingly. While you can use any naming scheme you'd like, we typically recommend: * **dev** — Your local development environment; configured in a `profiles.yml` file on your computer. * **ci** — A [continuous integration](https://docs.getdbt.com/docs/platform/git/connect-github.md) environment running on pull requests in GitHub, GitLab, and so on. * **prod** — The production deployment of your dbt project, like in dbt, Airflow, or [similar](https://docs.getdbt.com/docs/deploy/deployments.md). If your schema names are being generated incorrectly, double-check your target name in the relevant environment. For more information, consult the [managing environments in dbt](https://docs.getdbt.com/docs/local/dbt-environments.md) guide. #### Related docs * [Mapping project folders to warehouse schemas](https://docs.getdbt.com/best-practices/how-we-structure/6-mapping-folders-to-warehouse.md) to map staging, intermediate, and marts folders to warehouse schemas * [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md?step=1) to learn how to customize dbt models database, schema, and alias * [Custom database](https://docs.getdbt.com/docs/build/custom-databases.md) to learn how to customize dbt model database * [Custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) to learn how to customize dbt model alias name --- ### Custom target names #### dbt Scheduler You can define a custom target name for any dbt job to correspond to settings in your dbt project. This is helpful if you have logic in your dbt project that behaves differently depending on the specified target, for example: ```sql select * from a_big_table -- limit the amount of data queried in dev {% if target.name != 'prod' %} where created_at > date_trunc('month', current_date) {% endif %} ``` To set a custom target name for a job in dbt, configure the **Target Name** field for your job in the Job Settings page. [![Overriding the target name to 'prod'](/img/docs/dbt-platform/using-dbt-platform/jobs-settings-target-name.png?v=2 "Overriding the target name to 'prod'")](#)Overriding the target name to 'prod' #### dbt Studio IDE When developing in dbt, you can set a custom target name in your user credentials. Click your account name above the profile icon in the left panel, select **Account settings**, then go to **Credentials**. Choose the project to update the target name. [![Overriding the target name to 'dev'](/img/docs/dbt-platform/using-dbt-platform/development-credentials.png?v=2 "Overriding the target name to 'dev'")](#)Overriding the target name to 'dev' --- ### Data health signals Preview ### Data health signals [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Starter, Enterprise, Enterprise+ Data health signals offer a quick, at-a-glance view of data health when browsing your resources in Catalog. They keep you informed on the status of your resource's health using the indicators **Healthy**, **Caution**, **Degraded**, or **Unknown**. Note, we don’t calculate data health for non-dbt resources. * Supported resources are [models](https://docs.getdbt.com/docs/build/models.md), [sources](https://docs.getdbt.com/docs/build/sources.md), and [exposures](https://docs.getdbt.com/docs/build/exposures.md). * For accurate health data, ensure the resource is up-to-date and had a recent job run. * Each data health signal reflects key data health components, such as test success status, missing resource descriptions, missing tests, absence of builds in 30-day windows, [and more](#data-health-signal-criteria). [![View data health signals for your models.](/img/docs/collaborate/dbt-explorer/data-health-signal.png?v=2 "View data health signals for your models.")](#)View data health signals for your models. #### Access data health signals Access data health signals in the following places: * In the [search function](https://docs.getdbt.com/docs/explore/explore-projects.md#search-resources) or under **Models**, **Sources**, or **Exposures** in the **Resource** tab. * For sources, the data health signal also indicates the [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) status. * In the **Health** column on [each resource's details page](https://docs.getdbt.com/docs/explore/explore-projects.md#view-resource-details). Hover over or click the signal to view detailed information. * In the **Health** column of public models tables. * In the [DAG lineage graph](https://docs.getdbt.com/docs/explore/explore-projects.md#project-lineage). Click any node to open the node details panel where you can view it and its details. * In [Data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) through an embeddable iFrame and visible in your BI dashboard. [![Access data health signals in multiple places in dbt Catalog.](/img/docs/collaborate/dbt-explorer/data-health-signal.gif?v=2 "Access data health signals in multiple places in dbt Catalog.")](#)Access data health signals in multiple places in dbt Catalog. #### Data health signal criteria Each resource has a health state that is determined by specific set of criteria. Select the following tabs to view the criteria for that resource type. ##### Models The health state of a model is determined by the following criteria: | **Health state** | **Criteria** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ **Healthy** | All of the following must be true:<br /><br />- Built successfully in the last run<br />- Built in the last 30 days<br />- Model has tests configured<br />- All tests passed<br />- All upstream [sources are fresh](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) or freshness is not applicable (set to `null`)<br />- Has a description | | 🟡 **Caution** | One of the following must be true:<br /><br />- Not built in the last 30 days<br />- Tests are not configured<br />- Tests return warnings<br />- One or more upstream sources are stale:<br />    - Has a freshness check configured<br />    - Freshness check ran in the past 30 days<br />    - Freshness check returned a warning<br />- Missing a description | | 🔴 **Degraded** | One of the following must be true:<br /><br />- Model failed to build<br />- Model has failing tests<br />- One or more upstream sources are stale:<br />    - Freshness check hasn’t run in the past 30 days<br />    - Freshness check returned an error | | ⚪ **Unknown** | - Unable to determine health of resource; no job runs have processed the resource. | ##### Sources The health state of a source is determined by the following criteria: | **Health state** | **Criteria** | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ Healthy | All of the following must be true:<br /><br />- Freshness check configured<br />- Freshness check passed<br />- Freshness check ran in the past 30 days<br />- Has a description | | 🟡 Caution | One of the following must be true:<br /><br />- Freshness check returned a warning<br />- Freshness check not configured<br />- Freshness check not run in the past 30 days<br />- Missing a description | | 🔴 Degraded | - Freshness check returned an error | | ⚪ Unknown | Unable to determine health of resource; no job runs have processed the resource. | ##### Exposures The health state of an exposure is determined by the following criteria: | **Health state** | **Criteria** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ Healthy | All of the following must be true:<br /><br />- Underlying sources are fresh<br />- Underlying models built successfully<br />- Underlying models’ tests passing<br /> | | 🟡 Caution | One of the following must be true:<br /><br />- At least one underlying source’s freshness checks returned a warning<br />- At least one underlying model was skipped<br />- At least one underlying model’s tests returned a warning<br /> | | 🔴 Degraded | One of the following must be true:<br /><br />- At least one underlying source’s freshness checks returned an error<br />- At least one underlying model did not build successfully<br />- At least one model’s tests returned an error | --- ### Data health tile dbt platform | Enterprise, Enterprise+ With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. It allows teams to immediately go back into Catalog to see more details and investigate issues. The data health tile: * Distills [data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) for data consumers. * Deep links you into Catalog where you can further dive into upstream data issues. * Provides richer information and makes it easier to debug. * Revamps the existing, [job-based tiles](#job-based-data-health). Data health tiles rely on [exposures](https://docs.getdbt.com/docs/build/exposures.md) to surface data health signals in your dashboards. An exposure defines how specific outputs — like dashboards or reports — depend on your data models. Exposures in dbt can be configured in two ways: * Manual — Defined [manually](https://docs.getdbt.com/docs/build/exposures.md#declaring-an-exposure) and explicitly in your project’s YAML files. * Automatic — Pulled automatically for supported dbt integrations. dbt automatically [creates and visualizes downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md), removing the need for manual YAML definitions. These downstream exposures are stored in dbt’s metadata system, appear in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), and behave like manual exposures, however they don’t exist in YAML files. [![Example of passing Data health tile in your dashboard.](/img/docs/collaborate/dbt-explorer/data-tile-pass.png?v=2 "Example of passing Data health tile in your dashboard.")](#)Example of passing Data health tile in your dashboard. [![Embed data health tiles in your dashboards to distill data health signals for data consumers.](/img/docs/collaborate/dbt-explorer/data-tiles.png?v=2 "Embed data health tiles in your dashboards to distill data health signals for data consumers.")](#)Embed data health tiles in your dashboards to distill data health signals for data consumers. #### Prerequisites * You must have a dbt account on an [Enterprise-tier plan](https://www.getdbt.com/pricing/). * You must be an account admin to set up [service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). * You must have [develop permissions](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * You have [exposures](https://docs.getdbt.com/docs/build/exposures.md) defined in your project: * If using manual exposures, they must be explicitly defined in your YAML files. * If using automatic downstream exposures, ensure your BI tool is [configured](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) with dbt. * You have [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) enabled in the job that generates this exposure. * The exposure used for the data health tile must have the [`type` property](https://docs.getdbt.com/docs/build/exposures.md#available-properties) set to `dashboard`. Otherwise, you won't be able to view the **Embed data health tile in your dashboard** dropdown in Catalog. #### View exposure in dbt Catalog First, be sure to enable [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) in the job that generates this exposure. 1. Navigate to Catalog by clicking on the **Catalog** link in the navigation. 2. In the main **Overview** page, go to the left navigation. 3. Click on **Exposures** to view the [exposures](https://docs.getdbt.com/docs/build/exposures.md) list. 4. Select a dashboard exposure and go to the **General** tab to view the data health information. 5. In this tab, you’ll see: * The exposure name at the top of the page. * The resource type (for example, **Exposure**) and an **Updated at** timestamp. * A **Data health** section that shows when the **Last check completed** (relative time since the check ran). * A freshness summary (for example, **Data is fresh**) and a breakdown of upstream **Sources** (fresh, delayed, stale, skipped, outdated, unconfigured). * A quality summary (for example, **Data quality passed**) and a breakdown of upstream **Models** and **Tests**. 6. You can click the **Open in Dashboard** button in the upper right to immediately view this in your analytics tool. [![View an exposure in dbt Catalog.](/img/docs/collaborate/dbt-explorer/data-tile-exposures.png?v=2 "View an exposure in dbt Catalog.")](#)View an exposure in dbt Catalog. #### Embed in your dashboard Once you’ve navigated to the exposure in Catalog, you’ll need to set up your data health tile and [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). You can embed data health tile to any analytics tool that supports URL or iFrame embedding. Follow these steps to set up your data health tile: 1. Go to **Account settings** in dbt. 2. Select **API tokens** in the left sidebar and then **Service tokens**. 3. Click on **Create service token** and give it a name. 4. Select the [**Metadata Only**](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) permission. This token will be used to embed the tile in your dashboard in the later steps. [![Set up your dashboard status tile and service token to embed a data health tile](/img/docs/collaborate/dbt-explorer/data-tile-setup.png?v=2 "Set up your dashboard status tile and service token to embed a data health tile")](#)Set up your dashboard status tile and service token to embed a data health tile 5. Copy the **Metadata Only** token and save it in a secure location. You'll need it token in the next steps. 6. Navigate back to Catalog and select an exposure. tip The exposure used for the data health tile must have the [`type` property](https://docs.getdbt.com/docs/build/exposures.md#available-properties) set to `dashboard`. Otherwise, you won't be able to view the **Embed data health tile in your dashboard** dropdown in Catalog. 7. Below the **Data health** section, expand on the toggle for instructions on how to embed the exposure tile (if you're an account admin with develop permissions). 8. In the expanded toggle, you'll see a text field where you can paste your **Metadata Only token**. [![Expand the toggle to embed data health tile into your dashboard.](/img/docs/collaborate/dbt-explorer/data-tile-example.png?v=2 "Expand the toggle to embed data health tile into your dashboard.")](#)Expand the toggle to embed data health tile into your dashboard. 9. Once you’ve pasted your token, you can select either **URL** or **iFrame** depending on which you need to add to your dashboard. If your analytics tool supports iFrames, you can embed the dashboard tile within it. #### Examples The following examples show how to embed the data health tile in Omni, PowerBI, Tableau, and Sigma. ##### Omni example Follow these steps to embed the data health tile in [Omni](https://omni.co/): [![Embed data health tile in Omni](/img/docs/collaborate/dbt-explorer/omni-example.png?v=2 "Embed data health tile in Omni")](#)Embed data health tile in Omni 1. Create a dashboard in Omni. 2. Copy the iFrame snippet available in Catalog's **Data health** section, under the **Embed data health into your dashboard** toggle. 3. Add a new Text or Markdown [element](https://docs.omni.co/visualize-present/dashboards/text-markdown) in your Dashboard with the code from step 2, it should be in the following format: ```html/text <iframe src='https://YOUR_ACCOUNT_PREFIX.metadata.REGION.dbt.com/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=production&environmentId=ENV_ID_NUMBER&token=<YOUR_METADATA_TOKEN>' title='Exposure Status Tile'></iframe> ``` 4. Save the tile and your Omni dashboard should now have a dbt platform hosted data health tile that is automatically updated based on the state of your dbt environment. ##### PowerBI example You can embed the data health tile iFrame in PowerBI using PowerBI Pro Online, Fabric PowerBI, or PowerBI Desktop. [![Embed data health tile iFrame in PowerBI](/img/docs/collaborate/dbt-explorer/power-bi.png?v=2 "Embed data health tile iFrame in PowerBI")](#)Embed data health tile iFrame in PowerBI Follow these steps to embed the data health tile in PowerBI: 1. Create a dashboard in PowerBI and connect to your database to pull in the data. 2. Create a new PowerBI measure by right-clicking on your **Data**, **More options**, and then **New measure**. [![Create a new PowerBI measure.](/img/docs/collaborate/dbt-explorer/power-bi-measure.png?v=2 "Create a new PowerBI measure.")](#)Create a new PowerBI measure. 3. Navigate to Catalog, select the exposure, and expand the [**Embed data health into your dashboard**](https://docs.getdbt.com/docs/explore/data-tile.md#embed-in-your-dashboard) toggle. 4. Go to the **iFrame** tab and copy the iFrame code. Make sure the Metadata Only token is already set up. 5. In PowerBI, paste the iFrame code you copied into your measure calculation window. The iFrame code should look like this: ```html/text <iframe src='https://1234.metadata.ACCESS_URL/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=staging&environmentId=123456789&token=YOUR_METADATA_TOKEN' title='Exposure status tile' height='400'></iframe> ``` [![In the 'Measure tools' tab, replace your values with the iFrame code.](/img/docs/collaborate/dbt-explorer/power-bi-measure-tools.png?v=2 "In the 'Measure tools' tab, replace your values with the iFrame code.")](#)In the 'Measure tools' tab, replace your values with the iFrame code. 6. PowerBI desktop doesn't support HTML rendering by default, so you need to install an HTML component from the PowerBI Visuals Store. 7. To do this, go to **Build visuals** and then **Get more visuals**. 8. Login with your PowerBI account. 9. There are several third-party HTML visuals. The one tested for this guide is [HTML content](https://appsource.microsoft.com/en-us/product/power-bi-visuals/WA200001930?tab=Overview). Install it, but please keep in mind it's a third-party plugin not created or supported by dbt Labs. 10. Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile. [![Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile.](/img/docs/collaborate/dbt-explorer/power-bi-final.png?v=2 "Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile.")](#)Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile. *Refer to [this tutorial](https://www.youtube.com/watch?v=SUm9Hnq8Th8) for additional information on embedding a website into your Power BI report.* ##### Tableau example Follow these steps to embed the data health tile in Tableau: [![Embed data health tile iFrame in Tableau](/img/docs/collaborate/dbt-explorer/tableau-example.png?v=2 "Embed data health tile iFrame in Tableau")](#)Embed data health tile iFrame in Tableau 1. Create a dashboard in Tableau and connect to your database to pull in the data. 2. Copy the URL or iFrame snippet available in Catalog's **Data health** section, under the **Embed data health into your dashboard** toggle. 3. Insert a **Web Page** object. 4. Paste the copied URL and click **Ok**. The URL you copy from Catalog looks similar to the following but may include additional Tableau-specific parameters (such as `biResourceId` and `autoBiProvider=tableau`) that are required to resolve the exposure: ```text/html https://metadata.ACCESS_URL/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=production&environmentId=220370&token=<YOUR_METADATA_TOKEN> ``` 5. You should now see the data health tile embedded in your Tableau dashboard. ##### Sigma example Follow these steps to embed the data health tile in Sigma: [![Embed data health tile in Sigma](/img/docs/collaborate/dbt-explorer/sigma-example.jpg?v=2 "Embed data health tile in Sigma")](#)Embed data health tile in Sigma 1. Create a dashboard in Sigma and connect to your database to pull in the data. 2. Ensure you've copied the URL or iFrame snippet available in Catalog's **Data health** section, under the **Embed data health into your dashboard** toggle. 3. Add a new embedded UI element in your Sigma Workbook in the following format: ```html/text https://metadata.ACCESS_URL/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=production&environmentId=ENV_ID_NUMBER&token=<YOUR_METADATA_TOKEN> ``` *Note, replace the placeholders with your actual values.* 4. You should now see the data health tile embedded in your Sigma dashboard. #### Job-based data health Legacy The default experience is the [environment-based data health tile](#view-exposure-in-dbt-explorer) with Catalog. This section is for legacy job-based data health tiles. If you're using the revamped environment-based exposure tile, refer to the previous section. Expand the following to learn more about the legacy job-based data health tile.  Job-based data health In dbt, the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) can power dashboard status tiles, which are job-based. A dashboard status tile is placed on a dashboard (specifically: anywhere you can embed an iFrame) to give insight into the quality and freshness of the data feeding into that dashboard. This is done in dbt [exposures](https://docs.getdbt.com/docs/build/exposures.md). ###### Functionality The dashboard status tile looks like this: [![](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/passing-tile.jpeg?v=2)](#) The data freshness check fails if any sources feeding into the exposure are stale. The data quality check fails if any dbt tests fail. A failure state could look like this: [![](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/failing-tile.jpeg?v=2)](#) Clicking into **see details** from the Dashboard Status Tile takes you to a landing page where you can learn more about the specific sources, models, and tests feeding into this exposure. ###### Setup First, be sure to enable [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) in the job that generates this exposure. In order to set up your dashboard status tile, here is what you need: 1. **Metadata Only token.** You can learn how to set up a Metadata-Only token [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). 2. **Exposure name.** You can learn more about how to set up exposures [here](https://docs.getdbt.com/docs/build/exposures.md). 3. **Job iD.** Remember that you can select your job ID directly from the URL when looking at the relevant job in dbt. You can insert these three fields into the following iFrame, and then embed it **anywhere that you can embed an iFrame**: ```html/text <iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token>' title='Exposure Status Tile'></iframe> ``` Replace `YOUR_ACCESS_URL` with your region and plan's Access URL dbt is hosted in multiple regions in the world and each region has a different access URL. Replace `YOUR_ACCESS_URL` with the appropriate [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. For example, if your account is hosted in the EMEA region, you would use the following iFrame code: ```html/text <iframe src='https://metadata.emea.dbt.com/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token>' title='Exposure Status Tile'></iframe> ``` ###### Embedding with BI tools The dashboard status tile should work anywhere you can embed an iFrame. But below are some tactical tips on how to integrate with common BI tools. ##### Mode ###### Mode Mode allows you to directly [edit the HTML](https://mode.com/help/articles/report-layout-and-presentation/#html-editor) of any given report, where you can embed the iFrame. Note that Mode has also built its own [integration](https://mode.com/get-dbt/) with the dbt Discovery API! ##### Looker ###### Looker Looker does not allow you to directly embed HTML and instead requires creating a [custom visualization](https://docs.looker.com/admin-options/platform/visualizations). One way to do this for admins is to: * Add a [new visualization](https://fishtown.looker.com/admin/visualizations) on the visualization page for Looker admins. You can use [this URL](https://metadata.cloud.getdbt.com/static/looker-viz.js) to configure a Looker visualization powered by the iFrame. It will look like this: [![Configure a Looker visualization powered by the iFrame](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/looker-visualization.jpeg?v=2 "Configure a Looker visualization powered by the iFrame")](#)Configure a Looker visualization powered by the iFrame * Once you have set up your custom visualization, you can use it on any dashboard! You can configure it with the exposure name, job ID, and token relevant to that dashboard. [![](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/custom-looker.jpeg?v=2)](#) ##### Tableau ###### Tableau Tableau does not require you to embed an iFrame. You only need to use a Web Page object on your Tableau Dashboard and a URL in the following format: ```html/text https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` Replace `YOUR_ACCESS_URL` with your region and plan's Access URL dbt is hosted in multiple regions in the world and each region has a different access URL. Replace `YOUR_ACCESS_URL` with the appropriate [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. For example, if your account is hosted in the North American region, you would use the following code: ```html/text https://metadata.cloud.getdbt.com/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` [![Configure Tableau by using a Web page object.](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/tableau-object.png?v=2 "Configure Tableau by using a Web page object.")](#)Configure Tableau by using a Web page object. ##### Sigma ###### Sigma Sigma does not require you to embed an iFrame. Add a new embedded UI element in your Sigma Workbook in the following format: ```html/text https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` Replace `YOUR_ACCESS_URL` with your region and plan's Access URL dbt is hosted in multiple regions in the world and each region has a different access URL. Replace `YOUR_ACCESS_URL` with the appropriate [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. For example, if your account is hosted in the APAC region, you would use the following code: ```html/text https://metadata.au.dbt.com/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` [![Configure Sigma by using an embedded UI element.](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/sigma-embed.gif?v=2 "Configure Sigma by using an embedded UI element.")](#)Configure Sigma by using an embedded UI element. --- ### Databricks and Apache Iceberg dbt supports materializing Iceberg tables in Unity Catalog in two ways: * **Simplest:** The model config `table_format = 'iceberg'` instructs dbt to materialize this model as an Iceberg table in Unity Catalog * **Extensible:** Define an Iceberg catalog in `catalogs.yml` and configure this model with `catalog_name` dbt supports creating Iceberg tables for two Databricks materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) #### Databricks Iceberg support Databricks is built on [Delta Lake](https://docs.databricks.com/aws/en/delta/) and stores data in the [Delta table](https://docs.databricks.com/aws/en/introduction/delta-comparison#delta-tables-default-data-table-architecture) format. Databricks supports two methods for creating Iceberg tables in its data catalog, [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/): * Creating [Unity Catalog managed Iceberg tables](https://docs.databricks.com/aws/en/tables/managed). Databricks Runtime 16.4 LTS and later support this feature. * Enabling [Iceberg reads](https://docs.databricks.com/aws/en/delta/uniform) on Delta tables. These tables still use the Delta file format, but generate both Delta and Iceberg-compatible metadata. Databricks Runtime 14.3 LTS and later support this feature. dbt supports both creating managed Iceberg tables and Iceberg-enabled Delta tables (formerly [UniForm](https://www.databricks.com/blog/delta-uniform-universal-format-lakehouse-interoperability)). (Applies to dbt v2.0 and later) The [`use_uniform`](#choose-between-managed-iceberg-and-uniform) config determines whether dbt creates a managed Iceberg table or a Delta table. Because `use_uniform` defaults to `false`, setting `table_format: 'iceberg'` creates a managed Iceberg table. External Iceberg compute engines can read from and write to these Iceberg tables using Unity Catalog's [Iceberg REST API endpoint](https://docs.databricks.com/aws/en/external-access/iceberg). However, Databricks only has limited support for reading from external Iceberg catalogs (and externally managed Iceberg tables) through [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation) (configured outside of dbt). dbt doesn't yet support enabling [Iceberg v3](https://docs.databricks.com/aws/en/iceberg/iceberg-v3) on managed Iceberg tables. (Applies to dbt v2.0 and later) ##### Choose between managed Iceberg and UniForm Set `table_format: 'iceberg'` on a model to create a Unity Catalog managed Iceberg table. You don't need a `catalogs.yml` file or a `catalog_name` config. Two model configs control which kind of Iceberg table dbt creates: | Config | Type | Required | Description | Default | | -------------- | ------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | `table_format` | String | Yes, to create an Iceberg table | Set to `iceberg` to materialize the model as an Iceberg table. | `default` | | `use_uniform` | Boolean | No | When `false`, dbt creates a Unity Catalog managed Iceberg table (`create table ... using iceberg`). When `true`, dbt creates a Delta table with Iceberg reads enabled (`create table ... using delta tblproperties (...)`). | `false` | This model creates a managed Iceberg table: models/my\_iceberg\_model.sql ```sql {{ config( materialized='table', table_format='iceberg' ) }} select * from {{ ref('raw_orders') }} ``` To create an Iceberg-enabled Delta table instead, set `use_uniform` to `true`: models/my\_uniform\_model.sql ```sql {{ config( materialized='table', table_format='iceberg', use_uniform=true ) }} select * from {{ ref('raw_orders') }} ``` ##### External tables dbt also supports creating externally-managed Iceberg tables using the model configuration [`location_root`](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#configuring-tables). Databricks' DDL for creating tables requires a fully qualified `location`. dbt defines this parameter on the user's behalf to streamline usage and enforce basic isolation of table data: * When you set a `location_root` string, dbt generates a `location` string of the form: `{{ location_root }}/{{ model_name }}`. * If you set the configuration option `include_full_name_in_path` to `true`, dbt generates a `location` string of the form `{{ location_root }}/{{ database_name}}/{{ schema_name }}/{{ model_name }}`. In dbt v2, you may set `location_root` within the catalog definition in `catalogs.yml`, under `config.databricks` (in the new catalog spec) or `adapter_properties` (in the old catalog spec). ##### Catalogs Configure catalogs in order to: * Define multiple configurations for Databricks-managed Iceberg tables within one catalog * Support cross-platform Mesh Notes: * (Applies to dbt v2.0 and later) Every Databricks catalog configures `table_format`. `type: unity` requires `iceberg`, and `type: hive_metastore` requires `default`. * On Databricks, `catalog_name` takes precedence over the `catalog` config when determining the model's top-level namespace. #### Configure catalog integration for Iceberg tables 1. Create a `catalogs.yml` at the top level of your dbt project. An example of Unity Catalog as the catalog: ##### (Applies to dbt v2.0 and later) New spec catalogs.yml ```yaml catalogs: - name: unity_catalog type: unity table_format: iceberg config: databricks: # optional location_root: s3://cloud-storage-uri ``` ##### Old spec catalogs.yml ```yaml catalogs: - name: unity_catalog active_write_integration: unity_catalog_integration write_integrations: - name: unity_catalog_integration table_format: iceberg catalog_type: unity file_format: delta adapter_properties: location_root: s3://cloud-storage-uri ``` 2. Add the `catalog_name` config parameter in either a config block (inside the .sql model file), properties YAML file (model folder), or your project YAML file (`dbt_project.yml`). <br /> <br /> An example of `iceberg_model.sql`: ```yaml {{ config( materialized = 'table', catalog_name = 'unity_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` 3. Execute the dbt model with a `dbt run -s iceberg_model`. --- ### dbt AI FAQs Answers to common questions about dbt AI features, including dbt Wizard and dbt Copilot. dbt Wizard is an AI-powered assistant fully integrated into your dbt experience that handles the tedious tasks, speeds up workflows, and ensures consistency, helping you deliver exceptional data products faster. dbt Labs is committed to protecting your privacy and data. This page provides information about how dbt Wizard handles your data. For more information, check out the [dbt Labs AI development principles](https://www.getdbt.com/legal/ai-principles) page. #### Overview  What is dbt Wizard? dbt Wizard is the latest and recommended agentic experience for governed data development in dbt. Wizard is available in the dbt platform, the terminal CLI, and as a [Desktop app](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md). It helps teams ship trusted dbt changes faster and with less risk by understanding project context, routing to the right dbt tools, validating changes, and supporting review before changes are persisted. Use dbt Wizard to investigate lineage and downstream impact, generate or refactor SQL from natural-language prompts, create [documentation](https://docs.getdbt.com/docs/build/documentation.md), [data tests](https://docs.getdbt.com/docs/build/data-tests.md), [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md), and [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md), and validate changes with warehouse awareness. In the dbt platform, dbt Wizard is available in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md) and the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md). dbt Copilot is available in [Canvas](https://docs.getdbt.com/docs/platform/use-canvas.md) and [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md). In the CLI, you can use dbt Wizard from your terminal for local development workflows. In [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md), you get dbt Wizard as a native app. View rich data visualizations, data diffs, and lineage to understand your data and your changes, and keep several conversations going at once, each on its own branch, so you can work on more than one thing at a time.  Where can I find dbt Wizard? dbt Wizard is available in the dbt platform, as a terminal CLI, and as Wizard Desktop. * In the platform, you can use dbt Wizard in the [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) for governed data development in dbt. * In the CLI, use the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) for local development and automation. * In [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md), use a native app with rich data visualizations, data diffs, lineage, and multiple conversations running at once. To use dbt Wizard in the platform, you need any dbt [platform account](https://www.getdbt.com/contact). All dbt platform plans have access to dbt Wizard in Studio IDE and the [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md). AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md).  What are the benefits of using dbt Wizard? dbt Wizard helps teams ship trusted dbt changes faster and with less risk. Use it to: * Ask project-aware questions and investigate lineage, dependencies, and downstream impact. * Generate or refactor SQL from natural-language prompts. * Generate documentation, tests, metrics, and semantic models. * Validate changes with warehouse awareness before review. * Review proposed file changes as diffs before they are persisted. dbt Wizard is built into dbt experiences with dbt governance, privacy, and security controls.  Is dbt Wizard the same as dbt Copilot? No, dbt Wizard and dbt Copilot are separate products. dbt Wizard is the latest and next generation of agentic product available in the dbt platform and as a CLI. It uses your project context to help you develop governed dbt changes faster. Think of it like a smart AI agent that has a map of your project. Instead of having to read through each file and understand the context, it can answer questions and help you develop *and* validate your changes faster. dbt Copilot features for those who have access to them, include quick-action buttons in Studio IDE, the Copilot pane in Insights and Canvas.  Can I use my existing dbt Copilot action allotment with dbt Wizard? No, dbt Copilot actions apply only to dbt Copilot usage. Refer to [dbt Wizard billing and AI access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). #### Availability  Who has access to dbt Wizard? **In the dbt platform**: When enabled by an admin, dbt Wizard is available to users with a dbt [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) on any [dbt platform account](https://www.getdbt.com/contact). **In the CLI**: dbt Wizard CLI uses dbt Labs-managed models (OpenAI, Anthropic, or open weight models), billed through dbt's spend-limit billing. You can also bring your own API key or credentials for a supported provider using [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md): OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, or Snowflake Cortex (preview). BYOK is a good backup option if you want to manage AI costs directly — token costs are billed directly by whichever provider you choose. Install and configure the CLI on your local machine. Refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) for more information.  Is dbt Wizard available for all deployment types? Yes, dbt Wizard is deployed everywhere, including [multi-tenant and single-tenant deployments](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). #### How it works  What data/code is used to train the AI model supporting dbt Wizard? dbt Wizard is supported by dbt Labs-managed models (OpenAI, Anthropic, or open weight models), or by several third-party pre-trained AI models at your discretion (BYOK OpenAI, BYOK Anthropic, BYOK Azure AI Foundry, and so on). When using managed OpenAI, our agreement with OpenAI prohibits OpenAI from retaining your data persistently. Refer to our [dbt Labs AI principles page](https://www.getdbt.com/legal/ai-principles) for more information.  Which AI model providers does dbt Wizard use? In the dbt platform, dbt Wizard uses a managed OpenAI model by default. dbt Labs also offers managed open weight models. On any plan, you can also [bring your own provider keys](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) for OpenAI, Anthropic, or Azure AI Foundry. The [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) supports the same dbt Labs-managed models, or OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, and Snowflake Cortex (preview) in bring-your-own-key mode. Refer to [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) and [Supported AI providers](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#supported-ai-providers) for more information. For how model choice affects cost, which models draw from your consumption pool, and how BYOK billing works, refer to [dbt Wizard billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Do we support BYOK (bring your own key) at the project level? In dbt platform, the dbt Wizard BYOK option is currently an account-only configuration. However, there may be a future where we make this configurable on a project-level. BYOK is a good option if you want to manage AI costs directly, rather than using a dbt Labs-managed model. dbt Wizard CLI supports BYOK locally for OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, and Snowflake Cortex (preview). #### Privacy and data This section covers dbt Wizard in the dbt platform. For what the CLI collects and how to opt out, refer to [dbt Wizard CLI data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md). [![How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.](/img/docs/dbt-platform/wizard-architecture-platform.png?v=2 "How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.")](#)How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider. Your prompt, project metadata, and any query results you approve are sent to the AI provider over TLS. Warehouse credentials are never shared, queries run only after you approve them, and dbt Labs-managed providers can't retain your data or train on it.  Does dbt Wizard access my warehouse data? dbt Wizard can run dbt commands and queries on your behalf, and every query needs your explicit permission first. When a query runs, dbt Wizard sends those results — which may include row-level data — to the AI provider so it can respond in your session. For dbt-managed AI providers, we have zero data retention (ZDR) agreements in place that prevents the provider from retaining or using this data for training. If you bring your own AI provider (BYOK), that provider's terms will govern retention and training. Always review AI output for accuracy. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Does dbt Wizard store or use personal data? dbt Wizard stores your conversation history — including your prompts, responses, and any query results returned during your session — so you can revisit past chats. Conversation history is retained for 90 days; feedback you submit on a dbt Wizard conversation is retained for 400 days. You can delete your conversation history or feedback at any time in the product. You control the information you submit to dbt Wizard. dbt Labs does not use your prompts, conversation history, command results, or feedback to train AI models.  Is my data used by dbt Labs to train AI models? No. dbt Labs does not use customer content processed by dbt Wizard — including warehouse query results, prompts, or conversation history — for AI model training. A zero data retention (ZDR) policy is also in place with AI providers, which prevents training on the provider side as well.  Does dbt Labs share my personal data with third parties? dbt Labs only shares client personal information as needed to perform the services, under client instructions, or for legal, tax, or compliance reasons.  Can dbt Wizard data be deleted upon client written request? Yes. dbt Wizard conversation history is retained for 90 days by default, feedback you submit on a dbt Wizard conversation is retained for 400 days, and you can delete this information in the product at any time. To the extent a client identifies personal or sensitive information uploaded to dbt Labs systems, that data can be deleted within 30 days of written request.  Does dbt Labs own the output generated by dbt Wizard? dbt Labs will not dispute your ownership of any output (e.g., code or artifacts) that are unique to your company generated when you use dbt Wizard. Your code will not be used to train AI models for the benefit of dbt Labs or other third parties, including other dbt Labs customers.  Does dbt Labs have terms in place? dbt Wizard is governed by our [Terms of Use](https://www.getdbt.com/terms-of-use). In the event clients prefer additional terms, clients may enter into the presigned AI & Beta Addendum (the dbt Labs signature will be dated as of the date the client signs). Contact your account manager for more information. Clients who signed with terms after January 2024 don't need additional terms prior to enabling dbt Wizard. Longer term clients have also protected their data through confidentiality and data deletion obligations. In the event clients prefer additional terms, clients may enter into the presigned AI & Beta Addendum (the dbt Labs signature will be dated as of the date the client signs). #### Considerations  What are the considerations for using dbt Wizard? * dbt Wizard is not available in the dbt API. Future releases are planned that may bring dbt Wizard to even more parts of the dbt application. #### dbt Wizard allowlisting URLs  Allowlisting URLs dbt Wizard doesn't specifically block AI-related URLs. However, if your organization use endpoint protection platforms, firewalls, or network proxies (such as Zscaler), you may encounter the following issues with dbt Wizard: * Block unknown or AI-related domains. * Break TLS/SSL traffic to inspect it. * Disallow specific ports or services. We recommend the following URLs to be allowlisted: **For dbt Wizard in the IDE**: * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_generic_tests/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_documentation/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_semantic_model/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_inline` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_metrics/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/track_response` **For dbt Copilot in Canvas**: * `/api/private/visual-editor/v1/ai/llm-generate` * `/api/private/visual-editor/v1/ai/track-response` * `/api/private/visual-editor/v1/files/${fileId}/llm-generate-dag-through-chat` --- ### dbt Catalog FAQs dbt platform [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) is dbt’s new knowledge base and lineage visualization experience. It offers an interactive and high-level view of your company’s entire data estate, where you can dive deep into the context you need to understand and improve lineage so your teams can trust the data they’re using to make decisions. #### Overview  How does dbt Catalog help with data quality? Catalog makes it easy and intuitive to understand your entire lineage — from data source to the reporting layer — so you can troubleshoot, improve, and optimize your pipelines. With built-in features like project recommendations and model performance analysis, you can be sure you have appropriate test and documentation coverage across your estate and quickly spot and remediate slow-running models. With column-level lineage, you can quickly identify the potential downstream impacts of table changes or work backwards to quickly understand the root cause of an incident. Catalog gives teams the insights they need to improve data quality proactively, ensuring pipelines stay performant and data trust remains solid.  How is dbt Catalog priced? Catalog is generally available to all regions and deployment types on all dbt [Enterprise-tier and Starter plans](https://www.getdbt.com/). Certain features within Catalog, such as project recommendations, multi-project lineage, column-level lineage, and more are only available on the Enterprise and Enterprise+ plans. Catalog can be accessed by users with developer and read-only seats.  What happened to dbt Docs? Catalog is the default documentation experience for dbt customers. dbt Docs is still available but doesn't offer the same speed, metadata, or visibility as Catalog and will become a legacy feature. #### How dbt Catalog works  Can I use dbt Catalog on-premises or with my self-hosted dbt deployment? No. Catalog and all of its features are only available as a dbt platform user experience. Catalog reflects the metadata from your dbt platform project(s) and their runs.  How does dbt Catalog support dbt environments? Catalog supports a production or staging [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) for each project you want to explore. It defaults to the latest production or staging state of a project. Users can only assign one production and one staging environment per dbt project. Support for development (dbt platform CLI and Studio IDE) environments is coming soon.  How do I get started in Catalog? How does it update? Simply select **Catalog** from the dbt top navigation bar. Catalog automatically updates after each dbt run in the given project’s environment (production, by default). The dbt commands you run within the environment will generate and update the metadata in Catalog, so make sure to run the correct combination of commands within the jobs of the environment; for more details, refer to [Generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata).  Is it possible to export dbt lineage to an external system or catalog? Yes. The lineage that powers Catalog is also available through the Discovery API.  How does dbt Catalog integrate with third-party tools to show end-to-end lineage? Catalog reflects all the lineage defined within the dbt project. Our vision for Catalog is to incorporate additional metadata from external tools like data loaders (sources) and BI/analytics tools (exposures) integrated with dbt, all seamlessly incorporated into the lineage of the dbt project.  Why did previously visible data in dbt Catalog disappear? Catalog automatically deletes stale metadata after 3 months if no jobs were run to refresh it. To avoid this, make sure you schedule jobs to run more frequently than 3 months with the necessary commands. #### Key features  Does dbt Catalog support multi-project discovery (dbt Mesh)? Yes. Refer to [Explore multiple projects](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) to learn more.  What kind of search capabilities does dbt Catalog support? Resource search capabilities include using keywords, partial strings (fuzzy search), and set operators like `OR`. Meanwhile, lineage search supports using dbt selectors. For details, refer to [Keyword search](https://docs.getdbt.com/docs/explore/explore-projects.md#search-resources).  Can I view model execution information for a job that is currently being run? dbt updates the performance and metrics after a job run. However, **Model performance** charts only display data for *completed* UTC days. This means that runs from the current UTC day won't appear in the charts until the UTC day changes (midnight UTC). For example, if you're in US Pacific time, you won't see the current day's runs reflected until 4:00 PM PT.  Can I analyze the number of successful model runs within a month? A chart of models built by month is available in thedbt dashboard.  Can model or column descriptions be edited within dbt? Yes. Today, you can edit descriptions in the Studio IDE or dbt platform CLI by changing the YAML files within the dbt project. In the future, Catalog will support more ways of editing descriptions.  Where do recommendations come from? Can they be customized? Recommendations largely mirror the best practice rules from the `dbt_project_evaluator` package. At this time, recommendations can’t be customized. In the future, Catalog will likely support recommendation customization capabilities (for example, in project code). #### Column-level lineage  What are the best use cases for column-level lineage in dbt Catalog? Column-level lineage in Catalog can be used to improve many data development workflows, including: * **Audit** — Visualize how data moves through and is used in your dbt project * **Root cause** — Improve time to detect and resolve data quality issues, tracking back to the source * **Impact analysis** — Trace transformations and usage to avoid introducing issues for consumers * **Efficiency** — Prune unnecessary columns to reduce costs and data team overhead  Does the column-level lineage remain functional even if column names vary between models? Yes. Column-level lineage can handle name changes across instances of the column in the dbt project.  Can multiple projects leverage the same column definition? No. Cross-project column lineage is supported in the sense of viewing how a public model is used across projects, but not on a column-level.  Can column descriptions be propagated down in downstream lineage automatically? Yes, a reused column, labeled as passthrough or rename, inherits its description from source and upstream model columns. In other words, source and upstream model columns propagate their descriptions downstream whenever they are not transformed, meaning you don’t need to manually define the description. Refer to [Inherited column descriptions](https://docs.getdbt.com/docs/explore/column-level-lineage.md#inherited-column-descriptions) for more info.  Is column-level lineage also available in the development tab? If you develop in dbt platform, column-level lineage is not in the Studio IDE, but it is available in Catalog for Enterprise and Enterprise+ plans. If you develop locally with dbt v2, the [dbt VS Code extension](https://docs.getdbt.com/docs/dbt-extension-features.md#rich-lineage-in-context) shows column lineage in your editor. #### Availability, access, and permissions  How can non-developers interact with dbt Catalog? Read-only users can consume metadata in Catalog. More bespoke experiences and exploration avenues for analysts and less-technical contributors will be provided in the future.  Does dbt Catalog require a specific dbt plan? Catalog is available on dbt Starter and all Enterprise plans. Certain features within Catalog are only available on Enterprise and Enterprise+ plans, including Model Performance, project recommendations, multi-project lineage, column-level lineage, and more. Refer to the [availability by plan](https://docs.getdbt.com/docs/explore/explore-projects.md#availability-by-plan) table for a complete list.  Will self-hosted dbt users be able to leverage any of these new dbt Catalog features? No. Catalog is a dbt-only product experience.  Is it possible to access dbt Catalog using a read-only license? Yes, users with read-only access can use the Catalog. Specific feature availability within Catalog will depend on your dbt plan.  Is there an easy way to share useful dbt Catalog content with people outside of dbt? The ability to embed and share views is being evaluated as a potential future capability.   Is dbt Catalog accessible from other areas inside dbt? Yes, you can [access Catalog from various dbt features](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md), ensuring you have a seamless experience navigating between resources and lineage in your project. While the primary way to access Catalog is through the **Catalog** link in the navigation, you can also access it from the [Studio IDE](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md#studio-ide), [the lineage tab in jobs](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md#lineage-tab-in-jobs), and the [model timing tab in jobs](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md#model-timing-tab-in-jobs). --- ### dbt Copilot dbt platform | Starter, Enterprise, Enterprise+ dbt Copilot helps you generate SQL, documentation, tests, and semantic models in the dbt platform. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. *The earlier version of dbt Copilot in the Studio IDE is available only to a limited set of accounts. dbt Wizard is available to all accounts and is the recommended way to develop with AI in the Studio IDE — it covers everything dbt Copilot's quick actions did, plus multi-step changes with built-in validation.* #### Where to access dbt Copilot dbt Copilot is available across the following experiences in the dbt platform. Refer to the links for more info on how to use each experience. * [dbt Copilot in Canvas](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md): Build visual models using natural language prompts in Canvas * [dbt Copilot in Insights](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md): Chat with your data and get answers powered by the dbt Semantic Layer in Insights #### Action limits by plan dbt Copilot usage is metered in actions — one completed AI request counts as one action. Each plan includes a monthly action allotment per license: | Plan | Actions per month | | ----------- | ----------------- | | Developer | ❌ | | Starter | 100 | | Enterprise | 5,000 | | Enterprise+ | 10,000 | Enterprise and Enterprise+ limits don't apply if you [bring your own key (BYOK)](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md), since your AI provider bills that usage directly. Legacy Enterprise-tier plans enrolled before May 1, 2025 have a 1,000 action limit. Refer to [dbt AI usage](https://docs.getdbt.com/docs/platform/billing/dbt-ai-usage.md) for what counts as an action, what happens when you hit the limit, and how to check your usage. #### Configure AI provider for dbt Copilot You need dbt admin permissions to change account settings and configure providers. dbt Copilot is configured separately from dbt Wizard — for dbt Wizard providers, refer to [Manage AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider). dbt Copilot supports fewer providers than dbt Wizard, including bring your own key (BYOK) on Starter, Enterprise, and Enterprise+ plans: * dbt Labs-managed OpenAI API key * BYOK OpenAI API key * BYOK Azure OpenAI API key Snowflake Cortex, AWS Bedrock, Azure AI Foundry, and Anthropic aren't supported for dbt Copilot. dbt Copilot is available for inline assistance in Studio IDE, Canvas, and Insights. Configure it separately from dbt Wizard if your team uses these inline AI experiences. **To configure a provider:** 1. Click your account name and select **Account settings** in the side menu. 2. Under **Settings**, click **Copilot**. 3. Under **AI providers**, click **Edit** to configure the AI integration. 4. Select your **Key management** option from the dropdown, then follow the steps for your provider below. ##### dbt Labs OpenAI 1. Select the toggle for **dbt Labs** to use dbt Labs' managed OpenAI key. 2. Click **Save**. [![Example of the dbt Labs integration page](/img/docs/dbt-platform/account-integration-dbtlabs.png?v=2 "Example of the dbt Labs integration page")](#)Example of the dbt Labs integration page ##### OpenAI 1. Select the toggle for **OpenAI** to use your own OpenAI key. 2. Enter the API key. 3. Click **Save**. [![Example of the OpenAI integration page](/img/docs/dbt-platform/account-integration-openai.png?v=2 "Example of the OpenAI integration page")](#)Example of the OpenAI integration page Data residency limitation OpenAI projects with [data residency controls](https://platform.openai.com/docs/guides/your-data#data-residency-controls) enabled and configured for the United States (project region set to US) don't currently support BYOK. These projects can only use the API key in the dbt platform configuration. Specifying custom endpoints required for data residency isn’t yet supported, and we’re evaluating a solution for this. To use BYOK, ensure your OpenAI project doesn’t have data residency controls enabled. Projects without project region settings will use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. ##### Azure OpenAI To learn about deploying your own OpenAI model on Azure, refer to [Deploy models on Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-openai). Configure credentials for your Azure OpenAI deployment in dbt as follows: 1. Locate your Azure OpenAI configuration in your Azure Deployment details page. 2. Enter your Azure OpenAI API key. 3. Enter the **Endpoint**, **API Version**, and **Deployment / Model Name**. 4. Click **Save**. Use the full Azure Target URI For the **Endpoint** field, enter the full Azure Target URI from Azure — not just the base endpoint. Entering only the base endpoint, for example `https://<resource>.openai.azure.com`, prevents credential validation and blocks setup. Supported formats include: * `https://<resource>.openai.azure.com/openai/deployments/<deployment>/chat/completions?api-version=<version>` * `https://<resource>.openai.azure.com/openai/responses?api-version=<version>` [![Example of Azure OpenAI integration section](/img/docs/dbt-platform/account-integration-azure-manual.png?v=2 "Example of Azure OpenAI integration section")](#)Example of Azure OpenAI integration section * For BYOK, enable the latest text generation models as well as the `text-embedding-3-small` model. * Ensure your project doesn't have data residency controls enabled. #### Considerations * dbt Copilot is a separate experience from dbt Wizard. For agentic, full-lifecycle AI development, use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md). * AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * Certain features are only available on Enterprise and Enterprise+ plans. Refer to [Billing](https://docs.getdbt.com/docs/platform/billing.md) for details. * dbt Copilot doesn't yet support generating semantic models with the latest YAML spec. --- ### dbt Copilot in Insights Beta ### dbt Copilot in Insights [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ dbt Copilot in Insights lets you chat with your data and get accurate answers powered by the [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). Unlike generic AI chat interfaces, dbt Wizard in Insights provides consistent, explainable results with transparent SQL, lineage, and data policies. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. #### Prerequisites * Enable beta features under **Account settings** > **Personal profile** > **Experimental features**. See [Preview new dbt platform features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) for steps. * Have access to [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) and meet those prerequisites. * Be on a dbt platform [Enterprise-tier](https://www.getdbt.com/pricing) plan — [book a demo](https://www.getdbt.com/contact) to learn more about Insights. * Available on all [tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) configurations. * Have a dbt [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) with access to Insights. * Configured [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide). #### Using dbt Copilot in Insights Use dbt Copilot to analyze your data and get contextualized results in real time by asking natural language questions to the [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) dbt Wizard in Insights agent. 1. Click the **dbt Copilot** icon in the Query console sidebar menu. 2. In the dropdown menu above the dbt Copilot prompt box, select **Agent**. 3. In the dbt Copilot prompt box, enter your question. 4. Click **↑** to submit your question. The agent then translates natural language questions into structured queries, executes queries against governed dbt models and metrics, and returns results with references, assumptions, and possible next steps. The agent can loop through these steps multiple times if it hasn't reached a complete answer, allowing for complex, multi-step analysis.⁠ dbt Insights automatically executes the SQL query suggested by dbt Copilot in Insights, and you can preview the SQL results in the **Data** tab. 5. Confirm the results or continue asking the agent for more insights about your data. Your conversation with the agent remains even if you switch tabs within dbt Insights. However, they disappear when you navigate out of Insights or when you close your browser. [![Using dbt Copilot in Insights](/img/docs/dbt-insights/insights-copilot-agent.png?v=2 "Using dbt Copilot in Insights")](#)Using dbt Copilot in Insights --- ### dbt Information Schema Available in v2 The dbt Information Schema is a contracted interface into your project’s metadata. You can query it using SQL with [`dbt show`](#querying-with-dbt-show) or in your [checks](#using-the-information-schema-in-checks). The metadata is stored as [Parquet](https://parquet.apache.org/) files, which are more compact and efficient to query than JSON artifacts. For example, a project whose `manifest.json` and `catalog.json` total ~70 MB has an Information Schema of ~5 MB. The Information Schema doesn't use warehouse compute to build or to query it. dbt writes the files from its internal project metadata, using an embedded DuckDB engine that runs inside the dbt process. You don't need to install, configure, or manage DuckDB to use the Information Schema. When you use the [`--generate-info-schema`](#generating-the-information-schema) flag, dbt writes these files to a versioned directory, such as `target/info_schema/v1/`. The available metadata grows as dbt processes your project: * Parsing provides basic project metadata * Compiling adds model column types and column-level lineage (with `--static-analysis strict`) * Running or building adds runtime results Use `dbt show --inline` to run SQL queries or `dbt show --info` to query a view by name. You can also read the files with Parquet-compatible tools such as Pandas or Polars. For available tables and their descriptions, see [Information Schema tables](https://docs.getdbt.com/reference/info-schema.md). dbt Wizard also uses the Information Schema to understand your project. Refer to [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#dbt-information-schema) to learn more. #### Generating the Information Schema Use `--generate-info-schema` or `DBT_ENGINE_GENERATE_INFO_SCHEMA=true` with `dbt build`, `dbt run`, `dbt compile`, or `dbt parse`. By default, the `dbt.node_columns` table contains only YAML-declared columns for both models and sources. * To include inferred model column types and column-level lineage, add `--static-analysis strict`: ```shell dbt build --generate-info-schema --static-analysis strict ``` With this flag, dbt infers column types from your model SQL. Source columns are still limited to what's declared in YAML. * To include warehouse column types and all columns from sources used by the selected nodes, add [`--write-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md#--write-catalog-flag) on `dbt build` or `dbt run`: ```shell dbt build --generate-info-schema --write-catalog dbt build --generate-info-schema --static-analysis strict --write-catalog # to get inferred model column types and column-level lineage ``` * For [`dbt parse`](https://docs.getdbt.com/reference/commands/parse.md), the Information Schema contains no column types, no lineage, and no runtime results, because `dbt parse` doesn't connect to your warehouse. * `dbt show` reads your project's metadata directly, so you don't need to generate anything before using the command. Use `--generate-info-schema` when you want to export the Parquet files to disk (for example, to read them with [external tools](#querying-with-external-tools)). ##### Schema directory The Information Schema Parquet files live in a versioned subdirectory under your project's `target/info_schema/` directory (or wherever you've set with [`--info-schema-dir`](#overriding-the-output-directory)). They aren't stored in your data warehouse. ###### Overriding the output directory Use `--info-schema-dir` or `DBT_ENGINE_INFO_SCHEMA_DIR` to write the Information Schema to a custom directory. The versioned subdirectory (`v1/`) is still appended under whatever directory you set. ```shell dbt build --generate-info-schema --info-schema-dir /tmp/my_schema # writes to /tmp/my_schema/v1/ ``` #### Querying the Information Schema You can query the Information Schema locally with `dbt show` or any Parquet-compatible tool. ##### Querying with `dbt show` Use `dbt show --info <view>` to query a view directly from the CLI: ```shell dbt show --info models dbt show --info models --format json --limit 20 ``` This queries the intermediate views directly, without connecting to your warehouse. `--info <view>` is equivalent to `--inline "select * from {{ info_schema('<view>') }}"`. You can also use `--inline` SQL that calls `{{ info_schema() }}` directly: ```shell dbt show --inline "select name from {{ info_schema('models') }} order by name" ``` ##### Querying with external tools Point any Parquet-compatible tool at the files in `target/info_schema/v1/`. For example, with pandas: ```python import pandas as pd models = pd.read_parquet("target/info_schema/v1/dbt.models.parquet") ``` #### Using the Information Schema in checks [Checks](https://docs.getdbt.com/docs/build/checks.md) are SQL queries that run against the dbt Information Schema to check your project quality. Use the [`{{ info_schema() }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/info-schema-macro.md) macro in your check to reference a view in the dbt Information Schema. You must set the version of the `info_schema` you want to use in `dbt_project.yml`. Currently, `1` is the only available version. ```yaml info_schema: version: 1 ``` You can find the schema version in the versioned subdirectory name (for example, `target/info_schema/v1/`). #### Related docs * [Information Schema tables](https://docs.getdbt.com/reference/info-schema.md) * [Commands for generating metadata](https://docs.getdbt.com/reference/commands/cmd-docs.md#commands-for-generating-metadata) * [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) * [`dbt run`](https://docs.getdbt.com/reference/commands/run.md) * [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) * [`dbt parse`](https://docs.getdbt.com/reference/commands/parse.md) --- ### dbt licensing (Applies to dbt v2.0 and later) dbt v2 has the following distributions today, all free to install and run. | Distribution | Package | Use it when | | ------------ | --------- | ------------------------------------------------------------------------------------ | | dbt v2 | `dbt` | The recommended v2 experience. | | dbt OSS | `dbt-oss` | Your organization has a strict requirement to use the Apache 2.0 open-source runtime | If you have a older project that isn’t ready to move to v2, continue using v1.x for compatibility. For new or upgraded projects, we recommend [upgrading to v2](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md?version=2.0#dbt-v2). #### Which one should I use? For most people: dbt v2. It has more [capabilities](https://docs.getdbt.com/docs/dbt/dbt-availability.md?version=2.0#what-you-get-with-v2) out of the box than the open source v2 — including a built-in high-performance SQL linter — even if you never create a dbt account. We recommend everyone to just [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) and get dbt v2 by default. Typically you'd choose the open source installation directly only if you're in one of two specific situations: 1. Your organization's license policy requires a strict open-source distribution 2. You're building something custom on top of the OSS code itself. Already running dbt v1? You don't have to move to v2 — it's still fully supported. Over time, new capabilities will land in v2 only, so most people will eventually want to [upgrade](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md?version=2.0#dbt-v2). To check which distribution you're using, run `dbt --version` in the command line. #### What changed, and what didn't **Changed:** * v2 is available through two distributions: dbt v2 and dbt OSS. * dbt OSS, the Apache 2.0 open-source distribution for v2, is powered by the shared Rust engine code and is now available in `dbt-oss`. * dbt v2 builds on dbt OSS and extends it with additional proprietary capabilities under the dbt Product Licensing Agreement. **Unchanged:** * dbt v1 is still fully available and still Apache 2.0. * dbt v2 is still completely free to use, with some features unlocked by a free login or a paid dbt platform account — not required for any distribution. * Contributing to dbt is still open to everyone. #### Licensing details [dbt v1](https://github.com/dbt-labs/dbt) is released under the [Apache 2.0 license](http://www.apache.org/licenses/LICENSE-2.0). dbt v2 is proprietary to dbt Labs, made available under the [dbt Product Licensing Agreement](https://www.getdbt.com/dbt-product-license-agreement). For the full breakdown of what's permitted under each license — source visibility, contributions, modifications, self-hosting, and redistribution — see the [dbt Licensing FAQ](https://www.getdbt.com/licenses-faq). dbt platform is a separate hosted product governed by its own [terms of service](https://www.getdbt.com/terms-of-use). Also not to be confused with dbt platform [licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). --- ### dbt MCP FAQs and troubleshooting Find answers to common questions about the dbt Model Context Protocol (MCP) server and fixes for common setup issues. For client-specific steps, see the troubleshooting sections in the [Get started](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) and [Setup and config](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) pages. #### FAQs The `execute_sql` tool in the dbt MCP server lets your AI assistant run SQL in your warehouse. These FAQs cover how it works and what its limits are.  What does execute\_sql do? `execute_sql` lets your AI assistant run a SQL query against your data warehouse and read the results. It's part of the SQL toolset and runs on dbt platform, so you don't need dbt installed locally. Here's what happens when your assistant calls it: 1. The assistant writes a SQL query and calls `execute_sql`. 2. The dbt MCP server forwards the request to dbt platform, along with your token, development environment ID, and user ID. 3. dbt platform compiles the SQL (including any Jinja, such as `ref()`) and runs it in your warehouse using your development credentials. 4. The results come back to your assistant as a table. [![The AI assistant calls execute\_sql. The dbt MCP server forwards the request to dbt platform, which compiles the SQL, runs it in the warehouse with your development credentials, and returns the results to the assistant.](/img/docs/dbt-ai/mcp-execute-sql-flow.svg?v=2 "The AI assistant calls execute_sql. The dbt MCP server forwards the request to dbt platform, which compiles the SQL, runs it in the warehouse with your development credentials, and returns the results to the assistant.")](#)The AI assistant calls execute\_sql. The dbt MCP server forwards the request to dbt platform, which compiles the SQL, runs it in the warehouse with your development credentials, and returns the results to the assistant.  Is execute\_sql the same as dbt show? No. `dbt show` runs through the dbt platform CLI tools and previews a model's output while you develop. `execute_sql` sends SQL to dbt platform, which runs it in your warehouse.  Whose warehouse credentials does execute\_sql use? Your personal development credentials for the project's development environment. These are the same credentials you use in Studio IDE. It doesn't use your production deployment credentials. If you don't have development credentials, the query fails with a message that no warehouse credentials are configured. Ask a project admin to set them up.  Is execute\_sql read-only? No. `execute_sql` doesn't block write statements. What a query can do depends on the permissions of the warehouse credentials it runs with. If those credentials can create or drop tables, so can your assistant. To keep things safe: * **Use least-privilege warehouse roles.** Give each user's development credentials read-only access to production-like data and write access only to their own development schema. * **Keep tool approval prompts on.** Your assistant then asks before running SQL, so you can review it first. * **Don't count on disabling the tool as a security control.** You can turn off `execute_sql` with the `x-dbt-disable-tools` header, but whoever configures the MCP client sets it, so admins can't enforce it. Warehouse permissions are the control that holds.  Who can use execute\_sql? Anyone with a developer seat or a read-only seat, as long as they connect with OAuth or a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens don't work because development credentials belong to a person. Read-only seats can run plain SQL. SQL that contains Jinja, like `ref()` or `source()`, needs a developer seat. For setup details, see [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md).  Why doesn't ref() point to my development tables? Even though the query runs with your development credentials, `ref('model')` resolves to the production relation. It can't point to a table you built in your own development schema. To query a development table, use its full name instead, for example `my_dev_schema.my_model`. Avoid `source()` too, and use full table names or `ref()` instead.  Are there limits on query size or run time? Yes. Long-running queries time out, and dbt caps large result sets. Add filters and `limit` clauses to keep queries small and fast.  Can I see results as a chart or table? Yes you can! Your AI tool can ask for results to show as a chart or table in the UI. Otherwise, only your assistant sees the results.  When should I use something other than execute\_sql? | If you want to... | Use | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Ask governed business questions with defined metrics | The [Semantic Layer tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#semantic-layer), like `query_metrics` | | Preview a model's output while you develop | dbt platform CLI tools, like `show`, `build`, and `run` | | Turn a question into SQL without running it | `text_to_sql` | #### Troubleshooting Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). Configuration not working in WSL (VS Code) **Symptoms:** VS Code MCP config works on other machines or native Windows but not in WSL. **Cause:** Local user settings are not applied in WSL environments. **Solution:** Configure MCP in the WSL-specific settings instead of local user settings: 1. Open the Command Palette → **Preferences: Open Remote Settings**. 2. Or select the **Remote** tab in the Settings editor. 3. Add your MCP server configuration there. execute\_sql tool not working **Symptoms:** The `execute_sql` tool returns an authentication error or is unavailable. **Cause:** How you fix this depends on how you connected: * **Remote MCP with OAuth:** Sign in through your MCP client when prompted. You don't need a personal access token. * **Remote MCP with token-based auth:** You need a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. You also need `x-dbt-dev-environment-id` and `x-dbt-user-id` headers. The user ID must match the user who created the PAT. * **Local MCP:** Set `DBT_TOKEN` to a personal access token (PAT). Service tokens won't work. **Solution:** 1. If you use OAuth for remote MCP, confirm you completed sign-in and consent in your MCP client. 2. If you use token-based auth or local MCP, create a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) in **Account settings** → **API tokens** → **Personal tokens**. 3. For token-based remote MCP, add the PAT to the `Authorization` header as `Token YOUR_PAT`. For local MCP, set the PAT as your `DBT_TOKEN` value. 4. Also ensure `DBT_DEV_ENV_ID` and `DBT_USER_ID` are set (local MCP) or `x-dbt-dev-environment-id` and `x-dbt-user-id` headers are set (remote MCP). Refer to [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. Toolset unavailable or showing as disabled **Symptoms:** A toolset (Semantic Layer, Discovery, Admin API) is not available in your AI client even though you've configured credentials. **Cause:** Either the required variables are missing, or the toolset has been explicitly disabled. **Solution:** 1. Check that all required variables for the toolset are set — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). 2. Check whether you have any `DISABLE_*` variables set to `true` that might be turning off the toolset. 3. If you're using enable mode (`DBT_MCP_ENABLE_*`), make sure the toolset you need is listed. 4. Set `DBT_MCP_LOG_LEVEL=DEBUG` to see which toolsets are active at startup. Pasting full URLs instead of IDs **Symptoms:** Authentication errors, unexpected behavior, or the server failing to connect to the right environment. **Cause:** Environment variables like `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_ACCOUNT_ID` expect numeric integers, not full browser URLs. **Solution:** ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. Multi-cell or static subdomain account connection issues **Symptoms:** Connection errors when your account URL includes a prefix (for example, `abc123.us1.dbt.com`). **Solution (as of v1.14.0):** Set `DBT_HOST` to the full hostname including the prefix. If you're using PAT-based auth, also set `DBT_ACCOUNT_ID`. ```bash # ✅ Correct DBT_HOST=abc123.us1.dbt.com DBT_ACCOUNT_ID=12345 # required for PAT-based auth ``` You no longer need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. If you have these set from an older configuration, remove them. Claude Desktop errors Claude Desktop may return errors such as `Error: spawn uvx ENOENT`, `Error: spawn npx ENOENT`, `Could not connect to MCP server dbt-mcp`, or `Server disconnected`. For self-hosted installations, replace the `command` with the full path to `uvx`: run `which uvx` on Unix systems or `where uvx` on Windows and paste the full path into your JSON (for example, `"command": "/the/full/path/to/uvx"`). For token-based remote setups with `mcp-remote`, apply the same full-path fix for `npx`. If Claude reports that entries in `claude_desktop_config.json` are not valid and were skipped, check that you are not using `"type": "http"` in that file. Use the [token-based `mcp-remote` config](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md#desktop-remote) instead. For remote setups, double-check that the MCP URL ends in `/api/ai/v1/mcp/` (including the trailing slash) and that your `Authorization` header value is `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Logs are at `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). Claude Code can't connect to the dbt MCP server If the dbt MCP server doesn't connect, confirm `.mcp.json` is at the *project root* and that the `dbt` block matches one of the examples on the [Claude setup page](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md#claude-code). For self-hosted installations, apply the same full-path fix for `uvx` (and for `--env-file` paths). For remote setups, verify the URL and headers, and try the equivalent `claude mcp add --transport http` command to compare. The connector won't authorize or the OAuth flow fails * Confirm your account has a [static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). OAuth with MCP requires one. * Verify the host in `API_ALLOWED_PREFIXES`, `OAUTH_TOKEN_ENDPOINT`, `OAUTH_AUTHORIZATION_ENDPOINT`, and `OAUTH_RESOURCE_URL` all match your MCP URL host exactly, and that the integration `ENABLED = TRUE`. * AI features are enabled by default and admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * Make sure remote MCP OAuth is available for your account plan. The agent returns no metrics or empty results * Confirm the project you authorized has a configured [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) with metrics and dimensions. * Check that the user who connected has at least read-only access to that project — the agent only sees what the user can see. * If you scoped the OAuth connection to a single project, make sure it's the project that contains your metrics. Permission errors when creating the integration or MCP server `CREATE API INTEGRATION` and `CREATE EXTERNAL MCP SERVER` require `ACCOUNTADMIN` (or a role with account-level `CREATE INTEGRATION`). Run steps 1–2 as `ACCOUNTADMIN`, then grant `USAGE` on the MCP server and integration to the role that runs the agent (Step 3). --- ### dbt platform v1 Compatible track - changelog dbt platform | Free Select the **v1 Compatible** and **v1 Extended** release tracks if you need a less-frequent release cadence, the ability to test new dbt releases before they go live in production, and/or ongoing compatibility with the latest open source releases of dbt v1. Each monthly **v1 Compatible** release includes functionality matching up-to-date open source versions of dbt v1 and adapters at the time of release. For more information, see [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### October 2026 Release date: October 15, 2026 ##### dbt platform ###### Breaking changes * The default value of the following legacy behavior-change flags is now `true`: `require_batched_execution_for_custom_microbatch_strategy`, `skip_nodes_if_on_run_start_fails`, `state_modified_compare_more_unrendered_values`, `require_yaml_configuration_for_mf_time_spines`, `require_nested_cumulative_type_params`, `validate_macro_args`, and `require_all_warnings_handled_by_warn_error`. This change ports dbt-core#15489 now that the customer-communication deadline has passed. ###### Features * Add OpenTelemetry span instrumentation for node and hook execution, gated behind the `--snowflake-projects-otel` flag (off by default). * You can now resolve private packages (`packages.private` in `packages.yml`) through the dbt platform's private-packages resolve endpoint, authenticated with an orchestration-injected HMAC token (`DBT_CLOUD_PRIVATE_PACKAGES_AUTH_TOKEN`, `DBT_CLOUD_ACCOUNT_ID`, `DBT_CLOUD_ACCOUNT_HOST`, `DBT_CLOUD_PROJECT_ID`). dbt checks this endpoint after the existing `DBT_ENV_PRIVATE_GIT_PROVIDER_INFO` legacy resolution, which is unchanged and still takes precedence when configured. dbt resolves a private package discovered after the initial resolve (for example, a transitive private dependency) lazily through the same endpoint rather than falling back to SSH, because SSH credentials are never available under dbt platform orchestration. dbt raises an error immediately if it can't resolve any requested package, rather than falling back silently. * Accept Mesh environment routing metadata in cross-project dependencies. * dbt now warns you if you use a deprecated dbt version. * Add `otel_trace_id()` and `otel_span_id()` Jinja functions to correlate query comments and model SQL with OpenTelemetry trace and span IDs. ###### Fixes * Recognize Databricks adapter config keys (`query_tags`, `zorder`, `options`, `unique_tmp_table_suffix`, `skip_optimize`) in JSON schema validation so they no longer raise spurious `CustomKeyInConfigDeprecation` warnings (dbt-databricks#1540, dbt-databricks#1302). * Fix the `dbt build` node count ("X of Y") being inflated when you select unit tests. dbt now adds the unit test count to `num_nodes` exactly once, instead of on every dequeue while `run_count` is still 0. * Stop resolving a unit test's tested model eagerly at parse time. dbt now resolves it only in the single deterministic post-parse pass, so resolution no longer depends on filesystem parse order at all. * Fix OpenTelemetry hook spans: name them by hook and transaction phase, skip calls that run no hooks, and add node context and status. * Send the `x-ms-blob-type: BlockBlob` header and accept `201 Created` when uploading artifacts to an Azure Blob Storage signed URL, so artifact ingestion no longer fails with `400 MissingRequiredHeader` on Azure-hosted regions. Upload errors now include the storage provider's error code alongside the status code. * Resolve a function's `unrendered_config` from the `functions:` project config block instead of falling through to `models:`, so model-level defaults no longer appear in function configs and make functions compare as modified under `state:modified`. * Backport v2 parser fixes: default `--v2-parser` to the bundled dbt v2 binary instead of `dbt-core-experimental-parser`, re-parse adapter macros from the installed adapter package after loading a dbt v2-generated manifest (ports dbt-core#15529), and rediscover the built-in generic test macros so they aren't left undefined at test-compile time (ports dbt-core#15925). * Capture and re-emit dbt v2 parser subprocess output through dbt-core's event system instead of letting it pass through to raw stdout and stderr, re-leveling each line by its JSON-compatible severity when available. * Validate that a resolved package installation path stays within the configured `packages-install-path`. * Fix `Dimension` to satisfy the `metricflow-semantic-interfaces` protocol and bump the `metricflow` floor to 0.213.0. ###### Dependencies * Bump the `dbt-core-experimental-parser` lower bound to 2.0.0b1. * Require `sqlparse>=0.5.5,<0.7.0`. Version 0.6.0 is the fixed release for CVE-2026-54284, CVE-2026-59893, CVE-2026-59894, and CVE-2026-71491 (parser denial-of-service). This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.12.5 # shared interfaces dbt-adapters==1.24.5 dbt-common==1.38.0 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.4 dbt-state==2.61.0 # adapters dbt-athena==1.11.1 dbt-bigquery==1.12.1 dbt-databricks==1.12.7 dbt-fabric==1.9.4 dbt-postgres==1.11.0 dbt-redshift==1.11.1 dbt-snowflake==1.12.3 dbt-spark==1.10.3 dbt-synapse==1.8.5 dbt-teradata==1.11.0 dbt-trino==1.10.6 ``` Changelogs: * [dbt-core 1.12.5](https://github.com/dbt-labs/dbt/blob/1.12.latest/CHANGELOG.md) * [dbt-adapters 1.24.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.38.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-state 2.61.0](https://github.com/dbt-labs/dbt-state) * [dbt-athena 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.12.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.12.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.12.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.5](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.11.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.11.0) * [dbt-trino 1.10.6](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### August 2026 Release date: August 18, 2026 This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.12.0 # shared interfaces dbt-adapters==1.24.5 dbt-common==1.38.0 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.4 dbt-state==2.42.0 # adapters dbt-athena==1.11.0 dbt-bigquery==1.12.0 dbt-databricks==1.12.4 dbt-fabric==1.9.4 dbt-postgres==1.11.0 dbt-redshift==1.11.0 dbt-snowflake==1.12.0 dbt-spark==1.10.3 dbt-synapse==1.8.5 dbt-teradata==1.11.0 dbt-trino==1.10.3 ``` Changelogs: * [dbt-core 1.12.0](https://github.com/dbt-labs/dbt/blob/1.12.latest/CHANGELOG.md) * [dbt-adapters 1.24.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.38.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-state 2.42.0](https://github.com/dbt-labs/dbt-state) * [dbt-athena 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.12.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.12.4](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.12.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.5](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.11.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.11.0) * [dbt-trino 1.10.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### July 2026 Release date: July 21, 2026 ##### dbt platform ##### Fixes * Prune stale cross-project dependencies from injected external nodes before linking. A publication can list a transitive public ancestor that has since been deleted upstream; injecting it as a dangling `depends_on` edge previously failed compile with `GraphDependencyNotFoundError`. The dead edge is now dropped with a warning and restored once the producing project's publication is regenerated. ##### Dependencies * Pin `pyathena` to v3.35 and older This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.12 # shared interfaces dbt-adapters==1.22.10 dbt-common==1.37.5 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.4 # adapters dbt-athena==1.10.2 dbt-bigquery==1.11.1 dbt-databricks==1.12.2 dbt-fabric==1.9.4 dbt-postgres==1.10.2 dbt-redshift==1.10.2 dbt-snowflake==1.11.6 dbt-spark==1.10.3 dbt-synapse==1.8.5 dbt-teradata==1.10.3 dbt-trino==1.10.2 ``` Changelogs: * [dbt-core 1.11.12](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt-adapters 1.22.10](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.37.5](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-athena 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.12.2](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.11.6](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.5](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.3](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.3) * [dbt-trino 1.10.2](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### June 2026 Release date: June 15, 2026 ##### dbt cloud-based platform This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.11 # shared interfaces dbt-adapters==1.22.10 dbt-common==1.37.5 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.4 # adapters dbt-athena==1.10.1 dbt-bigquery==1.11.1 dbt-databricks==1.12.1 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.1 dbt-snowflake==1.11.5 dbt-spark==1.10.1 dbt-synapse==1.8.5 dbt-teradata==1.10.2 dbt-trino==1.10.2 ``` Changelogs: * [dbt-core 1.11.11](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt-adapters 1.22.10](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.37.5](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-athena 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.12.1](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.11.5](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.5](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.2) * [dbt-trino 1.10.2](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### May 2026 The compatible release scheduled for May 2026 will be skipped in order to stabilize `dbt-core 1.12.0` release across the dbt platform. Compatible releases will resume in June 2026. #### April 2026 Release date: April 21, 2026 ##### dbt cloud-based platform This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.8 # shared interfaces dbt-adapters==1.22.10 dbt-common==1.37.3 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.2 # adapters dbt-athena==1.10.0 dbt-bigquery==1.11.1 dbt-databricks==1.11.7 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.1 dbt-snowflake==1.11.4 dbt-spark==1.10.1 dbt-synapse==1.8.4 dbt-teradata==1.10.2 dbt-trino==1.10.1 ``` Changelogs: * [dbt-core 1.11.8](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt-adapters 1.22.10](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.37.3](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-athena 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.11.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.11.4](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.2) * [dbt-trino 1.10.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### March 2026 The compatible release scheduled for March 2026 will be skipped in order to further stabilize the minor upgrade of `dbt-core==1.11.6` across the dbt platform. Compatible releases will resume in April 2026. #### February 2026 Release date: February 27, 2026 ##### dbt cloud-based platform ##### Features * Support partial success result status for Advanced CI ##### Dependencies * Update dbt-databricks upper bound to 1.12 This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.6 # shared interfaces dbt-adapters==1.22.6 dbt-common==1.37.2 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.1 # adapters dbt-athena==1.10.0 dbt-bigquery==1.11.0 dbt-databricks==1.11.5 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.1 dbt-snowflake==1.11.2 dbt-spark==1.10.1 dbt-synapse==1.8.4 dbt-teradata==1.10.1 dbt-trino==1.10.1 ``` Changelogs: * [dbt-core 1.11.6](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md#dbt-core-1116---february-17-2026) * [dbt-adapters 1.22.6](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1226---february-17-2026) * [dbt-common 1.37.2](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1372---december-15-2025) * [dbt-athena 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md#dbt-athena-1100---december-22-2025) * [dbt-bigquery 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1110---december-22-2025) * [dbt-databricks 1.11.5](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-1115-feb-19-2026) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md#dbt-postgres-1100---december-22-2025) * [dbt-redshift 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md#dbt-redshift-1101---february-11-2026) * [dbt-snowflake 1.11.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md#dbt-snowflake-1112---february-11-2026) * [dbt-spark 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md#dbt-spark-1101---february-11-2026) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.1) * [dbt-trino 1.10.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-1101---january-16-2026) #### January 2026 Release date: January 22, 2026 ##### dbt cloud-based platform ##### Under the Hood * Add debug log for local md5 hash for dbt v2 conformance * Resolve Click CLI UserWarning regarding --target and --profile usage in Advanced CI This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.19 # shared interfaces dbt-adapters==1.22.5 dbt-common==1.37.2 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.1 # adapters dbt-athena==1.10.0 dbt-bigquery==1.11.0 dbt-databricks==1.10.19 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.0 dbt-snowflake==1.11.1 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.1 dbt-trino==1.10.1 ``` Changelogs: * [dbt-core 1.10.19](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11019---january-20-2026) * [dbt-adapters 1.22.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1225---january-14-2026) * [dbt-common 1.37.2](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1372---december-15-2025) * [dbt-athena 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md#dbt-athena-1100---december-22-2025) * [dbt-bigquery 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1110---december-22-2025) * [dbt-databricks 1.10.19](https://github.com/databricks/dbt-databricks/blob/1.10.latest/CHANGELOG.md#dbt-databricks-11019-jan-21-2026) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md#dbt-postgres-1100---december-22-2025) * [dbt-redshift 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md#dbt-redshift-1100---december-22-2025) * [dbt-snowflake 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md#dbt-snowflake-1111---january-08-2026) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.1) * [dbt-trino 1.10.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-1101---january-16-2026) #### December 2025 Release date: December 9, 2025 ##### dbt cloud-based platform This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.15 # shared interfaces dbt-adapters==1.16.7 dbt-common==1.33.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.3 dbt-databricks==1.10.15 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.3 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.15](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11015---november-12-2025) * [dbt-adapters 1.16.7](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.33.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1330---october-20-2025) * [dbt-athena 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.15](https://github.com/databricks/dbt-databricks/blob/1.10.latest/CHANGELOG.md#dbt-databricks-11015-nov-17-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.3](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### November 2025 Release date: November 11, 2025 ##### dbt cloud-based platform ##### Under the Hood * Record source column schemas when `DBT_RECORDER_MODE` is set * Issue additional column schema retrieval for hardcoded relation references in SQL * Make source schema recording cache thread-safe * Record column schemas for deferred relations and unselected dependencies This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.15 # shared interfaces dbt-adapters==1.16.7 dbt-common==1.33.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.3 dbt-databricks==1.10.15 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.3 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.15](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11015---november-12-2025) * [dbt-adapters 1.16.7](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.33.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1330---october-20-2025) * [dbt-athena 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.15](https://github.com/databricks/dbt-databricks/blob/1.10.latest/CHANGELOG.md#dbt-databricks-11015-nov-17-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.3](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### October 2025 Release date: October 23, 2025 ##### dbt cloud-based platform ##### Under the Hood * Add instrumentation to adapter methods for reliable debugging traces at the adapter boundary This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.13 # shared interfaces dbt-adapters==1.16.7 dbt-common==1.33.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.2 dbt-databricks==1.10.14 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.2 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.13](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11013---september-25-2025) * [dbt-adapters 1.16.7](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.33.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1330---october-20-2025) * [dbt-athena 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.14](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-11014-october-22-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.2](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### September 2025 Release date: September 10, 2025 This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.11 # shared interfaces dbt-adapters==1.16.6 dbt-common==1.29.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.2 dbt-databricks==1.10.12 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-protos==1.0.348 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.2 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.11](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11011---september-04-2025) * [dbt-adapters 1.16.6](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.29.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1290---september-04-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.12](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-11012-september-8-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.2](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### August 2025 Release date: August 12, 2025 ##### Notable dbt v1 OSS changes This compatible upgrade brings in a minor update to `dbt-core`, from `dbt-core==1.9.8` to `dbt-core==1.10.8`. Some noteworthy changes from this minor version include: * Introduction of several new [deprecations](https://docs.getdbt.com/reference/deprecations.md) that warn about project incompatibilities between dbt v1 and v2. * Support for defining `meta` and `tags` within `config` of columns and exposures, as well as defining `freshness` within `config` of sources. These changes lead to manifest.json minor schema evolutions which may cause an intermittent increase in false positives during `state:modified` comparisons. ##### dbt cloud-based platform ##### Fixes * Update generate publications script to add project and env id in generated publication file * Use JSON stream for publication artifact generation script * Get environment variables correctly from environment for publication artifacts * Adding `--resource-type` and `--exclude-resource-type` flags to Semantic Layer commands * Azure DevOps Private Packages are now properly matched with Private Package Definition in packages.yml ##### Under the Hood * Prepare support for Private Package's URLs with multiple levels * Disable telemetry client logger * Update semantic layer SDK to 0.11 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.8 # shared interfaces dbt-adapters==1.16.3 dbt-common==1.27.1 dbt-semantic-interfaces==0.9.0 dbt-extractor==0.6.0 dbt-protos==1.0.348 # dbt-adapters dbt-athena==1.9.4 dbt-bigquery==1.10.1 dbt-databricks==1.10.10 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-redshift==1.9.5 dbt-snowflake==1.10.0 dbt-spark==1.9.3 dbt-synapse==1.8.2 dbt-teradata==1.9.3 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.8](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-1108---august-12-2025) * [dbt-adapters 1.16.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1163---july-21-2025) * [dbt-common 1.25.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1271---july-21-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-1109-august-7-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-1100-rc3---june-24-2025) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.3](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.3) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### July 2025 The compatible release slated for July 2025 will be skipped in order to further stabilize the minor upgrade of `dbt-core==1.10.0` ([released June 16, 2025](https://pypi.org/project/dbt-core/1.10.0/)) across the dbt platform. Compatible releases will resume in August 2025. #### June 2025 Release date: June 12, 2025 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.8 # shared interfaces dbt-adapters==1.15.3 dbt-common==1.25.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.4 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-protos==1.0.317 dbt-redshift==1.9.5 dbt-sl-sdk-internal[sync]==0.7.0 dbt-sl-sdk[sync]==0.7.0 dbt-snowflake==1.9.4 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.2 dbt-trino==1.9.2 ``` Changelogs: * [dbt-core 1.9.8](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-198---june-10-2025) * [dbt-adapters 1.15.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1153---may-20-2025) * [dbt-common 1.25.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1250---may-20-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-194---may-02-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.2) * [dbt-trino 1.9.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-192---june-03-2025) #### May 2025 Release date: May 19, 2025 ##### dbt cloud-based platform These changes reflect capabilities that are only available in the dbt platform. ##### Fixes * Get environment variables correctly from the environment for publication artifacts ##### Under the hood * Create JSON schemas for PublicationArtifact and ResolvedProjectsArtifact This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.4 # shared interfaces dbt-adapters==1.14.8 dbt-common==1.24.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.4 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-redshift==1.9.5 dbt-snowflake==1.9.4 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.2 dbt-trino==1.9.1 ``` Changelogs: * [dbt-core 1.9.4](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-194---april-02-2025) * [dbt-adapters 1.14.8](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1148---april-25-2025) * [dbt-common 1.24.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1240---may-09-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-194---may-02-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.2) * [dbt-trino 1.9.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-191---march-26-2025) #### April 2025 Release date: April 9, 2025 ##### dbt Cloud These changes reflect capabilities that are only available in dbt Cloud. ##### Under the Hood * Add secondary profiles to profile.py This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.4 # shared interfaces dbt-adapters==1.14.5 dbt-common==1.17.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.3 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-redshift==1.9.3 dbt-snowflake==1.9.2 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.2 dbt-trino==1.9.1 ``` Changelogs: * [dbt-core 1.9.4](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-194---april-02-2025) * [dbt-adapters 1.14.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1145---april-07-2025) * [dbt-common 1.17.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1170---march-31-2025) * [dbt-athena 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-193---april-07-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-193---april-01-2025) * [dbt-snowflake 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-192---march-07-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.2) * [dbt-trino 1.9.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-191---march-26-2025) #### March 2025 Release date: March 11, 2025 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.3 # shared interfaces dbt-adapters==1.14.1 dbt-common==1.15.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.2 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-fabric==1.9.2 dbt-postgres==1.9.0 dbt-redshift==1.9.1 dbt-snowflake==1.9.2 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.1 dbt-trino==1.9.0 ``` Changelogs: * [dbt 1.9.3](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-193---march-07-2025) * [dbt-adapters 1.14.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1141---march-04-2025) * [dbt-common 1.15.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1150---february-14-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.2](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.2) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-191---march-07-2025) * [dbt-snowflake 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-192---march-07-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.1) * [dbt-trino 1.9.0](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-190---december-20-2024) #### February 2025 Release date: February 12, 2025 ##### dbt Cloud These changes reflect capabilities that are only available in dbt. ##### Features * Add [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) to cross-project ref artifact. * Include debug exception message in ObservabilityMetric. ##### Fixes * Adding support for deferral against the new time spine definition. * Fix error messages for SL query. * Semantic Layer commands now respect `--favor-state` when running with `--defer`. This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.2 # shared interfaces dbt-adapters==1.14.0 dbt-common==1.14.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.1 dbt-bigquery==1.9.1 dbt-databricks==1.9.4 dbt-fabric==1.9.0 dbt-postgres==1.9.0 dbt-redshift==1.9.0 dbt-snowflake==1.9.1 dbt-spark==1.9.1 dbt-synapse==1.8.2 dbt-teradata==1.9.1 dbt-trino==1.9.0 ``` Changelogs: * [dbt 1.9.2](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-192---january-29-2025) * [dbt-adapters 1.14.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1140---february-07-2025) * [dbt-common 1.14.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.4](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-194-jan-30-2024) * [dbt-fabric 1.9.0](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.0) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.0](https://github.com/dbt-labs/dbt-redshift/blob/1.9.latest/CHANGELOG.md#dbt-redshift-190---december-09-2024) * [dbt-snowflake 1.9.1](https://github.com/dbt-labs/dbt-snowflake/blob/1.9.latest/CHANGELOG.md#dbt-snowflake-191---february-07-2025) * [dbt-spark 1.9.1](https://github.com/dbt-labs/dbt-spark/blob/1.9.latest/CHANGELOG.md#dbt-spark-191---february-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.1) * [dbt-trino 1.9.0](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-190---december-20-2024) #### January 2025 Release date: January 14, 2025 ##### dbt Cloud These changes reflect capabilities that are only available in dbt. ##### Features * Filter out external exposures in dbt compare. ##### Fixes * Use `meta.dbt_cloud_id` to `build unique_id` for manually defined exposure for merging against a duplicated exposure. This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.1 # shared interfaces dbt-adapters==1.13.1 dbt-common==1.14.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.0 dbt-bigquery==1.9.1 dbt-databricks==1.9.1 dbt-fabric==1.9.0 dbt-postgres==1.9.0 dbt-redshift==1.9.0 dbt-snowflake==1.9.0 dbt-spark==1.9.0 dbt-synapse==1.8.2 dbt-teradata==1.9.0 dbt-trino==1.9.0 ``` Changelogs: * [dbt 1.9.1](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-191---december-16-2024) * [dbt-adapters 1.13.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1131---january-10-2025) * [dbt-common 1.14.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.1](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-191-december-16-2024) * [dbt-fabric 1.9.0](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.0) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.0](https://github.com/dbt-labs/dbt-redshift/blob/1.9.latest/CHANGELOG.md#dbt-redshift-190---december-09-2024) * [dbt-snowflake 1.9.0](https://github.com/dbt-labs/dbt-snowflake/blob/1.9.latest/CHANGELOG.md#dbt-snowflake-190---december-09-2024) * [dbt-spark 1.9.0](https://github.com/dbt-labs/dbt-spark/blob/1.9.latest/CHANGELOG.md#dbt-spark-190---december-10-2024) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.0) * [dbt-trino 1.9.0](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-190---december-20-2024) #### December 2024 Release date: December 12, 2024 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.0 # shared interfaces dbt-adapters==1.10.4 dbt-common==1.14.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.0 dbt-bigquery==1.9.0 dbt-databricks==1.9.0 dbt-fabric==1.8.8 dbt-postgres==1.9.0 dbt-redshift==1.9.0 dbt-snowflake==1.9.0 dbt-spark==1.9.0 dbt-synapse==1.8.2 dbt-teradata==1.8.2 dbt-trino==1.8.5 ``` Changelogs: * [dbt 1.9.0](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-190---december-09-2024) * [dbt-adapters 1.10.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1104---november-11-2024) * [dbt-common 1.14.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-bigquery 1.9.0](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-190---december-09-2024) * [dbt-databricks 1.9.0](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-190-december-9-2024) * [dbt-fabric 1.8.8](https://github.com/microsoft/dbt-fabric/blob/v1.8.latest/CHANGELOG.md) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.0](https://github.com/dbt-labs/dbt-redshift/blob/1.9.latest/CHANGELOG.md#dbt-redshift-190---december-09-2024) * [dbt-snowflake 1.9.0](https://github.com/dbt-labs/dbt-snowflake/blob/1.9.latest/CHANGELOG.md#dbt-snowflake-190---december-09-2024) * [dbt-spark 1.9.0](https://github.com/dbt-labs/dbt-spark/blob/1.9.latest/CHANGELOG.md#dbt-spark-190---december-10-2024) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.8.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.8.2) * [dbt-trino 1.8.5](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-185---december-11-2024) --- ### dbt release notes dbt platform dbt release notes for recent and historical changes. Release notes fall into one of the following categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by month for both multi-tenant and virtual private cloud (VPC) environments. ![RSS](/img/fontawesome/rss.svg)Subscribe to release note updates via [RSS](https://docs.getdbt.com/feeds/release-notes-rss.xml), [Atom](https://docs.getdbt.com/feeds/release-notes-atom.xml), or [JSON Feed](https://docs.getdbt.com/feeds/release-notes-rss.json). For dbt v2 updates, refer to the [v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md). #### October 2026 * **New:** dbt Wizard Desktop is now in public beta. [Download the app](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md#install-and-open-wizard-desktop) for macOS, Windows, or Linux and start your first session. * **New:** If your GitHub Enterprise Server can only be reached over a private connection (PrivateLink), you can now save that connection on the GitHub app. Set `private_link_endpoint_id` when you create or update the app through the API, and dbt routes the connection over your private network. * **New:** **Filter the Model timing table:** On the Model timing tab, you can now filter [Resource details](https://docs.getdbt.com/docs/deploy/run-visibility.md#resource-details) to find what's slowing a run down. Use the **Est. critical path** filter to show only the resources on the longest chain of dependent resources, which sets the minimum run time (Yes), or everything else (No). Use the resource type filter to narrow the list to **Model**, **Test**, **Snapshot**, or **Exposure**. You can still search by name. * **New:** Set [`allow_jinja_file_extensions`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) in your project config so dbt reads files with an extra Jinja ending, such as `model.sql.j2` or `doc.md.j2`. Jinja is the templating language dbt already uses inside SQL and Markdown files. * **New:** In the dbt Wizard [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), the conversation you have open appears in the browser URL. You can bookmark that conversation and open it again from the URL. * **Enhancement:** If a project can use dbt v2 but you do not have permission to select it, the dbt v2 choices stay visible and disabled in the environment, job, and development version menus. The tooltip says that an Account Admin or a [v2 Migration Admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin) has to select dbt v2. * **Enhancement:** If a [usage chart](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#viewing-usage-in-the-product) fails to load or refresh, you see a message and a **Retry** button. This applies to the monthly models built chart on **Project home** and the top jobs chart on an **Environment** page. A chart that loaded earlier keeps its last numbers and shows when they were updated while the retry runs. * **Enhancement:** In Studio [User development settings](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#override-dbt-version), the **v2 Stable** option no longer shows a **Preview** label. dbt v2 is generally available. * **Enhancement:** In the dbt VS Code extension, drag the edge of a column header in the query results panel to make that column wider or narrower. * **Enhancement:** **Longer email verification window**: Email verification short codes now expire after 10 minutes instead of 5, giving you more time to complete account verification. * **Fix:** If a signup uses an email address that already has an account, including when two signups use that email at the same time, dbt shows a page that says the email is already taken. You no longer get a generic error page. * **Fix:** A request with a bad value now returns a clear error instead of a server error. An unsupported `order_by` value on version 3 list endpoints, and a negative or too-large `limit` or `offset` on version 2 list endpoints, return a 400 error (the request was invalid). Creating a project with `POST /api/v2/accounts/{id}/projects/` also returns a 400 error when `type` is not text. Getting run timing with `GET /api/v3/accounts/:id/projects/:id/runs/:id/timing/` returns a 404 (not found) when that run is not in the account or project in the URL. * **Fix:** In Studio IDE, an [ephemeral](https://docs.getdbt.com/docs/build/materializations.md#ephemeral) model is a model that is not built as its own table or view. Those models were missing from the command result list, so a tab such as **Error**, **Warn**, **Skip**, **Reused**, or **Pass** could show a count and no rows. They now appear on the matching tab with the other models. * **Fix:** Open a job run, select a step, and open **System logs**. That tab now includes the logs from your models, so you can troubleshoot a failed run in one place. * **Fix:** [`dbt fmt`](https://docs.getdbt.com/reference/commands/lint.md#dbt-format) no longer shifts a Jinja tag to the left when that same line ends with a SQL comment. It also settles on one indentation for Jinja tags inside a `for` loop, and for an `{% if %}...{%- else -%}` block whose condition looks up warehouse information, such as the columns on a table. Running `dbt fmt` again does not keep changing those lines. * **Behavior change:** The **System logs** tab on a run step used to show only system messages. It now shows every log for that step, including logs from your models. If you used this tab for system messages only, you will see more lines there. * **Behavior change:** dbt v2 renamed the variable that selects an AI provider, and the two variables that write the [information schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md#generating-the-information-schema) (a local copy of project metadata you can query). If you set `DBT_AI_PROVIDER`, change it to [`DBT_ENGINE_AI_PROVIDER`](https://docs.getdbt.com/docs/dbt-ai/package-skills.md#set-the-ai_provider-flag). If you set `DBT_GENERATE_INFO_SCHEMA` or `DBT_INFO_SCHEMA_DIR`, change them to `DBT_ENGINE_GENERATE_INFO_SCHEMA` and `DBT_ENGINE_INFO_SCHEMA_DIR`. The command-line flags for these options are unchanged. Update the variables anywhere you set them, including continuous integration (CI). * **New:** You can now [route cross-project `ref()`](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#route-environments-across-projects) to a specific upstream environment, such as Staging, instead of only the default Production or Staging environment. * **New:** In dbt Wizard Desktop, you can search the current chat and jump between matches, including text in code blocks. * **New:** In dbt Wizard Desktop, use `/new`, `/clear`, and `/rename` to manage conversations. `/new` accepts an optional starting prompt and works while another response is running. * **New:** In dbt Wizard Desktop, you can discard all uncommitted changes and remove new files in one confirmed action. Conflicted files are left alone. * **New:** In dbt Wizard Desktop, you can opt into automatic background updates, choose the Stable or Unstable channel in **General** settings, and relaunch when an update is ready. Refer to [Update the app](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md#update-the-app). * **Enhancement:** Question cards in dbt Wizard Desktop now let you answer multiple questions, write your own response, or skip a question. You can also reply in the composer while questions are waiting, with masked inputs for sensitive answers. * **Enhancement:** In dbt Wizard Desktop, **Changes** and the target branch now follow the base branch of your open pull request. * **Enhancement:** In dbt Wizard Desktop, expand **Show error details** to see why a response failed, including provider errors and spending limits. * **Fix:** dbt Wizard Desktop now uses browser sign-in by default for OpenAI subscriptions, including for workspaces that disable device-code login. Expired or revoked credentials no longer block startup, and you get clearer reconnect guidance. * **Fix:** dbt Wizard Desktop keeps model, lineage, and metric lookups current after you edit files or run dbt outside dbt Wizard. * **Fix:** dbt Wizard validation now skips runs when no changed dbt models are in scope and explains why. * **Fix:** dbt Wizard Desktop now shows clearer ownership and trust prompts for shared repositories. You can also start chats and terminals in trusted repositories owned by another user, including on Windows subsystems for Linux (WSL). * **Fix:** Subagent approval requests in dbt Wizard Desktop now appear in the chat that owns them and name the requesting agent. * **New:** Billing admins now receive proactive email notifications about dbt State usage, including a warning at 80% of the monthly allowance (with a projected run-out date), a notification when the allowance is fully used and State is paused, and a reminder 7 days before a trial expires. All emails include plan-specific messaging and calls to action. * **New:** Account admins now receive an email when Cost Insights ingestion has been failing for 3 or more cycles, with follow-up emails at 14 and 28 days if it continues. * **Enhancement:** The dbt State usage graph now includes a dropdown to select the reporting period: this month, previous month, last 30 days, last 60 days, or last 90 days. The chart x-axis also shows month abbreviations when the selected range spans multiple months. * **Enhancement:** dbt Wizard now displays how long each turn took at the bottom of every completed turn. For previously loaded conversations, the completion timestamp is also shown. * **Fix:** If you manage access through SCIM (your identity provider, like Okta or Microsoft Entra ID), you can change the user's license type before they even log in. Previously this only worked for active users, but now it works for pending and not-yet-fully-signed-in users too. dbt still checks that you have an available seat of that license type before applying any change. * **Fix:** API requests with an invalid permission value or a malformed body now return a descriptive error instead of a server error. * **New:** dbt State trial and DATT allowance updates: * **New:** Self-hosted dbt users using dbt State now get a free plan after their trial ends, with 1,000 daily active target tables (DATT) per month at no cost. Developer and paid dbt platform accounts also receive a monthly DATT allowance, with amounts and availability varying by plan. For more information, refer to [dbt State usage](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md). * **New:** When your dbt State trial ends, you’ll receive an email confirming your plan, monthly DATT allowance, and allowance end date if your plan has one. * **New:** If your dbt State trial has ended, dbt State is now active again on your account with your plan’s free monthly DATT allowance. No action is required. * **New:** Admins can now set the [default license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#default-license-for-new-users) for new users who aren't assigned a license through SSO, SCIM, or license mapping. Choose Developer, Read-Only, or Analyst. Admins can check their settings and update the default as needed. This is available for all multi-tenant and single-tenant accounts. * **Enhancement:** You can now sort columns, search by connection name, and filter by connection type, credentials, and connection method (private link or OAuth). * **Fix:** Charts now show dates such as `2026-01` or `2026-01-15` from oldest to newest. Tables keep your query's row order until you sort a column. * **Behavior change:** You can no longer create a System for Cross-domain Identity Management (SCIM) token with a personal access token or a service token. Sign in to dbt and [create a SCIM token](https://docs.getdbt.com/docs/platform/manage-access/scim.md#set-up-dbt). * **New:** The jobs list API endpoint (`GET /api/v2/accounts/{account_id}/jobs/`) now accepts a `pk__in` query parameter, letting you retrieve up to 500 specific jobs by ID in a single request. * **New:** You can now specify `clickhouse_v0` as the adapter\_version when creating connections and credentials, filtering connections, and retrieving adapter schemas in the Administrative API v3. Contact your account representative to enable. * **New:** The `metrics` and `metricsPaginated` GraphQL queries now accept a `searchTerms` argument, letting you filter metrics by multiple terms in a single request with correct pagination. Note: `search` and `searchTerms` cannot be combined, and a maximum of 20 terms is allowed. * **New:** In dbt Wizard Desktop, you can attach files and images to a chat. Drag them in, paste them, or copy them from Finder into the composer, then preview them before you send. Sent files show up as readable cards. * **New:** In dbt Wizard Desktop, projects connected to dbt platform now connect to the remote dbt MCP server by default, using OAuth sign-in. This adds platform tools like job management, run status, and run errors. Go to **Settings** → **Connectors** to turn it on or off for each dbt account. * **New:** In dbt Wizard Desktop, you can pin chats and environments to a section at the top of the sidebar and reorder them. * **New:** In dbt Wizard Desktop, you can open SQL from a chat in the **Queries** pane. * **New:** In dbt Wizard Desktop, you can pick the Jaffle Shop sample project during onboarding. * **Enhancement:** Workspace @-mention queries now use server-side fuzzy matching instead of filtering a local list, returning better-ranked results for large repositories. The context file picker also prevents selecting stale results while a search is still in flight. * **Enhancement:** You can now add a `table` block type to `compose_report` reports to render query results as formatted, scrollable tables. Columns can be narrowed and reordered using the `columns` field, and the display caps at 50 rows on screen with a truncation notice for clipped results. * **Enhancement:** You can use the new transformation type badge on column cards to see how each column is derived. * **Enhancement:** The webhook subscription job picker now searches and paginates server-side. You see a search box with debounced filtering and scroll-to-load pagination instead of a slow or frozen dropdown when your account has many jobs. * **Enhancement:** The dbt State navigation option no longer carries a "Preview" badge. Incomplete projects (those without a repository or skipped setup) are also excluded from automatic project selection on the dbt State home page. * **Enhancement:** The GET, POST, and PATCH endpoints for Microsoft Entra applications now return `client_id` and `tenant_id` in their responses, giving you visibility into your Azure DevOps (ADO) integration configuration. * **Enhancement:** dbt Wizard Desktop now installs and manages `uv` itself when a project's Python requirement blocks setup. * **Enhancement:** dbt Wizard Desktop chats in the sidebar now show PR status, unread, and running indicators. * **Enhancement:** dbt Wizard follows updated dbt best practices, including better use of the Semantic Layer. * **Behavior change:** Pull requests that dbt Wizard creates now say in their description that dbt Wizard made them. * **Fix:** dbt Wizard now detects when deferral isn't working because the upstream model doesn't exist in production, and recommends turning off deferral for the session. * **Fix:** dbt Wizard Desktop now handles dbt platform sign-in errors better, fixes issues with the **Changes** and **Summary** panes (including generating a new summary when non-agent changes are detected), and fixes edge cases when creating environments with a missing adapter. * **Fix:** When the Wizard receives a malformed diff patch, it now displays a "Couldn't parse diff" error badge instead of crashing. * **Fix:** Charts with decimal data values (for example, 25.43 minutes) no longer push the y-axis name outside the visible canvas area. The axis gap is now calculated from rounded tick values. * **Fix:** The x-axis name, grid bottom margin, and legend centering on horizontal bar charts are now computed correctly so the axis label no longer overlaps the legend. * **Fix:** The Cost Insights dashboard no longer shows a perpetual loading spinner when partial cost data exists. You now see available cost data even when some rows have not been fully processed. #### September 2026 * **Enhancement:** If a CI job can't find the commit for its deferral baseline, [SQL linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) no longer fails the run. Instead, dbt shows a warning with the commit SHA it couldn't resolve and lints the entire project. To refresh the baseline, rerun the job you defer to. * **Enhancement:** You can now enable or disable [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) for each job using the **dbt State** dropdown: **On**, **Off**, or **Inherited from environment**. New jobs default to **Inherited from environment**. Make sure to manually update existing jobs. Refer to [Enabling dbt State on environments and jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md) for more information. * **Private beta:** Snowflake users can now configure [custom relation overrides](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#custom-relation-overrides) when setting up Cost Insights. If your credentials don't have access to the `SNOWFLAKE` system database, you can use your own tables or views in place of the default Snowflake system tables. Contact your account representative for access. * **Enhancement:** The [column lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md#column-lens) panel now shows SQL transformation expressions for all users. You can see how columns are derived across your lineage without any additional configuration. * **Enhancement:** The "last run finished at" column in the account Insights table now displays times in UTC, giving you a consistent reference regardless of your local timezone. * **Enhancement:** Per-query Snowflake Adaptive warehouse costs are now tracked automatically for all accounts in [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) when `QUERY_METERING_HISTORY` is accessible, with no additional configuration required. * **Enhancement:** Job settings now always show a dbt State mode dropdown with inherit, on, or off. This replaces the previous per-job checkbox. You can also set dbt State as the default for a deployment environment. * **Enhancement:** AI reasoning blocks in agent chat now render in the correct order relative to answer text, both during live streaming and after you reload the conversation. * **Enhancement:** BigQuery authentication failures now surface as distinct 401 errors, and access-denied failures surface as 403 permission errors, instead of generic data platform errors. You get a more specific message to help diagnose credential and permission issues. * **Enhancement:** When a Redshift connection fails because the credentials are wrong, or a role does not have the required permissions, you now see a specific error message instead of a generic failure. * **Enhancement:** The external metadata ingestion checkbox in connection settings is now hidden unless it is already enabled for that connection. The toggle has also been removed from the account-level settings page. * **Fix:** Models with a "Warned" status now appear correctly in the [Execution timeline](https://docs.getdbt.com/docs/deploy/run-visibility.md#execution-timeline) Gantt chart. The [critical path, peak concurrency, average active, and longest model](https://docs.getdbt.com/docs/deploy/run-visibility.md#metric-tiles) summary tiles now include warned models in their calculations. * **Private beta:** Enterprise and Enterprise+ accounts can now [bring their own GitHub application](https://docs.getdbt.com/docs/platform/git/connect-github.md#custom-github-application) to natively connect GitHub Enterprise Server (self-hosted) and GitHub Enterprise Cloud with data residency (hosted on `ghe.com`), instead of using a git URL and SSH. Reach out to your account manager to join the private beta. * **New:** [DuckDB support for Apache Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/duckdb-iceberg-support.md) is now generally available for local use with dbt v2. * **New:** The models table in Catalog navigation now shows a column count for each model. * **Enhancement:** Studio IDE now detects missing development credentials at startup and shows a **Development Credentials Required** modal with a direct link to add credentials, instead of failing silently. * **Enhancement:** The **Upstream Sources** table on the model detail pages now sorts by freshness severity by default (Error first, Pass last). Click **Name** or **Status** to change the sort order. The **Status** column also no longer overflows on wide screens. * **Enhancement:** The model timing Gantt chart no longer limits groups to 2,000 rows. You can now see all models in the timing view for large runs. * **Enhancement:** Read-only users can now see their assigned analytics connection in project settings instead of **Not configured**. * **Enhancement:** Account admins now receive email when dbt Wizard usage credits are exhausted, with separate notices for accounts that have no usage commitment and accounts that have used their full commitment. * **Enhancement:** When a GitLab host is unreachable, affected API endpoints now return HTTP 503 with the message "GitLab is unavailable, please try again." instead of an unclear failure. This applies when you create a repository or list GitLab groups. * **Fix:** Pinned models in the dbt Wizard context card now display the model name (for example, `customers`) instead of the raw metadata unique ID (for example, `model.jaffle_shop.customers`). * **Fix:** Cross-account write attempts now return HTTP 403 Forbidden instead of a generic 500 Internal Server Error, so you get a clear rejection when the account doesn't match. * **Fix:** The error for a missing Azure DevOps account link now shows the correct message: "Missing Azure user; link your Azure DevOps account in your personal profile." * **Behavior change:** Creating accounts via `POST /api/v2/accounts/` is now blocked for service tokens, account-scoped personal access tokens (PATs), and OAuth access tokens. Service tokens and account-scoped user API tokens receive HTTP 400; OAuth access tokens receive HTTP 403, with explicit error messages in each case. * **Behavior change:** Cost Insights aggregates now include unit tests when you filter by the Test resource type. Previously, unit tests were silently excluded, causing lower-than-expected execution counts. ##### dbt Summit 2026 announcements The following product name changes and features are part of dbt Labs announcements at [dbt Summit 2026](https://www.getdbt.com/dbt-summit) in Las Vegas, September 15-18, 2026. ###### Product naming updates We've updated some of our product names and we're excited to unveil them at dbt Summit! Our docs have been updated to use this terminology: | Previously | Now | What to know | | --------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | — | `dbt` | The industry standard for data transformation, available in self-host and cloud-hosted configurations | | Fusion, dbt Core v2.x | `dbt v2` (recommended), `dbt OSS` (open source, Apache 2.0 only) | Check out [our blog](https://docs.getdbt.com/blog/comparing-dbt-and-dbt-oss) for more information | | dbt Core v1.x | `dbt v1` | Remains open source | Two other things moved: * The GitHub repo for dbt (v1 and v2) is now at [dbt-labs/dbt](https://github.com/dbt-labs/dbt). * The [`pip install`](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) instructions: * dbt v2: `pip install dbt` (previously installed the dbt platform CLI). * dbt v1: `pip install dbt-core` remains for now, but over time will be migrated to `pip install dbt-oss`. ###### Now generally available * **[dbt v2.0](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md)** is the next major version of dbt, running on a single engine written in Rust. Install it with `pip install dbt` and run it wherever you deploy dbt: on your laptop, in your virtual private cloud, or on the dbt platform. Read more in [dbt v2.0 is GA](https://docs.getdbt.com/blog/dbt-v2-is-ga). * **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)** skips or clones nodes when the logic and data haven't changed, rather than rebuilding everything on every run. It works locally (dbt v1.7 through v2.0), on the dbt platform, and with external orchestrators. To get started, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). * **Adapters for dbt v2.0** — [Snowflake](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md?version=2), [BigQuery](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md?version=2), [Redshift](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md?version=2), and [Databricks](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md?version=2) are available for local and platform use. [DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md?version=2) is available for local use. Refer to [Adapter lifecycle](https://docs.getdbt.com/docs/supported-data-platforms.md?version=2.0#adapter-lifecycle) and [About connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md?version=2.0) for current status and platform connection support. ###### In preview and beta * **Private beta: [Lake Compute](https://docs.getdbt.com/docs/lake-compute.md)** is a dedicated engine, built on DuckDB, that runs dbt models directly against your Apache Iceberg™ tables, priced for transformation and nothing else. * **Private beta: [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md)** runs the same Wizard engine in a native app with rendered tables and charts, inline diffs, and parallel chats. [Sign up to get the download link](https://www.getdbt.com/wizard-desktop-waitlist). * **Private beta: [ClickHouse](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-clickhouse.md?version=2.0)** connections for dbt v2.0 on the dbt platform. Connect ClickHouse Cloud on a v2 release track using username and password over HTTPS. ClickHouse is also in beta for local use — refer to [Connect ClickHouse](https://docs.getdbt.com/docs/local/connect-data-platform/clickhouse-setup.md?version=2). * **Beta: [Apache Spark](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md?version=2)** adapter for dbt v2.0, for local use (CLI only). * **Public beta: [dbt Charts](https://docs.dbtcharts.com/)** turns YAML into interactive dashboards that query your dbt models directly, so boards stay in sync with the transformations your team already trusts. Develop locally with the `dct` CLI. ##### Pre-dbt Summit * **Beta:** When [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is enabled in a self-managed deployment and your project is connected to the dbt platform, `state:*` selectors use dbt State as their comparison source, unless a state manifest is explicitly provided. Each node (models, snapshots, seeds, and tests) is compared against its own last execution in the deferral target environment rather than a single `manifest.json` from the most recent job run. Refer to [dbt State-powered `state:*` selectors](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md#dbt-state-powered-state-selectors) for more information. * **New:** The **dbt State** page now includes a [**Lag tolerance recommendations**](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#lag-tolerance-recommendations) section that identifies models that could safely tolerate more lag. For each model, it shows the current lag tolerance, the recommended value, the estimated percentage of build time you'd save, and projected build time savings over the next 30 days. You can search by model name or filter by project, and apply recommendations by updating the [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) config. * **Beta:** The Snowflake adapter now supports the `interactive_table` materialization in beta on dbt v2 and `dbt-snowflake` v1.12+, covering both static and dynamic (auto-refreshing) interactive tables. For more information, see [Interactive tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md). * **Enhancement:** When you connect to Snowflake through the Semantic Layer, authentication failures and permission errors now return distinct messages prefixed with `[WAREHOUSE_AUTHENTICATION_FAILED]` or `[WAREHOUSE_PERMISSION_DENIED]`, so you can tell credential issues apart from missing grants. * **Fix:** Saving a job with an invalid day-of-month value such as `*,L` now shows a validation error and prevents the broken schedule from being saved. Use either `*` or `L` in the day-of-month field, not both. * **Behavior change:** The **Fusion admin** permission set is now called [**v2 Migration Admin**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin). The permissions it grants haven't changed, and existing assignments carry over. * **Behavior change:** When you assign a project-scoped permission set to a group or service token, you must now specify either all projects or a specific project. Requests that leave project scope unset return a `400` error. Account-level permission sets such as Billing Admin and Notification Manager are not affected. Existing legacy assignments continue to work until you change them. * **Preview:** Explore mode is now available in the dbt Wizard [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md#ask-questions-in-explore-mode) and [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#ask-questions-in-explore-mode). Explore mode lets users ask questions of governed production data in plain language, with the SQL or metric definition behind every answer. Read-only users can now be [invited](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) to ask questions about your data without a developer license in the Wizard home tab. * **New:** Jobs with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) enabled now include an **Explain** tab on the run details page. This tab shows why dbt State rebuilt, reused, or cloned each resource in the run, so you can audit State behavior and debug unexpected decisions directly on the dbt platform. Refer to [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) for more information. * **Behavior change:** The following behavior change flags on the dbt platform **Latest** release track have reached maturity, switching their default values from `false` to `true`. To preserve the previous behavior for any of these flags, set them to `false` in your `dbt_project.yml`. * [`skip_nodes_if_on_run_start_fails`](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md) * [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md) * [`require_yaml_configuration_for_mf_time_spines`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_yaml_configuration_for_mf_time_spines.md) * [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) * [`require_nested_cumulative_type_params`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md) * [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md) * [`require_all_warnings_handled_by_warn_error`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_all_warnings_handled_by_warn_error.md) * **Behavior change:** AI features in the dbt platform are being enabled by default. They're already on for new accounts and are rolling out soon to existing accounts. Organizations that opted out stay off, and admins can turn AI features on or off anytime in **Account settings**. Refer to [AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * **New:** dbt Wizard now uses usage-based billing for dbt-managed AI. Usage is metered per token against your consumption pool, with monthly included usage or a trial to start and admin-set spend limits. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) and the [Billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). * **New:** [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) onboarding and billing updates: * **New:** New users can now sign up for a free dbt platform account and start a 30-day dbt State trial through a guided onboarding flow. For details, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md?user-type=self-hosted). * **New:** dbt v1.7-1.12 users who are using the dbt State plugin can now create a free dbt platform account to start a 30-day dbt State trial. For details, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md?user-type=self-hosted\&version=1.12). * **New:** dbt State billing is now available for self-serve accounts. You can set up usage-based billing in the **Usage-based features** dashboard. Note that you're billed per target table that dbt State reuses each day, not per dbt platform seat. Refer to [Setting up usage-based billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md#setting-up-usage-based-billing) for more information. * **Behavior change**: The standalone dbt State app (`app.state.dbt.com`) is being retired and is no longer accepting new users. Existing dbt State app users should [create a free dbt platform account](http://us1.dbt.com/register?_dbtsrc=dbt-state) to continue using dbt State — your free trial will be extended upon account creation. #### August 2026 * **New:** A new **Explain** tab on the run details page shows dbt State's decision for each model in a run (rebuilt, reused, or cloned), with expandable details, search, and Comma-Separated Values (CSV) download. You can use this tab to investigate why each model was rebuilt or reused. Contact your account manager to enable. * **Enhancement:** Non-admin users now see an "Ask an admin to enable" message on the dbt Wizard and dbt State cards in Billing & Usage when a trial is available but they lack permission to start it, instead of a blank space. The same message appears if a non-admin tries to start a trial from a dbt State or dbt Wizard link. * **Enhancement:** dbt State is now available for jobs running on the Compatible, dbt v2 Extended, and dbt v2 Fallback release tracks, in addition to previously supported tracks. * **Enhancement:** When compare results are larger than 50 MB, pull request comments now show a "too large to summarize" notice with a link to the full compare report, instead of failing with no message. * **Enhancement:** The Studio IDE browser tab now displays "dbt Studio" instead of a generic editor title. * **Enhancement:** When you connect an MCP client with OAuth, more clients can now complete sign-in successfully. * **Enhancement:** [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) now reuses views that only use `select *` on CTEs. Previously, any `select *` anywhere in a view caused a rebuild. Views that use `select *` directly on a `ref()` or `source()` still force a rebuild, because dbt cannot safely determine the output columns at parse time. For more information, refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select). * **Enhancement:** dbt State now fetches table metadata in the background at the start of each run, so execution doesn't stall. Any node that is ready to skip or clone proceeds immediately, without waiting for the fetch to complete. Previously, dbt waited for the entire metadata fetch to complete before any node could execute. For more details, refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). * **Enhancement:** On Snowflake, when [`metadata_warehouse`](https://docs.getdbt.com/reference/resource-configs/metadata-warehouse.md) is configured, dbt State now issues multiple, individual queries (one per schema) in parallel on the dedicated warehouse — faster than the single, consolidated query dbt runs by default. Without a dedicated warehouse, dbt now emits a warning if the metadata fetch takes longer than 15 seconds. * **New:** The [`allow_clones`](https://docs.getdbt.com/reference/resource-configs/allow-clones.md) profile-level setting lets you control whether dbt State can clone tables into a target environment. Previously, there was no way to disable cloning — dbt State always cloned into any environment when a matching table was found. * **New**: [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md) is a new dbt State config that checks the Jinja template for unrendered code changes. If dbt detects unrendered code changes, it then compares the rendered SQL. A rebuild only occurs when *both* have changed. This prevents unnecessary rebuilds for nodes that use non-deterministic macros or environment variables. * **New:** When dbt State is enabled, you can run `dbt state explain` (dbt v2) or `dbt-state explain` (dbt v1 plugin) in the CLI after a job finishes to see why dbt State made each decision and whether each node was built, reused, or cloned. For a detailed breakdown, run the command with `--verbose -s my_node_name` to see the table analysis, query analysis, and data freshness analysis for a specific node. For more information, refer to [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md). * **Enhancement:** New sessions open on the Wizard tab when available, and the Studio IDE remembers your last-used tab for each project so you can pick up where you left off. * **Enhancement:** A new `relationName` field on the `ModelAppliedStateNode` and `ModelAppliedStateNestedNode` GraphQL types exposes the fully-qualified, adapter-rendered relation name (for example, `"database"."schema"."model_name"`) from the last successful model build. * **Beta**: [dbt Core 2.0](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) is now available in beta! * **New:** The [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set is now generally available (GA) for Enterprise plans. Analyst read is a project-level permission set that provides read-only access to analyze dbt models and project resources, and read-only users can connect to analysis features such as the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md). * **Enhancement:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) now supports cost attribution for [Snowflake Adaptive Warehouses](https://docs.snowflake.com/en/user-guide/warehouses-adaptive). For setup details, refer to [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * **New:** The [Model timing tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) in job run details has been redesigned with a richer, scalable view that includes metric tiles, an execution timeline with grouping and highlight controls, a concurrency-over-time chart, and a searchable resource details table. * **New:** Semantic Layer development connections to Redshift now support external OAuth using Okta or Microsoft Entra with AWS IAM Identity Center. * **Enhancement:** System for Cross-domain Identity Management (SCIM) API errors for seat or licensing failures now include email addresses so you can identify which users are blocking provisioning. * **Behavior change:** Semantic Layer GraphQL queries that exceed the complexity limit of 200,000 now return an error instead of completing with a warning. If you hit this error, request fewer fields, use pagination, narrow your filters, or split the query into smaller ones. * **Enhancement:** The [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set now includes read access to **Connections** (account and project), **Projects**, Git repository settings, Semantic Layer configuration, **Environments**, custom environment variables, and Catalog metadata. Analysts can view that configuration without assigning additional permission sets, once added to a group with Analyst read. #### July 2026 * **Enhancement:** The [dbt State usage page](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) now shows daily active target tables (DATTs) split into **Billable** and **Free**. During a trial, all DATTs are counted as free. * **Preview**: [The dbt Wizard home tab in dbt platform](https://docs.getdbt.com/docs/platform/wizard-home.md) is now available in public preview. You can build and change dbt projects through natural language, with inline diffs, DAG previews, and validation built in. * **New:** The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#discovery) now uses one `get_node_details` tool for all resource types. The older type-specific tools are deprecated and will be removed in a future release. * **New:** When a job is deactivated, the banner now shows a specific reason (repeated run failures, account inactivity, or a generic fallback) with tailored reactivation instructions for each case. * **Enhancement:** The agent now automatically retries transient LLM provider failures (network timeouts, rate limits, and server errors) with exponential backoff, so brief provider blips are less likely to surface as errors during your session. * **Enhancement:** Client tool loops now run until the agent finishes rather than stopping after 50 iterations, eliminating premature termination of long-running agentic workflows. * **Enhancement:** Code blocks in dbt Wizard responses now include a Copy button on hover, so you can reuse generated SQL or YAML more easily. * **Enhancement:** When a Bring-Your-Own-Key (BYOK) OpenAI model is configured with a deployment that does not support embeddings (for example, a `gpt-4o` Azure deployment), the similar models feature now returns an actionable error message prompting you to use a text-embedding model instead of a generic internal error. * **Enhancement:** The `get_lineage` tool now accepts a `direction` parameter (`upstream`, `downstream`, or `both`) to narrow results to only ancestors or only descendants of a target node, reducing response size for large graphs. The response also now includes a `description` field on each returned node. * **Enhancement:** The `list_metrics` tool now accepts a `meta_filter` parameter to restrict results to metrics whose `config.meta` contains specified key-value pairs (for example, `{"agent_accessible": true}`), keeping result sets small enough to preserve description and metadata in the response. * **Enhancement:** The `ModelAppliedFilter` input type now includes a `health` field, letting you filter applied models by health status (`unknown`, `degraded`, `caution`, or `healthy`) directly in the Discovery API. * **Enhancement:** The `RunStatus` enum and the `lastRunStatus` field on model execution information now include `warn`, so models whose last run completed with warnings correctly reflect that status. * **Enhancement:** When searching from within a project environment route (for example, a Staging page), the Catalog search now defaults the environment filter to that environment type rather than always defaulting to Production. * **Enhancement:** A redesigned search result card replaces tooltip-based match pills with inline expandable snippets for columns, tags, descriptions, and code matches. Please contact your account manager to enable. * **Enhancement:** The `latest-fusion` release track is now dbt v2 Stable across all settings. Existing configurations have been updated automatically. No action is needed. * **Enhancement:** On the Enable dbt v2 Environments page, environments already running dbt v2 now show a disabled checkbox, preventing unnecessary saves. * **Enhancement:** When saving a dbt v2 upgrade fails, the platform now displays the top-level user message from the API instead of internal field-level error details. * **Enhancement:** The command panel now shows live status updates as commands run, so you see progress sooner without waiting for a refresh. * **Enhancement:** The "Committed spend" card is now labeled "Consumption pool" with copy explaining that usage-based features like dbt State draw from it. The card now appears between the current plan metric tiles and the product-specific sections on billing Overview and usage tab pages. * **Enhancement:** The Daily Active Target Tables (DATTs) chart now stacks billable and free series, so trial users whose usage is entirely free see real bars instead of an empty chart. * **Enhancement:** Memory-tuning optimizations are now applied automatically to all dbt v2 runs, reducing out-of-memory kill rates and improving overall uptime. * **Fix:** Claude-backed agents can return longer answers and handle some previously broken interactions more reliably. * **New**: [Apache Ossie](https://github.com/apache/ossie) semantic layer support: * Open Semantic Interchange (OSI) has been renamed to Apache Ossie. For more information, refer to [OSI is now Apache Ossie (Incubating)](https://www.getdbt.com/blog/osi-is-now-apache-ossie). * dbt writes an `osi_document.json` file to your `target/` directory alongside `semantic_manifest.json` at parse time. This artifact provides an Ossie representation of your project's Semantic Layer. For more information, refer to [Semantic manifest](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). * dbt supports the Ossie standard for defining semantic models and metrics. You can place Ossie-format `.json` files in an `osi/` directory at the root of your project, and dbt parses them into the manifest alongside any native dbt semantic models. To use a different directory, configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml`. Ossie versions `0.1.0` and `0.1.1` are supported; any other version raises a parse error. For more information, refer to [Ossie semantic layer documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md). * **New:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) is now generally available (GA) for Snowflake, BigQuery, and Databricks. * **Preview:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) for Amazon Redshift is now in preview. * **Enhancement:** Users with `user_credential_write` access can now view and manage their credentials without needing `credentials_read` privileges. This update reduces the need for additional, broader permissions when performing credential updates. * **Enhancement:** The [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) in dbt platform has a redesigned empty state with updated suggested prompts to help you discover different ways to get started. A new wayfinder bar keeps your current project and branch visible and highlights the next step as you move from asking questions to changing code and opening a pull request. * **Enhancement:** Catalog now supports a **Warn** last-run status. Resources whose last run completed with warnings show a distinct status and tooltip, and you can filter by **Warn** alongside other run statuses. * **New:** You can now create hybrid jobs to track runs triggered by an external orchestrator. Hybrid jobs have a simplified setup that omits execution steps, triggers, advanced settings, and cost-optimization controls. They display **Externally triggered** as their next-run schedule and are available only for projects configured as [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md). * **Enhancement:** Runs using a dbt v2 dbt version now invoke the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. dbt v2 virtual environments do not include SQLFluff, so linting now works for all Fusion-version runs and runs faster. * **Enhancement:** When the agent compresses conversation context in the background, a spinner labeled **Optimizing conversation context…** now appears in the chat area. Submitting new messages and stopping the agent are disabled while compaction is in progress to prevent conflicts. * **Enhancement:** When [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is unavailable (not activated, trial expired, or spend limit reached), Studio IDE now shows a dedicated screen with the specific reason and an appropriate action instead of a generic message. * **Enhancement:** The users table, group member lists, and user edit drawer now search, filter, sort, and paginate server-side. On large accounts, all users are findable by name, email, or license type, group member search no longer misses results beyond the first page, and users beyond the first page can be opened and edited in the user edit drawer. * **Enhancement:** The **Enable global account discovery** setting on the **Account settings** page is now visible to all entitled accounts without requiring a feature flag. You can allow or restrict account discovery from [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). * **Enhancement:** Credential-level [connection overrides](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) (such as Databricks catalog, Snowflake warehouse, role, and database) are now surfaced as a read-only **Connection overrides** section in the profile details view, without requiring you to open the edit form. * **Fix:** When a run pod is Out of Memory (OOM)-killed and restarted, the platform now passes the correct status code and message to the config API so the run transitions to a failed state in the dbt platform UI instead of remaining **running** indefinitely. * **Fix:** The Secure Shell (SSH) connection and authentication timeouts for Semantic Layer data platform connections are now 30 seconds (previously 1 second). If your bastion host or network path has higher latency, you will no longer experience deterministic connection failures. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information. * **Fix:** Some types of Compile SQL queries are now rejected if they are too complex. If a request fails with a validation error, try reducing the number of metrics or group-by dimensions in the query. * **Behavior change:** You can no longer create a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) using an account-scoped [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) (PAT). Requests to the service tokens endpoint authenticated with a PAT now return a `400` error. Use a service token to create new ones instead. * **New:** You can now access dbt State settings from **Account settings** > **Billing & Usage**, previously found under **State**. You can manage your trial, enable dbt State on environments and jobs, and set spend alerts — all in one place. For details, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). * **New:** Redshift development connections now support [external OAuth](https://docs.getdbt.com/docs/platform/manage-access/redshift-external-oauth.md) (Okta or Entra ID) through AWS IAM Identity Center. ###### Docs changes To simplify the docs experience, clarify availability, and make it easier to find what applies to you, we made the following changes to the docs site: *tl;dr:* The docs are now organized around v1 and v2 for simplified docs versioning and navigation. We've clarified dbt Core and licensing, reorganized v2 content, and refreshed adapter and Fusion availability guidance. If you notice anything off or have any feedback, we'd love to hear it! Open up a [docs issue here](https://github.com/dbt-labs/docs.getdbt.com/issues). * **Enhancement**: We've updated the version switcher on the docs site. The version switcher now just shows v1 and v2. v2 is the current generation of dbt, built on Rust for a faster, richer dev experience; v1 is the Python-based generation of dbt. Refer to [dbt versions](https://docs.getdbt.com/docs/introduction.md#dbt-versions) for what's different between v1 and v2. * **New:** We've added a dedicated page explaining dbt Core and its distributions. dbt Core 2.0 is the Rust-based open-source runtime. dbt Core v1.x is the Python-based runtime. Refer to [About dbt Core](https://docs.getdbt.com/docs/introduction.md) for more info. * **New:** Licensing across dbt Core now has its own page, so you can see what applies to your setup in one place. Refer to [dbt licensing](https://docs.getdbt.com/docs/dbt-licensing.md). * **Enhancement:** [Static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) now lives with the rest of your build docs and available in v2. * **Enhancement:** The Fusion upgrade readiness checklist now sits right next to the [v2 upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md), and the networking and telemetry references moved in [local install](https://docs.getdbt.com/docs/local/dbt-networking-requirements.md) and [Reference](https://docs.getdbt.com/reference/telemetry-observability.md). * **Enhancement:** More adapters are closer to general availability — Snowflake, BigQuery, Databricks, and Redshift are now in **Preview**, and Spark and DuckDB are in **Beta**. Refer to [Adapter lifecycles](https://docs.getdbt.com/docs/dbt/dbt-availability.md?version=2.0#adapter-lifecycle) for the current status of each adapter. * **Enhancement:** Simplified and clarified the [Fusion feature tables](https://docs.getdbt.com/docs/dbt/dbt-availability.md?version=2.0#what-you-get-with-v2) to make it easier to see what's available and how to get it. * **New:** Added availability badges to pages and sections so you can quickly see what applies to your setup at a glance. #### June 2026 * **New:** The list users API endpoint (`GET /api/v3/accounts/{account_id}/users/`) supports new query parameters to help find users in large accounts without paging through the full list: * `search`: Matches email, first name, last name, or full name (case-insensitive). * `group_id`: Returns only active members of that group. * `license_type`: Filters by license, for example `developer` or `read_only`. You can combine them with `limit` and `offset`, and `total_count` reflects the filtered results. * **Enhancement:** [Column-level tags](https://docs.getdbt.com/reference/resource-configs/tags.md) defined in your dbt project now appear on the **Columns** tab of resource details pages in Catalog. You can click any tag badge to filter the lineage view, or search for columns directly by tag name. Refer to [View resource details](https://docs.getdbt.com/docs/explore/explore-projects.md#view-resource-details). * **Enhancement:** You can now enable [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) on continuous integration and merge job types, in addition to deploy jobs. For more information, refer to [Enabling dbt State on environments and jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md). * **Enhancement**: The [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) table view now includes **All** and **Jobs** buttons to switch between an aggregated cost view and a per-job cost breakdown. Available in the project dashboard and the **Model performance** section in Catalog. When **Jobs** is selected, the CSV export includes job-level data. For more information, refer to [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md). * **Enhancement:** [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) tool calls for dbt command invocations now stream their output live in chat, in both the Studio IDE and [Wizard home](https://docs.getdbt.com/docs/platform/wizard-home.md). * **Enhancement:** You can now download files from the Studio IDE File explorer. Right-click a file and select **Download** to save it to your computer. For more information, refer to the [Studio IDE user interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#basic-layout). * **Fix:** If you use the Administrator API to manage [SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md) to sync users from your identity provider, the `/api/v3/accounts/{account_id}/scim/v2/Users` response now returns `value` and `display` on each embedded group reference. `id` and `displayName` are retained so existing integrations keep working — this is a non-breaking change. * **Enhancement**: The [Administrative API v3](https://docs.getdbt.com/dbt-cloud/api-v3) now supports private endpoint operations — [`list`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/List%20Private%20Endpoints), [`create`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Create%20Private%20Endpoint), [`retrieve`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Retrieve%20Private%20Endpoint), [`update`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Update%20Private%20Endpoint), and [`delete`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Delete%20Private%20Endpoint). Use these endpoints to manage private connectivity programmatically. * **Enhancement**: You can [download OpenTelemetry (OTel) logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs) for individual dbt command steps in Fusion job runs. * **Enhancement**: You can now configure [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) for the Studio IDE directly in the dbt platform UI — either as a team-wide default on your development environment, or as a personal override. For more information, refer to [Enabling dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md). * **New:** [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) for Redshift and Databricks is now generally available (GA). * **Behavior change:** On September 1, 2026, several behavior change flags on the dbt platform **Latest** release track will reach maturity (enabled by default). Refer to [About behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) to see which flags may affect your project and how to opt out before then. * **Beta:** dbt v2 now supports the Salesforce Data 360 connection in the dbt platform. For more information, refer to [Connect Salesforce Data 360](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-salesforce.md). * **Private beta**: The [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set is available for Enterprise plans. * Analyst read is a project-level permission set that provides read-only access to analyze dbt models and project resources. The OAuth integration that lets read-only users connect to analysis features (such as the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md)) is available to use, while the Analyst read permission set and read-only permission changes are in private beta. To enable them, contact your account manager. * **Beta**: Workspace-level Private Link for Microsoft Fabric is now available in beta. Configure a private connection between the dbt platform and your Fabric workspace so SQL traffic stays on Azure's private network. For more information, refer to [Configuring Private Link for Microsoft Fabric](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-fabric.md). * **Beta**: [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) now supports Amazon Redshift Serverless and provisioned clusters. Configure your platform metadata credentials with the `sys:monitor` role or `SYSLOG ACCESS UNRESTRICTED` permission to allow dbt to read cross-user query history, then set your pricing in Cost Insights settings. For more information, refer to [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md). ##### Snowflake Summit 2026 announcements The following features are new or enhanced as part of dbt Labs announcements at [Snowflake Summit 2026](https://www.getdbt.com/events/snowflake-summit-2026) in San Francisco from June 1–4, 2026: * **Alpha**: [dbt Core 2.0](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) is now available in alpha! * **New**: dbt Core 2.0 is the open-source Apache 2.0 foundation that dbt v2 builds on, delivering a faster, Rust-based runtime. It ships as two distributions: `dbt-oss` (OSS, Apache 2.0) and `dbt` (dbt v2 distribution, proprietary). * **Beta**: [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) is now available in beta! * **New**: `dbt lint` is a high-performance SQL linter built into the dbt platform, available on projects running dbt v2. It is SQLFluff-compatible; it reads your existing `.sqlfluff` config, uses the same rule codes, and respects `-- noqa` suppression comments. In benchmarks, it runs roughly 50× faster than single-threaded SQLFluff.. * **Preview**: [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2) is now available in preview! * **New**: dbt Docs v2 is a next-generation open-source catalog experience available with dbt v2 and dbt Core 2.0. It uses a compact binary index instead of loading the full `manifest.json` in the browser, making it significantly faster for large projects. * **New**: dbt Docs v2 includes a redesigned UI, Semantic Layer metadata, column-level lineage (dbt v2 only), and a REST API at `/api/v1/` so AI agents and MCP servers can query your dbt project metadata without a browser. * **New**: Generate and serve [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2) with dbt v2 or dbt Core 2.0 by running a dbt command with `--use-index`, then `dbt docs serve`. Add [`--write-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md#--write-catalog-flag) for richer column type metadata. * **Preview**: [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is now available in preview! * **New**: dbt State skips or clones nodes when the logic and data haven't changed, rather than rebuilding everything on every run. Available natively in dbt v2.0, the dbt platform, and dbt v2, and as a plugin for dbt v1.7-1.12. To get started, refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). * **New**: [dbt State pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md) is usage-based at $0.094 per daily unique reuse. New organizations receive a 30-day free trial with no usage limit. * **Behavior change**: State-aware orchestration is no longer being enabled for new customers. Refer to [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) for more information. * **New**: dbt Wizard is available in dbt platform as a public preview. Introducing dbt Wizard CLI as a public beta. Purpose-built for agentic governed data development in dbt, dbt Wizard understands your project through the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md), unlike general-purpose coding agents. * **New**: [Support for Anthropic as a BYOK provider for dbt AI](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider). * **New**: [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) is a new CLI command available in dbt Core 2.0 and later. It opens browser-based authentication and shares your login state across the CLI, dbt VS Code extension, dbt State, and dbt Wizard CLI with no separate sign-in flows needed. * **New:** OAuth client registrations now accept custom-scheme redirect URIs (for example, `cursor://` or `vscode://`), so you can build native app OAuth integrations with Cursor and VS Code. * **New:** Public REST API endpoints at `/api/ide/v3/{environment_id}/files/` support Studio IDE workspace file operations, including stat, read, write, list, delete, mkdir, and rename. Pass file paths as query parameters. * **New:** The `GET /api/ide/v3/{environment_id}/status` endpoint returns the `dbt_version` and `is_fusion` status for a given environment. * **New:** The dbt platform CLI Python client's `create_invocation()` method now supports a `workspace` parameter, so you can run invocations against persisted workspace files on workers. #### May 2026 * **Enhancement:** Repository clone failures now surface a more actionable diagnostic message to help you resolve common issues faster. For guidance, refer to [Troubleshooting clone errors](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#troubleshooting-clone-errors). * **Fix:** The connection test failure message now prompts you to verify your connection details and confirm that your credentials have access to the data warehouse, rather than showing a generic failure message. * **Enhancement:** Users granted `user_credential_write` can access **Your profile** > **Credentials** without `develop_access` (including read-only users). Environment variable overrides and dbt version overrides still require `develop_access`. Refer to [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information. * **New:** The [Job creator permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-creator) is now available for Enterprise accounts. Assign it to users who need to create, edit, and run jobs within assigned projects and environments without access to edit environments or environment variables. * **Enhancement:** The admin API toolset (job management and run operations) is now always available in the dbt [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)and no longer requires a feature flag. You no longer need to contact your account manager to enable these tools. * **Fix:** When a job cannot clone its repository because no remote URL is configured, the error message now explains the most likely causes (an invalid Git remote URL, a Git provider outage, or a deprecated HTTPS connection) and directs you to verify the URL, confirm your provider is operational, and ensure the repository uses SSH with deploy keys before retrying. * **New:** The **Notification Manager** [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) is now available for Enterprise accounts. Assign it to users who need to manage Slack, Microsoft Teams, and email job notifications across all projects without requiring full Account Admin access. * **Beta**: [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md), available in public beta, shows estimated warehouse compute costs and run times for dbt projects and models in dbt platform, highlighting efficiency gains from [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md). Refer to [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) and [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md) to learn more. * **New:** Fusion release tracks are now being rolled out across accounts in phases. Refer to [Fusion release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md?version=2.0#dbt-v2-release-tracks) for more information. * **Enhancement:** Commands run by and the [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)now appear in the Studio IDE **Commands** tab with a icon and **Run by Copilot** tooltip, so you can tell agent-run commands apart from manually run ones. * **Fix:** [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state) now detects changes to [UDF](https://docs.getdbt.com/docs/build/udfs.md) properties (such as `arguments` and `returns`) defined in `.yml` files. Previously, only changes to the SQL or Python function body were detected. * **New:** [Native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) are now generally available (GA). * **Preview**: The [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) is now in preview. Use natural language prompts to build or refactor models, and generate SQL, tests, documentation, and semantic models from scratch. For more information, refer to the [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). * **Behavior change:** When you set up single sign-on (SSO) in the dbt platform, the SSO slug is now system-generated and read-only. Existing SSO configurations remain valid, but you can’t change the slug. If you delete and recreate your SSO configuration, the new configuration uses a new, system-generated slug. Refer to [Single sign-on overview](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for more information. * **Enhancement:** The [dbt VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md?version=2.0) now supports account creation. If you sign in with an existing dbt user that doesn't have an associated dbt platform account, the registration flow prompts you to create one instead of requiring a separate workflow. * **Enhancement:** Delete individual [dbt Wizard chat conversations](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#availability-and-considerations) from the conversation list (three dots → **Delete**). Deleting the open conversation clears the panel. * **New:** The Fusion + Snowflake connection experience is now generally available on the dbt platform. See our [Fusion upgrade guides](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=1) for information on enabling the upgrade workflows for your environments today! * **Enhancement:** In the Discovery API [Tests object schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-tests.md), you can now filter `environment.applied.tests` by multiple test result statuses in a single query using the new `lastKnownResults: [TestStatus]` filter field on `TestAppliedFilter`. The single-value `lastKnownResult` filter field is still supported but deprecated. Update your queries to use `lastKnownResults` going forward. * **Enhanced** dbt v2 eligibility job prompts now use a **Debug on dbt v2** dropdown instead of a standalone **Run once on dbt v2** button. For more information, refer to [Update your jobs](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=7). * **Enhancement:** The [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)input bar now supports arrow key history navigation. Press the up arrow at the start of the input to cycle through previous inputs, and the down arrow at the end to return to more recent ones. dbt stores up to 5 previous inputs per session. * **Enhancement:** Tool approval and file edit dialogs in the [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)now support number key shortcuts (1, 2, 3) to select options. The first option is auto-focused when a dialog appears, so you can act immediately without clicking. #### April 2026 * **Enhancement:** When a dbt command run by the [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)times out, the agent now automatically attempts to cancel the stuck invocation on the server and returns a retry-friendly message, letting you decide whether to retry. Previously, timeouts resulted in an unhandled error. This applies to both model invocations and autofix runs. * **Enhancement:** In dbt platform run logs, `dbt ls` and `dbt list` now display node results as **No-op** instead of **Unknown** when using dbt v2. Refer to [dbt ls (list)](https://docs.getdbt.com/reference/commands/list.md) for more information. * **New:** A universal login URL is available at <https://login.dbt.com>, making it easier for you to view accounts you have access to across instances (regions and tenancies). This is currently available for multi-tenant accounts with an account-specific domain, and support for single-tenant accounts is coming soon. For more information, refer to [Log in to dbt platform](https://docs.getdbt.com/docs/platform/about-platform/login.md). * **Fix:** Refreshing the same browser tab now restores your active dbt Wizard conversation instead of showing the empty state. Opening a new tab, or returning after closing the tab, still starts in the empty state. The dbt Wizard is currently in beta. * **Enhancement:** The dbt VS Code extension's **Get started** panel has been redesigned and surfaces the exact next setup step you need to install the extension and dbt v2. The new panel also supports a new **agentic migration** option that helps you upgrade your project to dbt v2 automatically in Copilot or Cursor. For more info, see [Getting started](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started). * **Beta**: [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) now also supports Databricks and Redshift. Refer to [Credential permissions](https://docs.getdbt.com/docs/explore/model-query-history.md#credential-permissions) for more information. * **Enhancement:** [Slack notifications (account-level)](https://docs.getdbt.com/docs/deploy/job-notifications.md#slack-notifications-account) and [Microsoft Teams notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md#microsoft-teams-notifications) are now generally available, enabling you to send job notifications directly to Slack channels configured at the account level, and to Teams channels. * **Enhancement:** When using the [dbt autofix](https://github.com/dbt-labs/dbt-autofix) tool in the Studio IDE, you can now compile your project directly from the results panel after a successful `dbt parse`. Click **Compile** next to the **Successfully resolved** result to kick off a compile. For more information, refer to [Fix deprecation warnings](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md). * **Beta**: DuckDB is now supported in the dbt v2 CLI, which lets you run local dbt projects without a warehouse account. For more information, refer to [Connect DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md). * **New**: You can now configure Snowflake PrivateLink endpoints directly in dbt platform without contacting dbt Support, available in private beta. Go to **Account settings → Integrations → Private endpoints** to request and manage Snowflake PrivateLink endpoints on AWS. This feature is available for Snowflake on AWS only. For more information, refer to [AWS PrivateLink for Snowflake](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-snowflake.md?version=1.12). * **Enhancement:** You can now use arrays as values for keys in the dbt platform extended attributes YAML editor. For example, `db_groups: [db_editor, db_viewer]` is now valid. Previously, array values were only supported using the API. For more information, refer to [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes). * **Beta**: The Redshift adapter now supports a `datasharing` profile credential on the dbt platform **Latest** release track. When set to `true`, dbt uses Redshift's native `SHOW` commands (for example, `SHOW TABLES`, `SHOW COLUMNS`, `SHOW SCHEMAS`) for metadata queries instead of PostgreSQL catalog tables, enabling cross-database and cross-cluster access with [Redshift Datasharing](https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html). For more information, refer to [Redshift setup](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md#datasharing). * **Enhancement:** When a connection does not have platform metadata credentials configured yet, the credentials form now renders in edit mode immediately — you no longer need to click **Add credentials** first. If you cancel, the **Add credentials** button appears so you can return to the form. Existing connections with configured platform metadata credentials are unaffected. Refer to [Configure the warehouse connection](https://docs.getdbt.com/docs/explore/external-metadata-ingestion.md#configure-the-warehouse-connection) for more information. * **New**: The [dbt Remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?version=2.0) now supports Admin API calls! This allows users to troubleshoot job-related errors in agents like Claude and Cursor. * **New**: The [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) is now in beta. Use the Developer agent to write or refactor dbt models from natural language, generate documentation, tests, semantic models, and SQL code from scratch, giving you the flexibility to modify or fix generated code. For more information, refer to the [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). * **Enhancement:** The Studio IDE now validates dbt YAML using dbt v2 aligned JSON Schema from [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema) across [dbt platform release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), including for development environments on dbt v1. This improves autocomplete and structural feedback in the editor. Diagnostics can occasionally disagree with what your environment accepts; use dbt runs and previews as the source of truth. For context, review [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) and [dbt YAML validation in Studio](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#dbt-yaml-validation). This will be a phased rollout starting the week of April 6th. * **Enhancement:** The Studio IDE status bar now offers more control, more detailed information, and quicker access to settings for deferral, dbt version, and project status. For more information, refer to the [Studio IDE docs](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#the-command-and-status-bar). These updates roll out in phases to existing accounts starting April 6. * **Enhancement:** In Snowflake **Private endpoints**, output validation errors now display inline beneath the text area (instead of as a page-level banner). The **Submit request** button is also disabled when the output is invalid (for example, empty, malformed JSON, or missing required fields). * **Enhancement:** The Studio IDE now supports deep links to a specific console tab using the `?consoleTab=` query parameter. For example, append `?consoleTab=problems` to open Studio with the **Problems** tab pre-selected. The `problems` tab applies only when it is available for the current session. #### March 2026 * **Enhancement:** The environment [Connection profiles](https://docs.getdbt.com/docs/platform/about-profiles.md#environment-profiles-table) page has been updated. The profile name is now a clickable button that opens the view/edit drawer, the Connection column links to the connection details page in a new tab, and in edit mode a **swap icon** button lets you change the assigned profile. The previous ellipsis menu has been removed. For details, refer to [About profiles](https://docs.getdbt.com/docs/platform/about-profiles.md). * **Beta:** Apache Spark is now supported in the dbt v2 CLI, enabling faster compilation and execution for Spark-based dbt projects. dbt v2 currently supports only Apache Spark 3.0. For more information, refer to [Connect Apache Spark to Fusion](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md). * **Enhancement:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) charts now include an **Assets** filter (**Models** / **Tests** / **All**) on the **Cost**, **Usage**, **Query run time**, and **Builds** tabs. Use the dropdown on each chart to filter the data you want to view; your selection is stored per tab. The former **Model builds** tab is now labeled **Builds**. For more information, refer to [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md). * **Enhancement:** [Deferral](https://docs.getdbt.com/reference/node-selection/defer.md) now supports [user-defined functions (UDFs)](https://docs.getdbt.com/docs/build/udfs.md). When you run a dbt command with `--defer` and `--state`, dbt resolves `function()` calls from the state manifest. This lets you run models that depend on UDFs without first building those UDFs in your current target. * **Fix**: Status messages that exceed the 1024 character limit are now automatically truncated to prevent validation errors and run timeouts. Previously, long status messages could cause runs to fail with unhandled exceptions or result in lost status information. The system now logs when truncation occurs to help identify and optimize verbose status messages. * **Fix:** Resolved an issue where [retrying failed runs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) that were triggered from Git tags would use the wrong commit. Previously, when runs were triggered from Git tags instead of branches, the system would enter a detached HEAD state, causing retries to use the latest commit on HEAD rather than the original tagged commit. The fix now correctly preserves and uses the original Git tag reference when retrying runs, ensuring consistency between the initial run and any retries. * **New**: The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?version=2.0#product-docs) now includes product docs tools (`search_product_docs` and `get_product_doc_pages`) that let your AI assistant search and fetch pages from docs.getdbt.com in real time. Get responses grounded in the latest official dbt documentation rather than relying on training data or web searches, so you can stay in your development flow and trust the answers. This allows you to stay in your development flow and trust. These tools are enabled by default with no additional configuration. Restart your MCP server if you don't see the product docs tools in your MCP config. For more information, refer to [the dbt MCP repo](https://github.com/dbt-labs/dbt-mcp?tab=readme-ov-file#product-docs). * **Enhancement**: The Model Timing tab displays an informative banner for dbt v2 runs instead of the timing chart. The banner explains "Model timing is not yet available for Fusion runs" and provides context about threading differences. Non-Fusion runs continue to show the timing chart normally. * **Behavior change**: [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed. [Assess impact and take any required actions](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#assess-impact-and-required-actions). * **New**: The new Semantic Layer YAML specification is now available on the dbt platform **Latest** release track. For an overview of the changes and steps how to migrate to the latest YAML spec, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). * **Behavior change:** New projects in trial, starter, or Enterprise accounts now default to **Fusion Stable** for all new environments with a supported adapter (Redshift, Snowflake, BigQuery, and Databricks). You can revert to another version by changing the dbt version in your [environment settings](https://docs.getdbt.com/docs/dbt-platform-environments.md#change-environment-settings). #### February 2026 * **New**: Advanced CI (dbt compare in orchestration) is now supported in dbt v2. For more information, review [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md). * **Beta**: The `dbt-salesforce` adapter available in the dbt v2 CLI is now in beta. For more information, refer to [Salesforce Data 360 setup](https://docs.getdbt.com/docs/local/connect-data-platform/salesforce-data-cloud-setup.md). * **Enhancement:** The Analyst permission now has the project-level access to read repositories. Review [Project access for project permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#project-access-for-project-permissions) for more information. * **Enhancement:** After a user accepts an email [invite](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) to access an [SSO-protected](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) dbt platform account, the UI now prompts them to log in with SSO to complete the process. This replaces the previous "Joined successfully" message, helping avoid confusion when users accept an invite but do not complete the SSO login flow. * **New:** [Profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) let you define and manage connections, credentials, and attributes for deployment environments at the project level. dbt automatically creates profiles for existing projects and environments based on the current configurations, so you don't need to take any action. This is being rolled out in phases during the coming weeks. * **New**: [Python UDFs](https://docs.getdbt.com/docs/build/udfs.md) are now supported and available in dbt v2 when using Snowflake or BigQuery. * **Enhancement:** Minor enhancements and UI updates to the Studio IDE, file explorer that replicate the VS Code IDE experience. * **Enhancement:** Profile creation now displays specific validation error messages (such as "Profile keys cannot contain spaces or special characters") instead of generic error text, making it easier to identify and fix configuration issues. * **Private beta**: [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) shows estimated warehouse compute costs and run times for your dbt projects and models, directly in the dbt platform. It highlights cost reductions and efficiency gains from optimizations like [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) across your project dashboard, model pages, and job details. Refer to [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) and [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md) to learn more. * **New**: The [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) now supports [Omni](https://docs.omni.co/integrations/dbt/semantic-layer) as a partner integration. For more information, refer to [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). * **Enhancement**: We clarified documentation for cumulative log size limits on run endpoints, originally introduced in [October 2025](https://docs.getdbt.com/docs/dbt-versions/2025-release-notes.md#october-2025). When logs exceed the cumulative size limit, dbt omits them and displays a banner. No functional changes were made in February 2026. For more information, review [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md#log-size-limits). * **New**: The `immutable_where` configuration is now supported for Snowflake dynamic tables. For more information, refer to [Snowflake configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#immutable-where). * **Fix**: The user invite details now show more information in invite status, giving admins visibility into users who accepted an invite to an SSO-protected account but haven't yet logged in via SSO. Previously, these invites were hidden, making it appear as if the user hadn't been invited. The Invites endpoints of the dbt platform Admin v2 API now include these additional statuses: * `4` (PENDINGEMAIL\_VERIFICATION) * `5` (EMAIL\_VERIFIED\_SSO). * **Enhancement**: Improved performance on Runs endpoint for Admin V2 API and run details in dbt platform when connecting with GCP. #### January 2026 * **Enhancement:** The `defer-env-id` setting for choosing which deployment environment to defer to is [now available](https://docs.getdbt.com/docs/platform/about-defer.md#configure-deferral-environment-id) in the Studio IDE. Previously, this configuration only worked for the dbt platform CLI * **Beta:** The [Analyst agent](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#dbt-copilot) in dbt Insights is now in beta. * dbt dbt Wizard's AI assistant in Insights now uses a dropdown menu to select between **Agent** and **Generate SQL**, replacing the previous tab interface. * **Enhancement:** The [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#search-your-project) now includes search and replace functionality and a command palette, enabling you to quickly find and replace text across your project, navigate files, jump to symbols, and run IDE configuration commands. This feature is being rolled out in phases and will become available to all dbt platform accounts by mid-February. * **Enhancement:** [State-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) improvements: * When a model fails a data test, state-aware orchestration rebuilds it on subsequent runs instead of reusing it from prior state to ensure dbt reevaluates data quality issues. * State-aware orchestration now detects and rebuilds models whose tables are deleted from the warehouse, even when there are no code or data changes. Previously, tables deleted externally were not detected, and therefore not rebuilt, unless code or data had changed. For more information, review [Handling deleted tables](https://docs.getdbt.com/docs/deploy/state-aware-about.md#handling-deleted-tables). State-aware orchestration is in private preview. refer to the [prerequisites for using the feature](https://docs.getdbt.com/docs/deploy/state-aware-setup.md#prerequisites). * **Enhancement:** [dbt dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) correctly detects column names across various `schema.yml` files, adds only missing descriptions, and preserves existing ones. * **Enhancement**: v2 now automatically reads environment variables from a `.env` file in your current working directory (the folder you `cd` into and run dbt commands from in your terminal), if one exists. This provides a simple way to manage credentials and configuration without hardcoding them in your `profiles.yml`. The [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) also supports `.env` files and LSP-powered features. For more information, refer to [Configure environment variables](https://docs.getdbt.com/docs/local/configure-environment-variables.md). * **New**: The new Semantic Layer YAML specification creates an open standard for defining metrics and dimensions that works across multiple platforms. The new spec is now live in dbt v2. Key changes: * Semantic models are now embedded within model YAML entries. This removes the need to manage YAML entries across multiple files. * Measures are now simple metrics. * Frequently used options are now top-level keys, reducing YAML nesting depth. For an overview of the changes and steps how to migrate to the latest YAML spec, check [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). * **Fix:** Debug logs in the **Run summary** tab are now properly truncated to improve performance and user interface responsiveness. Previously, debug logs were not truncated correctly, causing slower page loads. You can access the full debug logs by clicking **Download > Download all debug logs**. For more information, review [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md#run-summary-tab). * **New:** The [Semantic Layer querying](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#semantic-layer-querying) within dbt Insights is now generally available (GA), enabling you to build SQL queries against the Semantic Layer without writing SQL code. * **Enhancement**: Eligible dbt platform accounts in the dbt v2 private preview can now use [Exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md). --- ### dbt Semantic Layer dbt platform | Starter, Enterprise, Enterprise+ The dbt Semantic Layer eliminates duplicate coding by allowing data teams to define metrics on top of existing models and automatically handling data joins. Connect your AI to your metrics AI tools like Claude and ChatGPT can connect to the Semantic Layer through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=sl-connect-ai), so their answers use your governed metrics instead of guessing at raw tables. The dbt Semantic Layer, powered by [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md), simplifies the process of defining and using critical business metrics, like `revenue` in the modeling layer (your dbt project). By centralizing metric definitions, data teams can ensure consistent self-service access to these metrics in downstream data tools and applications. Moving metric definitions out of the BI layer and into the modeling layer allows data teams to feel confident that different business units are working from the same metric definitions, regardless of their tool of choice. If a metric definition changes in dbt, it’s refreshed everywhere it’s invoked and creates consistency across all applications. To ensure secure access control, the Semantic Layer implements robust [access permissions](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer) mechanisms. Refer to the [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) or [Why we need a universal semantic layer](https://www.getdbt.com/blog/universal-semantic-layer/) blog post to learn more. [YouTube video player](https://www.youtube.com/embed/DS7Ub_CmBR0?si=m92hLmxw1VuE6KKO) #### Get started with the dbt Semantic Layer To define and query metrics with the dbt Semantic Layer, you must be on a [dbt Starter or Enterprise-tier](https://www.getdbt.com/pricing/) account. [](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses)Suitable for both Multi-tenant and Single-tenant accounts. Note: Single-tenant accounts should contact their account representative for necessary setup and enablement. <br /> <br /> This page points to various resources available to help you understand, configure, deploy, and integrate the Semantic Layer. The following sections contain links to specific pages that explain each aspect in detail. Use these links to navigate directly to the information you need, whether you're setting up the Semantic Layer for the first time, deploying metrics, or integrating with downstream tools. Refer to the following resources to get started with the Semantic Layer: * [Quickstart with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md): Build and define metrics, set up the Semantic Layer, and query them in Google Sheets and other tools. * [Build your metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) — Use MetricFlow in dbt to centrally define your metrics. * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) — Discover answers to frequently asked questions about the Semantic Layer, such as availability, integrations, and more. #### Configure the dbt Semantic Layer The following resources provide information on how to configure the Semantic Layer: * [Administer the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) — Seamlessly set up the credentials and tokens to start querying the Semantic Layer. * [Architecture](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-architecture.md) — Explore the powerful components that make up the Semantic Layer. #### Deploy metrics This section provides information on how to deploy the Semantic Layer and materialize your metrics: * [Deploy your Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/deploy-sl.md) — Run a dbt job to deploy the Semantic Layer and materialize your metrics. * [Write queries with exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) — Use exports to write commonly used queries directly within your data platform, on a schedule. * [Cache common queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) — Leverage result caching and declarative caching for common queries to speed up performance and reduce query computation. #### Consume metrics and integrate Consume metrics and integrate the Semantic Layer with downstream tools and applications: * [Consume metrics](https://docs.getdbt.com/docs/use-dbt-semantic-layer/consume-metrics.md) — Query and consume metrics in downstream tools and applications using the Semantic Layer. * [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) — Review a wide range of partners you can integrate and query with the Semantic Layer. * [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) — Use the Semantic Layer APIs to query metrics in downstream tools for consistent, reliable data metrics. --- ### dbt Semantic Layer architecture dbt platform The Semantic Layer allows you to define metrics and use various interfaces to query them. The Semantic Layer does the heavy lifting to find where the queried data exists in your data platform and generates the SQL to make the request (including performing joins). [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack. [![The diagram displays how your data flows using the dbt Semantic Layer and the variety of integration tools it supports.](/img/docs/dbt-platform/semantic-layer/sl-architecture.jpg?v=2 "The diagram displays how your data flows using the dbt Semantic Layer and the variety of integration tools it supports.")](#)The diagram displays how your data flows using the dbt Semantic Layer and the variety of integration tools it supports. #### Components The Semantic Layer includes the following components: | Components | Information | local dbt | Developer plans | Starter plans | Enterprise-tier plans | License | | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | --------------- | ------------- | --------------------- | ------------------------------------------------------------------------------------------- | | **[MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md)** | MetricFlow in dbt allows users to centrally define their semantic models and metrics with YAML specifications. | ✅ | ✅ | ✅ | ✅ | [Apache 2.0 license](https://github.com/dbt-labs/metricflow/blob/main/LICENSE) | | **dbt Semantic interfaces** | A configuration spec for defining metrics, dimensions, and how they link to each other. The [dbt-semantic-interfaces](https://github.com/dbt-labs/dbt-semantic-interfaces) is available under Apache 2.0. | ✅ | ✅ | ✅ | ✅ | [Apache 2.0 license](https://github.com/dbt-labs/dbt-semantic-interfaces/blob/main/LICENSE) | | **Service layer** | Coordinates query requests and dispatching the relevant metric query to the target query engine. This is provided through dbt and is available to all users on dbt version 1.6 or later. The service layer includes a Gateway service for executing SQL against the data platform. | ❌ | ❌ | ✅ | ✅ | Proprietary, Cloud (Starter, Enterprise, Enterprise+) | | **[Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md)** | The interfaces allow users to submit metric queries using GraphQL and JDBC APIs. They also serve as the foundation for building first-class integrations with various tools. | ❌ | ❌ | ✅ | ✅ | Proprietary, Cloud (Starter, Enterprise, Enterprise+) | #### Feature comparison The following table compares the features available in dbt and source available in dbt v1: | Feature | MetricFlow Source available | Semantic Layer with dbt | | -------------------------------------------------------------------------------------- | --------------------------- | ----------------------- | | Define metrics and semantic models in dbt using the MetricFlow spec | ✅ | ✅ | | Generate SQL from a set of config files | ✅ | ✅ | | Query metrics and dimensions through the command line interface (CLI) | ✅ | ✅ | | Query dimension, entity, and metric metadata through the CLI | ✅ | ✅ | | Query metrics and dimensions through semantic APIs (ADBC, GQL) | ❌ | ✅ | | Connect to downstream integrations (Tableau, Hex, Dot, Mode, Google Sheets, and so on) | ❌ | ✅ | | Create and run Exports to save metrics queries as tables in your data platform | ❌ | ✅ | #### Related docs * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### dbt Semantic Layer FAQs dbt platform The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) is a dbt offering that allows users to centrally define their metrics within their dbt project using [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md). The Semantic Layer offers: * Dynamic SQL generation to compute metrics * APIs to query metrics and dimensions * First-class [integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) to query those centralized metrics in downstream tools The Semantic Layer is powered by MetricFlow, which is a source-available component. #### Overview of the dbt Semantic Layer  What are the main benefits of using the dbt Semantic Layer? The primary value of the dbt Semantic Layer is to centralize and bring consistency to your metrics across your organization. Additionally, it allows you to: * **Meet your users where they are** by being agnostic to where your end users consume data through the supporting of different APIs for integrations. * **Optimize costs** by spending less time preparing data for consumption. * **Simplify your code** by not duplicating metric logic and allowing MetricFlow to perform complex calculations for you. * **Empower stakeholders** with rich context and flexible, yet governed experiences. [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack.  What's the main difference between the dbt Semantic Layer and dbt Metrics? dbt Metrics is the now-deprecated dbt package that was used to define metrics within dbt. dbt Metrics has been replaced with [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md), a more flexible and powerful engine, which powers the foundation of the dbt Semantic Layer today. MetricFlow introduces SQL generation to the dbt Semantic Layer and offers more advanced capabilities than dbt Metrics, for example: * **Query construction** — MetricFlow iteratively constructs queries using a dataflow plan, our internal DAG for generating SQL. By comparison, dbt Metrics relied on templated Jinja to construct SQL. * **Joins** — MetricFlow also has a sophisticated way of handling joins, which dbt Metrics did not support. With MetricFlow you can effortlessly access all valid dimensions for your metrics on the fly, even when they are defined in different semantic models.  Is there a dbt Semantic Layer discussion hub? Yes, absolutely! Join the [dbt Slack community](https://app.slack.com/client/T0VLPD22H) and [#dbt-cloud-semantic-layer](https://getdbt.slack.com/archives/C046L0VTVR6) slack channel for all things related to the dbt Semantic Layer.  How does the dbt Semantic Layer fit with different modeling approaches (Medallion, Data Vault, Dimensional modeling)? The dbt Semantic Layer is flexible enough to work with many common modeling approaches. It references dbt models, which means how you configure your Semantic Layer will mirror the modeling approach you've taken with the underlying data. The primary consideration is the flexibility and performance of the underlying queries. For example: * A star schema data model offers more flexibility for dimensions that are available for a given metric, but will require more joins. * A fully denormalized data model is simpler, will be materialized to a specific grain, but won’t be able to join to other tables. While the dbt Semantic Layer will work for both cases, it's best to allow MetricFlow to handle some level of denormalization for you in order to provide more flexibility to metric consumers.  How is the dbt Semantic Layer priced? The dbt Semantic Layer measures usage in distinct 'Queried Metrics'. Refer to the [Billing](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#what-counts-as-a-queried-metric) to learn more about pricing. #### Availability  What data platforms are supported by the dbt Semantic Layer? The dbt Semantic Layer supports the following data platforms: * Snowflake * BigQuery * Databricks * Redshift * Postgres * Trino Support for other data platforms, such as Fabric, isn't available at this time. If you're interested in using the dbt Semantic Layer with a data platform not on the list, please [contact us](https://www.getdbt.com/contact).  Do I need to be on a specific version of dbt to use dbt Semantic Layer? Yes, the dbt Semantic Layer is compatible with [dbt v1.6 or higher](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md).  Does dbt Semantic Layer require a specific dbt plan? Yes, dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing) plan customers can access the dbt Semantic Layer. Certain features like caching and using multiple credentials are available for Enterprise and Enterprise+ plans.  Is there a way to leverage dbt Semantic Layer capabilities outside the dbt platform? The dbt Semantic Layer is proprietary to dbt, however some components of it are open-source. dbt users can use MetricFlow features, like defining metrics in their projects, without a dbt platform plan. dbt users can also query their semantic layer locally using the command line. However, they won't be able to use the [APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) or [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) to access metrics dynamically.  Is there a solution or licensing path for an organization that doesn't use dbt for pipelining, but might like to implement the dbt Semantic Layer? If you're interested in the this type of implementation, please reach out to us [here](https://www.getdbt.com/contact). #### How does the dbt Semantic Layer work?  Why is the dbt Semantic Layer better than using tables or dbt models to calculate metrics? You can use tables and dbt models to calculate metrics as an option, but it's a static approach that is rigid and cumbersome to maintain. That’s because metrics are seldom useful on their own: they usually need dimensions, grains, and attributes for business users to analyze (or slice and dice) data effectively. If you create a table with a metric, you’ll need to create numerous other tables derived from that table to show the desired metric cut by the desired dimension or time grain. Mature data models have thousands of dimensions, so you can see how this will quickly result in unnecessary duplication, maintenance, and costs. It's also incredibly hard to predict all the slices of data that a user is going to need ahead of time. With the dbt Semantic Layer, you don’t need to pre-join or build any tables; rather, you can simply add a few lines of code to your semantic model, and that data will only be computed upon request. [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack.  Do I materialize anything when I define a semantic model? No, you don't. When querying the dbt Semantic Layer through the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md), you're not materializing any data by default. The dbt Semantic Layer dynamically computes the metric using the underlying data tables. Then it returns the output to the end user.  Is the dbt Semantic Layer a physical copy of your data stored on your data warehouse? The dbt Semantic Layer does not store a physical copy of your data. It uses underlying tables to construct or compute the requested output.  How does the Semantic Layer handle data? The dbt Semantic Layer is part of the dbt platform. It allows data teams to define metrics once, centrally, and access them from any integrated analytics tool, ensuring consistent answers across diverse datasets. In providing this service, dbt Labs permits clients to access Semantic Layer metrics. Client data passes through the Semantic Layer on the way back from the data warehouse. dbt Labs handles this in a secure way using encryption and authentication from the client’s data warehouse. In certain cases, such data may be cached on dbt Labs system ephemerally (data is not persistently stored). dbt Labs employees cannot access cached data during normal business operations and must have a business need and/or direct manager approval for access to the underlying infrastructure. Access would only be when necessary for providing a client services and never with the purpose of enriching dbt Labs. No client warehouse data is retained on dbt Labs's systems. We offer a caching solution to optimize query performance. The caching feature uses client data warehouse storage rather than being stored on dbt Labs’s systems. In addition, this feature is activated only through a client opt-in. Therefore, caching is always in client hands and at client discretion  Does our agreement, the Terms of Service (ToS) for dbt, apply to the Semantic Layer? Yes it does.  Where is MetricFlow hosted? How do queries pass through MetricFlow and dbt and back to the end user? MetricFlow is hosted in dbt. Requests from the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) are routed from our API gateway to MetricFlow, which generates the SQL to compute what's requested by the user. MetricFlow hands the SQL back to our gateway, which then executes it against the data platform.  How do I configure the dbt Semantic Layer? 1. You define [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) in YAML files that describe your data, including entities (for joins), measures (with aggregation types as a building block to your metrics), and dimensions (to slice and dice your metrics). 2. Then you build your metrics on top of these semantic models. This is all done in `.yml` configurations alongside your dbt models in your projects. 3. Once you've defined your metrics and semantic models, you can [configure the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) in dbt. Read our [dbt Semantic Layer quickstart](https://docs.getdbt.com/guides/sl-qs.md) guide for more information.  How does caching work in the dbt Semantic Layer? The dbt Semantic Layer offers two layers of caching: * The result cache, which caches query results in the data platform so that subsequent runs of the same query are faster. * A declarative cache which also lives in your data platform.  Does the dbt Semantic Layer expect all models to be in normalized format? No, the dbt Semantic Layer is flexible enough to work with many data modeling approaches including Snowflake, Star schemas, Data vaults, or other normalized tables.  How are queries optimized to not scan more data than they should? MetricFlow always tries to generate SQL in the most performant way, while ensuring the metric value is correct. It generates SQL in a way that allows us to add optimizations, like predicate pushdown, to ensure we don’t perform full table scans.  What are the latency considerations of using the dbt Semantic Layer? The latency of query runtimes is low, in the order of milliseconds.  What if different teams have different definitions? If the underlying metric aggregation is different, then these would be different metrics. However, if teams have different definitions because they're using specific filters or dimensions, it's still the same metric. They're just using it in different ways. This can be managed by adjusting how the metric is viewed in downstream tools or setting up [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) to handle the various permutations of it. #### Build metrics and semantic models  Can I define my own aggregations? MetricFlow does not currently support custom aggregations on measures. You can find supported aggregation types [here](https://docs.getdbt.com/docs/build/measures.md#aggregation).  How are joins identified in the semantic model? [Joins](https://docs.getdbt.com/docs/build/join-logic.md) are identified through [entities](https://docs.getdbt.com/docs/build/entities.md) defined in a [semantic model](https://docs.getdbt.com/docs/build/semantic-models.md). These are the keys in your dataset. You can specify `foreign`, `unique`, `primary`, or `natural` joins. With multiple semantic models and the entities within them, MetricFlow creates a graph using the semantic models as nodes and the join paths as edges to perform joins automatically. MetricFlow chooses the appropriate join type and avoids fan-out or chasm joins with other tables based on the entity types. You can find supported join types [here](https://docs.getdbt.com/docs/build/join-logic.md#types-of-joins).  What is the benefit of “expr” used in semantic models and metric configurations? Expr (short for “expression”) allows you to put any arbitrary SQL supported by your data platform in any definition of a measure, entity, or dimension. This is useful if you want the object name in the semantic model to be different than what it’s called in the database. Or if you want to include logic in the definition of the component you're creating. The MetricFlow spec is deliberately opinionated, and we offer “expr” as an escape hatch to allow developers to be more expressive.  Do you support semi-additive metrics? Yes, we approach this by specifying a [dimension](https://docs.getdbt.com/docs/build/dimensions.md) that a metric cannot be aggregated across (such as `time`). You can learn how to configure semi-additive dimensions [here](https://docs.getdbt.com/docs/build/measures.md#non-additive-dimensions).  Can I use an entity as a dimension? Yes, while [entities](https://docs.getdbt.com/docs/build/entities.md) must be defined under “entities,” they can be queried like dimensions in downstream tools. Additionally, if the entity isn't used to perform joins across your semantic models, you may optionally define it as a dimension.  Can I test my semantic models and metrics? Yes! You can validate your semantic nodes (semantic models, metrics, saved queries) in a few ways: * [Query and validate you metrics](https://docs.getdbt.com/docs/build/metricflow-commands.md) in your development tool before submitting your code changes. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. #### Available integrations  What integrations are supported today? There are a number of data applications that have integrations with the dbt Semantic Layer, including Tableau, Google Sheets, Hex, Dot, and Mode, among others. Refer to [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) for more information.  How can I benefit from using the dbt Semantic Layer if my visualization tool is not currently supported? You can use [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to materialize your metrics into a table or view in your data platform. From there, you can connect your visualization tool to your data platform. Although this approach doesn't provide the dynamic benefits of the dbt Semantic Layer, you still benefit from centralized metrics and from using MetricFlow configurations to define, generate, and compute SQL for your metrics.  Why should I use exports as opposed to defining a view within my data platform? Creating an [export](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) allows you to bring your governed metric definitions into your data platform as a table or view. This means your metric logic is managed centrally in dbt, instead of as a view in your data platform and ensures that metric values remain consistent across all interfaces.  Can metric descriptions be viewed from third-party tools? Yes, all of our interfaces or APIs expose metric descriptions, which you can surface in downstream tools. #### Permissions and access  How do fine-grained access controls work with the dbt Semantic Layer? The dbt Semantic Layer uses service or personal tokens for authentication. [Service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) are mapped to underlying data platform credentials. These credentials control physical access to the raw data. The credential configuration allows admins to create a credential and map it to service tokens, which can then be shared to relevant teams for BI connection setup. You can configure credentials and service tokens to reflect your teams and their roles. Personal access tokens [(PATs)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) enable user-level authentication. When you use PATs to authenticate, your personal user credentials are used when running queries against the Semantic Layer. Currently, the credentials you configure when setting up the dbt Semantic Layer are used for every request. Any physical access policies you have tied to your credentials will be respected. #### Implementation  How can I implement dbt Mesh with the dbt Semantic Layer? When using the dbt Semantic Layer in a [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setting, we recommend the following: * You have one standalone project that contains your semantic models and metrics. * Then as you build your Semantic Layer, you can [cross-reference dbt models](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) across your various projects or packages to create your semantic models using the [two-argument `ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) (`ref('project_name', 'model_name')`). * Your dbt Semantic Layer project serves as a global source of truth across the rest of your projects. (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ###### Usage example For example, let's say you have a public model (`fct_orders`) that lives in the `jaffle_finance` project. As you build your semantic model, use the following syntax to ref the model: models/metrics/semantic\_model\_name.yml ```yaml semantic_models: - name: customer_orders defaults: agg_time_dimension: first_ordered_at description: | Customer grain mart that aggregates customer orders. model: ref('jaffle_finance', 'fct_orders') # ref('project_name', 'model_name') entities: ...rest of configuration... dimensions: ...rest of configuration... measures: ...rest of configuration... ``` Notice that in the `model` parameter, we're using the `ref` function with two arguments to reference the public model `fct_orders` defined in the `jaffle_finance` project. <br />  Which ‘staging layer’ should the dbt Semantic Layer talk to? Raw, staging, or marts? We recommend to build your semantic layer on top of the [marts layer](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md), which represents the clean and transformed data from your dbt models.  Should semantic layer credentials mirror those for production environments? Or should they be different? Semantic layer credentials are different than the credentials you use to run dbt models. Specifically, we recommend a less privileged set of credentials since consumers are only reading data.  How does the dbt Semantic Layer support a dbt Mesh architecture design? Currently, semantic models can be created from dbt models that live across projects ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)). In the future, users will also be able to use mesh concepts on semantic objects and define metrics across dbt projects. (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. --- ### dbt State trial and billing Login required | Usage-based Start a dbt State trial, continue on your plan's free monthly DATT allowance, or manage paid access from the dbt platform **Billing & Usage** dashboard. #### How the trial works * Eligible new organizations receive 30 days of free use with no usage limit. * To start a dbt State trial, you need a dbt account so you can manage dbt State usage, billing, and spend limits from one dashboard. A paid dbt platform plan is *not* required to use dbt State locally. To start your trial, refer to the instructions in [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). * Once started, you cannot pause the trial. * Seven days before your trial ends, your admin gets an email with what's coming: the trial end date, the monthly DATT allowance your account will receive, and how to upgrade if needed. * The [free monthly DATT allowance](#free-monthly-datt-allowance) includes a set amount of daily active target tables for your account. The allowance amount and what happens when you reach the limit depend on your plan. * If you were using state-aware orchestration prior to June 1, 2026, your dbt State trial will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. #### How billing works ##### Daily active target tables For purposes of pricing, daily active target tables (DATT) are measured as the number of distinct target tables (as defined below) for which dbt State performs at least one of the following unique operations on a given day (based on UTC time): a skip, clone, or test reuse. A target table is a database object managed by your dbt project for a given database and schema name. It includes seeds, snapshots, dbt models (including incremental models). It also includes each distinct test (even if the tests are not built into the database because `store_failures` is disabled). For example, if `stg_customers` has `not_null` and `unique` tests on its `id` column, that's three target tables: the model and its two tests. When you run `dbt build` or a similar command, a target table is selected for execution. It counts as an active target table if dbt State can reuse it based on your configuration rules. All reuses of the same active target table in a single day (based on UTC time) count as a single daily active target table (DATT). ##### Monthly cost calculation dbt State calculates cost per billing period using the unit price (USD $0.094) x sum of daily active target tables (DATT) for all account users and all days in that billing period. For example, if you have 100 DATT in a billing period, you'll be billed for 100 \* $0.094 = $9.40. For current unit price and more information, refer to the [dbt Labs Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table). ##### DATT chart You can view your DATT count in **Account settings** > **Billing & Usage** > **Usage-based features**. Under the **State** tab, the DATT chart shows DATTs split into **Billable** and **Free**. During a trial, all DATTs are counted as free. DATTs within your [monthly allowance](#free-monthly-datt-allowance) are also counted as free and do *not* appear on your invoice. ##### Cancellation Usage is tracked through your cancellation date. You're billed at month end for usage incurred before cancellation and not charged for usage after. #### Free monthly DATT allowance Every account gets a monthly allowance of free daily active target tables (DATT). Usage within your allowance doesn’t appear on your invoice. When your trial ends, you move to your plan's monthly allowance: * Self-hosted dbt users move to the free plan, which includes 1,000 DATT each month indefinitely. * Developer plan users also get 1,000 DATT each month indefinitely. * Paid dbt platform plans get a temporary monthly allowance. Check out [paid plans with a temporary allowance](#paid-plans-with-a-temporary-allowance) for more details. The allowance works the same way on every plan: * It resets on the 1st of each month. * Unused DATT don't carry over. * Your first month gets the full allowance for your plan. Usage during your trial doesn't count against it. ###### Plans with a permanent allowance | Plan | Monthly allowance | When you reach the limit | | --------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Self-hosted dbt (free plan) | 1,000 DATT | dbt State stops reusing nodes until the next reset. [Set up usage-based billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md#setting-up-usage-based-billing) to resume. | | Developer | 1,000 DATT | dbt State stops reusing nodes until the next reset. [Set up usage-based billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md#setting-up-usage-based-billing) to resume, which moves your account to the Developer (Usage) plan. | ###### Paid plans with a temporary allowance Paid dbt platform plans also get a monthly allowance for a limited time. | Plan | Monthly allowance | Available until | When you reach the limit | | ------------------------------------------ | ----------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Starter | 1,000 DATT | March 31, 2027 | **Card on file:** dbt State keeps running and bills your card for usage past the limit.<br />**No card on file:** dbt State stops reusing nodes until the next reset. | | Enterprise, Enterprise+, legacy Enterprise | 2,000 DATT | March 31, 2027 | **Prepaid State contract:** dbt State keeps running and draws from your contract.<br />**No prepaid contract:** dbt State stops reusing nodes until the next reset. | | Legacy Teams | 1,000 DATT | December 31, 2026 | dbt State stops reusing nodes until the next reset. To continue past the limit, migrate to Starter. | tip On Starter, you can pause dbt State at the limit by turning off card billing for State on the **Usage-based features** page. If you cancel paid access, dbt State keeps running on your plan's monthly allowance. ##### Viewing your allowance Go to **Account settings** > **Billing & Usage** > **Usage-based features** > **State** tab to see your current consumption, remaining allowance, and reset date. dbt sends an email to your admin when you've used up 80% of your DATT allowance, and again if dbt State pauses at the limit. #### Setting up usage-based billing dbt State pricing is usage-based; you're billed per target table that dbt State reuses each day, *not* per dbt platform seat. You can set up billing during your trial or at any time after. Follow the steps for your account type: ###### Managed accounts If your account has consumption spend on contract, go to the **State** tab of the **Usage-based features** page and click **Allow** to bill against your committed spend. Otherwise, [contact the dbt Labs sales team](https://www.getdbt.com/contact). ###### Self-hosted and Developer accounts 1. Go to **Billing & Usage** > **Usage-based features**. 2. Click **Set up billing** (if your trial is still active) or **Add billing info** (if your trial has ended). 3. In **Add a payment method**, enter your card details and billing address. 4. Click **Save card**. 5. Under the **State** tab, the card you added is displayed. Click **Confirm and activate**. After setting up billing, you can optionally [set a spend alert](#setting-spend-alerts) to get notified when your monthly costs reach a threshold. #### Setting spend alerts You can set a spend alert to get notified when your monthly dbt State costs reach a defined threshold. 1. In your dbt platform account, click your account name in the lower-left corner above your username and click **Account settings**. 2. Go to **Billing & Usage** > **Usage-based features**. 3. In the **Spend alert** section, click **Set a spend alert**. 4. Enable the toggle to receive email notifications when monthly spend reaches your threshold. 5. In the **Alert threshold** field, enter the amount in USD that triggers the alert. 6. Click **Save**. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage) --- ### dbt support Support for dbt is available to all users through the following channels: * Dedicated dbt Support team (dbt users). * [The Community Forum](https://discourse.getdbt.com/). * [dbt Community slack](https://www.getdbt.com/community/join-the-community/). #### dbt support If you're developing on the command line (CLI) and have questions or need some help — reach out to the helpful dbt community through [the Community Forum](https://discourse.getdbt.com/) or [dbt Community slack](https://www.getdbt.com/community/join-the-community/). #### dbt platform support The global dbt Support team is available to dbt customers by [email](mailto:support@getdbt.com) or by clicking **Create a support ticket** through the dbt navigation. ##### Create a support ticket To create a support ticket in dbt: 1. In the dbt navigation, click on **Get resources**. 2. Click **Create a support ticket**. 3. Fill out the form and click **Create Ticket**. 4. A dbt Support team member will respond to your ticket through email. ##### Ask dbt Support Assistant dbt Support Assistant is an AI widget that provides instant, AI-generated responses to common questions. This feature is available to dbt users and can help answer troubleshooting questions, give a synopsis of features and functionality, or link to relevant documentation. The dbt Support Assistant AI widget is separate from [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md), a powerful AI engine that helps with code generation to accelerate your analytics workflows. The dbt Support Assistant focuses on answering documentation and troubleshooting-related questions. Enabling or disabling AI features in dbt won't affect the dbt Support Assistant's availability. info We recommend validating information received in AI responses for any scenario using our documentation. Please [contact support](mailto:support@getdbt.com) to report incorrect information provided by the Support Assistant. ##### Support plans and resources We want to help you work through implementing and utilizing dbt platform at your organization. Have a question you can't find an answer to in [our docs](https://docs.getdbt.com/) or [the Community Forum](https://discourse.getdbt.com/)? Our Support team is here to `dbt help` you! * **Enterprise and Enterprise+ plans** — Priority [support](#severity-level-for-enterprise-support), optional premium plans, enhanced SLAs, implementation assistance, dedicated management, and dbt Labs security reviews depending on price point. * **Developer and Starter plans** — 24x5 support (no service level agreement (SLA); [contact Sales](https://www.getdbt.com/pricing/) for Enterprise plan inquiries). * **Support team help** — Assistance with [common dbt questions](https://docs.getdbt.com/category/troubleshooting.md), like project setup, login issues, error understanding, setup private packages, link to a new GitHub account, [how to generate a har file](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md), and so on. * **Resource guide** — Check the [guide](https://docs.getdbt.com/community/resources/getting-help.md) for effective help-seeking strategies. Example of common support questions Types of dbt cloud-based platform related questions our Support team can assist you with, regardless of your dbt plan:<br /><br />**How do I...**<br /> * set up a dbt project? <br /> * set up a private package in dbt? <br /> * configure custom branches on git repos? <br /> * link dbt to a new GitHub account? <br /> <br /> **Help! I can't...** <br /> * log in. <br /> * access logs. <br /> * update user groups. <br /> <br /> **I need help understanding...** <br /> * why this run failed. <br /> * why I am getting this error message in dbt? <br /> * why my CI jobs are not kicking off as expected. <br /> #### dbt Enterprise accounts Basic assistance with dbt project troubleshooting. Help with errors and issues in macros, models, and dbt Labs' packages. For strategic advice, best practices, or expansion conversations, consult your Account team. For customers on a dbt Enterprise-tier plan, we **also** offer basic assistance in troubleshooting issues with your dbt project: * **Something isn't working the way I would expect it to...** * in a macro I created... * in an incremental model I'm building... * in one of dbt Labs' packages like dbt\_utils or audit\_helper... * **I need help understanding and troubleshooting this error...** * `Server error: Compilation Error in rpc request (from remote system) 'dbt_utils' is undefined` * `SQL compilation error: syntax error line 1 at position 38 unexpected '<EOF>'.` * `Compilation Error Error reading name_of_folder/name_of_file.yml - Runtime Error Syntax error near line 9` Types of questions you should ask your Account team: * How should we think about setting up our dbt projects, environments, and jobs based on our company structure and needs? * I want to expand my account! How do I add more people and train them? * Here is our data road map for the next year - can we talk through how dbt fits into it and what features we may not be utilizing that can help us achieve our goals? * It is time for our contract renewal, what options do I have? ##### Severity level for Enterprise support Support tickets are assigned a severity level based on the impact of the issue on your business. The severity level is assigned by dbt Labs, and the level assigned determines the priority level of support you will receive. For specific ticket response time or other questions that relate to your Enterprise or Enterprise+ account’s SLA, please refer to your Enterprise contract. | Severity Level | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Severity Level 1 | Any Error which makes the use or continued use of the Subscription or material features impossible; Subscription is not operational, with no alternative available. | | Severity Level 2 | Feature failure, without a workaround, but Subscription is operational. | | Severity Level 3 | Feature failure, but a workaround exists. | | Severity Level 4 | Error with low-to-no impact on Client’s access to or use of the Subscription, or Client has a general question or feature enhancement request. | #### Leave feedback Leave feedback or submit a feature request for dbt or dbt v1. ###### Share feedback or feature request for the dbt platform 1. In the dbt navigation, click **Leave feedback**. 2. In the **Leave feedback** pop up, fill out the form. 3. Upload any relevant files to the feedback form (optional). 4. Confirm if you'd like dbt Labs to contact you about the feedback (optional). 5. Click **Send Feedback**. [![Leave feedback in dbt](/img/docs/leave-feedback.gif?v=2 "Leave feedback in dbt")](#)Leave feedback in dbt ###### Share feedback or feature request for dbt * [Create a GitHub issue here](https://github.com/dbt-labs/dbt/issues). #### External help For SQL writing, project performance review, or project building, refer to dbt Preferred Consulting Providers and dbt Labs' Services. For help writing SQL, reviewing the overall performance of your project, or want someone to actually help build your dbt project, refer to the following pages: * List of [dbt Consulting Partners](https://www.getdbt.com/partner-directory). * dbt Labs' [Services](https://www.getdbt.com/dbt-labs/services/). --- ### dbt tips and tricks Use this page for valuable insights and practical advice to enhance your dbt experience. Whether you're new to dbt or an experienced user, these tips are designed to help you work more efficiently and effectively. The following tips are organized into the following categories: * [Package tips](#package-tips) to help you streamline your workflow. * [Advanced tips and techniques](#advanced-tips-and-techniques) to help you get the most out of dbt. If you're developing with the Studio IDE, you can refer to the [keyboard shortcuts](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) page to help make development more productive and easier for everyone. #### YAML tips This section clarifies where you can use [Jinja](https://docs.getdbt.com/docs/build/jinja-macros.md), nest [vars](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) and [`env_var`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) in your YAML files. For a full file-by-file reference, refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). (Applies to dbt v2.0 and later) * You can use Jinja in almost every YAML file in dbt, including `dependencies.yml` on the dbt v2. If your project must also run on dbt v1, use `packages.yml` for Jinja-based package specs instead to maintain backwards compatibility. - Use `vars` in any YAML file that supports Jinja (like `schema.yml`, `snapshots.yml`). However, note that: * In `dbt_project.yml`, `packages.yml`, and `profiles.yml` files, you must pass `vars` through the CLI using `--vars`, not defined inside the `vars:` block in the YAML file. This is because these files are parsed before Jinja is rendered. - You can use `env_var()` in all YAML files that support Jinja. Only `profiles.yml` and `packages.yml` support environment variables for secure values (using the `DBT_ENV_SECRET_` prefix). These are masked in logs and intended for credentials or secrets. For additional information, check out [dbt v1's context docs](https://github.com/dbt-labs/dbt/blob/1.latest/core/dbt/context/README.md). #### Package tips Leverage these dbt packages to streamline your workflow: | Package | Description | | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`dbt_codegen`](https://hub.getdbt.com/dbt-labs/codegen/latest/) | Use the package to help you generate YML files for your models and sources and SQL files for your staging models. | | [`dbt_utils`](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) | The package contains macros useful for daily development. For example, `date_spine` generates a table with all dates between the ones provided as parameters. | | [`dbt_project_evaluator`](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest) | The package compares your dbt project against a list of our best practices and provides suggestions and guidelines on how to update your models. | | [`dbt_expectations`](https://hub.getdbt.com/metaplane/dbt_expectations/latest/) | The package contains many tests beyond those built into dbt. | | [`dbt_audit_helper`](https://hub.getdbt.com/#:~:text=adwords-,audit_helper,-codegen) | The package lets you compare the output of 2 queries. Use it when refactoring existing logic to ensure that the new results are identical. | | [`dbt_artifacts`](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest) | The package saves information about your dbt runs directly to your data platform so that you can track the performance of models over time. | | [`dbt_meta_testing`](https://hub.getdbt.com/tnightengale/dbt_meta_testing/latest) | This package checks that your dbt project is sufficiently tested and documented. | #### Advanced tips and techniques * Use your folder structure as your primary selector method. `dbt build --select marts.marketing` is simpler and more resilient than relying on tagging every model. * Think about jobs in terms of build cadences and SLAs. Run models that have hourly, daily, or weekly build cadences together. * Use the [where config](https://docs.getdbt.com/reference/resource-configs/where.md) for tests to test an assertion on a subset of records. * [store\_failures](https://docs.getdbt.com/reference/resource-configs/store_failures.md) lets you examine records that cause tests to fail, so you can either repair the data or change the test as needed. * Use [severity](https://docs.getdbt.com/reference/resource-configs/severity.md) thresholds to set an acceptable number of failures for a test. * Use [incremental\_strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) in your incremental model config to implement the most effective behavior depending on the volume of your data and reliability of your unique keys. * Set `vars` in your `dbt_project.yml` to define global defaults for certain conditions, which you can then override using the `--vars` flag in your commands. * Use [for loops](https://docs.getdbt.com/guides/using-jinja.md?step=3) in Jinja to DRY up repetitive logic, such as selecting a series of columns that all require the same transformations and naming patterns to be applied. * Instead of relying on post-hooks, use the [grants config](https://docs.getdbt.com/reference/resource-configs/grants.md) to apply permission grants in the warehouse resiliently. * Define [source-freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) thresholds on your sources to avoid running transformations on data that has already been processed. * Use the `+` operator on the left of a model `dbt build --select +model_name` to run a model and all of its upstream dependencies. Use the `+` operator on the right of the model `dbt build --select model_name+` to run a model and everything downstream that depends on it. * Use `dir_name` to run all models in a package or directory. * Use the `@` operator on the left of a model in a non-state-aware CI setup to test it. This operator runs all of a selection’s parents and children, and also runs the parents of its children, which in a fresh CI schema will likely not exist yet. * Use the [--exclude flag](https://docs.getdbt.com/reference/node-selection/exclude.md) to remove a subset of models out of a selection. * Use the [--full-refresh](https://docs.getdbt.com/reference/commands/run.md#refresh-incremental-models) flag to rebuild an incremental model from scratch. * Use [seeds](https://docs.getdbt.com/docs/build/seeds.md) to create manual lookup tables, like zip codes to states or marketing UTMs to campaigns. `dbt seed` will build these from CSVs into your warehouse and make them `ref` able in your models. * Use [target.name](https://docs.getdbt.com/docs/build/custom-schemas.md#an-alternative-pattern-for-generating-schema-names) to pivot logic based on what environment you’re using. For example, to build into a single development schema while developing, but use multiple schemas in production. #### Related docs * [Quickstart guide](https://docs.getdbt.com/guides.md) * [About dbt](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) * [Develop in the Cloud](https://docs.getdbt.com/docs/platform/about-develop-dbt.md) --- ### dbt v2 networking requirements dbt v2 requires outbound HTTPS access to several endpoints depending on your usage. This page describes each requirement and provides guidance for enterprise environments that restrict outbound traffic. The following table summarizes all endpoints. See each section below for details. | Resource | URL | Required for | | ----------------------------------------- | ---------------------------------------------- | --------------------------- | | [Adapter drivers](#adapter-drivers) | `https://public.cdn.getdbt.com` | All users | | [Telemetry](#telemetry) | `https://p.vx.dbt.com` | All users (can be disabled) | | [Manifest downloads](#manifest-downloads) | Cloud provider storage URLs (varies by region) | dbt platform users only | #### Adapter drivers The dbt v2 binary does *not* bundle database drivers. Instead, dbt v2 automatically downloads the correct [ADBC](https://arrow.apache.org/adbc/) driver for your data platform the first time you run a dbt command (such as `dbt run`, `dbt debug`, or `dbt compile`). dbt v2 detects which driver you need based on your `profiles.yml` configuration and downloads it from the dbt Labs CDN. dbt v2 distributes all checksums with the binary itself to guarantee authenticity of the downloaded drivers. Adapter driver downloads require outbound HTTPS access to the dbt CDN: | Resource | URL | Purpose | | ------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | **Adapter drivers** | `https://public.cdn.getdbt.com` | Downloads ADBC adapter driver libraries (`.dylib`, `.so`, `.dll`) on first use or when running `dbt system install-drivers` | info dbt v2 handles driver download automatically on first use. The `dbt system install-drivers` command downloads **all** supported drivers (Snowflake, BigQuery, Postgres, Databricks, Redshift, and DuckDB) at once. This is useful if you work across multiple data platforms and want to pre-cache every driver before going offline or switching projects. ##### Enterprise proxy considerations Adapter drivers are native shared libraries (`.dylib` on macOS, `.so` on Linux, `.dll` on Windows). Some enterprise proxy filters and security tools classify these file types as executables and may block the download — even if you allowlist `public.cdn.getdbt.com` at the domain level. If your organization's proxy blocks adapter driver downloads, work with your IT team to ensure both: 1. You allowlist the domain `public.cdn.getdbt.com`. 2. Content inspection rules permit downloading native library file types (`.dylib`, `.so`, `.dll`) from that domain. If you cannot change your proxy configuration, see [Restricted network installation](#restricted-network-installation). ##### Restricted network installation If your environment cannot access `public.cdn.getdbt.com` for adapter driver downloads, you can pre-build a bundle of the dbt v2 binary and the adapter drivers into a single `.tar.gz` or Docker image and host it on an internally approved fileshare. When you assemble a bundle for air-gapped or firewall-restricted machines: 1. Use the [version compatibility matrix](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#compatibility-matrix) to choose a dbt v2 version that matches the dbt VS Code extension your users run. 2. Verify the binary against its published SHA-256 checksum before distributing it. Refer to [Verify binaries for manual and air-gapped installs](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#verify-binaries-for-manual-and-air-gapped-installs). 3. Point the dbt VS Code extension at the bundled binary with the `dbt.fusionPath` setting. Refer to [dbt extension settings](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings). For supported adapters, refer to [v2 requirements](https://docs.getdbt.com/docs/dbt/supported-features.md#requirements). #### Telemetry dbt v2 sends anonymous usage statistics to help improve the product. If the telemetry endpoint is unreachable (for example, blocked by a firewall or proxy), dbt v2 logs errors on each invocation. | Resource | URL | Purpose | | ------------- | ---------------------- | -------------------------------- | | **Telemetry** | `https://p.vx.dbt.com` | Sends anonymous usage statistics | To suppress these errors without allowlisting the URL, disable telemetry using the options described in [Anonymous usage stats](https://docs.getdbt.com/reference/global-configs/usage-stats.md). #### Manifest downloads (dbt platform only) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") For [dbt platform](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) customers using dbt v2 locally, dbt v2 downloads production manifests from dbt platform to enable features like [deferral](https://docs.getdbt.com/reference/node-selection/defer.md) and [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md). The [cloud storage provider](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) hosting your dbt platform cell serves these manifests via **pre-signed URLs**. The specific hostnames depend on your dbt platform deployment region and the underlying cloud provider. To ensure dbt v2 can download manifests, allowlist the appropriate storage domain for your region: | Cloud provider | URL pattern | Example | | ------------------------ | ----------------------------------------- | ----------------------------------------------- | | **AWS (S3)** | `https://s3.<region>.amazonaws.com` | `https://s3.ap-northeast-1.amazonaws.com` (JP1) | | **Azure (Blob Storage)** | `https://<account>.blob.core.windows.net` | `https://prodeu2.blob.core.windows.net` (EU2) | | **GCP (Cloud Storage)** | `https://storage.googleapis.com` | `storage.googleapis.com` | Because pre-signed URLs contain region and account-specific hostnames that may change over time, we recommend allowlisting the **base storage domain** for your cloud provider rather than individual URLs: * **AWS** — `s3.*.amazonaws.com` * **Azure** — `*.blob.core.windows.net` * **GCP** — `storage.googleapis.com` --- ### dbt VS Code extension features Local development The dbt VS Code extension uses a dynamic Language Server Protocol (LSP) to provide a fast, intelligent, and cost-efficient dbt development experience with enhanced workflows and easy navigation. Feature availability The extension's editor features are available to all users. Some features depend on your project's [static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) mode, and a few read data from your dbt platform account. See the [feature availability](#feature-availability) tables for the full list of features and what each needs. #### Feature availability The dbt VS Code extension is free to install, and its editor features are available to all users. What's available depends on your project's [static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) mode rather than your account: | Feature | Availability | | ---------------------------------------------------------- | ------------------------------------ | | Error diagnostics for Jinja, YAML, and SQL syntax | All users | | Jinja LSP go-to ref, source, and macro | All users | | Linter warning diagnostics | All users | | Table-level lineage | All users | | Basic dbt command UI (run, build, test, and query results) | All users | | Ref autocomplete | All users | | Refactor ref names | All users | | Dialect-aware function autocomplete | All users | | Query cache for faster incremental compiles | All users | | Preview CTE | `baseline` (the default) or `strict` | | SQL type and schema error diagnostics | `static_analysis: strict` | | Refactor column names | `static_analysis: strict` | | Column-level lineage | `static_analysis: strict` | | SQL LSP go-to column and CTE | `static_analysis: strict` | | SQL LSP hover to see the schema for `select *` | `static_analysis: strict` | These features need you to [sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) so the extension can read data from your dbt platform account: | Feature | Why it needs a dbt platform account | | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | Model docs tab with platform metadata | Reads build status, descriptions, and test results from your dbt platform account. | | [Compare changes](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md) with dbt platform deferral | Fetches the deferred manifest from your dbt platform environment. You can also point it at a local `manifest.json` instead, with no account. | #### Lightning-fast parse times Parse even the largest projects up to 30x faster than with dbt v1. The LSP query cache makes incremental compiles faster still. [](/img/docs/extension/zoomzoom.mp4) #### View compiled code Get a live view of the SQL code your models will build — right alongside your dbt code. Usage: * Click the **code icon** to view compiled code side-by-side with source code. * Compiled code will update as you save your source code. * Clicking on a dbt macro will focus the corresponding compiled code. * Clicking on a compiled code block will focus the corresponding source code. [](/img/docs/extension/compiled-code.mp4) #### Build flexibly Use the command palette to quickly build models using complex selectors. Usage: * Click the **dbt icon** or use keyboard shortcut `cmd+shift+enter` (macOS) / `ctrl+shift+enter` (Windows/Linux) to launch a quickpick menu. * Select a command to run. [](/img/docs/extension/build-flexibly.mp4) #### Live error detection Automatically validate your SQL code to detect errors and surface warnings without hitting the warehouse. Syntax-tree diagnostics for Jinja, YAML, and SQL syntax errors (L1): * Syntax errors (missing commas, misspelled keywords, and more) * Hover over red squiggles to display errors * Full diagnostic information is available in the **Problems** panel L2 dbt v2 SQL comprehension diagnostics (requires [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2)): * Missing `group by` clauses, or columns that are neither grouped nor aggregated * Invalid function names or arguments * SQL type and schema errors * Linter warning diagnostics [](/img/docs/extension/live-error-detection.mp4) #### Powerful IntelliSense Autocomplete SQL functions, model names, macros, and more. * Autocomplete `ref`s and `source` calls. For example, type `{{ ref(` or `{{ source(` and you will see a list of available resources and their type complete the function call. Autocomplete doesn't trigger when replacing existing model names inside parentheses. * Dialect-aware SQL function autocomplete [![Example of the VS Code extension IntelliSense](/img/docs/extension/vsce-intellisense.gif?v=2 "Example of the VS Code extension IntelliSense")](#)Example of the VS Code extension IntelliSense #### Instant refactoring Rename models or columns and see references update project-wide. Renaming models: * Right-click on a file in the file tree and select **Rename**. * After renaming the file, you'll get a prompt asking if you want to make refactoring changes. * Select **OK** to apply the changes, or **Show Preview** to display a preview of refactorings. * After applying your changes, `ref`s should be updated to use the updated model name. Renaming columns (requires [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2)): Column renaming depends on strict static analysis, which validates column references across your project before the extension updates downstream models. * Right-click on a column alias and select **Rename Symbol**. * After renaming the column, you'll get a prompt asking if you want to make refactoring changes. * Select **OK** to apply the changes, or **Show Preview** to show a preview of refactorings. * After applying your changes, downstream references to the column should be updated to use the new column name. Note: Renaming models and columns is not yet supported for snapshots, or any resources defined in a .yml file. [](/img/docs/extension/refactor.mp4) #### Go-to-definition and reference Jump to the definition of any `ref`, macro, model, or column with a single click. Particularly useful in large projects with many models and macros. Excludes definitions from installed packages. * Command or Ctrl-click to go to the definition for an identifier. * Right-click an identifier and select **Go to Definition** or **Go to References**. * Jinja LSP go-to-definition for `ref()`, `source()`, and macros. Column and CTE go-to-definition (requires [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2)): * Go-to-definition for column names * Go-to-definition for CTE names [](/img/docs/extension/go-to-definition.mp4) #### Rich lineage in context See lineage at the column or table level as you develop — no context switching or breaking flow. Table-level lineage: Using the lineage tab in Cursor If you're using the dbt VS Code extension in Cursor, the lineage tab works best in Editor mode and doesn't render in Agent mode. If you're in Agent mode and the lineage tab isn't rendering, just switch to Editor mode to view your project's table and column lineage. View table lineage: * Open the **Lineage** tab in your editor. It will reflect table lineage focused on the currently-open file. * Double-click nodes to open the files in your editor. * The lineage pane updates as you navigate the files in your dbt project. * Right-click on a node to update the DAG, or view column lineage for a node. [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) (depends on [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2)): View column lineage: * Right-click on a filename, or in the SQL contents of a model file. * Select **dbt: View Lineage** --> **Show column lineage**. * Select the column to view lineage for. * Double-click on a node to update the DAG selector. * You can also use column selectors in the lineage window by adding the `column:` prefix and appending the column name. [](/img/docs/extension/lineage.mp4) #### Hover insights See context on tables, columns, and functions without leaving your code. Simply hover over any SQL element to see details like column names and data types. Hover insights depend on [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2), which lets the extension understand column types and function signatures across your project. Usage: * Hover over `*` to see expanded list of columns and their types. * Hover over column name or alias to see its type. [](/img/docs/extension/hover-insights.mp4) #### Live preview for models and CTEs Preview query output directly from inside your editor for faster validation and debugging. * Click the **table icon** or use keyboard shortcut `cmd+enter` (macOS) / `ctrl+enter` (Windows/Linux) to preview query results for a model or selected SQL snippet. * Results are displayed in the **Query Results** tab in the bottom panel. * The preview table is sortable and results are stored until the tab is closed. CTE preview: * Click the **Preview CTE** codelens to preview CTE results. [](/img/docs/extension/preview-cte.mp4) #### Explore your catalog This tab reads metadata from your dbt platform account, so you need to [sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to use it. Open the **Catalog** tab to see information for the model you're working on — enriched by your dbt platform account — without leaving your editor. For the current model, the catalog tab surfaces: * The build status, last build time, and run duration from the dbt platform. * The model's **Description**. * The model's **Columns**, including each column's type, description, and test results. Sort columns alphabetically or by test name. * A **View in dbt platform** link to open the resource in the dbt platform. [![Example of the Catalog tab in the dbt VS Code extension](/img/docs/extension/vsce-catalog-tab.png?v=2 "Example of the Catalog tab in the dbt VS Code extension")](#)Example of the Catalog tab in the dbt VS Code extension #### Generate a system report Generate a system report to collect your VS Code extension logs and system information into a zip file. This is useful when troubleshooting issues with the dbt VS Code extension. You can share the zip file with dbt Labs support to help diagnose problems. To generate and download a system report: 1. Open the Command Palette (`Cmd+Shift+P` on macOS, `Ctrl+Shift+P` on Windows/Linux). 2. Search for and select **dbt: Generate System Report**. 3. Choose a location to save the .zip file when prompted. 4. A notification will confirm where the file was saved. #### Compare changes in development [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Uses your dbt platform account This capability can read from your dbt platform account. [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) so the dbt VS Code extension can reach it. (Applies to dbt v2.0 and later) Authentication is handled by [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0), so your login state is shared across the CLI, dbt VS Code extension, and . You can use compare changes, powered by dbt v2, in your local development environment to compare your current working copy against your `manifest.json` (for example, your last production state) directly in your editor. For more details on how to use this feature, refer to [Compare changes in local development](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md). [![Example of the Compare tab](/img/docs/extension/vs-compare-changes.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab --- ### dbt Wizard billing and AI access FAQs Common questions about AI being turned on by default, how dbt Wizard usage is measured, what your usage credits covers, and how spend limits work. What's changing from September 1, 2026 From September 1, 2026, a couple of things are changing for dbt AI features: * **AI features are being enabled by default.** They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can turn AI features on or off anytime in **Account settings**. * **dbt Wizard is moving to usage-based billing** for [dbt-managed AI](#dbt-managed-inference). Usage is metered per token against your consumption pool, and an admin can set a monthly spend limit in dbt platform. #### AI enabled by default AI features are being enabled by default for dbt platform accounts. They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can turn AI features on or off anytime in **Account settings**. Turning AI on doesn't create a charge on its own — refer to [Billing FAQs](#billing-faqs) in the next section to understand how usage is metered.  Which AI features are enabled by default? The following surfaces are on by default: * dbt Wizard in Studio IDE * dbt Wizard home tab * dbt Copilot in dbt platform (includes Canvas and Insights). * Any future dbt AI features will automatically become available as well.  AI features aren't on for my account yet. How do I turn them on? AI features are already on for new accounts and are rolling out soon to existing accounts, so they may not be on for your account right away. An account admin can turn them on now in **Account settings** — refer to [Manage AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). If your organization opted out, they'll remain off until an admin turns them on.  I previously asked for AI to be permanently disabled. Will it turn on anyway? No. If your organization already opted out of AI features contractually or had them permanently disabled, they stay off. You don't need to do anything before September 1, 2026.  Can I opt out of AI features? Yes. An account admin can turn AI off at any time in **Account settings**. Refer to [Manage AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) for the steps — the same toggle controls both dbt Wizard and dbt Copilot.  If AI is enabled by default, will I be charged automatically? No. Enabling AI doesn't authorize paid usage by itself. dbt Wizard usage draws from your included consumption pool or trial pool. On Developer, Starter, and self-hosted plans, dbt-managed dbt Wizard pauses once that pool is used up, and going beyond it requires explicit purchase. Enterprise and Enterprise+ accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. If you keep AI disabled, you incur no AI charges after September 1, 2026.  Which AI features use consumption-based billing? Only dbt Wizard with dbt-managed inference, across dbt Wizard in dbt platform and the dbt Wizard CLI. dbt Copilot stays on its existing actions-based model and isn't moving to consumption-based billing.  How do I check whether AI is enabled and what my account has used? An account admin will be able to check the AI toggle in **Account settings**. To see usage and remaining credit from September 1st, 2026, go to **Account settings** > **Billing & Usage**. The overview and the dbt Wizard usage-based feature page will show your consumption pool, amount used and remaining, and the reset date. Historical dbt Copilot Actions usage appears there too.  Who do I contact about AI access or usage limits? Contact your dbt Labs account team for questions about enabling or disabling AI features, purchasing additional usage credits, or contract-specific billing questions. If you're on Developer or Starter plan, [reach out to dbt Support](mailto:support@getdbt.com) for help. #### Billing FAQs The following questions cover how dbt-managed inference is metered for dbt Wizard usage, what your plan's consumption pool will include, and how you can track and cap your spend once Wizard usage-based billing goes live on September 1st, 2026. ##### Wizard usage overview  How is Wizard usage measured and priced? Wizard usage with the dbt-managed inference will be measured per token. Every token is processed as input or output counts. Each model has its own unique per-token pricing. Cost will depend on the model used, prompt length and complexity, and response size. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for the currently available models.  What is the dbt Wizard usage consumption pool? The dbt Wizard consumption pool is the usage balance available when using dbt-managed inference for dbt Wizard — this includes Developer and Starter plan trial Wizard usage credits and the monthly Enterprise/Enterprise+ included usage credits. As you use dbt Wizard, your token usage will automatically convert into a dollar amount and is deducted from your active usage credit. Once your usage credit is used up, additional usage draws from a purchased consumption pool, if one exists. What happens next depends on your plan: * **Developer, Starter, and self-hosted:** dbt Wizard usage pauses until you add paid usage or your pool refreshes. * **Enterprise and Enterprise+**: Accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached.  Is the Wizard consumption pool shared between dbt Wizard in the dbt platform and Wizard CLI? Yes. The consumption pool and usage credits will be shared across all users within an account and across Wizard in dbt platform and local Wizard CLI. Usage from either surface draws from the same account-level consumption pool.  Which AI models are available for use through dbt managed inference? Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for the models available as of September 1, 2026. dbt Labs bills usage of these models through your dbt account. ##### Developer and Starter plan (self-serve free trial)  What does the dbt Wizard Developer and Starter plan free trial include? Developer and Starter plans get $100 in usage credits for dbt-managed Wizard inference, free for 30 days. The $100 is *per account, not per user*, which means everyone on the account shares the same credits. On Developer, that's a single user, since it's a single-user plan. The credit covers Wizard usage across both the dbt platform and the CLI, and the trial ends when your account hits $100 in usage or 30 days, whichever comes first. The trial credits can only be spent on dbt Wizard, not on dbt State or other consumption-based features.  Who is eligible to start a dbt Wizard 30-day free trial? Developer or Starter plan accounts are eligible for the 30-day, $100/account free trial. Trials require a business email address, so personal email domains such as Gmail aren't eligible. An account admin or billing admin must start the trial.  Do unused trial consumption pools roll over or expire? Unused trial usage credits don't roll over. Your trial ends when you use up the entire $100 usage credit or 30 days pass, whichever happens first.  What happens if I deplete my trial usage credit before the 30-day trial ends? Your trial ends when you reach 30 days or use up the full $100 credits, whichever comes first. Because the credit is shared across the account, usage from any user on it counts toward the same $100. To continue using dbt managed providers, you will need to add a payment method and set a monthly spend limit. You can also continue with your own AI provider ("Bring Your Own Key (BYOK)") if you configure credentials for a supported AI provider.  What happens when my dbt Wizard trial ends? dbt-managed Wizard usage pauses unless paid access is configured by purchasing additional consumption pools. Self-serve accounts will be able to add a payment method and choose a monthly spend limit. Enterprise and Enterprise+ accounts should contact their dbt Labs account team. BYOK usage remains separate and is billed by your provider.  Will dbt automatically charge me when my trial ends? No. Starting a trial doesn't automatically create paid usage. For self-serve access, you must add a payment method and choose a spend limit. If you set up payment while the trial is active, paid usage begins only after the trial ends or its credit is exhausted. ##### Enterprise & Enterprise+ plans  What does the dbt Wizard Enterprise plan monthly usage credits include? Enterprise plans automatically include $100/month in usage credits at no cost, and Enterprise+ includes $200/month. These amounts are *per account, not per user*, which means a 5-person account and a 500-person account both get the same monthly credit, and everyone on the account draws from the same shared balance. No billing setup or opt-in is required to receive the monthly included consumption usage credits as it renews each billing period and doesn't roll over. These included monthly credits can only be spent on dbt Wizard. They can't be used for dbt State or any other consumption-based feature.  What happens when your account uses up its monthly usage credits? Enterprise and Enterprise+ account should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. If you have a committed spend amount, usage beyond your included credits draws from that commitment and appears on your invoice. BYOK usage is unaffected because your AI provider bills it separately. ##### Consumption pool add-on  How does the consumption pool work? It's the balance that covers dbt-managed inference usage (which must be purchased once you've used up any freely available monthly or trial usage credits that may be available), metered per token at cost. Pool dollars don't roll over at the end of a committed term. Unlike free Wizard usage credits, purchased committed spend isn't limited to dbt Wizard — it covers both dbt Wizard and dbt State.  Do I pay the full consumption pool or only for what I use? It depends on how you purchase dbt Wizard: * **Pay-as-you-go (self-service):** You pay only for actual dbt-managed Wizard usage, up to your selected spend limit. The spend limit is a cap, not a prepaid charge. Typically for Developer, Starter, and self-hosted plans. * **Pre-committed spend:** You commit to a specific amount upfront through your account team and are billed for that amount. Your usage is deducted from the committed amount as you use dbt Wizard. Typically for Enterprise-tiered plans. Talk to your account team to set up a pre-committed spend.  Who can set or change the dbt Wizard consumption pool limit? An account admin or billing admin can manage Wizard billing and spend controls. In the dbt platform, go to **Billing & Usage** > **Usage-based features** > **Wizard** to view or update the limit.  Is the consumption pool for dbt Wizard also shared with dbt State or dbt Copilot? It depends on which credits you're using: * Free dbt Wizard usage credits (The Developer and Starter trial pool, and the Enterprise and Enterprise+ monthly included usage credits) are scoped to dbt Wizard only. * Consumption pool add-on that you purchase covers both dbt Wizard and dbt State, so usage from either feature draws down the same account-level pool. Either way, Copilot Actions are metered separately on an actions-based model and never touch your dbt Wizard consumption pool. dbt Wizard also has its own feature-level spend limit, configured separately from dbt State. ##### Tracking usage & spend limits  Do I need a paid dbt plan or credit card to try dbt Wizard? No. You need a free dbt account to manage usage, billing, and spend limits, but you don't need a paid dbt platform plan or credit card to start the trial. If you don't have an account, you can [create one](https://www.getdbt.com/signup) during setup.  How can I track my dbt Wizard usage and remaining trial credit? From September 1st, 2026, you'll be able to track in dbt platform by going to **Account settings** > **Billing & Usage**. The overview and Wizard usage-based feature pages will show your current consumption pool/usage, trial balance, and spend controls across the platform and CLI.  Where can I find the current token rates for each supported model? Refer to the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table), which includes the current input, cache-write, cached-read, and output token rates. Rates vary by model and can change over time.  How does the spend limit work? You'll be able to set a monthly spend limit for dbt Wizard in dbt platform. You'll be alerted as you approach it, and usage pauses if you reach it until the limit is raised or the next billing period begins. ##### Bring Your Own Key (BYOK)  Does bring your own key (BYOK) usage consume dbt Wizard consumption pools? No. With BYOK, your AI provider bills you directly. BYOK usage doesn't draw from your dbt-managed consumption pools.  How does BYOK work? With BYOK, you connect your own AI provider credentials and pay the provider directly. BYOK usage doesn't consume your dbt Wizard consumption pool. Refer to the [BYOK setup guide](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) for configuration details. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information. #### dbt-managed inference  Why use dbt-managed inference instead of bringing my own key? With dbt-managed inference, there's nothing to configure or maintain. dbt Labs selects and maintains the underlying models for cost, speed, and accuracy, so your team focuses on data work, not agent upkeep. Usage is billed through your existing dbt account and covered by your consumption pool, so there's one bill instead of a second vendor relationship to manage.  Which models are available with dbt-managed inference, and who picks them? dbt-managed inference includes several frontier models, including models from OpenAI and Anthropic, plus a set of open weight models. dbt Labs maintains and updates this list, so new models become available without you having to evaluate or configure a new provider yourself. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Do I need to worry about rate limits or provider outages with dbt-managed inference? No — dbt manages the underlying provider relationships and infrastructure for dbt-managed inference. You interact with a single consumption pool and spend limit in dbt platform, regardless of which model is handling a given request.  Can I mix dbt-managed inference and BYOK? Yes. BYOK usage is billed by your provider and never draws from your dbt-managed consumption pool, so you can use dbt-managed inference for some work and BYOK for other work without either affecting the other's usage or billing.  Is dbt-managed inference more expensive than using my own provider key? Cost depends on the model and your usage pattern. dbt-managed inference is metered per token at the model's rate, while BYOK usage is billed directly by your provider at their own rates. Compare the two based on which models and volume you expect to use. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Does read-only user usage in Explore mode count against my consumption pool? Yes. Questions asked in [Explore mode](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) — including by [read-only users](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) — use dbt-managed inference and draw from your account's dbt Wizard consumption pool, the same as any other dbt Wizard usage. #### Related docs * [Manage AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) to turn AI features on or off * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [dbt AI usage](https://docs.getdbt.com/docs/platform/billing/dbt-ai-usage.md) for how dbt AI usage is metered and limited * [BYOK for dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [Billing](https://docs.getdbt.com/docs/platform/billing.md) for general dbt platform billing --- ### dbt Wizard CLI data use and telemetry Local development dbt Wizard CLI collects anonymous product telemetry to improve the AI agent experience, understand usage patterns, optimize performance, and attribute compute costs without capturing your code, queries, prompts, responses, or file contents. #### Opt out of client telemetry dbt Wizard CLI respects the following opt-out mechanisms, checked in order: 1. Set `DO_NOT_TRACK=1`. 2. Set `DBT_SEND_ANONYMOUS_USAGE_STATS=false`. 3. Store telemetry consent in `~/.dbt/.user.yml`. Setting any of these disables telemetry from the dbt Wizard CLI client. #### What dbt Wizard does not collect dbt Wizard CLI does not collect: * Prompt or response content. * SQL queries, file paths, or dbt node names. * MCP tool call arguments or outputs. * Raw API keys or tokens. Identifiers are hashed before transmission. * Error messages that contain user content. Errors are limited to class or code. #### Events collected | Event | When it is collected | Why it is collected | | ------------------------------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | LLM request completed or failed | When an LLM request completes | Token usage, model, latency monitoring, error rates, and cost attribution | | Wizard session started or ended | When a user opens or closes a dbt Wizard CLI session | Weekly active users, session duration, model adoption, and client surface usage | | Wizard turn completed | After a user message and AI response complete | Engagement depth, token consumption, model usage, status, and duration | | Wizard tool use | Each time the agent invokes a tool | Tool adoption, reliability, and performance | | Conversation feedback | If you submit feedback on a dbt Wizard conversation. | We may retain the associated transcript for up to 400 days to investigate the feedback. These transcripts are not used for model training and can be deleted upon written request. | Tool telemetry records the tool type, tool name, whether the call failed, and execution time. Tool arguments and outputs are not collected. #### Data handling * Telemetry is transmitted over HTTPS to dbt Labs ingestion infrastructure. * Events are stored in an internal dbt Labs data warehouse. * Telemetry is not shared with third parties. * API keys and tokens are not transmitted in raw form. * Local development users who opt out with the supported environment variables generate no dbt Wizard CLI client telemetry events. #### Related docs * [Install dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) * [Configure BYOK for dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md) — including [privacy and data](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#privacy-and-data) for dbt Wizard in dbt platform --- ### dbt Wizard command reference Beta ### dbt Wizard command reference [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Full reference for all `wizard` subcommands and global flags. Not the same as dbt commands This page is auto-generated and covers `wizard` commands and flags. For standard dbt project commands (`dbt run`, `dbt build`, `dbt test`, and so on.) refer to the [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md). If you see any issues, please [file an issue](https://github.com/dbt-labs/docs.getdbt.com/issues) and we'll be happy to sort it out. #### Common commands Most people use a handful of commands to get started. View the following table and then refer to the rest of the page or the [examples](#examples) section for more details. | I want to... | Command | What it does | | ---------------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------- | | Start an interactive session | `wizard` | Opens the interactive TUI where you chat with the agent. | | Run a one-off task without the TUI | `wizard exec "add tests to my staging models"` | Runs the agent once, prints the result, and exits. Good for scripts and CI. | | Review my uncommitted changes | `wizard review --uncommitted` | Runs a code review on your staged, unstaged, and untracked changes. | | Pick up where I left off | `wizard resume --last` | Reopens your most recent session with its full history. | | Check that my install is healthy | `wizard doctor` | Diagnoses your install, config, auth, runtime health | | Update to the latest version | `wizard update` | Updates wizard to the newest release. | #### How to read this reference * **Flag** — the full flag as typed. Short flags (e.g. `-m`) are listed in the Short column. * **Type** — `boolean` (presence/absence), `string`, `path`, or an enumeration of allowed values. * **Description** — what the flag does and when to use it. * Global flags apply to the base interactive `wizard` command. Subcommands can have different flag sets; use each command's section for automation. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Global flags These flags work on the base interactive `wizard` command. | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### Commands | Command | Aliases | Description | | ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------- | | [`exec`](#exec) | `e` | Run wizard non-interactively | | [`review`](#review) | — | Run a code review non-interactively | | [`login`](#login) | — | Manage login | | [`logout`](#logout) | — | Remove stored authentication credentials | | [`mcp`](#mcp) | — | Manage external MCP servers for wizard | | [`plugin`](#plugin) | — | Manage wizard plugins | | [`providers`](#providers) | — | Manage model providers | | [`mcp-server`](#mcp-server) | — | Start wizard as an MCP server (stdio) | | [`app-server`](#app-server) | — | \[experimental] Run the app server or related tooling | | [`remote-control`](#remote-control) | — | \[experimental] Manage the app-server daemon with remote control enabled | | [`completion`](#completion) | — | Generate shell completion scripts | | [`update`](#update) | — | Update wizard to the latest version | | [`doctor`](#doctor) | — | Diagnose local wizard installation, config, auth, and runtime health | | [`sandbox`](#sandbox) | — | Run commands within a wizard-provided sandbox | | [`debug`](#debug) | — | Debugging tools | | [`apply`](#apply) | `a` | Apply the latest diff produced by wizard agent as a `git apply` to your local working tree | | [`resume`](#resume) | — | Resume a previous interactive session (picker by default; use --last to continue the most recent) | | [`fork`](#fork) | — | Fork a previous interactive session (picker by default; use --last to fork the most recent) | | [`cloud`](#cloud) | — | \[EXPERIMENTAL] Browse tasks from wizard in dbt platform and apply changes locally | | [`exec-server`](#exec-server) | — | \[EXPERIMENTAL] Run the standalone exec-server service | | [`features`](#features) | — | Inspect feature flags | #### exec Run wizard non-interactively ```bash wizard exec [OPTIONS] <COMMAND> [ARGS] ``` **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `[PROMPT]` | Initial instructions for the agent. If not provided as an argument (or if `-` is used), instructions are read from stdin. If stdin is piped and a prompt is also provided, stdin is appended as a `<stdin>` block | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `--skip-git-repo-check` | — | boolean | Allow running wizard outside a Git repository | | `--ephemeral` | — | boolean | Run without persisting session files to disk | | `--ignore-user-config` | — | boolean | Do not load `$DBT_WIZARD_HOME/config.toml`; auth still uses `DBT_WIZARD_HOME` | | `--ignore-rules` | — | boolean | Do not load user or project execpolicy `.rules` files | | `--output-schema <FILE>` | — | path | Path to a JSON Schema file describing the model's final response shape | | `--color <COLOR>` | — | enum | Specifies color settings for use in the output | | `--json` | — | boolean | Print events to stdout as JSONL | | `--include-subagent-events` | — | boolean | When `--json` is set, capture and emit sub-agent events tagged with their thread ID. Each sub-agent event gets a `sub_agent_thread_id` field added to the JSONL line so consumers can distinguish it from the primary agent's events | | `--no-validation` | — | boolean | When set, the validation gate always responds with "declined" so the validation sub-agent is never spawned. Useful for benchmarking runs where sub-agent overhead is not desired | | `-o, --output-last-message <FILE>` | `-o` | path | Specifies file where the last message from the agent should be written | #### review Run a code review non-interactively **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------- | | `[PROMPT]` | Custom review instructions. If `-` is used, read from stdin | | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--uncommitted` | — | boolean | Review staged, unstaged, and untracked changes | | `--base <BRANCH>` | — | string | Review changes against the given base branch | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--commit <SHA>` | — | string | Review the changes introduced by a commit | | `--title <TITLE>` | — | string | Optional commit title to display in the review summary | #### login Manage login | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### logout Remove stored authentication credentials | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp Manage external MCP servers for dbt Wizard | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### plugin Manage dbt Wizard plugins | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### providers Manage model providers | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp-server Start dbt Wizard as an MCP server (stdio) | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### app-server \[experimental] Run the app server or related tooling | Flag | Short | Type | Description | | --------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `stdio://` (default), `unix://`, `unix://PATH`, `ws://IP:PORT`, `off` | | `--analytics-default-enabled` | — | boolean | Controls whether analytics are enabled by default. | | `--ws-auth <MODE>` | — | enum | Websocket auth mode for non-loopback listeners | | `--ws-token-file <PATH>` | — | path | Absolute path to the capability-token file | | `--ws-token-sha256 <HEX>` | — | string | Hex-encoded SHA-256 digest of the capability token | | `--ws-shared-secret-file <PATH>` | — | path | Absolute path to the shared secret file for signed JWT bearer tokens | | `--ws-issuer <ISSUER>` | — | string | Expected issuer for signed JWT bearer tokens | | `--ws-audience <AUDIENCE>` | — | string | Expected audience for signed JWT bearer tokens | | `--ws-max-clock-skew-seconds <SECONDS>` | — | string | Maximum clock skew when validating signed JWT bearer tokens | #### remote-control \[experimental] Manage the app-server daemon with remote control enabled | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit machine-readable JSON | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### completion Generate shell completion scripts **Arguments:** | Argument | Description | | --------- | --------------------------------- | | `[SHELL]` | Shell to generate completions for | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### update Update dbt Wizard to the latest version | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### doctor Diagnose local dbt Wizard installation, config, auth, and runtime health | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit a redacted machine-readable report | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--summary` | — | boolean | Only show grouped check rows and the final count summary | | `--all` | — | boolean | Expand long lists in detailed human output | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--no-color` | — | boolean | Disable ANSI color in human output | | `--ascii` | — | boolean | Use ASCII status labels and separators in human output | #### sandbox Run commands within a dbt Wizard-provided sandbox | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### debug Debugging tools | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### apply Apply the latest diff produced by dbt Wizard agent as a `git apply` to your local working tree **Arguments:** | Argument | Description | | ----------- | ----------- | | `<TASK_ID>` | | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### resume Resume a previous interactive session (picker by default; use --last to continue the most recent) **Arguments:** | Argument | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID) or thread name. UUIDs take precedence if it parses. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Continue the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--include-non-interactive` | — | boolean | Include non-interactive sessions in the resume picker and --last selection | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### fork Fork a previous interactive session (picker by default; use --last to fork the most recent) **Arguments:** | Argument | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID). When provided, forks this session. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Fork the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### cloud \[EXPERIMENTAL] Browse tasks from dbt Wizard in dbt platform and apply changes locally | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### exec-server \[EXPERIMENTAL] Run the standalone exec-server service | Flag | Short | Type | Description | | --------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `ws://IP:PORT` (default), `stdio`, `stdio://` | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--remote <URL>` | — | string | Register this exec-server as a remote environment using the given base URL | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--environment-id <ID>` | — | string | Environment id to attach to when registering remotely | | `--name <NAME>` | — | string | Human-readable environment name | | `--use-agent-identity-auth` | — | boolean | Use Agent Identity auth from DBT\_WIZARD\_ACCESS\_TOKEN for remote registration | #### features Inspect feature flags | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### Examples Here are some examples and commands that you might use. Replace the example prompts with your own: * **Run a task without the TUI** Use `wizard exec` for one-off tasks, scripts, and CI. It runs the agent, prints the result, and exits. ```shell # Run a task and exit wizard exec "explain what the orders model does" # Pipe a prompt in from stdin echo "summarize my schema.yml files" | wizard exec - # Emit machine-readable output for scripting wizard exec --json "list my models" > result.jsonl ``` * **Review your changes** `wizard review` runs a code review without starting an interactive session. ```shell # Review staged, unstaged, and untracked changes wizard review --uncommitted # Review your branch against main wizard review --base main ``` * **Resume a session** ```shell # Continue your most recent session wizard resume --last # Pick from a list of past sessions wizard resume ``` * **Override a config value** Use `-c` to override any value from `~/.dbt/wizard/config.toml` for a single run, without editing the file. ```shell # Set the model for this run only wizard exec -c model="dbt/gpt-5.5" "your prompt" ``` #### Related docs * [Slash command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-slash-commands.md) for interactive TUI slash commands * [Headless mode](https://docs.getdbt.com/docs/dbt-ai/wizard-headless.md) * [Configuration reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md) for `dbt run`, `dbt build`, and other dbt commands --- ### dbt Wizard config reference Beta ### dbt Wizard config reference [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development dbt Wizard stores configuration in two TOML files under `~/.dbt/wizard/`, each controlling a different part of the product. #### The two config files See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! TOML (Tom's Obvious, Minimal Language) is a config file format. If you've written YAML or JSON before, it'll feel familiar — keys and values are separated by `=` (with spaces before and after the equals sign), and nested sections use `[section.name]` headers instead of indentation or curly braces. For example: \~/.dbt/wizard/config.toml ```toml model = "gpt-4o" [projects."/Users/you/jaffle-shop"] trust_level = "trusted" ``` This config sets the default AI model for all new sessions and applies to all future invocations across all projects. You can override the default AI model for a specific project by setting the `model` key in the `wizard_config.toml` file for that project or using the `/model` picker in the text-based user interface (TUI). | File | Controls | When to use | | ---------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------- | | `~/.dbt/wizard/config.toml` | Agent runtime — AI model, MCP servers, approval rules, trusted projects | Change agent-level behavior across all sessions | | `~/.dbt/wizard/wizard_config.toml` | Per-project settings — dbt executable path, deferral, saved model preference | Change how dbt Wizard works with a specific dbt project | Note that `dbt_project.yml` is separate from both and controls how dbt builds your project. #### Common tasks | Goal | Edit | Restart needed? | | ----------------------------------------- | ------------------------------------------------------------------ | --------------- | | Change the default AI model | `config.toml` → `model = "gpt-4o"` | Yes | | Point dbt Wizard at a specific dbt binary | `wizard_config.toml` → `path` | Yes | | Add the dbt MCP server | `config.toml` → `[mcp_servers.dbt]` | Yes | | Mark a repo as trusted | `config.toml` → `trust_level = "trusted"` under `[projects."..."]` | Yes | | Change approval or sandbox defaults | `config.toml` | Yes | #### Config precedence Settings resolve in this order (highest to lowest): 1. CLI flags (`-m`, `-c`) at invocation 2. In-session model picker (`/model`) 3. `~/.dbt/wizard/config.toml` 4. `~/.dbt/wizard/wizard_config.toml` 5. Built-in defaults #### config.toml Global agent settings for models, approvals, sandboxing, web search, and MCP servers. `~/.dbt/wizard/config.toml` controls how the agent session behaves. It's created automatically when dbt Wizard is installed. ##### Configuration keys This is the list of configuration keys that can be set in `config.toml`. * `model` * `approval_policy` * `sandbox_mode` * `web_search` You can view more by running `wizard config --help`. ###### AI model and API New dbt Wizard CLI installs use dbt Labs-managed models out of the box. Refer to [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) to use a different provider. | Key | Type | Default | Description | | ------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | string | — | Default AI model for new sessions. Refer to [AI model ID format](#ai-model-id-format) below. Change interactively with `/model` in the TUI. | ###### Behavior | Key | Type | Default | Description | | ----------------- | ------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `approval_policy` | string | `on-request` | When Wizard asks before executing commands. Options: `untrusted` (ask for non-trusted commands), `on-request` (dbt Wizard decides), `never` (run without asking). | | `sandbox_mode` | string | `read-only` | Shell sandbox policy. Options: `read-only` (default), `workspace-write` (allow writes inside project dir), `danger-full-access` (no restrictions). | | `web_search` | string | `cached` | Web search mode. Options: `disabled`, `cached`, or `live`. | ###### Project trust Set `trust_level` per-project to allow dbt Wizard to use project-local config: ```toml [projects."/absolute/path/to/your-repo"] trust_level = "trusted" ``` ###### MCP servers ```toml [mcp_servers.dbt] command = "DBT_MCP_ENDPOINT" ``` ##### AI model ID format AI model IDs in `config.toml` use the public model ID, such as `gpt-4o`. To list all available AI model IDs: ```bash wizard debug models ``` ##### Environment variables Any key can be set as an environment variable using the `DBT_WIZARD_` prefix in `SCREAMING_SNAKE_CASE`: ```bash export DBT_WIZARD_MODEL=gpt-4o export DBT_WIZARD_APPROVAL_POLICY=never export OPENAI_API_KEY=sk-... # OpenAI (no prefix needed) export ANTHROPIC_API_KEY=sk-ant-... # Anthropic (no prefix needed) ``` `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are read directly without the `DBT_WIZARD_` prefix, following each provider's convention. ##### When it's read `config.toml` is read at session start when you run `wizard`. Changes take effect only after you exit and restart the CLI. Per-invocation `-c` flags and the in-session `/model` picker override this file for that session only. ##### Example ```toml model = "gpt-4o" [projects."/Users/you/jaffle-shop"] trust_level = "trusted" [mcp_servers.dbt] command = "DBT_MCP_ENDPOINT" ``` #### wizard\_config.toml Per-project settings Wizard writes during onboarding, including dbt path and deferral settings. `~/.dbt/wizard/wizard_config.toml` stores what dbt Wizard knows about each of your dbt projects. dbt Wizard writes to this file automatically during onboarding and when you change project settings. You can also edit it manually. ##### Configuration keys | Key | Description | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `version` | Schema version for this file (currently `1`). Don't change this manually. | | `global.terms_of_use_accepted_at` | Timestamp of when you accepted the dbt Wizard Terms of Use. Written once under the `[global]` section and applies across all projects. | | `global.dbt_compile_on_startup` | Whether to compile the development project at startup. Defaults to `true`. | | `global.prod_parse_on_startup` | Whether to parse the production environment and refresh deferral state at startup. Defaults to `true`. | | `global.prod_parse_args` | Arguments for the production parse. Defaults to `["parse", "--target", "prod"]`. | | `global.prod_state_ttl_hours` | Number of hours a production state snapshot remains fresh. Defaults to `24`. | | `global.prod_state_dir` | Directory for production state snapshots. Defaults to `target/prod-state` in the project. | | `global.compile_extra_args` | Extra arguments to append when dbt Wizard compiles any project without a project-specific override. | | `onboarded_at` | Timestamp of when you onboarded this project in dbt Wizard. | | `path` | Path to the dbt binary dbt Wizard should use for this project. | | `compile_extra_args` | Extra arguments to append when dbt Wizard compiles this project. | | `prod_parse_args` | Project-specific arguments for the production parse command. | | `deferral.mode` | Who handles [deferral](https://docs.getdbt.com/reference/node-selection/defer.md) for this project. Refer to [Deferral](#deferral) below. | | `deferral.target` | The [target](https://docs.getdbt.com/docs/local/profiles.yml.md) from your `profiles.yml` that dbt Wizard compiles and defers to when `deferral.mode` is `"wizard"` (for example, `"prod"`). | | `deferral.state` | Path to a state directory when `deferral.mode` is `"manual"`. | | `deferral.favor_state` | Whether deferred relations take precedence. Defaults to `true`. | | `profile_override.path` | Path to an alternative `profiles.yml` file for this project. | | `profile_override.profile` | Profile name to use from the selected profiles file. | | `profile_override.target` | Target to use from the selected profile. | Settings are stored per-project under `[projects."/absolute/path/to/repo"]` blocks: \~/.dbt/wizard/wizard\_config.toml ```toml version = 1 [global] terms_of_use_accepted_at = "2026-05-28T15:06:54Z" dbt_compile_on_startup = true prod_parse_on_startup = true prod_state_ttl_hours = 24 [projects."/Users/you/jaffle-shop"] onboarded_at = "2026-05-20T11:09:01.410346" path = "/Users/you/jaffle-shop/.venv/bin/dbt" [projects."/Users/you/jaffle-shop".deferral] mode = "wizard" target = "prod" favor_state = true ``` ###### Deferral [Deferral](https://docs.getdbt.com/reference/node-selection/defer.md) lets dbt Wizard reuse models that are already built elsewhere (for example, in production) instead of rebuilding everything when you're only working on part of a project, saving you time and warehouse cost. The `deferral.mode` setting in `wizard_config.toml` controls who handles deferral. It accepts six values, and dbt Wizard usually sets it for you during onboarding: | `deferral.mode` value | What it means | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `"wizard"` | dbt Wizard handles deferral for you. You tell dbt Wizard which [target](https://docs.getdbt.com/docs/local/profiles.yml.md) from your `profiles.yml` to defer to (it tries to detect one automatically when you first set up the project). dbt Wizard then compiles that target and reuses its models for any upstream models you haven't built yourself. | | `"fusion_cloud"` | dbt platform handles deferral against your connected environment, so dbt Wizard doesn't manage any local state. Set automatically when you're connected to the platform. | | `"cloud_cli"` | The dbt platform CLI manages credentials and deferral through dbt platform, so dbt Wizard skips its production compile and deferral flag injection. | | `"dbt_state"` | dbt State or run cache handles deferral, so dbt Wizard skips its own production compile. | | `"manual"` | You provide the deferral manifest directory with `state`. | | `"disabled"` | Deferral is disabled for the project. | For more about dbt State, refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). Set `state` to a directory containing a compatible `manifest.json` when `mode = "manual"`: ```toml [projects."/Users/you/jaffle-shop".deferral] mode = "manual" state = "/Users/you/dbt-state/production" favor_state = true ``` **About [favor-state](https://docs.getdbt.com/reference/node-selection/defer.md?version=2.0#favor-state):** `favor_state` is configurable and defaults to `true`. When it is `true`, dbt Wizard passes `--favor-state` so deferred relations take precedence. Set it to `false` to use a relation that already exists in development and fall back to deferred state when it doesn't. You can also configure production snapshot and compile behavior with global `prod_parse_on_startup`, `prod_parse_args`, `prod_state_ttl_hours`, `prod_state_dir`, and `compile_extra_args` settings. Per-project `compile_extra_args` and `prod_parse_args` values let you override the global behavior for one project. Use a `profile_override` block to select a different profiles file, profile, or target for compilation. For configuration examples and a verification workflow, refer to [Developing with production deferral](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-6-production-deferral.md). #### Troubleshooting Here are some common config mistakes and how to fix them: | Symptom | Likely cause | Fix | | -------------------------------------------------------- | -------------------------------------------- | --------------------------------------------------- | | Still on old AI model after editing `wizard_config.toml` | `wizard_config.toml` doesn't store the model | Set `model` in `config.toml` and restart `wizard` | | AI model unchanged after editing `config.toml` | Old session still running | Press Ctrl+C, then run `wizard` again | | "Invalid model" error | Wrong AI model ID | Run `wizard debug models` to see valid AI model IDs | ##### Re-trigger onboarding flows If you need to re-run part of the setup — for example, after deleting config entries by mistake or switching to a new project — you can reset individual flows by deleting the relevant file. dbt Wizard will re-prompt you the next time it runs. To delete a file, run the command in your terminal (macOS/Linux). For example: ```bash rm ~/.dbt/wizard/auth.json ``` If you get a `No such file or directory` error, the file doesn't exist — which means that part of the setup hasn't run yet, or has already been reset. You can ignore the error. | What you want to redo | File to delete | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Re-run the trusted folder prompt | Remove your project's entry from `~/.dbt/wizard/config.toml` under `[projects."..."]` | | Re-authenticate with dbt platform | `~/.dbt/wizard/auth.json` | | Re-run dbt project setup | Remove your project's entry from `~/.dbt/wizard/wizard_config.toml` under `[projects."..."]`, or delete the file entirely to reset all projects | | Re-trigger BYOK key configuration | `~/.dbt/wizard/provider*` (this matches all provider files) | caution `config.toml` and `wizard_config.toml` are separate files that control different things. If you want to re-run project setup, edit `wizard_config.toml` — not `config.toml`. Editing the wrong file won't re-run the onboarding flow you expect. Refer to [The two config files](#the-two-config-files) at the top of this page for the difference. #### Related docs Links to BYOK setup, approval behavior, and CLI command reference. * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [Approval and sandboxing](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing) * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) --- ### dbt Wizard in Studio IDE Preview ### dbt Wizard in Studio IDE [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based Use dbt Wizard in the Studio IDE to ship trusted dbt changes faster. It understands your project, answers context-grounded questions, generates models, tests, docs, and Semantic Layer definitions, and shows file diffs before changes are persisted. dbt Wizard supports the dbt development lifecycle from investigation to review. Use it to: * Ask project-aware questions using lineage, metadata, and catalog context. * Build or refactor models from natural-language prompts. * Generate and validate YAML for tests, documentation, semantic models, and metrics. * Make scoped edits to logic, names, materializations, tests, and related YAML. * Investigate job and run failures with dbt Agent Skills. The agent comes with the following out of the box, meaning no configuration needed: * [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills): dbt-recommended guidance and instructions, managed by dbt Labs. * [dbt MCP server Product docs toolset](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#product-docs): Tools for searching and fetching content from dbt's official documentation. AI features are being enabled by default. They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can [turn AI off or back on and configure providers](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) anytime. Prefer to learn by video? Check out the free [dbt Wizard video course](https://learn.getdbt.com/courses/dbt-wizard) on dbt Learn! ##### Prerequisites * A [dbt platform account](https://www.getdbt.com/signup) and [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * [Legacy Team plans](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) don't have access to dbt Wizard. Move to a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing) to use it. * A [development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) and credentials set up in the Studio IDE. * Use a supported AI provider. Refer to [Supported AI providers](https://docs.getdbt.com/docs/platform/wizard-platform.md#supported-ai-providers), or the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full model list and rates. If dbt Wizard stops responding, your account may have used up its usage credits. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md).  Availability and considerations * **Where it runs:** Supported in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) only, all [deployment types](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md?version=2.0). Not supported in VS Code or the dbt platform CLI. * **Engines:** Works with dbt v2 and dbt v1. * **Conversations:** In the conversation list, open **More actions** menu (three dots) of the conversation you want to delete, then click **Delete** to remove one thread. Deleting the open thread clears the panel. * **Sessions:** Refreshing the same browser tab keeps your active session. A new tab, or returning after closing the tab, starts empty. * **Chat history:** Retained for 90 days only. Chat history isn't supported yet on single-tenant deployments, so save anything important before closing. * **Plan mode:** Not supported yet. The agent doesn't show a separate plan before applying changes, however you can use the **Ask for approval** mode to approve each file. * **New chat:** Click **Start new dbt Wizard chat** (top right of the dbt Wizard panel) to begin a new session. ##### Using dbt Wizard Use the dbt Wizard panel to generate resources with quick actions, or use the agent to build and refactor models end-to-end with natural language prompts. To use the dbt Wizard, follow these steps: 1. Open your dbt project in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), then click **dbt Wizard** in the command palette. 2. Start a prompt in several ways in the [dbt Wizard panel](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?version=2#panel-controls): * **Quick actions**: Use [quick-action resource generation](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#quick-action-resource-generation) at the top of the panel for quick action prompts. * **Plain text**: Type directly into the text field to describe what you want to build or change. * **Model context**: Type `@` to select a model as context. This scopes the agent's changes to that resource. 3. Select the [**Agent mode** button](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) to specify the mode for the dbt Wizard. Available modes are **Explore only**, **Ask for approval** (default), and **Edit files automatically**. 4. Select the dbt managed model you'd like to work with from the [model picker](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#choose-a-model) next to the **Agent mode** button. 5. [Review the agent's suggestions](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#reviewing-agent-suggestions) and approve or reject the changes. You can also use the **Start new dbt Wizard chat** button to start a new chat session. 6. [Approve dbt commands](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#granting-command-permissions) when the dbt Wizard requests to run commands like `dbt compile` or `dbt build`. 7. Repeat the process to build or change more models. 8. Commit the changes to your dbt project and open a pull request. Be warned, the wizard has been known to cast spells. The following images show how dbt Wizard displays its work and outcome: [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. [![Wizard final refactor result displayed as a diff](/img/docs/dbt-platform/wizard-ide-refactor-diff.png?v=2 "Wizard final refactor result displayed as a diff")](#)Wizard final refactor result displayed as a diff For more details on the dbt Wizard and how it works, expand the following sections to open additional information.  Panel controls The dbt Wizard panel contains: 1. **Quick actions** (center): Buttons at the top of the panel for common tasks like generating documentation, tests, semantic models, and metrics. When selected, the text field is pre-filled with a prompt. 2. **Agent mode button** (bottom left): Switch between **Explore only**, **Ask for approval**, and **Edit files automatically** mode. Click the button to change modes. 3. **Model picker** (bottom left): Select the dbt managed model to use for the session. Refer to [Choose a model](#choose-a-model) for the available models. 4. **dbt model context** (bottom left): Shows the currently open file. Use `@` in the text field to reference a different dbt model. Click **x** to remove the dbt model context. 5. **Text input field** (bottom left): Type your prompt in the text field to describe what you want to build or change. Type `@` to select a dbt model as context. This scopes the agent's changes to that resource. 6. **Start new dbt Wizard chat** (top right): Starts a new chat session. 7. **Stop** or **Enter** (bottom right): Press **Enter** to submit your prompt. Press **Stop** to stop the current session and agent processing. You cannot undo this action. [![The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field.](/img/docs/dbt-platform/wizard-panel.png?v=2 "The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field.")](#)The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field. dbt Wizard also has a simplified wayfinder bar above the text input field. The wayfinder bar shows your current project and branch and guides you through Git tasks, such as committing files or creating a branch. []()  Agent modes The dbt Wizard operates in three modes: | Mode | Behavior | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Explore only** | The agent queries and explains data but can't edit files or run builds. Best when you want to analyze data or validate a model's output without the agent proposing changes. Every answer comes with the SQL or metric definition behind it. Available for read-only users in the home tab. | | **Ask for approval** (default) | The agent drafts edits to files. You approve each file change before it is persisted. Best when you want tight control over what gets saved to your branch. | | **Edit files automatically** | The agent drafts and automatically saves file edits without per-file approval. Best for faster iteration when you're confident in the prompt. | Switch between modes anytime with the **Agent mode** button (bottom-left). The authoring modes keep the analytical tools available, so switching out of **Explore only** doesn't cost you anything. [![dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.](/img/docs/dbt-platform/wizard-modes.png?v=2 "dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.")](#)dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.  Reviewing agent suggestions When the dbt Wizard proposes code changes, you can review them before they are saved to your project: * **View the diff**: The agent displays a diff of the proposed changes. Click **Show all X lines** to expand and view the full suggestion. * **Line indicators**: Added and removed lines are highlighted with line number indicators so you can see exactly what changed. * **Copy or open in editor**: Use the options in the top-right corner of the diff view to copy the suggestion or open it directly in the editor.  Granting command permissions To validate or run models during a session, the agent may request to run dbt commands such as `dbt compile` or `dbt build`. You'll be prompted to approve each request before it executes. For example, the agent might request to run: ```bash dbt compile --select model_name ``` You can select one of the following options: | Option | Behavior | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | **Yes, run once** | Grants permission to run this specific command one time. | | **Yes, and allow `dbt_command_name` for the session** | Grants permission to run dbt commands for the remainder of your session without prompting again. | | **No** | Denies the request. The agent will not run the command. | After you run a command, dbt Wizard adds an icon and a tooltip to the Studio IDE [**Commands** tab](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#console-section) results. This helps you distinguish agent-run commands from manually run commands in the run results and logs. ##### Bringing your own skills You can extend dbt Wizard with custom skills to encode your team's SQL conventions, naming rules, and modeling workflows — so you don't repeat them in every prompt. See [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-skills.md) for the full reference, including how to create, structure, and invoke skills. ##### Debug job failures The dbt Wizard can investigate and troubleshoot dbt job and run failures directly from the Studio IDE. This capability is powered by the `troubleshooting-dbt-job-errors` [dbt Agent Skill](https://github.com/dbt-labs/dbt-agent-skills), which comes pre-configured with the agent — no setup required. You can ask the agent questions and issue commands like: * "What jobs have failed recently?" * "What is the root cause of the job failure?" * "How can I fix the recent job failure?" * "Fix the job failure." The agent notes when your local project state may differ from the job — for example, if you're on a different branch or have uncommitted changes — so you have full context before acting on any suggested fixes. ##### Timeout handling When a dbt command run by dbt Wizard runs for more than 5 minutes, the agent automatically attempts to stop the command on the server before returning control to you. Instead of hanging or showing a generic error, the agent returns a clear message that explains the command timed out and was aborted. The message also tells you whether the cancellation request succeeded. If cancellation fails, it's possible the command may still be running on the server. You can then choose whether to retry the command, narrow the request, or take another action. ##### dbt v2 migration workflow If you have access to [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) with [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled, you can use the [dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) skill. This skill can help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed. It classifies every error, applies validated fixes automatically, and surfaces what's blocked. info The dbt v2 migration workflow is accessible through the dbt Wizard in the Studio IDE. If you're using VS Code or the dbt platform CLI, use the [autofix tool](https://docs.getdbt.com/guides/dbt-package-compat?step=4) instead. 1. From the job list, click the **Review job** button for a job with a successful run. * If you don't see the **Review job** button, enable the **Show dbt v2 eligibility** toggle in the job list. 2. In the **dbt v2 eligibility unknown for this job** pop-up, click **Debug in Studio with dbt Wizard**. 3. dbt redirects you to the Studio IDE and sets your personal development environment to dbt v2. 4. dbt Wizard opens and automatically triggers the dbt v2 migration skill with this prompt: ```text I need help fixing dbt v2 compatibility issues in this project. Please investigate and resolve any deprecation warnings or incompatibilities. Please use the migrating-dbt-core-to-fusion skill to guide this. ``` 5. Review and approve dbt Wizard's permission requests so it can run the commands it needs. 6. The dbt Wizard iteratively runs `dbt compile`, reads the results, and applies fixes until it reaches a successful compile or encounters an error it can't resolve. If it gets blocked, it exits cleanly, explains what it could not fix, and creates and links to a markdown file summarizing all changes made. 7. When the project compiles with no warnings or errors, commit and publish your changes. 8. After you merge the changes, wait for the job to run again or run it manually on dbt v2. [![The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.](/img/docs/dbt-platform/v2-migration-workflow.gif?v=2 "The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.")](#)The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE. For more on how to prepare your project for dbt v2 and what to do when you hit compatibility errors, see the [dbt v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) and the [Upgrade to dbt v2 guides](https://docs.getdbt.com/guides/prepare-v2-upgrade.md). ##### Writing effective prompts Good prompts include the *scope* (which models or area of the project), the *intent* (the transformation or business logic you want), and any *constraints* (naming conventions, materialization, tests). Here are a few examples: | Task | Example prompt | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Build a new model | "Create a model called `fct_daily_revenue` that joins `stg_orders` and `stg_payments`, aggregates revenue by day, and materializes as a table." | | Refactor an existing model | "Refactor `fct_orders` to use incremental materialization. Keep existing tests and follow our naming conventions." | | Generate tests and docs | "Add `not_null` and `unique` tests to the primary key of `dim_customers`, and generate documentation for all columns." | For detailed guidance, patterns, and more examples across SQL, documentation, tests, and semantic models, see the [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md). Best practices for using dbt Wizard For recommended workflows on real project tasks — understanding a project, validating changes, building Semantic Layer definitions, and more — refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md). Most of these prompts work the same in Studio IDE. #### Ask questions in Explore mode Explore mode in Studio IDE lets you ask questions of your production data in plain language. dbt Wizard in Explore mode answers and explains but never changes your project. This option is great for exploratory data analysis and getting a quick understanding of your data. Explore mode queries with your personal warehouse credentials. 1. Open dbt Wizard and set the mode picker (bottom-left) to **Explore only** if needed. 2. Type your question and press **Enter**. Make sure you're specific and include the time period, grouping, and filter you care about: * `what was total revenue in Q2 2026, by month?` * `how many new customers signed up in July?` * `which regions grew fastest this year?` 3. dbt Wizard gives a plain-language summary of what it did, then the result. Switch between **Chart**, **Table**, and **SQL** to see the data your way. 4. Validate the data by using the **SQL** view to review the query or governed metric behind the answer. Note that you can't edit the SQL query but you can copy it. 5. Keep going by asking a follow-up in the same conversation. For example, `now break that out by region` works after your first question. Explore mode uses dbt-managed inference, so questions draw from your account's dbt Wizard [consumption pool](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) like any other dbt Wizard usage. In the Studio IDE, the answer and its visualization appear inline in the dbt Wizard panel alongside your project. [![Explore mode in the Studio IDE, showing a chart and Chart/Table/SQL toggles in the dbt Wizard panel.](/img/docs/dbt-platform/wizard-ide-explore-viz.png?v=2 "Explore mode in the Studio IDE, showing a chart and Chart/Table/SQL toggles in the dbt Wizard panel.")](#)Explore mode in the Studio IDE, showing a chart and Chart/Table/SQL toggles in the dbt Wizard panel. #### Choose a model With dbt managed inference, you can switch between the supported managed models at any using the model picker dropdown next to the **Agent mode** control (where you choose **Ask for approval** or **Edit files automatically**). * In [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) and the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), open the model picker dropdown next to the **Agent mode** control in the dbt Wizard panel, then select a model. * The picker lists the managed models available to you. Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the available models and their token rates. * If you [bring your own key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md), dbt Wizard uses the provider and model you configured with your key rather than the managed model picker. [![The model picker dropdown next to the Agent mode control in the Wizard panel.](/img/docs/dbt-platform/wizard-model-picker.png?v=2 "The model picker dropdown next to the Agent mode control in the Wizard panel.")](#)The model picker dropdown next to the Agent mode control in the Wizard panel. #### Related docs * [About dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [dbt v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) * [Develop with dbt Wizard](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md) * [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md) * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) * [About dbt AI and intelligence](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md) --- ### dbt Wizard slash command reference Beta ### dbt Wizard slash command reference [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Full reference for dbt Wizard interactive TUI slash commands. Auto-generated page This page is auto-generated. If you see any issues, please [file an issue](https://github.com/dbt-labs/docs.getdbt.com/issues) and we'll be happy to sort it out. This page covers the slash commands supported by dbt Wizard. These are `/commands` typed inside the interactive text based user interface (TUI) session to control dbt Wizard's behavior mid-session. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Slash commands Type `/` in the composer to open the command picker. Use arrow keys to navigate or keep typing to filter. Press **Tab** to queue a command while a task is running — it executes at the end of the current turn. ##### All slash commands | Command | Alias | Purpose | Available during task | | --------------------------------------------------- | -------- | ---------------------------------------------------------------------- | --------------------- | | [`/model`](#model-and-ai) | — | Choose what model and reasoning effort to use | ✗ | | [`/providers`](#model-and-ai) | — | Configure model providers and BYOK credentials | ✓ | | [`/ide`](#review-and-context) | — | Include current selection, open files, and other context from your IDE | ✓ | | [`/permissions`](#permissions-and-safety) | — | Choose what Wizard is allowed to do | ✗ | | [`/keymap`](#customization) | — | Remap TUI shortcuts | ✗ | | [`/vim`](#customization) | — | Toggle Vim mode for the composer | ✗ | | [`/setup-default-sandbox`](#permissions-and-safety) | — | Set up elevated agent sandbox | ✗ | | [`/sandbox-add-read-dir`](#permissions-and-safety) | — | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | ✗ | | [`/experimental`](#customization) | — | Toggle experimental features | ✗ | | [`/approve`](#customization) | — | Approve one retry of a recent auto-review denial | ✓ | | [`/memories`](#memory) | — | Configure memory use and generation | ✗ | | [`/skills`](#skills-and-extensions) | — | Use skills to improve how Wizard performs specific tasks | ✓ | | [`/hooks`](#skills-and-extensions) | — | View and manage lifecycle hooks | ✓ | | [`/review`](#review-and-context) | — | Review my current changes and find issues | ✗ | | [`/rename`](#session-management) | — | Rename the current thread | ✓ | | [`/new`](#session-management) | — | Start a new chat during a conversation | ✗ | | [`/resume`](#session-management) | — | Resume a saved chat | ✗ | | [`/fork`](#session-management) | — | Fork the current chat | ✗ | | [`/init`](#session-management) | — | Create an AGENTS.md file with instructions for Wizard | ✗ | | [`/compact`](#session-management) | — | Summarize conversation to prevent hitting the context limit | ✗ | | [`/plan`](#long-running-tasks) | — | Switch to Plan mode | ✗ | | [`/goal`](#long-running-tasks) | — | Set or view the goal for a long-running task | ✓ | | [`/agent`](#long-running-tasks) | — | Switch the active agent thread | ✓ | | [`/side`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/btw`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/copy`](#review-and-context) | — | Copy last response as markdown | ✓ | | [`/raw`](#review-and-context) | — | Toggle raw scrollback mode for copy-friendly terminal selection | ✓ | | [`/diff`](#review-and-context) | — | Show git diff (including untracked files) | ✓ | | [`/mention`](#review-and-context) | — | Mention a file | ✓ | | [`/status`](#session-info) | — | Show current session configuration and token usage | ✓ | | [`/config`](#session-info) | — | View and manage configuration | ✓ | | [`/debug-config`](#session-info) | — | Show config layers and requirement sources for debugging | ✓ | | [`/title`](#customization) | — | Configure which items appear in the terminal title | ✓ | | [`/statusline`](#customization) | — | Configure which items appear in the status line | ✓ | | [`/theme`](#customization) | — | Choose a syntax highlighting theme | ✗ | | [`/pets`](#customization) | `/pet` | Choose or hide the terminal pet | ✗ | | [`/overview`](#review-and-context) | — | Render the dbt project overview card | ✓ | | [`/mcp`](#skills-and-extensions) | — | List configured MCP tools; use /mcp verbose for details | ✓ | | [`/apps`](#skills-and-extensions) | — | Manage apps | ✓ | | [`/plugins`](#skills-and-extensions) | — | Browse plugins | ✓ | | [`/quit`](#exit) | — | Exit Wizard | ✓ | | [`/exit`](#exit) | — | Exit Wizard | ✓ | | [`/feedback`](#session-info) | — | Send feedback to maintainers | ✓ | | [`/ps`](#background-terminals) | — | List background terminals | ✓ | | [`/stop`](#background-terminals) | `/clean` | Stop all background terminals | ✓ | | [`/clear`](#session-management) | — | Clear the terminal and start a new chat | ✗ | | [`/personality`](#model-and-ai) | — | Choose a communication style for Wizard | ✗ | | [`/realtime`](#realtime-experimental) | — | Toggle realtime voice mode (experimental) | ✓ | | [`/settings`](#realtime-experimental) | — | Configure realtime microphone/speaker | ✓ | | [`/subagents`](#long-running-tasks) | — | Switch the active agent thread | ✓ | *** ##### Model and AI Control the AI model, provider, speed, and response style. | Command | Inline args | Description | | -------------- | ----------- | ---------------------------------------------- | | `/model` | — | Choose what model and reasoning effort to use | | `/providers` | — | Configure model providers and BYOK credentials | | `/personality` | — | Choose a communication style for Wizard | *** ##### Session management Start, resume, branch, and clean up sessions. | Command | Inline args | Description | | ---------- | ----------- | ----------------------------------------------------------- | | `/new` | — | Start a new chat during a conversation | | `/clear` | — | Clear the terminal and start a new chat | | `/resume` | Yes | Resume a saved chat | | `/fork` | — | Fork the current chat | | `/compact` | — | Summarize conversation to prevent hitting the context limit | | `/rename` | Yes | Rename the current thread | | `/init` | — | Create an AGENTS.md file with instructions for Wizard | *** ##### Review and context Pull information into the session or trigger a code review. | Command | Inline args | Description | | ----------- | ----------- | ---------------------------------------------------------------------- | | `/review` | Yes | Review my current changes and find issues | | `/diff` | — | Show git diff (including untracked files) | | `/mention` | — | Mention a file | | `/copy` | — | Copy last response as markdown | | `/raw` | Yes | Toggle raw scrollback mode for copy-friendly terminal selection | | `/overview` | — | Render the dbt project overview card | | `/ide` | Yes | Include current selection, open files, and other context from your IDE | *** ##### Permissions and safety Control what Wizard is allowed to execute. | Command | Inline args | Description | | ------------------------ | ----------- | --------------------------------------------------------------------- | | `/permissions` | — | Choose what Wizard is allowed to do | | `/setup-default-sandbox` | — | Set up elevated agent sandbox | | `/sandbox-add-read-dir` | Yes | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | *** ##### Customization Appearance, keybindings, and UI preferences. | Command | Inline args | Description | | --------------------- | ----------- | -------------------------------------------------- | | `/theme` | — | Choose a syntax highlighting theme | | `/keymap` | Yes | Remap TUI shortcuts | | `/vim` | — | Toggle Vim mode for the composer | | `/statusline` | — | Configure which items appear in the status line | | `/title` | — | Configure which items appear in the terminal title | | `/experimental` | — | Toggle experimental features | | `/approve` | — | Approve one retry of a recent auto-review denial | | `/pets` (alias: /pet) | Yes | Choose or hide the terminal pet | *** ##### Skills and extensions Manage capabilities Wizard can use during a session. | Command | Inline args | Description | | ---------- | ----------- | -------------------------------------------------------- | | `/skills` | — | Use skills to improve how Wizard performs specific tasks | | `/hooks` | — | View and manage lifecycle hooks | | `/mcp` | Yes | List configured MCP tools; use /mcp verbose for details | | `/apps` | — | Manage apps | | `/plugins` | — | Browse plugins | *** ##### Long-running tasks Manage multi-turn goals, parallel agents, and branched conversations. | Command | Inline args | Description | | ------------ | ----------- | ---------------------------------------------- | | `/plan` | Yes | Switch to Plan mode | | `/goal` | Yes | Set or view the goal for a long-running task | | `/agent` | — | Switch the active agent thread | | `/subagents` | — | Switch the active agent thread | | `/side` | Yes | Start a side conversation in an ephemeral fork | | `/btw` | Yes | Start a side conversation in an ephemeral fork | *** ##### Background terminals Inspect and control shell processes Wizard has running in the background. | Command | Inline args | Description | | ----------------------- | ----------- | ----------------------------- | | `/ps` | — | List background terminals | | `/stop` (alias: /clean) | — | Stop all background terminals | *** ##### Memory Control how Wizard stores and uses memory across sessions. | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------- | | `/memories` | — | Configure memory use and generation | *** ##### Session info Inspect the current session state without changing anything. | Command | Inline args | Description | | --------------- | ----------- | -------------------------------------------------------- | | `/status` | — | Show current session configuration and token usage | | `/config` | — | View and manage configuration | | `/debug-config` | — | Show config layers and requirement sources for debugging | | `/feedback` | — | Send feedback to maintainers | *** ##### Exit | Command | Inline args | Description | | ------- | ----------- | ----------- | | `/quit` | — | Exit Wizard | | `/exit` | — | Exit Wizard | *** ##### Realtime (experimental) | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------------- | | `/realtime` | — | Toggle realtime voice mode (experimental) | | `/settings` | — | Configure realtime microphone/speaker | *** #### Related docs * [Wizard CLI command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) for flags and subcommands * [Use subagents](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) for the `/agent` and `/subagents` commands * [Headless mode](https://docs.getdbt.com/docs/dbt-ai/wizard-headless.md) for using `wizard exec` in scripts and CI * [Configuration reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) for `config.toml` keys and precedence * [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md) for `dbt run`, `dbt build`, and other dbt commands --- ### dbt Wizard use cases Common use cases for dbt Wizard, with example prompts and what to expect. No spellbook needed. dbt Wizard works best when you give it a clear scope (which dbt model or area), an intent (what you want to change or learn), and any constraints (naming conventions, materialization, tests). The following examples follow that pattern. Start a new [chat or session](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md#use-one-chat-per-task) for each discrete task. For example, building a model and debugging an unrelated job failure should be two separate conversations. * [Build a new model](#build-a-new-model) * [Refactor to incremental](#refactor-to-incremental) * [Add tests and docs](#add-tests-and-docs) * [Debug a job failure](#debug-a-job-failure) * [Assess source impact](#assess-source-impact) * [Rename a column project-wide](#rename-a-column-project-wide) * [Multi-file changes](#multi-file-changes) * [Validate before shipping](#validate-before-shipping) * [Add a semantic model](#add-a-semantic-model) These prompts work the same across surfaces — the [CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md), the [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md), and dbt Wizard in [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md). For examples specific to the Studio IDE, refer to the [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md). To use dbt Wizard in the CLI, use the `wizard` [command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md). Best practices for using dbt Wizard Once you're set up, refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows on real project tasks. #### Build a new model You have clean source data and want a new mart model without writing all the SQL by hand. **Example prompt:** ```text Create a model called `fct_monthly_revenue` that joins `stg_orders` and `stg_payments`, groups by `month` and `customer_id`, and materializes as a table. Add `not_null` tests to the primary key and a unique test on the grain. ``` **What dbt Wizard does:** 1. Reads `stg_orders` and `stg_payments` from your project index to understand the available columns 2. Generates `fct_monthly_revenue.sql` with the join and aggregation logic 3. Creates a matching YAML block with the tests you asked for 4. Shows a diff for both files — you approve before anything is saved **Tips:** * Reference your existing staging models by name so dbt Wizard uses the right columns * If your project has a team style guide in `.agents/skills/`, dbt Wizard picks it up automatically and applies naming and materialization conventions #### Refactor to incremental A full-refresh mart is getting slow. You want to switch it to incremental without breaking existing tests. **Example prompt:** ```text Refactor `fct_orders` to use incremental materialization with a `unique_key` on `order_id` and an updated_at filter. Keep all existing tests and don't change the output schema. ``` **What dbt Wizard does:** 1. Reads the current `fct_orders.sql` and its YAML 2. Rewrites the model with an `{% if is_incremental() %}` filter 3. Adds the `unique_key` and `materialized = 'incremental'` config 4. Flags if any existing tests might behave differently on incremental runs **Tips:** * Tell dbt Wizard which column to use as the high-watermark: it can infer a likely candidate from your schema but an explicit instruction is more reliable * Ask it to add a `full_refresh` note to the model description so future maintainers understand the intent #### Add tests and docs You've inherited a model with no tests or YAML. You want coverage without writing boilerplate. **Example prompt:** ```text Generate tests and documentation for dim_customers. Add not_null and unique on customer_id, not_null on email, and accepted_values on customer_status with values active, churned, and prospect. Write a column description for each. ``` **What dbt Wizard does:** 1. Reads `dim_customers.sql` and any existing YAML 2. Generates a complete `dim_customers.yml` with model description, column descriptions, and the tests you specified 3. Infers reasonable descriptions from column names — you review and edit before approving **Tips:** * You can ask dbt Wizard to write descriptions in a specific voice or format: "Write the descriptions in plain language, one sentence each" * To document a whole layer at once: "Generate documentation for all models in `models/staging/` that don't have a YAML file yet" For a workflow that finds coverage gaps and checks candidate assertions against warehouse data, refer to [Add data-informed tests with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-4-data-informed-tests.md). #### Debug a job failure A dbt platform job failed overnight. You want to understand why without digging through logs manually. **Example prompt:** ```text The nightly job failed. What's the root cause and how do I fix it? ``` **What dbt Wizard does:** 1. Uses the run evidence you provide, or connected dbt MCP tools, to retrieve and inspect job run details 2. Identifies the failing model, the error message, and the likely cause 3. Proposes a fix and shows the diff 4. Notes if your local branch differs from the job's branch so you have full context **Tips:** * You can be more specific: "What caused the failure in `fct_orders` in the last run of the Production job?" * Wizard won't apply a fix without your approval, which is especially useful when the failure is in a production model For the evidence-gathering, diagnosis, and validation procedure, refer to [Debug a failed dbt job with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-5-debug-failed-job.md). #### Assess source impact Before modifying a stg model, you want to know what other downstream models might be affected. **Example prompt:** ```text If I change the grain of stg_payments from one row per payment to one row per payment attempt, which downstream models break? ``` **What dbt Wizard does:** 1. Runs a lineage and impact query against your project to find all downstream dependents of `stg_payments` 2. Identifies which models join on payment-level keys and would be affected by a grain change 3. Lists the models by severity — marts and exposures first, then intermediate models 4. Suggests which models need to be updated as part of the change **Tips:** * You don't need to be precise about the change: even a "what depends on stg\_payments?" gives you the blast radius * Follow up with: "Write a migration plan for making this change safely" #### Rename a column project-wide A source column has been renamed. You need to update all references without missing anything. **Example prompt:** ```text The source column user_id in raw_customers has been renamed to customer_id. Update stg_customers and find any downstream models that reference user_id directly. ``` **What dbt Wizard does:** 1. Updates the column alias in `stg_customers.sql` 2. Searches downstream models for direct references to `user_id` 3. Proposes updates to any models that would break 4. Updates YAML column descriptions to match **Tips:** * Always run `dbt compile` after dbt Wizard's changes to catch any references it might have missed * Ask Wizard to "check for user\_id in any raw SQL strings or Jinja macros too" for thorough coverage #### Multi-file changes You need to make a change that touches multiple files at once — a model rename, a contract update, or a schema change — and have all the related files stay in sync. **Example prompt:** ```text Rename dim_users to dim_customers. Update the model file, its YAML, every downstream ref(), the tests, the documentation, and any exposures that point to it. ``` **What dbt Wizard does:** 1. Renames `dim_users.sql` to `dim_customers.sql` 2. Updates the model name in the corresponding YAML 3. Searches the project for `ref('dim_users')` and rewrites each occurrence 4. Updates any tests, docs blocks, and exposures that reference the old name 5. Compiles the project to confirm every reference resolves before showing you the diff 6. Flags anything it couldn't update automatically (for example, raw SQL strings or external dashboards) **Tips:** * Multi-file changes are coordinated as a single diff: review them together rather than file by file * For column type or schema changes, name the new type explicitly: "Change `order_amount` in `fct_orders` from `numeric` to `decimal(18,2)` and update downstream models and tests" #### Validate before shipping For changes where correctness matters more than speed, ask dbt Wizard to assess impact and validate the result against your project. **Example prompt:** ```text Add not_null and unique tests to the primary key of dim_customers. Use heavy validation, investigate any failures, and summarize skipped checks. ``` **What dbt Wizard does:** 1. Generates the YAML for the new tests 2. Assesses the affected resources and proposes a validation plan 3. Runs the approved compile, build, test, and comparison steps for the selected validation level 4. Reports failures, differences, unresolved risks, and skipped checks 5. Shows the proposed changes for you to review In dbt Wizard CLI, choose light, medium, heavy, or skipped validation based on the risk and cost of the change. Follow the [validation workflow](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) for the checks included at each level. **Tips:** * State the business behavior that must remain true, not only the commands to run #### Add a semantic model You have a mart model and want to expose it via the dbt Semantic Layer. **Example prompt:** ```text Create a semantic model for fct_orders. Include a revenue metric that sums order_total, a count of orders, and time dimensions on order_date at day, week, and month granularity. ``` **What dbt Wizard does:** 1. Reads `fct_orders.sql` and its YAML to understand available columns 2. Determines the dbt version and selects the compatible Semantic Layer YAML structure 3. Proposes entities, dimensions, and metrics based on the model grain and business request 4. Adds the definitions you approve and validates their references **Tips:** * If you're unsure what entities to use, ask first: "What would be good entities for a semantic model on fct\_orders?" * dbt Wizard follows the [dbt Semantic Layer documentation](https://docs.getdbt.com/docs/build/semantic-models.md): you can ask it to explain any generated field For version-specific examples and validation steps, refer to [Build Semantic Layer definitions with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-7-semantic-layer.md). #### Related docs * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * [Understand a dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-2-understand-project.md) * [Validate dbt changes](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [Add data-informed tests](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-4-data-informed-tests.md) * [Debug a failed job](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-5-debug-failed-job.md) * [Build Semantic Layer definitions](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-7-semantic-layer.md) * [dbt Wizard overview](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md): same agent, in dbt platform * [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md): more prompt patterns for the dbt Wizard in Studio IDE (many apply to the CLI too) --- ### Deploy dbt dbt platform Use the dbt platform's capabilities to seamlessly run a dbt job in production or staging environments. Rather than run dbt commands manually from the command line, you can leverage the [dbt platform's in-app scheduling](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to automate how and when you execute dbt. The dbt platform offers the easiest and most reliable way to run your dbt project in production. Effortlessly promote high quality code from development to production and build fresh data assets that your business intelligence tools (like [dbt Charts](https://dbtcharts.com/)) and end users query to make business decisions. Deploying with dbt lets you: * Keep production data fresh on a timely basis * Ensure CI and production pipelines are efficient * Identify the root cause of failures in deployment environments * Maintain high-quality code and data in production * Gain visibility into the [health](https://docs.getdbt.com/docs/explore/data-tile.md) of deployment jobs, models, and tests * Uses [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to write [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) in your data platform for reliable and fast metric reporting * [Visualize](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and [orchestrate](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) downstream exposures to understand how models are used in downstream tools and proactively refresh the underlying data sources during scheduled dbt jobs. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Use [dbt's Git repository caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) to protect against third-party outages and improve job run reliability. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Use [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) to upload dbt artifacts into the dbt platform for central visibility, cross-project referencing, and easier collaboration. [Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Before continuing, make sure you understand dbt's approach to [deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md). Learn how to use dbt's features to help your team ship timely and quality production data more easily. #### Deploy with dbt [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-scheduler.md) ###### [Job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) [The job scheduler is the backbone of running jobs in the dbt platform, bringing power and simplicity to building data pipelines in both continuous integration and production environments.](https://docs.getdbt.com/docs/deploy/job-scheduler.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) ###### [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) [Create and schedule jobs for the job scheduler to run.](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) <br /> <br /> [Runs on a schedule, by API, or after another job completes.](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) ###### [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) [Intelligently determines which models to build by detecting changes in code or data at each job run.](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) ###### [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [Set up CI checks so you can build and test any modified code in a staging environment when you open PRs and push new commits to your dbt repository.](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) ###### [Continuous deployment](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) [Set up merge jobs to ensure the latest code changes are always in production when pull requests are merged to your Git repository.](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-commands.md) ###### [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) [Configure which dbt commands to execute when running a dbt job.](https://docs.getdbt.com/docs/deploy/job-commands.md) <br /> #### Monitor jobs and alerts [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) ###### [Visualize and orchestrate exposures](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) [Learn how to use dbt to automatically generate downstream exposures from dashboards and proactively refresh the underlying data sources during scheduled dbt jobs.](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/artifacts.md) ###### [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) [dbt generates and saves artifacts for your project, which it uses to power features like creating docs for your project and reporting the freshness of your sources.](https://docs.getdbt.com/docs/deploy/artifacts.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-notifications.md) ###### [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) [Receive email or Slack channel notifications when a job run succeeds, fails, or is canceled so you can respond quickly and begin remediation if necessary.](https://docs.getdbt.com/docs/deploy/job-notifications.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/model-notifications.md) ###### [Model notifications](https://docs.getdbt.com/docs/deploy/model-notifications.md) [Receive email notifications in real time about issues encountered by your models and tests while a job is running.](https://docs.getdbt.com/docs/deploy/model-notifications.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/run-visibility.md) ###### [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) [View the history of your runs and the model timing dashboard to help identify where improvements can be made to the scheduled jobs.](https://docs.getdbt.com/docs/deploy/run-visibility.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/retry-jobs.md) ###### [Retry jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) [Rerun your errored jobs from start or the failure point.](https://docs.getdbt.com/docs/deploy/retry-jobs.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/source-freshness.md) ###### [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) [Enable snapshots to capture the freshness of your data sources and configure how frequent these snapshots should be taken. This can help you determine whether your source data freshness is meeting your SLAs.](https://docs.getdbt.com/docs/deploy/source-freshness.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/webhooks.md) ###### [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) [Create outbound webhooks to send events about your dbt jobs' statuses to other systems in your organization.](https://docs.getdbt.com/docs/deploy/webhooks.md) <br /> #### Hybrid projects [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) ###### [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) [Use Hybrid projects to upload dbt artifacts into the dbt platform for central visibility, cross-project referencing, and easier collaboration.](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) <br /> #### Related docs * [Use exports to materialize saved queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) * [Integrate with other orchestration tools](https://docs.getdbt.com/docs/deploy/deployment-tools.md) --- ### Deploy jobs dbt platform You can use deploy jobs to build production data assets. Deploy jobs make it easy to run dbt commands against a project in your cloud data platform, triggered either by schedule or events. Each job run in dbt will have an entry in the job's run history and a detailed run overview, which provides you with: * Job trigger type * Commit SHA * Environment name * Sources and documentation info, if applicable * Job run details, including run timing, [model timing data](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab), and [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * Detailed run steps with logs and their run step statuses. For dbt v2 runs, you can also download OpenTelemetry logs from individual steps. Refer to [Downloading logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs). You can create a deploy job and configure it to run on [scheduled days and times](#schedule-days), enter a [custom cron schedule](#cron-schedule), or [trigger the job after another job completes](#trigger-on-job-completion). #### Prerequisites * You must have a [dbt account](https://www.getdbt.com/signup/) and [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * For the [Trigger on job completion](#trigger-on-job-completion) feature, your dbt account must be on the [Starter or an Enterprise-tier](https://www.getdbt.com/pricing/) plan. * You must have a dbt project connected to a [data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * You must have [access permission](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to view, create, modify, or run jobs. * You must set up a [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md). #### Create and schedule jobs info dbt uses [Coordinated Universal Time](https://en.wikipedia.org/wiki/Coordinated_Universal_Time) (UTC) for all jobs, including those configured with cron. It does not adjust for your local timezone or daylight saving time. For example: * 0 means 12am (midnight) UTC * 12 means 12pm (afternoon) UTC * 23 means 11pm UTC 1. On your deployment environment page, click **Create job** > **Deploy job** to create a new deploy job. 2. Options in the **Job settings** section: * **Job name** — Specify the name for the deploy job. For example, `Daily build`. * (Optional) **Description** — Provide a description of what the job does (for example, what the job consumes and what the job produces). * **Environment** — By default, it’s set to the deployment environment you created the deploy job from. 3. Options in the **Execution settings** section: * [**Commands**](https://docs.getdbt.com/docs/deploy/job-commands.md#built-in-commands) — By default, it includes the `dbt build` command. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when the job runs. During a job run, [built-in commands](https://docs.getdbt.com/docs/deploy/job-commands.md#built-in-commands) are "chained" together and if one run step fails, the entire job fails with an "Error" status. - [**Run source freshness**](https://docs.getdbt.com/docs/deploy/source-freshness.md) — Enable this option to invoke the `dbt source freshness` command before running the deploy job. If the step fails, the job can succeed if subsequent steps pass. Refer to [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) for more details. - [**dbt State**](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) — \[dbt State] reuses nodes when their logic and data haven’t changed, avoiding unnecessary rebuilds. Select **On**, **Off**, or **Inherited from environment** to use the environment’s setting. This option appears only when dbt State is [enabled](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) on your account. Learn more about [enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md#enabling-dbt-state-on-individual-jobs). 4. Options in the **Triggers** section: * **Run on schedule** — Run the deploy job on a set schedule. * **Timing** — Specify whether to [schedule](#schedule-days) the deploy job using **Intervals** that run the job every specified number of hours, **Specific hours** that run the job at specific times of day, or **Cron schedule** that run the job specified using [cron syntax](#cron-schedule). * **Days of the week** — By default, it’s set to every day when **Intervals** or **Specific hours** is chosen for **Timing**. Using `state:modified` on a scheduled job Using a [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state) selector on a scheduled job can result in the job completing successfully with zero models built when no changes are detected since the last deferred run. Refer to [Scheduled jobs and state:modified](#scheduled-jobs-and-statemodified) for details and recommendations. * **Run when another job finishes** — Run the deploy job when another *upstream* deploy [job completes](#trigger-on-job-completion). * **Project** — Specify the parent project that has that upstream deploy job. * **Job** — Specify the upstream deploy job. * **Completes on** — Select the job run status(es) that will [enqueue](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) the deploy job. 5. (Optional) Options in the **Advanced settings** section: * **Environment variables** — Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when the deploy job runs. * **Target name** — Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably. * **Run timeout** — Cancel the deploy job if the run time exceeds the timeout value. * **Compare changes against** — By default, it’s set to **No deferral**. Select either **Environment** or **This Job** to let dbt know what it should compare the changes against. info Older versions of dbt only allow you to defer to a specific job instead of an environment. Deferral to a job compares state against the project code that was run in the deferred job's last successful run. While deferral to an environment is more efficient as dbt will compare against the project representation (which is stored in the `manifest.json`) of the last successful deploy job run that executed in the deferred environment. By considering *all* deploy jobs that run in the deferred environment, dbt will get a more accurate, latest project representation state. * **dbt version** — By default, it’s set to inherit the [dbt version](https://docs.getdbt.com/docs/dbt-versions.md) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior. * **Threads** — By default, it’s set to 4 [threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). Increase the thread count to increase model execution concurrency. [![Example of Advanced Settings on the Deploy Job page](/img/docs/dbt-platform/using-dbt-platform/deploy-job-adv-settings.png?v=2 "Example of Advanced Settings on the Deploy Job page")](#)Example of Advanced Settings on the Deploy Job page ##### Schedule days To set your job's schedule, use the **Run on schedule** option to choose specific days of the week, and select customized hours or intervals. Under **Timing**, you can either use regular intervals for jobs that need to run frequently throughout the day or customizable hours for jobs that need to run at specific times: * **Intervals** — Use this option to set how often your job runs, in hours. For example, if you choose **Every 2 hours**, the job will run every 2 hours from midnight UTC. This doesn't mean that it will run at exactly midnight UTC. However, subsequent runs will always be run with the same amount of time between them. For example, if the previous scheduled pipeline ran at 00:04 UTC, the next run will be at 02:04 UTC. This option is useful if you need to run jobs multiple times per day at regular intervals. * **Specific hours** — Use this option to set specific times when your job should run. You can enter a comma-separated list of hours (in UTC) when you want the job to run. For example, if you set it to `0,12,23,` the job will run at midnight, noon, and 11 PM UTC. Job runs will always be consistent between both hours and days, so if your job runs at 00:05, 12:05, and 23:05 UTC, it will run at these same hours each day. This option is useful if you want your jobs to run at specific times of day and don't need them to run more frequently than once a day. ##### Cron schedule To fully customize the scheduling of your job, choose the **Cron schedule** option and use cron syntax. With this syntax, you can specify the minute, hour, day of the month, month, and day of the week, allowing you to set up complex schedules like running a job on the first Monday of each month. **Note:** Cron schedules in dbt use UTC and don't convert to your local timezone or adjust for daylight saving time. **Cron frequency** To enhance performance, job scheduling frequencies vary by dbt plan: * Developer plans: dbt sets a minimum interval of every 10 minutes for scheduling jobs. This means scheduling jobs to run more frequently, or at less than 10 minute intervals, is not supported. * Starter, Enterprise, and Enterprise+ plans: No restrictions on job execution frequency. **Examples** Use tools such as [crontab.guru](https://crontab.guru/) to generate the correct cron syntax. This tool allows you to input cron snippets and return their plain English translations. The dbt job scheduler supports using `L` to schedule jobs on the last day of the month. Examples of cron job schedules: * `0 * * * *`: Every hour, at minute 0. * `*/5 * * * *`: Every 5 minutes. (Not available on Developer plans) * `5 4 * * *`: At exactly 4:05 AM UTC. * `30 */4 * * *`: At minute 30 past every 4th hour (such as 4:30 AM, 8:30 AM, 12:30 PM, and so on, all UTC). * `0 0 */2 * *`: At 12:00 AM (midnight) UTC every other day. * `0 0 * * 1`: At midnight UTC every Monday. * `0 0 L * *`: At 12:00 AM (midnight), on the last day of the month. * `0 0 L 1,2,3,4,5,6,8,9,10,11,12 *`: At 12:00 AM, on the last day of the month, only in January, February, March, April, May, June, August, September, October, November, and December. * `0 0 L 7 *`: At 12:00 AM, on the last day of the month, only in July. * `0 0 L * FRI,SAT`: At 12:00 AM, on the last day of the month, and on Friday and Saturday. * `0 12 L * *`: At 12:00 PM (afternoon), on the last day of the month. * `0 7 L * 5`: At 07:00 AM, on the last day of the month, and on Friday. * `30 14 L * *`: At 02:30 PM, on the last day of the month. * `0 4 * * MON#1`: At 4:00 AM on the first Monday of every month. ##### Scheduled jobs and `state:modified` [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state) and `state:modified+` only detect changes to your project's code, configuration, or manifest-relevant metadata — not changes to the data itself, like new rows landing in a source table. When dbt detects no code or configuration changes since the deferred manifest was last produced, the job succeeds without building any models. This is expected behavior. If your job needs to build models on every scheduled run regardless of code changes, remove the `state:modified` selector from that job. Reserve it for [CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md) or [merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md), where the intent is to build only what changed in a given pull request or merge. ##### Trigger on job completion [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") To *chain* deploy jobs together: 1. In the **Triggers** section, enable the **Run when another job finishes** option. 2. Select the project that has the deploy job you want to run after completion. 3. Specify the upstream (parent) job that, when completed, will trigger your job. * You can also use the [Create Job API](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) to do this. 4. In the **Completes on** option, select the job run status(es) that will [enqueue](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) the deploy job. [![Example of Trigger on job completion on the Deploy job page](/img/docs/deploy/deploy-job-completion.png?v=2 "Example of Trigger on job completion on the Deploy job page")](#)Example of Trigger on job completion on the Deploy job page 5. You can set up a configuration where an upstream job triggers multiple downstream (child) jobs and jobs in other projects. You must have proper [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#project-role-permissions) to the project and job to configure the trigger. If another job triggers your job to run, you can find a link to the upstream job in the [run details section](https://docs.getdbt.com/docs/deploy/run-visibility.md#job-run-details). #### Delete a job To delete a job or multiple jobs in dbt: 1. Click **Deploy** on the navigation header. 2. Click **Jobs** and select the job you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete job** to delete the job. <br /> [![Delete a job](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-job.png?v=2 "Delete a job")](#)Delete a job 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. 6. Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Job monitoring On the **Environments** page, there are two sections that provide an overview of the jobs for that environment: * **In progress** — Lists the currently in progress jobs with information on when the run started * **Top jobs by models built** — Ranks jobs by the number of models built over a specific time [![In progress jobs and Top jobs by models built](/img/docs/deploy/in-progress-top-jobs.png?v=2 "In progress jobs and Top jobs by models built")](#)In progress jobs and Top jobs by models built #### Job settings history You can view historical job settings changes over the last 90 days. To view the change history: 1. Navigate to **Orchestration** from the main menu and click **Jobs**. 2. Click a **job name**. 3. Click **Settings**. 4. Click **History**. [![Example of the job settings history.](/img/docs/deploy/job-history.png?v=2 "Example of the job settings history.")](#)Example of the job settings history. #### Related docs * [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) * [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) --- ### Deploy your metrics dbt platform | Starter, Enterprise, Enterprise+ This section explains how you can perform a job run in your deployment environment in dbt to materialize and deploy your metrics. Currently, the deployment environment is only supported. 1. Once you’ve [defined your semantic models and metrics](https://docs.getdbt.com/guides/sl-qs.md?step=6), commit and merge your metric changes in your dbt project. 2. In dbt, create a new [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) or use an existing environment on dbt 1.6 or higher. * Note: Deployment environment is currently supported (*development experience coming soon*) 3. To create a new environment, navigate to **Orchestration** in the navigation menu, select **Environments**, and then select **Create new environment**. 4. Fill in your deployment credentials with the username and password for your data platform. You can name the schema anything you want. Click **Save** to create your new production environment. 5. [Create a new deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) that runs in the environment you just created. Go back to the **Orchestration** menu, select **Jobs**, select **Create job**, and click **Deploy job**. 6. Set the job to run a `dbt parse` job to parse your projects and generate a [`semantic_manifest.json` artifact](https://docs.getdbt.com/reference/artifacts/sl-manifest.md) file(Applies to dbt v1.12 and later) and an [`osi_document.json` artifact](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). Although running `dbt build` isn't required, you can choose to do so if needed. note If you are on dbt v2, add the `dbt docs generate` command to your job to successfully deploy your metrics. 7. Run the job by clicking the **Run now** button. Monitor the job's progress in real-time through the **Run summary** tab. Once the job completes successfully, your dbt project, including the generated documentation, will be fully deployed and available for use in your production environment. If any issues arise, review the logs to diagnose and address any errors. What’s happening internally? * Merging the code into your main branch allows dbt to pull those changes and build the definition in the manifest produced by the run. <br /> * Re-running the job in the deployment environment helps materialize the models, which the metrics depend on, in the data platform. It also makes sure that the manifest is up to date. <br /> * The Semantic Layer APIs pull in the most recent manifest and enables your integration to extract metadata from it. #### Next steps After you've executed a job and deployed your Semantic Layer: * [Set up your Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) in dbt. * Discover the [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), such as Tableau, Google Sheets, Microsoft Excel, and more. * Start querying your metrics with the [API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata). #### Related docs * [Optimize querying performance](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) using declarative caching. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. * If you haven't already, learn how to [build your metrics and semantic models](https://docs.getdbt.com/docs/build/build-metrics-intro.md) in your development tool of choice. --- ### Deployment environments dbt platform Deployment environments in dbt are crucial for deploying dbt jobs in production and using features or integrations that depend on dbt metadata or results. To execute dbt, environments determine the settings used during job runs, including: * The version of dbt that will be used to run your project * The warehouse connection information (including the target database/schema settings) * The [connection profile](https://docs.getdbt.com/docs/platform/about-profiles.md) (the credentials dbt uses to connect) * The version of your code to execute * Whether [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is enabled for the environment, which jobs in the environment can inherit. For more information, refer to [Enabling dbt State on environments and jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md). A dbt project can have multiple deployment environments, providing you the flexibility and customization to tailor the execution of dbt jobs. You can use deployment environments to [create and schedule jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs), [enable continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md), or more based on your specific needs or requirements. Learn how to manage dbt environments To learn different approaches to managing dbt environments and recommendations for your organization's unique needs, read [dbt environment best practices](https://docs.getdbt.com/guides/set-up-ci.md). Learn more about development vs. deployment environments in [dbt Environments](https://docs.getdbt.com/docs/dbt-platform-environments.md). There are three types of deployment environments: * **Production**: Environment for transforming data and building pipelines for production use. * **Staging**: Environment for working with production tools while limiting access to production data. * **General**: General use environment for deployment development. We highly recommend using the `Production` environment type for the final, source of truth deployment data. There can be only one environment marked for final production workflows and we don't recommend using a `General` environment for this purpose. #### Create a deployment environment To create a new dbt deployment environment, navigate to **Orchestration** > **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment. [![Navigate to Orchestration > Environments to create a deployment environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/create-deploy-env.png?v=2 "Navigate to Orchestration > Environments to create a deployment environment")](#)Navigate to Orchestration > Environments to create a deployment environment ##### Set as production environment In dbt, each project can have one designated deployment environment, which serves as its production environment. This production environment is *essential* for using features like Catalog and cross-project references. It acts as the source of truth for the project's production state in dbt. [![Set your production environment as the default environment in your Environment Settings](/img/docs/dbt-platform/using-dbt-platform/prod-settings-1.png?v=2 "Set your production environment as the default environment in your Environment Settings")](#)Set your production environment as the default environment in your Environment Settings ##### Semantic Layer For customers using the Semantic Layer, the next section of environment settings is the Semantic Layer configurations. [The Semantic Layer setup guide](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) has the most up-to-date setup instructions. You can also leverage the dbt Job scheduler to [validate your semantic nodes in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. ##### dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) reduces unnecessary node rebuilds by reusing nodes when neither the logic nor the data has changed. You can enable it on a deployment environment so that jobs in the environment can inherit the setting. This option is only visible if dbt State is enabled on your account. For steps, refer to [Enabling dbt State on environments and jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md). #### Staging environment Use a staging environment to grant developers access to deployment workflows and tools while controlling access to production data. Staging environments enable you to achieve more granular control over permissions, data warehouse connections, and data isolation — within the purview of a single project in dbt. ##### Git workflow You can approach this in a couple of ways, but the most straightforward is configuring staging with a long-living branch (for example, `staging`) similar to, but separate from the primary branch (for example, `main`). In this scenario, the workflows would ideally move upstream from the Development environment -> Staging environment -> Production environment with developer branches feeding into the `staging` branch, then ultimately merging into `main`. In many cases, the `main` and `staging` branches will be identical after a merge and remain until the next batch of changes from the `development` branches are ready to be elevated. We recommend setting branch protection rules on `staging` similar to `main`. Some customers prefer to connect Development and Staging to their `main` branch and then cut release branches on a regular cadence (daily or weekly), which feeds into Production. ##### Why use a staging environment These are the primary motivations for using a staging environment: 1. An additional validation layer before changes are deployed into production. You can deploy, test, and explore your dbt models in staging. 2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments. 3. Provide developers with the ability to create, edit, and trigger ad hoc jobs in the staging environment, while keeping the production environment locked down using [environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md). **Conditional configuration of sources** enables you to point to "prod" or "non-prod" source data, depending on the environment you're running in. For example, this source will point to `<DATABASE>.sensitive_source.table_with_pii`, where `<DATABASE>` is dynamically resolved based on an environment variable. models/sources.yml ```yaml sources: - name: sensitive_source database: "{{ env_var('SENSITIVE_SOURCE_DATABASE') }}" tables: - name: table_with_pii ``` There is exactly one source (`sensitive_source`), and all downstream dbt models select from it as `{{ source('sensitive_source', 'table_with_pii') }}`. The code in your project and the shape of the DAG remain consistent across environments. By setting it up in this way, rather than duplicating sources, you get some important benefits. **Cross-project references in dbt Mesh:** Let's say you have `Project B` downstream of `Project A` with cross-project refs configured in the models. When developers work in the IDE for `Project B`, cross-project refs will resolve to the staging environment of `Project A`, rather than production. You'll get the same results with those refs when jobs are run in the staging environment. Only the production environment will reference the production data, keeping the data and access isolated without needing separate projects. **Faster development enabled by deferral:** If `Project B` also has a staging deployment, then references to unbuilt upstream models(Applies to dbt v1.11 and later) and [user-defined functions (UDFs)](https://docs.getdbt.com/docs/build/udfs.md) within `Project B` will resolve to that environment using [deferral](https://docs.getdbt.com/docs/platform/about-defer.md), rather than resolving to the models(Applies to dbt v1.11 and later) and functions in production. This saves developers time and warehouse spend, while preserving clear separation of environments. Finally, the staging environment has its own view in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), giving you a full view of your prod and pre-prod data. ##### Create a Staging environment [![Create a staging environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/create-staging-environment.png?v=2 "Create a staging environment")](#)Create a staging environment Follow the steps outlined in [connection profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) to complete the remainder of the environment setup. We recommend that the data warehouse credentials be for a dedicated user or service principal. #### Deployment connection A deployment environment needs two settings to run jobs: | Setting | Scope | Controls | | ------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------- | | **Deployment connection** | Environment-level | Where dbt builds objects in your warehouse (database, schema, and warehouse) | | **Connection profile** | Project-level (assigned to the environment) | How dbt authenticates, and the credentials it uses to connect | You need to complete both settings as a deployment connection on its own isn't enough for jobs to run. You also need a [connection profile](#connection-profiles) assigned to the environment. Deployment connections Deployment connections are created and managed at the account-level for dbt accounts and assigned to an environment. To change warehouse type, we recommend creating a new environment. Each project can have multiple connections (Snowflake account, Redshift host, Bigquery project, Databricks host, and so on.) of the same warehouse type. Some details of that connection (databases/schemas/and so on.) can be overridden within this section of the dbt environment settings. This section determines the exact location in your warehouse dbt should target when building warehouse objects! This section will look a bit different depending on your warehouse provider. For all warehouses, set [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the environment's connection profile to override missing or inactive (grayed-out) settings. ##### Postgres This section will not appear if you are using Postgres, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the environment's connection profile to override these values. ##### Redshift This section will not appear if you are using Redshift, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the environment's connection profile to override these values. ##### Snowflake [![Snowflake Deployment Connection Settings](/img/docs/collaborate/snowflake-deploy-env-deploy-connection.png?v=2 "Snowflake Deployment Connection Settings")](#)Snowflake Deployment Connection Settings ###### Editable fields * **Role**: Snowflake role * **Database**: Target database * **Warehouse**: Snowflake warehouse ##### Bigquery This section will not appear if you are using Bigquery, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the environment's connection profile to override these values. ##### Spark This section will not appear if you are using Spark, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the environment's connection profile to override these values. ##### Databricks [![Databricks Deployment Connection Settings](/img/docs/collaborate/databricks-deploy-env-deploy-connection.png?v=2 "Databricks Deployment Connection Settings")](#)Databricks Deployment Connection Settings ###### Editable fields * **Catalog** (optional): [Unity Catalog namespace](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) #### Connection profiles The deployment connection sets *where* dbt builds. The connection profile sets *how* dbt authenticates. Deployment credentials are managed through connection profiles, which are created at the project level and assigned to deployment environments. Profiles define the credentials and attributes dbt uses to connect to your warehouse. To configure credentials for this environment, refer to [About dbt platform profiles](https://docs.getdbt.com/docs/platform/about-profiles.md). Jobs need both a deployment connection and a profile assigned to the environment. You can configure any [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on that profile. #### Delete an environment Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. Follow these steps to delete an environment in dbt: 1. Navigate to **Orchestration** > **Environments**. 2. Select the environment you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete** to delete the environment. [![Delete an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-environment.png?v=2 "Delete an environment")](#)Delete an environment 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. 6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Related docs * [dbt environment best practices](https://docs.getdbt.com/guides/set-up-ci.md) * [About dbt platform profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Delete a job or environment in dbt](https://docs.getdbt.com/faqs/Environments/delete-environment-job.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) --- ### Derived metrics In MetricFlow, derived metrics are metrics created by defining an expression using other metrics. They enable you to perform calculations with existing metrics. This is helpful for combining metrics and doing math functions on aggregated columns, like creating a profit metric. The parameters, description, and type for derived metrics are: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ------ | | `name` | The name of the metric. | Required | String | | `description` | A human-readable summary of the metric. | Optional | String | | `type` | The metric type (`simple`, `cumulative`, `ratio`, `derived`, or `conversion`). | Required | String | | `label` | Display name for downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `expr` | The expression that combines other metrics. Validation warns if it is missing or references undefined metrics. | Required | String | | `input_metrics` | Defines aliases, filters, or offsets for metrics referenced in the expression. Needed only when you customize those attributes. | Optional | List | | `input_metrics.name` | The name of the referenced metric defined elsewhere in the project. | Required when `metric_aliases` provided | String | | `input_metrics.alias` | Alternate name you can reference in `expr`. | Optional | String | | `input_metrics.filter` | Filter to apply to the referenced metric. | Optional | String | | `input_metrics.offset_window` | Offset applied to the referenced metric (for example, `1 week`). Allowed only for derived metrics. | Optional | String | The following displays the complete specification for derived metrics, along with an example. (Applies to dbt v1.12 and later) models/model.yml ```yaml metrics: - name: my_derived_metric description: cool derived metric # Optional label: my derived metric label # Optional type: derived # Required expr: my_simple_metric - my_simple_metric_a_week_ago # Required for derived input_metrics: # Required for derived if using aliases / filters / offset_window for portions of the expression - name: my_simple_metric alias: my_simple_metric_a_week_ago filter: "{{ Dimension('my_primary_entity__my_categorical_dimension_column') }} > 10" offset_window: 1 week # Allowed for derived metrics ``` For advanced data modeling, you can use `fill_nulls_with` and `join_to_timespine` to [set null metric values to zero](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md), ensuring numeric values for every data row. #### Derived metrics example (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: fct_orders semantic_model: enabled: true name: order ... rest of config ... metrics: - name: order_gross_profit description: "Gross profit from each order." label: Order gross profit type: derived expr: revenue - cost input_metrics: - name: order_total alias: revenue - name: order_cost alias: cost - name: food_order_gross_profit label: Food order gross profit description: "The gross profit for each food order." type: derived expr: revenue - cost input_metrics: - name: order_total alias: revenue filter: | {{ Dimension('order__is_food_order') }} = True - name: order_cost alias: cost filter: | {{ Dimension('order__is_food_order') }} = True - name: order_total_growth_mom description: "Percentage growth of orders total compared to 1 month ago" label: Order total growth % M/M type: derived expr: (order_total - order_total_prev_month) * 100 / order_total_prev_month input_metrics: - name: order_total - name: order_total alias: order_total_prev_month offset_window: 1 month ``` #### Derived metric offset To perform calculations using a metric's value from a previous time period, you can add an offset parameter to a derived metric. For example, if you want to calculate period-over-period growth or track user retention, you can use this metric offset. **Note:** You must include the [`metric_time` dimension](https://docs.getdbt.com/docs/build/dimensions.md#time) when querying a derived metric with an offset window. The following example displays how you can calculate monthly revenue growth using a 1-month offset window: (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: customers semantic_model: enabled: true name: customers_semantic_model metrics: - name: customer_retention description: Percentage of customers that are active now and those active 1 month ago label: customer_retention type: derived expr: current_active_customers / active_customers_prev_month input_metrics: - name: active_customers alias: current_active_customers - name: active_customers alias: active_customers_prev_month offset_window: 1 month ``` ##### Offset windows and granularity You can query any granularity and offset window combination. The following example queries a metric with a 7-day offset and a monthly grain: (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: customers semantic_model: enabled: true name: customers_semantic_model ... rest of config ... metrics: - name: d7_booking_change description: Difference between bookings now and 7 days ago type: derived label: d7 bookings change expr: current_bookings - bookings_7_days_ago input_metrics: - name: bookings alias: current_bookings - name: bookings offset_window: 7 days alias: bookings_7_days_ago ``` When you run the query `dbt sl query --metrics d7_booking_change --group-by metric_time__month` for the metric, here's how it's calculated. For dbt v1, you can use the `mf query` prefix. 1. Retrieve the raw, unaggregated dataset with the specified (Applies to dbt v1.12 and later) simple metric and dimensions at the smallest level of detail, which is currently 'day'. 2. Then, perform an offset join on the daily dataset, followed by performing a date trunc and aggregation to the requested granularity. For example, to calculate `d7_booking_change` for July 2017: * First, sum up all the booking values for each day in July to calculate the bookings metric. * The following table displays the range of days that make up this monthly aggregation. | | Orders | Metric\_time | | ----- | ------ | ------------------------ | | | 330 | 2017-07-31 | | | 7030 | 2017-07-30 to 2017-07-02 | | | 78 | 2017-07-01 | | Total | 7438 | 2017-07-01 | 3. Calculate July's bookings with a 7-day offset. The following table displays the range of days that make up this monthly aggregation. Note that the month begins 7 days later (offset by 7 days) on 2017-07-24. | | Orders | Metric\_time | | ----- | ------ | ------------------------ | | | 329 | 2017-07-24 | | | 6840 | 2017-07-23 to 2017-06-30 | | | 83 | 2017-06-24 | | Total | 7252 | 2017-07-01 | 4. Lastly, calculate the derived metric and return the final result set: ```bash bookings - bookings_7_days_ago would be compile as 7438 - 7252 = 186. ``` | d7\_booking\_change | metric\_time\_\_month | | ------------------- | --------------------- | | 186 | 2017-07-01 | #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Dimensions (Applies to dbt v1.12 and later) Dimensions represent the non-aggregatable columns in your data set, which are the attributes, features, or characteristics that describe or categorize data. In the context of the Semantic Layer, dimensions are part of a larger structure called a semantic model. They are created along with other elements like [entities](https://docs.getdbt.com/docs/build/entities.md) and [simple metrics](https://docs.getdbt.com/docs/build/simple.md) and used to add more details to your data. In SQL, dimensions are typically included in the `group by` clause of your SQL query. (Applies to dbt v1.12 and later) Dimensions are defined on the column they map to, so they only require a `type`. The `name` defaults to the column name and must be unique within the same semantic model. note You can't use `expr` inside a `dimension` block. To define a dimension from a SQL expression instead of a single column, use [`derived_semantics`](#derived_semantics-in-dimensions). | Parameter | Description | Required | Type | | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ---------- | | `name` | The name of the dimension that will be visible to the user in downstream tools. It can also serve as an alias for derived dimensions<br /><br />Dimension names should be unique within a semantic model, but they can be non-unique across different models as MetricFlow uses [joins](https://docs.getdbt.com/docs/build/join-logic.md) to identify the right dimension. | Optional (defaults to the column name) | String | | `type` | Specifies the type of group created in the semantic model. There are two types:<br /><br />- **Categorical**: Describe attributes or features like geography or sales region.<br />- **Time**: Time-based dimensions like timestamps or dates. | Required | String | | `description` | A clear description of the dimension. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | Dictionary | Refer to the following for the complete specification for dimensions: ```yaml models: - name: Model name # Required semantic_model: enabled: true # bool. Required columns: # Any column can have either an entity or a dimension, but not both - name: my_dimension_column # Required description: Column description # Optional dimension: name: my_dimension # Optional, defaults to column name type: categorical # Required. Accepted values: categorical | time label: Recommended adding a string that defines the display value in downstream tools # Optional description: Same as always # Optional, defaults to the column description if not otherwise specified ``` Refer to the following example to see how dimensions are used in a semantic model: (Applies to dbt v1.12 and later) ```yaml models: - name: fct_transactions semantic_model: enabled: true name: transactions agg_time_dimension: order_date columns: # --- entities --- - name: transaction_column entity: type: primary name: transaction # --- dimensions tied 1:1 to columns --- - name: another_transaction_column granularity: day dimension: type: time name: order_date label: "Date of transaction" description: "A record for every transaction that takes place. Carts are considered multiple transactions for each SKU." - name: type dimension: type: categorical name: type ``` #### `derived_semantics` in `dimensions` Use the `derived_semantics` key in the model YAML entry when you need to derive a dimension definition that is not a direct 1:1 mapping to a single physical column. The `expr` field is required when using `derived_semantics`. ```yaml models: - name: my_model semantic_model: enabled: true ... derived_semantics: dimensions: - name: is_bulk type: categorical expr: "case when quantity > 10 then true else false end" # Required ``` (Applies to dbt v1.12 and later) Dimensions are bound to the primary entity of the semantic model they are defined in. MetricFlow requires that all semantic models have a primary entity. This is to guarantee unique dimension names. If your data source doesn't have a primary entity, you need to assign the entity a name using the `entity` key. It doesn't necessarily have to map to a column in that table and assigning the name doesn't affect query generation. We recommend making these "virtual primary entities" unique across your semantic model. See the following example on how to define a primary entity: ```yaml models: - name: bookings_monthly_source semantic_model: enabled: true agg_time_dimension: ds columns: # Primary entity - name: booking_id entity: type: primary name: booking_id - name: order_date granularity: day dimension: type: time label: "Date" metrics: - name: bookings_monthly type: simple agg: sum ``` If your table doesn't have a physical primary key column, you can still declare a primary entity. Set the model’s grain by declaring a `primary_entity`. The new YAML spec supports this and treats the model as being at that entity’s grain. ```yaml models: - name: model_without_pk semantic_model: enabled: true primary_entity: order # "virtual primary entity" columns: - name: customer_id entity: foreign ``` #### Dimensions types This section further explains the dimension definitions, along with examples. Dimensions have the following types: * [`derived_semantics` in `dimensions`](#derived_semantics-in-dimensions) * [Dimensions types](#dimensions-types) * [Categorical](#categorical) * [Time](#time) * [SCD Type II](#scd-type-ii) * [Basic structure](#basic-structure) * [Semantic model parameters and keys](#semantic-model-parameters-and-keys) * [Implementation](#implementation) * [SCD examples](#scd-examples) #### Categorical Categorical dimensions are used to group metrics by different attributes, features, or characteristics such as product type. (Applies to dbt v1.12 and later) Define a categorical dimension on the column it maps to. For example, the `transaction_type` column becomes a dimension you can group or filter metrics by: ```yaml models: - name: fct_transactions semantic_model: enabled: true columns: - name: transaction_type dimension: type: categorical config: meta: usage: "Group transactions by their type, like refund or purchase." ``` To calculate a categorical dimension from a SQL expression instead of a single column, define it under [`derived_semantics`](#derived_semantics-in-dimensions) and use `expr`. For example, `is_bulk_transaction` applies a case statement to the underlying `quantity` column so users can group or filter by bulk transactions: ```yaml models: - name: fct_transactions semantic_model: enabled: true derived_semantics: dimensions: - name: is_bulk_transaction type: categorical expr: "case when quantity > 10 then true else false end" ``` #### Time (Applies to dbt v1.12 and later) Time dimensions no longer use `type_params`. * For dimensions defined on a column entry, add the column’s `granularity` at the column level. * For derived dimensions, add `granularity` in the dimension configuration. A semantic model’s default aggregation time dimension is set with the `agg_time_dimension` property at the model's top level. A metric can override this with its own `agg_time_dimension`. For more information, see [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). You can use multiple time groups in separate metrics. For example, the `users_created` metric uses `created_at`, and the `users_deleted` metric uses `deleted_at`: ```bash # dbt users dbt sl query --metrics users_created,users_deleted --group-by metric_time__year --order-by metric_time__year # dbt v1 users mf query --metrics users_created,users_deleted --group-by metric_time__year --order-by metric_time__year ``` You can set `is_partition` for time to define specific time spans. ##### is\_partition Use `is_partition: True` to show that a dimension exists over a specific time window. For example, a date-partitioned dimensional table. When you query metrics from different tables, the Semantic Layer uses this parameter to ensure that the correct dimensional values are joined to measures. (Applies to dbt v1.12 and later) ```yaml models: - name: orders semantic_model: enabled: true agg_time_dimension: created_at columns: - name: ts_created granularity: day dimension: type: time name: created_at label: "Date of creation" config: meta: notes: "Only valid for orders from 2022 onward" is_partition: true - name: ts_deleted granularity: day dimension: type: time name: deleted_at label: "Date of deletion" is_partition: true metrics: - name: users_deleted type: simple agg: sum expr: 1 agg_time_dimension: deleted_at - name: users_created type: simple agg: sum expr: 1 ``` ##### time\_granularity (Applies to dbt v1.12 and later) `granularity` specifies the grain of a time dimension. MetricFlow will transform the underlying column to the specified granularity. For example, if you add hourly granularity to a time dimension column, MetricFlow will run a `date_trunc` function to convert the timestamp to hourly. You can easily change the time grain at query time and aggregate it to a coarser grain, for example, from hourly to monthly. However, you can't go from a coarser grain to a finer grain (monthly to hourly). Our supported granularities are: * nanosecond (Snowflake only) * microsecond * millisecond * second * minute * hour * day * week * month * quarter * year Aggregation between metrics with different granularities is possible, with the Semantic Layer returning results at the coarsest granularity by default. For example, when querying two metrics with daily and monthly granularity, the resulting aggregation will be at the monthly level. ```yaml models: - name: your_model_name semantic_model: enabled: true agg_time_dimension: created_at columns: - name: ts_created granularity: hour dimension: type: time name: created_at label: "Date of creation" is_partition: true - name: ts_deleted granularity: day dimension: type: time name: deleted_at label: "Date of deletion" is_partition: true metrics: - name: users_deleted type: simple agg: sum expr: 1 agg_time_dimension: deleted_at - name: users_created type: simple agg: sum expr: 1 ``` (Applies to dbt v1.12 and later) To create a time dimension from a SQL expression instead of a single column, define it under [`derived_semantics`](#derived_semantics-in-dimensions) and add `expr` and `granularity`. For example, `order_month` truncates `ts_created` to the month: ```yaml models: - name: orders semantic_model: enabled: true derived_semantics: dimensions: - name: order_month type: time granularity: month expr: "date_trunc('month', ts_created)" ``` ##### SCD Type II (Applies to dbt v1.12 and later) caution Currently, semantic models with SCD Type II dimensions cannot contain simple metrics. MetricFlow supports joins against dimensions values in a semantic model built on top of a slowly changing dimension (SCD) Type II table. This is useful when you need a particular metric sliced by a group that changes over time, such as the historical trends of sales by a customer's country. ###### Basic structure SCD Type II are groups that change values at a coarser time granularity. SCD Type II tables typically have two time columns that indicate the validity period of a dimension: `valid_from` (or `tier_start`) and `valid_to` (or `tier_end`). This creates a range of valid rows with different dimension values for a metric. MetricFlow associates the metric with the earliest available dimension value within a coarser time window, such as a month. By default, it uses the group valid at the start of this time granularity. MetricFlow supports the following basic structure of an SCD Type II data platform table: | entity\_key | dimensions\_1 | dimensions\_2 | ... | dimensions\_x | valid\_from | valid\_to | | ----------- | ------------- | ------------- | --- | ------------- | ----------- | ---------- | | 123 | value\_a | value\_x | ... | value\_n | 2024-01-01 | 2024-06-30 | | 123 | value\_b | value\_y | ... | value\_m | 2024-07-01 | 2024-12-31 | * `entity_key` (required): A unique identifier for each row in the table, such as a primary key or another unique identifier specific to the entity. * `valid_from` (required): Start date timestamp for when the dimension is valid. Use `validity_params: is_start: True` in the semantic model to specify this. * `valid_to` (required): End date timestamp for when the dimension is valid. Use `validity_params: is_end: True` in the semantic model to specify this. ###### Semantic model parameters and keys When configuring an SCD Type II table in a semantic model, use `validity_params` to specify the start (`valid_from`) and end (`valid_to`) of the validity window for each dimension. * `validity_params`: Parameters that define the validity window. * `is_start: True`: Indicates the start of the validity period. Displayed as `valid_from` in the SCD table. * `is_end: True`: Indicates the end of the validity period. Displayed as `valid_to` in the SCD table. Here’s an example configuration: (Applies to dbt v1.12 and later) ```yaml models: - name: tiers semantic_model: enabled: true agg_time_dimension: tier_start columns: - name: start_date granularity: day dimension: type: time # The type of dimension name: tier_start # The name of the dimension. label: "Start date of tier" # A readable label for the dimension validity_params: # Defines the validity window is_start: true # Indicates the start of the validity period - name: end_date granularity: day dimension: type: time name: tier_end label: "End date of tier" validity_params: is_end: true # Indicates the end of the validity period ``` SCD Type II tables have a specific dimension with a start and end date. To join tables: * Set the additional [entity `type`](https://docs.getdbt.com/docs/build/entities.md#entity-types) parameter to the `natural` key. * Use a `natural` key as an [entity `type`](https://docs.getdbt.com/docs/build/entities.md#entity-types), which means you don't need a `primary` key. * In most instances, SCD tables don't have a logically usable `primary` key because `natural` keys map to multiple rows. ###### Implementation Here are some guidelines to follow when implementing SCD Type II tables: * The SCD table must have `valid_to` and `valid_from` time dimensions, which are logical constructs. * The `valid_from` and `valid_to` properties must be specified exactly once per SCD table configuration. * The `valid_from` and `valid_to` properties shouldn't be used or specified on the same time dimension. * The `valid_from` and `valid_to` time dimensions must cover a non-overlapping period where one row matches each natural key value (meaning they must not overlap and should be distinct). * We recommend defining the underlying dbt model with [dbt snapshots](https://docs.getdbt.com/docs/build/snapshots.md). This supports the SCD Type II table layout and ensures that the table is updated with the latest data. This is an example of SQL code that shows how a sample metric called `num_events` is joined with versioned dimensions data (stored in a table called `scd_dimensions`) using a primary key made up of the `entity_key` and `timestamp` columns. ```sql select metric_time, dimensions_1, sum(1) as num_events from events a left outer join scd_dimensions b on a.entity_key = b.entity_key and a.metric_time >= b.valid_from and (a.metric_time < b. valid_to or b.valid_to is null) group by 1, 2 ``` ###### SCD examples The following are examples of how to use SCD Type II tables in a semantic model:  SCD dimensions for sales tiers and the time length of that tier. This example shows how to create slowly changing dimensions (SCD) using a semantic model. The SCD table contains information about salespersons' tier and the time length of that tier. Suppose you have the underlying SCD table: | sales\_person\_id | tier | start\_date | end\_date | | ----------------- | ---- | ----------- | ---------- | | 111 | 1 | 2019-02-03 | 2020-01-05 | | 111 | 2 | 2020-01-05 | 2048-01-01 | | 222 | 2 | 2020-03-05 | 2048-01-01 | | 333 | 2 | 2020-08-19 | 2021-10-22 | | 333 | 3 | 2021-10-22 | 2048-01-01 | As mentioned earlier, the `validity_params` include two important arguments that specify the columns in the SCD table that mark the start and end dates (or timestamps) for each tier or dimension: * `is_start` * `is_end` Additionally, the entity is tagged as `natural` to differentiate it from a `primary` entity. In a `primary` entity, each entity value has one row. In contrast, a `natural` entity has one row for each combination of entity value and its validity period. (Applies to dbt v1.12 and later) ```yaml models: - name: sales_person_tiers semantic_model: enabled: true agg_time_dimension: tier_start # You can use a virtual primary entity name primary_entity: sales_person columns: - name: start_date granularity: day dimension: type: time name: tier_start label: "Start date of tier" validity_params: is_start: true - name: end_date granularity: day dimension: type: time name: tier_end label: "End date of tier" validity_params: is_end: true - name: tier dimension: type: categorical name: tier - name: sales_person_id entity: type: natural name: sales_person ``` The following code represents a separate semantic model that holds a fact table for `transactions`: (Applies to dbt v1.12 and later) ```yaml models: - name: fct_transactions semantic_model: enabled: true agg_time_dimension: metric_time columns: - name: transaction_id entity: type: primary name: transaction_id - name: customer_id entity: type: foreign name: customer - name: product_id entity: type: foreign name: product - name: sales_person_id entity: type: foreign name: sales_person - name: metric_time granularity: day dimension: type: time name: metric_time label: "Date of transaction" is_partition: true - name: sales_geo dimension: type: categorical name: sales_geo metrics: - name: transactions type: simple agg: sum expr: 1 - name: gross_sales type: simple agg: sum expr: sales_price - name: sales_persons_with_a_sale type: simple agg: count_distinct expr: sales_person_id ``` You can now access the metrics in the `transactions` semantic model organized by the slowly changing dimension of `tier`. In the sales tier example, For instance, if a salesperson was Tier 1 from 2022-03-01 to 2022-03-12, and gets promoted to Tier 2 from 2022-03-12 onwards, all transactions from March would be categorized under Tier 1 since the dimensions value of Tier 1 comes earlier (and is the default starting point), even though the salesperson was promoted to Tier 2 on 2022-03-12.  SCD dimensions with sales tiers and group transactions by month when tiers are missing This example shows how to create slowly changing dimensions (SCD) using a semantic model. The SCD table contains information about salespersons' tier and the time length of that tier. Suppose you have the underlying SCD table: | sales\_person\_id | tier | start\_date | end\_date | | ----------------- | ---- | ----------- | ---------- | | 111 | 1 | 2019-02-03 | 2020-01-05 | | 111 | 2 | 2020-01-05 | 2048-01-01 | | 222 | 2 | 2020-03-05 | 2048-01-01 | | 333 | 2 | 2020-08-19 | 2021-10-22 | | 333 | 3 | 2021-10-22 | 2048-01-01 | In the sales tier example, if sales\_person\_id 456 is Tier 2 from 2022-03-08 onwards, but there is no associated tier level dimension for this person from 2022-03-01 to 2022-03-08, then all transactions associated with sales\_person\_id 456 for the month of March will be grouped under 'NA' since no tier is present prior to Tier 2. The following command or code represents how to return the count of transactions generated by each sales tier per month: ```bash # dbt platform users dbt sl query --metrics transactions --group-by metric_time__month,sales_person__tier --order-by metric_time__month,sales_person__tier # dbt v1 users mf query --metrics transactions --group-by metric_time__month,sales_person__tier --order-by metric_time__month,sales_person__tier ``` --- ### Discover data with Catalog dbt platform | Starter, Enterprise, Enterprise+ With Catalog, you can view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics), their lineage, and [model consumption](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) to gain a better understanding of its latest production state. Use Catalog to navigate and manage your projects within dbt to help you and other data developers, analysts, and consumers discover and leverage your dbt resources. Catalog integrates with the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md), [Orchestrator](https://docs.getdbt.com/docs/deploy/deployments.md), and [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to help you develop or view your dbt resources. #### Prerequisites * You have a dbt account on the [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) or [staging](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-staging-environment) deployment environment for each project you want to explore. * You have at least one successful job run in the deployment environment. Note that [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) do not update Catalog. * You are on the Catalog page. To do this, select **Catalog** from the top-level navigation in dbt. #### Generate metadata Catalog uses the metadata provided by the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) to display the details about [the state of your dbt project](https://docs.getdbt.com/docs/dbt-apis/project-state.md). The metadata that's available depends on the [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) you've designated as *production* or *staging* in your dbt project. #### dbt metadata If you're using a [hybrid project setup](https://docs.getdbt.com/docs/deploy/hybrid-setup.md) and uploading artifacts from dbt v1, make sure to follow the [setup instructions](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#connect-project-in-dbt-cloud) to connect your project in dbt. This enables Catalog to access and display your metadata correctly. * (Applies to dbt v2.0 and later) To ensure all metadata is available in Catalog, run `dbt build` as part of your job in your production or staging environment. Jobs running v2 refresh catalog metadata automatically on every job run, so you don't need a `dbt docs generate` step. * Catalog automatically retrieves the metadata updates after each job run in the production or staging deployment environment so it always has the latest results for your project. This includes deploy and merge jobs. * Note that CI jobs don't update Catalog. This is because they don't reflect the production state and don't provide the necessary metadata updates. * To view a resource and its metadata, you must define the resource in your project and run a job in the production or staging environment. * The resulting metadata depends on the [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) executed by the jobs. ##### When dbt creates model metadata dbt populates a model's metadata in Catalog when both of the following conditions are met: * The model is defined in your dbt project (it exists in the manifest). * The model appears in the `run_results` of a [`dbt build`](https://docs.getdbt.com/reference/commands/build.md), [`dbt run`](https://docs.getdbt.com/reference/commands/run.md), or [`dbt clone`](https://docs.getdbt.com/reference/commands/clone.md) command, regardless of the run's success or failure status. ##### When dbt removes model metadata dbt removes a model's metadata from Catalog in these two cases: * **Model removed from project**: If a model is deleted from your dbt project (and therefore no longer exists in the manifest), its metadata is removed after a subsequent job run in which the model is no longer included. * **Environment inactivity**: If an environment has had no job runs in the past 3 months, all metadata for that environment is purged. To prevent this, schedule jobs to run at least once every 3 months. (Applies to dbt v2.0 and later) | To view in Catalog | You must successfully run | | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | All metadata | [dbt build](https://docs.getdbt.com/reference/commands/build.md) and [dbt source freshness](https://docs.getdbt.com/reference/commands/source.md#freshness) together as part of the same job in the environment | | Model lineage, details, or results | [dbt run](https://docs.getdbt.com/reference/commands/run.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) on a given model within a job in the environment | | Columns and statistics for models, sources, and snapshots | Any [job](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) run in the environment. Jobs running v2 refresh catalog metadata automatically. | | Data test results | [dbt test](https://docs.getdbt.com/reference/commands/test.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment | | Unit test results | [dbt test](https://docs.getdbt.com/reference/commands/test.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment. Unit tests are typically run in development or CI environments, so their results rarely appear in production Catalog. | | Source freshness results | [dbt source freshness](https://docs.getdbt.com/reference/commands/source.md#freshness) within a job in the environment | | Snapshot details | [dbt snapshot](https://docs.getdbt.com/reference/commands/snapshot.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment | | Seed details | [dbt seed](https://docs.getdbt.com/reference/commands/seed.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment | tip If your organization works in both dbt v1 and Cloud, you can unify these workflows by automatically uploading dbt v1 artifacts into dbt Cloud and viewing them in Catalog for a more connected dbt experience. To learn more, visit [hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md). #### Catalog overview [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Catalog introduces the ability to widen your search by including dbt resources (models, seeds, snapshots, sources, exposures, and more) across your entire account. This broadens the results returned and gives you greater insight into all the assets across your dbt projects. Learn more in [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) or in our [video overview](https://www.loom.com/share/ae93b3d241cd439fbe5f98f5e6872113?). Navigate the Catalog overview page to access your project's resources and metadata. The page includes the following sections: * **Search bar** — [Search](#search-resources) for resources in your project by keyword. You can also use filters to refine your search results. * **Sidebar** — Use the left sidebar to browse your project's [resources, file tree, and database](#browse-with-the-sidebar) in the lower section of the sidebar. You can also browse model [performance](https://docs.getdbt.com/docs/explore/model-performance.md) and [project recommendations](https://docs.getdbt.com/docs/explore/project-recommendations.md) depending on your plan. Refer to [Availability by plan](#availability-by-plan) for more info. * Find your project recommendations within your project's landing page.\* * **Lineage graph** — Explore your project's or account's [lineage graph](#project-lineage) to visualize the relationships between resources. * **Latest updates** — View the latest changes or issues related to your project's resources, including the most recent job runs, changed properties, lineage, and issues. * **Marts and public models** — View the [marts](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md#guide-structure-overview) and [public models](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers) in your project. You can also navigate to all public models in your account through this view. * **Model query history** — Use [model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) to track consumption queries on your models for deeper insights. * **Visualize downstream exposures** — [Set up](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and [visualize downstream exposures](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) to automatically expose relevant data models from Tableau to enhance visibility. * **Data health signals** — View the [data-health-signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) for each resource to understand its health and performance. ##### Catalog permissions When using global navigation and searching across your projects, the following permissions apply. * Your project access permissions determine which dbt projects appear in the left-hand menu of the global navigation. * In Catalog searches, we use soft access controls, you'll see all matching resources in search results, with clear indicators for items you don't have access to. ##### Availability by plan Catalog is available on all Starter, Enterprise, and Enterprise+ plans. However, certain features are only available on Enterprise and Enterprise+ plans: | Feature | Starter | Enterprise | Enterprise+ | | ------------------------------------------------------------------------------------------ | ------- | ---------- | ----------- | | dbt lineage & resource browsing | ✅ | ✅ | ✅ | | [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) | ✅ | ✅ | ✅ | | [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) | ✅ | ✅ | ✅ | | [Model performance](https://docs.getdbt.com/docs/explore/model-performance.md) | ❌ | ✅ | ✅ | | [Project recommendations](https://docs.getdbt.com/docs/explore/project-recommendations.md) | ❌ | ✅ | ✅ | | [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) | ❌ | ✅ | ✅ | | [Multi-project lineage](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) | ❌ | ✅ | ✅ | | [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) | ❌ | ✅ | ✅ | | [Downstream exposures](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) | ❌ | ✅ | ✅ | | [Data health tile](https://docs.getdbt.com/docs/explore/data-tile.md) | ❌ | ✅ | ✅ | This table covers Catalog only. Refer to [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) to learn how you can use this feature locally. On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### Explore your project's lineage graph Catalog provides a visualization of your project's DAG that you can interact with. To access the project's full lineage graph, select **Overview** in the left sidebar and click the **Explore Lineage** button on the main (center) section of the page. If you don't see the project lineage graph immediately, click **Render Lineage**. It can take some time for the graph to render depending on the size of your project and your computer's available memory. The graph of very large projects might not render so you can select a subset of nodes by using selectors, instead. The nodes in the lineage graph represent the project's resources and the edges represent the relationships between the nodes. Nodes are color-coded and include iconography according to their resource type. By default, Catalog shows the project's [applied state](https://docs.getdbt.com/docs/dbt-apis/project-state.md#definition-logical-vs-applied-state-of-dbt-nodes) lineage. That is, it shows models that have been successfully built and are available to query, not just the models defined in the project. To explore the lineage graphs of tests and macros, view [their resource details pages](#view-resource-details). By default, Catalog excludes these resources from the full lineage graph unless a search query returns them as results.  How can I interact with the full lineage graph? * Hover over any item in the graph to display the resource's name and type. * Zoom in and out on the graph by mouse-scrolling. * Grab and move the graph and the nodes. * Right-click on a node (context menu) to: * Refocus on the node, including its upstream and downstream nodes * Refocus on the node and its downstream nodes only * Refocus on the node and it upstream nodes only * View the node's [resource details](#view-resource-details) page * Select a resource to highlight its relationship with other resources in your project. A panel opens on the graph's right-hand side that displays a high-level summary of the resource's details. The side panel includes a **General** tab for information like description, materialized type, and other details. In the side panel's upper right corner: * Click the View Resource icon to [view the resource details](#view-resource-details). * Click the [Open in IDE](#open-in-ide) icon to examine the resource using the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). * Click the Copy Link to Page icon to copy the page's link to your clipboard. * Use [selectors](https://docs.getdbt.com/reference/node-selection/methods.md) (in the search bar) to select specific resources or a subset of the DAG. This can help narrow the focus on the resources that interest you. All selectors are available for use, except those requiring a state comparison (result, source status, and state). You can also use the `--exclude` and the `--select` flag (which is optional). Examples: * `resource_type:model [RESOURCE_NAME]` — Returns all models matching the name search * `resource_type:metric,tag:nightly` — Returns metrics with the tag `nightly` * Use [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md) (in the search bar) to select specific resources or a subset of the DAG. This can help narrow the focus on the resources that interest you. Examples: * `+orders` — Returns all the upstream nodes of `orders` * `+dim_customers,resource_type:source` — Returns all sources that are upstream of `dim_customers` * Use [set operators](https://docs.getdbt.com/reference/node-selection/set-operators.md) (in the search bar) to select specific resources or a subset of the DAG. This can help narrow the focus on the resources that interest you. For example: * `+snowplow_sessions +fct_orders` — Use space-delineated arguments for a union operation. Returns resources that are upstream nodes of either `snowplow_sessions` or `fct_orders`. * [View resource details](#view-resource-details) by selecting a node (double-clicking) in the graph. * Click **Lenses** (lower right corner of the graph) to use Catalog [lenses](#lenses) feature. ##### Example of full lineage graph To explore a model in your project's lineage graph in Catalog: 1. Select your project in the left sidebar. 2. Click the **View lineage** button. 3. Click a model to view the description. [![Example of full lineage graph](/img/docs/collaborate/dbt-explorer/example-project-lineage-graph.png?v=2 "Example of full lineage graph")](#)Example of full lineage graph #### Lenses The **Lenses** feature is available from your [project's lineage graph](#project-lineage) (lower right corner). Lenses are like map layers for your DAG. Lenses make it easier to understand your project's contextual metadata at scale, especially to distinguish a particular model or a subset of models. When you apply a lens, tags become visible on the nodes in the lineage graph, indicating the layer value along with coloration based on that value. If you're significantly zoomed out, only the tags and their colors are visible in the graph. Lenses are helpful to analyze a subset of the DAG if you're zoomed in, or to find models/issues from a larger vantage point.  List of available lenses A resource in your project is characterized by resource type, materialization type, or model layer, as well as its latest run or latest test status. Lenses are available for the following metadata: * **Resource type**: Organizes resources by resource type, such as models, tests, seeds, saved query, and [more](https://docs.getdbt.com/docs/build/projects.md). Resource type uses the `resource_type` selector. * **Materialization type**: Identifies the strategy for building the dbt models in your data platform. * **Latest status**: The status from the latest execution of the resource in the current environment. For example, diagnosing a failed DAG region. * **Model layer**: The modeling layer that the model belongs to according to [best practices guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md#guide-structure-overview). For example, discovering marts models to analyze. * **Marts** — A model with the prefix `fct_` or `dim_` or a model that lives in the `/marts/` subdirectory. * **Intermediate** — A model with the prefix `int_`. Or, a model that lives in the `/int/` or `/intermediate/` subdirectory. * **Staging** — A model with the prefix `stg_`. Or, a model that lives in the `/staging/` subdirectory. * **Test status**: The status from the latest execution of the tests that ran again this resource. In the case that a model has multiple tests with different results, the lens reflects the 'worst case' status. * **Consumption query history**: The number of queries against this resource over a given time period. ##### Example of lenses Example of applying the **Materialization type** *lens* on a zoomed-out lineage graph. In the bottom-left corner, open **Lenses** and select **Materialization type**. Each node displays a colored badge for its related materialization type: * Table * View * Incremental * Ephemeral * Materialized View. [![Example of the Materialization type lens](/img/docs/collaborate/dbt-explorer/example-materialization-type.png?v=2 "Example of the Materialization type lens")](#)Example of the Materialization type lens Example of applying the **Test status** *lens* on a lineage graph filtered to models. In the bottom-left corner, open **Lenses** and select **Test status**. Each node displays a colored badge for its latest test status: * Pass * Error * Fail * Warn * Skipped * Reused. [![Example of the Test Status lens](/img/docs/collaborate/dbt-explorer/example-test-status.png?v=2 "Example of the Test Status lens")](#)Example of the Test Status lens #### Keyword search With Catalog, global navigation provides a search experience allowing you to find dbt resources across all your projects, as well as non-dbt resources in Snowflake. You can locate resources in your project by performing a keyword search in the search bar. All resource names, column names, resource descriptions, warehouse relations, and code matching your search criteria will be displayed as a list on the main (center) section of the page. When searching for an exact column name, the results show all relational nodes containing that column in their schemas. If there's a match, a notice in the search result indicates the resource contains the specified column. Also, you can apply filters to further refine your search results.  Search features * **Partial keyword search** — Also referred to as fuzzy search. Catalog uses a "contains" logic to improve your search results. This means you can search for partial terms without knowing the exact root word of your search term. * **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. * **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. * **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. * **SQL keyword search** — Use SQL keywords in your search. For example, the search results `int github users joined` will include matches that contain that specific string of keywords (similar to phrase searching).  Filters side panel The **Filters** side panel becomes available after you perform a keyword search. Use this panel to further refine the results from your keyword search. By default, Catalog searches across all resources in the project. You can filter on: * [Resource type](https://docs.getdbt.com/docs/build/projects.md) (like models, sources, and so on) * [Model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md) (like public, private) * [Model layer](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) (like marts, staging) * [Model materialization](https://docs.getdbt.com/docs/build/materializations.md) (like view, table) * [Tags](https://docs.getdbt.com/reference/resource-configs/tags.md) (supports multi-select) Under the **Models** option, you can filter on model properties (access or materialization type). Also available are **Advanced** options, where you can limit the search results to column name, model code, and more.  Global navigation Catalog builds on the functionality of the old navigation and introduces exciting new capabilities to enhance your experience. For more information, refer to [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md). ##### Example of keyword search Example of results from searching on the keyword `customers` and applying the filters models, description, and code. [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) are visible to the right of the model name in the search results. #### Browse with the sidebar From the sidebar, you can browse your project's resources, its file tree, and the database. * **Resources** tab — All resources in the project organized by type. Select any resource type in the list and all those resources in the project will display as a table in the main section of the page. For a description on the different resource types (like models, metrics, and so on), refer to [About dbt projects](https://docs.getdbt.com/docs/build/projects.md). * [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) are visible to the right of the resource name under the **Health** column. * **File Tree** tab — All resources in the project organized by the file in which they are defined. This mirrors the file tree in your dbt project repository. * **Database** tab — All resources in the project organized by the database and schema in which they are built. This mirrors your data platform's structure that represents the [applied state](https://docs.getdbt.com/docs/dbt-apis/project-state.md) of your project. #### Integrated tool access Users with a [developer license](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#license-based-access-control) or an analyst seat\* can open a resource directly from the Catalog in the Studio IDE to view its model files, in Insights to query it, or in Canvas for visual editing. \* The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. #### View model versions If models in the project are versioned, you can see which [version of the model](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) is being applied — `prerelease`, `latest`, and `old` — in the title of the model's details page and in the model list from the sidebar. #### View resource details You can view the definition and latest run results of any resource in your project. To find a resource and view its details, you can interact with the lineage graph, use search, or browse the Catalog. The details (metadata) available to you depends on the resource's type, its definition, and the [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that run within jobs in the production environment. In the upper right corner of the resource details page, you can: * Click the [Open in Studio IDE](#open-in-ide) icon to examine the resource using the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). * Click the Share icon to copy the page's link to your clipboard.  What details are available for a model? * **Data health signals** — [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) offer a quick, at-a-glance view of data health. These icons indicate whether a model is Healthy, Caution, Degraded, or Unknown. Hover over an icon to view detailed information about the model's health. * **Status bar** (below the page title) — Information on the last time the model ran, whether the run was successful, how the data is materialized, number of rows, and the size of the model. * **General** tab includes: * **Lineage** graph — The model's lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the model. Click the Expand icon in the graph's upper right corner to view the model in full lineage graph mode. * **Description** section — A [description of the model](https://docs.getdbt.com/docs/build/documentation.md#adding-descriptions-to-your-project). * **Recent** section — Information on the last time the model ran, how long it ran for, whether the run was successful, the job ID, and the run ID. * **Tests** section — [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) for the model, including a status indicator for the latest test status. A ✅ denotes a passing test. * **Details** section — Key properties like the model's relation name (for example, how it's represented and how you can query it in the data platform: `database.schema.identifier`); model governance attributes like access, group, and if contracted; and more. * **Relationships** section — The nodes the model **Depends On**, is **Referenced by**, and (if applicable) is **Used by** for projects that have declared the models' project as a dependency. * **Code** tab — The source code and compiled code for the model. * **Columns** tab — The available columns in the model, including [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) defined on columns in your dbt YAML. Column tags appear as clickable badges that filter the lineage view by that tag. This tab also shows test results (if any) that you can select to view the test's details page. A ✅ denotes a passing test. To filter the columns in the resource (including by tag name), use the search bar at the top of the columns view.  What details are available for an exposure? * **Status bar** (below the page title) — Information on the last time the exposure was updated. * **Data health signals** — [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the exposure's health. * **General** tab includes: * **Data health** — The status on data freshness and data quality. * **Status** section — The status on data freshness and data quality. * **Lineage** graph — The exposure's lineage graph. Click the **Expand** icon in the graph's upper right corner to view the exposure in full lineage graph mode. Integrates natively with Tableau and auto-generates downstream lineage. * **Description** section — A description of the exposure. * **Details** section — Details like exposure type, maturity, owner information, and more. * **Relationships** section — The nodes the exposure **Depends On**.  What details are available for a test? * **Status bar** (below the page title) — Information on the last time the test ran, whether the test passed, test name, test target, and column name. Defaults to all if not specified. * **Test Type** (next to the Status bar) — Information on the different test types available: Unit test or Data test. Defaults to all if not specified. When you select a test, the following details are available: * **General** tab includes: * **Lineage** graph — The test's lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the test resource. Click the Expand icon in the graph's upper right corner to view the test in full lineage graph mode. * **Description** section — A description of the test. * **Recent** section — Information on the last time the test ran, how long it ran for, whether the test passed, the job ID, and the run ID. * **Details** section — Details like schema, severity, package, and more. * **Relationships** section — The nodes the test **Depends On**. * **Code** tab — The source code and compiled code for the test. Example of the Tests view:  What details are available for each source table within a source collection? * **Status bar** (below the page title) — Information on the last time the source was updated and the number of tables the source uses. * **Data health signals** — [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the source's health. * **General** tab includes: * **Lineage** graph — The source's lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the source. Click the Expand icon in the graph's upper right corner to view the source in full lineage graph mode. * **Description** section — A description of the source. * **Source freshness** section — Information on whether refreshing the data was successful, the last time the source was loaded, the timestamp of when a run generated data, and the run ID. * **Details** section — Details like database, schema, and more. * **Relationships** section — A table that lists all the sources used with their freshness status, the timestamp of when freshness was last checked, and the timestamp of when the source was last loaded. * **Columns** tab — The available columns in the source, including [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) defined on columns in your dbt YAML. Column tags appear as clickable badges that filter the lineage view by that tag. This tab also shows test results (if any) that you can select to view the test's details page. A ✅ denotes a passing test. ##### Example of model details To view a model's details page in Catalog: 1. Select your project in the sidebar. 2. Click **Models**. 3. Under **Models**, select a model (for example, `customers`). On the **General** tab, you can review the description, latest status, and lineage graph. Select **Full Lineage** to open the full graph view. [![Example of resource details](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "Example of resource details")](#)Example of resource details [![Example of downstream exposure details for Tableau.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "Example of downstream exposure details for Tableau.")](#)Example of downstream exposure details for Tableau. #### Staging environment Catalog supports views for [staging deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment), in addition to the production environment. This gives you a unique view into your pre-production data workflows, with the same tools available in production, while providing an extra layer of scrutiny. You can explore the metadata from your production or staging environment to inform your data development lifecycle. Just [set a single environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) per dbt project as "production" or "staging," and ensure the proper metadata has been generated then you'll be able to view it in Catalog. Refer to [Generating metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details. #### Related content * [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) * [About model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) * Blog on [What is data mesh?](https://www.getdbt.com/blog/what-is-data-mesh-the-definition-and-importance-of-data-mesh) --- ### DuckDB and Apache Iceberg dbt v2 only DuckDB support for `catalogs.yml` requires [dbt v2](https://docs.getdbt.com/docs/introduction.md) with the `use_catalogs_v2` behavior flag enabled. It isn't available in the legacy Python `dbt-duckdb` adapter for dbt v1. dbt\_project.yml ```yaml flags: use_catalogs_v2: true ``` Unlike Snowflake, Databricks, and BigQuery, DuckDB doesn't ship with a single built-in "managed" Iceberg catalog. This means there's no `table_format='iceberg'`-only shortcut for DuckDB — every Iceberg model requires a `catalog_name` that points to an entry in [`catalogs.yml`](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md). dbt supports creating Iceberg tables for two DuckDB materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) #### How DuckDB attaches catalogs When you configure a catalog with a `duckdb` block in `catalogs.yml`, dbt generates and runs the appropriate DuckDB [`ATTACH`](https://duckdb.org/docs/sql/statements/attach.html) statement on your behalf — you don't need to write `ATTACH` SQL yourself. dbt then resolves any model with a matching `catalog_name` to that attached database. #### Iceberg REST catalogs DuckDB can attach to any catalog that implements the Iceberg REST protocol, including self-hosted catalogs (such as [Lakekeeper](https://github.com/lakekeeper/lakekeeper) or [Nessie](https://github.com/projectnessie/nessie)), AWS Glue, and AWS S3 Tables. catalogs.yml ```yaml catalogs: - name: rest_catalog type: iceberg_rest table_format: iceberg config: duckdb: endpoint: "https://my-iceberg-rest.example.com" secret: my_iceberg_secret ``` models/my\_iceberg\_model.sql ```sql {{ config( materialized = 'table', catalog_name = 'rest_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` Run the model with `dbt run -s my_iceberg_model`. Instead of `endpoint`, you can use `endpoint_type: GLUE` or `endpoint_type: S3_TABLES` to attach one of these well-known AWS-managed Iceberg REST endpoints without specifying a URL: catalogs.yml ```yaml catalogs: - name: s3_tables_catalog type: iceberg_rest table_format: iceberg config: duckdb: endpoint_type: S3_TABLES warehouse: "arn:aws:s3tables:us-east-1:123456789012:bucket/example" ``` `endpoint` and `endpoint_type` are mutually exclusive. ##### Writing to S3 Tables To write to S3 Tables, use an explicit `endpoint` with `authorization_type: SIGV4`. In the current v2 preview builds, the `endpoint_type: S3_TABLES` shortcut works for *reading*, but incorrectly defaults to OAuth2 instead of SigV4 when writing. This surfaces as either errors: ```text Invalid Configuration Error: AUTHORIZATION_TYPE is 'oauth2', yet no 'secret' was provided... ``` ```text InvalidArguments: HTTP Error: Failed to retrieve OAuth2 token from (sqlstate: ...) ``` S3 Tables requires SigV4 authentication. Until this is fixed, use the explicit `endpoint` + `authorization_type: SIGV4` config. Refer to [Write to Amazon S3 Tables](#write-to-amazon-s3-tables) for any S3 Tables catalog you write to. #### Write to Amazon S3 Tables To materialize Iceberg models into an S3 Tables bucket, configure your catalog, AWS credentials, and model as follows. ##### Configure your catalog In `catalogs.yml`, define an `iceberg_rest` catalog with an explicit endpoint form together with SigV4 authentication, and set the S3 Tables write-compatibility options: catalogs.yml ```yaml catalogs: - name: s3_tables_catalog type: iceberg_rest table_format: iceberg config: duckdb: endpoint: "https://s3tables.<region>.amazonaws.com/iceberg" warehouse: "arn:aws:s3tables:<region>:<account-id>:bucket/<bucket-name>" authorization_type: SIGV4 secret: s3_tables_secret default_schema: <namespace> stage_create_tables: false # S3 Tables rejects staged CREATE TABLE AS SELECT disable_multi_table_commit: true # S3 Tables has no multi-table transaction commit endpoint purge_requested: true # S3 Tables only allows DROP TABLE with purge enabled ``` Replace the placeholders with your AWS Region, account ID, table bucket name, and namespace. `secret` references an `s3` credential-chain secret defined in `profiles.yml` (refer to [Secrets](#secrets)) — this supplies the SigV4 credentials, not an OAuth2 token. ##### Configure AWS credentials In `profiles.yml`, define an `s3` secret with the same name you used in `catalogs.yml`. The `credential_chain` provider supplies AWS credentials for SigV4 authentication. profiles.yml ```yaml my_profile: target: dev outputs: dev: type: duckdb path: ':memory:' extensions: - iceberg - aws - httpfs secrets: - type: s3 name: s3_tables_secret provider: credential_chain region: <region> ``` ##### Configure your model Set `catalog_name` to your catalog's name and `schema` to your S3 Tables namespace: models/my\_s3\_tables\_model.sql ```sql {{ config( materialized = 'table', catalog_name = 's3_tables_catalog', schema = '<namespace>' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` ##### Match the schema to your namespace Because DuckDB's default schema is `main`, dbt's default schema-naming logic concatenates it with your model's custom schema (for example, `main_my_namespace`), which won't match an existing S3 Tables namespace. Add a project-level override so the schema resolves to your namespace exactly: macros/generate\_schema\_name.sql ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- if custom_schema_name is none -%} {{ target.schema }} {%- else -%} {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` #### Cross-platform Mesh: reading catalogs managed by other platforms Because a single catalog entry in `catalogs.yml` can carry configuration for multiple platforms at once, you can point DuckDB at the same physical catalog that Snowflake or Databricks writes to — enabling [cross-platform Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh.md) without copying data. ##### Snowflake Horizon [Snowflake Horizon](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) is Snowflake's managed Iceberg catalog. Add a `duckdb` block alongside the `snowflake` block to let DuckDB attach to the same catalog: catalogs.yml ```yaml catalogs: - name: horizon_catalog type: horizon table_format: iceberg config: snowflake: external_volume: my_external_volume duckdb: warehouse: horizon_wh endpoint: "https://horizon.example.com/catalog" secret: horizon_secret default_schema: demo ``` ##### Databricks Unity Catalog Similarly, for [Databricks Unity Catalog](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md): catalogs.yml ```yaml catalogs: - name: unity_catalog type: unity table_format: iceberg config: databricks: file_format: delta use_uniform: true duckdb: warehouse: unity_wh endpoint: "https://dbc-example.cloud.databricks.com/api/2.1/unity-catalog/iceberg" default_schema: demo ``` ##### Read-only vs. read-write By default, dbt attaches Horizon and Unity catalogs read-write (`read_only: false`) and applies write-compat `ATTACH` defaults for each (for example, disabling multi-table commits on Unity). Writing to these catalogs from DuckDB requires DuckDB 1.5.4+ and [duckdb-iceberg#1017](https://github.com/duckdb/duckdb-iceberg/issues/1017). If you only need to *read* Iceberg tables that another platform wrote, set `read_only: true`: ```yaml duckdb: warehouse: horizon_wh endpoint: "https://horizon.example.com/catalog" read_only: true ``` #### DuckLake [DuckLake](https://ducklake.select/) is a separate open table format (not Apache Iceberg) built for DuckDB, but you configure it the same way, through `catalogs.yml`. Because DuckLake isn't Iceberg, its catalog entries use `table_format: default`. catalogs.yml ```yaml catalogs: - name: local_lake type: ducklake table_format: default config: duckdb: metadata_path: "metadata.ducklake" data_path: "s3://my-bucket/lake" # optional ``` models/my\_ducklake\_model.sql ```sql {{ config( materialized = 'table', catalog_name = 'local_lake' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` dbt installs the DuckLake extension and attaches the catalog before running your model: ```sql INSTALL ducklake ATTACH IF NOT EXISTS 'ducklake:metadata.ducklake' AS local_lake (DATA_PATH 's3://my-bucket/lake') ``` #### Secrets The `secret` field in a `duckdb` catalog block references a named secret defined in `profiles.yml`, which dbt turns into a DuckDB [`CREATE SECRET`](https://duckdb.org/docs/configuration/secrets_manager.html) statement: profiles.yml ```yaml my_profile: target: dev outputs: dev: type: duckdb path: ':memory:' secrets: - type: iceberg name: my_iceberg_secret # additional key-value pairs become CREATE SECRET parameters # (for example, token, client_id, client_secret) -- see DuckDB's # iceberg extension docs for the parameters your catalog needs. ``` For Amazon S3 Tables specifically, use a `type: s3` / `provider: credential_chain` secret instead (SigV4, not OAuth2/token-based) — see [Amazon S3 Tables (write path)](#amazon-s3-tables-write-path) above. #### DuckDB-specific configs for Iceberg catalogs You can supply these configs, nested under `config.duckdb`, for `horizon`, `unity`, and `iceberg_rest` catalogs: | Field | Required | Description | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `endpoint` | One of `endpoint`/`endpoint_type` | Full Iceberg REST catalog URL. | | `endpoint_type` | One of `endpoint`/`endpoint_type` | `GLUE` or `S3_TABLES`, for well-known AWS-managed endpoints. For writing to S3 Tables, prefer the explicit `endpoint` form (see caution above). | | `warehouse` | Required for `horizon`; required for any Amazon S3 Tables catalog (both the `endpoint_type: S3_TABLES` and the explicit `endpoint` write path) | Warehouse identifier passed as the `ATTACH` source. | | `secret` | Optional | Name of a DuckDB secret from `profiles.yml` to use for authentication. | | `attach_as` | Optional | Overrides the DuckDB attach alias. Defaults to the catalog's `name`. | | `default_region` | Optional | AWS region, when applicable. | | `default_schema` | Optional | Default schema/namespace within the catalog. | | `max_table_staleness` | Optional | How long DuckDB may serve cached metadata before refreshing. | | `authorization_type` | Optional | `OAUTH2`, `SIGV4`, or `NONE`. Can't be combined with `endpoint_type`. Required as `SIGV4` when writing to S3 Tables. | | `access_delegation_mode` | Optional | `VENDED_CREDENTIALS` or `NONE`. | | `read_only` | Optional | Attach the catalog read-only. Defaults to `false` (read-write). | | `support_nested_namespaces` | Optional | Whether the catalog supports nested namespaces. | | `stage_create_tables` | Optional | Write-compat: stage `CREATE TABLE AS SELECT` writes. Requires DuckDB 1.5.4+. Set to `false` for Amazon S3 Tables. | | `disable_multi_table_commit` | Optional | Write-compat: disable multi-table commits. Requires DuckDB 1.5.4+. Set to `true` for Amazon S3 Tables. | | `skip_create_table_metadata_updates` | Optional | Write-compat: skip metadata updates on `CREATE TABLE`. Requires DuckDB 1.5.4+. | | `remove_files_on_delete` | Optional | Write-compat: remove underlying data files when a table is dropped. Requires DuckDB 1.5.4+. | | `purge_requested` | Optional | Purge underlying files when supported by the catalog. Set to `true` for Amazon S3 Tables, which only allows `DROP TABLE` with purge enabled. | | `encode_entire_prefix` | Optional | Percent-encode the entire object key prefix. | For `ducklake` catalogs, `config.duckdb` accepts: | Field | Required | Description | | ------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | `metadata_path` | Required | Path to the DuckLake metadata store, for example `metadata.ducklake` or a database connection string. | | `data_path` | Optional | Where DuckLake writes data files. | | `attach_as` | Optional | Overrides the DuckDB attach alias. Defaults to the catalog's `name`. | | `metadata_schema` | Optional | Schema within the metadata store to use. | | `metadata_catalog` | Optional | Catalog/database name within the metadata store. | | `data_inlining_row_limit` | Optional | Inline row groups smaller than this many rows into the metadata catalog instead of writing a Parquet file. | | `create_if_not_exists` | Optional | Create the DuckLake catalog if it doesn't already exist. | | `read_only` | Optional | Attach read-only. | | `encrypted` | Optional | Encrypt the DuckLake catalog. | | `automatic_migration` | Optional | Automatically migrate the catalog's DuckLake format version on attach. | | `override_data_path` | Optional | Allow attaching with a `data_path` that differs from the one recorded in an existing catalog. | --- ### Enabling dbt State in Studio Login required | Usage-based When you enable dbt State in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), it runs automatically on every `dbt run` or `dbt build` during development — skipping unchanged models and reusing production results so your runs are *faster*. You can [turn it on for your development environment](#enabling-dbt-state-on-a-development-environment) so it's the default for everyone, or you can [override that setting just for your own account](#overriding-dbt-state-setting-per-user). **Prerequisite**: An account admin must [enable dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) before you can use it. #### Enabling dbt State on a development environment Enabling dbt State on your development environment turns it on for everyone using the Studio IDE, unless they override it for their own account. 1. Go to **Orchestration** > **Environments** and select your development environment. 2. Click **Settings** > **Edit**. 3. In the **dbt State** section, select **Enable dbt State**. 4. Click **Save**. 5. In the pop-up box, click **Continue** if you want to go ahead with the changes and restart all IDE sessions for this project. #### Overriding dbt State setting per user You can override the development environment's dbt State setting for your own account without affecting other users. Because the user-level setting takes precedence over the environment-level setting, you can turn dbt State on for yourself before enabling it for your whole team, or turn it off when it's enabled at the environment level. 1. Click your account name in the lower-left corner and select **Account settings**. 2. Under **Your profile**, go to **Credentials**. 3. Select the project you want to enable dbt State for. 4. Click **Edit** and go to the **User development settings** section. 5. Under **dbt State**, select one of the following options: * **Enabled**: Enables dbt State for your user regardless of the development environment setting. * **Disabled**: Disables dbt State for your user regardless of the development environment setting. * **Reset (inherit from development)**: Only appears after you've saved an **Enabled** or **Disabled** override. Clears your override and falls back to the dbt State setting configured on your development environment. 6. Click **Save**. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Enable dbt State on environments and jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md) --- ### Enabling dbt State on environments and jobs Login required | Usage-based Learn how to enable [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) on deployment environments and individual jobs. This option is only visible if dbt State is [enabled on your account](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). #### Enabling dbt State on environments You can enable [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) at the environment level, allowing jobs in that environment to inherit the setting automatically. When you enable dbt State on a deployment environment (production, staging, or general): * New jobs default to **Inherited from environment** and have dbt State enabled without any additional configuration. * Existing jobs are not automatically updated. You must [configure each job manually](#enabling-dbt-state-on-individual-jobs). To enable dbt State on a deployment environment: 1. Go to **Orchestration** > **Environments**. 2. Select the environment you want to enable dbt State for. 3. Click **Settings** > **Edit**. 4. In the **dbt State** section, select **Enable dbt State**. 5. Click **Save**. For development environments, refer to [Enabling dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md). #### Enabling dbt State on individual jobs dbt State is available on all job types: deploy, continuous integration (CI), and merge jobs. Each job has a **dbt State** dropdown in its execution settings with **On**, **Off**, or **Inherited from environment** options. New jobs default to **Inherited from environment** — no additional configuration needed as long as dbt State is enabled on the environment. Existing jobs default to **Off**. To follow the environment's setting, you must configure each job manually. To enable dbt State on individual jobs: 1. Go to **Orchestration** > **Jobs**. 2. Select the job you want to enable dbt State for. 3. Click **Settings** > **Edit**. 4. In the **Execution settings** section, set the **dbt State** dropdown to **On** to enable it explicitly, or **Inherited from environment** to follow the environment's setting. If you select **Inherited from environment**, make sure dbt State is [enabled on the environment](#enabling-dbt-state-on-environments) first — otherwise, the job will inherit it as off. 5. Click **Save**. note dbt State and State-aware orchestration are mutually exclusive. Selecting **Inherited from environment** or **On** automatically clears the **State-aware orchestration** option, and enabling **State-aware orchestration** sets **dbt State** to **Off**. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Enable dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md) --- ### Enhance your code [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/environment-variables.md) ###### [Environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) [Learn how you can use environment variables to customize the behavior of a dbt project.](https://docs.getdbt.com/docs/build/environment-variables.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/hooks-operations.md) ###### [Hooks and operations](https://docs.getdbt.com/docs/build/hooks-operations.md) [Learn how to use hooks to trigger actions and operations to invoke macros.](https://docs.getdbt.com/docs/build/hooks-operations.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/packages.md) ###### [Packages](https://docs.getdbt.com/docs/build/packages.md) [Learn how you can leverage code reuse through packages (libraries).](https://docs.getdbt.com/docs/build/packages.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/project-variables.md) ###### [Project variables](https://docs.getdbt.com/docs/build/project-variables.md) [Learn how to use project variables to provide data to models for compilation.](https://docs.getdbt.com/docs/build/project-variables.md) --- ### Enhance your models [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/materializations.md) ###### [Materializations](https://docs.getdbt.com/docs/build/materializations.md) [Learn how to use materializations to make dbt models persist in a data platform.](https://docs.getdbt.com/docs/build/materializations.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/incremental-models.md) ###### [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) [Learn how to use incremental models so you can limit the amount of data that needs to be transformed.](https://docs.getdbt.com/docs/build/incremental-models.md) <br /> --- ### Entities Entities are real-world concepts in a business, such as customers, transactions, and ad campaigns. We often focus our analyses on specific entities, such as customer churn or annual recurring revenue modeling. In our Semantic Layer models, these entities serve as a join key across semantic models. (Applies to dbt v1.12 and later) Within a semantic graph, the required parameters for an entity are `name` and `type`. The `name` refers to either the key column name from the underlying data table, or it may serve as an alias with the column name referenced in the `expr` parameter. The `name` for your entity must be unique to the semantic model and can not be the same as an existing simple metric or `dimension` within that same model. Entities can be specified with a single column or multiple columns. Entities (join keys) in a semantic model are identified by their name. Each entity name must be unique within a semantic model, but it doesn't have to be unique across different semantic models. There are four entity types: * [Primary](#primary) — Has only one record for each row in the table and includes every record in the data platform. This key uniquely identifies each record in the table. * [Unique](#unique) — Contains only one record per row in the table and allows for null values. May have a subset of records in the data warehouse. * [Foreign](#foreign) — A field (or a set of fields) in one table that uniquely identifies a row in another table. This key establishes a link between tables. * [Natural](#natural) — Columns or combinations of columns in a table that uniquely identify a record based on real-world data. This key is derived from actual data attributes. Use entities as dimensions You can also use entities as dimensions, which allows you to aggregate a metric to the granularity of that entity. #### Entity types MetricFlow's join logic depends on the entity `type` you use and determines how to join semantic models. Refer to [Joins](https://docs.getdbt.com/docs/build/join-logic.md) for more info on how to construct joins. ##### Primary A primary key has *only one* record for each row in the table and includes every record in the data platform. It must contain unique values and can't contain null values. Use the primary key to ensure that each record in the table is distinct and identifiable.  Primary key example For example, consider a table of employees with the following columns: ```sql employee_id (primary key) first_name last_name ``` In this case, `employee_id` is the primary key. Each `employee_id` is unique and represents one specific employee. There can be no duplicate `employee_id` and can't be null. ##### Unique A unique key contains *only one* record per row in the table but may have a subset of records in the data warehouse. However, unlike the primary key, a unique key allows for null values. The unique key ensures that the column's values are distinct, except for null values.  Unique key example For example, consider a table of students with the following columns: ```sql student_id (primary key) email (unique key) first_name last_name ``` In this example, `email` is defined as a unique key. Each email address must be unique; however, multiple students can have null email addresses. This is because the unique key constraint allows for one or more null values, but non-null values must be unique. This then creates a set of records with unique emails (non-null) that could be a subset of the entire table, which includes all students. ##### Foreign A foreign key is a field (or a set of fields) in one table that uniquely identifies a row in another table. The foreign key establishes a link between the data in two tables. It can include zero, one, or multiple instances of the same record. It can also contain null values.  Foreign key example For example, consider you have two tables, `customers` and `orders`: customers table: ```sql customer_id (primary key) customer_name ``` orders table: ```sql order_id (primary key) order_date customer_id (foreign key) ``` In this example, the `customer_id` in the `orders` table is a foreign key that references the `customer_id` in the `customers` table. This link means each order is associated with a specific customer. However, not every order must have a customer; the `customer_id` in the orders table can be null or have the same `customer_id` for multiple orders. ##### Natural Natural keys are columns or combinations of columns in a table that uniquely identify a record based on real-world data. For instance, if you have a `sales_person_department` dimension table, the `sales_person_id` can serve as a natural key. You can only use natural keys for [SCD type II dimensions](https://docs.getdbt.com/docs/build/dimensions.md#scd-type-ii). #### Entities configuration The following is the complete spec for entities: (Applies to dbt v1.12 and later) ```yaml models: - name: model_name semantic_model: enabled: true name: semantic_model_name # ...rest of the semantic model config (labels, groups, etc.) columns: - name: transaction_id entity: type: primary # Required; primary | natural | foreign | unique name: entity_name # Optional; defaults to column name description: "A description of the field or role the entity takes in this table" # Additional configuration config: # Optional meta: owner: "Team Name" ``` Here's an example of how to define entities in a semantic model: (Applies to dbt v1.12 and later) ```yaml models: - name: your_model_name semantic_model: enabled: true columns: - name: id_transaction entity: type: primary name: transaction - name: id_order entity: type: foreign name: order ``` #### `derived_semantics` in `entities` Use `derived_semantics` when you need an entity that is not a direct 1:1 mapping to a single physical column. The `expr` field is required when using `derived_semantics`. For example: ```yaml # Derived entities (expression-based) derived_semantics: entities: - name: user type: foreign expr: "substring(id_order from 2)" # Required ``` #### Combine columns with a key If a table doesn't have any key (like a primary key), use *surrogate combination* to form a key that will help you identify a record by combining two columns. This applies to any [entity type](https://docs.getdbt.com/docs/build/entities.md#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a *surrogate key*. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator. (Applies to dbt v1.12 and later) ```yaml models: - name: your_model_name semantic_model: enabled: true derived_semantics: entities: - name: brand_target_key type: foreign expr: "date_key || '|' || brand_code" ``` #### Examples As mentioned, entities serve as our join keys, using the unique entity name. Therefore, we can join a single `unique` key to multiple `foreign` keys. Consider a `date_categories` table with the following columns: ```sql date_id (primary key) date_day (unique key) fiscal_year_name ``` And an `orders` table with the following columns: ```sql order_id (primary key) ordered_at delivered_at order_total ``` How might we define our Semantic Layer YAML so that we can query `order_total` by `ordered_at` `fiscal_year_name`, and `delivered_at` `fiscal_year_name`? (Applies to dbt v1.12 and later) First, we need to define two `unique` entities in the `date_categories` with the expression set to `date_day`: ```yaml models: - name: date_categories semantic_model: enabled: true agg_time_dimension: date_day columns: - name: date_id entity: type: primary name: date_id - name: date_day granularity: day dimension: type: time name: date_day - name: fiscal_year_name dimension: type: categorical name: fiscal_year_name description: "Formatted fiscal year string (e.g. 'FY2025')" derived_semantics: # Unique entities entities: - name: ordered_at_entity type: unique expr: date_day - name: delivered_at_entity type: unique expr: date_day ``` (Applies to dbt v1.12 and later) Then, we need to add these same entities as `foreign` keys to our `orders` model, with the expression set to `ordered_at` and `delivered_at`: ```yaml models: - name: orders semantic_model: enabled: true agg_time_dimension: ordered_at columns: - name: order_id entity: type: primary name: order_id - name: ordered_at granularity: day dimension: type: time name: ordered_at derived_semantics: # Foreign keys entities: - name: ordered_at_entity type: foreign expr: ordered_at - name: delivered_at_entity type: foreign expr: delivered_at metrics: - name: order_total type: simple agg: sum description: "Total amount for each order including taxes." ``` With this configuration, our semantic models can join on `ordered_at = date_day` via the `ordered_at_entity`, and on `delivered_at = date_day` via the `delivered_at_entity`. To validate our output, we can run: * `dbt sl query --metrics order_total --group-by ordered_at_entity__fiscal_year_name` or * `dbt sl query --metrics order_total --group-by delivered_at_entity__fiscal_year_name` --- ### Environment variables Environment variables can be used to customize the behavior of a dbt project depending on where the project is running. Refer to the docs on [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) for more information on how to call the Jinja function `{{env_var('DBT_KEY','OPTIONAL_DEFAULT')}}` in your project code. Environment Variable Naming and Prefixing Environment variables in dbt must be prefixed with either `DBT_`, `DBT_ENV_SECRET_`, or `DBT_ENV_CUSTOM_ENV_`. Environment variable keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt's UI. ##### Setting and overriding environment variables This section explains how to set and override environment variables in dbt. * [Order of precedence](#order-of-precedence) * [Setting environment variables](#setting-environment-variables) * [Overriding environment variables at the job level](#overriding-environment-variables-at-the-job-level) * [Overriding environment variables at the personal level](#overriding-environment-variables-at-the-personal-level) * [Local environment variables](#local-environment-variables) ###### Order of precedence Environment variable values can be set in multiple places within dbt. As a result, dbt will interpret environment variables according to the following order of precedence (lowest to highest): [![Environment variables order of precedence](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/env-var-precdence.png?v=2> "Environment variables order of precedence")](#)Environment variables order of precedence There are four levels of environment variables: 1. The optional default argument supplied to the `env_var` Jinja function in code, which can be overridden at (*lowest precedence*) 2. The project-wide level by its default value, which can be overridden at 3. The environment level, which can in turn be overridden again at 4. The job level (job override) or in the Studio IDE for an individual dev (personal override). (*highest precedence*) ###### Setting environment variables To set environment variables at the project and environment level, click **Orchestration** in the left-side menu, then select **Environments**. Click **Environment variables** to add and update your environment variables. [![Environment variables tab](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/navigate-to-env-vars.png?v=2> "Environment variables tab")](#)Environment variables tab You'll notice there is a **Project default** column. This is a great place to set a value that will persist across your whole project, independent of where the code is run. We recommend setting this value when you want to supply a catch-all default or add a project-wide token or secret. To the right of the **Project default** column, you can see all your environments. Values set at the environment level take priority over the project-level default value. This is where you can tell dbt to interpret an environment value differently in your staging versus production environment, as an example. [![Setting project level and environment level values](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/project-environment-view.png?v=2> "Setting project level and environment level values")](#)Setting project level and environment level values ###### Overriding environment variables at the job level You may have multiple jobs that run in the same environment, and you'd like the environment variable to be interpreted differently depending on the job. When setting up or editing a job, you will see a section where you can override environment variable values defined at the environment or project level. [![Navigating to environment variables job override settings](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/job-override.gif?v=2> "Navigating to environment variables job override settings")](#)Navigating to environment variables job override settings Every job runs in a specific deployment environment, and by default, a job will inherit the values set at the environment level (or the highest precedence level set) for the environment in which it runs. If you'd like to set a different value at the job level, edit the value to override it. [![Setting a job override value](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/job-override.png?v=2> "Setting a job override value")](#)Setting a job override value ###### Overriding environment variables at the personal level You can also set a personal value override for an environment variable when you develop in the dbt-integrated developer environment (Studio IDE). By default, dbt uses environment variable values set in the project's development environment. To see and override these values, from dbt: 1. Click your account name in the left side menu and select **Account settings**. 2. Under the **Your profile** section, click **Credentials** and then select your project. 3. Scroll to the **Environment variables** section and click **Edit** to make the necessary changes. To supply an override, developers can edit and specify a different value to use. These values will be respected in the Studio IDE both for the Results and Compiled SQL tabs. [![Setting a personal override value](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/personal-override.png?v=2> "Setting a personal override value")](#)Setting a personal override value Appropriate coverage If you have not set a project level default value for every environment variable, it may be possible that dbt does not know how to interpret the value of an environment variable in all contexts. In such cases, dbt will throw a compilation error: "Env var required but not provided". Changing environment variables mid-session in the Studio IDE If you change the value of an environment variable mid-session while using the Studio IDE, you may have to refresh the Studio IDE for the change to take effect. To refresh the Studio IDE mid-development, click on either the green 'ready' signal or the red 'compilation error' message at the bottom right corner of the Studio IDE. A new modal will pop up, and you should select the **Restart IDE** button. This loads your environment variable values into your development environment. [![Refreshing IDE mid-session](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/refresh-ide.png?v=2> "Refreshing IDE mid-session")](#)Refreshing IDE mid-session There are some known issues with partial parsing of a project and changing environment variables mid-session in the IDE. If you find that your dbt project is not compiling to the values you've set, try deleting the `target/partial_parse.msgpack` file in your dbt project which will force dbt to re-compile your whole project. ###### Local environment variables If you are using the dbt VS Code extension, you can set environment variables locally in your shell profile (`~/.zshrc` or `~/.bashrc`) or in a `.env` file at the root level of your dbt project. For more information, refer to [Configure the dbt VS Code extension](https://docs.getdbt.com/docs/configure-dbt-extension.md#set-environment-variables-locally). ##### Handling secrets While all environment variables are encrypted at rest in dbt, dbt has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in dbt, you can prefix the key with `DBT_ENV_SECRET_`. Environment variables prefixed with `DBT_ENV_SECRET_` are protected with additional security controls. They are encrypted at rest using an encryption key (for example, AWS KMS when your deployment is hosted on AWS) and can only be accessed by decrypting them with that key. Decryption is restricted to specific flows where the value is required, such as when a job runs. Secret keys are never written to logs or error messages and are obfuscated in dbt, so they are not exposed in the UI or artifacts, and are only available to dbt at runtime as needed. [![DBT\_ENV\_SECRET prefix obfuscation](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/DBT_ENV_SECRET.png?v=2> "DBT_ENV_SECRET prefix obfuscation")](#)DBT\_ENV\_SECRET prefix obfuscation **Note**: An environment variable can be used to store a [git token for repo cloning](https://docs.getdbt.com/docs/build/environment-variables.md#clone-private-packages). We recommend you make the git token's permissions read only and consider using a machine account or service user's PAT with limited repo access in order to practice good security hygiene. ##### Special environment variables dbt has a number of pre-defined variables built in. Variables are set automatically and cannot be changed. This means that the order of precedence for overriding environment variables doesn't apply to these pre-defined variables at the project, environment, or job level. ###### Studio IDE details The following environment variable is set automatically for the Studio IDE: * `DBT_CLOUD_GIT_BRANCH` — Provides the development Git branch name in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). * The variable changes when the branch is changed. * Doesn't require restarting the Studio IDE after a branch change. * Currently not available in the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). Use case — This is useful in cases where you want to dynamically use the Git branch name as a prefix for a [development schema](https://docs.getdbt.com/docs/build/custom-schemas.md) (`{{ env_var('DBT_CLOUD_GIT_BRANCH') }}`). ###### dbt platform context The following environment variables are set automatically: * `DBT_ENV` — This key is reserved for the dbt application and will always resolve to 'prod'. For deployment runs only. * `DBT_CLOUD_ENVIRONMENT_NAME` — The name of the dbt environment in which `dbt` is running. * `DBT_CLOUD_ENVIRONMENT_TYPE` — The type of dbt environment in which `dbt` is running. The valid values are `dev`, `staging`, or `prod`. The value will be empty for [General deployment environments](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments), so use a default like `{{ env_var('DBT_CLOUD_ENVIRONMENT_TYPE', '') }}`. * `DBT_CLOUD_INVOCATION_CONTEXT` — The context type in which `dbt` is invoked. The values are `dev`, `staging`, `prod`, or `ci`. * Additionally, use `DBT_CLOUD_INVOCATION_CONTEXT` in the `generate_schema_name()` macro to define explicit guidelines to use the default schema only (with the `dbt_cloud_pr prefix`) in CI job runs, even if those CI jobs run in the same environment as production jobs. ###### Run details * `DBT_CLOUD_PROJECT_ID` — The ID of the dbt Project for this run * `DBT_CLOUD_JOB_ID` — The ID of the dbt Job for this run * `DBT_CLOUD_RUN_ID` — The ID of this particular run * `DBT_CLOUD_RUN_REASON_CATEGORY` — The "category" of the trigger for this run (one of: `scheduled`, `github_pull_request`, `gitlab_merge_request`, `azure_pull_request`, `other`) * `DBT_CLOUD_RUN_REASON` — The specific trigger for this run (for example, `Scheduled`, `Kicked off by <email>`, or a custom value set through the API) * `DBT_CLOUD_ENVIRONMENT_ID` — The ID of the environment for this run * `DBT_CLOUD_ACCOUNT_ID` — The ID of the dbt account for this run ###### Git details *The following variables are currently only available for GitHub, GitLab, and Azure DevOps PR builds triggered using a webhook* * `DBT_CLOUD_PR_ID` — The Pull Request ID in the connected version control system * `DBT_CLOUD_GIT_SHA` — The git commit SHA which is being run for this Pull Request build ##### Example usage Environment variables can be used in many ways, and they give you the power and flexibility to do what you want to do more easily in dbt.  Clone private packages Now that you can set secrets as environment variables, you can pass git tokens into your package HTTPS URLs to allow for on-the-fly cloning of private repositories. Read more about enabling [private package cloning](https://docs.getdbt.com/docs/build/packages.md#private-packages).  Dynamically set your warehouse in your Snowflake connection Environment variables make it possible to dynamically change the Snowflake virtual warehouse size depending on the job. Instead of calling the warehouse name directly in your project connection, you can reference an environment variable which will get set to a specific virtual warehouse at runtime. For example, suppose you'd like to run a full-refresh job in an XL warehouse, but your incremental job only needs to run in a medium-sized warehouse. Both jobs are configured in the same dbt environment. In your connection configuration, you can use an environment variable to set the warehouse name to `{{env_var('DBT_WAREHOUSE')}}`. Then in the job settings, you can set a different value for the `DBT_WAREHOUSE` environment variable depending on the job's workload. Currently, it's not possible to dynamically set environment variables across models within a single run. This is because each env\_var can only have a single set value for the entire duration of the run. **Note** — You can also use this method with Databricks SQL Warehouse. [![Adding environment variables to your connection credentials](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/warehouse-override.png?v=2> "Adding environment variables to your connection credentials")](#)Adding environment variables to your connection credentials Environment variables and Snowflake OAuth limitations Env vars work well with username/password and keypair, including scheduled jobs, because dbt v1 consumes the Jinja inserted into the autogenerated [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) and resolves it to do an `env_var` lookup. However, there are some limitations when using env vars with Snowflake OAuth Connection settings: * You can't use them in the account/host field, but they can be used for database, warehouse, and role. For these fields, [use extended attributes](https://docs.getdbt.com/docs/deploy/deploy-environments.md#deployment-connection). Something to note, if you supply an environment variable in the account/host field, Snowflake OAuth Connection will **fail** to connect. This happens because the field doesn't pass through Jinja rendering, so dbt simply passes the literal `env_var` code into a URL string like `{{ env_var("DBT_ACCOUNT_HOST_NAME") }}.snowflakecomputing.com`, which is an invalid hostname. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) instead.  Audit your run metadata Here's another motivating example that uses the dbt run ID, which is set automatically at each run. This additional data field can be used for auditing and debugging: ```sql {{ config(materialized='incremental', unique_key='user_id') }} with users_aggregated as ( select user_id, min(event_time) as first_event_time, max(event_time) as last_event_time, count(*) as count_total_events from {{ ref('users') }} group by 1 ) select *, -- Inject the run id if present, otherwise use "manual" '{{ env_var("DBT_CLOUD_RUN_ID", "manual") }}' as _audit_run_id from users_aggregated ```  Configure Semantic Layer credentials Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. --- ### Example usage for dbt State Login required | Usage-based These examples use the Jaffle Shop project to show side-by-side comparisons of CLI output with and without dbt State enabled. To enable dbt State, follow the steps in [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). The following examples use this [Jaffle Shop project](https://github.com/dbt-labs/jaffle-shop) DAG as a reference. You can refer to it to understand the model lineage for each scenario. [![The Jaffle Shop DAG](/img/docs/dbt-state/dbt_state_dag.png?v=2 "The Jaffle Shop DAG")](#)The Jaffle Shop DAG Each of the following scenarios shows how a run differs between dbt alone and dbt with dbt State, using the same command and project. | Scenario | Command | What dbt State changes | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | [Initial run in empty schema](#initial-run-in-empty-schema) | `dbt run --target prod` | Same result | | [Second run](#second-run) | `dbt run --target prod` | Reuses unchanged table models and most views; may rebuild views with `select *` directly on a `ref()` or `source()` | | [Selecting a model in a fresh dev environment after changing the customers model](#selecting-a-model-in-a-fresh-dev-environment-after-changing-the-customers-model) | `dbt run --target dev --select "customers"` | Defers to prod for upstream models | | [Selecting a model in a new dev schema with no model changes](#selecting-a-model-in-a-new-dev-schema-with-no-model-changes) | `dbt run --target dev --select "customers"` | Defers and clones unchanged models | <br /> Every skipped model is a model you didn't pay to rebuild. dbt State tracks what's changed and skips the rest — reducing run time and warehouse costs. #### Initial run in empty schema ```shell dbt run --target prod ``` With no prior state to compare against, dbt builds every model from scratch. dbt State captures metadata from this run for future comparisons. ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 644 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [SUCCESS 1 in 0.54s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [SUCCESS 1 in 0.48s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [SUCCESS 1 in 0.51s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [SUCCESS 1 in 0.77s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [SUCCESS 1 in 0.58s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [SUCCESS 1 in 0.63s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [SUCCESS 1 in 1.81s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [SUCCESS 1 in 1.33s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [SUCCESS 1 in 2.09s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [SUCCESS 1 in 1.26s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [SUCCESS 1 in 1.70s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [SUCCESS 1 in 1.77s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 17.77 seconds (17.77s). Completed successfully Done. PASS=12 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=12 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 644 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] State adapter: Fetching freshness metadata 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [SUCCESS 1 in 0.54s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [SUCCESS 1 in 0.48s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [SUCCESS 1 in 0.51s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [SUCCESS 1 in 0.77s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [SUCCESS 1 in 0.58s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [SUCCESS 1 in 0.63s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [SUCCESS 1 in 1.81s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [SUCCESS 1 in 1.33s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [SUCCESS 1 in 2.09s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [SUCCESS 1 in 1.26s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [SUCCESS 1 in 1.70s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [SUCCESS 1 in 1.77s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 17.77 seconds (17.77s). Completed successfully. Total cache hits: 0. Estimated time saved: 0.00s. Freshness tolerance: 45m. Done. PASS=12 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=12 ``` #### Second run ```shell dbt run --target prod ``` For each model, dbt State compares the current logic and upstream data against the previous run. If nothing has changed, dbt State skips the build or clones the result from another environment. With dbt State enabled, all twelve models are reused — nothing changed, so there's nothing to rebuild. dbt State reuses views when their SQL logic is unchanged, even if new data has arrived upstream. Because views don't store data, they always read directly from the underlying tables when queried. The staging views shown here use `select *` on a CTE, which dbt State can reuse. For details on `select *` patterns that can force a rebuild, refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select). ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 644 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [SUCCESS 1 in 0.54s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [SUCCESS 1 in 0.48s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [SUCCESS 1 in 0.51s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [SUCCESS 1 in 0.77s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [SUCCESS 1 in 0.58s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [SUCCESS 1 in 0.63s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [SUCCESS 1 in 1.81s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [SUCCESS 1 in 1.33s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [SUCCESS 1 in 2.09s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [SUCCESS 1 in 1.26s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [SUCCESS 1 in 1.70s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [SUCCESS 1 in 1.77s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 17.77 seconds (17.77s). Completed successfully Done. PASS=12 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=12 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Unable to do partial parsing because of a version mismatch Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] State adapter: Fetching freshness metadata 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [No new changes in 2.73s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [No new changes in 0.87s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [No new changes in 1.04s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [No new changes in 0.74s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [No new changes in 1.47s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [No new changes in 0.92s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [No new changes in 1.84s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [No new changes in 1.99s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [No new changes in 2.35s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [No new changes in 0.24s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [No new changes in 1.07s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [No new changes in 2.19s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 19.79 seconds (19.79s). Completed successfully. Total cache hits: 12. Estimated time saved: 17.77s. Freshness tolerance: 45m. Done. PASS=0 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=12 TOTAL=12 ``` #### Selecting a model in a fresh dev environment after changing the customers model ```shell dbt run --target dev --select "customers" ``` Imagine you've made a small change to the `customers` model and run it in a fresh developer schema. Without dbt State, dbt v1 fails because upstream relations are missing. With dbt State, dbt [defers](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md) to prod for upstream models and runs only the updated `customers` model. ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] 1 of 1 ERROR creating sql table model dbt_schema.customers ..................... [ERROR in 0.36s] Finished running 1 table model in 0 hours 0 minutes and 5.11 seconds (5.11s). Completed with 1 error, 0 partial successes, and 0 warnings: [ERROR]: in model customers (models/marts/customers.sql) Database Error in model customers (models/marts/customers.sql) 002003 (42S02): SQL compilation error: Object 'ANALYTICS.DBT_SCHEMA.STG_CUSTOMERS' does not exist or not authorized. compiled code at target/run/jaffle_shop/models/marts/customers.sql compiled code at target/compiled/jaffle_shop/models/marts/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] State adapter: Fetching freshness metadata 1 of 1 OK created sql table model dbt_schema.customers ......................... [SUCCESS 128 in 52.46s] Finished running 1 table model in 0 hours 0 minutes and 58.06 seconds (58.06s). Completed successfully Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 ``` #### Selecting a model in a new dev schema with no model changes ```shell dbt run --target dev --select "customers" ``` Suppose you create a fresh dev schema and run only the `customers` model. Without dbt State, dbt v1 fails because there is no data in the schema. With dbt State, dbt knows `customers` just ran in another schema: it defers to prod for upstream models and clones `customers` because the outcome is unchanged. ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] 1 of 1 ERROR creating sql table model dbt_schema.customers ..................... [ERROR in 0.36s] Finished running 1 table model in 0 hours 0 minutes and 5.11 seconds (5.11s). Completed with 1 error, 0 partial successes, and 0 warnings: [ERROR]: in model customers (models/marts/customers.sql) Database Error in model customers (models/marts/customers.sql) 002003 (42S02): SQL compilation error: Object 'ANALYTICS.DBT_SCHEMA.STG_CUSTOMERS' does not exist or not authorized. compiled code at target/run/jaffle_shop/models/marts/customers.sql compiled code at target/compiled/jaffle_shop/models/marts/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] State adapter: Fetching freshness metadata 1 of 1 OK created sql table model dbt_schema.customers ......................... [Cloned from other environment in 57.19s] Finished running 1 table model in 0 hours 1 minutes and 3.50 seconds (63.50s). Completed successfully. Total cache hits: 1. Estimated time saved: 2.20s. Freshness tolerance: 45m. Done. PASS=0 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=1 TOTAL=1 ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Set up authentication for non-interactive environments](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) * [Configuring deferral in dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md) * [Monitoring dbt State activity in dbt platform](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) * [Migrating from state-aware orchestration to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### Explore cost data dbt platform | Enterprise, Enterprise+ You can access Cost Insights in these different dbt platform areas: * [Project dashboard](#project-dashboard) * [Catalog on Model page](#model-performance-in-catalog) * [Job details page](#job-details) Each view provides different levels of detail to help you understand your warehouse spending and optimization impact. Cost and cost reduction estimates are based on historical runs and reflect actual usage, *not* forecasts of future costs. #### Prerequisites To view cost data, ensure you have: * One of the roles listed in [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * A supported data warehouse: * Snowflake * BigQuery * Databricks * Amazon Redshift [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md). note For accounts already using dbt State or state-aware orchestration before Cost Insights is enabled, at least one full model build must occur within the last 10 days to establish a baseline for cost reduction calculations. If you don't see cost reduction data, try running a full build to establish the baseline. #### Project dashboard The Cost Insights section in your project dashboard gives you a high-level view of warehouse costs and the impact of optimization through dbt State or state-aware orchestration. [![Cost Insights in the project dashboard](/img/docs/dbt-platform/cost-insights/cost-insights-project.png?v=2 "Cost Insights in the project dashboard")](#)Cost Insights in the project dashboard ##### Access To go to your project dashboard, select your project in the main menu and click **Dashboard**. ##### Key metrics The project dashboard displays the following metrics that summarize the overall cost and optimization impact for your project: * **Total cost reduction** * **Total % reduction** * **Total query run time reduction** * **Reused assets** ##### Filters You can customize the cost data you want to view by: * **Deployment type**: Production or Staging * **Last**: 30 days, 60 days, 90 days, 6 months, or 1 year * **View**: Daily, Weekly, or Monthly * **Assets**: All, Models, Tests ##### Visualization tabs The project dashboard includes the following tabs that help you analyze cost and optimization trends over time. * **Cost**: Shows the estimated build cost reduction when using dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed and the reduction in usage from dbt State or state-aware orchestration over the selected timeframe. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated reduction in build time when using dbt State or state-aware orchestration. * **Builds**: Shows total builds split into number of assets rebuilt and assets reused by dbt State or state-aware orchestration. ##### Table view Access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. #### Model performance in Catalog The **Model performance** section in Catalog displays historical trends to help you identify optimization opportunities and understand model resource consumption. [![Cost Insights in Catalog](/img/docs/dbt-platform/cost-insights/cost-insights-model.png?v=2 "Cost Insights in Catalog")](#)Cost Insights in Catalog ##### Access To access model performance data: 1. From the main menu, go to **Catalog**. 2. Click your project from the file tree. 3. Navigate to the model whose cost data you want to view. You can search for it or click **Models** under **Project assets** in the sidebar to view all available models in the project. 4. Go to the **Performance** tab on the model's details page. ##### Key metrics The **Model performance** section displays the following metrics that summarize the overall cost and optimization impact for your project: * **Total cost reduction** * **Total % reduction** * **Total query run time deduction** * **Reused assets** (when dbt State or state-aware orchestration is enabled) ##### Filters Use the time period filter to customize the data you want to view: from the last 3 months up to the last 1 week. For **Cost insights**, **Usage**, and **Query run time** tabs, you can set the view granularity by **Daily**, **Weekly**, or **Monthly**. ##### Visualization tabs * **Cost insights**: Shows the estimated warehouse costs incurred by this model and cost reduction from dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed by this model over time. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated query execution time and the reduction in run duration from dbt State or state-aware orchestration. * **Build time**: Shows average execution time for the model and how it trends over the selected period. * **Build count**: Tracks how many times the model was built or reused, including any failures or errors. * **Test results**: Displays test execution outcomes and pass/fail rates for tests on this model. * **Consumption queries**: Shows queries running against this model, helping you understand downstream usage patterns. ##### Table view For **Cost insights**, **Usage**, and **Query run time** tabs, you can access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. ##### Chart interactions For **Build time** and **Build count** tabs: * Click on any data point in the charts to see a detailed table listing all job runs for that day. * Each row in the table provides a direct link to the run details if you want to investigate further. #### Job details The **Insights** section on the Job details page provides cost and performance data for individual jobs. [![Cost Insights in job details](/img/docs/dbt-platform/cost-insights/cost-insights-job.png?v=2 "Cost Insights in job details")](#)Cost Insights in job details ##### Access To access job details, select your project in the main menu and go to **Orchestration** > **Jobs**. Select the job whose cost data you want to view. ##### Filters For the **Runs** tab, you can use the **Last** filter to view data from the past week, 14 days, or 30 days. For **Cost**, **Usage**, **Query run time**, and **Builds** tabs, you can customize the cost data you want to view by: * **Last**: 30 days, 60 days, 90 days, 6 months, or 1 year * **View**: Daily, Weekly, Monthly * **Assets**: All, Models, Tests ##### Visualization tabs * **Runs**: Displays the success rate and run duration in minutes for recent runs. You can select a time period with options for **Last week**, **Last 14 days**, and **Last 30 days**. * **Cost**: Shows the estimated build cost reduction when using dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed and the reduction in usage from dbt State or state-aware orchestration over the selected timeframe. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated query execution time and the reduction in run duration from dbt State or state-aware orchestration. * **Builds**: Shows the number of assets built versus reused by dbt State or state-aware orchestration. ##### Table view For **Cost**, **Usage**, **Query run time**, and **Builds** tabs, you can access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. --- ### Explore multiple projects dbt platform | Enterprise, Enterprise+ View all the projects and public models in your account (where public models are defined) and gain a better understanding of your cross-project resources and how they're used. On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! The resource-level lineage graph for a project displays cross-project relationships in the DAG. Nodes that represent another dbt project show a project icon to the left of the project name. From a project's **Overview** page, click **View lineage** in the upper right corner to open that project's resource-level lineage graph. In that graph: * View an upstream (parent) project to see the downstream (child) projects that depend on it. * Select a model to reveal its dependent projects in the lineage. * Click on an upstream (parent) project to view the other projects that reference it in the **Relationships** tab, showing the number of downstream (child) projects that depend on them. * This includes all projects listing the upstream one as a dependency in its `dependencies.yml` file, even without a direct `{{ ref() }}`. * Select a project node from a public model to open its detailed lineage graph if you have the [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to do so. Indirect dependencies When viewing a project's lineage, Catalog shows only *directly* [referenced](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) public models. It doesn't show [indirect dependencies](https://docs.getdbt.com/faqs/Project_ref/indirectly-reference-upstream-model.md). If a referenced model in your project depends on another upstream public model, the second-level model won't appear in Catalog, however it will appear in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) lineage view. [![View your cross-project lineage in a parent project and the other projects that reference it by clicking the 'Relationships' tab.](/img/docs/collaborate/dbt-explorer/cross-project-lineage-parent.png?v=2 "View your cross-project lineage in a parent project and the other projects that reference it by clicking the 'Relationships' tab.")](#)View your cross-project lineage in a parent project and the other projects that reference it by clicking the 'Relationships' tab. When viewing a downstream (child) project that imports and refs public models from upstream (parent) projects: * Public models will show up in the lineage graph and you can click on them to view the model details. * Clicking on a model opens a side panel containing general information about the model, such as the specific dbt project that produces that model, description, package, and more. * Double-clicking on a model from another project opens the resource-level lineage graph of the parent project, if you have the permissions to do so. [![View a downstream (child) project that imports and refs public models from the upstream (parent) project.](/img/docs/collaborate/dbt-explorer/cross-project-child.png?v=2 "View a downstream (child) project that imports and refs public models from the upstream (parent) project.")](#)View a downstream (child) project that imports and refs public models from the upstream (parent) project. #### Explore the project-level lineage graph For cross-project collaboration, you can interact with the DAG in all the same ways as described in [Explore your project's lineage](https://docs.getdbt.com/docs/explore/explore-projects.md#project-lineage). You can also interact with it at the project level and view the details. If you have permissions for a project in the account, you can view all public models used across the entire account. However, you can only view full public model details and private models if you have permissions for the specific project where those models are defined. ##### View account-level lineage To view all projects in your account as a lineage graph or list: 1. In **Catalog**, select your project from the sidebar. 2. Click **Account lineage** in the upper right corner. On the account-level lineage graph, each project appears as its own node with the project name and a count of public models it exposes. Arrows between nodes show how projects depend on one another. The top toolbar includes a search bar and **List view** and lineage view toggles. Zoom controls appear in the lower right corner. [![Account-level lineage graph showing cross-project dependencies between projects.](/img/docs/collaborate/dbt-explorer/account-level-lineage.gif?v=2 "Account-level lineage graph showing cross-project dependencies between projects.")](#)Account-level lineage graph showing cross-project dependencies between projects. From the account-level lineage graph, you can also: * Click **List view** to switch to a table of projects and public models. * Double-click a project node to open that project's resource-level lineage graph. ##### Explore a project's resource-level lineage From a project's **Overview** page, click **View lineage** to open the resource-level lineage graph for that project. In this view, you can see: * Sources, seeds, and models in the current project, with connectors between them. * Downstream projects that reference the project's public models, shown as separate project nodes connected by dashed lines. * A search bar, **Lenses**, a **Resource type** filter, a resource-type legend, and zoom controls. [![Resource-level lineage graph with a selected model and the Relationships tab.](/img/docs/collaborate/dbt-explorer/multi-project-overview.gif?v=2 "Resource-level lineage graph with a selected model and the Relationships tab.")](#)Resource-level lineage graph with a selected model and the Relationships tab. When you select a model, a details panel opens on the right: * **General** tab: the model description and metadata fields such as **Project** and **Relation**. * **Columns** tab: each column's name, data type, and description, plus column test results when tests are defined. A **Search for columns** field appears at the top of the tab. * **Relationships** tab: downstream projects that reference the model, listed under **Referenced by**. --- ### Explore your data dbt platform dbt provides a variety of tools for you to explore your data, models, and other resources. Many of the features you'd traditionally use your data warehouse services to explore are at your fingertips in your dbt account. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/cost-insights.md) ###### [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) [Track warehouse compute costs and see realized savings from state-aware orchestration across your dbt projects and models.](https://docs.getdbt.com/docs/explore/cost-insights.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/explore-projects.md) ###### [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) [Interact with dbt Catalog to understand, improve, and leverage your dbt projects.](https://docs.getdbt.com/docs/explore/explore-projects.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/dbt-insights.md) ###### [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) [Query data and perform exploratory data analysis using dbt Insights.](https://docs.getdbt.com/docs/explore/dbt-insights.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) ###### [Documentation](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) [Document your dbt projects so stakeholders, engineers, and analysts can understand your resources and lineage from start to finish.](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) <br /> Some features are only available on [selected plans](https://www.getdbt.com/pricing/). #### Related docs * [dbt plans and pricing](https://www.getdbt.com/pricing/) * [Quickstart guides](https://docs.getdbt.com/docs/get-started-dbt.md) * [Reference material](https://docs.getdbt.com/reference/references-overview.md) --- ### External metadata ingestion Preview ### External metadata ingestion [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ With external metadata ingestion, you can connect directly to your data warehouse, giving you visibility into tables, views, and other resources that aren't defined in dbt with Catalog. External metadata ingestion support Currently, external metadata ingestion is supported for Snowflake only. External metadata credentials enable ingestion of metadata that exists *outside* your dbt runs like tables, views, or cost information; typically at a higher level than what dbt environments access. This is useful for enriching Catalog with warehouse-native insights (for example, Snowflake views or access patterns) and creating a unified discovery experience. These credentials are configured separately from dbt environment credentials and are scoped at the account level, not the project level. #### Prerequisites * Have a dbt account on the [Enterprise or Enterprise+](https://www.getdbt.com/pricing) plan. * You must be an [account admin with permission](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) to edit connections. * The credentials must have [sufficient read-level access to fetch metadata](https://docs.getdbt.com/docs/explore/external-metadata-ingestion.md#configuration-instructions). * Have [**global navigation**](https://docs.getdbt.com/docs/explore/explore-projects.md#catalog-overview) enabled. * Use Snowflake as your data platform. * Support for additional adapters is coming soon! #### Configuration instructions ##### Enable external metadata ingestion 1. Click your account name at the bottom of the left-side menu and click **[Account settings](https://docs.getdbt.com/docs/platform/account-settings.md)**. 2. Under Account information, go to **Settings** and click **Edit** at the top right corner of the page. 3. Select the **Ingest external metadata in dbt Catalog (formerly dbt Explorer)** option (if not already enabled). ##### Configure the warehouse connection 1. Go to **Account settings**. 2. Click **Connections** from the left-hand side panel. 3. Select an existing connection or create a [**New connection**](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md) where you want to ingest metadata from. 4. Scroll to the **Platform metadata credentials** section. Depending on your current configuration, you may see one of the following: * If platform metadata ingestion is not yet configured, the credentials form is immediately in edit mode. * If you previously canceled and see **Add credentials**, click the button to reopen the form. * If you have multiple connections that reference the same account identifier, you will only be prompted to configure platform metadata ingestion for one of them. Other connections using the same account identifier will display a message indicating that platform metadata ingestion is already configured. 5. Enter the necessary credentials. These should have warehouse-level visibility across relevant databases and schemas. 6. Select the **External metadata ingestion** option. * This allows metadata from this connection to populate the Catalog. * *Optional*: Enable additional features such as **Cost insights** in the **Features** section under **Platform metadata credentials**. 7. Under **Catalog filters**, apply filters to restrict which metadata is ingested: * You can filter by **database**, **schema**, **table**, or **view**. * **Note:** To include all databases or schemas, enter `.*` in the **Allow** field. * It is strongly recommended to filter by certain schemas. See [Important considerations](https://docs.getdbt.com/docs/explore/external-metadata-ingestion.md#important-considerations) for more information. * These fields accept CSV-formatted regular expressions: * Example: `DIM` matches `DIM_ORDERS` and `DIMENSION_TABLE` (basic "contains" match). * Wildcards are supported. For example: `DIM*` matches `DIM_ORDERS` and `DIM_PRODUCTS`. #### Required credentials This section sets up the foundational access for dbt in Snowflake. It creates a role (`dbt_metadata_role`) with minimal permissions and a user (`dbt_metadata_user`) dedicated to dbt’s metadata access. This ensures a clear, controlled separation of access, so dbt can read metadata without broader permissions. This setup ensures dbt can read metadata for profiling, documentation, and lineage, without the ability to modify data or manage resources. 1. Create role: ```sql CREATE OR REPLACE ROLE dbt_metadata_role; ``` 2. Grant access to a warehouse to run queries to view metadata: ```sql GRANT USAGE ON WAREHOUSE "<your-warehouse>" TO ROLE dbt_metadata_role; ``` If your warehouse needs to be restarted for metadata ingestion (doesn't have auto-resume enabled), you may need to grant `OPERATE` permissions to the role as well. If you do not already have a user, create a dbt-specific user for metadata access. Replace `<your-password>` with a strong password and `<your-warehouse>` with the warehouse name used above: ```sql CREATE USER dbt_metadata_user DISPLAY_NAME = 'dbt Metadata Integration' PASSWORD = '<your-password>' DEFAULT_ROLE = dbt_metadata_role TYPE = 'LEGACY_SERVICE' DEFAULT_WAREHOUSE = '<your-warehouse>'; ``` 3. Grant the role to the user: ```sql GRANT ROLE dbt_metadata_role TO USER dbt_metadata_user; ``` Note: Use read-only service accounts for least privilege and better auditing. #### Assign metadata access privileges This section outlines the minimum necessary privileges to read metadata from each required Snowflake database. It provides access to schemas, tables, views, and lineage information, ensuring dbt can profile and document your data while preventing any modifications. Replace `your-database` with the name of a Snowflake database to grant metadata access. Repeat this block for each relevant database: ```sql SET db_var = '"<your-database>"'; -- Grant access to view the database and its schemas GRANT USAGE ON DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT USAGE ON ALL SCHEMAS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT USAGE ON FUTURE SCHEMAS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; -- Grant REFERENCES to enable lineage and dependency analysis GRANT REFERENCES ON ALL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON FUTURE TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON ALL EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON FUTURE EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON ALL VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON FUTURE VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; -- Recommended grant SELECT for privileges to enable metadata introspection and profiling GRANT SELECT ON ALL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON ALL EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON ALL VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON ALL DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; -- Grant MONITOR on dynamic tables (e.g., for freshness or status checks) GRANT MONITOR ON ALL DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT MONITOR ON FUTURE DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; ``` #### Grant access to Snowflake metadata This step grants the dbt role (`dbt_metadata_role`) access to Snowflake’s system-level database, enabling it to read usage statistics, query histories, and lineage information required for comprehensive metadata insights. Grant privileges to read usage stats and lineage from Snowflake’s system-level database: ```sql GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE dbt_metadata_role; ``` #### Important considerations The following are best practices for external metadata ingestion, designed to ensure consistent, reliable, and scalable integration of metadata from third-party systems. * Catalog unifies the shared resources between dbt and Snowflake. For example, if there’s a Snowflake table that represents a dbt model, these are represented as a single resource in Catalog. In order for proper unification to occur, the same connection must be used by both the [production environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) and the external metadata ingestion credential. * Avoid duplicates: Use one metadata connection per platform if possible (for example, one for Snowflake, one for BigQuery). * Having multiple connections pointing to the same warehouse can cause duplicate metadata. * Align with dbt environment: To unify asset lineage and metadata, ensure the same warehouse connection is used by both the dbt environment and the external metadata ingestion. * Use filters to limit ingestion to relevant assets: * For example: restrict to production schemas only, or ignore transient/temp schemas. External metadata ingestion runs daily at 5 PM UTC, and also runs immediately each time you update and save credentials. --- ### Fill null values for metrics Understanding and implementing strategies to fill null values in metrics is key for accurate analytics. This guide explains `fill_nulls_with` and `join_to_timespine` to ensure data completeness, helping end users make more informed decisions and enhancing your dbt workflows. ##### About null values You can use `fill_nulls_with` to replace null values in metrics with a value like zero (or your chosen integer). This ensures every data row shows a numeric value. This guide explains how to ensure there are no null values in your metrics: * Use `fill_nulls_with` for `simple`, `cumulative`, and `conversion` metrics * Use `join_to_timespine` and `fill_nulls_with` together for derived and ratio metrics to avoid null values appearing. ##### Fill null values for simple metrics For example, if you'd like to handle days with site visits but no leads, you can use `fill_nulls_with` to set the value for leads to zero on days when there are no conversions. Let's say you have three metrics: * `website_visits` and `leads` * and a derived metric called `leads_to_website_visit` that calculates the ratio of leads to site visits. (Applies to dbt v1.12 and later) On the days when there are no conversions, you can set the value for leads to zero by adding the `fill_nulls_with` parameter to the leads metric: models/website\_visits.yml ```yaml # Define simple metrics within a semantic model models: - name: website_visits_model semantic_model: enabled: true # ... other configs ... metrics: - name: website_visits type: simple agg: count - name: leads type: simple agg: count fill_nulls_with: 0 # This fills null values with zero - name: leads_to_website_visit type: derived expr: leads/website_visits input_metrics: - name: leads - name: website_visits ``` The `website_visits` and `leads` metrics have the following data: | metric\_time | website\_visits | | ------------ | --------------- | | 2024-01-01 | 50 | | 2024-01-02 | 37 | | 2024-01-03 | 79 | | metric\_time | leads | | ------------ | ----- | | 2024-01-01 | 5 | | 2024-01-03 | 8 | * Note that there is no data for `2024-01-02` in the `leads` metric. Although there are no days without visits, there are days without leads. After applying `fill_nulls_with: 0` to the `leads` metric, querying these metrics together shows zero for leads on days with no conversions: | metric\_time | website\_visits | leads | | ------------ | --------------- | ----- | | 2024-01-01 | 50 | 5 | | 2024-01-02 | 37 | 0 | | 2024-01-03 | 79 | 8 | ##### Use join\_to\_timespine for derived and ratio metrics (Applies to dbt v1.12 and later) To ensure you have a complete set of data for every and daily coverage for metrics calculated from other metrics, you can use `join_to_timespine` to fill null values for `derived` and `ratio` metrics. These metrics are built from other metrics (other calculations), not direct aggregations, requiring MetricFlow to have an extra subquery layer to render the metric. The subquery nesting is as follows: * For `derived` and `ratio` metrics, there are three levels of subquery nesting — derived or ratio metric → input metrics → simple metrics with aggregations. * For `simple` and `cumulative` metrics, there are only two levels of subquery nesting — simple or cumulative metric → aggregation. Because `coalesce` isn't applied to the third, subquery layer for `derived` or `ratio` metrics, this means you could still have nulls in the final result set. Note you can use `join_to_timespine` with simple metrics as well if you want to include a row for every date, even if there is no data. ##### Fill null values for derived and ratio metrics To fill null values for derived and ratio metrics, you can link them with a time spine to ensure daily data coverage. As mentioned in [the previous section](#use-join_to_timespine-for-derived-and-ratio-metrics), this is because `derived` and `ratio` metrics take *metrics* as inputs. For example, the following structure leaves nulls in the final results (`leads_to_website_visit` column) because `COALESCE` isn't applied at the third outer rendering layer for the final metric calculation in `derived` metrics: | metric\_time | website\_visits | leads | leads\_to\_website\_visit | | ------------ | --------------- | ----- | ------------------------- | | 2024-01-01 | 50 | 5 | .1 | | 2024-01-02 | 37 | 0 | null | | 2024-01-03 | 79 | 8 | .1 | To display a zero value for `leads_to_website_visit` for `2024-01-02`, you would join the `leads` metric to a time spine model to ensure a value for each day. You can do this by adding `join_to_timespine` to the (Applies to dbt v1.12 and later) simple metric in the `leads` metric configuration: (Applies to dbt v1.12 and later) models/leads.yml ```yaml models: - name: leads_model semantic_model: enabled: true # ... other configs ... metrics: - name: leads type: simple agg: count fill_nulls_with: 0 join_to_timespine: true ``` Once you do this, if you query the `leads` metric after the timespine join, there will be a record for each day and any null values will get filled with zero. | metric\_time | leads | leads\_to\_website\_visit | | ------------ | ----- | ------------------------- | | 2024-01-01 | 5 | .1 | | 2024-01-02 | 0 | 0 | | 2024-01-03 | 8 | .1 | Now, if you combine the metrics in a `derived` metric, there will be a zero value for `leads_to_website_visit` on `2024-01-02` and the final result set will not have any null values. #### FAQs  How to handle null values in derived metrics defined on top of multiple tables For additional examples and discussion on how to handle null values in derived metrics that use data from multiple tables, check out [MetricFlow issue #1031](https://github.com/dbt-labs/metricflow/issues/1031). --- ### Frequently asked questions #### [Accounts](https://docs.getdbt.com/category/accounts.md) [13 items](https://docs.getdbt.com/category/accounts.md) --- ### Get started with Wizard Desktop Beta ### Get started with Wizard Desktop [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Wizard Desktop is dbt Wizard as a native app. View rich data visualizations, data diffs, lineage, and more to get to your results faster than ever before. This page walks you through installing the app, connecting it to the services it needs, and starting your first session. For everything the app can do, refer to [Use Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-use.md). Be warned, the wizard has been known to cast spells . [Download for macOSFor Apple silicon Macs · .dmg](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-aarch64-apple-darwin.dmg)[Download for WindowsFor x64 processors · .exe](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-pc-windows-msvc.exe)[Download for LinuxFor x86\_64, Ubuntu and Debian · .deb](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-unknown-linux-gnu.deb) Need a different build? [Intel Mac (.dmg)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-apple-darwin.dmg) · [Linux ARM64 (.deb)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-aarch64-unknown-linux-gnu.deb) Desktop is powered by the same technology as the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md), with a visual interface to help you follow the work, guide it, and manage several threads at once: * **Parallel chats:** each chat gets its own Git worktree, so several pieces of work run at once without conflicts. * **Visible validation:** see structured results, including a `dbt-compare` view of what your changes did to the data. * **Contextual comments:** hover almost anything to comment on that exact spot, then send several together. * **Charts and tables:** ask a question about your data and see the answer inline. * **Inline impact:** see which models dbt Wizard's changes touch, with diffs you open inline. * **Lineage, file, and metrics browsing:** browse your project right alongside the chat. [![The dbt Wizard home screen](/img/wizard/desktop/wizard-desktop-intro.png?v=2 "The dbt Wizard home screen")](#)The dbt Wizard home screen [![A session in progress, with a chart in the chat and validation results in the side pane](/img/wizard/desktop/wizard-desktop-validation.png?v=2 "A session in progress, with a chart in the chat and validation results in the side pane")](#)A session in progress, with a chart in the chat and validation results in the side pane #### Prerequisites To use Wizard Desktop, you'll need: * A computer using macOS, Linux, or Windows. * A dbt account. You can create a free account when you sign in or use an existing account. * A local copy of your dbt project on your machine if you want dbt Wizard to work on project files. The app scans for existing projects, and you can also select a project folder yourself. * Warehouse credentials if you want dbt Wizard to run your project or query data. You don't need to configure them before opening the app. #### Install Wizard Desktop 1. Download the installer for your operating system: * [macOS, Apple silicon (.dmg)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-aarch64-apple-darwin.dmg) or [macOS, Intel (.dmg)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-apple-darwin.dmg) * [Windows, x64 (.exe)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-pc-windows-msvc.exe) * [Linux, x86\_64 (.deb)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-unknown-linux-gnu.deb) or [Linux, ARM64 (.deb)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-aarch64-unknown-linux-gnu.deb) 2. Open Wizard Desktop and select **Get started** on the welcome screen. 3. Follow the onboarding prompts. When prompted, create a free dbt account or sign in to an existing account. #### Add your dbt projects On the **Scan for your dbt projects** screen, choose how dbt Wizard finds projects: * **Scan for projects**: dbt Wizard searches local Git repos and adds any dbt projects it finds. Allow macOS folder access when prompted so projects are discoverable. * **Add from dbt platform**: connect your dbt platform account. dbt Wizard authenticates, then lists your platform projects with all of them selected by default. Choose which projects to include and, optionally, the folder to clone them into, then select **Clone**. dbt Wizard clones the projects locally so you can work on the files. * **Add a folder instead**: pick a project folder from your computer. * **Start chatting without one**: skip project setup. A project-free chat doesn't use your project files or warehouse connection, so skip to [Start your first session](#start-your-first-session). * Select **Continue** when you're done. #### Choose how to access an AI model On the **Connect an AI model provider** screen, choose how dbt Wizard accesses an AI model: * **Start free trial with dbt-managed keys**: dbt Labs manages the keys and keeps you on current models from OpenAI, Anthropic, and open source. Usage is metered against your dbt usage credits. * **Use my own API key**: bring your own provider credentials with [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) for OpenAI, Anthropic, AWS Bedrock, Azure, Snowflake Cortex, Google Gemini, Databricks, and more. Your provider bills you directly. * **Skip, I'll do this later**: continue without choosing a provider. To configure one later, go to **Settings** → **AI providers**. Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. #### Set up your warehouse connection To run a project or query its data, dbt Wizard needs a warehouse connection. On the home screen, select **Set a connection**. By default dbt Wizard connects through your `profiles.yml`, which may not be configured for the project yet. Choose one of: * **Set warehouse credentials**: enter your credentials and dbt Wizard writes them locally to your [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) and connects directly. Use the same developer credentials you'd use to develop in dbt platform. Refer to [Get started with the Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) and [dbt platform environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) for what to enter. * **Use the dbt platform CLI**: dbt Wizard runs through your dbt platform account, so your credentials stay there. Requires a separate [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) install. If a connection already exists, either through a live dbt platform session or a local `profiles.yml`, dbt Wizard will then use it and you can continue to the [Start your first session](#start-your-first-session) section. #### Start your first session Before sending your first prompt, check the session settings in the prompt area: 1. Confirm the project you selected, or leave it empty for a project-free chat. 2. For a project session, select where dbt Wizard should run the project, for example `default · dev`. If no options appear, [set up your warehouse connection](#set-up-your-warehouse-connection). 3. Choose the model and how much reasoning it should use. If no models are available, configure a provider in **Settings** → **AI providers**. 4. Choose how much dbt Wizard can do before asking for your approval. Available permissions listed in the [next section](#review-and-accept-changes) are **Plan changes**, **Ask for approval**, or **Auto**. 5. Describe what you want dbt Wizard to do, or choose a quick start such as **Audit for test coverage**, **Tell me about this project**, or **Help me refactor a model**. Some things worth asking on a first run: * `Tell me about this project` * `Which models have no tests?` * `Refactor stg_orders and show me what changes in the data` A chat is one conversation about your project. Each project chat works in its own copy of your repo on its own branch, called a Git worktree, so you can keep a few pieces of work going at once without them stepping on each other. Each chat gets a name like `docs-fox`, and its branch and schema are named after it. Refer to [Worktree and schema names](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-use.md#worktree-and-schema-names) for what those names mean. #### Review and accept changes dbt Wizard reads your project's lineage, tests, and metadata, runs tools, and conjures changes as a diff. What happens next depends on your [permission mode](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-use.md#permission-modes): * **Plan changes**: dbt Wizard proposes an approach without editing anything. * **Ask for approval**: dbt Wizard proposes each change and waits. Nothing is written until you accept. * **Auto**: dbt Wizard applies edits and shows a change indicator you can open to review them. Every change lands as a reviewable diff, and you can comment on an exact line, table cell, or chart to send feedback back into the chat. Refer to [Review changes](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-use.md#review-what-changed). #### Update the app When a new version of Wizard Desktop is available, the app generally will tell you a few seconds after launching it. Select **Install** to update, or **Dismiss** to skip it until next launch. To update at any time: 1. Go to **Settings** → **General** → **Application updates**. This shows the version you're on. 2. Select **Check for updates**. 3. If there's a new version, select **Install version** and wait for the install progress to finish. 4. Select **Restart to finish update**. The app won't restart on its own. You can also enable automatic updates by clicking on the **Automatic updates** toggle in the same section. #### Coming from the CLI? It's the same dbt Wizard you already know, running the same engine as the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). The app has a built-in terminal, so you can keep running dbt commands the way you always have, and you can select **Open in** at any time to jump to the same branch in iTerm, Terminal, VS Code, Cursor, or Finder. [Download Wizard Desktop](#install-and-open-wizard-desktop) and install the app. Refer to [the config reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) for the configuration files used by the CLI. | | Wizard Desktop | [CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) | [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) | | ---------------------------------- | ----------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------ | | Where it runs | Native app on your machine | Your terminal | Browser | | Best for | Seeing and steering the work, several threads at once | Scripting and terminal-native workflows | Full development and business-user analytics | | Rendered tables, charts, and diffs | Yes | Plain text in your terminal | Yes | | Parallel work | Yes, each chat gets its own worktree | One session per terminal | Yes | | Chat history | Kept in the app | Resume a past session with `wizard resume` | Kept in your dbt account | | Availability | Public beta, download the app | Public beta, install the CLI | Public preview, in your dbt account | #### What's next * [Use Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-use.md): parallel chats, validation, commenting, diffs, and troubleshooting * [Wizard Desktop settings](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-settings.md): providers, connectors, appearance, shortcuts, and GitHub * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### Global navigation Preview ### Global navigation [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Starter, Enterprise, Enterprise+ Search, explore, and analyze data assets across all your dbt projects and connected metadata sources. Discover cross-project lineage, data discovery, and unified analytics governance. **Plan availability** Global navigation search varies depending on your [dbt platform](https://www.getdbt.com/pricing) plan: * Enterprise plans — Catalog lets you search across all [dbt resources](https://docs.getdbt.com/docs/build/projects.md) (models, seeds, snapshots, sources, exposures, and more) in your account. * Starter plans (single project) — Use global navigation to search and navigate resources within your project #### About Global navigation Global navigation in Catalog lets you search, explore, and analyze data assets across all your dbt projects and connected metadata sources—giving you a unified, account-wide view of your analytics ecosystem. With global navigation, you can: * Search data assets — expand your search by including dbt resources (models, seeds, snapshots, sources, exposures, and more) across your entire account. This broadens the results returned and gives you greater insight into all the assets across your dbt projects. * Explore lineage — explore an interactive map of data relationships across all your dbt projects. It lets you: * View upstream/downstream dependencies for models, sources, and more. * Drill into project and column-level lineage, including multi-project (Mesh) links. * Filter with "lineage lenses" by resource type, materialization, layer, or run status. * Troubleshoot data issues by tracing root causes and downstream impacts. * Optimize pipelines by spotting slow, failing, or unused parts of your DAG. * See recommendations — global navigation offers a project-wide snapshot of dbt health, highlighting actionable tips to enhance your analytics engineering. These insights are automatically generated using dbt metadata and best practices from the project evaluator ruleset. * View model query history — see how often each dbt model is queried in your warehouse, helping you: * Track real usage via successful `SELECT`s (excluding builds/tests) * Identify most/least used models for optimization or deprecation * Guide investment and maintenance with data-driven insights * Track downstream exposures — monitor how your dbt models and sources are used by BI tools, apps, ML models, and reports across all connected projects --- ### Google Sheets dbt platform | Starter, Enterprise, Enterprise+ The Semantic Layer offers a seamless integration with Google Sheets through a custom menu. This add-on allows you to build Semantic Layer queries and return data on your metrics directly within Google Sheets #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and are using dbt v1.6 or higher. If you haven't created metrics yet, start with the [Quickstart for the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md). * You need a Google account with access to Google Sheets and the ability to install Google add-ons. * You have a [dbt Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer). * You have a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to authenticate with from a dbt account. * You must have a dbt Starter or Enterprise-tier [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. If you're using [IP restrictions](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md), ensure you've added [Google’s IP addresses](https://www.gstatic.com/ipranges/goog.txt) to your IP allowlist. Otherwise, the Google Sheets connection will fail. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Installing the add-on 1. Navigate to the [Semantic Layer for Sheets App](https://gsuite.google.com/marketplace/app/foo/392263010968) to install the add-on. You can also find it in Google Sheets by going to [**Extensions -> Add-on -> Get add-ons**](https://support.google.com/docs/answer/2942256?hl=en\&co=GENIE.Platform%3DDesktop\&oco=0#zippy=%2Cinstall-add-ons%2Cinstall-an-add-on) and searching for it there. 2. After installing, open the **Extensions** menu and select **Semantic Layer for Sheets**. This will open a custom menu on the right-hand side of your screen. 3. [Find your](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer) **Host** and **Environment ID** in dbt. * Navigate to **Account Settings** and select **Projects** on the left sidebar. * Select your project and then navigate to the **Semantic Layer** settings. You'll need this to authenticate in Google Sheets in the following step. * You can generate your service token by clicking **Generate service token** within the Semantic Layer configuration page or navigating to **API tokens** in dbt. Alternatively, you can also create a personal access token by going to **API tokens** > **Personal tokens**. [![Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings](/img/docs/dbt-platform/semantic-layer/sl-and-gsheets.png?v=2 "Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings")](#)Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings 4. In Google Sheets, authenticate with your Host, dbt Environment ID, and service or personal token. 5. Start querying your metrics using the **Query Builder**. For more info on the menu functions, refer to [Query Builder functions](#query-builder-functions). To cancel a query while running, press the "Cancel" button. When querying your data with Google Sheets: * It returns the data to the cell you clicked on. * The custom menu operation has a timeout limit of six (6) minutes. * If you're using this extension, make sure you're signed into Chrome with the same Google profile you used to set up the Add-On. Log in with one Google profile at a time as using multiple Google profiles at once might cause issues. * Note that only standard granularities are currently available, custom time granularities aren't currently supported for this integration. #### Query Builder functions The Google Sheets **Query Builder** custom menu has the following capabilities: | Menu items | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metrics | Search and select metrics. | | Group By | Search and select dimensions or entities to group by. Dimensions are grouped by the entity of the semantic model they come from. You may choose dimensions on their own without metrics. | | Time Range | Quickly select time ranges to look at the data, which applies to the main time series for the metrics (metric time), or do more advanced filter using the "Custom" selection. | | Where | Filter your data. This includes categorical and time filters. | | Order By | Return your data order. | | Limit | Set a limit for the rows of your output. | Note: Click the **info** button next to any metric or dimension to see its defined description from your dbt project. ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). ###### Filtering data To use the filter functionality, choose the [dimension](https://docs.getdbt.com/docs/build/dimensions.md) you want to filter by and select the operation you want to filter on. * For categorical dimensions, you can type a value into search or select from a populated list. * For entities, you must type the value you are looking for as we do not load all of them given the large number of values. * Continue adding additional filters as needed with AND and OR. * For time dimensions, you can use the time range selector to filter on presets or custom options. The time range selector applies only to the primary time dimension (`metric_time`). For all other time dimensions that aren't `metric_time`, you can use the "Where" option to apply filters. ###### Other settings If you would like to just query the data values without the headers, you can optionally select the **Exclude column names** box. To return your results and keep any previously selected data below it intact, un-select the **Clear trailing rows** box. By default, we'll clear all trailing rows if there's stale data. [![Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.](/img/docs/dbt-platform/semantic-layer/query-builder.png?v=2 "Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.")](#)Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings. #### Using saved selections Saved selections allow you to save the inputs you've created in the Google Sheets **Query Builder** and easily access them again so you don't have to continuously build common queries from scratch. To create a saved selection: 1. Run a query in the **Query Builder**. 2. Save the selection by selecting the arrow next to the **Query** button and then select **Query & Save Selection**. 3. The application saves these selections, allowing you to view and edit them from the hamburger menu under **Saved Selections**. You can also make these selections private or public. Public selections mean your inputs are available in the menu to everyone on the sheet. Private selections mean your inputs are only visible to you. Note that anyone added to the sheet can still see the data from these private selections, but they won't be able to interact with the selection in the menu or benefit from the automatic refresh. ##### Refreshing selections Set your saved selections to automatically refresh every time you load the addon. You can do this by selecting **Refresh on Load** when creating the saved selection. When you access the addon and have saved selections that should refresh, you'll see "Loading..." in the cells that are refreshing. Public saved selections will refresh for anyone who edits the sheet. What's the difference between saved selections and saved queries? * Saved selections are saved components that you can create only when using the application. * Saved queries, explained in the next section, are code-defined sections of data you create in your dbt project that you can easily access and use for building selections. You can also use the results from a saved query to create a saved selection. #### Using saved queries Access [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), powered by MetricFlow, in Google Sheets to quickly get results from pre-defined sets of data. To access the saved queries in Google Sheets: 1. Open the hamburger menu in Google Sheets. 2. Navigate to **Saved Queries** to access the ones available to you. 3. You can also select **Build Selection**, which allows you to explore the existing query. This won't change the original query defined in the code. * If you use a `WHERE` filter in a saved query, Google Sheets displays the advanced syntax for this filter. **Limited use policy disclosure** The Semantic Layer for Sheet's use and transfer to any other app of information received from Google APIs will adhere to [Google API Services User Data Policy](https://developers.google.com/terms/api-services-user-data-policy), including the Limited Use requirements. #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### Headless mode Beta ### Headless mode [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development dbt Wizard can run without the interactive TUI — useful for scripts, CI pipelines, pre-commit hooks, and any workflow where you want a one-shot result without human-in-the-loop approval. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### `exec` — one-shot prompts Run a single prompt and exit: ```bash wizard exec "list all models with no tests" ``` Pipe input via stdin: ```bash echo "which sources have stale freshness?" | wizard exec - ``` Use `exec` in CI to gate on quality checks: ```bash # Check test coverage before merging wizard exec "are there any models in models/marts/ with no tests?" ``` ##### JSON output For downstream processing, emit a structured JSON event stream: ```bash wizard exec --json "summarize test coverage by schema" > coverage.json ``` With a JSON Schema to constrain the response shape: ```bash wizard exec \ --json \ --output-schema ./schemas/coverage-response.json \ "summarize test coverage by schema" ``` Write the final message to a file: ```bash wizard exec \ --output-last-message ./review-output.md \ "review the changes in this branch for correctness" ``` #### `review` — automated code review Review uncommitted changes: ```bash wizard review --uncommitted ``` Review a branch diff in CI: ```bash wizard review --base main ``` Review a specific commit: ```bash wizard review --commit abc1234 ``` ##### Example: GitHub Actions code review ```yaml - name: dbt Wizard review run: | wizard review \ --base ${{ github.base_ref }} > review.md env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ``` #### Permissions in headless mode In headless `exec` mode, Wizard runs without interactive approval prompts. Pre-grant the sandbox permissions you need: ```bash # Read-only analysis (default — safe for CI) wizard exec "list models with no documentation" # Allow file writes inside the workspace wizard exec -s workspace-write "add not_null tests to all primary keys in staging" # Allow shell commands like dbt compile wizard exec -s workspace-write "compile and validate fct_orders" ``` For read-only analysis tasks (coverage checks, impact queries, documentation gaps), the default permissions are sufficient. For tasks that write files or run dbt commands, pass the appropriate flags explicitly. #### Related docs * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) --- ### Hooks and operations #### Related documentation * [pre-hook & post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start & on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) * [`run-operation` command](https://docs.getdbt.com/reference/commands/run-operation.md) ##### Assumed knowledge * [Project configurations](https://docs.getdbt.com/reference/dbt_project.yml.md) * [Model configurations](https://docs.getdbt.com/reference/model-configs.md) * [Macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) #### Getting started with hooks and operations Effective database administration sometimes requires additional SQL statements to be run, for example: * Creating UDFs * Managing row- or column-level permissions * Vacuuming tables on Redshift * Creating partitions in Redshift Spectrum external tables * Resuming/pausing/resizing warehouses in Snowflake * Refreshing a pipe in Snowflake * Create a share on Snowflake * Cloning a database on Snowflake dbt provides hooks and operations so you can version control and execute these statements as part of your dbt project. #### About hooks Hooks are snippets of SQL that are executed at different times: * `pre-hook`: executed *before* a model, seed or snapshot is built. * `post-hook`: executed *after* a model, seed or snapshot is built. * `on-run-start`: executed at the *start* of `dbt build`, `dbt compile`, `dbt docs generate`, `dbt run`, `dbt seed`, `dbt snapshot`, or `dbt test`. * `on-run-end`: executed at the *end* of `dbt build`, `dbt compile`, `dbt docs generate`, `dbt run`, `dbt seed`, `dbt snapshot`, or `dbt test`. Hooks are a more-advanced capability that enable you to run custom SQL, and leverage database-specific actions, beyond what dbt makes available out-of-the-box with standard materializations and configurations. If (and only if) you can't leverage the [`grants` resource-config](https://docs.getdbt.com/reference/resource-configs/grants.md), you can use `post-hook` to perform more advanced workflows: * Need to apply `grants` in a more complex way, which the dbt `grants` config doesn't (yet) support. * Need to perform post-processing that dbt does not support out-of-the-box. For example, `analyze table`, `alter table set property`, `alter table ... add row access policy`, etc. ##### Examples using hooks You can use hooks to trigger actions at certain times when running an operation or building a model, seed, or snapshot. For more information about when hooks can be triggered, see reference sections for [`on-run-start` and `on-run-end` hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) and [`pre-hook`s and `post-hook`s](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). You can use hooks to provide database-specific functionality not available out-of-the-box with dbt. For example, you can use a `config` block to run an `ALTER TABLE` statement right after building an individual model using a `post-hook`: models/\<model\_name>.sql ```sql {{ config( post_hook=[ "alter table {{ this }} ..." ] ) }} ``` ##### Calling a macro in a hook You can also use a [macro](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) to bundle up hook logic. Check out some of the examples in the reference sections for [on-run-start and on-run-end hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) and [pre- and post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). models/\<model\_name>.sql ```sql {{ config( pre_hook=[ "{{ some_macro() }}" ] ) }} ``` models/properties.yml ```yaml models: - name: <model_name> config: pre_hook: - "{{ some_macro() }}" ``` dbt\_project.yml ```yaml models: <project_name>: +pre-hook: - "{{ some_macro() }}" ``` #### About operations Operations are [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) that you can run using the [`run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) command. As such, operations aren't actually a separate resource in your dbt project — they are just a convenient way to invoke a macro without needing to run a model. Explicitly execute the SQL in an operation Unlike hooks, you need to explicitly execute a query within a macro, by using either a [statement block](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks.md) or a helper macro like the [run\_query](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) macro. Otherwise, dbt will return the query as a string without executing it. This macro performs a similar action as the above hooks: macros/grant\_select.sql ```sql {% macro grant_select(role) %} {% set sql %} grant usage on schema {{ target.schema }} to role {{ role }}; grant select on all tables in schema {{ target.schema }} to role {{ role }}; grant select on all views in schema {{ target.schema }} to role {{ role }}; {% endset %} {% do run_query(sql) %} {% do log("Privileges granted", info=True) %} {% endmacro %} ``` To invoke this macro as an operation, execute `dbt run-operation grant_select --args '{role: reporter}'`. ```text $ dbt run-operation grant_select --args '{role: reporter}' Running with dbt=1.6.0 Privileges granted ``` Full usage docs for the `run-operation` command can be found [here](https://docs.getdbt.com/reference/commands/run-operation.md). #### Additional examples These examples from the community highlight some of the use-cases for hooks and operations! * [In-depth discussion of granting privileges using hooks and operations, for dbt v1 versions prior to 1.2](https://discourse.getdbt.com/t/the-exact-grant-statements-we-use-in-a-dbt-project/430) * [Staging external tables](https://github.com/dbt-labs/dbt-external-tables) * [Performing a zero copy clone on Snowflake to reset a dev environment](https://discourse.getdbt.com/t/creating-a-dev-environment-quickly-on-snowflake/1151/2) * [Running `vacuum` and `analyze` on a Redshift warehouse](https://github.com/dbt-labs/redshift/tree/0.2.3/#redshift_maintenance_operation-source) * [Creating a Snowflake share](https://discourse.getdbt.com/t/how-drizly-is-improving-collaboration-with-external-partners-using-dbt-snowflake-shares/1110) * [Unloading files to S3 on Redshift](https://github.com/dbt-labs/redshift/tree/0.2.3/#unload_table-source) * [Creating audit events for model timing](https://github.com/dbt-labs/dbt-event-logging) * [Creating UDFs](https://discourse.getdbt.com/t/using-dbt-to-manage-user-defined-functions/18) --- ### How dbt Wizard works dbt Wizard helps teams develop, troubleshoot, harden, and ship trusted dbt projects faster and with less risk. Built for governed data development in dbt, dbt Wizard understands your project, routes to the right dbt tools, and validates work with awareness of warehouse operations. Use it to investigate failed runs, debug models, assess impact, make changes, and ship trusted data work in one place. Most of how dbt Wizard works is the same in [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md), [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md), and in the [terminal CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md). The following sections explain shared behavior first, then call out what differs in each environment. #### dbt Information Schema dbt Wizard uses the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) — a contracted interface into your project’s metadata that's ready before your first prompt. Think of it like a map of your whole city: dbt Wizard knows how everything connects before it starts, rather than walking every street to figure out the layout. (No crystal ball required.) That metadata gives dbt Wizard four capabilities that aren't possible from file-reading alone: | Capability | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Impact analysis | When you ask "what breaks if I change this column?", dbt Wizard returns the exact set of downstream models, metrics, tests, and exposures affected — instantly, from the dbt Information Schema. Column-level lineage is tracked, so dbt Wizard knows not just which models reference a table, but which ones reference that specific column. | | Health checks | dbt Wizard knows which models have tests, which don't, which have stale data, which have failing contracts, and which had recent run failures — as a structured dataset. You can ask "what's unhealthy in this part of my DAG?" and get a precise answer without running anything. | | Data profiling | dbt Wizard can profile your data — row counts, column distributions, null rates — and use that context when deciding how to build or refactor a model. It can reason about your data without materializing models or running expensive queries. | | Validation planning | dbt Wizard uses project metadata to identify affected resources and choose relevant checks. In the CLI, you control the depth of structured validation before commands run. | dbt Wizard builds and updates this metadata from dbt artifacts. In dbt platform, project state comes from your connected development environment. In the terminal, run `dbt parse`, `dbt compile`, or `dbt build` before a session so dbt Wizard has your latest local project state. dbt version shown in the status panel The dbt version dbt Wizard displays comes from your project's manifest (`target/manifest.json`), not the `dbt` executable on your `PATH`. If you generated the manifest with a different binary, dbt Wizard reports that version until you recompile. Run `dbt compile` (or `dbt parse`/`dbt build`) with your intended dbt to refresh the manifest and the displayed version. #### Validation mechanics Validation can combine static checks, dbt commands, development builds, downstream impact analysis, and development-to-production comparisons. The checks that run depend on the surface, available tools, project state, permissions, and the validation depth you approve. In dbt Wizard CLI or [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md), choose light, medium, heavy, or skipped validation. Medium validation is the default: * Light validation focuses on syntax, linting where supported, tests, and code review without materializing the changed models. * Medium validation adds development materialization and downstream checks. * Heavy validation adds explicit expectations and development-to-production comparisons when the required relations are available. dbt Wizard reports failures and checks it couldn't complete. A passing check doesn't remove the need to review business logic, and a skipped check should remain visible in your review. For a complete procedure and the approval points for warehouse commands, refer to [Validate dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md). #### Tools and capabilities dbt Wizard takes action through a defined set of tools — from reading files to running dbt commands — so you can see exactly what it's doing and why. | Tool | Purpose | Where available | | --------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- | | File read/write | Read files and propose edits as diffs | Platform, CLI, and Desktop | | Project queries | Query lineage, tests, metadata, metrics, and run results | Platform, CLI, and Desktop | | dbt commands | Run commands like `dbt compile`, `dbt build`, and `dbt test` | Platform, CLI, and Desktop | | Bash | Execute shell commands in your project directory | CLI and Desktop | | Web search | Look up dbt docs and troubleshooting information | Platform, CLI, and Desktop | | MCP | Access connected MCP servers | CLI and Desktop (add servers); platform includes built-in docs and platform context | dbt Wizard never runs destructive commands (such as `dbt build --full-refresh`, `dbt run --full-refresh`, or `git reset --hard`) without approval. #### Skills and memories dbt Wizard supports reusable skills and memories that help it apply your team's conventions across sessions. | Feature | Description | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Skills | Standardize repeatable workflows and best practices. For example, a skill can tell dbt Wizard how your team structures staging models, writes tests, or documents columns. | | Memories | Store project-specific context that should carry between sessions. For example, dbt Wizard can remember that your project uses `customer_id` as the standard customer key or that finance models require extra review. | dbt Wizard automatically loads skills from your project and local directories in both the platform and the CLI. In the platform, start a new chat after adding or changing skills so they are picked up. Refer to the [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) page for more details. #### In dbt platform Use dbt Wizard in [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) from the home app or Studio IDE. [![How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.](/img/docs/dbt-platform/wizard-architecture-platform.png?v=2 "How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.")](#)How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider. Your prompt, project metadata, and any query results you approve are sent to the AI provider over TLS. Warehouse credentials are never shared, queries run only after you approve them, and dbt Labs-managed providers can't retain your data or train on it. ##### Approval and review By default, dbt Wizard keeps you in control before it changes your project or runs commands. In the platform: * dbt Wizard shows file edits as diffs for you to accept or reject before anything is persisted * dbt commands ask for confirmation before running * In the home app and Studio IDE, toggle **Ask for approval** or **Edit files automatically** to choose how much freedom the agent has per session There is no bash sandbox in the platform — shell access is not exposed the way it is in the CLI. ##### Sessions and conversations In the platform, a session is a saved conversation in the dbt Wizard panel or home app: * Ask follow-up questions without restating full context * Continue an earlier investigation, refactor, or validation task * Review prior prompts, responses, and proposed changes in the conversation list Start a new session with **Start new dbt Wizard chat** in the panel. Chat history is retained for 90 days. Refreshing the same browser tab keeps your active session; opening a new tab starts empty. Start a new chat for each task Use a separate chat for each discrete unit of work, such as building one model, debugging one job failure, or reviewing a change. A focused conversation keeps the context relevant to the task, so dbt Wizard doesn't carry over assumptions from unrelated work. Return to an existing chat only when you're continuing that same task. For Studio-specific behavior and availability, refer to [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). #### Locally (CLI and Desktop) Use the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) or [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) for local development. Everything in this section applies to both, since the app runs the same engine as the CLI. ##### Connections and authentication (MCP) dbt Wizard can connect to MCP servers from the CLI, including the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md), for access to platform APIs, Semantic Layer metadata, and cross-project context. For the complete setup (like supported server types, configuration keys, authentication, and examples), refer to [Use MCP servers with the dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md). MCP connector support for Wizard Desktop is coming soon. To connect to an MCP server, run the following commands, replacing `MCP_NAME` with a name of your choice and `YOUR_MCP_URL` with the URL of the MCP server: ```bash wizard mcp add MCP_NAME --url https://YOUR_MCP_URL wizard mcp login MCP_NAME ``` You can see more options by running `wizard mcp --help`. For example, to connect to the dbt MCP server, replace `DBT_MCP_ENDPOINT` with your endpoint and run: ```bash wizard mcp add dbt --url DBT_MCP_ENDPOINT ``` You should see output similar to `Added global MCP server 'dbt'.` Then authenticate: ```bash wizard mcp login dbt ``` The dbt MCP server reads its connection settings — such as `DBT_HOST`, `DBT_TOKEN`, `DBT_PROJECT_DIR`, `DBT_PATH`, `DBT_PROD_ENV_ID`, and `DBT_ACCOUNT_ID` — from environment variables, typically a `.env` file in your dbt project root. If dbt Wizard can't reach the server, confirm these values are set, then restart `wizard` so it picks up the changes. For the full list of variables and an example `.env` file, refer to [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) and the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). ##### Deferral and state When you work on part of a project, dbt Wizard uses [deferral](https://docs.getdbt.com/reference/node-selection/defer.md) so it can reuse models that are already built elsewhere (for example, in production) instead of rebuilding everything. This saves time and warehouse cost. How deferral is handled depends on the mode set up in `wizard_config.toml` for your project: | `deferral.mode` value | Behavior | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `"wizard"` | dbt Wizard handles deferral for you. You tell dbt Wizard which target from your `profiles.yml` to defer to (it tries to detect one automatically when you first set up the project). dbt Wizard then compiles that target and reuses its models for any upstream models you haven't built yourself. | | `"fusion_cloud"` | dbt platform handles deferral against your connected environment, so dbt Wizard doesn't manage local state. | | `"cloud_cli"` | The dbt platform CLI handles credentials and deferral through dbt platform, so dbt Wizard doesn't manage local state or inject deferral flags. | | `"dbt_state"` | dbt State or run cache handles deferral, so dbt Wizard skips its own production compile. | | `"manual"` | You maintain the deferral manifest path manually. | | `"disabled"` | Deferral is disabled for the project. | For more about dbt State, refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). The per-project `favor_state` setting defaults to `true`. With favor-state on, deferred relations take precedence. Set `favor_state = false` when you want dbt to use relations you have already built in development and fall back to the deferred environment for relations that aren't available there. dbt Wizard stores the deferral mode for each project in `wizard_config.toml` under `deferral.mode`. For a complete setup and verification procedure, refer to [Developing with production deferral](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-6-production-deferral.md). ##### Approval and sandboxing By default, the CLI keeps you in control before it changes your project or runs commands. In the terminal: * dbt Wizard shows file changes as diffs so you can review them * Commands that need permission under the active approval policy request confirmation before running * The active sandbox profile limits where shell commands can read and write. dbt Wizard shows the active profile when the session starts. Choose the sandbox profile and approval behavior that match the task: ```bash # Restrict shell commands to read-only access. wizard --sandbox read-only # Never prompt for approval during this session. # Useful for trusted tasks where you want Wizard to iterate without stopping. wizard --ask-for-approval never # Allow shell commands to write inside your workspace directory. # Useful for commands that generate files. wizard --sandbox workspace-write ``` Use relaxed settings when you want dbt Wizard to move faster on a scoped task you trust, such as a large refactor, generating documentation, or applying repetitive test updates. The tradeoff is that dbt Wizard has more freedom to act before you review each step, so use these settings in a clean working tree or feature branch. Approval settings apply to the current session. They are not scoped per action type in the command line flags shown above; for example, `--ask-for-approval never` relaxes prompts broadly for that session. To set defaults that persist across sessions or apply only to a specific project, use the [configuration file](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md). You can view more options by running: ```bash wizard --help ``` ##### Sessions Locally, a session is a saved conversation and task history from a previous run on your machine. Sessions help you return to earlier work, continue a multi-step task, or review what the agent did in a past interaction. Within a session, you can: * Ask follow-up questions without restating the full context * Continue work on an earlier investigation, refactor, or validation task * Review prior prompts, responses, tool calls, and proposed changes Resume a previous session with: ```bash wizard resume # choose from a list of saved sessions wizard resume --last # resume the most recent session ``` Local sessions are saved on your machine, in the CLI and in Wizard Desktop alike. This is separate from platform conversations, which are stored in your dbt platform account. Start a new session for each task Use a separate session for each discrete unit of work, such as building one model, debugging one job failure, or reviewing one change. Resume a previous session only when you're continuing the same task. In the CLI, type `/new` to start fresh without leaving the TUI, or type `/compact` to summarize the conversation if a long session for a single task approaches the context limit. For more information, refer to [Slash commands](https://docs.getdbt.com/docs/dbt-ai/wizard-slash-commands.md). #### Related docs * [dbt Wizard overview](https://docs.getdbt.com/docs/platform/wizard-overview.md) * [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows --- ### How to find your dbt MCP IDs Several dbt MCP environment variables and headers require numeric IDs from your dbt platform account. This guide shows exactly where to find each one. Use numeric IDs, not full URLs ID variables expect integers, not URLs. A common mistake is copying the URL from your browser address bar. ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` #### DBT\_HOST (account hostname) Your hostname is the domain you use to access dbt platform. Both `cloud.getdbt.com` and `https://cloud.getdbt.com` are accepted. 1. Log in to your dbt platform account. 2. Go to **Account settings**. 3. Copy the **Access URL** value. | Account type | Example Access URL | DBT\_HOST value | | -------------------------------- | ------------------------- | ------------------------- | | US multi-tenant | `cloud.getdbt.com` | `cloud.getdbt.com` | | Accounts with a subdomain prefix | `abc123.us1.dbt.com` | `abc123.us1.dbt.com` | | Single-tenant | `your-company.getdbt.com` | `your-company.getdbt.com` | For more information on regions and hosting, refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). #### DBT\_ACCOUNT\_ID (account ID) 1. Log in to your dbt platform account. 2. Go to **Account settings**. 3. The account ID is displayed on the settings page, or you can find it in the URL: `https://cloud.getdbt.com/settings/accounts/ACCOUNT_ID/`. So for example, if the URL is `https://YOUR_ACCESS_URL/settings/accounts/12345/`, the account ID would be `12345`. Alternatively, see [Finding your user and account IDs](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for additional methods. #### DBT\_PROD\_ENV\_ID (production environment ID) 1. Log in to your dbt platform account. 2. Go to **Orchestration** → **Environments**. 3. Click on your production environment. 4. The environment ID is in the URL: `https://cloud.getdbt.com/deploy/ACCOUNT_ID/projects/PROJECT_ID/environments/ENVIRONMENT_ID`. So for example, if the URL is `https://YOUR_ACCESS_URL/deploy/12345/projects/67890/environments/54321`, the environment ID would be `54321`. Copy only the number at the end — for example, `54321`. #### DBT\_DEV\_ENV\_ID (development environment ID) Follow the same steps as for `DBT_PROD_ENV_ID`, but click on your development environment instead of production. This variable is required for `execute_sql`. If you don't have a dedicated development environment, you can use your production environment ID here, though a separate development environment is recommended. #### DBT\_USER\_ID (user ID) 1. Log in to your dbt platform account. 2. Go to **Account settings** → **Profile** (or click your profile/avatar). 3. Your user ID is in the URL: `https://cloud.getdbt.com/settings/profile/USER_ID`. So for example, if the URL is `https://YOUR_ACCESS_URL/settings/profile/123`, the user ID would be `123`. Copy only the number. Alternatively, see [Finding your user and account IDs](https://docs.getdbt.com/faqs/Accounts/find-user-id.md). #### DBT\_TOKEN (access token) The following sections explain how to find your Personal Access Token (PAT) and Service token. ##### Personal Access Token (PAT) Required for `execute_sql`. Tied to your personal account. 1. Go to **Account settings** → **API tokens** → **Personal tokens**. 2. Click **+ New token**, give it a name, and copy the token value. 3. Store it somewhere safe — you can't view it again after closing the dialog. ##### Service token Used for shared or team setups. Better for CI/automation. 1. Go to **Account settings** → **API tokens** → **Service tokens**. 2. Click **+ New token**, assign the required permissions, and copy the token value. 3. Assign the `Developer` permission set, scoped either to the project that owns the environment you'll use for `x-dbt-prod-environment-id` or to all projects. A token without this permission set is rejected with a `401` when the MCP client connects. No other permission sets are required. `Developer` can only be assigned to a service token on Enterprise and Enterprise+ plans; on other plans, use [OAuth](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). For more information, see [User tokens (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) and [Service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). --- ### Hybrid setup dbt platform | Enterprise+ Set up Hybrid projects to upload dbt v1 artifacts into dbt for better collaboration and visibility. Available in public preview Hybrid projects is available in public preview to [dbt Enterprise accounts](https://www.getdbt.com/pricing). #### Set up Hybrid projects In a hybrid project, you use dbt v1 locally and can upload artifacts of that dbt v1 project to dbt for central visibility, cross-project referencing, and easier collaboration. This setup requires connecting your dbt v1 project to a dbt project and configuring a few environment variables and access settings. Follow these steps to set up a dbt Hybrid project and upload dbt v1 artifacts into dbt: * [Make dbt models public](#make-dbt-models-public) (optional) * [Create hybrid project](#create-hybrid-project) * [Generate service token and artifact upload values](#generate-service-token-and-artifact-upload-values) * [Configure dbt v1 project and upload artifacts](#configure-dbt-project-and-upload-artifacts) * [Review artifacts in dbt](#review-artifacts-in-dbt-cloud) Make sure to enable the hybrid projects toggle in dbt’s **Account settings** page. ##### Make dbt models public (optional) This step is optional and only needed if you want to share your dbt v1 models with other dbt projects using the [cross-project referencing](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) feature. Before connecting your dbt v1 project to a dbt project, make sure models that you want to share have `access: public` in their model configuration. This setting makes those models visible to other dbt projects for better collaboration, such as [cross-project referencing](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref). 1. The easiest way to set this would be in your `dbt_project.yml` file, however you can also set this in the following places: * `dbt_project.yml` (project-level) * `properties.yml` (for individual models) * A model's `.sql` file using a `config` block Here's an example using a `dbt_project.yml` file where the marts directory is set as public so they can be consumed by downstream tools: dbt\_project.yml ```yaml models: define_public_models: # This is my project name, remember it must be specified marts: +access: public ``` 2. After defining `access: public`, rerun a dbt execution in the dbt v1 command line interface (CLI) (like `dbt run`) to apply the change. 3. For more details on how to set this up, see [access modifier](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers) and [`access` config](https://docs.getdbt.com/reference/resource-configs/access.md). ##### Create hybrid project Create a hybrid project in dbt to allow you to upload your dbt v1 artifacts to dbt. A [dbt account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#permission-sets) should perform the following steps and share the artifacts information with a dbt v1 user: 1. To create a new project in dbt, navigate to **Account home**. 2. Click on **+New project**. 3. Fill out the **Project name**. Name the project something that allows you to recognize it's a dbt v1 project. * You don't need to set up a [data warehouse](https://docs.getdbt.com/docs/supported-data-platforms.md) or [Git connection](https://docs.getdbt.com/docs/platform/git/configure-git.md), however to upgrade the hybrid project to a full dbt project, you'd need to set up data warehouse and Git connection. 4. Select the **Advanced settings** toggle and then select the **Hybrid development** checkbox. Click **Continue**. * The hybrid project will have a visible **Hybrid** indicator in the project list to help you identify it. [![Hybrid project new project](/img/docs/deploy/hp-new-project.jpg?v=2 "Hybrid project new project")](#)Hybrid project new project 5. After creating a project, create a corresponding [production environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) and click **Save**. You will need to create a placeholder [profile](https://docs.getdbt.com/docs/platform/about-profiles.md) and assign it to the environment to save. 6. (Optional) To update an existing dbt project to a hybrid project, navigate to **Account settings** and then select the **Project**. Click **Edit** and then check the **Hybrid development** checkbox. [![Hybrid project for an existing project](/img/docs/deploy/hp-existing-project.jpg?v=2 "Hybrid project for an existing project")](#)Hybrid project for an existing project ##### Generate service token and artifact upload values A dbt admin should perform these steps to generate a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#enterprise-plans-using-service-account-tokens) (with both **Job Runner** *and* **Job Viewer** permissions) and copy the values needed to configure a dbt v1 project so it's ready to upload generated artifacts to dbt. The dbt admin should share the values with a dbt v1 user. 1. Go to the Hybrid project environment you created in the previous step by navigating to **Deploy** > **Environments** and selecting the environment. 2. Select the **Artifact upload** button and copy the following values, which the dbt v1 user will need to reference in their dbt v1's `dbt_project.yml` configuration: * **[Tenant URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md)** * **Account ID** * **Environment ID** * **Create a service token** * dbt creates a service token with both **Job Runner** *and* **Job Viewer** permissions. * Note if you don't see the **Create service token** button, it's likely you don't have the necessary permissions to create a service token. Contact your dbt admin to either get the necessary permissions or create the service token for you. [![Generate hybrid project service token](/img/docs/deploy/hp-artifact-upload.png?v=2 "Generate hybrid project service token")](#)Generate hybrid project service token 3. Make sure to copy and save the values as they're needed to configure your dbt v1 project in the next step. Once the service token is created, you can't access it again. ##### Configure dbt project and upload artifacts Once you have the values from the previous step, you can prepare your dbt v1 project for artifact upload by following these steps: 1. Check your dbt version by running `dbt --version` and you should see the following: ```bash Core: - installed: 1.11.0 - latest: 1.11.0 - Up to date! ``` 2. If you don't have the [latest supported version](https://docs.getdbt.com/docs/dbt-versions.md), [upgrade](https://docs.getdbt.com/docs/local/install-dbt.md?version=1#change-dbt-core-versions) your dbt v1 project by running `python -m pip install --upgrade dbt-core`. 3. Set the following environment variables in your dbt v1 project by running the following commands in the CLI. Replace the `your_account_id`, `your_environment_id`, and `your_token` with the actual values in the [previous step](#generate-service-token-and-artifact-upload-values). ```bash export DBT_CLOUD_ACCOUNT_ID=your_account_id export DBT_CLOUD_ENVIRONMENT_ID=your_environment_id export DBT_CLOUD_TOKEN=your_token export DBT_ENGINE_UPLOAD_TO_ARTIFACTS_INGEST_API=True ``` * Set the environment variables in whatever way you use them in your project. * To unset an environment variable, run `unset environment_variable_name`, replacing `environment_variable_name` with the actual name of the environment variable. 4. In your local dbt v1 project, add the following items you copied in the [previous section](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#enable-artifact-upload) to the dbt v1's `dbt_project.yml` file: * `tenant_hostname` ```yaml name: "jaffle_shop" version: "3.0.0" require-dbt-version: ">=1.5.0" ....rest of dbt_project.yml configuration... dbt-cloud: tenant_hostname: cloud.getdbt.com # Replace with your Tenant URL ``` 5. Once you set the environment variables using the `export` command in the same dbt CLI session, you can execute a `dbt run` in the CLI. ```bash dbt run ``` To override the environment variables set, execute a `dbt run` with the environment variable prefix. For example, to use a different account ID and environment ID: ```bash DBT_CLOUD_ACCOUNT_ID=1 DBT_CLOUD_ENVIRONMENT_ID=123 dbt run ``` 6. After the run completes, you should see a `Artifacts uploaded successfully to artifact ingestion API: command run completed successfully` message and a run in dbt under your production environment. ##### Review artifacts in the dbt platform Now that you've uploaded dbt v1 artifacts into the dbt platform and executed a `dbt run`, you can view the artifacts job run: 1. Navigate to **Deploy** 2. Click on **Jobs** and then the **Runs** tab. 3. You should see a job run with the status **Success** with a `</> Artifact ingestion` indicator. 4. Click on the job run to review the logs to confirm a successfully artifacts upload message. If there are any errors, resolve them by checking out the debug logs. [![Hybrid project job run with artifact ingestion](/img/docs/deploy/hp-artifact-job.jpg?v=2 "Hybrid project job run with artifact ingestion")](#)Hybrid project job run with artifact ingestion #### Benefits of using Hybrid projects Now that you've integrated dbt v1 artifacts with your dbt project, you can now: * Collaborate with dbt users by enabling them to visualize and perform [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) to dbt models that live in dbt v1 projects. * (Coming soon) New users interested in the [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) can build off of dbt models already created by a central data team in dbt v1 rather than having to start from scratch. * dbt v1 users can navigate to [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and view their models and assets. To view Catalog, you must have a [read-only seat](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). --- ### Install dbt OSS Local development (Applies to dbt v2.0 and later) The open-source v2 foundation is licensed under Apache 2.0. Most users don't need this page — [install dbt normally](https://docs.getdbt.com/docs/local/install-dbt.md) with the standard instructions. This page is for organizations that require the Apache 2.0 codebase specifically. Use an isolated environment Use `uv tool` to install `dbt-oss` in its own isolated environment. Avoid installing `dbt-oss` in the same environment as `dbt` or `dbt-core` because they all use the same entry point. If you already have `dbt` installed, refer to [Run dbt-oss alongside other dbt installs](#run-dbt-oss-alongside-other-dbt-installs). #### Install Before installing dbt OSS, make sure you have [uv](https://docs.astral.sh/uv/getting-started/installation/) installed on your machine. Install `dbt-oss` with `uv tool`: ```shell uv tool install dbt-oss ``` Confirm the installed version begins with `2.`: ```shell dbt --version ``` For adapter install details, refer to the [`dbt` repository](https://github.com/dbt-labs/dbt). #### Run `dbt-oss` alongside other dbt installs If you have `dbt` installed and want to run `dbt-oss` alongside it, use `uvx` with a shell alias to avoid conflicts. Using `uvx` ensures dbt OSS runs in its own isolated environment without interfering with your existing dbt installation. Add the following to your shell profile. This example uses `dbto`, but you can choose a different name: ```shell # ~/.zshrc or ~/.bashrc alias dbto='uvx --from dbt-oss dbt' ``` Reload your shell profile: ```shell source ~/.zshrc # or source ~/.bashrc ``` You can then use `dbto <command>` (or your chosen alias) instead of `dbt <command>` for dbt OSS invocations. #### What's included * The open-source, Rust-based dbt runtime. * The dbt project language and DAG semantics. * The standard dbt command set (`run`, `build`, `test`, `compile`, `parse`, and more). #### What's not included The [standard dbt install](https://docs.getdbt.com/docs/local/install-dbt.md) gives you dbt v2, which adds the following on top of the open source layer: * SQL comprehension and static analysis * LSP features (autocomplete, hover info, inline errors) * `dbt lint` and error diagnostics * dbt VS Code extension integration For the full picture of what you get with dbt, refer to [v2 availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md). #### Contributing To contribute, refer to the [`dbt` repository](https://github.com/dbt-labs/dbt) and its [CONTRIBUTING guide](https://github.com/dbt-labs/dbt/blob/HEAD/CONTRIBUTING.md), or ask in the [dbt Community](https://docs.getdbt.com/community/resources/getting-help.md). #### License dbt OSS is licensed under Apache 2.0. Refer to the [LICENSE file](https://github.com/dbt-labs/dbt/blob/HEAD/LICENSE) in the repository. Refer to [dbt licensing](https://docs.getdbt.com/docs/dbt-licensing.md?version=2.0) for more info. #### Related * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) (standard install) * [Upgrade to v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [`dbt` repository on GitHub](https://github.com/dbt-labs/dbt) --- ### Install dbt Wizard CLI Beta ### Install dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Install the dbt Wizard CLI from your terminal for agentic and governed data development in dbt. This guide explains how to install, verify, update, and uninstall the dbt Wizard CLI on your local machine. Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. You can run the dbt Wizard CLI locally from any project on any dbt engine. Be warned, the wizard has been known to cast spells . #### Prerequisites * macOS, Windows, or Linux * A dbt project with a built `target/` directory (`dbt parse`, `dbt compile`, or `dbt build`) * Access to a supported AI provider. You can use a managed provider in dbt, or configure [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) with your own provider credentials. ##### Supported AI models dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). New to the terminal? If you've never used the terminal before, check out the [terminal guide](https://docs.getdbt.com/guides/terminal-guide.md) for some helpful tips to help you get started! #### Install and set up dbt Wizard CLI 12345 View all stepsNext 1 Install the dbt Wizard CLI Run the install script for your operating system: macOS/Linux: ```bash curl -fsSL https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.sh | sh ``` Windows (PowerShell): ```powershell irm https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.ps1 | iex ``` This installs dbt Wizard to `/usr/local/bin/wizard`, along with the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) that powers dbt Wizard's project-aware answers. For install and update details, refer to [Install dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md); to remove them, refer to [Uninstall](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md#uninstall). #### Update Run the following command to update dbt Wizard to the latest version: ```bash wizard update ``` Prefer to learn by video? Check out the free [dbt Wizard video course](https://learn.getdbt.com/courses/dbt-wizard) on dbt Learn! #### Uninstall 1. Run the built-in uninstall command. It lists every binary, config, and data directory it's about to remove, then asks you to confirm (`Proceed? [Y/N]`) before deleting anything: ```shell wizard system uninstall ``` Uninstalling Wizard Removing `~/.dbt/wizard` deletes your local config, logs, and cache, and can't be undone. Your dbt profiles (`~/.dbt/`) and dbt projects aren't part of dbt Wizard and won't be touched. 2. Confirm the binary is deleted by checking your system path: ```bash which wizard ``` If no output path is returned, dbt Wizard is successfully uninstalled. #### Telemetry dbt Wizard collects anonymous product telemetry to improve the AI agent experience, understand usage patterns, optimize performance, and attribute compute costs without capturing your code, queries, prompts, responses, or file contents. For details about what is collected, what is not collected, and how to opt out of client telemetry, refer to [dbt Wizard CLI data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md). Best practices for using dbt Wizard Once you're set up, refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows on real project tasks. #### Related docs * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md): Install dbt Wizard and start a local terminal session * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md): Manage your API key and choose an AI model * [Command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md): Full reference for all `wizard` subcommands and global flags * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md): Realistic analytics engineering scenarios * [Migrate from another AI agent](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md): Migrate from another AI agent to dbt Wizard * [CLI data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md): What dbt Wizard CLI collects and how to opt out * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### Install the dbt VS Code extension Local development The dbt extension — available for [VS Code and Cursor](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt\&ssr=false#overview), and [Windsurf](https://open-vsx.org/extension/dbtLabsInc/dbt) — makes dbt development smoother and more efficient. dbt v1 and v2 both support the extension — refer to [Version compatibility](https://docs.getdbt.com/docs/about-dbt-extension.md#version-compatibility) for which features need which setup. note This is the only official dbt Labs VS Code extension. Other extensions *can* work alongside the dbt VS Code extension, but they aren't tested or supported by dbt Labs. #### Prerequisites To use the extension, you need the following: | Prerequisite | Details | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Project files** | You need a `profiles.yml` file. You may also need a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file for some dbt platform features or credential-based workflows. You don't need a dbt platform project to use the extension. | | **Editor** | [VS Code](https://code.visualstudio.com/), [Cursor](https://www.cursor.com/en), or [Windsurf](https://windsurf.com/editor). | | **Operating system** | macOS, Windows, or Linux. | | **Local configuration** (optional) | [Configure the extension](https://docs.getdbt.com/docs/configure-dbt-extension.md) to mirror your dbt environment locally and set any environment variables required by your project. | | **Project migration support** (optional) | The extension has [dbt-autofix](https://github.com/dbt-labs/dbt-autofix) built in, so you can fix deprecations from the **Problems** pane or in a single pull request. You can also use the **Migrate dbt v1 to dbt v2** agent skill or [run dbt-autofix](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=5#what-the-autofix-tool-handles) yourself. | #### Install the extension 1. In your editor, open the **Extensions** tab and search for `dbt`. 2. Locate the extension from the publisher `dbtLabsInc` or `dbt Labs Inc`, then click **Install**. [![Search for the extension](/img/docs/extension/extension-marketplace.png?v=2 "Search for the extension")](#)Search for the extension 3. Open a dbt project in your editor. Make sure the project is added to your current workspace. 4. Confirm that the extension is active by checking for the **dbt Extension** label in the status bar. Hover over the label to view diagnostic information. [![If you see the 'dbt Extension' label, the extension is activated](/img/docs/extension/dbt-extension-statusbar.png?v=2 "If you see the 'dbt Extension' label, the extension is activated")](#)If you see the 'dbt Extension' label, the extension is activated 5. After the extension activates, it automatically downloads the correct dbt Language Server (LSP) for your operating system. [![The dbt Language Server will be installed automatically](/img/docs/extension/extension-lsp-download.png?v=2 "The dbt Language Server will be installed automatically")](#)The dbt Language Server will be installed automatically 6. If dbt isn't installed, the extension prompts you to download and install it. Follow the notification steps, or [install it manually from the command line](#install-dbt-v2-from-the-command-line-if-you-havent-already). [![Follow the prompt to install v2](/img/docs/extension/install-dbt-v2-engine.png?v=2 "Follow the prompt to install v2")](#)Follow the prompt to install v2 7. Run the VS Code extension [upgrade tool](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) to check whether your project is ready for v2 and fix any errors or deprecations. 8. Optional: [Configure your local environment](https://docs.getdbt.com/docs/configure-dbt-extension.md) to mirror your dbt platform environment and [set environment variables](https://docs.getdbt.com/docs/configure-dbt-extension.md#configure-environment-variables) required by your project. The language server ships with dbt v2 The dbt language server is part of the v2 binary rather than a separate download — the extension runs it through v2. If you install v2 manually (for example, in an air-gapped environment) instead of letting the extension manage it, use the [version compatibility matrix](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) to pick a binary that matches your extension version. You're ready to use the dbt extension. Next, you can: * Follow the [getting started](#getting-started) workflow to finish setup. * [Upgrade your project to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) if you're migrating from v1. * [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to a dbt platform account if you want to use the extension's dbt platform capabilities, such as the **Catalog** tab. * Review the [limitations and unsupported features](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations).  Install dbt v2 from the command line, if you haven't already. Choose your preferred installation method:  Pip installation for Windows, macOS, and Linux ```shell python -m pip install dbt ``` To upgrade to a newer version: ```shell python -m pip install --upgrade dbt ```  Standalone installation for macOS and Linux ```shell curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh -s -- --update ``` To use `dbt` immediately after installation, close and reopen your terminal or reload your shell so that the new `$PATH` is recognized: ```shell exec $SHELL ``` To upgrade to a newer version: ```shell dbt system update ``` note `dbtf system update` installs dbt v2 globally — it updates your `PATH` in `~/.zshrc` and creates a `dbtf` alias. To manage multiple versions or isolate your install, use separate shell profiles or virtual environments.  Standalone installation for Windows ```powershell irm https://public.cdn.getdbt.com/fs/install/install.ps1 | iex ``` To use `dbt` immediately after installation, close and reopen or reload your shell so that the new `Path` is recognized: ```powershell Start-Process powershell ``` To upgrade to a newer version: ```shell dbt system update ```  Homebrew installation for macOS ```shell brew tap dbt-labs/dbt brew install dbt-labs/dbt/dbt ``` To upgrade to a newer version: ```shell brew tap dbt-labs/dbt brew upgrade dbt ```  Winget installation for Windows ```shell winget install --id dbtLabs.dbt --exact ``` To upgrade to a specific version: ```shell winget install --id dbtLabs.dbt --exact --version <version> ``` Run the following command to verify your installation: ```bash dbt --version ``` You can use `dbt` or its dbt v2 alias `dbtf` (handy if you already have another dbt CLI installed). Default install path: * macOS/Linux: `$HOME/.local/bin/dbt` * Windows: `C:\Users\<username>\.local\bin\dbt.exe` The installer adds this path automatically, but you may need to reload your shell for the `dbtf` command to work. After installation, follow the [getting started](#getting-started) workflow. You can get started by: * Running `dbt init --fusion-upgrade` to start terminal onboarding. * Running **dbt: Register dbt extension** from the command palette. * Selecting **Get started** from the extension menu. #### Getting started After v2 and the dbt VS Code extension are installed, the dbt logo appears in the sidebar. Click it to open the **Get started** panel. The **Get started** panel is a persistent setup companion available in VS Code and Cursor. It monitors your environment and shows the most important next action. As new v2 releases ship or your project changes, the panel resurfaces relevant steps automatically. For more details refer to [Upgrade to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md). [![The Get started panel in VS Code showing the setup steps, with the active step highlighted.](/img/docs/extension/vsce-get-started.png?v=2 "The Get started panel in VS Code showing the setup steps, with the active step highlighted.")](#)The Get started panel in VS Code showing the setup steps, with the active step highlighted. The panel guides you through: 1. **Install or update dbt v2**: Detects whether the v2 binary is missing or outdated and installs or updates it with a single click. 2. **Open project**: Checks for a `dbt_project.yml` file in your workspace to confirm a valid dbt project is open before proceeding. 3. **Check dbt v2 compatibility**: Guides you through upgrading your project to v2. You can choose between an agentic migration or manual CLI onboarding — refer to [Upgrade to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md). 4. **Sign in**: Confirms you've signed in to a dbt platform account, which the extension uses for dbt platform capabilities like the **Catalog** tab — refer to [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md). When all setup steps are complete, the panel shows a green **Extension setup complete** button. [![The Get started panel showing Extension setup complete with all four steps checked.](/img/docs/extension/vsce-get-started-complete.png?v=2 "The Get started panel showing Extension setup complete with all four steps checked.")](#)The Get started panel showing Extension setup complete with all four steps checked. #### Next steps Once you've installed the dbt VS Code extension, go to the next pages to get started: 1. Review the [Upgrade to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) page to upgrade your dbt project to the next-gen engine today! 2. [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to a dbt platform account if you want to use the extension's dbt platform capabilities, such as the **Catalog** tab. 3. Review the [limitations and unsupported features](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations). --- ### Installing agent skills from dbt packages Available in v2 | Local development Ship a set of skills in a dbt package, and `dbt deps` installs them into the directory your coding agent reads from. Your whole team picks up the same skills from a versioned dependency instead of managing them separately. A skill is a directory containing a `SKILL.md` file that gives your coding agent reusable instructions, in the [Agent Skills](https://agentskills.io/specification) format. Any dbt project can ship skills, and any project that installs that project as a package can install its skills. Let's say your team has rules about how to name models or where tests go. You can write those rules down as a skill *once*, ship them in a package, and everyone's coding agent picks them up the next time they run `dbt deps`. This means everyone on the team gets the same rules, and you only update them in one place. Agent skills install locally where your coding agent runs. Skills install only when you configure an `ai_provider`. Without one, `dbt deps` installs your packages as usual and leaves your agent directories untouched. #### Set the ai\_provider flag Set [`ai_provider` in the `flags`](https://docs.getdbt.com/reference/global-configs/about-global-configs.md?version=2#available-flags) block of your root project to tell dbt which directory to write skills to. For example, if you use claude as your AI provider, you'd set it as such: dbt\_project.yml ```yml flags: ai_provider: claude ``` Each provider has a directory it reads skills from. Most providers share `.agents/skills`; claude reads skills via its own directory. Supported providers are: | `ai_provider` value | Installs to | | ------------------- | ---------------- | | `wizard` | `.agents/skills` | | `claude` | `.claude/skills` | | `openai` | `.agents/skills` | | `codex` | `.agents/skills` | | `cursor` | `.agents/skills` | | `gemini` | `.agents/skills` | Values are case-insensitive, so `wizard`, `Wizard`, and `WIZARD` all resolve the same way. If you use dbt Wizard, set `wizard` even if you [bring your own key](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) or use dbt-managed AI. You can also list more than one provider, which installs the same skills into each provider's directory: dbt\_project.yml ```yml flags: ai_provider: - wizard - claude ``` dbt resolves `ai_provider` from three places, in order of precedence: 1. The `--ai-provider` command-line option, which accepts a comma-separated list, for example `dbt deps --ai-provider claude,cursor`. 2. The `DBT_ENGINE_AI_PROVIDER` environment variable. 3. The `flags` block in your root project's `dbt_project.yml`, as shown previously. Only the root project's `flags` block is read. Setting `ai_provider` in a package has no effect, because the choice of coding agent belongs to the user, not to the package author. If your project or its packages ship skills and `ai_provider` isn't set, `dbt deps` succeeds and warns that it found skills but installed none. For example: ```text [warning] [AiProviderUnset (dbt1801)]: Found 3 agent skill(s) in this project and its packages, but 'ai_provider' is not set, so none were installed. Set it in dbt_project.yml (flags: {ai_provider: claude}), via --ai-provider, or with DBT_ENGINE_AI_PROVIDER. Known providers: wizard, claude, openai, codex, cursor, gemini. ``` #### Ship skills in a package Create one directory per skill under your project's `skills` directory, each containing a `SKILL.md` file: ```text skills/ naming-conventions/ SKILL.md add-a-data-test/ SKILL.md ``` To keep skills somewhere other than `skills`, set [skill-paths](https://docs.getdbt.com/reference/project-configs/skill-paths.md). Each `SKILL.md` needs YAML frontmatter with a `name` that matches its directory name, and a `description` that tells the agent when to apply the skill: skills/naming-conventions/SKILL.md ```markdown --- name: naming-conventions description: Use when creating or renaming a model, so that staging, intermediate, and mart models follow the house naming convention. --- # Naming conventions Apply these prefixes when adding a new model: - Staging models use the prefix `stg_` and live in `models/staging/`. - Intermediate models use the prefix `int_` and live in `models/intermediate/`. - Mart models use no prefix and are named for the business concept they expose. ``` For more detail on writing effective skills, refer to [Skill file format](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md#skill-file-format). dbt reads a package's `SKILL.md` files and never modifies them. Only the installed copy is written to. A `SKILL.md` that dbt can't read — because it's missing frontmatter, or its `name` doesn't match its directory — produces a warning and is skipped. One malformed skill in a package doesn't stop the rest from installing. #### Install the skills Add the package to your project as you would any other dependency: packages.yml ```yaml packages: - local: "../packages/demo_skills" ``` Then run `dbt deps`. dbt reports each skill it writes: ```shell dbt deps ``` ```text Installing packages Installing demo_skills Installed demo_skills Installed 1 package Installing add-a-data-test (demo_skills) -> .agents/skills Installing naming-conventions (demo_skills) -> .agents/skills ``` With more than one provider set, dbt writes a copy per directory: ```text Installing add-a-data-test (demo_skills) -> .agents/skills Installing add-a-data-test (demo_skills) -> .claude/skills ``` dbt installs skills whenever it installs packages, so `dbt build`, `dbt run`, and `dbt parse` install them too. You don't have to run `dbt deps` yourself. dbt collects skills from your root project as well as from every installed package, including packages installed as transitive dependencies. A `skills` directory in your own project installs alongside the ones your packages ship. Re-running a command when nothing has changed writes nothing and reports nothing. dbt rewrites the installed copy only when the source has changed. #### Understand how dbt tracks installed skills When dbt installs a skill, it records what it did in the installed copy's frontmatter, under the `metadata` field: .agents/skills/naming-conventions/SKILL.md ```markdown --- name: naming-conventions description: Use when creating or renaming a model, so that staging, intermediate, and mart models follow the house naming convention. metadata: dbt.managed_by: dbt dbt.source: package dbt.package: demo_skills dbt.source_path: skills/naming-conventions dbt.source_hash: sha256:fbdd25363736f004d0599b754323c661ddf79d8ffb76f54c0b8c263e7bc52def dbt.installed_at: 2026-09-09T19:21:14.595255+00:00 --- ``` That record is the only way dbt recognizes a skill as its own, which has three consequences worth knowing: * **Skills you write by hand are never touched.** dbt only overwrites or removes a skill directory whose `SKILL.md` carries `dbt.managed_by: dbt`. Your own skills can sit in the same directory safely. * **Removing the record makes dbt forget the copy.** If you delete the `dbt.` entries from an installed skill, dbt no longer recognizes it and leaves it alone from then on, including during [dbt clean](https://docs.getdbt.com/reference/commands/clean.md). * **Modifications to skills are overwritten.** dbt always overwrites skill directories whose `SKILL.md` carries `dbt.managed_by: dbt`. To permanently modify a skill installed via a package, remove the dbt metadata within the frontmatter of the `SKILL.md`. If the skill's author included their own `metadata` entries, dbt preserves them and adds its own entries after them. `dbt.version` records the version dbt resolved for the package. Local and tarball packages have no pinned version, so the field is absent for those; `dbt.source_hash` still identifies the exact content that was installed. #### Disable a skill Use the `skills` config in your root project, keyed by the package that ships the skill: dbt\_project.yml ```yml skills: demo_skills: naming-conventions: +enabled: false ``` A package's own `skills` config sets the defaults for the skills it ships, and your root project's `skills` config overrides it. This works the same way as [enabling and disabling other resources](https://docs.getdbt.com/reference/resource-configs/enabled.md). #### More examples Set `+enabled` at the package level to control a whole package, or nest it under a skill name to control a single skill. Disable every skill from one package, and leave all others enabled: dbt\_project.yml ```yml skills: demo_skills: +enabled: false ``` Enable skills from only one package, and disable them from all others: dbt\_project.yml ```yml skills: +enabled: false demo_skills: +enabled: true ``` Enable just two skills, and disable everything else: dbt\_project.yml ```yml skills: demo_skills: +enabled: false naming-conventions: +enabled: true adding-exposures: +enabled: true ``` Disable just two skills, and leave everything else enabled: dbt\_project.yml ```yml skills: demo_skills: naming-conventions: +enabled: false adding-exposures: +enabled: false ``` Skill names must be unique Because skills install under their own name, two enabled skills with the same name would occupy the same directory. Rather than choose between them, dbt fails the command before writing anything: ```text Two or more agent skills are named 'naming-conventions' (this project, package 'demo_skills'), so they would install into the same directory. Skill names must be unique across your project and its packages. ``` Resolve it the same way you'd resolve duplicate model names: disable all but one with `+enabled: false`, or ask the package maintainer to rename theirs. #### Remove installed skills [dbt clean](https://docs.getdbt.com/reference/commands/clean.md) removes the skills dbt installed and leaves everything else in place: ```shell dbt clean ``` ```text Removing add-a-data-test -> .agents/skills Removing naming-conventions -> .agents/skills ``` #### Related docs * [Use skills with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Packages](https://docs.getdbt.com/docs/build/packages.md) * [skill-paths](https://docs.getdbt.com/reference/project-configs/skill-paths.md) * [About dbt deps command](https://docs.getdbt.com/reference/commands/deps.md) * [About dbt clean command](https://docs.getdbt.com/reference/commands/clean.md) --- ### Integrate Claude with dbt MCP Claude is an AI assistant from Anthropic with two primary interfaces: * [Claude Desktop](#claude-desktop): A GUI with MCP support for file access and commands, plus basic coding features. * [Claude Code](#claude-code): A terminal/IDE tool for development. Both interfaces can connect to either: * Self-hosted dbt MCP server (runs on your machine, supports CLI commands like `dbt run`) * Remote dbt MCP server (HTTP, consumption-focused). OAuth needs no local install. Token-based auth on Claude Desktop uses the `mcp-remote` proxy and requires Node.js. #### Prerequisites * You use Claude for AI or agentic work * For OAuth (self-hosted or remote), use your [access URL with a static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). * Remote MCP OAuth is available for Starter, Enterprise, and Enterprise+ accounts. Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. #### Claude Desktop [Claude Desktop](https://claude.ai/download) reads MCP servers from `claude_desktop_config.json`. Open it from **Settings → Developer → Edit Config**. ##### Set up with self-hosted dbt MCP server For a fast first install, you can download the prebuilt `.mcpb` file; for more control, edit the JSON directly. tip You don't need to clone the dbt-mcp repository — for self-hosted setups, install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` (or use the configs later in this page). Only clone the repository if you want to [contribute to dbt MCP](https://github.com/dbt-labs/dbt-mcp/issues). ###### Quick install with the .mcpb file 1. Go to the [latest dbt MCP release](https://github.com/dbt-labs/dbt-mcp/releases/latest) and download the `dbt-mcp.mcpb` file. 2. Double-click the downloaded file to open it in Claude Desktop. 3. Configure the **dbt platform Host**. You can find this in your dbt platform account by navigating to **Account settings** and copying the **Access URL**. 4. Enable the server in Claude Desktop. 5. Ask Claude a data-related question and see dbt MCP in action! ###### Advanced config with Claude Desktop Use advanced configuration when you want to define the dbt MCP server yourself in Claude's configuration file — the same JSON where Claude stores every MCP server, under `mcpServers`, with fields like `command`, `args`, and `env`. See the [MCP install pattern](https://modelcontextprotocol.io/quickstart/user#installing-the-filesystem-server) for the underlying convention. To open the configuration file and add or replace the dbt MCP server entry: 1. From Claude, go to **Settings…** 2. In the Settings window, select the **Developer** tab. 3. Click **Edit Config** and open the file in a text editor. 4. Add your server configuration under `mcpServers`. Choose the option that fits your use case:  Self-hosted MCP with OAuth[Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Configuration for users who want seamless OAuth authentication with the dbt platform. ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication.  Self-hosted MCP (CLI only) Self-hosted configuration for users who only want to use dbt commands with dbt v1 or dbt v2 ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` Finding your paths: * **DBT\_PROJECT\_DIR**: Full path to the folder containing your `dbt_project.yml` file * **DBT\_PATH**: Find by running `which dbt` in Terminal (macOS/Linux) or `where dbt` (Windows) in Powershell  Self-hosted MCP with .env Advanced configuration for users who need custom [environment variables](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). Put your `.env` file in your *dbt project root* (same folder as `dbt_project.yml`) and use an absolute path with `--env-file`. Using the `env` field (single-file configuration): IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_DEV_ENV_ID`, and `DBT_USER_ID` must be numeric IDs (for example, `54321`), not full URLs copied from your browser. `DBT_HOST` accepts both `cloud.getdbt.com` and `https://cloud.getdbt.com`. Using an `.env` file (use an absolute path to `.env` in your dbt project root): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` Using an .env file (alternative): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` 5. Save the file and restart Claude Desktop. You'll see an MCP server indicator in the bottom-right corner of the conversation input box. For more configuration options (env vars, service tokens, tool-access controls), refer to [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md). ##### Set up with remote dbt MCP server The remote dbt MCP server runs in dbt platform. For OAuth, Claude Desktop connects over HTTP with a custom connector. For token-based auth, use the `mcp-remote` proxy described in the token tab (Claude Desktop does not accept `"type": "http"` in `claude_desktop_config.json`). info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. Get your MCP URL first. You'll need it for both auth methods: You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Then follow the tab that matches your auth method: ##### OAuth (remote) *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). For OAuth, add dbt as a custom connector through Claude Desktop's settings. You don't need to edit `claude_desktop_config.json`. The following steps show how to connect dbt as a custom connector in Claude Desktop. The exact UI varies by tool, but the flow is the same: add a custom connector with your MCP URL, complete the OAuth consent flow, then connect. 1. In your AI tool, go to its connector settings and choose to add a custom connector (in Claude Desktop, go to **Chat → Customize → Connectors**, then click **Add custom connector**). 2. Enter a name (for example, `dbt`) and paste your dbt platform MCP URL (for example, `https://abc123.us1.dbt.com/api/ai/v1/mcp`), then click **Add**. [![Custom connector dialog showing the dbt MCP URL](/img/docs/dbt-cloud/oauth-add-custom-connector.png?v=2 "Custom connector dialog showing the dbt MCP URL")](#)Custom connector dialog showing the dbt MCP URL 3. Click **Connect**. The tool redirects you to dbt to complete the OAuth consent flow, where you can approve or deny individual [scopes](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent). [![OAuth consent screen showing requested scopes and project access](/img/docs/dbt-cloud/oauth-consent-screen.png?v=2 "OAuth consent screen showing requested scopes and project access")](#)OAuth consent screen showing requested scopes and project access 4. After you approve, the connector is added to the **Custom connectors** table and shows as connected. [![Adding a custom dbt connector in an AI tool's connector settings](/img/docs/dbt-cloud/oauth-connectors-page.png?v=2 "Adding a custom dbt connector in an AI tool's connector settings")](#)Adding a custom dbt connector in an AI tool's connector settings 5. That's it 🎉! Ask your tool a data question like *"What is the total revenue for the last 30 days?"* to confirm the connection. ##### Token-based Use token-based auth when your client doesn't yet support OAuth for HTTP MCP servers, or when you need a shared or CI-style setup. `"type": "http"` is not valid in Claude Desktop Claude Desktop doesn't accept `"type": "http"` in `claude_desktop_config.json`. If you paste that block (the format that works in [Claude Code](#code-remote)'s `.mcp.json`), Claude Desktop skips the server and shows a message like "some MCP servers could not be loaded." Use the `mcp-remote` proxy config in step 2 instead. Token-based remote MCP on Claude Desktop requires [Node.js](https://nodejs.org/) so `npx` can run `mcp-remote`. 1. From Claude Desktop, go to **Settings → Developer → Edit Config** to open `claude_desktop_config.json`. 2. Add a `dbt-remote` entry under `mcpServers`: ```json { "mcpServers": { "dbt-remote": { "command": "npx", "args": [ "mcp-remote", "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "--header", "Authorization: Token YOUR_DBT_ACCESS_TOKEN", "--header", "x-dbt-prod-environment-id: DBT_PROD_ENV_ID" ] } } } ``` Replace `YOUR_DBT_HOST_URL` with your hostname (for example, `abc123.us1.dbt.com`), and replace the token and environment ID placeholders with your values. To pass optional headers such as `x-dbt-user-id` or `x-dbt-dev-environment-id`, add more `"--header"` and `"Header-Name: value"` pairs to the `args` array. For token-based remote MCP, set these headers in your client's MCP config: * **`Authorization`** *(required)* — `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Use a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with Developer access to the project that owns the environment in `x-dbt-prod-environment-id`. Assign service tokens the `Developer` permission set. PATs inherit their creator's permissions. Without Developer access, the connection fails with a `401` error. * **`x-dbt-prod-environment-id`** *(required)* — your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **`x-dbt-dev-environment-id`** — required for `execute_sql` and dbt v2 tools. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. * **`x-dbt-user-id`** — required for `execute_sql`. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. Use numeric IDs, not full URLs Headers like `x-dbt-prod-environment-id`, `x-dbt-dev-environment-id`, and `x-dbt-user-id` expect numeric IDs (for example, `54321`), not full URLs copied from your browser. The MCP host URL must include `https://`, whether you set it in a `url` field or pass it as an `mcp-remote` argument. If you use token-based auth, `execute_sql` requires a personal access token (PAT). Service tokens won't work. To run SQL without creating a PAT, connect with [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp) instead. Refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) for the full list of headers. Full path to `npx` If Claude Desktop shows `Server disconnected` or can't find `npx`, set `"command"` to the full path from `which npx` (macOS or Linux) or `where npx` (Windows), for example `"command": "/opt/homebrew/bin/npx"`. 3. Save the file and restart Claude Desktop. In **Settings → Developer**, confirm `dbt-remote` is running. Ask Claude a data question to confirm the server is connected. #### Claude Code [Claude Code](https://www.anthropic.com/claude-code) reads MCP servers from `.mcp.json` at the root of your project (the repository root for your workspace). For self-hosted setups, you can reuse the same `command` / `args` / `env` shape you use in other clients. For remote token-based auth, Claude Code supports `"type": "http"` in `.mcp.json`, which Claude Desktop does not. ##### Set up with self-hosted dbt MCP server tip You don't need to clone the dbt-mcp repository — for self-hosted setups, install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` (or use the configs later in this page). Only clone the repository if you want to [contribute to dbt MCP](https://github.com/dbt-labs/dbt-mcp/issues). 1. Follow [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) to choose your auth pattern: * OAuth with the dbt platform * [CLI only](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#cli-only) * [Environment variables](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#environment-variable-configuration) (including an `.env` file with `--env-file`) 2. Create `.mcp.json` at the project root and add a `dbt` entry under the top-level `mcpServers` key. Pick the option that fits your use case:  Self-hosted MCP with OAuth[Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Configuration for users who want seamless OAuth authentication with the dbt platform. ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication.  Self-hosted MCP (CLI only) Self-hosted configuration for users who only want to use dbt commands with dbt v1 or dbt v2 ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` Finding your paths: * **DBT\_PROJECT\_DIR**: Full path to the folder containing your `dbt_project.yml` file * **DBT\_PATH**: Find by running `which dbt` in Terminal (macOS/Linux) or `where dbt` (Windows) in Powershell  Self-hosted MCP with .env Advanced configuration for users who need custom [environment variables](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). Put your `.env` file in your *dbt project root* (same folder as `dbt_project.yml`) and use an absolute path with `--env-file`. Using the `env` field (single-file configuration): IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_DEV_ENV_ID`, and `DBT_USER_ID` must be numeric IDs (for example, `54321`), not full URLs copied from your browser. `DBT_HOST` accepts both `cloud.getdbt.com` and `https://cloud.getdbt.com`. Using an `.env` file (use an absolute path to `.env` in your dbt project root): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` Using an .env file (alternative): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` About `claude mcp add` The Claude Code CLI can register MCP servers with `claude mcp add`, but it typically writes to the user-level config (`~/.claude.json`) rather than the project's `.mcp.json`. For dbt MCP, we recommend committing `.mcp.json` to your repository so the setup is project-scoped and easier to share. ##### Set up with remote dbt MCP server Claude Code can connect to the remote dbt MCP server over HTTP using `"type": "http"` in `.mcp.json`. That HTTP config shape is for Claude Code (and similar clients), not for Claude Desktop's `claude_desktop_config.json`. info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. 1. Open `.mcp.json` at the root of your project (create it if it doesn't exist). 2. Get your MCP URL: You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). 3. Add the `dbt` entry to `.mcp.json` using the tab that matches your auth method: ##### OAuth (remote) *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). Add the following to `.mcp.json` at your project root. On first connect, Claude Code opens a browser for sign-in and consent. ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` Replace `YOUR_DBT_HOST_URL` with your hostname (for example, `abc123.us1.dbt.com`). You can find the URL in dbt platform under **Account settings** → **Access URLs** → **MCP Endpoint URL**. You can also register the same server from the CLI: ```bash claude mcp add --transport http dbt https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/ ``` ##### Token-based ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` For token-based remote MCP, set these headers in your client's MCP config: * **`Authorization`** *(required)* — `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Use a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with Developer access to the project that owns the environment in `x-dbt-prod-environment-id`. Assign service tokens the `Developer` permission set. PATs inherit their creator's permissions. Without Developer access, the connection fails with a `401` error. * **`x-dbt-prod-environment-id`** *(required)* — your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **`x-dbt-dev-environment-id`** — required for `execute_sql` and dbt v2 tools. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. * **`x-dbt-user-id`** — required for `execute_sql`. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. Use numeric IDs, not full URLs Headers like `x-dbt-prod-environment-id`, `x-dbt-dev-environment-id`, and `x-dbt-user-id` expect numeric IDs (for example, `54321`), not full URLs copied from your browser. The MCP host URL must include `https://`, whether you set it in a `url` field or pass it as an `mcp-remote` argument. If you use token-based auth, `execute_sql` requires a personal access token (PAT). Service tokens won't work. To run SQL without creating a PAT, connect with [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp) instead. Refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) for the full list of headers. 4. Save the file. Claude Code picks up `.mcp.json` on startup — ask it a data question to confirm the connection. #### Troubleshooting Claude Desktop errors Claude Desktop may return errors such as `Error: spawn uvx ENOENT`, `Error: spawn npx ENOENT`, `Could not connect to MCP server dbt-mcp`, or `Server disconnected`. For self-hosted installations, replace the `command` with the full path to `uvx`: run `which uvx` on Unix systems or `where uvx` on Windows and paste the full path into your JSON (for example, `"command": "/the/full/path/to/uvx"`). For token-based remote setups with `mcp-remote`, apply the same full-path fix for `npx`. If Claude reports that entries in `claude_desktop_config.json` are not valid and were skipped, check that you are not using `"type": "http"` in that file. Use the [token-based `mcp-remote` config](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md#desktop-remote) instead. For remote setups, double-check that the MCP URL ends in `/api/ai/v1/mcp/` (including the trailing slash) and that your `Authorization` header value is `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Logs are at `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). Claude Code can't connect to the dbt MCP server If the dbt MCP server doesn't connect, confirm `.mcp.json` is at the *project root* and that the `dbt` block matches one of the examples on the [Claude setup page](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md#claude-code). For self-hosted installations, apply the same full-path fix for `uvx` (and for `--env-file` paths). For remote setups, verify the URL and headers, and try the equivalent `claude mcp add --transport http` command to compare. --- ### Integrate Cursor with dbt MCP [Cursor](https://docs.cursor.com/context/model-context-protocol) is an AI-powered code editor, powered by Microsoft Visual Studio Code (VS Code). After setting up your MCP server, you connect it to Cursor. Log in to Cursor and follow the steps that align with your use case. #### Set up with self-hosted dbt MCP server Choose your setup based on your workflow: * OAuth for dbt platform connections * CLI only if using dbt v1 or dbt v2 locally. * Configure environment variables if you're using them in your dbt platform account. ##### OAuth or CLI Click one of the following application links with Cursor open to automatically configure your MCP server: ##### CLI only (dbt v1 and dbt v2) Self-hosted configuration for users who only want to use dbt commands with dbt v1 or dbt v2 (no dbt platform features). [Add dbt v1 or dbt v2 to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX1BST0pFQ1RfRElSIjoiL3BhdGgvdG8veW91ci9kYnQvcHJvamVjdCIsIkRCVF9QQVRIIjoiL3BhdGgvdG8veW91ci9kYnQvZXhlY3V0YWJsZSJ9LCJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJkYnQtbWNwIl19) After clicking: 1. Update (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` with the full path to your dbt project (the folder containing `dbt_project.yml`). 2. Update `DBT_PATH` with the full path to your dbt executable: * macOS/Linux: Run `which dbt` in Terminal. * Windows: Run `where dbt` in Command Prompt or PowerShell. 3. Save the configuration. ##### OAuth with dbt platform *OAuth with the self-hosted dbt MCP server is available for Starter, Enterprise, and Enterprise+ accounts.* Configuration settings for users who want OAuth authentication with the dbt platform. AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * [dbt platform only](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJodHRwczovLzx5b3VyLWRidC1ob3N0LXdpdGgtY3VzdG9tLXN1YmRvbWFpbj4iLCJESVNBQkxFX0RCVF9DTEkiOiJ0cnVlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) * [dbt platform + CLI](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJodHRwczovLzx5b3VyLWRidC1ob3N0LXdpdGgtY3VzdG9tLXN1YmRvbWFpbj4iLCJEQlRfUFJPSkVDVF9ESVIiOiIvcGF0aC90by9wcm9qZWN0IiwiREJUX1BBVEgiOiJwYXRoL3RvL2RidC9leGVjdXRhYmxlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) After clicking: 1. Replace `<your-dbt-host-with-custom-subdomain>` with your actual host (for example, `abc123.us1.dbt.com`). 2. (For dbt platform + CLI) Update (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` and `DBT_PATH` as described above. 3. Save the configuration. ##### Custom environment variables Use this method if you need custom environment variables or prefer to use service tokens. Refer to the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) for the complete list of available environment variables for the self-hosted MCP server. 1. Click the following link with Cursor open: [Add to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJkYnQtbWNwIl0sImVudiI6e319) 2. In the template, add your environment variables to the `env` section based on your needs. 3. Save the configuration. ###### Using an `.env` file If you prefer to manage environment variables in a separate file, put the `.env` file in your *dbt project root* (same folder as `dbt_project.yml`). Click this link: [Add to Cursor (with .env file)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt-mcp\&config=eyJjb21tYW5kIjoidXZ4IC0tZW52LWZpbGUgPGVudi1maWxlLXBhdGg%252BIGRidC1tY3AifQ%3D%3D) Then update `env-file-path` with the absolute path to your `.env` file (for example, `/absolute/path/to/your-dbt-project/.env`). #### Set up with remote dbt MCP server Remote MCP supports **OAuth** or **token-based** headers. * *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* * For either method, the MCP URL is `https://<Access URL>/api/ai/v1/mcp`. You can find the URL in dbt platform under **Account settings** → **Access URLs** → **MCP Endpoint URL**. **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). The deeplink below configures **token-based** authentication (URL and headers). For OAuth setup, follow the [remote MCP quickstart](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md#5-configure-your-mcp-client). 1. Click the following application link with Cursor open: [Add to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJ1cmwiOiJodHRwczovLzxob3N0Pi9hcGkvYWkvdjEvbWNwLyIsImhlYWRlcnMiOnsiQXV0aG9yaXphdGlvbiI6InRva2VuIDx0b2tlbj4iLCJ4LWRidC1wcm9kLWVudmlyb25tZW50LWlkIjoiPHByb2QtaWQ%252BIn19) 2. Provide your URL/headers by updating the **host**, **production environment ID**, and **service token** in the template. Use `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp` as the server URL (no trailing slash). IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_DEV_ENV_ID` must be numeric IDs (for example, `54321`), not full URLs. `DBT_HOST` field accepts the `https://` prefix and without the `https://` prefix. The following are valid examples: ```bash DBT_HOST=https://ab123.us1.dbt.com DBT_HOST=ab123.us1.dbt.com ``` 3. Save, and now you have access to the dbt MCP server! --- ### Integrate Snowflake Cortex agents with dbt MCP dbt platform [Snowflake Cortex agents](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents) can call external [MCP servers](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents-mcp-connectors) as tools. This guide walks you through connecting a Cortex agent to the remote dbt MCP server so it can query your Semantic Layer metrics and dimensions in plain English from Snowflake Intelligence. The connection uses OAuth: Snowflake registers itself with dbt platform through Dynamic Client Registration (DCR) with PKCE, so you don't store a client secret in Snowflake. Each user completes their own OAuth consent the first time they use the agent, which means the agent respects each user's existing dbt permissions and project access. #### Prerequisites Before connecting a Cortex agent to the remote dbt MCP server, make sure you have the following in place. ###### In dbt platform Make sure the following are set up before connecting from Snowflake: * **Account setup** * [Remote MCP OAuth enabled](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md). The remote MCP server is generally available, but the OAuth connection method is in public beta for Starter and Enterprise-tiered accounts. * A [static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) configured, for example `abc123` in `abc123.us1.dbt.com`. If your account doesn't have a subdomain, contact support. * **Access and permissions** * Read-only or higher access to dbt platform. The agent inherits each connected user's permissions. * **Semantic Layer setup** * A configured [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) with [metrics and dimensions](https://docs.getdbt.com/docs/build/build-metrics-intro.md). * **MCP endpoint** You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Use the host portion of this URL in the Snowflake SQL, for example `abc123.us1.dbt.com`. ###### In Snowflake * Snowflake Intelligence and Cortex agents enabled in your account and region. * External MCP connectors available for your account. Confirm availability with your Snowflake account team. * The `ACCOUNTADMIN` role, or a role with account-level `CREATE INTEGRATION`. * A database, schema, and role for creating the MCP server and agent. Optional: native SQL execution Cortex agents can compile and execute Semantic Layer queries without extra setup. To also allow ad hoc SQL against Snowflake, configure project read credentials in **Settings → Credentials** in dbt platform. Without those credentials, the agent can still compile Semantic Layer SQL and execute it natively on Snowflake. #### Parameters The SQL on this page uses placeholders. Replace each one with your own value before running: | Placeholder | Description | | ------------------------------- | ------------------------------------------------------------------------------ | | `YOUR_DBT_HOST_URL` | Your dbt host, with no `https://` (for example, `abc123.us1.dbt.com`). | | `INTEGRATION_NAME` | A name for the Snowflake API integration (for example, `dbt_mcp_integration`). | | `TARGET_DATABASE.TARGET_SCHEMA` | The database and schema where the MCP server and agent live. | | `TARGET_ROLE` | The Snowflake role that uses the agent and MCP server. | | `MCP_SERVER_NAME` | A name for the external MCP server object. | | `AGENT_NAME` | A name for the Cortex agent. | | `CORTEX_MODEL` | The orchestration model for the agent (for example, `claude-4-sonnet`). | #### Set up the connection The following steps register the remote dbt MCP server with Snowflake and connect it to a Cortex agent. Steps 1–3 run as Snowflake SQL. Step 4 starts in Snowflake Intelligence and redirects each user to dbt platform to complete OAuth authorization. ##### Step 1: Create the API integration In Snowflake, run this as `ACCOUNTADMIN`. The integration tells Snowflake how to reach the remote dbt MCP endpoint and how to complete OAuth using Dynamic Client Registration with PKCE (no client secret). ```sql -- Create an API integration for the remote dbt MCP server. -- Uses OAuth Dynamic Client Registration + PKCE (no client secret). CREATE API INTEGRATION IF NOT EXISTS INTEGRATION_NAME API_PROVIDER = EXTERNAL_MCP API_ALLOWED_PREFIXES = ('https://YOUR_DBT_HOST_URL/api/ai/v1/mcp') API_USER_AUTHENTICATION = ( TYPE = OAUTH_DYNAMIC_CLIENT OAUTH_CLIENT_AUTH_METHOD = NONE OAUTH_TOKEN_ENDPOINT = 'https://YOUR_DBT_HOST_URL/oauth/token' OAUTH_AUTHORIZATION_ENDPOINT = 'https://YOUR_DBT_HOST_URL/oauth/authorize' OAUTH_RESOURCE_URL = 'https://YOUR_DBT_HOST_URL/api/ai/v1/mcp' OAUTH_ALLOWED_SCOPES = ('user_access', 'offline_access') ) ENABLED = TRUE; ``` The `OAUTH_TOKEN_ENDPOINT` and `OAUTH_AUTHORIZATION_ENDPOINT` use the same host as your MCP URL. The `user_access` and `offline_access` scopes let the agent act on your behalf and refresh its session without you re-authenticating each time. ##### Step 2: Create the external MCP server In Snowflake, grant your role the ability to create an external MCP server, then create one that references the integration from Step 1. ```sql GRANT CREATE EXTERNAL MCP SERVER ON SCHEMA TARGET_DATABASE.TARGET_SCHEMA TO ROLE TARGET_ROLE; -- Create an external MCP server pointing to the remote dbt MCP endpoint. CREATE EXTERNAL MCP SERVER IF NOT EXISTS TARGET_DATABASE.TARGET_SCHEMA.MCP_SERVER_NAME WITH DISPLAY_NAME = 'dbt Semantic Layer MCP' URL = 'https://YOUR_DBT_HOST_URL/api/ai/v1/mcp' API_INTEGRATION = INTEGRATION_NAME; ``` ##### Step 3: Create the Cortex agent In Snowflake, grant your role the privileges to create an agent and to use the MCP server and its integration, then create the agent. ```sql GRANT CREATE AGENT ON SCHEMA TARGET_DATABASE.TARGET_SCHEMA TO ROLE TARGET_ROLE; -- Grant the role access to the MCP server and its underlying integration. GRANT USAGE ON EXTERNAL MCP SERVER TARGET_DATABASE.TARGET_SCHEMA.MCP_SERVER_NAME TO ROLE TARGET_ROLE; GRANT USAGE ON INTEGRATION INTEGRATION_NAME TO ROLE TARGET_ROLE; CREATE AGENT IF NOT EXISTS TARGET_DATABASE.TARGET_SCHEMA.AGENT_NAME COMMENT = 'Analytics agent powered by the dbt Semantic Layer via MCP' PROFILE = '{"display_name": "dbt Semantic Layer Agent"}' FROM SPECIFICATION $$ models: orchestration: "CORTEX_MODEL" instructions: response: 'Answer questions about business data using the dbt Semantic Layer. Present results clearly in plain English with context about what the metrics mean. When data is returned, summarize the key insight before showing details.' orchestration: 'Always use the dbt MCP tools to query metrics and dimensions rather than writing raw SQL. First explore available metrics and dimensions if you are unsure what is available, then construct and execute the appropriate semantic layer query.' sample_questions: - question: 'What are the top 10 products by revenue this quarter?' - question: 'How has inventory turnover trended over the last 12 months?' - question: 'Which regions have the highest order volume?' mcp_servers: - server_spec: name: "TARGET_DATABASE.TARGET_SCHEMA.MCP_SERVER_NAME" $$; ``` Update the `instructions` and `sample_questions` to match the metrics and dimensions in your own Semantic Layer. The `orchestration` instruction steers the agent toward the dbt Semantic Layer tools (like `list_metrics`, `get_dimensions`, and `query_metrics`) instead of writing raw SQL. ##### Step 4: Complete the OAuth flow The agent can't query dbt until each user authorizes it. Complete the OAuth flow once per user: 1. In the Snowflake Intelligence user interface, open your MCP connectors. Depending on your Snowflake version, this is under **Settings → User → MCP Connectors**, or in the **Connectors** panel of the agent's sources. 2. Find the **dbt Semantic Layer MCP** connector you created and select **Connect**. 3. Snowflake redirects you to dbt platform to sign in and approve the requested [scopes](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) on the consent screen. You can scope the connection to a specific project (recommended) so the agent only sees that project's data. 4. After you approve, the connector shows as **Connected** and you're returned to Snowflake. Snowflake self-registers with dbt platform through Dynamic Client Registration on first connect, so no admin action is needed to register it. In dbt platform, admins can review and audit the connected client, and manage sessions and scopes, in **Account settings → Integrations → App integrations**. For the full registration, consent, and session model, see [Connect apps with OAuth](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md). #### Verify the connection Open your agent in Snowflake Intelligence and ask one of its sample questions, such as *"What are the top 10 products by revenue this quarter?"* If the connection is working, the agent calls the dbt Semantic Layer tools and returns an answer grounded in your metrics. Remote MCP doesn't require AI features You don't need [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled to use remote MCP. If an admin turns AI features off, [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features) is the only tool hidden and every other tool keeps working. `text_to_sql` is also the only tool that consumes your dbt Copilot action allotment. Other MCP tools don't. If your account runs out of dbt Copilot actions, `text_to_sql` will not work until the limit resets. Other remote MCP tools keep working. If you need help, contact your account manager. #### Troubleshooting The connector won't authorize or the OAuth flow fails * Confirm your account has a [static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). OAuth with MCP requires one. * Verify the host in `API_ALLOWED_PREFIXES`, `OAUTH_TOKEN_ENDPOINT`, `OAUTH_AUTHORIZATION_ENDPOINT`, and `OAUTH_RESOURCE_URL` all match your MCP URL host exactly, and that the integration `ENABLED = TRUE`. * AI features are enabled by default and admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * Make sure remote MCP OAuth is available for your account plan. The agent returns no metrics or empty results * Confirm the project you authorized has a configured [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) with metrics and dimensions. * Check that the user who connected has at least read-only access to that project — the agent only sees what the user can see. * If you scoped the OAuth connection to a single project, make sure it's the project that contains your metrics. Permission errors when creating the integration or MCP server `CREATE API INTEGRATION` and `CREATE EXTERNAL MCP SERVER` require `ACCOUNTADMIN` (or a role with account-level `CREATE INTEGRATION`). Run steps 1–2 as `ACCOUNTADMIN`, then grant `USAGE` on the MCP server and integration to the role that runs the agent (Step 3). #### Related docs * [Set up the remote MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md) * [Available MCP tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md) * [Connect apps with OAuth](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md) * [Snowflake: MCP Connectors](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents-mcp-connectors) --- ### Integrate VS Code with MCP [Microsoft Visual Studio Code (VS Code)](https://code.visualstudio.com/mcp) is a powerful and popular integrated development environment (IDE). VS Code can connect to either the **self-hosted** dbt MCP server (runs on your machine, supports CLI commands like `dbt run`) or the **remote** dbt MCP server (HTTP, no install, consumption-focused). Before starting, make sure you have: * VS Code installed with the latest updates. * For self-hosted MCP: completed the [self-hosted MCP setup](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) and configured your dbt project paths. * For remote MCP: your **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. #### Set up with self-hosted dbt MCP server To get started, in VS Code: 1. Open the **Settings** menu and select the correct tab atop the page for your use case: * **Workspace**: Configures the server in the context of your workspace * **User**: Configures the server in the context of your user <br /> **Note for WSL users**: If you're using VS Code with Windows Subsystem for Linux (WSL), you'll need to configure WSL-specific settings. Run the **Preferences: Open Remote Settings** command from the **Command Palette** (F1) or select the **Remote** tab in the **Settings** editor. Local user settings are reused in WSL but can be overridden with WSL-specific settings. Configuring MCP servers in the local user settings will not work properly in a WSL environment. 2. Select **Features** --> **Chat** 3. Ensure that **MCP** is **Enabled** [![mcp-vscode-settings](/img/mcp/vscode_mcp_enabled_image.png?v=2 "mcp-vscode-settings")](#)mcp-vscode-settings 4. Open the command palette `Control/Command + Shift + P`, and select either: * **MCP: Open Workspace Folder MCP Configuration** — if you want to install the MCP server for this workspace * **MCP: Open User Configuration** — if you want to install the MCP server for the user 5. Add your server configuration (`dbt`) to the provided `mcp.json` file as one of the servers: tip You do not need to clone the dbt-mcp repository. Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` (or use the config below); cloning is only for contributing.  Self-hosted MCP with dbt platform OAuth Self-hosted MCP with OAuth is for users who want to use the dbt platform features. AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Choose your configuration based on your use case: ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication.  Self-hosted MCP (CLI only) For users who only want to use dbt commands with dbt v1 or dbt v2 (Applies to dbt v1.11 and later) ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_ENGINE_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` **Finding your paths:** * (Applies to dbt v1.11 and later) **DBT\_ENGINE\_PROJECT\_DIR**: Full path to the folder containing your `dbt_project.yml` file * macOS/Linux: Run `pwd` from your project folder. * Windows: Run `cd` from your project folder in Command Prompt. * **DBT\_PATH**: Path to dbt executable * macOS/Linux: Run `which dbt`. * Windows: Run `where dbt`.  Self-hosted MCP with .env For advanced users who need custom environment variables or service token authentication. Put your `.env` file in your *dbt project root* (same folder as `dbt_project.yml`) and use an absolute path with `--env-file`. Refer to the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) for the complete list of available environment variables for the self-hosted MCP server. Using the `env` field (single-file configuration): IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_DEV_ENV_ID`, and `DBT_USER_ID` must be numeric IDs (for example, `54321`), not full URLs copied from your browser. `DBT_HOST` accepts both `cloud.getdbt.com` and `https://cloud.getdbt.com`. (Applies to dbt v1.11 and later) ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_ENGINE_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_ENGINE_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` Using an `.env` file (alternative - two-file configuration): ```json { "servers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` 6. You can start, stop, and configure your MCP servers by: * Running the `MCP: List Servers` command from the Command Palette (Control/Command + Shift + P) and selecting the server. * Utilizing the keywords inline within the `mcp.json` file. [![VS Code inline management](/img/mcp/vscode_run_server_keywords_inline.png?v=2 "VS Code inline management")](#)VS Code inline management Now, you can access the dbt MCP server in VS Code through interfaces like GitHub Copilot. #### Set up with remote dbt MCP server The remote dbt MCP server runs in dbt platform — no `uvx` or self-hosted install needed. VS Code connects to it over HTTP from the same `mcp.json` you use for self-hosted servers. info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. 1. Open the command palette (`Control/Command + Shift + P`) and select one of: * **MCP: Open Workspace Folder MCP Configuration** — for this workspace. * **MCP: Open User Configuration** — for your user. 2. Get your MCP URL: You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). 3. Add a `dbt` entry under the top-level `servers` key. (VS Code uses `servers`, not `mcpServers`.) Pick the tab that matches your auth method: ##### OAuth (remote) *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). Add the following to `mcp.json`. VS Code opens a browser for sign-in and consent the first time the server connects. ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` Replace `YOUR_DBT_HOST_URL` with your hostname (for example, `abc123.us1.dbt.com`). You can find the URL in dbt platform under **Account settings** → **Access URLs** → **MCP Endpoint URL**. ##### Token-based Use token-based auth when your client doesn't yet support OAuth for HTTP MCP servers, or when you need a shared/CI setup. ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` For token-based remote MCP, set these headers in your client's MCP config: * **`Authorization`** *(required)* — `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Use a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with Developer access to the project that owns the environment in `x-dbt-prod-environment-id`. Assign service tokens the `Developer` permission set. PATs inherit their creator's permissions. Without Developer access, the connection fails with a `401` error. * **`x-dbt-prod-environment-id`** *(required)* — your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **`x-dbt-dev-environment-id`** — required for `execute_sql` and dbt v2 tools. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. * **`x-dbt-user-id`** — required for `execute_sql`. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. Use numeric IDs, not full URLs Headers like `x-dbt-prod-environment-id`, `x-dbt-dev-environment-id`, and `x-dbt-user-id` expect numeric IDs (for example, `54321`), not full URLs copied from your browser. The MCP host URL must include `https://`, whether you set it in a `url` field or pass it as an `mcp-remote` argument. If you use token-based auth, `execute_sql` requires a personal access token (PAT). Service tokens won't work. To run SQL without creating a PAT, connect with [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp) instead. Refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) for the full list of headers. 4. Save the file. Use **MCP: List Servers** from the command palette to start the server, then ask Copilot Chat a data-related question to confirm the connection. #### Troubleshooting This section contains troubleshooting steps for errors you might encounter when integrating VS Code with MCP. Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. Configuration not working in WSL (VS Code) **Symptoms:** VS Code MCP config works on other machines or native Windows but not in WSL. **Cause:** Local user settings are not applied in WSL environments. **Solution:** Configure MCP in the WSL-specific settings instead of local user settings: 1. Open the Command Palette → **Preferences: Open Remote Settings**. 2. Or select the **Remote** tab in the Settings editor. 3. Add your MCP server configuration there. Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). #### Resources * [Microsoft VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers) --- ### Integrate with other orchestration tools Alongside [dbt](https://docs.getdbt.com/docs/deploy/jobs.md), discover other ways to schedule and run your dbt jobs with the help of tools such as the ones described on this page. Build and install these tools to automate your data workflows, trigger dbt jobs (including those hosted on dbt), and enjoy a hassle-free experience, saving time and increasing efficiency. #### Airflow If your organization uses [Airflow](https://airflow.apache.org/), there are a number of ways you can run your dbt jobs, including: ##### dbt platform Installing the [dbt Provider](https://airflow.apache.org/docs/apache-airflow-providers-dbt-cloud/stable/index.html) to orchestrate dbt jobs. This package contains multiple Hooks, Operators, and Sensors to complete various actions within dbt. [![Airflow DAG using DbtCloudRunJobOperator](/img/docs/running-a-dbt-project/airflow_dbt_connector.png?v=2 "Airflow DAG using DbtCloudRunJobOperator")](#)Airflow DAG using DbtCloudRunJobOperator [![dbt job triggered by Airflow](/img/docs/running-a-dbt-project/dbt_cloud_airflow_trigger.png?v=2 "dbt job triggered by Airflow")](#)dbt job triggered by Airflow ##### dbt v1 Invoking dbt v1 jobs through the [BashOperator](https://registry.astronomer.io/providers/apache-airflow/modules/bashoperator). In this case, be sure to install dbt into a virtual environment to avoid issues with conflicting dependencies between Airflow and dbt. For more details on both of these methods, including example implementations, check out [this guide](https://docs.astronomer.io/learn/airflow-dbt-cloud). #### Automation servers Automation servers (such as CodeDeploy, GitLab CI/CD ([video](https://youtu.be/-XBIIY2pFpc?t=1301)), Bamboo and Jenkins) can be used to schedule bash commands for dbt. They also provide a UI to view logging to the command line, and integrate with your git repository. #### Azure Data Factory Integrate dbt and [Azure Data Factory](https://learn.microsoft.com/en-us/azure/data-factory/) (ADF) for a smooth data process from data ingestion to data transformation. You can seamlessly trigger dbt jobs upon completion of ingestion jobs by using the [dbt API](https://docs.getdbt.com/docs/dbt-apis/overview.md) in ADF. The following steps provides you with a detailed overview of how to trigger a dbt job using the API in Azure Data Factory. To use the dbt API to trigger a job in dbt through ADF: 1. In dbt, go to the job settings of the daily production job and turn off the scheduled run in the **Trigger** section. 2. You'll want to create a pipeline in ADF to trigger a dbt job. 3. Securely fetch the dbt service token from a key vault in ADF, using a web call as the first step in the pipeline. 4. Set the parameters in the pipeline, including the dbt account ID and job ID, as well as the name of the key vault and secret that contains the service token. * You can find the dbt job and account id in the URL, for example, if your URL is `https://YOUR_ACCESS_URL/deploy/88888/projects/678910/jobs/123456`, the account ID is 88888 and the job ID is 123456 5. Trigger the pipeline in ADF to start the dbt job and monitor the status of the dbt job in ADF. 6. In dbt, you can check the status of the job and how it was triggered in dbt. #### Cron Cron is a decent way to schedule bash commands. However, while it may seem like an easy route to schedule a job, writing code to take care of all of the additional features associated with a production deployment often makes this route more complex compared to other options listed here. #### Dagster If your organization uses [Dagster](https://dagster.io/), you can use the [dagster\_dbt](https://docs.dagster.io/_apidocs/libraries/dagster-dbt) library to integrate dbt commands into your pipelines. This library supports the execution of dbt through dbt or dbt v1. Running dbt from Dagster automatically aggregates metadata about your dbt runs. Refer to the [example pipeline](https://dagster.io/blog/dagster-dbt) for details. #### Databricks workflows Use Databricks workflows to call the dbt job API, which has several benefits such as integration with other ETL processes, utilizing dbt job features, separation of concerns, and custom job triggering based on custom conditions or logic. These advantages lead to more modularity, efficient debugging, and flexibility in scheduling dbt jobs. For more info, refer to the guide on [Databricks workflows and dbt jobs](https://docs.getdbt.com/guides/databricks-workflows.md). #### Kestra If your organization uses [Kestra](http://kestra.io/), you can leverage the [dbt plugin](https://kestra.io/plugins/plugin-dbt) to orchestrate dbt and dbt v1 jobs. Kestra's user interface (UI) has built-in [Blueprints](https://kestra.io/docs/user-interface-guide/blueprints), providing ready-to-use workflows. Navigate to the Blueprints page in the left navigation menu and [select the dbt tag](https://demo.kestra.io/ui/blueprints/community?selectedTag=36) to find several examples of scheduling dbt v1 commands and dbt jobs as part of your data pipelines. After each scheduled or ad-hoc workflow execution, the Outputs tab in the Kestra UI allows you to download and preview all dbt build artifacts. The Gantt and Topology view additionally render the metadata to visualize dependencies and runtimes of your dbt models and tests. The dbt task provides convenient links to easily navigate between Kestra and dbt UI. #### Orchestra If your organization uses [Orchestra](https://getorchestra.io), you can trigger dbt jobs using the dbt API. Create an API token from your dbt account and use this to authenticate Orchestra in the [Orchestra Portal](https://app.getorchestra.io). For details, refer to the [Orchestra docs on dbt](https://orchestra-1.gitbook.io/orchestra-portal/integrations/transformation/dbt-cloud). Orchestra automatically collects metadata from your runs so you can view your dbt jobs in the context of the rest of your data stack. The following is an example of the run details in dbt for a job triggered by Orchestra: [![Example of Orchestra triggering a dbt job](/img/docs/running-a-dbt-project/dbt_cloud_orchestra_trigger.png?v=2 "Example of Orchestra triggering a dbt job")](#)Example of Orchestra triggering a dbt job The following is an example of viewing lineage in Orchestra for dbt jobs: [![Example of a lineage view for dbt jobs in Orchestra](/img/docs/running-a-dbt-project/orchestra_lineage_dbt_cloud.png?v=2 "Example of a lineage view for dbt jobs in Orchestra")](#)Example of a lineage view for dbt jobs in Orchestra #### Prefect If your organization uses [Prefect](https://www.prefect.io/), the way you will run your jobs depends on the dbt version you're on, and whether you're orchestrating dbt or dbt v1 jobs. Refer to the following variety of options: [![Prefect DAG using a dbt job run flow](/img/docs/running-a-dbt-project/prefect_dag_dbt_cloud.jpg?v=2 "Prefect DAG using a dbt job run flow")](#)Prefect DAG using a dbt job run flow ##### Prefect 2 ##### dbt platform * Use the [trigger\_dbt\_cloud\_job\_run\_and\_wait\_for\_completion](https://prefecthq.github.io/prefect-dbt/cloud/jobs/#prefect_dbt.cloud.jobs.trigger_dbt_cloud_job_run_and_wait_for_completion) flow. * As jobs are executing, you can poll dbt to see whether or not the job completes without failures, through the [Prefect user interface (UI)](https://docs.prefect.io/ui/overview/). ##### dbt v1 * Use the [trigger\_dbt\_cli\_command](https://prefecthq.github.io/prefect-dbt/cli/commands/#prefect_dbt.cli.commands.trigger_dbt_cli_command) task. * For details on both of these methods, see [prefect-dbt docs](https://prefecthq.github.io/prefect-dbt/). ##### Prefect 1 ##### dbt platform * Trigger dbt jobs with the [DbtCloudRunJob](https://docs.prefect.io/api/latest/tasks/dbt.html#dbtcloudrunjob) task. * Running this task will generate a markdown artifact viewable in the Prefect UI. * The artifact will contain links to the dbt artifacts generated as a result of the job run. ##### dbt v1 * Use the [DbtShellTask](https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask) to schedule, execute, and monitor your dbt runs. * Use the supported [ShellTask](https://docs.prefect.io/api/latest/tasks/shell.html#shelltask) to execute dbt commands through the shell. #### Related docs * [dbt plans and pricing](https://www.getdbt.com/pricing/) * [Quickstart guides](https://docs.getdbt.com/guides.md) * [Webhooks for your jobs](https://docs.getdbt.com/docs/deploy/webhooks.md) * [Orchestration guides](https://docs.getdbt.com/guides.md) * [Commands for your production deployment](https://discourse.getdbt.com/t/what-are-the-dbt-commands-you-run-in-your-production-deployment-of-dbt/366) --- ### Jinja and macros #### Related reference docs * [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md) * [Jinja Template Designer Documentation](https://jinja.palletsprojects.com/page/templates/) (external link) * [dbt Jinja context](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) * [Macro properties](https://docs.getdbt.com/reference/macro-properties.md) #### Overview In dbt, you can combine SQL with [Jinja](https://jinja.palletsprojects.com), a templating language. Using Jinja turns your dbt project into a programming environment for SQL, giving you the ability to do things that aren't normally possible in SQL. It's important to note that Jinja itself isn't a programming language; instead, it acts as a tool to enhance and extend the capabilities of SQL within your dbt projects. For example, with Jinja, you can: * Use control structures (for example, `if` statements and `for` loops) in SQL * Use [environment variables](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) in your dbt project for production deployments * Change the way your project builds based on the current target. * Operate on the results of one query to generate another query, for example: * Return a list of payment methods, to create a subtotal column per payment method (pivot) * Return a list of columns in two relations, and select them in the same order to make it easier to union them together * Abstract snippets of SQL into reusable [**macros**](#macros) — these are analogous to functions in most programming languages. If you've used the [`{{ ref() }}` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), you're already using Jinja! Jinja can be used in any SQL in a dbt project, including [models](https://docs.getdbt.com/docs/build/sql-models.md), [analyses](https://docs.getdbt.com/docs/build/analyses.md), [data tests](https://docs.getdbt.com/docs/build/data-tests.md), and even [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md). Ready to get started with Jinja and macros? Check out the [tutorial on using Jinja](https://docs.getdbt.com/guides/using-jinja.md) for a step-by-step example of using Jinja in a model, and turning it into a macro! #### Getting started ##### Jinja Here's an example of a dbt model that leverages Jinja: /models/order\_payment\_method\_amounts.sql ```sql {% set payment_methods = ["bank_transfer", "credit_card", "gift_card"] %} select order_id, {% for payment_method in payment_methods %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount, {% endfor %} sum(amount) as total_amount from app_data.payments group by 1 ``` This query will get compiled to: /models/order\_payment\_method\_amounts.sql ```sql select order_id, sum(case when payment_method = 'bank_transfer' then amount end) as bank_transfer_amount, sum(case when payment_method = 'credit_card' then amount end) as credit_card_amount, sum(case when payment_method = 'gift_card' then amount end) as gift_card_amount, sum(amount) as total_amount from app_data.payments group by 1 ``` You can recognize Jinja based on the delimiters the language uses, which we refer to as "curlies": * **Expressions `{{ ... }}`**: Expressions are used when you want to output a string. You can use expressions to reference [variables](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) and call [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros). * **Statements `{% ... %}`**: Statements don't output a string. They are used for control flow, for example, to set up `for` loops and `if` statements, to [set](https://jinja.palletsprojects.com/en/3.1.x/templates/#assignments) or [modify](https://jinja.palletsprojects.com/en/3.1.x/templates/#expression-statement) variables, or to define macros. * **Comments `{# ... #}`**: Jinja comments are used to prevent the text within the comment from executing or outputting a string. Don't use `--` for comment. When used in a dbt model, your Jinja needs to compile to a valid query. To check what SQL your Jinja compiles to: * **Using dbt:** Click the compile button to see the compiled SQL in the Compiled SQL pane * **Using dbt v1:** Run `dbt compile` from the command line. Then open the compiled SQL file in the `target/compiled/{project name}/` directory. Use a split screen in your code editor to keep both files open at once. ##### Macros [Macros](https://docs.getdbt.com/docs/build/jinja-macros.md) in Jinja are pieces of code that can be reused multiple times – they are analogous to "functions" in other programming languages, and are extremely useful if you find yourself repeating code across multiple models. Macros are defined in `.sql` files, typically in your `macros` directory ([docs](https://docs.getdbt.com/reference/project-configs/macro-paths.md)). Macro files can contain one or more macros — here's an example: macros/cents\_to\_dollars.sql ```sql {% macro cents_to_dollars(column_name, scale=2) %} ({{ column_name }} / 100)::numeric(16, {{ scale }}) {% endmacro %} ``` A model which uses this macro might look like: models/stg\_payments.sql ```sql select id as payment_id, {{ cents_to_dollars('amount') }} as amount_usd, ... from app_data.payments ``` This would be *compiled* to: target/compiled/models/stg\_payments.sql ```sql select id as payment_id, (amount / 100)::numeric(16, 2) as amount_usd, ... from app_data.payments ``` 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). ##### Using a macro from a package A number of useful macros have also been grouped together into [packages](https://docs.getdbt.com/docs/build/packages.md) — our most popular package is [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/). After installing a package into your project, you can use any of the macros in your own project — make sure you qualify the macro by prefixing it with the [package name](https://docs.getdbt.com/reference/dbt-jinja-functions/project_name.md): ```sql select field_1, field_2, field_3, field_4, field_5, count(*) from my_table {{ dbt_utils.dimensions(5) }} ``` You can also qualify a macro in your own project by prefixing it with your [package name](https://docs.getdbt.com/reference/dbt-jinja-functions/project_name.md) (this is mainly useful for package authors). #### FAQs What parts of Jinja are dbt-specific? There are certain expressions that are specific to dbt — these are documented in the [Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) section of these docs. Further, docs blocks, snapshots, and materializations are custom Jinja *blocks* that exist only in dbt. Which docs should I use when writing Jinja or creating a macro? If you are stuck with a Jinja issue, it can get confusing where to check for more information. We recommend you check (in order): 1. [Jinja's Template Designer Docs](https://jinja.palletsprojects.com/page/templates/): This is the best reference for most of the Jinja you'll use 2. [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md): This maps which file types support Jinja expressions, built-in functions, and custom macros. 3. [Our Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md): This documents any additional functionality we've added to Jinja in dbt. 4. [Agate's table docs](https://agate.readthedocs.io/page/api/table.html): If you're operating on the result of a query, dbt will pass it back to you as an agate table. This means that the methods you call on the table belong to the Agate library rather than Jinja or dbt. Why do I need to quote column names in Jinja? In the [macro example](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) we passed the column name `amount` quotes: ```sql {{ cents_to_dollars('amount') }} as amount_usd ``` We have to use quotes to pass the *string* `'amount'` to the macro. Without the quotes, the Jinja parser will look for a variable named `amount`. Since this doesn't exist, it will compile to nothing. Quoting in Jinja can take a while to get used to! The rule is that you're within a Jinja expression or statement (i.e. within `{% ... %}` or `{{ ... }}`), you'll need to use quotes for any arguments that are strings. Single and double quotes are equivalent in Jinja – just make sure you match them appropriately. And if you do need to pass a variable as an argument, make sure you [don't nest your curlies](https://docs.getdbt.com/best-practices/dont-nest-your-curlies.md). My compiled SQL has a lot of spaces and new lines, how can I get rid of it? This is known as "whitespace control". Use a minus sign (`-`, e.g. `{{- ... -}}`, `{%- ... %}`, `{#- ... -#}`) at the start or end of a block to strip whitespace before or after the block (more docs [here](https://jinja.palletsprojects.com/page/templates/#whitespace-control)). Check out the [tutorial on using Jinja](https://docs.getdbt.com/guides/using-jinja.md#use-whitespace-control-to-tidy-up-compiled-code) for an example. Take caution: it's easy to fall down a rabbit hole when it comes to whitespace control! How do I debug my Jinja? You should get familiar with checking the compiled SQL in `target/compiled/<your_project>/` and the logs in `logs/dbt.log` to see what dbt is running behind the scenes. You can also use the [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) function to debug Jinja by printing objects to the command line. How do I document macros? To document macros, use a [properties file](https://docs.getdbt.com/reference/macro-properties.md) and nest the configurations under a `macros:` key #### Example macros/properties.yml ```yml macros: - name: cents_to_dollars description: A macro to convert cents to dollars arguments: - name: column_name type: column description: The name of the column you want to convert - name: precision type: integer description: Number of decimal places. Defaults to 2. ``` tip From dbt v1.10, you can opt into validating the arguments you define in macro documentation using the `validate_macro_args` behavior change flag. When enabled, dbt will: * Infer arguments from the macro and includes them in the [manifest.json](https://docs.getdbt.com/reference/artifacts/manifest-json.md) file if no arguments are documented. * Raise a warning if documented argument names don't match the macro definition. * Raise a warning if `type` fields don't follow [supported formats](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). Learn more about [macro argument validation](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). #### Document a custom materialization When you create a [custom materialization](https://docs.getdbt.com/guides/create-new-materializations.md), dbt creates an associated macro with the following format: ```text materialization_{materialization_name}_{adapter} ``` To document a custom materialization, use the previously mentioned format to determine the associated macro name(s) to document. macros/properties.yml ```yaml macros: - name: materialization_my_materialization_name_default description: A custom materialization to insert records into an append-only table and track when they were added. - name: materialization_my_materialization_name_xyz description: A custom materialization to insert records into an append-only table and track when they were added. ``` Why does my dbt output have so many macros in it? The output of a dbt run counts over 100 macros in your project! ```shell $ dbt run Running with dbt=1.7.0 Found 1 model, 0 tests, 0 snapshots, 0 analyses, 138 macros, 0 operations, 0 seed files, 0 sources ``` This is because dbt ships with its own project, which also includes macros! You can learn more about this [here](https://discourse.getdbt.com/t/did-you-know-dbt-ships-with-its-own-project/764). #### dbtonic Jinja Just like well-written python is pythonic, well-written dbt code is dbtonic. ##### Favor readability over DRY-ness Once you learn the power of Jinja, it's common to want to abstract every repeated line into a macro! Remember that using Jinja can make your models harder for other users to interpret — we recommend favoring readability when mixing Jinja with SQL, even if it means repeating some lines of SQL in a few places. If all your models are macros, it might be worth re-assessing. ##### Leverage package macros Writing a macro for the first time? Check whether we've open sourced one in [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) that you can use, and save yourself some time! ##### Set variables at the top of a model `{% set ... %}` can be used to create a new variable, or update an existing one. We recommend setting variables at the top of a model, rather than hardcoding it inline. This is a practice borrowed from many other coding languages, since it helps with readability, and comes in handy if you need to reference the variable in two places: ```sql -- 🙅 This works, but can be hard to maintain as your code grows {% for payment_method in ["bank_transfer", "credit_card", "gift_card"] %} ... {% endfor %} -- ✅ This is our preferred method of setting variables {% set payment_methods = ["bank_transfer", "credit_card", "gift_card"] %} {% for payment_method in payment_methods %} ... {% endfor %} ``` --- ### Job commands dbt platform A dbt production job allows you to set up a system to run a dbt job and job commands on a schedule, rather than running dbt commands manually from the command line or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). A job consists of commands that are "chained" together and executed as run steps. Each run step can succeed or fail, which may determine the job's run status (Success, Cancel, or Error). Each job allows you to: * Configure job commands * View job run details, including timing, artifacts, and detailed run steps * Access logs to view or help debug issues and historical invocations of dbt * Set up notifications, and [more](https://docs.getdbt.com/docs/deploy/deployments.md#dbt-cloud) #### Job command types Job commands are specific tasks executed by the job, and you can configure them seamlessly by either adding [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) or using the checkbox option in the **Commands** section. During a job run, the commands are "chained" together and executed as run steps. When you add a dbt command in the **Commands** section, you can expect different outcomes compared to the checkbox option. [![Configuring checkbox and commands list](/img/docs/dbt-platform/using-dbt-platform/job-commands.gif?v=2 "Configuring checkbox and commands list")](#)Configuring checkbox and commands list ##### Built-in commands Every job invocation automatically includes the [`dbt deps`](https://docs.getdbt.com/reference/commands/deps.md) command, meaning you don't need to add it to the **Commands** list in your job settings. You will also notice every job will include a run step to reclone your repository and connect to your data platform, which can affect your job status if these run steps aren't successful. **Job outcome** — During a job run, the built-in commands are "chained" together. This means if one of the run steps in the chain fails, then the next commands aren't executed, and the entire job fails with an "Error" job status. [![A failed job that had an error during the dbt deps run step.](/img/docs/dbt-platform/using-dbt-platform/fail-dbtdeps.png?v=2 "A failed job that had an error during the dbt deps run step.")](#)A failed job that had an error during the dbt deps run step. ##### Command list You can add or remove as many dbt commands as necessary for every job. However, you need to have at least one dbt command. There are few commands listed as "dbt CLI" or "dbt v1" in the [dbt Command reference page](https://docs.getdbt.com/reference/dbt-commands.md) page. This means they are meant for use in dbt v1 or dbt CLI, and not in Studio IDE. To run linting in a job, enable **Linting** when you [set up a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs). Deploy jobs don't have this option, and you can't add the `dbt lint` command to the **Commands** list. Refer to [SQL linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) for details on how it works on v1 and v2, which files it checks, and what happens when it fails. (Applies to dbt v2.0 and later) For v2 jobs: * You don't need to add a docs command. Every job run refreshes [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) metadata automatically. Read [Set up a documentation job](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md?version=2#set-up-a-documentation-job) for more information. * To validate that your source data is up to date before your models are built, add `dbt source freshness` to the **Commands** list or select the **Run source freshness** checkbox. Read [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) for more information. Using selectors Use [selectors](https://docs.getdbt.com/reference/node-selection/syntax.md) as a powerful way to select and execute portions of your project in a job run. For example, to run tests for `one_specific_model`, use the selector: `dbt test --select one_specific_model`. The job will still run if a selector doesn't match any models. ###### Compare changes custom commands For users that have Advanced CI's [compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) feature enabled and selected the **dbt compare** checkbox, you can add custom dbt commands to optimize running the comparison (for example, to exclude specific large models, or groups of models with tags). Running comparisons on large models can significantly increase the time it takes for CI jobs to complete. [![Add custom dbt commands to when using dbt compare.](/img/docs/deploy/dbt-compare.jpg?v=2 "Add custom dbt commands to when using dbt compare.")](#)Add custom dbt commands to when using dbt compare. The following examples highlight how you can customize the dbt compare command box: * Exclude the large `fct_orders` model from the comparison to run a CI job on fewer or smaller models and reduce job time/resource consumption. Use the following command: ```sql --select state:modified --exclude fct_orders ``` * Exclude models based on tags for scenarios like when models share a common feature or function. Use the following command: ```sql --select state:modified --exclude tag:tagname_a tag:tagname_b ``` * Include models that were directly modified and also those one step downstream using the `modified+1` selector. Use the following command: ```sql --select state:modified+1 ``` ###### Job outcome During a job run, the commands are "chained" together and executed as run steps. If one of the run steps in the chain fails, then the subsequent steps aren't executed, and the job will fail. In the following example image, the first four run steps are successful. However, if the fifth run step (`dbt run --select state:modified+ --full-refresh --fail-fast`) fails, then the next run steps aren't executed, and the entire job fails. The failed job returns a non-zero [exit code](https://docs.getdbt.com/reference/exit-codes.md) and "Error" job status: [![A failed job run that had an error during a run step](/img/docs/dbt-platform/using-dbt-platform/skipped-jobs.png?v=2 "A failed job run that had an error during a run step")](#)A failed job run that had an error during a run step #### Job command failures Job command failures can mean different things for different commands. Some common reasons why a job command may fail: * **Failure at `dbt run`** — [`dbt run`](https://docs.getdbt.com/reference/commands/run.md) executes compiled SQL model files against the current target database. It will fail if there is an error in any of the built models. By default, if a model fails, its downstream models are also skipped. In dbt v1.12+, you can set [`on_error: continue`](https://docs.getdbt.com/reference/resource-configs/on_error.md) on a model to allow its downstream models to still attempt to run despite the failure. * **Failure at `dbt test`** — [`dbt test`](https://docs.getdbt.com/reference/commands/test.md) runs tests defined on models, sources, snapshots, and seeds. A test can pass, fail, or warn depending on its [severity](https://docs.getdbt.com/reference/resource-configs/severity.md). Unless you set [warnings as errors](https://docs.getdbt.com/reference/global-configs/warnings.md), only an error fails the command. Tests on upstream resources prevent downstream resources from running and a failed test will skip them. * **Failure at `dbt build`** — [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) runs models, tests, snapshots, and seeds. This command executes resources in the DAG-specified order. If any upstream resource fails, all downstream resources are skipped, and the command exits with an error code of `1`. In dbt v1.12+, you can set [`on_error: continue`](https://docs.getdbt.com/reference/resource-configs/on_error.md) on a model to allow its downstream models to still attempt to run when that model fails. * **Selector failures** * If a [`select`](https://docs.getdbt.com/reference/node-selection/set-operators.md) matches multiple nodes and one of the nodes fails, then the job will have an exit code `1` and the subsequent command will fail. If you specified the [`--fail-fast`](https://docs.getdbt.com/reference/global-configs/failing-fast.md) flag, then the first failure will stop the entire connection for any models that are in progress. * If a selector doesn't match any nodes, it's not considered a failure. #### Related docs * [Job creation best practices](https://discourse.getdbt.com/t/job-creation-best-practices-in-dbt-cloud-feat-my-moms-lasagna/2980) * [dbt Command reference](https://docs.getdbt.com/reference/dbt-commands.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) * [Build and view your docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) --- ### Job notifications dbt platform Set up notifications in dbt platform to receive alerts about the outcome of a job run. You can choose to be notified by one or more of the following job run outcomes: * **Succeeds** option — A job run completed successfully with no warnings or errors. * **Warns** option — A job run encountered warnings from [data tests](https://docs.getdbt.com/docs/build/data-tests.md) or [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) checks. * This notification is triggered by warning-level log lines from those steps, not the job's overall run status. A job that shows "success" in the user interface can still trigger a warn notification if test or freshness steps logged warnings. * **Fails** option — A job run failed to complete. * **Is canceled** option — A job run is canceled. * You may not see Slack notifications for runs canceled by dbt platform's inactivity-timeout cleanup process (which cancels runs that have been inactive for 10 minutes). To receive notifications for these cancellations, use [email notifications](#email-notifications) or [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md). ##### Notification options dbt platform currently supports the following notification channels: * [Email](#email-notifications) * [Slack (user-linked)](#slack-notifications) * [Slack (account-level)](#slack-notifications-account) * [Microsoft Teams](#microsoft-teams-notifications) Microsoft Teams If you can’t use the native Microsoft Teams integration, you can still send job notifications to a Teams channel by using the channel’s email address as an external email, as explained in the next section. #### Email notifications You can receive email alerts about jobs by configuring the dbt email notification settings. ##### Prerequisites * You must be a *developer user*, *account admin*, or have the **Notification Manager** [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to configure email notifications in dbt. For more details, refer to [Users and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * As a developer user, you can set up email notifications for yourself. Once an account admin links the Slack app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access ##### Configure email notifications 1. Select your profile icon and then click **Notification settings**. 2. By default, dbt sends notifications to the email address that's in your **User profile** page. If you're an account admin, you can choose a different email address to receive notifications: 1. Under Job notifications, click the **Notification email** dropdown. 2. Select another address from the list. The list includes **Internal Users** with access to the account and **External Emails** that have been added. 3. To add an external email address, click the **Notification email** dropdown 4. Click **Add external email**. 5. Enter the email address, and click Add user. After adding an external email, it becomes available for selection in the **Notification email** dropdown list. External emails can be addresses that are outside of your dbt account and also for third-party integrations like [channels in Microsoft Teams](https://support.microsoft.com/en-us/office/tip-send-email-to-a-channel-2c17dbae-acdf-4209-a761-b463bdaaa4ca) and [PagerDuty email integration](https://support.pagerduty.com/docs/email-integration-guide). note External emails and their notification settings persist until edited or removed even if you remove the admin who added them from the account. [![Example of the Notification email dropdown](/img/docs/deploy/example-notification-external-email.png?v=2 "Example of the Notification email dropdown")](#)Example of the Notification email dropdown 3. Select the **Environment** for the jobs you want to receive notifications about from the dropdown. 4. Click **Edit** to configure the email notification settings. Choose one or more of the run statuses for each job you want to receive notifications about. 5. When you're done with the settings, click **Save**. As an account admin, you can add more email recipients by choosing another **Notification email** from the dropdown, **Edit** the job notification settings, and **Save** the changes. To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. [![Example of the Email notifications page](/img/docs/deploy/example-email-notification-settings-page.png?v=2 "Example of the Email notifications page")](#)Example of the Email notifications page ##### Unsubscribe from email notifications 1. Select your profile icon and click on **Notification settings**. 2. On the **Email notifications** page, click **Unsubscribe from all email notifications**. ##### Send job notifications to a Microsoft Teams channel (email) You can send dbt job [notification emails](#configure-email-notifications) directly to a Microsoft Teams channel by using the channel’s email address. 1. In Microsoft Teams, get the email address for the channel you want to send notifications to. See [Send an email to a channel](https://support.microsoft.com/en-us/office/tip-send-email-to-a-channel-2c17dbae-acdf-4209-a761-b463bdaaa4ca). 2. In dbt platform, click on your profile in the left sidebar and then click **Notification settings**. 3. Under **Job notifications**, click the **Notification email** dropdown. 4. To add an external email address, click **Add external email** at the bottom of the dropdown. 5. Enter the Teams channel email address, and click **Add user**. 6. Make sure you select the Teams channel email from the **Notification email** dropdown (it might be selected already). 7. Then choose the environment for the jobs you want to receive notifications from. 8. Click **Edit**, select the job statuses you want. Then click **Save** to save. #### Slack notifications (user) You can receive Slack alerts about jobs by setting up the Slack integration and then configuring the dbt Slack notification settings. dbt integrates with Slack via OAuth to ensure secure authentication. This Slack integration is set at the user level, not to be confused with the [Slack notifications at the account level](#slack-notifications-account) feature. note Virtual Private Cloud (VPC) admins must [contact support](mailto:support@getdbt.com) to complete the Slack integration. If there has been a change in user roles or Slack permissions where you no longer have access to edit a configured Slack channel, please [contact support](mailto:support@getdbt.com) for assistance. ##### Prerequisites * You have a Slack workspace that you want to receive job notifications from. * You must be a Slack Workspace Owner. * You must be an account admin to configure Slack notifications in dbt. For more details, refer to [Users and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). The integration only supports *public* channels in the Slack workspace at this moment. Once an account admin links the Slack app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access ##### Set up the Slack integration 1. Select **Account settings** and then select **Integrations** from the left sidebar. 2. Locate the **OAuth** section with the Slack application and click **Link**. [![Link for the Slack app](/img/docs/dbt-platform/Link-your-Slack-Profile.png?v=2 "Link for the Slack app")](#)Link for the Slack app ###### Logged in to Slack If you're already logged in to Slack, the handshake only requires allowing the app access. If you're a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner. [![Allow dbt access to Slack](/img/docs/dbt-platform/Allow-dbt-to-access-slack.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ###### Logged out If you're logged out or the Slack app/website is closed, you must authenticate before completing the integration. 1. Complete the field defining the Slack workspace you want to integrate with dbt. [![Define the workspace](/img/docs/dbt-platform/define-workspace.png?v=2 "Define the workspace")](#)Define the workspace 2. Sign in with an existing identity or use the email address and password. 3. Once you have authenticated successfully, accept the permissions. [![Allow dbt access to Slack](/img/docs/dbt-platform/accept-permissions.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ##### Configure Slack notifications 1. Select your profile icon and then click on **Notification settings**. 2. Select **Slack notifications** in the left sidebar. 3. Select the **Notification channel** you want to receive the job run notifications from the dropdown. [![Example of the Notification channel dropdown](/img/docs/deploy/example-notification-slack-channels.png?v=2 "Example of the Notification channel dropdown")](#)Example of the Notification channel dropdown 4. Select the **Environment** for the jobs you want to receive notifications about from the dropdown. 5. Click **Edit** to configure the Slack notification settings. Choose one or more of the run statuses for each job you want to receive notifications about. 6. When you're done with the settings, click **Save**. To send alerts to another Slack channel, select another **Notification channel** from the dropdown, **Edit** those job notification settings, and **Save** the changes. To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. [![Example of the Slack notifications page](/img/docs/deploy/example-slack-notification-settings-page.png?v=2 "Example of the Slack notifications page")](#)Example of the Slack notifications page ##### Disable the Slack integration 1. Select **Account settings** and on the **Integrations** page, scroll to the **OAuth** section. 2. Click the **X** icon (on the far right of the Slack integration) and click **Unlink**. Channels that you configured will no longer receive Slack notifications. *This is not an account-wide action.* Channels configured by other account admins will continue to receive Slack notifications if they still have active Slack integrations. To migrate ownership of a Slack channel notification configuration, have another account admin edit their configuration. #### Slack notifications (account) Integrate Slack with dbt platform at the account level to receive job notifications in Slack. dbt integrates with Slack via OAuth to ensure secure authentication. A single dbt platform account can integrate with one Slack workspace. ##### Prerequisites * You have a Slack workspace that you want to receive job notifications from. * A dbt platform account admin must link the Slack app at the account level. * Install the official dbt platform Slack app using the [steps outlined in the next section](#set-up-the-slack-integration-1). * To install the Slack app to a workspace, your Slack org must permit app installations. In some orgs this requires a Slack admin approval. * The integration only supports *public* channels in the Slack workspace at this moment. Once an account admin links the Slack app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access IT licenses can receive job notifications but can't [configure them](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md?version=1.12). Only **Account Admins** can connect or disconnect the Slack workspace. ##### Set up the Slack integration The account-level Slack integration uses the official dbt platform Slack app, which is separate from the [user-linked Slack integration](#slack-notifications). To use Slack notifications, you must unlink the old Slack app and then connect the official app: 1. Go to **Account settings** > **Integrations** > **OAuth**. 2. Click the **X** icon next to Slack and select **Unlink**. 3. In the same OAuth section, click **Link** to connect the official Slack app. Until you do this, the account-level Slack option will not appear. [![Link for the Slack app](/img/docs/dbt-platform/Link-your-Slack-Profile.png?v=2 "Link for the Slack app")](#)Link for the Slack app ##### Logged in to Slack If you're already logged in to Slack, the integration only requires allowing the app access. If you're a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner. [![Allow dbt access to Slack](/img/docs/dbt-platform/Allow-dbt-to-access-slack.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ##### Logged out If you're logged out or the Slack app/website is closed, you must authenticate before completing the integration. 1. Complete the field defining the Slack workspace you want to integrate with dbt. [![Define the workspace](/img/docs/dbt-platform/define-workspace.png?v=2 "Define the workspace")](#)Define the workspace 2. Sign in with an existing identity or use the email address and password. 3. Once you have authenticated successfully, accept the permissions. [![Allow dbt access to Slack](/img/docs/dbt-platform/accept-permissions.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ##### Configure Slack notifications Configure the Slack channel you want to receive job notifications from. 1. Select your profile icon and then click on **Notification settings**. 2. Select **Slack notifications** in the left sidebar. 3. From the first dropdown, select the **Notification channel** you want to receive the job run notifications. [![Example of the Notification channel dropdown](/img/docs/deploy/example-notification-slack-channels.png?v=2 "Example of the Notification channel dropdown")](#)Example of the Notification channel dropdown 4. From the second dropdown, select the **Environment** for the jobs you want to receive notifications about. 5. Click **Edit** to configure the Slack notification settings. Choose one or more of the run statuses for each job you want to receive notifications about. 6. When you're done with the settings, click **Save**. * To send alerts to another Slack channel, select another **Notification channel** from the dropdown, **Edit** those job notification settings, and **Save** the changes. * To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. [![Example of the Slack notifications page](/img/docs/deploy/example-slack-notification-settings-page.png?v=2 "Example of the Slack notifications page")](#)Example of the Slack notifications page That's it! Your Slack channel is now set up to receive dbt job notifications at the account level. This integration is now available throughout the account for all licensed users. ##### Migrating legacy Slack notification settings If you previously configured Slack notifications with the legacy integration, you may see a migration banner on the **Slack notifications** page. [![Example of the Slack migration banner](/img/docs/deploy/migrating-legacy-slack-notifications.png?v=2 "Example of the Slack migration banner")](#)Example of the Slack migration banner The banner appears when all of the following are true: * You have notification settings from a previous Slack integration. * Your account is connected to the dbt platform app. * You have not yet configured Slack notifications with the dbt platform app. Before migrating, you must unlink the legacy Slack integration and link the dbt platform app. Unlinking the legacy integration is a manual step, and only one Slack app can be linked at a time. info The dbt platform Slack app sends job notifications to *public* only channels in your workspace. Private channels aren't supported. 1. Click **Migrate settings** to copy your existing settings to the dbt platform app, including: * Your selected **Notification channel** and **Environment** * Your selected jobs * Your notification toggles (for example, **Succeeds**, **Warns**, **Fails**, and **Is canceled**) 2. Click **Dismiss** to hide the banner for your current session — it reappears on reload until migration completes. When migration succeeds, dbt hides the banner and refreshes your Slack notification settings. If migration fails, the banner remains so you can try again. [![Example of invite guidance for the dbt platform app](/img/docs/deploy/dbt-platform-slack-invite.png?v=2 "Example of invite guidance for the dbt platform app")](#)Example of invite guidance for the dbt platform app ##### Disable the Slack integration In this step, you'll disable the Slack integration and remove the account-level Slack credentials. You can always re-enable the integration by following the [Set up the Slack integration](#set-up-the-slack-integration-1) steps. 1. Select **Account settings** and on the **Integrations** page, scroll to the **OAuth** section. 2. Click the **X** icon (on the far right of the Slack integration) and click **Unlink**. * This removes the account-level Slack credentials. All Slack notifications that rely on the account-level integration will stop sending. * If any legacy, user-linked Slack integrations still exist, those notifications may continue until the legacy link is removed. We recommend migrating to the new account-level app and removing legacy links. #### Microsoft Teams notifications You can receive Microsoft Teams alerts for your dbt jobs by connecting your Teams account to the dbt platform, and configuring your notification preferences. dbt integrates with Teams through Microsoft Entra to authenticate securely when you link Teams and configure notification settings. ##### Prerequisites Before you begin: * You must have a dbt platform account * You have a Microsoft Teams account that you want to receive job notifications from. * Make sure you have permission to view the **Account integrations** and **Job notifications** pages in dbt platform. Once an account admin links the Microsoft Teams app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access ##### Set up Microsoft Teams To enable Microsoft Teams job notifications, complete the following sections: 1. [Link dbt platform account to Teams](#link-dbt-platform-account-to-teams) — A user-level connection that links an individual dbt platform account (or a dedicated service account) to a Microsoft Teams user profile within your tenant. 2. [Configure Teams notifications](#configure-teams-notifications) — Configures which Teams channels receive job notifications. 3. [Disable the Teams integration](#disable-the-teams-integration) (optional) — Remove or reset the connection between dbt platform and Microsoft Teams. ##### Link dbt platform account to Teams info You can link any Teams user account from your tenant, but we recommend creating a dedicated account just for posting dbt notifications. During the OAuth process, you’ll need to sign in to a Microsoft account to complete the integration. * If you’re logged into a single Microsoft account, the integration will complete automatically. * If you’re logged into multiple accounts (or none), you’ll be prompted to select or log in to one.  Image of the Microsoft account selection popup [![Example of the Microsoft account popup](/img/docs/deploy/pick-account.png?v=2 "Example of the Microsoft account popup")](#)Example of the Microsoft account popup To link your dbt platform account to Microsoft Teams: 1. In dbt platform, go to the **Account settings** page by clicking on your account name and selecting **Account settings**. 2. In the left sidebar, select **Integrations**. 3. Scroll to the **OAuth** section. 4. Next to **Teams** and click on the **Link** button. 5. After doing this, you’ll either be prompted to choose your Microsoft account before completing the setup, or return directly to the dbt platform with your Teams profile linked. 6. Your dbt platform account is now linked to Microsoft Teams! dbt will now add the **dbt-cloud-integration app** to your Microsoft Entra tenant. This app manages authentication requests and permissions securely. [![Example of the dbt-cloud-integration app overview](/img/docs/deploy/dbt-cloud-integrations.png?v=2 "Example of the dbt-cloud-integration app overview")](#)Example of the dbt-cloud-integration app overview * The current Entra app permissions are: * `profile` * `openid` * `offline_access` * `Team.ReadBasic.All` * `TeamsActivity.Send` * `ChannelMessage.Send` * `ChannelMessage.Read.All` * `Channel.ReadBasic.All` info The `dbt-cloud-integration` app is a multi-tenant Microsoft Entra app that dbt Labs already owns. You don't need to pre-register anything in your tenant. Approving the consent prompt during **Link** registers the app as an Enterprise Application in your tenant. This follows the same pattern as other multi-tenant SaaS apps, for example Slack or Zoom. ##### Troubleshooting the Teams link If the **Link** flow fails with `AADSTS90094: Admin consent is required`, your Microsoft Entra tenant restricts user consent for third-party apps. To resolve this, have an Entra admin (Cloud Application Administrator or Global Administrator) do one of the following: * Complete the **Link** step themselves and approve the consent prompt. * Grant admin consent for the `dbt-cloud-integration` app from **Enterprise applications** in the Entra admin center. After admin consent is granted, retry the **Link** flow in dbt platform. For more information, refer to the [Microsoft documentation](https://learn.microsoft.com/en-us/answers/questions/5637987/corporate-account-apps-registration-error-aadsts90). ##### Configure Teams notifications Once you’ve connected dbt platform and Teams, you can configure which Teams channels receive job notifications. The **Teams notifications** menu requires that you have an active integration with Teams on the account. info Currently, dbt only sends notifications to Teams channels (standard, shared, or private) that you belong to. 1. In the dbt platform, click your profile icon and select **Notification settings**. 2. Select **Teams notifications** in the left sidebar. 3. From the first dropdown, select the **Notification team** that you want to send notifications to. 4. From the second dropdown, select the **Notification channel** you want to send notifications to. * dbt platform only sends notifications to Teams channels (standard, shared, or private) that *you* belong to. 5. In the dropdown, choose the environment for the jobs you want to receive notifications about. 6. Click **Edit** on the top right to configure the Teams job notification settings and customize which job statuses trigger job notifications. 7. When finished, click **Save**. Your Teams channel is now set up to receive dbt job notifications! [![Example of the configure Teams notification page](/img/docs/deploy/configure-teams-notification.png?v=2 "Example of the configure Teams notification page")](#)Example of the configure Teams notification page ##### Disable the Teams integration Disabling and unlinking the Teams integration in the dbt platform removes it for the entire account. To disable it: 1. In the dbt platform, go to **Account settings**. 2. Click on **Integrations** and scroll down to **OAuth**. 3. On the far right of the **Teams** integration, click the **X** icon. 4. Confirm the unlinking by selecting **Unlink**. The Teams integration has been disabled. You can always re-enable the integration by following the [Set up Microsoft Teams](#set-up-microsoft-teams) steps. --- ### Job scheduler dbt platform info Use the [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) to investigate and troubleshoot dbt job and run failures by asking the agent about recent failures, root causes, and fixes — powered by the `troubleshooting-dbt-job-errors` skill in dbt Agent Skills. The job scheduler is the backbone of running jobs in dbt, bringing power and simplicity to building data pipelines in both continuous integration and production contexts. The scheduler frees teams from having to build and maintain their own infrastructure, and ensures the timeliness and reliability of data transformations. The scheduler enables both cron-based and event-driven execution of dbt commands in the user’s data platform. Specifically, it handles: * Cron-based execution of dbt jobs that run on a predetermined cadence * Event-driven execution of dbt jobs that run based on the completion of another job ([trigger on job completion](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion)) * Event-driven execution of dbt CI jobs triggered when a pull request is merged to the branch ([merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md)) * Event-driven execution of dbt jobs triggered by API * Event-driven execution of dbt jobs manually triggered by a user to **Run now** The scheduler handles various tasks including: * Queuing jobs * Creating temporary environments to run the dbt commands required for those jobs * Providing logs for debugging and remediation * Storing dbt artifacts for direct consumption/ingestion by the Discovery API The scheduler also: * Uses [dbt's Git repository caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) to protect against third-party outages and improve job run reliability. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Powers running dbt in staging and production environments, bringing ease and confidence to CI/CD workflows and enabling observability and governance in deploying dbt at scale. * Uses [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) to upload dbt v1 artifacts into dbt for central visibility, cross-project referencing, and easier collaboration. [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Uses [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) to decide what needs to be rebuilt based on upstream data freshness and code changes. #### Scheduler terms Familiarize yourself with these useful terms to help you understand how the job scheduler works. | Term | Definition | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Scheduler | The dbt engine that powers job execution. The scheduler queues scheduled or API-triggered job runs, prepares an environment to execute job commands in your cloud data platform, and stores and serves logs and artifacts that are byproducts of run execution. | | Job | A collection of run steps, settings, and a trigger to invoke dbt commands against a project in the user's cloud data platform. | | Job queue | The job queue acts as a waiting area for job runs when they are scheduled or triggered to run; runs remain in queue until execution begins. More specifically, the Scheduler checks the queue for runs that are due to execute, ensures the run is eligible to start, and then prepares an environment with appropriate settings, credentials, and commands to begin execution. Once execution begins, the run leaves the queue. | | Over-scheduled job | A situation when a cron-scheduled job's run duration becomes longer than the frequency of the job’s schedule, resulting in a job queue that will grow faster than the scheduler can process the job’s runs. | | Deactivated job | A situation where a job has reached 100 consecutive failing runs or belongs to an inactive account. | | Prep time | The time dbt takes to create a short-lived environment to execute the job commands in the user's cloud data platform. Prep time varies most significantly at the top of the hour when the dbt Scheduler experiences a lot of run traffic. | | Run | A single, unique execution of a dbt job. | | Run slot | Run slots control the number of jobs that can run concurrently. Each running job occupies a run slot for the duration of the run. To view the number of run slots available in your plan, check out the [dbt pricing page](https://www.getdbt.com/pricing).<br /><br />Starter and Developer plans are limited to one project each. For additional projects or more run slots, consider upgrading to an [Enterprise-tier plan](https://www.getdbt.com/pricing/). | | Threads | When dbt builds a project's DAG, it tries to parallelize the execution by using threads. The [thread](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md) count is the maximum number of paths through the DAG that dbt can work on simultaneously. The default thread count in a job is 4. | | Wait time | Amount of time that dbt waits before running a job, either because there are no available slots or because a previous run of the same job is still in progress. | #### Scheduler queue The scheduler queues a deployment job to be processed when it's triggered to run by a [set schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#schedule-days), [a job completed](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion), an API call, or manual action. Before the job starts executing, the scheduler checks these conditions to determine if the run can start executing: * **Is there a run slot that's available on the account for use?** — If all run slots are occupied, the queued run will wait. The wait time is displayed in dbt. If there are long wait times, [upgrading to an Enterprise-tier plan](https://www.getdbt.com/contact/) can provide more run slots and allow for higher job concurrency. * **Does this same job have a run already in progress?** — The scheduler executes distinct runs of the same dbt job serially to avoid model build collisions. If there's a job already running, the queued job will wait, and the wait time will be displayed in dbt. If there is an available run slot and there isn't an actively running instance of the job, the scheduler will prepare the job to run in your cloud data platform. This prep involves readying a Kubernetes pod with the right version of dbt installed, setting environment variables, loading data platform credentials, and Git provider authorization, amongst other environment-setting tasks. The time it takes to prepare the job is displayed as **Prep time** in the UI. [![An overview of a dbt job run](/img/docs/dbt-platform/deployment/deploy-scheduler.png?v=2 "An overview of a dbt job run")](#)An overview of a dbt job run ##### Treatment of CI jobs When compared to deployment jobs, the scheduler behaves differently when handling [continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md). It queues a CI job to be processed when it's triggered to run by a Git pull request, and the conditions the scheduler checks to determine if the run can start executing are also different: * **Will the CI run consume a run slot?** — CI runs don't consume run slots and will never block production runs. * **Does this same job have a run already in progress?** — CI runs can execute concurrently (in parallel). CI runs build into unique temporary schemas, and CI checks execute in parallel to help increase team productivity. Teammates never have to wait to get a CI check review. ##### Treatment of merge jobs When triggered by a *merged* Git pull request, the scheduler queues a [merge job](https://docs.getdbt.com/docs/deploy/merge-jobs.md) to be processed. * **Will the merge job run consume a run slot?** — Yes, merge jobs do consume run slots. * **Does this same job have a run already in progress?** — A merge job can only have one run in progress at a time. If there are multiple runs queued up, the scheduler will enqueue the most recent run and cancel all the other runs. If there is a run in progress, it will wait until the run completes before queuing the next run. #### Job memory In dbt, the setting to provision memory available to a job is defined at the account-level and applies to each job running in the account; the memory limit cannot be customized per job. If a running job reaches its memory limit, the run is terminated with a "memory limit error" message. Jobs consume a lot of memory in the following situations: * A high thread count was specified * Custom dbt macros attempt to load data into memory instead of pushing compute down to the cloud data platform Refer to [dbt architecture](https://docs.getdbt.com/docs/platform/about-platform/architecture.md) for an architecture diagram and to learn how the data flows. #### Run cancellation for over-scheduled jobs Scheduler won't cancel API-triggered jobs The scheduler will not cancel over-scheduled jobs triggered by the [API](https://docs.getdbt.com/docs/dbt-apis/overview.md). The dbt scheduler prevents too many job runs from clogging the queue by canceling unnecessary ones. If a job takes longer to run than its scheduled frequency, the queue will grow faster than the scheduler can process the runs, leading to an ever-expanding queue with runs that don’t need to be processed (called *over-scheduled jobs*). The scheduler prevents queue clog by canceling runs that aren't needed, ensuring there is only one run of the job in the queue at any given time. If a newer run is queued, the scheduler cancels any previously queued run for that job and displays an error message. [![The cancelled runs display an error message explaining why the run was cancelled and recommendations](/img/docs/dbt-platform/deployment/run-error-message.png?v=2 "The cancelled runs display an error message explaining why the run was cancelled and recommendations")](#)The cancelled runs display an error message explaining why the run was cancelled and recommendations To prevent over-scheduling, users will need to take action by either refactoring the job so it runs faster or modifying its [schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#schedule-days). #### Deactivation of jobs To reduce unnecessary resource consumption and reduce contention for run slots in your account, dbt will deactivate a [deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) or a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) if it reaches 100 consecutive failing runs. A banner containing this message is displayed when a job is deactivated: "Job has been deactivated due to repeated run failures. To reactivate, verify the job is configured properly and run manually or reenable any trigger". When this happens, scheduled and triggered-to-run jobs will no longer be enqueued. Jobs can also be deactivated when a dbt account is inactive. Account owners receive a warning after 90 days without account activity. If there is no activity for another 7 days, dbt deactivates jobs in the account. A banner containing this message is displayed when a job is deactivated because the account is inactive: "Job has been deactivated because the account is inactive. To reactivate, either log out and log back in to dbt, then wait up to 30 minutes for the system to reactivate the job, or manually edit and save the job." To reactivate a job deactivated due to repeated run failures, you can either: * Update the job's settings to fix the issue and save the job (recommended) * Perform a manual run by clicking **Run now** on the job's page To reactivate jobs deactivated because the account is inactive, either log out and log back in to dbt, then wait up to 30 minutes for the system to reactivate the job, or manually edit and save the job. #### FAQs I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job If you're receiving a `This run exceeded your account's run memory limits` error in your failed job, it means that the job exceeded the [memory limits](https://docs.getdbt.com/docs/deploy/job-scheduler.md#job-memory) set for your account. All dbt accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices. ##### Common reasons Some common reasons for higher memory usage are: * dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient. * dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries. ##### Resolution There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. Video example As an additional resource, check out [this example video](https://www.youtube.com/watch?v=sTqzNaFXiZ8), which demonstrates how to refactor the sample code by reducing the number of rows returned. If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! ##### Additional resources * [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) #### Related docs * [dbt architecture](https://docs.getdbt.com/docs/platform/about-platform/architecture.md#dbt-cloud-features-architecture) * [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * [dbt continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) --- ### Jobs in the dbt platform dbt platform info Use the [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) to investigate and troubleshoot dbt job and run failures by asking the agent about recent failures, root causes, and fixes — powered by the `troubleshooting-dbt-job-errors` skill in dbt Agent Skills. These are the available job types in dbt: * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) — Build production data assets. Runs on a schedule, by API, or after another job completes. * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) — Test and validate code changes before merging. Triggered by commit to a PR or by API. * [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) — Deploy merged changes into production. Runs after a successful PR merge or by API. * [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) — Reuse an existing node by cloning it or skipping the rebuild when the logic and data haven't changed. The following comparison table describes the behaviors of the different job types: | | **Deploy jobs** | **CI jobs** | **Merge jobs** | **dbt State** | | ---------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. | Build merged changes into production or update state for deferral. | Reuse a node when its logic and data haven't changed. Rebuild it only when the result would be different. | | Trigger types | Triggered by a schedule, API, or the successful completion of another job. | Triggered by a commit to a PR or by API. | Triggered by a successful merge into the environment's branch or by API. | Evaluated on every run. A node is skipped or cloned when its logic is unchanged and upstream data is still within `lag_tolerance`. | | Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. | Builds into a production database and schema. | Builds into a production database and schema. | | Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. | Runs execute sequentially, so as to not have collisions on the underlying DAG. | | | Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. | N/A | Skips the rebuild, or clones an existing node, when the logic and data haven't changed. | | State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | | | Job run duration | Limit is 24 hours. | Limit is 24 hours. | Limit is 24 hours. | Limit is 24 hours. | --- ### Joins Joins are a powerful part of MetricFlow and simplify the process of making all valid dimensions available for your metrics at query time, regardless of where they are defined in different semantic models. With Joins, you can also create metrics using (Applies to dbt v1.12 and later) simple metrics from different semantic models. Joins use `entities` defined in your semantic model configs as the join keys between tables. Assuming entities are defined in the semantic model, MetricFlow creates a graph using the semantic models as nodes and the join paths as edges to perform joins automatically. MetricFlow chooses the appropriate join type and avoids fan-out or chasm joins with other tables based on the entity types.  What are fan-out or chasm joins? * Fan-out joins are when one row in a table is joined to multiple rows in another table, resulting in more output rows than input rows. * Chasm joins are when two tables have a many-to-many relationship through an intermediate table, and the join results in duplicate or missing data. #### Types of joins Joins are auto-generated MetricFlow automatically generates the necessary joins to the defined semantic objects, eliminating the need for you to create new semantic models or configuration files. This section explains the different types of joins that can be used with entities and how to query them. Metricflow uses these specific join strategies: * Primarily uses left joins when joining `fct` and `dim` models. Left joins make sure all rows from the "base" table are retained, while matching rows are included from the joined table. * For queries that involve multiple `fct` models, MetricFlow uses full outer joins to ensure all data points are captured, even when some `dim` or `fct` models are missing in certain tables. * MetricFlow restricts the use of fan-out and chasm joins. Refer to [SQL examples](#sql-examples) for more information on how MetricFlow handles joins in practice. The following table identifies which joins are allowed based on specific entity types to prevent the creation of risky joins. This table primarily represents left joins unless otherwise specified. For scenarios involving multiple `fct` models, MetricFlow uses full outer joins. | entity type - Table A | entity type - Table B | Join type | | --------------------- | --------------------- | ------------------------ | | Primary | Primary | ✅ Left | | Primary | Unique | ✅ Left | | Primary | Foreign | ❌ Fan-out (Not allowed) | | Unique | Primary | ✅ Left | | Unique | Unique | ✅ Left | | Unique | Foreign | ❌ Fan-out (Not allowed) | | Foreign | Primary | ✅ Left | | Foreign | Unique | ✅ Left | | Foreign | Foreign | ❌ Fan-out (Not allowed) | ##### Semantic validation MetricFlow performs semantic validation by executing `explain` queries in the data platform to ensure that the generated SQL gets executed without errors. This validation includes: * Verifying that all referenced tables and columns exist. * Ensuring the data platform supports SQL functions, such as `date_diff(x, y)`. * Checking for ambiguous joins or paths in multi-hop joins. If validation fails, MetricFlow surfaces errors for users to address before executing the query. #### Example The following example uses two semantic models with a common entity and shows a MetricFlow query that requires a join between the two semantic models: `transactions` and `user_signup`. (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: transactions_model semantic_model: enabled: true name: transactions columns: - name: id entity: name: transaction type: primary - name: user_id entity: name: user type: foreign metrics: - name: average_purchase_price type: simple agg: average expr: purchase_price - name: user_model semantic_model: enabled: true name: user_signup columns: - name: user_id entity: name: user type: primary - name: type dimension: type: categorical ``` * MetricFlow uses `user_id` as the join key to link two semantic models, `transactions` and `user_signup`. This allows you to query the `average_purchase_price` metric in the `transactions` semantic model, grouped by the `type` dimension in the `user_signup` semantic model. * Note that the `average_purchase_price` simple metric is defined in `transactions`, where `user_id` is a foreign entity. However, `user_signup` has `user_id` as a primary entity. * Since `user_id` is a foreign key in `transactions` and a primary key in `user_signup`, MetricFlow performs a left join where `transactions` joins `user_signup` to access the `average_purchase_price` simple metric defined in `transactions`. * To query dimensions from different semantic models, add a double underscore (or dunder) to the dimension name after joining the entity in your editing tool. The following query, `user_id__type` is included as a dimension using the `--group-by` flag (`type` is the dimension). ```yaml dbt sl query --metrics average_purchase_price --group-by metric_time,user_id__type # In dbt platform /> ``` ```yaml mf query --metrics average_purchase_price --group-by metric_time,user_id__type # In dbt v1 ``` ###### SQL examples These SQL examples show how MetricFlow handles both left join and full outer join scenarios in practice: ##### SQL example for left join Using the previous example for `transactions` and `user_signup` semantic models, this shows a left join between those two semantic models. ```sql select transactions.user_id, transactions.purchase_price, user_signup.type from transactions left outer join user_signup on transactions.user_id = user_signup.user_id where transactions.purchase_price is not null group by transactions.user_id, user_signup.type; ``` ##### SQL example for outer joins If you have multiple `fct` models, let's say `sales` and `returns`, MetricFlow uses full outer joins to ensure all data points are captured. This example shows a full outer join between the `sales` and `returns` semantic models. ```sql select sales.user_id, sales.total_sales, returns.total_returns from sales full outer join returns on sales.user_id = returns.user_id where sales.user_id is not null or returns.user_id is not null; ``` #### Multi-hop joins MetricFlow allows users to join (Applies to dbt v1.12 and later) simple metrics and dimensions across a graph of entities by moving from one table to another within a graph. This is referred to as "multi-hop join". MetricFlow can join up to three tables, supporting multi-hop joins with a limit of two hops. This does the following: * Enables complex data analysis without ambiguous paths. * Supports navigating through data models, like moving from `orders` to `customers` to `country` tables. While direct three-hop paths are limited to prevent confusion from multiple routes to the same data, MetricFlow does allow joining more than three tables if the joins don’t exceed two hops to reach a dimension. For example, if you have two models, `country` and `region`, where customers are linked to countries, which in turn are linked to regions, you can join all of them in a single SQL query and can dissect `orders` by `customer__country_country_name` but not by `customer__country__region_name`. [![Multi-Hop-Join](/img/docs/building-a-dbt-project/multihop-diagram.png?v=2 "Multi-Hop-Join")](#)Multi-Hop-Join (Applies to dbt v1.12 and later) Notice how the schema can be translated into the following three MetricFlow semantic models to create the metric 'Average purchase price by country' using the `purchase_price` simple metric from the sales table and the `country_name` dimension from the `country_dim` table. models/model.yml ```yaml models: - name: sales_model semantic_model: enabled: true name: sales agg_time_dimension: first_ordered_at columns: - name: id entity: type: primary - name: user_id entity: type: foreign - name: purchase_price # Column for the simple metric expression - name: first_ordered_at granularity: day dimension: type: time name: metric_time metrics: - name: average_purchase_price type: simple agg: avg expr: purchase_price - name: user_signup_model semantic_model: enabled: true name: user_signup columns: - name: user_id entity: type: primary - name: country_id entity: type: unique - name: signup_date granularity: day dimension: type: time - name: country_dim dimension: type: categorical - name: country_model semantic_model: enabled: true name: country columns: - name: country_id entity: type: primary - name: country_name dimension: type: categorical ``` ##### Query multi-hop joins To query dimensions *without* a multi-hop join involved, you can use the fully qualified dimension name with the syntax entity double underscore (dunder) dimension, like `entity__dimension`. For dimensions retrieved by a multi-hop join, you need to additionally provide the entity path as a list, like `user_id`. --- ### Lake Compute FAQs Private beta ### Lake Compute FAQs [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Lake Compute is in private beta To request access, fill out the [Lake Compute signup form](https://docs.google.com/forms/d/e/1FAIpQLSdWXHU3VFqq1t8GVAA2BMfY_AfeVJH26OXJFlNwXiVv4pcuXQ/viewform). Lake Compute is in active development and likely to change. Breaking changes may occur, documentation may be incomplete, technical support may be limited, and no service-level agreement covers it during beta. Refer to [Private beta limitations](https://docs.getdbt.com/docs/lake-compute.md#private-beta-limitations) before you plan real work around it. These questions cover how Lake Compute fits into an existing dbt project during private beta. For an introduction to the product and its requirements, refer to [About Lake Compute](https://docs.getdbt.com/docs/lake-compute.md). #### Setup and requirements  Do I need to migrate my ingestion pipelines to Fivetran or MDLS? No. You don't need to migrate your existing ingestion pipelines or configure Fivetran connectors to load into an MDLS destination. During private beta, Lake Compute uses a Fivetran account and an MDLS destination to provision and authenticate access. Your existing ingestion pipelines stay in place.  Do I need to set an environment variable to use Lake Compute? Yes. dbt gates Lake Compute behind an experimental opt-in, so set `DBT_ENGINE_EXPERIMENTAL_MULTI_ADAPTER=true` before you run anything: ```bash export DBT_ENGINE_EXPERIMENTAL_MULTI_ADAPTER=true ``` One variable covers `type: lakecompute` in `profiles.yml`, the `adapter` config on a node, and the `--adapter` flag, so you need it whether you run Lake Compute as a "sidecar" or as your default adapter. While it's off, dbt fails at parse time with a message naming the variable rather than silently building on your default adapter. `DBT_ALLOW_EXPERIMENTAL_ADAPTERS` doesn't lift this gate. Refer to [Required environment variable](https://docs.getdbt.com/docs/lake-compute.md#required-environment-variable) for the full behavior. #### Projects, models, and data  Does Lake Compute require moving or splitting my dbt project? No. dbt v2 supports an experimental feature called multi-adapter invocations, which runs the same dbt project as one unified DAG across both your primary data warehouse and Lake Compute. You select the adapter, or compute engine, at the model level.  What table formats can Lake Compute read and write? Lake Compute reads from and writes to Iceberg tables. Any upstream source or table that a Lake Compute model references must therefore be available as an Iceberg table. Downstream models can reference Lake Compute models as normal, and you can materialize those downstream models however you like: as views, warehouse-native tables, Iceberg tables, and more.  Does Lake Compute automatically identify which models I should move? No. Review your warehouse workload and select the models to evaluate on Lake Compute. Start with a small number of models, test their outputs, and compare their behavior and performance against the existing warehouse implementation.  Does Lake Compute automatically translate SQL dialects? No. Lake Compute runs DuckDB SQL and doesn't automatically translate SQL written for another warehouse. Some straightforward models work with zero or minimal changes, while more complex models require more substantial updates and testing. When you migrate from one SQL dialect to another, pay attention to: * Warehouse-native functions, including proprietary AI or ML functions * Semi-structured types such as `VARIANT`, `OBJECT`, and `ARRAY` * Implicit casting behavior for column data types * Window functions, whose behavior varies by engine * Timestamp and other data-type representations when you move data between your warehouse and Iceberg  Which models should stay on my warehouse? Keep models on your warehouse when they need distributed or multi-node execution, large two-sided joins, warehouse-native AI or ML functions, proprietary SQL behavior, high concurrency, or other capabilities that Lake Compute doesn't support.  Where does my data live? For models that Lake Compute executes, your data lives in your own cloud storage bucket, in an open catalog that Fivetran MDLS manages. Lake Compute reads and writes those tables. It doesn't store a separate copy of the data. For models that your data platform executes and writes to Iceberg tables, your data lives in the platform's managed Iceberg catalog, either Snowflake Horizon or Databricks Unity. That catalog may use your own cloud storage bucket or platform-managed storage.  Can my warehouse still read tables that Lake Compute writes? Yes. Your warehouse accesses the same Iceberg tables by integrating with MDLS or your storage bucket. Models running on your warehouse and models living in Lake Compute can reference each other within one unified DAG, without maintaining two independent copies of the data.  What happens if a model doesn't work well on Lake Compute? Reconfigure the model to run on your primary data warehouse. All the data remains available in an Iceberg catalog, so switching the execution engine for one model doesn't require moving the data or splitting the dbt project. #### Pricing  Does Lake Compute cost anything during private beta? No. Lake Compute is free during private beta. #### Support  Is Lake Compute covered by a service-level agreement during private beta? No. No service-level agreement (SLA) covers Lake Compute during private beta. Use Lake Compute for evaluation and feedback only, and expect limited technical support. Don't treat beta workloads as production-critical unless your team has evaluated the risks and has a fallback plan. For the full list of what Lake Compute doesn't support during private beta, refer to [Private beta limitations](https://docs.getdbt.com/docs/lake-compute.md#private-beta-limitations).  What should I include in a support ticket? Include enough detail for the support team to reproduce and route the issue: * State that the issue involves **Lake Compute private beta**. * Your MDLS destination name and cloud region. Don't include secrets, private keys, or access tokens. * Whether the same model succeeds when you run it on your warehouse. * Minimal SQL or a reproducible example, with sensitive values removed.  Should I contact dbt Support or Fivetran Support? Start with the team that owns the component where the failure occurs: | Contact | Examples | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **dbt Support** | dbt project parsing or compilation; dbt v2 or Fusion behavior; model selection; DAG or `ref` behavior; job orchestration; dbt-side connection or model configuration; a model runs on the warehouse, but dbt doesn't route it to Lake Compute as expected. | | **Fivetran Support** | Fivetran account or MDLS setup; MDLS destination health; write credentials; the MDLS-managed catalog; storage or catalog provisioning performed through Fivetran; a Fivetran service error. | | **Either team, if ownership is unclear** | Lake Compute execution, authentication, networking, or a failure at the boundary between dbt, Lake Compute, MDLS, and the warehouse. Identify the ticket as a Lake Compute private beta issue and include both account IDs so support can route it. Don't open duplicate tickets unless support asks you to. | #### Related docs * [About Lake Compute](https://docs.getdbt.com/docs/lake-compute.md) * [Lake Compute setup](https://docs.getdbt.com/docs/lake-compute/compute-onboarding.md) --- ### Materializations #### Overview Materializations are strategies for persisting dbt models in a warehouse. There are five types of materializations built into dbt. They are: * table * view * incremental * ephemeral * materialized view You can also configure [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md?step=1) in dbt. Custom materializations are a powerful way to extend dbt's functionality to meet your specific needs. For a detailed guide on materializations, refer to [Materializations best practices](https://docs.getdbt.com/best-practices/materializations/1-guide-overview.md). For information about data streaming, refer to [How to handle real-time data](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/1-intro.md). Learn by video! For video tutorials on Materializations, go to dbt Learn and check out the [Materializations fundamentals course](https://learn.getdbt.com/courses/materializations-fundamentals). #### Configuring materializations By default, dbt models are materialized as "views". Models can be configured with a different materialization by supplying the [`materialized` configuration](https://docs.getdbt.com/reference/resource-configs/materialized.md) parameter as shown in the following tabs. ##### Project file dbt\_project.yml ```yaml # The following dbt_project.yml configures a project that looks like this: # . # └── models # ├── csvs # │   ├── employees.sql # │   └── goals.sql # └── events # ├── stg_event_log.sql # └── stg_event_sessions.sql name: my_project version: 1.0.0 config-version: 2 models: my_project: events: # materialize all models in models/events as tables +materialized: table csvs: # this is redundant, and does not need to be set +materialized: view ``` ##### Model file Alternatively, materializations can be configured directly inside of the model SQL files. This can be useful if you are also setting \[Performance Optimization] configs for specific models (for example, [Redshift specific configurations](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md) or [BigQuery specific configurations](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md)). models/events/stg\_event\_log.sql ```sql {{ config(materialized='table', sort='timestamp', dist='user_id') }} select * from ... ``` ##### Property file Materializations can also be configured in the model's `properties.yml` file. The following example shows the `table` materialization type. For a complete list of materialization types, refer to [materializations](https://docs.getdbt.com/docs/build/materializations.md#materializations). models/properties.yml ```yaml models: - name: events config: materialized: table ``` #### Materializations ##### View When using the `view` materialization, your model is rebuilt as a view on each run, via a `create view as` statement. * **Pros:** No additional data is stored, views on top of source data will always have the latest records in them. * **Cons:** Views that perform a significant transformation, or are stacked on top of other views, are slow to query. * **Advice:** * Generally start with views for your models, and only change to another materialization when you notice performance problems. * Views are best suited for models that do not do significant transformation, for example, renaming, or recasting columns. ##### Table When using the `table` materialization, your model is rebuilt as a table on each run, via a `create table as` statement. * **Pros:** Tables are fast to query * **Cons:** * Tables can take a long time to rebuild, especially for complex transformations * New records in underlying source data are not automatically added to the table * **Advice:** * Use the table materialization for any models being queried by BI tools, to give your end user a faster experience * Also use the table materialization for any slower transformations that are used by many downstream models ##### Incremental `incremental` models allow dbt to insert or update records into a table since the last time that model was run. * **Pros:** You can significantly reduce the build time by just transforming new records * **Cons:** Incremental models require extra configuration and are an advanced usage of dbt. Read more about using incremental models [here](https://docs.getdbt.com/docs/build/incremental-models.md). * **Advice:** * Incremental models are best for event-style data * Use incremental models when your `dbt run`s are becoming too slow (i.e. don't start with incremental models) ##### Ephemeral `ephemeral` models are not directly built into the database. Instead, dbt will interpolate the code from an ephemeral model into its dependent models using a common table expression (CTE). You can control the identifier for this CTE using a [model alias](https://docs.getdbt.com/docs/build/custom-aliases.md), but dbt will always prefix the model identifier with `__dbt__cte__`. * **Pros:** * You can still write reusable logic * Ephemeral models can help keep your data warehouse clean by reducing clutter (also consider splitting your models across multiple schemas by [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md)). * **Cons:** * You cannot select directly from this model. * [Operations](https://docs.getdbt.com/docs/build/hooks-operations.md#about-operations) (for example, macros called using [`dbt run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) cannot `ref()` ephemeral nodes) * Overuse of ephemeral materialization can also make queries harder to debug. * Ephemeral materialization doesn't support [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md#where-are-contracts-supported). * **Advice:** Use the ephemeral materialization for: * Very light-weight transformations that are early on in your DAG * Are only used in one or two downstream models, and * Don't need to be queried directly ##### Materialized View The `materialized_view` materialization allows the creation and maintenance of materialized views in the target database. Materialized views are a combination of a view and a table, and serve use cases similar to incremental models. * **Pros:** * Materialized views combine the query performance of a table with the data freshness of a view * Materialized views operate much like incremental materializations, however they are usually able to be refreshed without manual interference on a regular cadence (depending on the database), forgoing the regular dbt batch refresh required with incremental materializations * `dbt run` on materialized views corresponds to a code deployment, just like views - **Cons:** * Due to the fact that materialized views are more complex database objects, database platforms tend to have fewer configuration options available; see your database platform's docs for more details * Materialized views may not be supported by every database platform * **Advice:** * Consider materialized views for use cases where incremental models are sufficient, but you would like the data platform to manage the incremental logic and refresh. ###### Configuration Change Monitoring This materialization makes use of the [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) config, which aligns with the incremental nature of the namesake database object. This setting tells dbt to attempt to make configuration changes directly to the object when possible, as opposed to completely recreating the object to implement the updated configuration. Using `dbt-postgres` as an example, indexes can be dropped and created on the materialized view without the need to recreate the materialized view itself. ###### Scheduled Refreshes In the context of a `dbt run` command, materialized views should be thought of as similar to views. For example, a `dbt run` command is only needed if there is the potential for a change in configuration or sql; it's effectively a deploy action. By contrast, a `dbt run` command is needed for a table in the same scenarios *AND when the data in the table needs to be updated*. This also holds true for incremental and snapshot models, whose underlying relations are tables. In the table cases, the scheduling mechanism is either dbt or your local scheduler; there is no built-in functionality to automatically refresh the data behind a table. However, most platforms (Postgres excluded) provide functionality to configure automatically refreshing a materialized view. Hence, materialized views work similarly to incremental models with the benefit of not needing to run dbt to refresh the data. This assumes, of course, that auto refresh is turned on and configured in the model. info `dbt-snowflake` *does not* support materialized views, it uses Dynamic Tables instead. For details, refer to [Snowflake specific configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables). Beginning in v1.12, `dbt-snowflake` supports interactive tables, which are optimized for low-latency, interactive queries and can refresh automatically like dynamic tables. For details, refer to [Interactive tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#interactive-tables). #### Python materializations Python models support two materializations: * `table` * `incremental` Incremental Python models support all the same [incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) as their SQL counterparts. The specific strategies supported depend on your adapter. Python models can't be materialized as `view` or `ephemeral`. Python isn't supported for non-model resource types (like tests and snapshots). For incremental models, like SQL models, you will need to filter incoming tables to only new rows of data: ##### Snowpark models/my\_python\_model.py ```python import snowflake.snowpark.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.dateadd("day", F.lit(-3), F.current_timestamp())) ... return df ``` ##### PySpark models/my\_python\_model.py ```python import pyspark.sql.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.date_add(F.current_timestamp(), F.lit(-3))) ... return df ``` **Note:** Incremental models are supported on BigQuery/Dataproc for the `merge` incremental strategy. The `insert_overwrite` strategy is not yet supported. --- ### MCP environment variables reference Local development This page is a condensed reference for all environment variables used by the self-hosted dbt MCP server. For full detail on each variable (including examples), see [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md). #### Self-hosted CLI These variables are required to use dbt commands through MCP. | Variable | Required | Description | | ------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_PROJECT_DIR` | Required | Full path to your dbt project folder (the folder containing `dbt_project.yml`). | | `DBT_PATH` | Required | Full path to your dbt executable. Find it by running `which dbt` (macOS/Linux) or `where dbt` (Windows). | | `DBT_PROFILES_DIR` | Optional | Path to a custom `profiles.yml` directory. Defaults to `~/.dbt/`. | | `DBT_CLI_TIMEOUT` | Optional | Seconds before dbt commands time out. Defaults to `60`. Compile runs the whole project, so large projects may need a higher limit to avoid timeouts. | #### dbt platform These variables are required for dbt platform features (Semantic Layer, Discovery API, Admin API, SQL execution). | Variable | Required | Description | | ----------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_HOST` | Required | Your dbt platform [hostname](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Accepts both `cloud.getdbt.com` (default) and `https://cloud.getdbt.com`.<br /><br />Include the full hostname with subdomain — for example, `DBT_HOST=abc123.us1.dbt.com`. You no longer need to separate the prefix using `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. | | `DBT_TOKEN` | Required | A service token or Personal Access Token (PAT).<br /><br />The `execute_sql` tool requires a PAT — service tokens *do not* work for that tool. | | `DBT_PROD_ENV_ID` | Required | Your production environment ID (numeric integer). | | `DBT_DEV_ENV_ID` | Required for `execute_sql` | Your development environment ID (numeric integer). | | `DBT_USER_ID` | Required for `execute_sql` | Your numeric user ID. | | `DBT_ACCOUNT_ID` | Required for Admin API and PAT-based auth | Your numeric account ID. Required when using a Personal Access Token (PAT) as your `DBT_TOKEN`. | Legacy prefix variables `MULTICELL_ACCOUNT_PREFIX` and `DBT_HOST_PREFIX` are legacy environment variables. They are backwards compatible — if you already have them set, they will continue to work. However, the recommended approach is to set the full hostname directly in `DBT_HOST` (for example, `DBT_HOST=abc123.us1.dbt.com`). See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions on locating each value. #### Tool configuration dbt MCP has two modes for controlling which tools are available: disable mode (default) and enable mode (allowlist). Toolsets may grow over time as new capabilities are added. The naming pattern generalizes to `DISABLE_{TOOLSET}=` and `DBT_MCP_ENABLE_{TOOLSET}=`, so new toolsets follow the same convention. For the latest list of toolset names, refer to the tables below or the [dbt-mcp README](https://github.com/dbt-labs/dbt-mcp). Which mode should I use? * **Disable mode:** Use when you want most tools available and only need to turn a few off. * **Enable mode:** Use when you want only a specific set of tools available. Do not mix both modes for the same toolset. For example avoid this pattern: ```bash # ❌ Mixing modes leads to unpredictable behavior DISABLE_SEMANTIC_LAYER=false DBT_MCP_ENABLE_SEMANTIC_LAYER=true ``` Also avoid empty `DBT_MCP_ENABLE_*=` lines as they activate enable mode and will silently disable all other toolsets. ##### Disable mode (default) All tools are available by default. To disable a toolset, set any of the following variables to `true` in your MCP client configuration: | Variable | Default | Description | | ----------------------------- | ------- | ----------------------------------------------------------------------------------------------- | | `DISABLE_DBT_CLI` | `false` | Disable local dbt CLI tools (dbt v1.x and v2). | | `DISABLE_SEMANTIC_LAYER` | `false` | Disable Semantic Layer tools. | | `DISABLE_DISCOVERY` | `false` | Disable Discovery API tools. | | `DISABLE_ADMIN_API` | `false` | Disable Admin API tools. | | `DISABLE_SQL` | `true` | SQL tools are disabled by default. Set to `false` to enable. | | `DISABLE_DBT_CODEGEN` | `true` | Codegen tools are disabled by default. Set to `false` to enable (requires dbt-codegen package). | | `DISABLE_LSP` | `false` | Disable LSP/dbt v2 tools. | | `DISABLE_MCP_SERVER_METADATA` | `true` | Server metadata tools are disabled by default. Set to `false` to enable. | | `DISABLE_TOOLS` | `""` | Comma-separated list of specific tool names to disable. | ##### Enable mode (allowlist) If any `DBT_MCP_ENABLE_*` variable is set, only the explicitly enabled toolsets are active: | Variable | Description | | ------------------------------- | -------------------------------------------------------------- | | `DBT_MCP_ENABLE_DBT_CLI` | Set to `true` to enable local dbt CLI tools (dbt v1.x and v2). | | `DBT_MCP_ENABLE_SEMANTIC_LAYER` | Set to `true` to enable Semantic Layer tools. | | `DBT_MCP_ENABLE_DISCOVERY` | Set to `true` to enable Discovery API tools. | | `DBT_MCP_ENABLE_ADMIN_API` | Set to `true` to enable Admin API tools. | | `DBT_MCP_ENABLE_SQL` | Set to `true` to enable SQL tools. | | `DBT_MCP_ENABLE_DBT_CODEGEN` | Set to `true` to enable codegen tools. | | `DBT_MCP_ENABLE_LSP` | Set to `true` to enable LSP/dbt v2 tools. | | `DBT_MCP_ENABLE_TOOLS` | Comma-separated list of specific tool names to enable. | ##### Precedence When multiple variables are set, they're evaluated in this order (highest priority first): 1. `DBT_MCP_ENABLE_TOOLS` (enable specific tools by name) 2. `DISABLE_TOOLS` (disable specific tools by name) 3. Toolset enable (`DBT_MCP_ENABLE_*=true`) 4. Toolset disable (`DISABLE_*=true`) 5. Default behavior #### Semantic Layer These variables control the behavior of Semantic Layer tools. | Variable | Default | Description | | -------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `DBT_MCP_SL_METRICS_RELATED_MAX` | `10` | Maximum number of metrics for which `list_metrics` also returns dimension and entity names inline, reducing the number of tool calls needed to answer data questions. When the metric count is at or below this value, dimensions and entities are embedded directly in the `list_metrics` response. When above this value, only metric names are returned and the LLM calls `get_dimensions`/`get_entities` separately. Set to `0` to always return metrics only and never inline dimension or entity data. | | `DBT_MCP_SL_MAX_RESPONSE_CHARS` | `16000` | Maximum character length of the CSV returned by `list_metrics`. Must be an integer `>= 0`. When the response would exceed this length *and* the metric count is above `DBT_MCP_SL_METRICS_RELATED_MAX`, the `description` and `metadata` columns are dropped from the response to save tokens. Set to `0` to disable trimming and always return the full response. | #### Logging and debugging | Variable | Default | Description | | ------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_MCP_LOG_LEVEL` | `INFO` | Log level for the MCP server. Options: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. Set to `DEBUG` temporarily to diagnose issues. | | `MCP_TRANSPORT` | `stdio` | Transport protocol. Set to `streamable-http` for local debugging only (for example, attaching a Python debugger). See the [dbt-mcp CONTRIBUTING guide](https://github.com/dbt-labs/dbt-mcp/blob/main/CONTRIBUTING.md). | --- ### Measures Measures are deprecated in the new spec Heads up, measures have been deprecated in favor of simple metrics under the `metrics:` key. Migrate by converting each measure to a `type: simple` metric. For more info, check out [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) and [upgrade to v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). Measures are aggregations performed on columns in your model. They can be used as final metrics or as building blocks for more complex metrics. Measures have several inputs, which are described in the following table along with their field types. | Parameter | Description | Required | Type | | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | | [`name`](https://docs.getdbt.com/docs/build/measures.md#name) | Provide a name for the measure, which must be unique and can't be repeated across all semantic models in your dbt project. | Required | String | | [`description`](https://docs.getdbt.com/docs/build/measures.md#description) | Describes the calculated measure. | Optional | String | | [`agg`](https://docs.getdbt.com/docs/build/measures.md#aggregation) | dbt supports the following aggregations: `sum`, `max`, `min`, `average`, `median`, `count_distinct`, `percentile`, and `sum_boolean`. | Required | String | | [`expr`](https://docs.getdbt.com/docs/build/measures.md#expr) | Either reference an existing column in the table or use a SQL expression to create or derive a new one. | Optional | String | | [`non_additive_dimension`](https://docs.getdbt.com/docs/build/measures.md#non-additive-dimensions) | Non-additive dimensions can be specified for measures that cannot be aggregated over certain dimensions, such as bank account balances, to avoid producing incorrect results. | Optional | String | | `agg_params` | Specific aggregation properties, such as a percentile. | Optional | Dict | | `agg_time_dimension` | The time field. Defaults to the default agg time dimension for the semantic model. | Optional | String | | `label` | String that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). Available in dbt version 1.7 or higher. | Optional | String | | `create_metric` | Create a `simple` metric from a measure by setting `create_metric: True`. The `label` and `description` attributes will be automatically propagated to the created metric. Available in dbt version 1.7 or higher. | Optional | Boolean | | `config` | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your metric. Supports the [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) property, nested under `config`. | Optional | | #### Measure spec An example of the complete YAML measures spec is below. The actual configuration of your measures will depend on the aggregation you're using. ```yaml semantic_models: - name: semantic_model_name ..rest of the semantic model config measures: - name: The name of the measure description: 'same as always' ## Optional agg: the aggregation type. expr: the field agg_params: 'specific aggregation properties such as a percentile' ## Optional agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional config: Use the config property to specify configurations for your measure. ## Optional meta: {<dictionary>} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional ``` ##### Name When you create a measure, you can either give it a custom name or use the `name` of the data platform column directly. If the measure's `name` differs from the column name, you need to add an `expr` to specify the column name. The `name` of the measure is used when creating a metric. Measure names must be unique across all semantic models in a project and can not be the same as an existing `entity` or `dimension` within that same model. ##### Description The description describes the calculated measure. It's strongly recommended you create verbose and human-readable descriptions in this field. ##### Aggregation The aggregation determines how the field will be aggregated. For example, a `sum` aggregation type over a granularity of `day` would sum the values across a given day. Supported aggregations include: | Aggregation types | Description | | ----------------- | ------------------------------------------ | | sum | Sum across the values | | min | Minimum across the values | | max | Maximum across the values | | average | Average across the values | | sum\_boolean | A sum for a boolean type | | count\_distinct | Distinct count of values | | median | Median (p50) calculation across the values | | percentile | Percentile calculation across the values. | ###### Percentile aggregation example If you're using the `percentile` aggregation, you must use the `agg_params` field to specify details for the percentile aggregation (such as what percentile to calculate and whether to use discrete or continuous calculations). ```yaml name: p99_transaction_value description: The 99th percentile transaction value expr: transaction_amount_usd agg: percentile agg_params: percentile: .99 use_discrete_percentile: False # False calculates the continuous percentile, True calculates the discrete percentile. ``` ###### Percentile across supported engine types The following table lists which SQL engine supports continuous, discrete, approximate, continuous, and approximate discrete percentiles. | | Cont. | Disc. | Approx. cont | Approx. disc | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Snowflake | [Yes](https://docs.snowflake.com/en/sql-reference/functions/percentile_cont.html) | [Yes](https://docs.snowflake.com/en/sql-reference/functions/percentile_disc.html) | [Yes](https://docs.snowflake.com/en/sql-reference/functions/approx_percentile.html) (t-digest) | No | | Bigquery | No (window) | No (window) | [Yes](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#approx_quantiles) | No | | Databricks | [Yes](https://docs.databricks.com/sql/language-manual/functions/percentile_cont.html) | [No](https://docs.databricks.com/sql/language-manual/functions/percentile_disc.html) | No | [Yes](https://docs.databricks.com/sql/language-manual/functions/approx_percentile.html) | | Redshift | [Yes](https://docs.aws.amazon.com/redshift/latest/dg/r_PERCENTILE_CONT.html) | No (window) | No | [Yes](https://docs.aws.amazon.com/redshift/latest/dg/r_APPROXIMATE_PERCENTILE_DISC.html) | | [Postgres](https://www.postgresql.org/docs/9.4/functions-aggregate.html) | Yes | Yes | No | No | | [DuckDB](https://duckdb.org/docs/sql/aggregates.html) | Yes | Yes | Yes (t-digest) | No | ##### Expr If the `name` you specified for a measure doesn't match a column name in your model, you can use the `expr` parameter instead. This allows you to use any valid SQL to manipulate an underlying column name into a specific output. The `name` parameter then serves as an alias for your measure. **Notes**: When using SQL functions in the `expr` parameter, **always use data platform-specific SQL**. This is because outputs may differ depending on your specific data platform. For Snowflake users For Snowflake users, if you use a week-level function in the `expr` parameter, it'll now return Monday as the default week start day based on ISO standards. If you have any account or session level overrides for the `WEEK_START` parameter that fixes it to a value other than 0 or 1, you will still see Monday as the week starts. If you use the `dayofweek` function in the `expr` parameter with the legacy Snowflake default of `WEEK_START = 0`, it will now return ISO-standard values of 1 (Monday) through 7 (Sunday) instead of Snowflake's legacy default values of 0 (Monday) through 6 (Sunday). ##### Model with different aggregations ```yaml semantic_models: - name: transactions description: A record of every transaction that takes place. Carts are considered multiple transactions for each sku. model: ref('schema.transactions') defaults: agg_time_dimension: transaction_date # --- entities --- entities: - name: transaction_id type: primary - name: customer_id type: foreign - name: store_id type: foreign - name: product_id type: foreign # --- measures --- measures: - name: transaction_amount_usd description: Total usd value of transactions expr: transaction_amount_usd agg: sum config: meta: used_in_reporting: true - name: transaction_amount_usd_avg description: Average usd value of transactions expr: transaction_amount_usd agg: average - name: transaction_amount_usd_max description: Maximum usd value of transactions expr: transaction_amount_usd agg: max - name: transaction_amount_usd_min description: Minimum usd value of transactions expr: transaction_amount_usd agg: min - name: quick_buy_transactions description: The total transactions bought as quick buy expr: quick_buy_flag agg: sum_boolean - name: distinct_transactions_count description: Distinct count of transactions expr: transaction_id agg: count_distinct - name: transaction_amount_avg description: The average value of transactions expr: transaction_amount_usd agg: average - name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition description: The total usd value of valid transactions only expr: case when is_valid = True then transaction_amount_usd else 0 end agg: sum - name: transactions description: The average value of transactions. expr: transaction_amount_usd agg: average - name: p99_transaction_value description: The 99th percentile transaction value expr: transaction_amount_usd agg: percentile agg_params: percentile: .99 use_discrete_percentile: False # False calculates the continuous percentile, True calculates the discrete percentile. - name: median_transaction_value description: The median transaction value expr: transaction_amount_usd agg: median # --- dimensions --- dimensions: - name: transaction_date type: time expr: date_trunc('day', ts) # expr refers to underlying column ts type_params: time_granularity: day - name: is_bulk_transaction type: categorical expr: case when quantity > 10 then true else false end ``` ##### Non-additive dimensions Some measures cannot be aggregated over certain dimensions, like time, because it could result in incorrect outcomes. Examples include bank account balances where it does not make sense to carry over balances month-to-month, and monthly recurring revenue where daily recurring revenue cannot be summed up to achieve monthly recurring revenue. You can specify non-additive dimensions to handle this, where certain dimensions are excluded from aggregation. To demonstrate the configuration for non-additive measures, consider a subscription table that includes one row per date of the registered user, the user's active subscription plan(s), and the plan's subscription value (revenue) with the following columns: * `date_transaction`: The daily date-spine. * `user_id`: The ID of the registered user. * `subscription_plan`: A column to indicate the subscription plan ID. * `subscription_value`: A column to indicate the monthly subscription value (revenue) of a particular subscription plan ID. Parameters under the `non_additive_dimension` will specify dimensions that the measure should not be aggregated over. | Parameter | Description | Field type | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `name` | This will be the name of the time dimension (that has already been defined in the data source) that the measure should not be aggregated over. | Required | | `window_choice` | Choose either `min` or `max`, where `min` reflects the beginning of the time period and `max` reflects the end of the time period. | Required | | `window_groupings` | Provide the entities that you would like to group by. | Optional | ```yaml semantic_models: - name: subscriptions description: A subscription table with one row per date for each active user and their subscription plans. model: ref('your_schema.subscription_table') defaults: agg_time_dimension: subscription_date entities: - name: user_id type: foreign primary_entity: subscription dimensions: - name: subscription_date type: time expr: date_transaction type_params: time_granularity: day measures: - name: count_users description: Count of users at the end of the month expr: user_id agg: count_distinct non_additive_dimension: name: subscription_date window_choice: max - name: mrr description: Aggregate by summing all users' active subscription plans expr: subscription_value agg: sum non_additive_dimension: name: subscription_date window_choice: max - name: user_mrr description: Group by user_id to achieve each user's MRR expr: subscription_value agg: sum non_additive_dimension: name: subscription_date window_choice: max window_groupings: - user_id metrics: - name: mrr_metrics type: simple type_params: measure: mrr ``` We can query the semi-additive metrics using the following syntax: For dbt: ```bash dbt sl query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__month --order subscription__subscription_date__month dbt sl query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__week --order subscription__subscription_date__week ``` For dbt v1: ```bash mf query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__month --order subscription__subscription_date__month mf query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__week --order subscription__subscription_date__week ``` #### Dependencies (Applies to dbt v1.12 and later) Metric nodes will reflect dependencies on semantic models based on their simple metrics. However, dependencies based on filters should not be reflected in: * [dbt selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) * Visualization of the DAG in dbt-docs and the [integrated development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (IDE). This is because metrics need to source nodes for their `depends_on` attribute from a few different places: * `RATIO` metrics depend on `numerator` and `denominator` metrics, while `DERIVED` metrics depend on `input_metrics`. * `SIMPLE` type metrics depend on their semantic\_model. For example, when you run the command `dbt list --select my_semantic_model+`, it will show you the metrics that belong to the specified semantic model. But there's a condition: Only the metrics that actually use simple metrics or derived metrics from that semantic model will be included in the list. In other words, if a metric only uses a dimension from the semantic model in its filters, it won't be considered as part of that semantic model. --- ### Merge jobs in dbt dbt platform | Starter, Enterprise You can set up a merge job to implement a continuous deployment (CD) workflow in dbt. The merge job triggers a dbt job to run when someone merges Git pull requests into production. This workflow creates a seamless development experience where changes made in code will automatically update production data. You can also use a merge job to refresh an environment’s `manifest.json` so downstream [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) stay fast. For example, run [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) or, to avoid warehouse work entirely, [`dbt parse --no-partial-parse`](https://docs.getdbt.com/reference/commands/parse.md) in a dedicated environment—the same one your CI jobs defer to under **Compare changes against**, so the manifest those pull request runs compare against updates quickly. Use the same base branch that environment tracks (for example, `main` or a [custom branch](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) such as `develop`). If merge also kicks off a long `dbt build`, consider a separate merge triggered job that runs *only* `dbt parse --no-partial-parse` so the manifest updates before the build finishes. By using CD in dbt, you can take advantage of deferral to build only the edited model and any downstream changes. With merge jobs, state will be updated almost instantly, always giving the most up-to-date state information in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). Triggering merge jobs in monorepos If you have a monorepo with several dbt projects, merging a single pull request in one of your projects will trigger jobs for all projects connected to the monorepo. To address this, you can use separate target branches per project (for example, `main-project-a`, `main-project-b`) to separate CI triggers. #### Prerequisites * You have a dbt account. * You have set up a [connection with your Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md). This integration lets dbt run jobs on your behalf for job triggering. * If you're using a native [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) integration, you need a paid or self-hosted account that includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). If you're using GitLab Free, merge requests will trigger CI jobs but CI job status updates (success or failure of the job) will not be reported back to GitLab. * For deferral (which is the default), make sure there has been at least one successful job run in the environment you defer to. #### Set up job trigger on Git merge 1. On your deployment environment page, click **Create job** > **Merge job**. 2. Options in the **Job settings** section: * **Job name** — Specify the name for the merge job. * **Description** — Provide a description about the job. * **Environment** — By default, it’s set to the environment you created the job from. 3. In the **Git trigger** section, the **Run on merge** option is enabled by default. Every time a PR merges (to a base branch configured in the environment) in your Git repo, this job will get triggered to run. 4. Options in the **Execution settings** section: * **Commands** — By default, it includes the `dbt build --select state:modified+` command. This informs dbt to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when this job runs. * **Compare changes against** — By default, it's set to compare changes against the environment you created the job from. This option allows dbt to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG. To change the default settings, you can select **No deferral**, **This job** for self-deferral, or choose a different environment. * **dbt State** — [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) reuses nodes when their logic and data haven’t changed, avoiding unnecessary rebuilds. Select **On**, **Off**, or **Inherited from environment** to use the environment’s setting. This option appears only when dbt State is [enabled](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) on your account. Learn more about [enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md#enabling-dbt-state-on-individual-jobs). 5. (optional) Options in the **Advanced settings** section: * **Environment variables** — Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when this job runs. * **Target name** — Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md). Similar to environment variables, this option lets you customize the behavior of the project. * **Run timeout** — Cancel this job if the run time exceeds the timeout value. * **dbt version** — By default, it’s set to inherit the [dbt version](https://docs.getdbt.com/docs/dbt-versions.md) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior. * **Threads** — By default, it’s set to 4 [threads](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads). Increase the thread count to increase model execution concurrency. [![Example of creating a merge job](/img/docs/dbt-platform/using-dbt-platform/example-create-merge-job.png?v=2 "Example of creating a merge job")](#)Example of creating a merge job #### Verify push events in Git Merge jobs require push events so make sure they've been enabled in your Git provider, especially if you have an already-existing Git integration. However, for a new integration setup, you can skip this check since push events are typically enabled by default.  GitHub example The following is a GitHub example of when the push events are already set: [![Example of the Pushes option enabled in the GitHub settings](/img/docs/dbt-platform/using-dbt-platform/example-github-push-events.png?v=2 "Example of the Pushes option enabled in the GitHub settings")](#)Example of the Pushes option enabled in the GitHub settings  GitLab example The following is a GitLab example of when the push events are already set: [![Example of the Push events option enabled in the GitLab settings](/img/docs/dbt-platform/using-dbt-platform/example-gitlab-push-events.png?v=2 "Example of the Push events option enabled in the GitLab settings")](#)Example of the Push events option enabled in the GitLab settings  Azure DevOps example The following is an example of creating a new **Code pushed** trigger in Azure DevOps. Create a new service hooks subscription when code pushed events haven't been set: [![Example of creating a new trigger to push events in Azure Devops](/img/docs/dbt-platform/using-dbt-platform/example-azuredevops-new-event.png?v=2 "Example of creating a new trigger to push events in Azure Devops")](#)Example of creating a new trigger to push events in Azure Devops --- ### MetricFlow commands MetricFlow allows you to define and query metrics in your dbt project in [dbt platform](https://docs.getdbt.com/docs/platform/about-develop-dbt.md) or [self-hosted dbt](https://docs.getdbt.com/docs/local/install-dbt.md) locally. To experience the power of the universal [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and dynamically query those metrics in downstream tools, you'll need a dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing/) account. Once you define metrics in your dbt project, you can query metrics, dimensions, and dimension values, and validate your configs using the MetricFlow commands, available across [dbt versions](https://docs.getdbt.com/docs/introduction.md). MetricFlow is compatible with Python versions 3.8, 3.9, 3.10, 3.11, and 3.12. #### MetricFlow (Applies to dbt v1.12 and later) MetricFlow is a Python library that allows you to define and query metrics in your dbt project. You can use MetricFlow to query metrics in your dbt project two ways: in the dbt platform or locally with self-hosted dbt, where you install and manage the engine yourself. ##### MetricFlow in dbt platform This section applies to dbt platform users running either the v1 or v2 engine, where commands and validations execute remotely in dbt platform. * Run MetricFlow commands using the `dbt sl` prefix in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or using the [VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md). * For CLI or VS Code/Cursor users, MetricFlow commands are embedded, which means you can immediately run them once you install the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or [VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md) and don't need to install MetricFlow separately. * Using MetricFlow with dbt platform doesn't require you to manage versioning — your dbt account will automatically manage the versioning. * dbt jobs support the `dbt sl validate` command to [automatically test your semantic nodes](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci). You can also add MetricFlow validations with your Git provider (such as GitHub Actions) by installing MetricFlow (`python -m pip install metricflow`). This allows you to run MetricFlow commands as part of your continuous integration checks on PRs. ##### MetricFlow with self-hosted dbt This section applies to local self-hosted dbt users who aren't on dbt platform. You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to install MetricFlow on Windows or Linux operating systems: 1. Create or activate your virtual environment `python -m venv venv`. 2. Run `pip install dbt-metricflow`. * You can install MetricFlow using PyPI as an extension of your dbt adapter in the command line. To install the adapter, run `python -m pip install "dbt-metricflow[adapter_package_name]"` and add the adapter name at the end of the command. For example, for a Snowflake adapter, run `python -m pip install "dbt-metricflow[dbt-snowflake]"`. **Note**, you'll need to manage versioning yourself: between dbt, your adapter, and MetricFlow if you execute with v1, or between dbt and MetricFlow if you execute with v2. Something to note, MetricFlow `mf` commands return an error if you have a Metafont latex package installed. To run `mf` commands, uninstall the package. #### MetricFlow commands Use MetricFlow commands to retrieve metadata and query metrics. The following table lists the compatibility matrix for MetricFlow commands and where you can run them. | Development setup | Engine | Hosted on | Prefix | Notes | | ------------------------------------------------ | ------------ | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Studio IDE/dbt platform CLI or VS Code extension | dbt v2 or v1 | dbt platform | `dbt sl` | Remote execution; Platform manages versions.<br />VS Code extension users must have a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file with a valid token to connect to dbt platform | | Self-hosted<br />(no dbt platform project) | dbt v2 or v1 | Local machine | `mf` | Install and manage MetricFlow locally. | * If you’re using v2 with dbt platform and have a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file with a valid token to connect to dbt platform, run MetricFlow commands using the `dbt sl` prefix. * This allows you to interact with metrics that are executed remotely on dbt platform (for example, from the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md)). * If you’re using [v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) and aren't connected to dbt platform, install MetricFlow separately and use the `mf` prefix to run commands locally. * If you’re using v1 locally without v2, run MetricFlow commands using the `mf` prefix. ##### Commands for dbt platform This section applies to dbt platform users running the v2 or v1 engine where commands and validations execute remotely in dbt platform. * Use the `dbt sl` prefix before the command name to execute them in the dbt platform (Studio IDE, VS Code/Cursor, dbt platform CLI) (like `dbt sl list metrics` to list all metrics). * For dbt platform users developing with a CLI or an editor (like VS Code), run the `dbt sl --help` command in the terminal to view a complete list of the MetricFlow commands and flags. * The following table lists the commands compatible with dbt platform (Studio IDE, VS Code/Cursor, dbt platform CLI) powered by the v2 or v1 engine: | Command | Description | Studio IDE | dbt platform CLI | VS Code/Cursor | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------------- | -------------- | | [`list metrics`](#list-metrics) | Lists metrics with dimensions. | ✅ | ✅ | ✅ | | [`list dimensions`](#list) | Lists unique dimensions for metrics. | ✅ | ✅ | ✅ | | [`list dimension-values`](#list-dimension-values) | List dimensions with metrics. | ✅ | ✅ | ✅ | | [`list entities`](#list-entities) | Lists all unique entities. | ✅ | ✅ | ✅ | | [`list saved-queries`](#list-saved-queries) | Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query or `--show-parameters` to show the full query parameters each saved query uses. | ✅ | ✅ | ✅ | | [`query`](#query) | Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to query metrics and dimensions (such as querying metrics, using the `where` filter, adding an `order`, and more). | ✅ | ✅ | ✅ | | [`validate`](#validate) | Validates semantic model configurations. | ✅ | ✅ | ✅ | | [`export`](#export) | Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. | ❌ | ✅ | ✅ | | [`export-all`](#export-all) | Runs exports for multiple saved queries at once, saving time and effort. | ❌ | ✅ | ✅ | Run dbt parse to reflect metric changes When you make changes to metrics, make sure to run `dbt parse` at a minimum to update the Semantic Layer. This updates the `semantic_manifest.json` file(Applies to dbt v1.12 and later) and `osi_document.json`, reflecting your changes when querying metrics. By running `dbt parse`, you won't need to rebuild all the models.  How can I query or preview metrics with the dbt platform CLI? Check out the following video for a short video demo of how to query or preview metrics with the dbt platform CLI: ##### Commands for local self-hosted dbt This section applies to local self-hosted users who aren't on dbt platform, whether you execute with v1 or v2. Commands and validations execute locally and use the `mf` prefix before the command name to execute them. For example, to list all metrics, run `mf list metrics`. * [`list metrics`](#list-metrics) — Lists metrics with dimensions. * [`list dimensions`](#list) — Lists unique dimensions for metrics. * [`list dimension-values`](#list-dimension-values) — List dimensions with metrics. * [`list entities`](#list-entities) — Lists all unique entities. * [`validate-configs`](#validate-configs) — Validates semantic model configurations. * [`health-checks`](#health-checks) — Performs data platform health check. * [`tutorial`](#tutorial) — Dedicated MetricFlow tutorial to help get you started. * [`query`](#query) — Query metrics and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started. #### List metrics This command lists the metrics with their available dimensions: ```bash dbt sl list metrics <metric_name> # For dbt platform users (dbt v1 or v2) mf list metrics <metric_name> # For self-hosted users (dbt v1 or v2) Options: --search TEXT Filter available metrics by this search term --show-all-dimensions Show all dimensions associated with a metric. --help Show this message and exit. ``` #### List dimensions This command lists all unique dimensions for a metric or multiple metrics. It displays only common dimensions when querying multiple metrics: ```bash dbt sl list dimensions --metrics <metric_name> # For dbt platform users (dbt v1 or v2) mf list dimensions --metrics <metric_name> # For self-hosted users (dbt v1 or v2) Options: --metrics SEQUENCE List dimensions by given metrics (intersection). Ex. --metrics bookings,messages --help Show this message and exit. ``` #### List dimension-values This command lists all dimension values with the corresponding metric: ```bash dbt sl list dimension-values --metrics <metric_name> --dimension <dimension_name> # For dbt platform users (dbt v1 or v2) mf list dimension-values --metrics <metric_name> --dimension <dimension_name> # For self-hosted users (dbt v1 or v2) Options: --dimension TEXT Dimension to query values from [required] --metrics SEQUENCE Metrics that are associated with the dimension [required] --end-time TEXT Optional iso8601 timestamp to constraint the end time of the data (inclusive) *Not available in the dbt platform/dbt v2 yet --start-time TEXT Optional iso8601 timestamp to constraint the start time of the data (inclusive) *Not available in the dbt platform/dbt v2 yet --help Show this message and exit. ``` #### List entities This command lists all unique entities: ```bash dbt sl list entities --metrics <metric_name> # For dbt platform users (dbt v1 or v2) mf list entities --metrics <metric_name> # For self-hosted users (dbt v1 or v2) Options: --metrics SEQUENCE List entities by given metrics (intersection). Ex. --metrics bookings,messages --help Show this message and exit. ``` #### List saved queries This command lists all available saved queries: ```bash dbt sl list saved-queries # For dbt platform users (dbt v1 or v2) ``` You can also add the `--show-exports` flag (or option) to show each export listed under a saved query: ```bash dbt sl list saved-queries --show-exports # For dbt platform users (dbt v1 or v2) ``` **Output** ```bash dbt sl list saved-queries --show-exports The list of available saved queries: - new_customer_orders exports: - Export(new_customer_orders_table, exportAs=TABLE) - Export(new_customer_orders_view, exportAs=VIEW) - Export(new_customer_orders, alias=orders, schemas=customer_schema, exportAs=TABLE) ``` #### Validate The following command performs validations against the defined semantic model configurations. * For dbt v2 and dbt platform CLI users with a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md): ```bash dbt sl validate ``` * For self-hosted users (dbt v1 or v2): ```bash mf validate-configs ``` ```bash Options: --timeout # dbt platform only Optional timeout for data warehouse validation in dbt platform. --dw-timeout INTEGER # dbt v1 only Optional timeout for data warehouse validation steps. Default None. --skip-dw # dbt v1 only Skips the data warehouse validations. --show-all # dbt v1 only Prints warnings and future errors. --verbose-issues # dbt v1 only Prints extra details about issues. --semantic-validation-workers INTEGER # dbt v1 only Uses specified number of workers for large configs. --help Show this message and exit. ``` #### Health checks The following command performs a health check against the data platform you provided in the configs. Note, in dbt, the `health-checks` command isn't required since it uses dbt's credentials to perform the health check. ```bash mf health-checks # For self-hosted users (dbt v1 or v2) ``` #### Tutorial Follow the dedicated MetricFlow tutorial to help you get started: ```bash mf tutorial # For self-hosted users (dbt v1 or v2) ``` #### Query Create a new query with MetricFlow and execute it against your data platform. The query returns the following result: ```bash dbt sl query --metrics <metric_name> --group-by <dimension_name> # For dbt platform users (dbt v1 or v2) dbt sl query --saved-query <name> # For dbt platform users (dbt v1 or v2) mf query --metrics <metric_name> --group-by <dimension_name> # For self-hosted users (dbt v1 or v2) Options: --metrics SEQUENCE Syntax to query single metrics: --metrics metric_name For example, --metrics bookings To query multiple metrics, use --metrics followed by the metric names, separated by commas without spaces. For example, --metrics bookings,messages --group-by SEQUENCE Syntax to group by single dimension/entity: --group-by dimension_name For example, --group-by ds For multiple dimensions/entities, use --group-by followed by the dimension/entity names, separated by commas without spaces. For example, --group-by ds,org --end-time TEXT Optional iso8601 timestamp to constraint the end time of the data (inclusive). *Not available in the dbt platform/dbt v2 yet --start-time TEXT Optional iso8601 timestamp to constraint the start time of the data (inclusive) *Not available in the dbt platform/dbt v2 yet --where TEXT SQL-like where statement provided as a string and wrapped in quotes. All filter items must explicitly reference fields or dimensions that are part of your model. To query a single statement: ---where "{{ Dimension('order_id__revenue') }} > 100" To query multiple statements: --where "{{ Dimension('order_id__revenue') }} > 100" --where "{{ Dimension('user_count') }} < 1000" # make sure to wrap each statement in quotes To add a dimension filter, use the `Dimension()` template wrapper to indicate that the filter item is part of your model. Refer to the FAQ for more info on how to do this using a template wrapper. --limit TEXT Limit the number of rows out using an int or leave blank for no limit. For example: --limit 100 --order-by SEQUENCE Specify metrics, dimension, or group bys to order by. Add the `-` prefix to sort query in descending (DESC) order. Leave blank for ascending (ASC) order. For example, to sort metric_time in DESC order: --order-by -metric_time To sort metric_time in ASC order and revenue in DESC order: --order-by metric_time,-revenue --csv FILENAME Provide filepath for data frame output to csv --compile (dbt platform/dbt v2) In the query output, show the query that was --explain (dbt v1) executed against the data warehouse --show-dataflow-plan Display dataflow plan in explain output --display-plans Display plans (such as metric dataflow) in the browser --decimals INTEGER Choose the number of decimal places to round for the numerical values --show-sql-descriptions Shows inline descriptions of nodes in displayed SQL --help Show this message and exit. ``` #### Query examples This section shares various types of query examples that you can use to query metrics and dimensions. The query examples listed are: * [Query metrics](#query-metrics) * [Query dimensions](#query-dimensions) * [Add `order`/`limit` function](#add-orderlimit) * [Add `where` clause](#add-where-clause) * [Filter by time](#filter-by-time) * [Query saved queries](#query-saved-queries) ##### Query metrics Use the example to query multiple metrics by dimension and return the `order_total` and `users_active` metrics by `metric_time.` **Query** ```bash dbt sl query --metrics order_total,users_active --group-by metric_time # For dbt platform users (dbt v1 or v2) mf query --metrics order_total,users_active --group-by metric_time # For self-hosted users (dbt v1 or v2) ``` **Result** ```bash ✔ Success 🦄 - query completed after 1.24 seconds | METRIC_TIME | ORDER_TOTAL | |:--------------|---------------:| | 2017-06-16 | 792.17 | | 2017-06-17 | 458.35 | | 2017-06-18 | 490.69 | | 2017-06-19 | 749.09 | | 2017-06-20 | 712.51 | | 2017-06-21 | 541.65 | ``` ##### Query dimensions You can include multiple dimensions in a query. For example, you can group by the `is_food_order` dimension to confirm if orders were for food or not. Note that when you query a dimension, you need to specify the primary entity for that dimension. In the following example, the primary entity is `order_id`. **Query** ```bash dbt sl query --metrics order_total --group-by order_id__is_food_order # For dbt platform users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order # For self-hosted users (dbt v1 or v2) ``` **Result** ```bash Success 🦄 - query completed after 1.70 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-06-16 | True | 499.27 | | 2017-06-16 | False | 292.90 | | 2017-06-17 | True | 431.24 | | 2017-06-17 | False | 27.11 | | 2017-06-18 | True | 466.45 | | 2017-06-18 | False | 24.24 | | 2017-06-19 | False | 300.98 | | 2017-06-19 | True | 448.11 | ``` ##### Add order/limit You can add order and limit functions to filter and present the data in a readable format. The following query limits the data set to 10 records and orders them by `metric_time`, descending. Note that using the `-` prefix will sort the query in descending order. Without the `-` prefix sorts the query in ascending order. Note that when you query a dimension, you need to specify the primary entity for that dimension. In the following example, the primary entity is `order_id`. **Query** ```bash # For dbt platform users (dbt v1 or v2) dbt sl query --metrics order_total --group-by order_id__is_food_order --limit 10 --order-by -metric_time # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order --limit 10 --order-by -metric_time ``` **Result** ```bash ✔ Success 🦄 - query completed after 1.41 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-08-31 | True | 459.90 | | 2017-08-31 | False | 327.08 | | 2017-08-30 | False | 348.90 | | 2017-08-30 | True | 448.18 | | 2017-08-29 | True | 479.94 | | 2017-08-29 | False | 333.65 | | 2017-08-28 | False | 334.73 | ``` ##### Add where clause You can further filter the data set by adding a `where` clause to your query. The following example shows you how to query the `order_total` metric, grouped by `is_food_order` with multiple `where` statements (orders that are food orders and orders from the week starting on or after Feb 1st, 2024). **Query** ```bash # For dbt platform users (dbt v1 or v2) dbt sl query --metrics order_total --group-by order_id__is_food_order --where "{{ Dimension('order_id__is_food_order') }} = True" --where "{{ TimeDimension('metric_time', 'week') }} >= '2024-02-01'" # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order --where "{{ Dimension('order_id__is_food_order') }} = True" --where "{{ TimeDimension('metric_time', 'week') }} >= '2024-02-01'" ``` Notes: * The type of dimension changes the syntax you use. So if you have a date field, use `TimeDimension` instead of `Dimension`. * When you query a dimension, you need to specify the primary entity for that dimension. In the example just shared, the primary entity is `order_id`. **Result** ```bash ✔ Success 🦄 - query completed after 1.06 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-08-31 | True | 459.90 | | 2017-08-30 | True | 448.18 | | 2017-08-29 | True | 479.94 | | 2017-08-28 | True | 513.48 | | 2017-08-27 | True | 568.92 | | 2017-08-26 | True | 471.95 | | 2017-08-25 | True | 452.93 | | 2017-08-24 | True | 384.40 | | 2017-08-23 | True | 423.61 | | 2017-08-22 | True | 401.91 | ``` ##### Filter by time To filter by time, there are dedicated start and end time options. Using these options to filter by time allows MetricFlow to further optimize query performance by pushing down the where filter when appropriate. Note that when you query a dimension, you need to specify the primary entity for that dimension. In the following example, the primary entity is `order_id`. **Query** ```bash # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' ``` **Result** ```bash ✔ Success 🦄 - query completed after 1.53 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-08-27 | True | 568.92 | | 2017-08-26 | True | 471.95 | | 2017-08-25 | True | 452.93 | | 2017-08-24 | True | 384.40 | | 2017-08-23 | True | 423.61 | | 2017-08-22 | True | 401.91 | ``` ##### Query saved queries You can use this for frequently used queries. Replace `<name>` with the name of your [saved query](https://docs.getdbt.com/docs/build/saved-queries.md). **Query** ```bash dbt sl query --saved-query <name> # For dbt platform users (dbt v1 or v2) mf query --saved-query <name> # For self-hosted users (dbt v1 or v2) ``` For example, if you use dbt and have a saved query named `new_customer_orders`, you would run `dbt sl query --saved-query new_customer_orders`. A note on querying saved queries When querying [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), you can use parameters such as `where`, `limit`, `order`, `compile`, and so on. However, keep in mind that you can't access `metric` or `group_by` parameters in this context. This is because they are predetermined and fixed parameters for saved queries, and you can't change them at query time. If you would like to query more metrics or dimensions, you can build the query using the standard format. #### Additional query examples The following tabs present additional query examples, like exporting to a CSV. Select the tab that best suits your needs: ##### --compile/--explain flag Add `--compile` (or `--explain` for v1 users) to your query to view the SQL generated by MetricFlow. **Query** ```bash # For dbt platform users (dbt v1 or v2) dbt sl query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --compile # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --explain ``` **Result** ```bash ✔ Success 🦄 - query completed after 0.28 seconds 🔎 SQL (remove --compile to see data or add --show-dataflow-plan to see the generated dataflow plan): select metric_time , is_food_order , sum(order_cost) as order_total from ( select cast(ordered_at as date) as metric_time , is_food_order , order_cost from analytics.js_dbt_sl_demo.orders orders_src_1 where cast(ordered_at as date) between cast('2017-08-22' as timestamp) and cast('2017-08-27' as timestamp) ) subq_3 where is_food_order = True group by metric_time , is_food_order order by metric_time desc limit 10 ``` ##### Export to CSV Add the `--csv file_name.csv` flag to export the results of your query to a CSV. The `--csv` flag is available to local self-hosted users only (with the `mf` prefix) and isn't supported by `dbt sl` in dbt platform. **Query** ```bash # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --csv query_example.csv ``` **Result** ```bash ✔ Success 🦄 - query completed after 0.83 seconds 🖨 Successfully written query output to query_example.csv ``` #### Time granularity Optionally, you can specify the time granularity you want your data to be aggregated at by appending two underscores and the unit of granularity you want to `metric_time`, the global time dimension. You can group the granularity by: `day`, `week`, `month`, `quarter`, and `year`. Below is an example for querying metric data at a monthly grain: ```bash dbt sl query --metrics revenue --group-by metric_time__month # For dbt platform users (dbt v1 or v2) mf query --metrics revenue --group-by metric_time__month # For self-hosted users (dbt v1 or v2) ``` #### Export Run [exports for a specific saved query](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-for-single-saved-query). Use this command to test and generate exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. Refer to [exports in development](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-in-development) for more info. Export is available in dbt. ```bash dbt sl export # For dbt platform users (dbt v1 or v2) ``` #### Export-all Run [exports for multiple saved queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-for-multiple-saved-queries) at once. This command provides a convenient way to manage and execute exports for several queries simultaneously, saving time and effort. Refer to [exports in development](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-in-development) for more info. Export is available in dbt. ```bash dbt sl export-all # For dbt platform users (dbt v1 or v2) ``` #### FAQs  How can I add a dimension filter to a where filter? To add a dimension filter to a where filter, you have to indicate that the filter item is part of your model and use a template wrapper: `{{Dimension('primary_entity__dimension_name')}}`. Here's an example query: `dbt sl query --metrics order_total --group-by metric_time --where "{{Dimension('order_id__is_food_order')}} = True"`. Before using the template wrapper, however, set up your terminal to escape curly braces for the filter template to work. Details How to set up your terminal to escape curly braces? To configure your `.zshrc`profile to escape curly braces, you can use the `setopt` command to enable the `BRACECCL` option. This option will cause the shell to treat curly braces as literals and prevent brace expansion. Refer to the following steps to set it up: <br /> 1. Open your terminal. 2. Open your `.zshrc` file using a text editor like `nano`, `vim`, or any other text editor you prefer. You can use the following command to open it with `nano`: ```bash nano ~/.zshrc ``` 3. Add the following line to the file: ```bash setopt BRACECCL ``` 4. Save and exit the text editor (in `nano`, press Ctrl + O to save, and Ctrl + X to exit). 5. Source your `.zshrc` file to apply the changes: ```bash source ~/.zshrc ``` 6. After making these changes, your Zsh shell will treat curly braces as literal characters and will not perform brace expansion. This means that you can use curly braces without worrying about unintended expansions. Keep in mind that modifying your shell configuration files can have an impact on how your shell behaves. If you're not familiar with shell configuration, it's a good idea to make a backup of your `.zshrc` file before making any changes. If you encounter any issues or unexpected behavior, you can revert to the backup.  Why is my query limited to 100 rows in the dbt platform CLI? The default `limit` for query issues from the dbt platform CLI is 100 rows. We set this default to prevent returning unnecessarily large data sets as the dbt platform CLI is typically used to query the dbt Semantic Layer during the development process, not for production reporting or to access large data sets. For most workflows, you only need to return a subset of the data. However, you can change this limit if needed by setting the `--limit` option in your query. For example, to return 1000 rows, you can run `dbt sl list metrics --limit 1000`.  How can I query multiple metrics, group bys, or where statements? To query multiple metrics, group bys, or where statements in your command, follow this guidance: * To query multiple metrics and group bys, use the `--metrics` or `--group-by` syntax followed by the metric or dimension/entity names, separated by commas without spaces: * Multiple metrics example: `dbt sl query --metrics accounts_active,users_active` * Multiple dimension/entity example: `dbt sl query --metrics accounts_active,users_active --group-by metric_time__week,accounts__plan_tier` * To query multiple where statements, use the `--where` syntax and wrap the statement in quotes: * Multiple where statement example: `dbt sl query --metrics accounts_active,users_active --group-by metric_time__week,accounts__plan_tier --where "metric_time__week >= '2024-02-01'" --where "accounts__plan_tier = 'coco'"`  How can I sort my query in ascending or descending order? When you query metrics, use `--order-by` to specify metrics or groupings to order by. The `order_by` option applies to metrics, dimensions, and group bys. Add the `-` prefix to sort your query in descending (DESC) order. Leave blank for ascending (ASC) order: * For example, to query a metric and sort `metric_time` in descending order, run `dbt sl query --metrics order_total --group-by metric_time --order-by -metric_time`. Note that the `-` prefix in `-metric_time` sorts the query in descending order. * To query a metric and sort `metric_time` in ascending order and `revenue` in descending order, run `dbt sl query --metrics order_total --order-by metric_time,-revenue`. Note that `metric_time` without a prefix is sorted in ascending order and `-revenue` with a `-` prefix sorts the query in descending order. --- ### MetricFlow time spine (Applies to dbt v1.9 and later) It's common in analytics engineering to have a date dimension or "time spine" table as a base table for different types of time-based joins and aggregations. The structure of this table is typically a base column of daily or hourly dates, with additional columns for other time grains, like fiscal quarters, defined based on the base column. You can join other tables to the time spine on the base column to calculate metrics like revenue at a point in time, or to aggregate to a specific time grain. To use MetricFlow with time-based metrics and dimensions, you *must* provide a time spine. This table serves as the foundation for time-based joins and aggregations. You can either: * Create a time spine from scratch (check out the [example time spine](#example-time-spine-tables) section for examples), or * Use an existing table in your project, like a `dim_date` table And once you have a time spine, you need to configure it in YAML to tell MetricFlow how to use it. #### Prerequisites MetricFlow requires you to define at least one dbt model which provides a time-spine, and then specify (in YAML) the columns to be used for time-based joins. This means you need to: * Define at least one [time spine](#example-time-spine-tables) at whichever granularity needed for your metrics (like daily or hourly). You can optionally define additional tables for coarser grains (like monthly or yearly). * [Configure each time spine in a properties YAML file](#configuring-time-spine-in-yaml) to define how MetricFlow recognizes and uses its columns. Note that you can't have overlapping time spines. MetricFlow will then join against the time spine model for the following types of metrics and dimensions: * [Cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md) * [Metric offsets](https://docs.getdbt.com/docs/build/derived.md#derived-metric-offset) * [Conversion metrics](https://docs.getdbt.com/docs/build/conversion.md) * [Slowly Changing Dimensions](https://docs.getdbt.com/docs/build/dimensions.md#scd-type-ii) * [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) with the `join_to_timespine` configuration set to true To see the generated SQL for the metric and dimension types that use time spine joins, refer to the respective documentation or add the `compile=true` flag when querying the Semantic Layer to return the compiled SQL. #### Configuring time spine in YAML Use our mini guide to create a time spine table For a quick start guide on how to create a time spine table, check out our [MetricFlow time spine mini guide](https://docs.getdbt.com/guides/mf-time-spine.md)! Time spine models are normal dbt models with extra configurations that tell dbt and MetricFlow how to use specific columns by defining their properties. Add the [`models` key](https://docs.getdbt.com/reference/model-properties.md) for the time spine in your `models/` directory. If your project already includes a calendar table or date dimension, you can configure that table as a time spine. Otherwise, review the [example time-spine tables](#example-time-spine-tables) to create one. If the relevant model file doesn't exist, create it and add the configuration mentioned in the [next section](#creating-a-time-spine-table). Some things to note when configuring time spine models: * Make sure you already have a time spine SQL table defined in your project. * Add the configurations under the `time_spine` key for that [model's properties](https://docs.getdbt.com/reference/model-properties.md), just as you would add a description or tests. * You only need to configure time-spine models that the Semantic Layer should recognize. * At a minimum, define a time-spine table for a daily grain. * You can optionally define additional time-spine tables for different granularities, like hourly. Review the [granularity considerations](#granularity-considerations) when deciding which tables to create. * If you're looking to specify the grain of a time dimension so that MetricFlow can transform the underlying column to the required granularity, refer to the [Time granularity documentation](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran) tip * If you previously used a `metricflow_time_spine.sql` model, you can delete it after configuring the `time_spine` property in YAML. The Semantic Layer automatically recognizes the new configuration. No additional `.yml` files are needed. * You can also configure MetricFlow to use any date dimension or time spine table already in your project by updating the `model` setting in the Semantic Layer. * If you don’t have a date dimension table, you can still create one by using the code snippet in the [next section](#creating-a-time-spine-table) to build your time spine model. ##### Creating a time spine table MetricFlow supports granularities ranging from milliseconds to years. Refer to the [Dimensions page](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (time\_granularity tab) to find the full list of supported granularities. To create a time spine table from scratch, you can do so by adding the following code to your dbt project. This example creates a time spine at an hourly grain and a daily grain: `time_spine_hourly` and `time_spine_daily`. (Applies to dbt v1.9 and later) models/\_models.yml ```yaml models: # Hourly time spine - name: time_spine_hourly description: my favorite time spine time_spine: standard_granularity_column: date_hour # column for the standard grain of your table, must be date time type. custom_granularities: - name: fiscal_year column_name: fiscal_year_column # must refer to a column defined in the model columns: - name: date_hour granularity: hour # set granularity at column-level for standard_granularity_column # Daily time spine - name: time_spine_daily time_spine: standard_granularity_column: date_day # column for the standard grain of your table columns: - name: date_day granularity: day # set granularity at column-level for standard_granularity_column ``` * This example configuration shows a time spine model called `time_spine_hourly` and `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. * The `standard_granularity_column` is the column that maps to one of our [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran). This column must be set under the `columns` key and should have a grain that is finer or equal to any custom granularity columns defined in the same model. * It needs to reference a column defined under the `columns` key, in this case, `date_hour` and `date_day`, respectively. * It sets the granularity at the column-level using the `granularity` key, in this case, `hour` and `day`, respectively. * MetricFlow will use the `standard_granularity_column` as the join key when joining the time spine table to another source table. * [The `custom_granularities` field](#custom-calendar), (available in dbt Latest and dbt v1.9 and higher) lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use. * The `column_name` field must reference a column that exists in the same model. For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example. ##### Migrating from SQL to YAML If you already have a SQL model that defines your time spine, you can reference that model directly in the YAML file. If you don't have a SQL model that defines your time spine, add one before proceeding to the following steps. 1. Add the following configuration to a new or existing properties YAML file using the [`models` key](https://docs.getdbt.com/reference/model-properties.md) for the time spine in your `models/` directory. Name the properties YAML file whatever you want (for example, `util/_models.yml`): models/\_models.yml ```yaml models: - name: all_days description: A time spine with one row per day, ranging from 2020-01-01 to 2039-12-31. time_spine: standard_granularity_column: date_day # Column for the standard grain of your table columns: - name: date_day granularity: day # Set the granularity of the column ``` 2. After adding the YAML configuration and ensuring you have a SQL model that defines the time spine, you can delete the existing `metricflow_time_spine.sql` file from your project to avoid any deprecation warnings or errors. 3. Test the configuration to ensure compatibility with your production jobs. Note that if you're migrating from a `metricflow_time_spine.sql` file: * Replace its functionality by adding the `time_spine` property to YAML as shown in the previous example. * Once configured, MetricFlow will recognize the YAML settings, and then the SQL model file can be safely removed. ##### Considerations when choosing which granularities to create * MetricFlow will use the time spine with the largest compatible granularity for a given query to ensure the most efficient query possible. For example, if you have a time spine at a monthly grain, and query a dimension at a monthly grain, MetricFlow will use the monthly time spine. If you only have a daily time spine, MetricFlow will use the daily time spine and `date_trunc` to month. * You can add a time spine for each granularity you intend to use if query efficiency is more important to you than configuration time, or storage constraints. For most engines, the query performance difference should be minimal and transforming your time spine to a coarser grain at query time shouldn't add significant overhead to your queries. * We recommend having a time spine at the finest grain used in any of your dimensions to avoid unexpected errors. For example, if you have dimensions at an hourly grain, you should have a time spine at an hourly grain. #### Example time spine tables The following examples show how to create time spine tables at different granularities: * [Seconds](#seconds) * [Minutes](#minutes) * [Daily](#daily) * [Daily (BigQuery)](#daily-bigquery) * [Hourly](#hourly) ##### Seconds metricflow\_time\_spine.sql ```sql {{ config(materialized='table') }} with seconds as ( {{ dbt.date_spine( 'second', "date_trunc('second', dateadd(second, -10, current_timestamp()))", "date_trunc('second', current_timestamp())" ) }} ), final as ( select cast(date_second as timestamp) as second_timestamp from seconds ) select * from final ``` ##### Minutes metricflow\_time\_spine.sql ```sql {{ config(materialized='table') }} with minutes as ( {{ dbt.date_spine( 'minute', "date_trunc('minute', dateadd(minute, -5, current_timestamp()))", "date_trunc('minute', current_timestamp())" ) }} ), final as ( select cast(date_minute as timestamp) as minute_timestamp from minutes ) select * from final ``` ##### Daily metricflow\_time\_spine.sql ```sql {{ config( materialized = 'table', ) }} with days as ( {{ dbt.date_spine( 'day', "to_date('01/01/2000','mm/dd/yyyy')", "to_date('01/01/2025','mm/dd/yyyy')" ) }} ), final as ( select cast(date_day as date) as date_day from days ) select * from final where date_day > dateadd(year, -4, current_timestamp()) and date_day < dateadd(day, 30, current_timestamp()) ``` ##### Daily (BigQuery) Use this model if you're using BigQuery. BigQuery supports `DATE()` instead of `TO_DATE()`: metricflow\_time\_spine.sql ```sql {{config(materialized='table')}} with days as ( {{dbt.date_spine( 'day', "DATE(2000,01,01)", "DATE(2025,01,01)" ) }} ), final as ( select cast(date_day as date) as date_day from days ) select * from final -- filter the time spine to a specific range where date_day > date_add(DATE(current_timestamp()), INTERVAL -4 YEAR) and date_day < date_add(DATE(current_timestamp()), INTERVAL 30 DAY) ``` ##### Hourly time\_spine\_hourly.sql ```sql {{ config( materialized = 'table', ) }} with hours as ( {{ dbt.date_spine( 'hour', "to_date('01/01/2000','mm/dd/yyyy')", "to_date('01/01/2025','mm/dd/yyyy')" ) }} ), final as ( select cast(date_hour as timestamp) as date_hour from hours ) select * from final -- filter the time spine to a specific range where date_day > dateadd(year, -4, current_timestamp()) and date_hour < dateadd(day, 30, current_timestamp()) ``` #### Custom calendar [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") tip Check out our mini guide on [how to create a time spine table](https://docs.getdbt.com/guides/mf-time-spine.md) to get started! (Applies to dbt v1.9 and later) Custom date transformations can be complex, and organizations often have unique needs that can’t be easily generalized. Creating a custom calendar model allows you to define these transformations in SQL, offering more flexibility than native transformations in MetricFlow. This approach lets you map custom columns back to MetricFlow granularities, ensuring consistency while giving you control over the transformations. For example, if you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations. * This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks. * Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. * This feature provides more control over how time-based metrics are calculated.  Data types and time zone considerations When working with custom calendars in MetricFlow, it's important to ensure: * Consistent data types — Both your dimension column and the time spine column should use the same data type to allow accurate comparisons. Functions like `DATE_TRUNC` don't change the data type of the input in some databases (like Snowflake). Using different data types can lead to mismatches and inaccurate results. We recommend using `DATETIME` or `TIMESTAMP` data types for your time dimensions and time spine, as they support all granularities. The `DATE` data type may not support smaller granularities like hours or minutes. * Time zones — MetricFlow currently doesn't perform any timezone manipulation. When working with timezone-aware data, inconsistent time zones may lead to unexpected results during aggregations and comparisons. For example, if your time spine column is `TIMESTAMP` type and your dimension column is `DATE` type, comparisons between these columns might not work as intended. To fix this, convert your `DATE` column to `TIMESTAMP`, or make sure both columns are the same data type. ##### Add custom granularities To add custom granularities, the Semantic Layer supports custom calendar configurations that allow users to query data using non-standard time periods like `fiscal_year` or `retail_month`. You can define these custom granularities (all lowercased) by modifying your model's YAML configuration like this: models/\_models.yml ```yaml models: - name: my_time_spine description: my favorite time spine time_spine: standard_granularity_column: date_day custom_granularities: - name: fiscal_year column_name: fiscal_year_column # must refer to a column defined in the model ``` ###### Coming soon Note that features like calculating offsets and period-over-period will be supported soon! #### Related docs * [MetricFlow time granularity](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) * [MetricFlow time spine mini guide](https://docs.getdbt.com/guides/mf-time-spine.md) --- ### Metrics as dimensions with metric filters [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) provide users with valuable insights into their data, like number of active users and overall performance trends to inform business decisions. [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md), on the other hand, help categorize data through attributes, like user type or number of orders placed by a customer. To make informed business decisions, some metrics need the value of another metric as part of the metric definition, leading us to "metrics as dimensions". This document explains how you can use metrics as dimensions with metric filters, enabling you to create more complex metrics and gain more insights. #### Reference a metric in a filter Use the `Metric()` object syntax to reference a metric in the `where` filter for another metric. The function for referencing a metric accepts a metric name and exactly one entity: ```yaml {{ Metric('metric_name', group_by=['entity_name']) }} ``` ##### Usage example As an example, a Software as a service (SaaS) company wants to count activated accounts. In this case, the definition of an activated account is an account with more than five data model runs. To express this metric in SQL, the company will: * Write a query to calculate the number of data model runs per account. * Then count the number of accounts who have more than five data model runs. models/model\_name.sql ```sql with data_models_per_user as ( select account_id as account, count(model_runs) as data_model_runs from {{ ref('fct_model_runs') }} group by account_id ), activated_accounts as ( select count(distinct account_id) as activated_accounts from {{ ref('dim_accounts') }} left join data_models_per_user on {{ ref('dim_accounts') }}.account_id = data_models_per_user.account where data_models_per_user.data_model_runs > 5 ) select * from activated_accounts ``` This SQL query calculates the number of `activated_accounts` by using the `data_model_runs` metric as a dimension for the user entity. It filters based on the metric value scoped to the account entity. You can express this logic at the query level or in the metric's YAML configuration. ###### YAML configuration Using the same `activated_accounts` example mentioned in [the usage example](#usage-example), the following YAML example explains how a company can create [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) and [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md), and use the `Metric()` object to reference the `data_model_runs` metric in the `activated_accounts` metric filter: (Applies to dbt v1.12 and later) * Create two semantic models: `model_runs` and `accounts`. * Create metrics to count data model runs and users. * Specify the foreign entity `account` in the `model_runs` semantic model. * Then create the `Activated Accounts` metric by filtering accounts that have more than five data model runs. models/model\_runs.yml ```yaml models: - name: model_runs semantic_model: enabled: true # ... other configurations ... columns: - name: model_run_id entity: type: primary name: model_run - name: account_id entity: type: foreign name: account metrics: - name: data_model_runs type: simple agg: sum expr: 1 - name: accounts semantic_model: enabled: true # ... other configurations ... columns: - name: account_id entity: type: primary name: account metrics: - name: accounts type: simple agg: sum expr: 1 # Metric that filters accounts based on another metric - name: activated_accounts label: Activated Accounts type: simple agg: sum expr: 1 filter: | {{ Metric('data_model_runs', group_by=['account']) }} > 5 ``` Let’s break down the SQL the system generates based on the metric definition when you run `dbt sl query --metrics activated_accounts` from the command line interface: * The filter `{{ Metric('data_model_runs', group_by=['account']) }}` generates SQL similar to the `data_models_per_user` sub-query shown earlier: ```sql select sum(1) as data_model_runs, account from data_model_runs group by account ``` * MetricFlow joins this query to the query generated by `accounts` on the group by elements and applies the filter conditions: ```sql select sum(1) as activated_accounts from accounts left join ( select sum(1) as data_model_runs, account from data_model_runs group by account ) as subq on accounts.account = subq.account where data_model_runs > 5 ``` The intermediate tables used to create this metric is: Accounts with the `data_model_runs` dimension | account | data\_model runs | | ------- | ---------------- | | 1 | 4 | | 2 | 7 | | 3 | 9 | | 4 | 1 | MetricFlow then filters this table to accounts with more than 5 data model runs and counts the number of accounts that meet this criteria: | activated\_accounts | | ------------------- | | 2 | ###### Query filter You can also use metrics in filters at the query level. Run this command in the command line interface (CLI) to generate the same SQL query referenced earlier: `dbt sl query --metrics accounts --where "{{ Metric('data_model_runs', group_by=['account']) }} > 5"` The resulting SQL and data will be the same, except with the `accounts` metric name instead of `activated_accounts`. #### Considerations * When using a metric filter, ensure the sub-query can join to the outer query without fanning out the result (unexpectedly increasing the number of rows). * The example that filters accounts using `{{ Metric('data_model_runs', group_by=['account']) }}` is valid because it aggregates the model runs to the account level. * However, filtering 'accounts' by `{{ Metric('data_model_runs', group_by=['model']) }}` isn't valid due to a one-to-many relationship between accounts and model runs, leading to duplicate data. * You can only group a metric by one entity. The ability to support grouping by multiple entities and dimensions is pending. * In the future, you can use metrics as dimensions for some of the following example use cases: * User segments: Segment users by using the number of orders placed by a user in the last 7 days as a dimension. * Churn prediction: Use the number of support tickets an account submitted in the first 30 days to predict potential churn. * Activation tracking: Define account or user activation based on the specific actions taken within a certain number of days after signing up. * Support for metric filters requiring multi-hop joins is pending. --- ### Microsoft Excel dbt platform | Starter, Enterprise, Enterprise+ The Semantic Layer offers a seamless integration with Excel Online and Desktop through a custom menu. This add-on allows you to build Semantic Layer queries and return data on your metrics directly within Excel. #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and are using dbt v1.6 or higher. * You need a Microsoft Excel account with access to install add-ons. * You have a [dbt Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer). * You have a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to authenticate with from a dbt account. * You must have a dbt Starter, Enterprise, or Enterprise+ [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. tip 📹 For on-demand video learning, explore the [Querying the Semantic Layer with Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel) course to learn how to query metrics with Excel. #### Installing the add-on The Semantic Layer Microsoft Excel integration is available to download directly on [Microsoft AppSource](https://appsource.microsoft.com/en-us/product/office/WA200007100?tab=Overview). You can choose to download this add-on in for both [Excel Desktop](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationId=4132ecd1-425d-982d-efb4-de94ebc83f26) and [Excel Online](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationid=4132ecd1-425d-982d-efb4-de94ebc83f26\&isWac=True) 1. In Excel, authenticate with your Host, dbt Environment ID, and service token. * Access your Environment ID, Host, and URLs in your Semantic Layer settings. Generate a service token in the Semantic Layer settings or **API tokens** settings. Alternatively, you can also create a personal access token by going to **API tokens** > **Personal tokens**. [![Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings](/img/docs/dbt-platform/semantic-layer/sl-and-gsheets.png?v=2 "Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings")](#)Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings 2. Start querying your metrics using the **Query Builder**. For more info on the menu functions, refer to [Query Builder functions](#query-builder-functions). To cancel a query while running, press the **Cancel** button. When querying your data with Microsoft Excel: * It returns the data to the cell you clicked on. * Results that take longer than one minute to load into Excel will fail. This limit only applies to the loading process, not the time it takes for the data platform to run the query. * If you're using this extension, make sure you're signed into Microsoft with the same Excel profile you used to set up the Add-In. Log in with one profile at a time as using multiple profiles at once might cause issues. * Note that only standard granularities are currently available, custom time granularities aren't currently supported for this integration. #### Query Builder functions The Microsoft Excel **Query Builder** custom menu has the following capabilities: | Menu items | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metrics | Search and select metrics. | | Group By | Search and select dimensions or entities to group by. Dimensions are grouped by the entity of the semantic model they come from. You may choose dimensions on their own without metrics. | | Time Range | Quickly select time ranges to look at the data, which applies to the main time series for the metrics (metric time), or do more advanced filter using the "Custom" selection. | | Where | Filter your data. This includes categorical and time filters. | | Order By | Return your data order. | | Limit | Set a limit for the rows of your output. | Note: Click the **info** button next to any metric or dimension to see its defined description from your dbt project. ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). ###### Filtering data To use the filter functionality, choose the [dimension](https://docs.getdbt.com/docs/build/dimensions.md) you want to filter by and select the operation you want to filter on. * For categorical dimensions, you can type a value into search or select from a populated list. * For entities, you must type the value you are looking for as we do not load all of them given the large number of values. * Continue adding additional filters as needed with AND and OR. * For time dimensions, you can use the time range selector to filter on presets or custom options. The time range selector applies only to the primary time dimension (`metric_time`). For all other time dimensions that aren't `metric_time`, you can use the "Where" option to apply filters. ###### Other settings If you would like to just query the data values without the headers, you can optionally select the **Exclude column names** box. To return your results and keep any previously selected data below it intact, un-select the **Clear trailing rows** box. By default, we'll clear all trailing rows if there's stale data. [![Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.](/img/docs/dbt-platform/semantic-layer/query-builder.png?v=2 "Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.")](#)Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings. #### Using saved selections Saved selections allow you to save the inputs you've created in the Microsoft Excel **Query Builder** and easily access them again so you don't have to continuously build common queries from scratch. To create a saved selection: 1. Run a query in the **Query Builder**. 2. Save the selection by selecting the arrow next to the **Query** button and then select **Query & Save Selection**. 3. The application saves these selections, allowing you to view and edit them from the hamburger menu under **Saved Selections**. ##### Refreshing selections Set your saved selections to automatically refresh every time you load the addon. You can do this by selecting **Refresh on Load** when creating the saved selection. When you access the addon and have saved selections that should refresh, you'll see "Loading..." in the cells that are refreshing. Public saved selections will refresh for anyone who edits the sheet. What's the difference between saved selections and saved queries? * Saved selections are saved components that you can create only when using the application. * Saved queries, explained in the next section, are code-defined sections of data you create in your dbt project that you can easily access and use for building selections. You can also use the results from a saved query to create a saved selection. #### Using saved queries Access [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), powered by MetricFlow, in Microsoft Excel to quickly get results from pre-defined sets of data. To access the saved queries in Microsoft Excel: 1. Open the hamburger menu in Microsoft Excel. 2. Navigate to **Saved Queries** to access the ones available to you. 3. You can also select **Build Selection**, which allows you to explore the existing query. This won't change the original query defined in the code. * If you use a `WHERE` filter in a saved query, Microsoft Excel displays the advanced syntax for this filter. #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### Migrate to dbt Wizard Local development Move from Claude Code to dbt Wizard while keeping your project conventions. dbt Wizard automatically imports Claude Code instructions, skills, and settings from your repo. Use this page if you already use Claude Code on dbt projects and want to bring over project context, skills, or model settings. If you're new to AI agents, start with [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). #### Prerequisites You'll need: * [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) installed or dbt Wizard enabled in [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) configured for a supported CLI provider (OpenAI, Anthropic, AWS Bedrock, or Snowflake Cortex in preview) * Any existing Claude Code files you want to migrate, such as `CLAUDE.md`, `.claude/CLAUDE.md`, or `.claude/skills/` #### Review project context and skills dbt Wizard automatically reads common agent instruction files and skills from your repo. You usually don't need to move existing project context before using dbt Wizard. | File or folder | What Wizard does | Migration needed? | | ---------------------------------------- | -------------------------------------------------------------- | ----------------- | | `AGENTS.override.md` at the project root | Checks this first for local override instructions. | No | | `AGENTS.md` at the project root | Uses this as the primary project instruction file. | No | | `CLAUDE.md` at the project root | Uses this as a built-in fallback when `AGENTS.md` is absent. | No | | `.claude/CLAUDE.md` | Uses this as another fallback instruction file. | No | | `.claude/skills/NAME/SKILL.md` | Auto-discovers these as skills. | No | | Reusable prompts or new skills | Place them in `.agents/skills/NAME/SKILL.md` at the repo root. | Yes | dbt Wizard also walks the directory tree from your project root to your current working directory, picks up instruction files at each level, and combines them for the session. For the full skill format, refer to [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md). ##### Convert reusable prompts to skills Use skills for reusable workflows or specialized instructions, not for general project context that already lives in `AGENTS.md` or `CLAUDE.md`. Each repo-level skill lives in `.agents/skills/SKILL_NAME/SKILL.md`. ```bash mkdir -p .agents/skills/my-team-style touch .agents/skills/my-team-style/SKILL.md ``` Then add frontmatter and instructions: ```markdown --- name: my-team-style description: Apply team-specific dbt modeling conventions when creating, editing, refactoring, testing, or documenting models in this project. --- # Team style (add your reusable instructions here) ``` Write a specific `description:` because dbt Wizard uses it to decide when to load the skill. #### Configure model settings Set provider credentials in [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md). To set a default model across sessions, add this to `~/.dbt/wizard/config.toml`: ```toml model = "claude-sonnet-4-6" # use a model ID from `wizard debug models` ``` You can also pick a model in the TUI with `/model` without editing config or the `--config` flag. Run `wizard --help` to see all available flags. #### Verify the migration From your dbt project root, start a new session: ```bash wizard ``` Ask dbt Wizard to do something that should use your migrated skill, such as: ```text create a new staging model for raw_invoices ``` And if your conventions aren't applied, check: * General project context is in an instruction file such as `AGENTS.md` or `CLAUDE.md` * Reusable skills are under `.agents/skills/` at the project root or `~/.agents/skills/` for CLI-global skills * The skill has a clear `description:` * You started a new session after adding the skill #### Related docs * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows --- ### Migrate to the latest YAML spec The latest Semantic Layer specification creates an open standard for defining metrics and dimensions that works across multiple platforms. It simplifies authorship by embedding semantic annotations alongside each model, replacing measures with simple metrics, and promoting frequently used options to top-level keys. With the new spec, you get simpler configuration without losing flexibility, faster onboarding for new contributors, and a clearer path to consistent, governed metrics across your organization. Availability The latest YAML spec is supported in the following environments: * **dbt platform (v1 Latest release track)** * **dbt v2** * **dbt v1.12** #### Changes in the latest spec This section highlights the key updates in the latest metrics spec and compares them to the legacy spec. * [Semantic models](#semantic-models): These define the business logic for your metrics by specifying entities, dimensions, and how they relate to your data models. In the new spec, `semantic_model` is nested directly under each model in `models:` instead of being a top-level key. * [Entities and dimensions](#entities-and-dimensions): Entities are the people, places, or things you want to group or join your metrics by (like `user_id` or `order_id`), while dimensions are the attributes you use to filter or slice your data (like `status` or `region`). In the new spec, both are defined directly under `columns:`. * [Time dimension](#time-dimension): Time dimensions are the date or timestamp columns that let you analyze metrics over time (like `order_date` or `created_at`). In the new spec, set `agg_time_dimension` at the model level as the default time dimension for all metrics, with the option to override per metric. Define `granularity` at the column level instead of using the deprecated `time_granularity`. * [Simple metrics](#simple-metrics): Metrics that directly reference a single column expression within a semantic model, without any additional columns involved. Simple metrics replace measures in the new spec. Use `type: simple` metrics defined directly within the model to replace measures. * [Advanced metrics](#advanced-metrics): These are metrics that combine or build upon other metrics, such as ratios, conversions, or derived calculations. In the new spec, define simple metrics inside the model, and create cross‑model metrics under a top‑level `metrics` block. Top-level key is required for any metric that depends on metrics or dimensions defined in a different semantic model. * [`type_params`](#type_params): This is a wrapper key in the legacy spec that contains metric-specific configurations (for example, `expr`, `join_to_timespine`). `type_params` is deprecated in the new spec and these parameters are promoted to top-level keys within each metric definition. ##### Semantic models The `semantic_model` key is embedded under `models`. ###### New spec ```yml models: - name: fct_orders semantic_model: enabled: true # required name: fct_orders_semantic_model # optional override; defaults to value of model.name ``` ###### Legacy spec ```yml semantic_models: - name: orders model: ref('orders') ``` Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ##### Entities and dimensions Entities and dimensions are defined directly under columns, creating a 1:1 relationship between the physical columns and their semantic definitions. ###### New spec ```yml models: - name: orders semantic_model: enabled: true agg_time_dimension: ordered_at columns: # entities - name: order_id entity: type: primary name: order - name: customer_id entity: type: foreign name: customer # time dimension - name: ordered_at granularity: day dimension: type: time # categorical dimension - name: order_status dimension: type: categorical ``` ###### Legacy spec ```yml semantic_models: - name: orders model: ref('orders') entities: - name: order type: primary expr: order_id - name: customer type: foreign expr: customer_id dimensions: - name: ordered_at type: time type_params: time_granularity: day - name: status type: categorical expr: order_status ``` ##### Time dimension * `agg_time_dimension`: Set once at the model level as the default time dimension for all metrics in that semantic model. You can still override it per metric with `agg_time_dimension`. * `time granularity`: Deprecated in the new spec. Define the native grain on the time dimension column with `granularity` (for example, `hour`, `day`). ###### New spec ```yml models: - name: subscriptions semantic_model: enabled: true # default aggregation time dimension for metrics in this model agg_time_dimension: activated_at columns: - name: activated_at granularity: day # native grain on the column dimension: type: time - name: created_at granularity: hour # another time column with a different native grain dimension: type: time metrics: - name: active_subscriptions type: simple agg: count expr: 1 # inherits agg_time_dimension: activated_at - name: signups_by_created_day type: simple agg: count expr: 1 agg_time_dimension: created_at # override to use created_at as the time dimension ``` ###### Legacy spec ```yml semantic_models: - name: subscriptions model: ref('subscriptions') defaults: agg_time_dimension: activated_at dimensions: - name: activated_at type: time type_params: time_granularity: day - name: created_at type: time type_params: time_granularity: hour measures: - name: active_subscriptions agg: count metrics: - name: active_subscriptions type: simple type_params: measure: active_subscriptions ``` ##### Simple metrics Measures are deprecated in the new spec and are replaced with simple metrics. ###### New spec ```yml models: - name: customers semantic_model: enabled: true agg_time_dimension: first_ordered_at columns: - name: customer_id entity: name: customer type: primary - name: first_ordered_at dimension: type: time granularity: day metrics: - name: lifetime_spend_pretax type: simple # simple metrics agg: sum expr: amount_pretax ``` ###### Legacy spec ```yml semantic_models: - name: customers model: ref('customers') entities: - name: customer type: primary expr: customer_id dimensions: - name: first_ordered_at type: time type_params: time_granularity: day measures: - name: lifetime_spend_pretax agg: sum metrics: - name: lifetime_spend_pretax type: simple type_params: measure: lifetime_spend_pretax ``` ##### Advanced metrics Define simple metrics inside the model, and create cross‑model metrics under a top‑level `metrics` block. Top-level key is required for any metric that depends on metrics or dimensions defined in a different semantic model. ###### New spec ```yml # define simple metrics where the data lives models: - name: orders ... semantic_model: enabled: true metrics: - name: orders type: simple agg: count expr: 1 - name: website semantic_model: enabled: true metrics: - name: sessions type: simple agg: count expr: 1 # advanced metrics under top-level metrics key metrics: - name: orders_per_session type: ratio numerator: orders denominator: sessions ``` ###### Legacy spec ```yml semantic_models: - name: orders model: ref('orders') measures: - name: orders agg: count - name: website model: ref('website') measures: - name: sessions agg: count metrics: - name: orders_per_session type: ratio type_params: numerator: { measure: orders } denominator: { measure: sessions } ``` ##### `type_params` The `type_params` key is deprecated. The following are direct keys on the metric: * `expr` * `percentile` * `percentile_type` * `non_additive_dimension: { name, window_agg, group_by }` * `join_to_timespine` * `fill_nulls_with` ###### New spec ```yml models: - name: payments semantic_model: enabled: true metrics: - name: revenue_p95 type: simple agg: percentile expr: amount percentile: 95.0 percentile_type: discrete ``` ###### Legacy spec ```yml metrics: - name: revenue_p95 type: simple type_params: expr: amount percentile: 95.0 percentile_type: discrete ``` For [derived metrics](https://docs.getdbt.com/docs/build/derived.md), `type_params.metrics` is renamed `input_metrics`. ###### New spec ```yaml metrics: - name: d7_booking_change description: Difference between bookings now and 7 days ago type: derived label: d7 bookings change expr: current_bookings - bookings_7_days_ago input_metrics: - name: bookings alias: current_bookings - name: bookings offset_window: 7 days alias: bookings_7_days_ago ``` ###### Legacy spec ```yaml - name: d7_booking_change description: Difference between bookings now and 7 days ago type: derived label: d7 bookings change type_params: expr: bookings - bookings_7_days_ago metrics: - name: bookings alias: current_bookings - name: bookings offset_window: 7 days alias: bookings_7_days_ago ``` For [ratio metrics](https://docs.getdbt.com/docs/build/ratio.md), `numerator` and `denominator` are now direct keys on the metric. ###### New spec ```yaml metrics: - name: conversion_rate type: ratio numerator: conversions denominator: sessions ``` ###### Legacy spec ```yaml metrics: - name: conversion_rate type: ratio type_params: numerator: conversions denominator: sessions ``` For [cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md): * `type_params.measure` is renamed `input_metric` and must reference a metric. * `type_params.cumulative_type_params` values are direct keys on the metric: `window`, `grain_to_date`, and `period_agg`. ###### New spec ```yaml metrics: - name: revenue_mtd_cumulative type: cumulative input_metric: revenue_daily window: 30d grain_to_date: month period_agg: sum ``` ###### Legacy spec ```yaml metrics: - name: revenue_mtd_cumulative type: cumulative type_params: measure: revenue_daily cumulative_type_params: window: 30d grain_to_date: month period_agg: sum ``` For [conversion metrics](https://docs.getdbt.com/docs/build/conversion.md), the following `type_params.conversion_type_params` values are direct keys on the metric: * `entity` * `calculation` * `base_metric` (previously `base_measure`) * `conversion_metric` (previously `conversion_measure`) * `constant_properties` ###### New spec ```yaml metrics: - name: paid_signup_conversion type: conversion entity: user_id calculation: conversion_rate base_metric: signups conversion_metric: paid_signups constant_properties: - base_property: plan conversion_property: plan ``` ###### Legacy spec ```yaml metrics: - name: paid_signup_conversion type: conversion type_params: conversion_type_params: entity: user_id calculation: conversion_rate base_measure: signups conversion_measure: paid_signups constant_properties: plan: pro ``` #### Migrating to the latest spec Studio IDE YAML validation The Studio IDE validates dbt YAML using JSON Schema from the [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema) project. These definitions are aligned with dbt v2 and apply across all [dbt platform release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), including when your development environment is still running dbt v1. If the Studio IDE flags your YAML as invalid but dbt commands succeed, trust your run results. Share examples with [dbt Support](mailto:support@getdbt.com) or your account team so the schema can be updated. Migrate your legacy metrics to the latest YAML spec using the dbt-autofix tool in your CLI, the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md), or dbt platform's Studio IDE. note dbt Copilot doesn't yet support generating semantic models with the latest YAML spec. ##### Package compatibility If your project uses dbt packages (listed in `packages.yml`) that define metrics or semantic models, the package maintainer must update those packages to use the latest YAML spec. The [dbt-autofix tool](https://github.com/dbt-labs/dbt-autofix) only updates files in your current dbt project (like models, marts, and so on) and does not update installed packages under `dbt_packages/`. If an installed package still uses the legacy metrics spec, dbt may raise parsing or validation errors after migration. To update packages, a package maintainer should: 1. Run `dbt-autofix deprecations --semantic-layer` in the package repository. 2. Validate the changes by running: * For dbt v2 and dbt users in the dbt platform CLI or locally with a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md): ```bash dbt parse dbt sl validate ``` When using `dbt sl validate` locally, the command validates your local semantic manifest, and not the platform's manifest. This means your uncommitted local changes are included in the validation. * For dbt v2 CLI users not connected to dbt platform and using local MetricFlow: ```bash dbt parse mf validate-configs ``` 3. Release a new version of the package with the updated metrics definitions. After a compatible version is released, update your project to [install the new package version](https://docs.getdbt.com/docs/build/packages.md). You can then migrate your metrics to the latest spec with the following steps, depending on which tool you're using. * [Using the CLI or VS Code extension](#using-the-cli-or-vs-code-extension) * [Using the Studio IDE](#using-the-studio-ide) ##### Using the CLI or VS Code extension The [dbt-autofix tool](https://github.com/dbt-labs/dbt-autofix) rewrites legacy metrics YAML into the latest format and produces a clear, reviewable diff in version control. Make sure you have installed the latest version of the autofix tool before migrating to the new spec using the CLI or the dbt VS Code extension. 1. In your CLI or in the VS Code extension, run the following command: ```bash dbt-autofix deprecations --semantic-layer ``` 2. Review the diff and resolve all flagged items. 3. Run parsing and validations: ```bash dbt parse mf validate-configs ``` ##### Using the Studio IDE Convert your metrics in the Studio IDE in the dbt platform without having to install the `dbt-autofix` tool. 1. Navigate to the Studio IDE by clicking **Studio** in the left menu. 2. Make sure to save and commit your work before proceeding. The autofix command may overwrite any unsaved changes. 3. In the Studio IDE, run the following command: ```bash dbt-autofix deprecations --semantic-layer ``` 4. Click **Commit and sync** in the top left of the Studio IDE to commit these changes to the project repository. --- ### Migrating from state-aware orchestration to dbt State Login required | Usage-based dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). #### Migrating your configuration Much of dbt State's configuration will feel familiar if you've used state-aware orchestration, but there is one significant difference: the `build_after` configs have moved out of the `freshness` block and into a new `state` block. To migrate to dbt State, move your configs from `freshness.build_after` to the new `state` block. Refer to the following table for the full mapping. | State-aware orchestration | dbt State | Notes | | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `freshness.build_after.updates_on` | [`state.require_fresh_data_from`](https://docs.getdbt.com/reference/resource-configs/require-fresh-data-from.md) | Same `any` and `all` options with the same behavior:<br />- `any` (default): rebuilds when *any* direct parent has fresh data<br />- `all`: rebuilds only when *all* direct parents have fresh data | | `freshness.build_after.count` + `freshness.build_after.period` | [`state.lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) | Combined into a single field with shorthand values (for example, `1800s`, `30m`, `12h`, `1d`, `2w`) or Jinja expressions | Backward compatibility in dbt v2 In dbt v2, you can enable dbt State without updating your project configs first. * If `lag_tolerance` and `require_fresh_data_from` are not set, dbt State falls back to your existing `build_after` configs until `build_after` is deprecated. * If neither `build_after` nor the `state` configs exist, dbt State uses its [default configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md): `lag_tolerance: 45m` and `require_fresh_data_from: any`. ##### Examples You can set these configs at the project level in `dbt_project.yml` or at the model level in a `.yml` file. **Project-level:** **Before (state-aware orchestration)** dbt\_project.yml ```yaml models: +freshness: build_after: count: 1 period: day updates_on: any ``` **After (dbt State)** dbt\_project.yml ```yaml models: +state: lag_tolerance: 1d require_fresh_data_from: any ``` **Model-level:** **Before (state-aware orchestration)** models/my\_model.yml ```yaml models: - name: my_model config: freshness: build_after: count: 1 period: day updates_on: any ``` **After (dbt State)** models/my\_model.yml ```yaml models: - name: my_model config: state: lag_tolerance: 1d require_fresh_data_from: any ``` #### Testing the migration You can run both configs side by side while validating. State-aware orchestration reads `freshness.build_after` and dbt State reads the `state` block, so they won't interfere with each other. Once you're confident everything looks right, remove the `build_after` configs. dbt\_project.yml ```yaml models: +freshness: build_after: # state-aware orchestration (remove once migrated) count: 1 period: day updates_on: any +state: # dbt State lag_tolerance: 1d require_fresh_data_from: any ``` #### Known differences from state-aware orchestration State-aware orchestration and dbt State differ in a few ways: * **More models rebuilding than expected**: If you notice more rebuilds after you migrate, the most common causes are: * **Views with `select *` on a `ref()` or `source()`**: dbt State can't determine which columns `select *` resolves to without querying the upstream schema, so it rebuilds these views rather than risk reusing a stale result. Views that use `select *` on a CTE are reused, because dbt can resolve the columns from the CTE definition. For more information, refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select). * **Non-determinism in Jinja-templated SQL**: Macros like `dbt_utils.get_relations_by_pattern` with `dbt_utils.union_relations` can return relations in a different order on each run, which produces different compiled SQL. dbt State detects a new hash and rebuilds the model. If that model has downstream dependencies, those models rebuild, too. * **Models with external sources on BigQuery**: Models that use external sources (such as Google Sheets) always rebuild because BigQuery doesn't expose modification timestamps for external sources, so dbt State can't determine freshness. To avoid this, configure [`loaded_at_field`](https://docs.getdbt.com/reference/resource-configs/freshness.md) or [`loaded_at_query`](https://docs.getdbt.com/reference/resource-configs/freshness.md) in your source definition to point to a timestamp field — this lets dbt State query a timestamp field directly to determine freshness, instead of relying on warehouse metadata. Refer to [Why is my model being rebuilt instead of reused?](https://docs.getdbt.com/faqs/State/views-rebuilt.md) for details on each cause and how to diagnose them. * **`build_after` vs `lag_tolerance`**: Both configs reduce how often a model runs when upstream data is frequently fresh, but they work differently: * `freshness.build_after` (for example, `{count: 4, period: hour}`) skips the model unless the configured interval has elapsed *and* upstream sources have new data since the last run. A SQL change alone does not trigger a rebuild; both conditions must be met. * `state.lag_tolerance` (for example, `4h`) skips the model unless upstream data is newer than the model's last run by at least the configured interval. Unlike `build_after`, a detected SQL change triggers a rebuild. * **No concurrent build detection**: In most cases, dbt State handles overlapping nodes the same way as state-aware orchestration: if a node is built by Job 1 before Job 2 starts, Job 2 sees the model as already built and reuses it. If Job 1 finishes building a node after Job 2 starts — but before Job 2 builds that node — Job 2 still builds it and uses compute to re-determine there's no new data. However, if Job 2 starts building the same snapshot or incremental model before Job 1 executing the same node, both jobs can detect the same changes in their separate transactions and commit them, which can lead to duplicate records or other data corruption. To prevent duplicate builds, ensure jobs that share the same nodes don't run at the same time — whether by configuring job schedules through the [dbt platform job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) or an external orchestration tool (for example, Airflow or Dagster), or by avoiding manually triggered runs that overlap. * **Efficient Testing not yet available**: State-aware orchestration offers Efficient Testing (private beta); dbt State doesn't support it yet. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### Model access "Model access" is not "User access" **Model groups and access** and **user groups and access** mean two different things. "User groups and access" is a specific term used in dbt to manage permissions. Refer to [User access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) for more info. The two concepts will be closely related, as we develop multi-project collaboration workflows this year: * Users with access to develop in a dbt project can view and modify **all** models in that project, including private models. * Users in the same dbt account *without* access to develop in a project cannot view that project's private models, and they can take a dependency on its public models only. #### Related documentation * [`groups`](https://docs.getdbt.com/docs/build/groups.md) * [`access`](https://docs.getdbt.com/reference/resource-configs/access.md) #### Groups Models can be grouped under a common designation with a shared owner. For example, you could group together all models owned by a particular team, or related to modeling a specific data source (`github`). Why define model `groups`? There are two reasons: * It turns implicit relationships into an explicit grouping, with a defined owner. By thinking about the interface boundaries *between* groups, you can have a cleaner (less entangled) DAG. In the future, those interface boundaries could be appropriate as the interfaces between separate projects. * It enables you to designate certain models as having "private" access—for use exclusively within that group. Other models will be restricted from referencing (taking a dependency on) those models. In the future, they won't be visible to other teams taking a dependency on your project—only "public" models will be. If you follow our [best practices for structuring a dbt project](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md), you're probably already using subdirectories to organize your dbt project. It's easy to apply a `group` label to an entire subdirectory at once: dbt\_project.yml ```yml models: my_project_name: marts: customers: +group: customer_success finance: +group: finance ``` Each model can only belong to one `group`, and groups cannot be nested. If you set a different `group` in that model's YAML or in-file config, it will override the `group` applied at the project level. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. #### Access modifiers Some models are implementation details, meant for reference only within their group of related models. Other models should be accessible through the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function across groups and projects. Models can set an [access modifier](https://en.wikipedia.org/wiki/Access_modifiers) to indicate their intended level of accessibility. | Access | Referenceable by | | --------- | ---------------------------------------------------------------------------------------- | | private | Same group | | protected | Same project (or installed as a package) | | public | Any group, package, or project. When defined, rerun a production job to apply the change | If you try to reference a model outside of its supported access, you will see an error: ```shell dbt run -s marketing_model ... dbt.exceptions.DbtReferenceError: Parsing Error Node model.jaffle_shop.marketing_model attempted to reference node model.jaffle_shop.finance_model, which is not allowed because the referenced node is private to the finance group. ``` (Applies to dbt v1.12 and later) When a macro is invoked with `dbt run-operation`, dbt doesn't enforce model access or group controls, so you can use `ref()` to reference all models (including `private` and `protected` ones). By default, all models are `protected`. This means that other models in the same project can reference them, regardless of their group. This is largely for backward compatibility when assigning groups to an existing set of models, as there may already be existing references across group assignments. However, it is recommended to set the access modifier of a new model to `private` to prevent other project resources from taking dependencies on models not intentionally designed for sharing across groups. models/marts/customers.yml ```yaml # First, define the group and owner groups: - name: customer_success owner: name: Customer Success Team email: cx@jaffle.shop # Then, add 'group' + 'access' modifier to specific models models: # This is a public model -- it's a stable & mature interface for other teams/projects - name: dim_customers config: group: customer_success # changed to config in v1.10 access: public # changed to config in v1.10 # This is a private model -- it's an intermediate transformation intended for use in this context *only* - name: int_customer_history_rollup config: group: customer_success # changed to config in v1.10 access: private # changed to config in v1.10 # This is a protected model -- it might be useful elsewhere in *this* project, # but it shouldn't be exposed elsewhere - name: stg_customer__survey_results config: group: customer_success # changed to config in v1.10 access: protected # changed to config in v1.10 ``` Models with `materialized` set to `ephemeral` cannot have the access property set to public. For example, if you have a model config set as: models/my\_model.sql ```sql {{ config(materialized='ephemeral') }} ``` And the model access is defined: models/my\_project.yml ```yaml models: - name: my_model config: access: public # changed to config in v1.10 ``` It will lead to the following error: ```text ❯ dbt parse 02:19:30 Encountered an error: Parsing Error Node model.jaffle_shop.my_model with 'ephemeral' materialization has an invalid value (public) for the access field ``` #### FAQs ##### How does model access relate to database permissions? These are different! Specifying `access: public` on a model does not trigger dbt to automagically grant `select` on that model to every user or role in your data platform when you materialize it. You have complete control over managing database permissions on every model/schema, as makes sense to you & your organization. Of course, dbt can facilitate this by means of [the `grants` config](https://docs.getdbt.com/reference/resource-configs/grants.md), and other flexible mechanisms. For example: * Grant access to downstream queriers on public models * Restrict access to private models, by revoking default/future grants, or by landing them in a different schema As we continue to develop multi-project collaboration, `access: public` will mean that other teams are allowed to start taking a dependency on that model. This assumes that they've requested, and you've granted them access, to select from the underlying dataset. ##### How do I ref a model from another project? You can `ref` a model from another project in two ways: 1. [Project dependency](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md): In dbt Enterprise, you can use project dependencies to `ref` a model. dbt uses a behind-the-scenes metadata service to resolve the reference, enabling efficient collaboration across teams and at scale. 2. ["Package" dependency](https://docs.getdbt.com/docs/build/packages.md): Another way to `ref` a model from another project is to treat the other project as a package dependency. This requires installing the other project as a package, including its full source code, as well as its upstream dependencies. ##### How do I restrict access to models defined in a package? Source code installed from a package becomes part of your runtime environment. You can call macros and run models as if they were macros and models that you had defined in your own project. For this reason, model access restrictions are "off" by default for models defined in packages. You can reference models from that package regardless of their `access` modifier. The project is installed as a package can optionally restrict external `ref` access to just its public models. The package maintainer does this by setting a `restrict-access` config to `True` in `dbt_project.yml`. By default, the value of this config is `False`. This means that: * Models in the package with `access: protected` may be referenced by models in the root project, as if they were defined in the same project * Models in the package with `access: private` may be referenced by models in the root project, so long as they also have the same `group` config When `restrict-access: True`: * Any `ref` from outside the package to a protected or private model in that package will fail. * Only models with `access: public` can be referenced outside the package. dbt\_project.yml ```yml restrict-access: True # default is False ``` --- ### Model contracts #### Related documentation * [`contract`](https://docs.getdbt.com/reference/resource-configs/contract.md) * [`columns`](https://docs.getdbt.com/reference/resource-properties/columns.md) * [`constraints`](https://docs.getdbt.com/reference/resource-properties/constraints.md) #### Why define a contract? Defining a dbt model is as easy as writing a SQL `select` statement. Your query naturally produces a dataset with columns of names and types based on the columns you select and the transformations you apply. While this is ideal for quick and iterative development, for some models, constantly changing the shape of its returned dataset poses a risk when other people and processes are querying that model. It's better to define a set of upfront "guarantees" that define the shape of your model. We call this set of guarantees a "contract." While building your model, dbt will verify that your model's transformation will produce a dataset matching up with its contract, or it will fail to build. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. #### Prerequisites **These places support model contracts:** * `dbt_project.yml` file * `properties.yml` file * SQL models * Models materialized as one of the following: * `table` * `view` — views offer support for column names and data types, but not `constraints` * `incremental` — with `on_schema_change: append_new_columns` or `on_schema_change: fail` * Certain data platforms, but the supported and [enforced `constraints`](https://docs.getdbt.com/reference/resource-properties/constraints.md) vary by platform **These places do *NOT* support model contracts:** * Python models * `materialized view` or `ephemeral` — materialized SQL models * Custom materializations (unless added by the author) * Models with recursive CTE's in BigQuery * Other resource types, such as `sources`, `seeds`, `snapshots`, and so on #### Define a contract Let's say you have a model with a query like: models/marts/dim\_customers.sql ```sql -- lots of SQL final as ( select customer_id, customer_name, -- ... many more ... from ... ) select * from final ``` To enforce a model's contract, set `enforced: true` under the `contract` configuration. When enforced, your contract *must* include every column's `name` and `data_type` (where `data_type` matches one that your data platform understands). If your model is materialized as `table` or `incremental`, and depending on your data platform, you may optionally specify additional [constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md), such as `not_null` (containing zero null values). models/marts/customers.yml ```yaml models: - name: dim_customers config: contract: enforced: true columns: - name: customer_id data_type: int constraints: - type: not_null - name: customer_name data_type: string ... ``` When building a model with a defined contract, dbt will do two things differently: 1. dbt will run a "preflight" check to ensure that the model's query will return a set of columns with names and data types matching the ones you have defined. This check is agnostic to the order of columns specified in your model (SQL) or YAML spec. 2. dbt will include the column names, data types, and constraints in the DDL statements it submits to the data platform, which will be enforced while building or updating the model's table, and order the columns per the contract instead of your dbt model. #### Platform constraint support Select the adapter-specific tab for more information on [constraint](https://docs.getdbt.com/reference/resource-properties/constraints.md) support across platforms. Constraints fall into three categories based on definability and platform enforcement: * **Definable and enforced** — The model won't build if it violates the constraint. * **Definable and not enforced** — The platform supports specifying the type of constraint, but a model can still build even if building the model violates the constraint. This constraint exists for metadata purposes only. This approach is more typical in cloud data warehouses than in transactional databases, where strict rule enforcement is more common. * **Not definable and not enforced** — You can't specify the type of constraint for the platform. ##### Redshift | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### Snowflake | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### BigQuery | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ❌ | ❌ | | check | ❌ | ❌ | ##### Postgres | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ✅ | | foreign\_key | ✅ | ✅ | | unique | ✅ | ✅ | | check | ✅ | ✅ | ##### Spark Currently, `not_null` and `check` constraints are enforced only after a model is built. Because of this platform limitation, dbt considers these constraints definable but not enforced, which means they're not part of the *model contract* since they can't be enforced at build time. This table will change as the features evolve. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ❌ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ✅ | ❌ | ##### Databricks Currently, `not_null` and `check` constraints are enforced only after a model is built. Because of this platform limitation, dbt considers these constraints definable but not enforced, which means they're not part of the *model contract* since they can't be enforced at build time. This table will change as the features evolve. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ❌ | ❌ | | check | ✅ | ✅ | ##### Fabric Microsoft Fabric creates the table first, then adds some constraints in separate steps. Only `not_null` is checked when the model builds. Primary key, foreign key, and unique constraints are recorded on the table for reference, but Fabric does not check them when new data is added. Define `primary_key`, `foreign_key`, and `unique` at the model level. The adapter ignores those constraint types at the column level. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### Athena | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ❌ | ❌ | | primary\_key | ❌ | ❌ | | foreign\_key | ❌ | ❌ | | unique | ❌ | ❌ | | check | ❌ | ❌ | #### FAQs  Which models should have contracts? Any model meeting the criteria described above *can* define a contract. We recommend defining contracts for ["public" models](https://docs.getdbt.com/docs/mesh/govern/model-access.md) that are being relied on downstream. * Inside of dbt: Shared with other groups, other teams, and [other dbt projects](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md). * Outside of dbt: Reports, dashboards, or other systems & processes that expect this model to have a predictable structure. You might reflect these downstream uses with [exposures](https://docs.getdbt.com/docs/build/exposures.md).  How are contracts different from tests? A model's contract defines the **shape** of the returned dataset. If the model's logic or input data doesn't conform to that shape, the model does not build. [Data Tests](https://docs.getdbt.com/docs/build/data-tests.md) are a more flexible mechanism for validating the content of your model *after* it's built. So long as you can write the query, you can run the data test. Data tests are more configurable, such as with [custom severity thresholds](https://docs.getdbt.com/reference/resource-configs/severity.md). They are easier to debug after finding failures because you can query the already-built model, or [store the failing records in the data warehouse](https://docs.getdbt.com/reference/resource-configs/store_failures.md). In some cases, you can replace a data test with its equivalent constraint. This has the advantage of guaranteeing the validation at build time, and it probably requires less compute (cost) in your data platform. The prerequisites for replacing a data test with a constraint are: * Making sure that your data platform can support and enforce the constraint that you need. Most platforms only enforce `not_null`. * Materializing your model as `table` or `incremental` (**not** `view` or `ephemeral`). * Defining a full contract for this model by specifying the `name` and `data_type` of each column. **Why aren't tests part of the contract?** In a parallel for software APIs, the structure of the API response is the contract. Quality and reliability ("uptime") are also very important attributes of an API's quality, but they are not part of the contract per se. When the contract changes in a backwards-incompatible way, it is a breaking change that requires a bump in major version.  Do I need to define every column for a contract? Yes. dbt contracts apply to *all* columns defined in a model, and they require declaring explicit expectations about *all* of those columns. The explicit declaration of a contract is not an accident — it's very much the intent of this feature. At the same time, for models with many columns, we understand that this can mean a *lot* of YAML. See [dbt-labs/dbt#11764](https://github.com/dbt-labs/dbt/issues/11764) for discussion of potential approaches to generate and update model contract definitions.  How are breaking changes handled? When comparing to a previous project state, dbt will look for breaking changes that could impact downstream consumers. If breaking changes are detected, dbt will present a contract error. Breaking changes include: * Removing an existing column * Changing the data\_type of an existing column * Removing or modifying one of the `constraints` on an existing column (dbt v1.6 or higher) * Removing a contracted model by deleting, renaming, or disabling it (dbt v1.9 or higher). * versioned models will raise an error. unversioned models will raise a warning. More details are available in the [contract reference](https://docs.getdbt.com/reference/resource-configs/contract.md#incremental-models-and-on_schema_change).  Can I use model contracts with nested fields? Yes, but only on BigQuery, because nested fields inside [struct columns](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-types#struct_type) are strongly typed. Model contracts are useful when a model contains nested JSON or event data, where a nested field that changes type could break downstream models. Nested fields (structs) are columns inside other columns. List each inner field in YAML with a dotted name, such as `b.id` or `b.double_nested.another.even_more`. You must declare a `name` and `data_type` for every inner field. You may optionally declare the struct column (in this example, `b` or `b.double_nested`) in order to add a description or a constraint on the struct. models/nested\_fields.yml ```yaml models: - name: double_nested_fields config: contract: enforced: true materialized: table columns: - name: a data_type: string - name: b.id data_type: integer - name: b.name data_type: string - name: b.double_nested.id data_type: integer - name: b.double_nested.another.again data_type: string - name: b.double_nested.another.even_more data_type: integer ``` models/double\_nested\_fields.sql ```sql select 'string' as a, struct( 1 as id, 'name' as name, struct(2 as id, struct('test' as again, 2 as even_more) as another) as double_nested ) as b ``` When every nested field's `name` and `data_type` match the SQL, the model builds as usual. If they do not match, the contract fails and the model does not build. For example, if `even_more` is a string in SQL but `integer` in the contract, dbt returns: ```text This model has an enforced contract that failed. Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition. | column_name | definition_type | contract_type | mismatch_reason | | --------------------------------- | --------------- | ------------- | ------------------ | | b.double_nested.another.even_more | STRING | INT64 | data type mismatch | ``` Arrays of structs are supported too. Declare the parent with `data_type: array`, then list each inner field: ```yaml - name: my_array data_type: array - name: my_array.id data_type: integer - name: my_array.name data_type: string ``` To add constraints such as `not_null` on nested fields, refer to [Constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md). --- ### Model notifications dbt platform Set up dbt to notify model owners through email about issues in your deployment environments. Configure dbt to send email notifications to model owners about issues in deployment [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments) as soon as they happen — while the job is still running. Model owners can specify which statuses to receive notifications about: * **Success** and **Fails** for models * **Warning**, **Success**, and **Fails** for tests With model-level notifications, model owners can be the first ones to know about issues before anyone else (like the stakeholders). To be timely and keep the number of notifications to a reasonable amount when multiple models or tests trigger them, dbt observes the following guidelines when notifying the owners: * Send a notification to each unique owner/email during a job run about any models (with status of failure/success) or tests (with status of warning/failure/success). Each owner receives only one notification, the initial one. * No notifications sent about subsequent models or tests while a dbt job is still running. * Each owner/user who subscribes to notifications for one or more statuses (like failure, success, warning) will receive only *one* email notification at the end of the job run. * The email includes a consolidated list of all models or tests that match the statuses the user subscribed to, instead of sending separate emails for each status. Create configuration YAML files in your project for dbt to send notifications about the status of your models and tests in your deployment environments. #### Prerequisites * Your dbt administrator has [enabled the appropriate account setting](#enable-access-to-model-notifications) for you. * Your deployment environment(s) must be on a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) instead of a legacy dbt v1 version. #### Configure groups Define your [groups](https://docs.getdbt.com/docs/build/groups.md) in any `.yml` file in your [models directory](https://docs.getdbt.com/reference/project-configs/model-paths.md). Each group's owner can now specify one or multiple email addresses to receive model-level notifications. The `email` field supports a single email address as a string or a list of multiple email addresses. The following example shows how to define groups in a `groups.yml` file. models/groups.yml ```yml groups: - name: finance owner: # Email is required to receive model-level notifications, additional properties are also allowed. name: "Finance team" email: finance@dbtlabs.com - name: marketing owner: name: "Marketing team" email: marketing@dbtlabs.com config: meta: slack: '#marketing-team' # Example of multiple emails supported - name: documentation team owner: name: "Docs team" email: - docs@dbtlabs.com - community@dbtlabs.com - product@dbtlabs.com config: meta: slack: '#docs-fox' ``` tip The `owner` field supports `name` and `email`, which are required values. Additional arbitrary fields (such as `favorite_food`) are deprecated and will no longer be allowed in a future release. To store additional metadata (like Slack channels, team info, or custom attributes), use `config.meta` instead. #### Attach groups to models Attach groups to models as you would any other config, in either the `dbt_project.yml` or `whatever.yml` files. For example: models/marts.yml ```yml models: - name: sales description: "Sales data model" config: group: finance - name: campaigns description: "Campaigns data model" config: group: marketing ``` By assigning groups in the `dbt_project.yml` file, you can capture all models in a subdirectory at once. In this example, model notifications related to staging models go to the data engineering group, `marts/sales` models to the finance team, and `marts/campaigns` models to the marketing team. dbt\_project.yml ```yml config-version: 2 name: "jaffle_shop" [...] models: jaffle_shop: staging: +group: data_engineering marts: sales: +group: finance campaigns: +group: marketing ``` Attaching a group to a model also encompasses its tests, so you will also receive notifications for a model's test failures. #### Enable access to model notifications Provide dbt account members the ability to configure and receive alerts about issues with models or tests that are encountered during job runs. To use model-level notifications, your dbt account must have access to the feature. Ask your dbt administrator to enable this feature for account members by following these steps: 1. Navigate to **Notification settings** from your profile name in the sidebar (lower left-hand side). 2. From **Email notifications**, enable the setting **Enable group/owner notifications on models** under the **Model notifications** section. Then, specify which statuses to receive notifications about (Success, Warning, and/or Fails). 3. Click **Save**. [![Example of the setting Enable group/owner notifications on models](/img/docs/dbt-platform/example-enable-model-notifications.png?v=2 "Example of the setting Enable group/owner notifications on models")](#)Example of the setting Enable group/owner notifications on models --- ### Model performance dbt platform | Enterprise, Enterprise+ Catalog provides metadata on dbt runs for in-depth model performance and quality analysis. This feature assists in reducing infrastructure costs and saving time for data teams by highlighting where to fine-tune projects and deployments — such as model refactoring or job configuration adjustments. To view model performance in Catalog: 1. Navigate to **Catalog** in the left-hand menu. 2. Select your project in the left sidebar. 3. Click the **Performance** tab. [![Overview of Performance page navigation.](/img/docs/collaborate/dbt-explorer/explorer-model-performance.gif?v=2 "Overview of Performance page navigation.")](#)Overview of Performance page navigation. On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### The Performance overview page You can pinpoint areas for performance enhancement by using the Performance overview page. This page presents a comprehensive analysis across all project models and displays the longest-running models, those most frequently executed, and the ones with the highest failure rates during runs/tests. Data can be segmented by environment and job type which can offer insights into: * Most executed models (total count). * Models with the longest execution time (average duration). * Models with the most failures, detailing run failures (percentage and count) and test failures (percentage and count). Each data point links to individual models in Catalog. [![Example of Performance overview page](/img/docs/collaborate/dbt-explorer/example-performance-overview-page.png?v=2 "Example of Performance overview page")](#)Example of Performance overview page You can view historical metadata for up to the past three months. Select the time horizon using the filter, which defaults to a two-week lookback. [![Example of dropdown](/img/docs/collaborate/dbt-explorer/ex-2-week-default.png?v=2 "Example of dropdown")](#)Example of dropdown #### The Model performance tab The **Model performance** section in Catalog displays historical trends to help you identify optimization opportunities and understand model resource consumption. [![Cost Insights in Catalog](/img/docs/dbt-platform/cost-insights/cost-insights-model.png?v=2 "Cost Insights in Catalog")](#)Cost Insights in Catalog ##### Key metrics The **Model performance** section displays the following metrics that summarize the overall cost and optimization impact for your project: * **Total cost reduction** * **Total % reduction** * **Total query run time deduction** * **Reused assets** (when dbt State or state-aware orchestration is enabled) ##### Filters Use the time period filter to customize the data you want to view: from the last 3 months up to the last 1 week. For **Cost insights**, **Usage**, and **Query run time** tabs, you can set the view granularity by **Daily**, **Weekly**, or **Monthly**. ##### Visualization tabs * **Cost insights**: Shows the estimated warehouse costs incurred by this model and cost reduction from dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed by this model over time. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated query execution time and the reduction in run duration from dbt State or state-aware orchestration. * **Build time**: Shows average execution time for the model and how it trends over the selected period. * **Build count**: Tracks how many times the model was built or reused, including any failures or errors. * **Test results**: Displays test execution outcomes and pass/fail rates for tests on this model. * **Consumption queries**: Shows queries running against this model, helping you understand downstream usage patterns. ##### Table view For **Cost insights**, **Usage**, and **Query run time** tabs, you can access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. ##### Chart interactions For **Build time** and **Build count** tabs: * Click on any data point in the charts to see a detailed table listing all job runs for that day. * Each row in the table provides a direct link to the run details if you want to investigate further. --- ### Model query history dbt platform | Enterprise, Enterprise+ Model query history helps data teams track model usage by analyzing query logs. Model query history allows you to: * View the count of consumption queries for a model based on the data warehouse's query logs. * Provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products. * Enable analysts to find the most popular models used by other people. Model query history is powered by a single consumption query of the query log table in your data warehouse aggregated on a daily basis.  What is a consumption query? Consumption query is a metric of queries in your dbt project that has used the model in a given time. It filters down to `select` statements only to gauge model consumption and excludes dbt model build and test executions. So for example, if `model_super_santi` was queried 10 times in the past week, it would count as having 10 consumption queries for that particular time period. Model query history is supported in the following data warehouses: * Snowflake (Enterprise-tier or higher) * BigQuery * Redshift * Databricks #### Prerequisites To access the features, you should meet the following requirements: * You have a dbt account on an [Enterprise-tier plan](https://www.getdbt.com/pricing/). Single-tenant accounts should contact their account representative for setup. * You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run. * You have [admin permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) in dbt to edit project settings or production environment settings. * You use Snowflake, BigQuery, Redshift, or Databricks as your data warehouse and can enable [query history permissions](#credential-permissions) or work with an admin to do so. * For Snowflake users: You must have a Snowflake Enterprise-tier or higher subscription. #### Enable query history in dbt New and existing production environments * **New production environments:** dbt automatically enables query history when you create a new [production deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) in **Orchestration** (marked **PROD**). This applies to Snowflake, BigQuery, Redshift, and Databricks on [Enterprise-tier plans](https://www.getdbt.com/pricing/). Development environments are not included. * **Existing production environments:** You must enable query history manually in your **PROD** environment settings. Configure the [credential permissions](#credential-permissions) for your warehouse prior to starting the steps in this section. To enable model query history in dbt, follow these steps: 1. Navigate to **Orchestration** and then **Environments**. 2. Select the environment marked **PROD** and click **Settings**. 3. Click **Edit** and scroll to the **Query History** section. 4. Click the **Test Permissions** button to validate the deployment credentials permissions are sufficient to support query history. 5. Click the **Enable query history** box to enable. 6. **Save** your settings. If query history fails to retrieve data, dbt automatically disables it to prevent unintended warehouse costs. * If the failure is temporary (like a network timeout), dbt may retry. * If the problem keeps happening (for example, missing permissions), dbt turns off query history so customers don’t waste warehouse compute. To turn it back on, click **Test Permissions** in **Environment settings**. If the test succeeds, dbt re-enables the environment. [![Enable query history in your environment settings.](/img/docs/collaborate/dbt-explorer/enable-query-history.png?v=2 "Enable query history in your environment settings.")](#)Enable query history in your environment settings. #### Credential permissions This section explains the permissions and steps you need to enable and view model query history in Catalog. The model query history feature uses the credentials in your production environment to gather metadata from your data warehouse’s query logs. This means you may need elevated permissions with the warehouse. Before making any changes to your data platform permissions, confirm the configured permissions in dbt: 1. Navigate to **Orchestration** and then **Environments**. 2. Select the Environment marked **PROD** and click **Settings**. 3. Click **Edit**. 4. Click the sync icon in the **Connection profiles** section. 5. Look at the information under **Deployment credentials**. * Note: Querying query history entails warehouse costs / uses credits. [![Click the sync icon in the Connection profiles section.](/img/docs/dbt-platform/sync-icon.png?v=2 "Click the sync icon in the Connection profiles section.")](#)Click the sync icon in the Connection profiles section. [![Confirm your deployment credentials in your environment settings page.](/img/docs/collaborate/dbt-explorer/model-query-credentials.png?v=2 "Confirm your deployment credentials in your environment settings page.")](#)Confirm your deployment credentials in your environment settings page. 6. Copy or cross reference those credential permissions with the warehouse permissions and grant your user the right permissions. ##### Snowflake model query history Model query history uses metadata tables available to [Snowflake Enterprise-tier](https://docs.snowflake.com/en/user-guide/intro-editions#enterprise-edition) accounts or higher: `QUERY_HISTORY` and `ACCESS_HISTORY`. The Snowflake user in the production environment must have the `GOVERNANCE_VIEWER` permission to view the data. Before enabling model query history, your `ACCOUNTADMIN` must run the following grant statement in Snowflake to ensure access: ```sql GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE <YOUR_DBT_CLOUD_DEPLOYMENT_ROLE>; ``` Without this grant, model query history won't display any data. For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/account-usage#enabling-other-roles-to-use-schemas-in-the-snowflake-database). ##### BigQuery model query history The model query history uses metadata from the [`INFORMATION_SCHEMA.JOBS` view](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs) in BigQuery. To access the metadata, the production environment user must have the correct [IAM role](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.resourceViewer) or permission to access this data: * If you use a BigQuery provided role, we recommend `roles/bigquery.resourceViewer`. * If you use a custom role, ensure it includes the `bigquery.jobs.listAll permission`. ##### Redshift model query history Model query history uses the `SYS_QUERY_HISTORY` and `SYS_QUERY_DETAIL` system views in Redshift. By default, users can only see their own queries in these views. To surface query history across all warehouse users, your database admin must grant the production environment credentials one of the following: * The `sys:monitor` role: ```sql GRANT ROLE sys:monitor TO <YOUR_DBT_USER>; ``` * Unrestricted syslog access on the user: ```sql ALTER USER <YOUR_DBT_USER> SYSLOG ACCESS UNRESTRICTED; ``` Without one of these, model query history won't display data from other users. For more information, refer to the [Redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/cm_chap_system-tables.html#c_visibility-of-data). ###### Redshift considerations Redshift model query history is derived from physical table scans (`SYS_QUERY_DETAIL` where `step_name = 'scan'`). This means usage data reflects a high-quality signal rather than an exact query count. Because Redshift expands regular views at execution time, scans are attributed to underlying base tables and view names do not appear in scan metadata. Materialized views do not have this limitation. Keep the following in mind when reviewing usage data: * Models materialized as `view` will likely show zero usage. * Models materialized as `table` or `incremental` are attributed correctly. * `ephemeral` models cannot be attributed. If your project relies heavily on views, usage may appear lower than expected. This is a known limitation of scan-based attribution rather than missing data. ##### Databricks model query history Model query history uses two Unity Catalog system tables: `system.query.history` and `system.access.table_lineage`. Before granting access, confirm the following prerequisites are met: * Your account has at least one Unity Catalog-enabled workspace. Users who are still on the legacy Hive Metastore cannot use model query history. * The metastore is on Privilege Model Version 1.0. For more information, refer to [Unity Catalog privileges reference](https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference). * The `access` and `query` schemas are enabled in the `system` catalog. A Databricks account admin can enable them under **Account Console** > **Settings** > **System tables**. Grant the service principal used in your production environment the following privileges: ```sql GRANT USE CATALOG ON CATALOG system TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT USE SCHEMA ON SCHEMA system.access TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT USE SCHEMA ON SCHEMA system.query TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT SELECT ON TABLE system.access.table_lineage TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT SELECT ON TABLE system.query.history TO `<YOUR_SERVICE_PRINCIPAL>`; ``` note `SELECT` on the tables alone is not sufficient. Databricks also requires `USE CATALOG` on the parent catalog and `USE SCHEMA` on the parent schemas. For more information, refer to the [Databricks Unity Catalog privileges documentation](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges). ###### Databricks considerations Keep the following in mind when using model query history with Databricks: * **Usage data may be incomplete**. Databricks does not capture lineage for all queries, so some queries may not appear in usage data. This is a known Databricks limitation. For more information, refer to [Databricks lineage system tables](https://docs.databricks.com/aws/en/admin/system-tables/lineage). * **Lineage data is not real-time**. `system.access.table_lineage` is updated throughout the day with no guaranteed timeframe. Delays of up to 24 hours are possible. * **Views may produce inconsistent lineage records**. Materialized views and streaming tables are included in lineage events. Standard views may or may not produce lineage records, depending on how Databricks Unity Catalog resolves them. * **Statement text encryption affects query filtering**. If statement text encryption is enabled in Databricks, `statement_text` in `system.query.history` is `NULL`. In this case, dbt test queries cannot be filtered and may be counted as user queries. * **Serverless compute lineage may be incomplete**. `system.query.history` includes queries from SQL warehouses and serverless compute, but lineage attribution for serverless executions may not always be captured. #### View query history in Catalog To enhance your discovery, you can view your model query history in various locations within Catalog: * [View from Performance charts](#view-from-performance-charts) * [View from Project lineage](#view-from-project-lineage) * [View from Model list](#view-from-model-list) ##### View from Performance charts 1. Navigate to Catalog by clicking **Catalog** in the navigation. 2. In the main **Overview** page, click on **Performance** under the **Project details** section. Scroll down to view the **Most consumed models**. 3. Use the dropdown menu on the right to select the desired time period, with options available for up to the past 3 months. [![View most consumed models on the 'Performance' page in dbt Catalog.](/img/docs/collaborate/dbt-explorer/most-consumed-models.png?v=2 "View most consumed models on the 'Performance' page in dbt Catalog.")](#)View most consumed models on the 'Performance' page in dbt Catalog. 4. Click on a model for more details and go to the **Performance** tab. 5. On the **Performance** tab, scroll down to the **Model performance** section. 6. Select the **Consumption queries** tab to view the consumption queries over a given time for that model. [![View consumption queries over time for a given model.](/img/docs/collaborate/dbt-explorer/model-consumption-queries.png?v=2 "View consumption queries over time for a given model.")](#)View consumption queries over time for a given model. ##### View from Project lineage 1. To view your model in your project lineage, go to the main **Overview page** and click on **Project lineage.** 2. In the lower left of your lineage, click on **Lenses** and select **Consumption queries**. [![View model consumption queries in your lineage using the 'Lenses' feature.](/img/docs/collaborate/dbt-explorer/model-consumption-lenses.png?v=2 "View model consumption queries in your lineage using the 'Lenses' feature.")](#)View model consumption queries in your lineage using the 'Lenses' feature. 3. Your lineage should display a small purple box above each model, indicating the consumption query number. The number for each model represents the model consumption over the last 30 days. ##### View from Model list 1. To view a list of models, go to the main **Overview page**. 2. In the left navigation, go to the **Resources** tab and click on **Models** to view the models list. 3. You can view the consumption query count for the models and sort by most or least consumed. The consumption query number for each model represents the consumption over the last 30 days. [![View models consumption in the 'Models' list page under the 'Consumption' column.](/img/docs/collaborate/dbt-explorer/model-consumption-list.jpg?v=2 "View models consumption in the 'Models' list page under the 'Consumption' column.")](#)View models consumption in the 'Models' list page under the 'Consumption' column. --- ### Model versions Model versions, dbt\_project.yml versions, and .yml versions The word "version" appears in multiple places in docs site and with different meanings: * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) — A dbt Mesh feature that enables better governance and data model management by allowing you to track changes and updates to models over time. * [dbt\_project.yml version](https://docs.getdbt.com/reference/project-configs/version.md#dbt_projectyml-versions)(optional) — `dbt_project.yml` version is unrelated to Mesh and refers to the compatibility of the dbt project with a specific version of dbt. * [.yml property file version](https://docs.getdbt.com/reference/project-configs/version.md#yml-property-file-versions)(optional) — Version numbers within .yml property files inform how dbt parses those YAML files. Unrelated to Mesh. Versioning APIs is a hard problem in software engineering. The root of the challenge is that the producers and consumers of an API have competing incentives: * Producers of an API need the ability to modify its logic and structure. There is a real cost to maintaining legacy endpoints forever, but losing the trust of downstream users is far costlier. * Consumers of an API need to trust in its stability: their queries will keep working, and won't break without warning. Although migrating to a newer API version incurs an expense, an unplanned migration is far costlier. When sharing a final dbt model with other teams or systems, that model is operating like an API. When the producer of that model needs to make significant changes, how can they avoid breaking the queries of its users downstream? Model versioning is a tool to tackle this problem, thoughtfully and head-on. The goal is not to make the problem go away entirely, nor to pretend it's easier or simpler than it is. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. #### Related documentation * [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md) * [`latest_version`](https://docs.getdbt.com/reference/resource-properties/latest_version.md) * [`include` and `exclude`](https://docs.getdbt.com/reference/resource-properties/versions.md#include) * [`ref` with `version` argument](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#versioned-ref) #### Why version a model? If a model defines a ["contract"](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) (a set of guarantees for its structure), it's also possible to change that model's structure in a way that breaks the previous set of guarantees. This could be as obvious as removing or renaming a column, or more subtle, like changing its data type or nullability. One approach is to force every model consumer to immediately handle the breaking change as soon as it's deployed to production. This is actually the appropriate answer at many smaller organizations, or while rapidly iterating on a not-yet-mature set of data models. But it doesn’t scale well beyond that. Instead, for mature models at larger organizations, powering queries inside & outside dbt, the model owner can use **model versions** to: * Test "prerelease" changes (in production, in downstream systems) * Bump the latest version, to be used as the canonical source of truth * Offer a migration window off the "old" version During that migration window, anywhere that model is being used downstream, it can continue to be referenced at a specific version. dbt 1.6 introduced first-class support for **deprecating models** by specifying a [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md). Taken together, model versions and deprecation offer a pathway for model producers to *sunset* old models, and consumers the time to *migrate* across breaking changes. It's a way of managing change across an organization: develop a new version, bump the latest, slate the old version for deprecation, update downstream references, and then remove the old version. There is a real trade-off that exists here—the cost to frequently migrate downstream code, and the cost (and clutter) of materializing multiple versions of a model in the data warehouse. Model versions do not make that problem go away, but by setting a deprecation date, and communicating a clear window for consumers to gracefully migrate off old versions, they put a known boundary on the cost of that migration. #### When should you version a model? By enforcing a model's contract, dbt can help you catch unintended changes to column names and data types that could cause a big headache for downstream queriers. If you're making these changes intentionally, you should create a new model version. If you're making a non-breaking change, you don't need a new version—such as adding a new column, or fixing a bug in an existing column's calculation. Of course, it's possible to change a model's definition in other ways—recalculating a column in a way that doesn't change its name, data type, or enforceable characteristics—but would substantially change the results seen by downstream queriers. This is always a judgment call. As the maintainer of a widely-used model, you know best what's a bug fix and what's an unexpected behavior change. The process of sunsetting and migrating model versions requires real work, and likely significant coordination across teams. You should opt for non-breaking changes whenever possible. Inevitably, however, these non-breaking additions will leave your most important models with lots of unused or deprecated columns. Rather than constantly adding a new version for each small change, you should opt for a predictable cadence (once or twice a year, communicated well in advance) where you bump the "latest" version of your model, removing columns that are no longer being used. #### How is this different from "version control"? [Version control](https://docs.getdbt.com/docs/platform/git/git-version-control.md) allows your team to collaborate simultaneously on a single code repository, manage conflicts between changes, and review changes before deploying into production. In that sense, version control is an essential tool for versioning the deployment of an entire dbt project—always the latest state of the `main` branch. In general, only one version of your project code is deployed into an environment at a time. If something goes wrong, you have the ability to roll back changes by reverting a commit or pull request, or by leveraging data platform capabilities around "time travel." When you make updates to a model's source code — its logical definition, in SQL or Python, or related configuration — dbt can [compare your project to the previous state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection), enabling you to rebuild only models that have changed, and models downstream of a change. In this way, it's possible to develop changes to a model, quickly test in CI, and efficiently deploy into production — all coordinated via your version control system. **Versioned models are different.** Defining model `versions` is appropriate when people, systems, and processes beyond your team's control, inside or outside of dbt, depend on your models. You can neither simply go migrate them all, nor break their queries on a whim. You need to offer a migration path, with clear diffs and deprecation dates. Multiple versions of a model will live in the same code repository at the same time, and be deployed into the same data environment simultaneously. This is similar to how web APIs are versioned: Multiple versions live simultaneously, two or three, and not more). Over time, newer versions come online, and older versions are sunsetted . #### How is this different from just creating a new model? Honestly, it's only a little bit different! There isn't much magic here, and that's by design. You've always been able to copy-paste, create a new model file, and name it `dim_customers_v2.sql`. Why should you opt for a "real" versioned model instead? As the **producer** of a versioned model: * You keep track of all live versions in one place, rather than scattering them throughout the codebase * You can reuse the model's configuration, and highlight just the diffs between versions * You can select models to build (or not) based on whether they're a `latest`, `prerelease`, or `old` version * dbt will notify consumers of your versioned model when new versions become available, or when they are slated for deprecation As the **consumer** of a versioned model: * You use a consistent `ref`, with the option of pinning to a specific live version * You will be notified throughout the life cycle of a versioned model All versions of a model preserve the model's original name. They are `ref`'d by that name, rather than the name of the file that they're defined in. By default, the `ref` resolves to the latest version (as declared by that model's maintainer), but you can also `ref` a specific version of the model, with a `version` keyword. Let's say that `dim_customers` has three versions defined: `v2` is the "latest", `v3` is "prerelease," and `v1` is an old version that's still within its deprecation window. Because `v2` is the latest version, it gets some special treatment: it can be defined in a file without a suffix, and `ref('dim_customers')` will resolve to `v2` if a version pin is not specified. The table below breaks down the standard conventions: | v | version | `ref` syntax | File name | Database relation | | - | ------------ | ---------------------------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------- | | 3 | "prerelease" | `ref('dim_customers', v=3)` | `dim_customers_v3.sql` | `analytics.dim_customers_v3` | | 2 | "latest" | `ref('dim_customers', v=2)` **and** `ref('dim_customers')` | `dim_customers_v2.sql` **or** `dim_customers.sql` | `analytics.dim_customers_v2` **and** `analytics.dim_customers` (recommended) | | 1 | "old" | `ref('dim_customers', v=1)` | `dim_customers_v1.sql` | `analytics.dim_customers_v1` | As you'll see in the implementation section below, a versioned model can reuse the majority of its YAML properties and configuration. Each version needs to only say how it *differs* from the shared set of attributes. This gives you, as the producer of a versioned model, the opportunity to highlight the differences across versions—which is otherwise difficult to detect in models with dozens or hundreds of columns—and to clearly track, in one place, all versions of the model which are currently live. dbt also supports [`version`-based selection](https://docs.getdbt.com/reference/node-selection/methods.md#version). For example, you could define a [default YAML selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md#default) that avoids running any old model versions in development, even while you continue to run them in production through a sunset and migration period. (You could accomplish something similar by applying `tags` to these models, and cycling through those tags over time.) selectors.yml ```yml selectors: - name: exclude_old_versions default: "{{ target.name == 'dev' }}" definition: method: fqn value: "*" exclude: - method: version value: old ``` Because dbt knows that these models are *actually the same model*, it can notify downstream consumers as new versions become available, and as older versions are slated for deprecation. ```bash Found an unpinned reference to versioned model 'dim_customers'. Resolving to latest version: my_model.v2 A prerelease version 3 is available. It has not yet been marked 'latest' by its maintainer. When that happens, this reference will resolve to my_model.v3 instead. Try out v3: {{ ref('my_dbt_project', 'my_model', v='3') }} Pin to v2: {{ ref('my_dbt_project', 'my_model', v='2') }} ``` #### How to create a new version of a model Most often, you'll start with a model that is not yet versioned. Let's go back in time to when `dim_customers` was a simple standalone model, with an enforced contract. For simplicity, let's pretend it has only two columns, `customer_id` and `country_name`, though most mature models will have many more. models/dim\_customers.sql ```sql -- lots of sql final as ( select customer_id, country_name from ... ) select * from final ``` models/schema.yml ```yaml models: - name: dim_customers config: materialized: table contract: enforced: true columns: - name: customer_id description: This is the primary key data_type: int - name: country_name description: Where this customer lives data_type: varchar ``` Let's say you need to make a breaking change to the model: Removing the `country_name` column, which is no longer reliable. First, create a new model file (SQL or Python) encompassing those breaking changes. The default convention is naming the new file with a `_v<version>` suffix. Let's make a new file, named `dim_customers_v2.sql`. (We don't need to rename the existing model file just yet, while it's still the "latest" version.) models/dim\_customers\_v2.sql ```sql -- lots of sql final as ( select customer_id -- country_name has been removed! from ... ) select * from final ``` Now, you could define properties and configuration for `dim_customers_v2` as a new standalone model, with no actual relation to `dim_customers` save a striking resemblance. Instead, we're going to declare that these are versions of the same model, both named `dim_customers`. We can define their properties in common, and then **just** highlight the diffs between them. (Or, you can choose to define each model version with full specifications, and repeat the values they have in common.) ##### Diffs only (recommended) models/schema.yml ```yaml models: - name: dim_customers latest_version: 1 config: materialized: table contract: {enforced: true} columns: - name: customer_id description: This is the primary key data_type: int - name: country_name description: Where this customer lives data_type: varchar # Declare the versions, and highlight the diffs versions: - v: 1 # Matches what's above -- nothing more needed - v: 2 # Removed a column -- this is the breaking change! columns: # This means: use the 'columns' list from above, but exclude country_name - include: all exclude: [country_name] ``` ##### Fully specified models/schema.yml ```yaml models: - name: dim_customers latest_version: 1 # declare the versions, and fully specify them versions: - v: 2 config: materialized: table contract: {enforced: true} columns: - name: customer_id description: This is the primary key data_type: int # no country_name column - v: 1 config: materialized: table contract: {enforced: true} columns: - name: customer_id description: This is the primary key data_type: int - name: country_name description: Where this customer lives data_type: varchar ``` Note: If none of your model versions specify columns, you don't need to define columns at all and can omit the `columns/include`/`exclude` keys from the versioned model. In this case, dbt will automatically use all top-level columns for all versions. The configuration above says: Instead of two unrelated models, I have two versioned definitions of the same model: `dim_customers_v1` and `dim_customers_v2`. **Where are they defined?** dbt expects each model version to be defined in a file named `<model_name>_v<v>`. In this case: `dim_customers_v1.sql` and `dim_customers_v2.sql`. It's also possible to define the "latest" version in `dim_customers.sql` (no suffix), without additional configuration. Finally, you can override this convention by setting [`defined_in: any_file_name_you_want`](https://docs.getdbt.com/reference/resource-properties/versions.md#defined_in)—but we strongly encourage you to follow the convention, unless you have a very good reason. **Where will they be materialized?** Each model version will create a database relation with alias `<model_name>_v<v>`. In this case: `dim_customers_v1` and `dim_customers_v2`. See [the section below](#configuring-database-location-with-alias) for more details on configuring aliases. **Which version is "latest"?** If not specified explicitly, the `latest_version` would be `2`, because it's numerically greatest. In this case, we've explicitly specified that `latest_version: 1`. That means `v2` is a "prerelease," in early development and testing. When we're ready to roll out `v2` to everyone by default, we would bump `latest_version: 2`, or remove `latest_version` from the specification. ##### Configuring versioned models You can reconfigure each version independently. For example, you could materialize `v2` as a table and `v1` as a view: models/schema.yml ```yml versions: - v: 2 config: materialized: table - v: 1 config: materialized: view ``` Like with all config inheritance, any configs set *within* the versioned model's definition (`.sql` or `.py` file) will take precedence over the configs set in YAML. ##### Configuring database location with `alias` Following the example, let's say you wanted `dim_customers_v1` to continue populating the database table named `dim_customers`. That's what the table was named previously, and you may have several other dashboards or tools expecting to read its data from `<dbname>.<schemaname>.dim_customers`. You could use the `alias` configuration: models/schema.yml ```yml - v: 1 config: alias: dim_customers # keep v1 in its original database location ``` (Applies to dbt v1.12 and later) If you want a view that always tracks the latest model version instead of pinning to a specific one, see [Pointing to the latest version](#pointing-to-the-latest-version). ##### Pointing to the latest version The [`latest_version_pointer`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) config automatically creates a view named after the model's base name (for example, `dim_customers`) that always points to the latest versioned relation (for example, `dim_customers_v2`). When you enable it, querying outside of dbt always returns the current version. This config only applies to versioned models. Enable this feature in your project by setting the [`latest_version_pointer_enabled_by_default`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) flag to `true` in `dbt_project.yml`, or enable it per model with the `latest_version_pointer.enabled` config: ##### Enable globally dbt\_project.yml ```yaml flags: latest_version_pointer_enabled_by_default: true ``` ##### Enable per model models/schema.yml ```yaml models: - name: dim_customers versions: - v: 1 - v: 2 config: latest_version_pointer: enabled: true # overrides the project flag when set alias: dim_customers_current # optional custom name ``` The pointer view uses the model's base name by default (for example, `dim_customers`). You can override the alias per model with `latest_version_pointer.alias`, or globally by overriding the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro in your project. ###### Naming collisions To prevent naming collisions, dbt raises a `dbt1005` error if the latest version's alias is the same as the pointer view name. In dbt v2, where `latest_version_pointer` is enabled by default, this error can surface on models that have an explicit `alias` matching the model's base name, even if you never configured the pointer yourself. For example, the following configuration would raise `dbt1005` because both `dim_customers_v2` and the pointer view would resolve to `dim_customers`: ```text dbt1005 (Cannot create latest version pointer: the latest version of 'dim_customers' is already aliased to 'dim_customers') ``` ```yaml models: - name: dim_customers versions: - v: 1 - v: 2 config: alias: dim_customers # collides with the pointer view name config: latest_version_pointer: enabled: true ``` To fix this, select one of the following options: * [Remove the `alias` (recommended)](#remove-the-alias-recommended) * [Disable the latest version pointer for that model](#disable-the-latest-version-pointer-for-that-model) * [Set a unique `alias`](#set-a-unique-alias) * [Override the `generate_latest_version_pointer_alias` macro](#override-the-generate_latest_version_pointer_alias-macro) ###### Remove the `alias` (recommended) Remove the `alias` from the latest version and let the automatic pointer handle it: ```yaml config: alias: dim_customers ``` ###### Disable the latest version pointer for that model This approach is immediately backward-compatible for pre-existing `alias` configs: ```yaml config: alias: dim_customers config: latest_version_pointer: enabled: false ``` ###### Set a unique `alias` ```yaml config: alias: dim_customers_latest ``` ###### Override the `generate_latest_version_pointer_alias` macro Override the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro to use a different naming convention globally: macros/generate\_latest\_version\_pointer\_alias.sql ```sql {% macro generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {{ node.name ~ "_latest" }} {%- endmacro %} ``` ##### Run a model with multiple versions To run a model with multiple versions, you can use the [`--select` flag](https://docs.getdbt.com/reference/node-selection/syntax.md). For example: * Run all versions of `dim_customers`: ```bash dbt run --select dim_customers # Run all versions of the model ``` * Run only version 2 of `dim_customers`: You can use either of the following commands (both achieve the same result): ```bash dbt run --select dim_customers.v2 # Run a specific version of the model dbt run --select dim_customers_v2 # Alternative syntax for the specific version ``` * Run the latest version of `dim_customers` using the `--select` flag shorthand: ```bash dbt run -s dim_customers,version:latest # Run the latest version of the model ``` These commands provide flexibility in managing and executing different versions of a dbt model. ##### Optimizing model versions How you define each model version is completely up to you. While it's easy to start by copy-pasting from one model's SQL definition into another, you should think about *what actually is changing* from one version to another. For example, if your new model version is only renaming or removing certain columns, you could define one version as a view on top of the other one: models/dim\_customers\_v2.sql ```sql {{ config(materialized = 'view') }} {% set dim_customers_v1 = ref('dim_customers', v=1) %} select {{ dbt_utils.star(from=dim_customers_v1, except=["country_name"]) }} from {{ dim_customers_v1 }} ``` Of course, if one model version makes meaningful and substantive changes to logic in another, it may not be possible to optimize it in this way. At that point, the cost of human intuition and legibility is more important than the cost of recomputing similar transformations. We expect to develop more opinionated recommendations as teams start adopting model versions in practice. One recommended pattern we can envision: Prioritize the definition of the `latest_version`, and define other versions (old and prerelease) based on their diffs from the latest. How? * Define the properties and configuration for the latest version in the top-level model YAML, and the diffs for other versions below (via `include`/`exclude`) * Where possible, define other versions as `select` transformations, which take the latest version as their starting point * When bumping the `latest_version`, migrate the SQL and YAML accordingly. In the example above, the third point might be tricky. It's easier to *exclude* `country_name`, than it is to add it back in. Instead, we might need to keep around the full original logic for `dim_customers_v1`—but materialize it as a `view`, to minimize the data warehouse cost of building it. If downstream queriers see slightly degraded performance, it's still significantly better than broken queries, and all the more reason to migrate to the new "latest" version. #### Coordinate model versioning Safely releasing a new model version requires coordination between model producers (who build the models) and model consumers (who depend on them). For practical guidance on how producers and consumers should communicate, test, and roll out versioned models across projects, refer to [Coordinating model versions best practices](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-6-coordinate-versions.md). --- ### Models and pricing Learn how dbt Wizard usage is metered and how model access works across the dbt platform and local CLI. This page covers which AI models dbt Wizard can use and how usage is priced. To start a trial, add a credit card, or set a spend limit, refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). Get started with free dbt-managed usage credits. Every credit balance is *per account, not per user*, which means everyone on your account shares the same pool. Enterprise and Enterprise+ accounts get monthly usage credits, and all other plans get a 30-day free trial with $100 in credits. There are two ways to pay for AI usage: let dbt Labs handle the models and the billing, or bring your own provider key. Here's how they compare: | | [dbt managed](#dbt-managed-providers) | [Bring your own key (BYOK)](#bring-your-own-key-byok) | | ------------------ | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | **Choose it when** | You want to start now, with no AI provider account of your own | You already have provider credits, or your company requires a specific vendor | | **Who bills you** | dbt Labs, through your dbt account | Your AI provider, directly | | **Spend controls** | Consumption pool and spend limits in **Billing & Usage** | Managed with your provider | | **Setup** | Nothing to configure | [Configure a provider key](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) | Both options are metered per token. For dbt managed rates, refer to the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model). With BYOK, you pay whatever your provider charges. Most users start with the dbt managed model and only reach for BYOK or a specific managed frontier provider when they need it. #### Key terms These four terms come up on every dbt Wizard billing page: | Term | What it means | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Token | The unit of AI model usage. How many tokens you use, and the rate you pay for them, depends on the model, the context window, the length and complexity of your prompt, and the size of the response | | Usage credits | The free balance included with your plan, shared by everyone on the account, whether that's trial credits on Developer and Starter, or monthly credits on Enterprise and Enterprise+. Scoped to dbt Wizard only | | Consumption pool | Your overall dbt managed usage balance, including any pool you purchase once your credits run out. A purchased pool covers both dbt Wizard and dbt State. Shown as **Consumption pool** in **Billing & Usage** | | Spend limit | The maximum dbt managed usage your account can consume in a billing period | Usage draws down from your credits first, then your consumption pool, and stops at your spend limit. Usage costs are passed through directly from the AI provider. #### dbt managed providers In the dbt platform, dbt Wizard offers a managed OpenAI model by default. dbt Labs also offers managed access to Anthropic and to open weight models. You'll see two kinds of models across these pages: * **Frontier models** are the flagship models hosted by providers like OpenAI and Anthropic. They're the most capable and the most expensive per token. * **Open weight models**, such as DeepSeek and Kimi, are openly published models available through dbt Labs. They offer a strong balance of capability and cost compared to frontier models. Refer to the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Supported AI providers](#supported-ai-providers) for the full list of what's available where. #### Bring your own key (BYOK) With BYOK, you supply your own credentials for a provider — OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, Snowflake Cortex, or Databricks. BYOK in dbt platform is available on Starter, Enterprise, and Enterprise+ plans. To set up BYOK, refer to [Configure BYOK for dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [Configure BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md). #### Supported AI providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). #### Choose a model With dbt managed inference, you can switch between the supported managed models at any using the model picker dropdown next to the **Agent mode** control (where you choose **Ask for approval** or **Edit files automatically**). * In [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) and the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), open the model picker dropdown next to the **Agent mode** control in the dbt Wizard panel, then select a model. * The picker lists the managed models available to you. Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the available models and their token rates. * If you [bring your own key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md), dbt Wizard uses the provider and model you configured with your key rather than the managed model picker. [![The model picker dropdown next to the Agent mode control in the Wizard panel.](/img/docs/dbt-platform/wizard-model-picker.png?v=2 "The model picker dropdown next to the Agent mode control in the Wizard panel.")](#)The model picker dropdown next to the Agent mode control in the Wizard panel. #### Related docs * [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for trials, spend limits, and paid access * [Billing](https://docs.getdbt.com/docs/platform/billing.md) for general dbt platform billing * [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for per-model token rates * [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) --- ### Monitor dbt State activity Login required | Usage-based Learn how to monitor dbt State activity in dbt platform for better visibility into model builds and cost savings. dbt State monitoring helps you: * **Track effectiveness of dbt State** — See how dbt State reduces unnecessary model rebuilds by only building models when there are changes to the data or code. dbt State provides transparency into how the optimization works across your projects. * **Analyze build patterns** — Gain insights into your project's build frequency and identify opportunities for further optimization. #### dbt State metrics When you go to **Account settings** > **Billing & Usage** > **Usage-based features**, the **State** tab shows how many days remain in your trial period. Once dbt State is enabled, it displays the following for the current month: * **Models reused this month**: How many model builds dbt State skipped or cloned instead of rebuilding from scratch. * **Total % build reduction**: The overall reduction in model builds across your account. * **Total query run time reduction**: The total time dbt State saved by not executing unnecessary model builds. The **State** tab also displays the following charts: * **DATT** — Shows the target tables processed by dbt State, split into **Billable** and **Free**. Daily active target tables (DATTs) are the [billable units](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md#daily-active-target-tables) for dbt State. During a trial, all DATTs are counted as free. * **Asset builds** — Shows all model builds for the month, including models reused and cloned. #### Lag tolerance recommendations The **dbt State** page, which you can access from the left-side menu of the dbt platform, includes a **Lag tolerance recommendations** section that identifies models that could safely tolerate more lag, letting dbt State skip more runs and save additional compute. This is available to all dbt State users, including those running dbt locally, through CI/CD, or through an external orchestrator. The recommendations table displays the following columns: | Column | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Model name** | The name of the model that could benefit from a higher `lag_tolerance` value. | | **Project** | The dbt project the model belongs to, as set by `name:` in `dbt_project.yml`. This may differ from the project name in the dbt platform. | | **Current lag** | The model's current `lag_tolerance` setting. | | **Recommended lag** | The `lag_tolerance` value dbt State recommends based on your model's 30-day build history. | | **% time saved** | The estimated percentage of build time you'd save by applying the recommended `lag_tolerance`. | | **Projected 30d time savings** | The estimated build time you could save over the next 30 days by applying the recommended `lag_tolerance`, based on redundant builds in the previous 30 days. This estimate includes only this model, so actual savings may be higher if downstream models also do not rebuild. | You can search for a specific model using the search bar, or filter recommendations by project using the **Project** dropdown menu. To apply a recommendation, update the model's `lag_tolerance` config. For configuration syntax and examples, refer to the [`lag_tolerance` config page](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). ##### How dbt State calculates recommendations dbt State analyzes each model’s build history from the previous 30 days. Models with fewer than 10 recorded builds are excluded because there isn't enough history to make a reliable recommendation. For each eligible model, dbt State: 1. Identifies builds where the model’s definition and inputs had not changed since the previous build. 2. Estimates the build time that different `lag_tolerance` values would have saved. 3. Recommends the smallest value that would have saved more than 30 minutes, helping reduce redundant builds while keeping data as fresh as possible. A model doesn’t appear in the table if it’s a view or its current `lag_tolerance` value is already equal to or greater than the recommended value. Each account displays 20 models with the highest projected savings. [![Lag tolerance recommendations](/img/docs/dbt-platform/using-dbt-platform/lag-tolerance-recommendations.png?v=2 "Lag tolerance recommendations")](#)Lag tolerance recommendations #### Models built and reused chart When you go to your **Account home**, you'll see a chart showing the number of models built and reused, giving you visibility into how dbt State is optimizing your data builds. You can also view the number of reused models per project on **Account home**. #### Logs view of built resources When you run a job, or when you run `dbt run` or `dbt build` locally, a structured logs view shows which resources were built, skipped, or reused. [![Logs view of built resources](/img/docs/dbt-platform/using-dbt-platform/sao-logs-view.png?v=2 "Logs view of built resources")](#)Logs view of built resources 1. Each resource has an icon indicating its status. 2. The **Reused** tab indicates the total number of reused resources. 3. You can use the search bar or filter the logs to show **All**, **Success**, **Warning**, **Failed**, **Running**, **Skipped**, **Reused**, or **Debugged** messages. 4. Detailed log messages provide context on why resources were built, reused, or skipped. These messages are highlighted in the logs. #### Reused tag in the Latest status lens Lineage lenses are interactive visual filters in [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md#lenses) that show additional context on your lineage graph to understand how resources are defined or performing. When you apply a lens, tags become visible on the nodes in the lineage graph, indicating the layer value along with coloration based on that value. If you're significantly zoomed out, only the tags and their colors are visible in the graph. The **Latest status** lens shows the status from the latest execution of the resource in the current environment. When you use this lens to view your lineage, dbt State tags reused models with **Reused**. [![Latest status lens showing reused models](/img/docs/dbt-platform/using-dbt-platform/sao-latest-status-lens.png?v=2 "Latest status lens showing reused models")](#)Latest status lens showing reused models To view your lineage with the **Latest status** lens: 1. From the main menu, go to **Orchestration** > **Runs**. 2. Select your run. 3. Go to the **Lineage** tab. You'll see your project's lineage. 4. In the **Lenses** field, select **Latest status**. #### Explain tab To see why dbt State rebuilt, reused, or cloned a specific resource, go to **Orchestration** > **Runs**. Select a run and go to the **Explain** tab. The **Explain** tab appears on job runs that used [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). It shows why dbt State rebuilt, reused, or cloned each resource, so you can investigate unexpected behavior or verify that State is working as expected. The tab is available while a run is in progress and updates as resources finish. The tab displays an **Explain results** table with one row per resource. You can search by resource name and download the full results as a text file. Expand a row to see the full decision details. Not all analyses apply to every resource type: | Field | Description | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Resource name** | The name of the resource. | | **Resource type** | The resource type: model, seed, snapshot, test, and so on. | | **Decision** | The reason dbt State rebuilt, reused, or cloned this resource. | | **Run step** | The job command that ran this resource (for example, `dbt build --exclude tag:ml_pipeline`). | | **Table analysis** | Whether the target table already exists in the schema. | | **Query analysis** | Whether the resource query or its upstream queries have changed. | | **Data freshness analysis** | Whether upstream data is fresh or within the configured [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). | [![Explain tab showing the decision breakdown](/img/docs/dbt-platform/deployment/explain-tab.png?v=2 "Explain tab showing the decision breakdown")](#)Explain tab showing the decision breakdown #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [`lag_tolerance` config reference](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### Monitor jobs and alerts dbt platform Monitor your dbt jobs to help identify improvement and set up alerts to proactively alert the right people or team. This portion of our documentation will go over dbt's various capabilities that help you monitor your jobs and set up alerts to ensure seamless orchestration, including: * [Visualize and orchestrate downstream exposures](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") — Automatically visualize and orchestrate exposures from dashboards and proactively refresh the underlying data sources during scheduled dbt jobs. * [Leverage artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) — dbt generates and saves artifacts for your project, which it uses to power features like creating docs for your project and reporting freshness of your sources. * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) — Receive email, Slack, or Microsoft Teams notifications when a job run succeeds, encounters warnings, fails, or is canceled. * [Model notifications](https://docs.getdbt.com/docs/deploy/model-notifications.md) — Receive email notifications about any issues encountered by your models and tests as soon as they occur while running a job. * [Retry jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) — Rerun your errored jobs from start or the failure point. * [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) — View your run history to help identify where improvements can be made to scheduled jobs. * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) — Monitor data governance by enabling snapshots to capture the freshness of your data sources. * [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) — Use webhooks to send events about your dbt jobs' statuses to other systems. To set up and add data health tiles to view data freshness and quality checks in your dashboard, refer to [data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md). [![An overview of a dbt job run which contains run summary, job trigger, run duration, and more.](/img/docs/dbt-platform/deployment/deploy-scheduler.png?v=2 "An overview of a dbt job run which contains run summary, job trigger, run duration, and more.")](#)An overview of a dbt job run which contains run summary, job trigger, run duration, and more. [![Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.](/img/docs/dbt-platform/deployment/run-history.png?v=2 "Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.")](#)Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more. [![Access logs for run steps](/img/docs/dbt-platform/deployment/access-logs.gif?v=2 "Access logs for run steps")](#)Access logs for run steps --- ### Navigate the dbt Insights interface dbt platform | Enterprise, Enterprise+ Learn how to navigate Insights interface and use the main components. Insights provides an interactive interface for writing, running, and analyzing SQL queries. This section highlights the main components of Insights. #### Query console The query console is the main component of Insights. It allows you to write, run, and analyze SQL queries. The Query console supports: 1. Query console editor, which allows you to write, run, and analyze SQL queries: * It supports syntax highlighting and autocomplete suggestions * Hyperlink from SQL code `ref` to the corresponding Explorer page 2. [Query console menu](#query-console-menu), which contains **Bookmark (icon)**, **Develop**, and **Run** buttons. 3. [Query output panel](#query-output-panel), below the query editor and displays the results of a query: * Has three tabs: **Data**, **Chart**, and **Details**, which allow you to analyze query execution and visualize results. 4. [Query console sidebar menu](#query-console-sidebar-menu), which contains the **Catalog**, **Bookmark**, **Query history**, and **dbt Copilot** icons. [![dbt Insights main interface with blank query editor](/img/docs/dbt-insights/insights-main.png?v=2 "dbt Insights main interface with blank query editor")](#)dbt Insights main interface with blank query editor ##### Query console menu The Query console menu is located at the top right of the Query editor. It contains the **Bookmark**, **Develop**, and **Run** buttons: * **Bookmark** button — Save your frequently used SQL queries as favorites for easier access. * When you click **Bookmark**, a **Bookmark Query Details** modal (pop up box) will appear where you can add a **Title** and **Description**. * Let dbt Copilot do the writing for you — use the AI assistant to automatically generate a helpful description for your bookmark. * Access the newly created bookmark from the **Bookmark** icon in the [Query console sidebar menu](#query-console-sidebar-menu). * **Develop**: Open the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to continue editing your SQL query. * **Run** button — Run your SQL query and view the results in the **Data** tab. #### Semantic Layer querying The Semantic Layer querying in dbt Insights lets you build queries against the Semantic Layer without writing SQL code. It guides you in creating queries based on available metrics, dimensions, and entities. With this feature, you can: * Build analyses from your predefined semantic layer metrics. * Have filters, time ranges, and aggregates tailored to the semantic model. * View the underlying SQL code for each metric query. To build a query in dbt Insights: 1. From the main menu, go to **Insights**. 2. Click **Build a query**. 3. From the Environment dropdown, select **PROD** or **STG**. Even thought you see **DEV** as an option, queries against development environments are deferred based on deferral rules, so selecting it will not query your development environment directly. 4. Select what you want to include in your query. * Click **Add Metric** to select the metrics for your query. * Click **Add Group by** to choose the dimensions that break down your metric, such as time grain (day, week, month), region, product, or customer. * Click **Add Filter** to create a filter to narrow your results. * Click **Add Order by** to select how you want to sort the results of your query. * Click **Add Limit**, select the amount of results you want to see when you run your query. If left blank, you will get all results. 5. Click **Run** to run your query. Results are available in the **Data** tab. You can see the SQL code generated in the **Details** tab. [![Semantic Layer querying within dbt Insights](/img/docs/dbt-insights/insights-query-builder-interface.png?v=2 "Semantic Layer querying within dbt Insights")](#)Semantic Layer querying within dbt Insights [![Results are displayed in the Data tab](/img/docs/dbt-insights/insights-query-builder.png?v=2 "Results are displayed in the Data tab")](#)Results are displayed in the Data tab [![The generated SQL code in the Details tab](/img/docs/dbt-insights/insights-query-builder-sql.png?v=2 "The generated SQL code in the Details tab")](#)The generated SQL code in the Details tab #### Query output panel The Query output panel is below the query editor and displays the results of a query. It displays the following tabs to analyze query execution and visualize results: * **Data** tab — Preview your SQL results, with results paginated. * **Details** tab — Generates succinct details of executed SQL query: * Query metadata — dbt Copilot's AI-generated title and description. Along with the supplied SQL and compiled SQL. * Connection details — Relevant data platform connection information. * Query details — Query duration, status, column count, row count. * **Chart** tab — Visualizes query results with built-in charts. * Use the chart icon to select the type of chart you want to visualize your results. Available chart types are **line chart, bar chart, or scatterplot**. * Use the **Chart settings** to customize the chart type and the columns you want to visualize. * Available chart types are **line chart, bar chart, or scatterplot**. * **Download** button — Allows you to export the results to CSV [![dbt Insights Data tab](/img/docs/dbt-insights/insights-chart-tab.png?v=2 "dbt Insights Data tab")](#)dbt Insights Data tab [![dbt Insights Chart tab](/img/docs/dbt-insights/insights-chart.png?v=2 "dbt Insights Chart tab")](#)dbt Insights Chart tab [![dbt Insights Details tab](/img/docs/dbt-insights/insights-details.png?v=2 "dbt Insights Details tab")](#)dbt Insights Details tab #### Query console sidebar menu The Query console sidebar menu and icons contains the following options: ##### dbt Catalog **Catalog icon** — View your project's models, columns, metrics, and more using the integrated Catalog view. [![dbt Insights dbt Catalog icon](/img/docs/dbt-insights/insights-explorer.png?v=2 "dbt Insights dbt Catalog icon")](#)dbt Insights dbt Catalog icon ##### Bookmark Save and access your frequently used queries from the query console sidebar. To save and manage bookmarks: 1. Click **Bookmark** in the [Query console menu](#query-console-menu) to save the current query. In the **Bookmark Query Details** modal, add a **Title** and **Description**. 2. Select the **Bookmark** icon in the sidebar to open your list of saved queries. 3. Select a saved Insight to view its details or open it in the query editor. For sharing saved Insights and version history, check out [Save your Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md#save-your-insights). [![Bookmark Query Details modal](/img/docs/dbt-insights/save-insights-modal.png?v=2 "Bookmark Query Details modal")](#)Bookmark Query Details modal [![Manage your query bookmarks](/img/docs/dbt-insights/manage-bookmarks.png?v=2 "Manage your query bookmarks")](#)Manage your query bookmarks [![List view of saved Insights](/img/docs/dbt-insights/view-saved-insights.png?v=2 "List view of saved Insights")](#)List view of saved Insights ##### Query history View past queries, their statuses (All, Success, Error, or Pending), start time, and duration. Search for past queries and filter by status. You can also re-run a query from the Query history. [![dbt Insights Query history icon](/img/docs/dbt-insights/insights-query-history.png?v=2 "dbt Insights Query history icon")](#)dbt Insights Query history icon ##### dbt Copilot dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. Use dbt Copilot to modify or generate queries using natural language prompts or to chat with the Analyst agent to gather insights about your data. Use dbt Copilot in Insights in two ways: * **Generate SQL**: Build queries in Insights with natural language prompts to explore and query data with an intuitive, context-rich interface. For more information, see [Build queries in dbt Insights](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md). * **dbt Copilot Analyst agent** [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles"): Ask questions to get intelligent data analysis with automated workflows, governed insights, and actionable recommendations. This is a conversational AI feature where you can ask natural language prompts and receive analysis in real-time. Enable beta features under **Account settings** > **Personal profile** > **Experimental features** to use the Analyst agent. For steps on how to enable, see [Preview new dbt platform features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md). Some sample questions you can ask the agent: * *What region are my sales growing the fastest?* * *What was the revenue last month?* * *How should I optimize my marketing spend next quarter?* * *How many customers do I have, broken down by customer type?* dbt Copilot Analyst agent creates an analysis plan based on your question. dbt Copilot Analyst agent: 1. Gets context using your semantic models and metrics. 2. Generates SQL queries using your project's definitions. 3. Executes the SQL query and returns results with context. 4. Reviews and summarizes the generated insights and provides a comprehensive answer. The agent can loop through these steps multiple times if it hasn't reached a complete answer, allowing for complex, multi-step analysis.⁠ For more information, see [dbt Copilot in Insights](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md). [![dbt Copilot in Insights](/img/docs/dbt-insights/insights-copilot-tabs.png?v=2 "dbt Copilot in Insights")](#)dbt Copilot in Insights #### LSP features The following Language Server Protocol (LSP) features are available for projects upgraded to dbt v2: * **Live CTE previews:** Preview a CTE’s output for faster validation and debugging. [![Preview CTE in Insights](/img/docs/dbt-insights/preview-cte.png?v=2 "Preview CTE in Insights")](#)Preview CTE in Insights * **Real-time error detection:** Automatically validate your SQL code to detect errors and surface warnings, without hitting the warehouse. This includes both dbt errors (like invalid `ref`) and SQL errors (like invalid column name or SQL syntax). [![Live error detection](/img/docs/dbt-insights/sql-validation.png?v=2 "Live error detection")](#)Live error detection * **`ref` suggestions:** Autocomplete model names when using the `ref()` function to reference other models in your project. [![ref suggestions in Insights](/img/docs/dbt-insights/ref-autocomplete.png?v=2 "ref suggestions in Insights")](#)ref suggestions in Insights * **Hover insights:** View context on tables, columns, and functions without leaving your code. Hover over any SQL element to see details like column names and data types. [![Sample column details](/img/docs/dbt-insights/column-info.png?v=2 "Sample column details")](#)Sample column details [![Sample column details](/img/docs/dbt-insights/column-hover.png?v=2 "Sample column details")](#)Sample column details --- ### Navigating the state-aware interface dbt platform Learn how to navigate the state-aware orchestration interface for better visibility into model builds and cost tracking. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). #### Models built and reused chart When you go to your **Account home**, you'll see a chart showing the number of models built and reused, giving you visibility into how state-aware orchestration is optimizing your data builds. This chart helps you to: * **Track effectiveness of state-aware orchestration** — See how state-aware orchestration reduces unnecessary model rebuilds by only building models when there are changes to the data or code⁠. This chart provides transparency into how the optimization is working across your dbt implementation. * **Analyze build patterns** — Gain insights into your project's build frequency and identify opportunities for further optimization. You can also view the number of reused models per project in the **Accounts home** page. [![Models built and reused chart in Account home](/img/docs/dbt-platform/using-dbt-platform/account-home-chart.png?v=2 "Models built and reused chart in Account home")](#)Models built and reused chart in Account home [![View reused models count per project in the Accounts home page](/img/docs/deploy/sao-model-reuse.png?v=2 "View reused models count per project in the Accounts home page")](#)View reused models count per project in the Accounts home page #### Model consumption view in jobs State-aware jobs provide charts that show information about your job runs, and how many models were built and reused by your job in the past week, in the last 14 days, or in the last 30 days. In the **Overview** section of your job, the following charts are available: Under the **Runs** tab: * **Recent runs** * **Total run duration time** [![Charts for Recent runs and Total run duration time](/img/docs/dbt-platform/using-dbt-platform/sao-runs-chart.png?v=2 "Charts for Recent runs and Total run duration time")](#)Charts for Recent runs and Total run duration time Under the **Models** tab: * **Models built** * **Models reused** [![Charts for Models built and Models reused](/img/docs/dbt-platform/using-dbt-platform/sao-models-chart.png?v=2 "Charts for Models built and Models reused")](#)Charts for Models built and Models reused #### Logs view of built models When running a job, a structured logs view shows which models were built, skipped, or reused. [![Logs view of built models](/img/docs/dbt-platform/using-dbt-platform/sao-logs-view.png?v=2 "Logs view of built models")](#)Logs view of built models 1. Each model has an icon indicating its status. 2. The **Reused** tab indicates the total number of reused models. 3. You can use the search bar or filter the logs to show **All**, **Success**, **Warning**, **Failed**, **Running**, **Skipped**, **Reused**, or **Debugged** messages. 4. Detailed log messages are provided to get more context on why models were built, reused, or skipped. These messages are highlighted in the logs. #### Reused tag in the Latest status lens Lineage lenses are interactive visual filters in [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md#lenses) that show additional context on your lineage graph to understand how resources are defined or performing. When you apply a lens, tags become visible on the nodes in the lineage graph, indicating the layer value along with coloration based on that value. If you're significantly zoomed out, only the tags and their colors are visible in the graph. The **Latest status** lens shows the status from the latest execution of the resource in the current environment. When you use this lens to view your lineage, models that were reused from state-aware orchestration are tagged with **Reused**. [![Latest status lens showing reused models](/img/docs/dbt-platform/using-dbt-platform/sao-latest-status-lens.png?v=2 "Latest status lens showing reused models")](#)Latest status lens showing reused models To view your lineage with the **Latest status** lens: 1. From the main menu, go to **Orchestration** > **Runs**. 2. Select your run. 3. Go to the **Lineage** tab. The lineage of your project appears. 4. In the **Lenses** field, select **Latest status**. #### Clear cache button State-aware orchestration uses a cached hash of both code and data state for each model in an environment stored in Redis. When running a job, dbt checks if there are changes in the hash for the model being built between the saved state in Redis and the current state that would be built by the job. If there is a change, dbt builds the model. If there are no changes, dbt reuses the model from the last time it was built. * To wipe this state clean and start again, clear the cache by going to **Orchestration** > **Environments**. Select your environment and click the **Clear cache** button. * The **Clear cache** button is only available if you have enabled state-aware orchestration. * After clearing the cache, the next run rebuilds every model from scratch. Subsequent runs rely on the regenerated cache. [![Clear cache button](/img/docs/dbt-platform/using-dbt-platform/sao-clear-cache.png?v=2 "Clear cache button")](#)Clear cache button #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### OAuth client credentials for dbt State Deprecated Login required | Usage-based The dbt State app is deprecated The standalone dbt State app (`app.state.dbt.com`) is no longer accepting new users. If you have an existing dbt State app account, [create a free dbt platform account](http://us1.dbt.com/register?_dbtsrc=dbt-state) to continue using dbt State — your free trial will be extended upon account creation. If you're using the standalone [dbt State web app](https://app.state.dbt.com/), authenticate dbt State with OAuth client credentials. #### Prerequisites * dbt State installed and configured. Refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) for more information. * A standalone dbt State account at [app.state.dbt.com](https://app.state.dbt.com/). * An **Admin** or **Owner** role in your dbt State organization. Refer to [Roles and tab access](#roles-and-tab-access) for details. #### Roles and tab access The dbt State web app has four tabs under **Organization**: | Tab | Description | | ----------- | --------------------------------------------------------------------------------- | | **Usage** | View your project reuses and compute time saved once dbt State is enabled. | | **Users** | Invite team members and grant or revoke access. | | **Billing** | View daily active target tables (DATTs) for the current billing period. | | **Clients** | Create and manage OAuth clients for CI/CD and other non-interactive environments. | <br /> Your role determines which tabs you can access. | Role | Access | Notes | | ------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Owner** | Usage, Users, Billing, Clients | The user who created the organization is the Owner by default. An Owner can transfer their role to another user, which demotes the original Owner to Admin. | | **Admin** | Usage, Users, Billing, Clients | — | | **Developer** | Usage | Default role when users are added. | An existing **Owner** or **Admin** can grant or revoke admin access from the **Users** tab. #### Creating an OAuth client 1. In the [dbt State web app](https://app.state.dbt.com/), navigate to the **Clients** tab. 2. Click **Add OAuth Client**. 3. Enter a name and description for the new client and click **Create**. 4. Copy the client ID and secret to use in your environment configuration. #### Configuring OAuth authentication Once you have the client ID and secret, set the following environment variables in your environment. Using environment variables is the recommended approach as it keeps sensitive credentials out of your code repository. ```bash DBT_ENGINE_STATE_OAUTH_CLIENT_ID=YOUR_CLIENT_ID DBT_ENV_SECRET_STATE_OAUTH_CLIENT_SECRET=YOUR_CLIENT_SECRET ``` Replace `YOUR_CLIENT_ID` and `YOUR_CLIENT_SECRET` with the values from your OAuth client. Once configured, [verify dbt State is active](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md#verifying-dbt-state-is-active). #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Setting up authentication for non-interactive environments](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) --- ### Onboarding to Lake Compute Private beta ### Onboarding to Lake Compute [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Lake Compute is in private beta To request access, fill out the [Lake Compute signup form](https://docs.google.com/forms/d/e/1FAIpQLSdWXHU3VFqq1t8GVAA2BMfY_AfeVJH26OXJFlNwXiVv4pcuXQ/viewform). Lake Compute is in active development and likely to change. Breaking changes may occur, documentation may be incomplete, technical support may be limited, and no service-level agreement covers it during beta. Refer to [Private beta limitations](https://docs.getdbt.com/docs/lake-compute.md#private-beta-limitations) before you plan real work around it. #### Set up MDLS MDLS is Fivetran's managed data lake product, including a hosted Iceberg catalog and compute service (Lake Compute). To start using Lake Compute, you need a Fivetran account and an MDLS instance. ##### 1. Create a Fivetran account (if you don't already have one) A Fivetran account itself is free to create and use indefinitely, even after any trial period ends, as long as you don't create connectors — billing only applies to connectors, and there is not yet any billing for MDLS or Lake Compute usage. If you also want to try connectors, you get a 14-day full-feature trial. ##### 2. Create an MDLS destination (if you don't already have one) Follow the [Managed Data Lake Service setup guide for AWS](https://fivetran.com/docs/managed-data-lake-service/setup-guide#setupinstructionsforaws) on Fivetran's docs site. Things to watch for * Provide an S3 bucket in the `us-east-1` region to reduce latency and egress costs. * Make sure the bucket name and prefix don't contain any of these characters: `['#', '%', '^', '{', '}', '[', ']', '"', '?', '|', '', ' ']`. * After you create the MDLS destination, Fivetran provides the required `External ID` (also known as `group_id`). Pass this along with Fivetran's AWS account (`834469178297`) to the IAM trust policy. ##### 3. Enable Lake Compute and generate a token Before you enable Lake Compute, click **Save & Test** on your MDLS destination and confirm every test passes. Only enable Lake Compute and generate a token after the MDLS connection tests are all green. You don't need to generate a token as part of the MDLS connection setup itself. On the destination page for your MDLS instance, go to the **Compute** tab and turn on **Enable Lake Compute**. [![Enable Lake Compute](/img/docs/lake-compute/enable-lake-compute.png?v=2 "Enable Lake Compute")](#)Enable Lake Compute Then generate a token (user-scoped or service account). Save this token in a password manager, because you need it for your dbt connection profile. [![Generate a Lake Compute token](/img/docs/lake-compute/generate-token.png?v=2 "Generate a Lake Compute token")](#)Generate a Lake Compute token #### Configure Snowflake These steps require `ACCOUNTADMIN` permissions. ##### Supported authentication methods Lake Compute supports the following authentication methods: * `sso` (with `authenticator: externalbrowser` for local development) * `keypair` (with `private_key`) * `self_signed_jwt` * `workload_identity` These are the only methods currently supported. The following authentication methods can create a Snowflake PAT, but Lake Compute doesn't support them yet — coming soon: * `user`/`password` * `user`/`password` with MFA * OAuth Lake Compute uses your Snowflake connection profile to generate a token that can access Iceberg tables in Snowflake Horizon. This token has the exact same access as the user/role specified in your Snowflake connection profile. By default, this is a [personal access token (PAT)](https://docs.snowflake.com/en/user-guide/programmatic-access-tokens) that lasts for 1 month; you can configure a different `pat_duration` if required by your organization's security policies. To use keypair authentication, follow Snowflake's guide to [key-pair authentication and key-pair rotation](https://docs.snowflake.com/en/user-guide/key-pair-auth) to create a Privacy Enhanced Mail (PEM) private-public key pair for your user, then set the public key on the user: ```sql ALTER USER USER_NAME SET RSA_PUBLIC_KEY='PUBLIC_KEY_BODY'; ``` Replace `USER_NAME` with the Snowflake user dbt connects as, and `PUBLIC_KEY_BODY` with the body of the public key you generated. Save the private key somewhere so you can specify it in your `profiles.yml` later. Lake Compute connects to Snowflake in one of two ways: * **Option 1 (PAT)** — compatible with any of the methods above. Lake Compute creates a Snowflake personal access token (PAT) using your connection profile, which requires a network policy (see [Allow Lake Compute's egress through a network policy](#allow-lake-computes-egress-through-a-network-policy-pat-only) below). * **Option 2 (JWT)** — compatible with `keypair` (private key) authentication. Lake Compute authenticates directly with a JWT instead of creating a PAT, so you don't need a network policy. `self_signed_jwt` looks similar to `keypair` by name, but it still creates a PAT under the hood — it just caches that PAT locally so it can be reused across runs. It follows Option 1, not Option 2, and still needs a network policy. ##### Create a Snowflake catalog integration for MDLS Create a catalog integration in Snowflake, using the pre-generated SQL in the Fivetran UI under **Catalog integration > Snowflake**. Make the following changes to that SQL: | Old value | New value | | ----------------------------------------- | ------------------------------------------------------------------------------------------------ | | `OAUTH_CLIENT_ID` / `OAUTH_CLIENT_SECRET` | Enter your existing credentials, or click **Regenerate client secret** if you've forgotten them. | Lake Compute uses the MDLS catalog integration to propagate Iceberg table metadata from MDLS to Snowflake after building each model. Therefore, you need to `grant usage on integration CATALOG_INTEGRATION_NAME to role ROLE_NAME` for all roles that are running dbt with Lake Compute. ##### Allow Lake Compute's egress through a network policy (PAT only) This section applies to Option 1 (PAT) only Snowflake requires a network policy to be in effect before it creates a PAT, even if your account doesn't otherwise enforce one. If you're using Option 2 (`keypair` JWT authentication), Lake Compute never creates a PAT, and you can **skip this section entirely.** First, check whether your account or your user already has a network policy assigned: ```sql SHOW PARAMETERS LIKE 'NETWORK_POLICY' IN ACCOUNT; SHOW PARAMETERS LIKE 'NETWORK_POLICY' IN USER USER_NAME; ``` * **If neither returns a policy:** Snowflake doesn't restrict egress by default, and no further action is needed — skip the rest of this section. * **If either returns a policy:** continue below. In Snowflake, a network policy attached to a user *replaces* the account-level policy for that user rather than supplementing it, so your existing policy already accounts for your own IP or VPN range — you only need to add Lake Compute's egress IPs to it. 1. Create a network rule with Lake Compute's egress IPs: ```sql CREATE NETWORK RULE lakecompute_ipv4 TYPE = IPV4 MODE = INGRESS VALUE_LIST = ( '34.85.252.27/32', '35.197.105.90/32', '3.239.194.48/29', -- AWS us-east-1 '35.80.36.104/29' -- AWS us-west-2 ); ``` 2. Add `lakecompute_ipv4` to your existing policy's allowed rule list. `ALTER NETWORK POLICY ... SET ALLOWED_NETWORK_RULE_LIST` replaces the entire list, so include your existing rule(s) as well as the new one. Replace `EXISTING_POLICY_NAME` with your account's or user's existing policy, and `EXISTING_RULE_NAME` with the rule(s) already on that policy (check with `DESC NETWORK POLICY EXISTING_POLICY_NAME;`): ```sql ALTER NETWORK POLICY EXISTING_POLICY_NAME SET ALLOWED_NETWORK_RULE_LIST = ( 'EXISTING_RULE_NAME', 'lakecompute_ipv4' ); ``` note Only security administrators (that is, users with the `SECURITYADMIN` role) or higher, or a role with the global `CREATE NETWORK POLICY` privilege, can create network rules or modify network policies. Some accounts also restrict which authentication methods a user or account can use. If your account blocks PAT authentication, you need an [authentication policy](https://docs.snowflake.com/en/sql-reference/sql/create-authentication-policy) that explicitly permits it before Snowflake creates a PAT for Lake Compute. #### Set up your connection profile You can use Lake Compute in one of two ways: 1. As a "sidecar" for an existing data warehouse, like Snowflake. Lake Compute reads upstream models out of the data warehouse's managed Iceberg catalog, like Snowflake Horizon. 2. As your default adapter, with Lake Compute for compute and Fivetran MDLS for storage. Before you continue, update to the latest dbt client: `dbt system update --version VERSION`. Lake Compute support depends on a recent dbt v2 release. ##### Opt in to multi-adapter invocations dbt gates Lake Compute behind an experimental opt-in. Set this before you run anything, for **either** of the two patterns above: ```bash export DBT_ENGINE_EXPERIMENTAL_MULTI_ADAPTER=true ``` The variable gates `type: lakecompute` in `profiles.yml`, the `adapter` config on a node, and the `--adapter` flag. While it's off, dbt fails at parse time with a message naming the variable rather than falling back to your default adapter, and `DBT_ALLOW_EXPERIMENTAL_ADAPTERS` won't substitute for it. Refer to [Required environment variable](https://docs.getdbt.com/docs/lake-compute.md#required-environment-variable) for the full behavior. Add it to your terminal profile so it persists across sessions. ##### Local `profiles.yml` ###### Use Lake Compute as a "sidecar" Replace `YOUR_ACCOUNT` with your Snowflake account identifier, `DATABASE_NAME` and `SCHEMA_NAME` with the database and schema dbt writes to, and `LAKE_COMPUTE_TOKEN` with the token you generated in step 3: ```yaml my_profile: target: default outputs: default: - name: snowflake type: snowflake default: true account: YOUR_ACCOUNT database: DATABASE_NAME schema: SCHEMA_NAME ... - name: lakecompute type: lakecompute method: fivetran fivetran_credential: LAKE_COMPUTE_TOKEN # database + schema are inherited from default adapter config ``` With this profile, Lake Compute: * Uses the default Snowflake adapter config to create a PAT to Snowflake's managed catalog, Horizon * Reads all upstream Iceberg tables from the Horizon catalog * Writes models as Iceberg tables to MDLS, and propagates them to the Horizon catalog, using the `database` and `schema` from the default Snowflake adapter config as the default write location ###### Use Lake Compute as your default adapter In this pattern, your entire project runs on Lake Compute, and Lake Compute materializes every model as an Iceberg table in your MDLS catalog instance. Replace `MDLS_INSTANCE_NAME` with your MDLS instance name, `SCHEMA_NAME` with the schema dbt writes to, and `LAKE_COMPUTE_TOKEN` with the token you generated in step 3: ```yaml # new spec my_profile: target: default outputs: default: - name: lakecompute type: lakecompute method: fivetran fivetran_credential: LAKE_COMPUTE_TOKEN # database + schema are required when lakecompute is default database: MDLS_INSTANCE_NAME schema: SCHEMA_NAME ``` This means: * **Every model in the project is configured `compute: 'lakecompute'` by default.** * Every model also gets `materialized: table` and `table_format: 'iceberg'` as defaults. ##### dbt platform ###### Use Lake Compute as a "sidecar" As a temporary workaround during private beta, you can set Lake Compute configurations using [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes), by adding a key named `lakecompute`: ```yaml lakecompute: method: fivetran fivetran_credential: "{{ env_var('DBT_ENV_SECRET_LAKE_COMPUTE_TOKEN') }}" method: self_signed_jwt ``` Don't put your token in Extended Attributes Extended Attributes aren't a safe place for secrets, so never paste the token itself into this field. Reference it with `env_var()`, as shown above, and store the token in a [secret environment variable](https://docs.getdbt.com/docs/build/environment-variables.md#handling-secrets) instead. Then within your environment variables, set: * `DBT_ALLOW_EXPERIMENTAL_ADAPTERS`: `true` * `DBT_ENGINE_EXPERIMENTAL_MULTI_ADAPTER`: `true` * `DBT_ENV_SECRET_LAKE_COMPUTE_TOKEN`: the token you generated in step 3 This differs from running locally, where you set both `DBT_ENGINE_EXPERIMENTAL_MULTI_ADAPTER` and `DBT_ALLOW_EXPERIMENTAL_ADAPTERS`. Refer to [Required environment variable](https://docs.getdbt.com/docs/lake-compute.md#required-environment-variable). ##### Verify your setup Run `dbt debug` to confirm your profile is configured correctly. Normally, `debug` just makes sure that you can connect to your data platform, but for Lake Compute it is more involved with the goal of giving you actionable feedback should something be misconfigured. A failure can indicate potential issues with credentials, a malformed profile, a missing `database`/`schema` (see above), or a network policy blocking egress (see [Allow Lake Compute's egress through a network policy](#allow-lake-computes-egress-through-a-network-policy-pat-only)). Here's what it does at a high-level: 1. This parses `profiles.yml` and `dbt_project.yml` 2. tests authentication against every configured adapter in output — including `lakecompute` — with both a write test and a read-back test. 3. Tests "propagation" that a table written to MDLS has a corresponding table created on the Horizon catalog so that downstream Snowflake nodes may `ref` it. #### Configure models to run on Lake Compute ##### Use Lake Compute as a "sidecar" Given a dbt DAG of `model_a -> model_b -> model_c`, you can configure: * `model_a` to run on Snowflake and materialize an Iceberg table in Snowflake Horizon, using Snowflake-managed storage * `model_b` to run on Lake Compute * `model_c` to run on Snowflake ```sql -- models/model_a.sql {{ config( table_format = 'iceberg', materialized = 'table' ) }} select 1 as id ``` ```sql -- models/model_b.sql {{ config( adapter = 'lakecompute' ) }} select * from {{ ref('model_a') }} ``` ```sql -- models/model_c.sql select * from {{ ref('model_b') }} ``` ##### Use Lake Compute as your default adapter All your models run on Lake Compute by default and write to Iceberg tables in MDLS. Also check for existing `materialized: view` configs — Lake Compute doesn't support views (see [Private beta limitations](https://docs.getdbt.com/docs/lake-compute.md#private-beta-limitations)), so switch any views to `table` or `ephemeral` before converting a project over. For an existing project, convert models over in two steps rather than all at once: 1. Identify the model you want to run on Lake Compute and its direct upstream dependencies. 2. Add `{{ config(materialized='table', table_format='iceberg') }}` to the target model and its upstream dependencies. Run the project and verify it works. Any runtime errors at this stage are due to Iceberg/Snowflake type incompatibilities and require casting. 3. On the target model, add `{{ config(adapter='lakecompute') }}` and look for any Snowflake-specific functions. If you find any, rewrite the query using the corresponding DuckDB function. Run the project again to verify. --- ### Optimize your dbt builds By default, dbt rebuilds every selected node on every run — even if nothing has changed. This makes it easy to get started, but adds up for larger projects: longer job times, higher warehouse costs, and slower feedback during development. dbt has features designed to help you skip unnecessary work and reuse existing results instead. #### dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is a service that makes dbt smarter about what to build. It integrates into any dbt deployment — including self-hosted deployments using dbt v2 or dbt v1 — without requiring a recurring dbt platform subscription. Instead of rebuilding every node on every run, it compares each node's logic and upstream data against the previous run and picks the most efficient path: * **Skip** — If the node's logic and upstream data haven't changed, dbt reuses the existing object in your target schema as-is. * **Clone** — If the data is fresh but exists in a different schema (for example, production), dbt clones it rather than rebuilding from scratch. * **Build** — If reuse isn't possible, dbt rebuilds normally and automatically defers unselected upstream nodes to production — no `--defer` or `--state` flags required. To enable dbt State: * **dbt v1.7–1.12**: ```bash cd path/to/your/project pip install dbt-state ``` * **dbt v2**: ```bash cd path/to/your/project dbt login ``` Authentication requires a dbt platform account with a [30-day free trial](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). dbt State pricing is usage-based — you're billed per target table that dbt State reuses each day, not per dbt platform seat. For full setup instructions, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). #### Deferral [Deferral](https://docs.getdbt.com/reference/node-selection/defer.md) lets you build a subset of your project without building all upstream dependencies first. Instead of running everything upstream, dbt points unbuilt references at existing objects in another environment — typically production. This is useful in development and CI environments, where you want to test one or two models without waiting for a full pipeline run. ```bash dbt build --select my_model --defer --state path/to/prod/artifacts ``` #### dbt clone [`dbt clone`](https://docs.getdbt.com/reference/commands/clone.md) creates copies of selected nodes in a target schema. On warehouses that support zero-copy cloning (for example, Snowflake), it creates lightweight database clones without duplicating the underlying data. On other warehouses, it creates views pointing at the upstream relations. This is useful in development when you want to quickly populate a dev environment with production-like objects without running the full pipeline. ```bash dbt clone --select my_model ``` #### dbt's selection syntax dbt has a [variety of selectors](https://docs.getdbt.com/reference/node-selection/syntax.md) you can use to target specific parts of your project instead of building everything every time. For example, `+my_model` selects `my_model` and all of its upstream dependencies, while `my_model+2` selects `my_model` and two levels of downstream dependents. This lets you test your changes in isolation without running your entire project. ```bash dbt build --select +my_model ``` --- ### Orchestrate downstream exposures Beta ### Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Use [dbt platform job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to proactively refresh downstream exposures and the underlying data sources (extracts) that power your Tableau Workbooks. (Applies to dbt v2.0 and later) dbt v2 not supported for this beta Orchestrating downstream exposures *isn’t* available when your deployments use [**v2 Stable**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Supported jobs must use [**v1 Latest**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with the dbt v1 engine. On the [**v2 Stable**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) release track, orchestrating downstream exposures isn’t supported yet. Setting `DBT_ACTIVE_EXPOSURES` and `DBT_ACTIVE_EXPOSURES_BUILD_AFTER` won’t enable orchestration behavior (like Tableau extract refreshes) or the related job log entries. The private beta is for dbt Enterprise accounts. Contact your account representative for access. Orchestrating exposures integrates with [downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and uses your `dbt build` job to ensure that Tableau extracts are updated regularly. Control the frequency of these refreshes by configuring environment variables in your dbt environment.  Differences between visualizing and orchestrating downstream exposures The following table summarizes the differences between visualizing and orchestrating downstream exposures: | Info | Set up and visualize downstream exposures | Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") | | ----------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Automatically brings downstream assets into your dbt lineage. | Proactively refreshes the underlying data sources during scheduled dbt jobs. | | Benefits | Provides visibility into data flow and dependencies. | Ensures BI tools always have up-to-date data without manual intervention. | | Location | Exposed in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) | Exposed in [dbt scheduler](https://docs.getdbt.com/docs/deploy/deployments.md) | | Supported BI tool | Tableau | Tableau | | Use case | Helps users understand how models are used and reduces incidents. | Optimizes timeliness and reduces costs by running models when needed. | #### Prerequisites To orchestrate downstream exposures, you should meet the following: * [Configured downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and ensured desired exposures are included in your lineage. * Verified your environment and jobs are on a supported dbt [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). * Have a dbt account on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/). * Created a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run. * Have [admin permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) in dbt to edit project settings or production environment settings. * Configured a [Tableau personal access token (PAT)](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) whose creator has privileges to view and refresh the data sources used by your exposures. The PAT inherits the permissions of its creator. Use a PAT created by: * A Tableau server or site administrator * A data source owner or a project leader #### Orchestrate downstream exposures To orchestrate downstream exposures and see refreshes happen automatically during scheduled jobs on [Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with the dbt v1 engine: 1. In the dbt platform, click **Deploy**, then **Environments**, and select the **Environment variables** tab. 2. Click **Add variable** and set the [environment level variable](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables) `DBT_ACTIVE_EXPOSURES` to `1` within the environment you want the refresh to happen. 3. Then set the `DBT_ACTIVE_EXPOSURES_BUILD_AFTER` to control the maximum refresh frequency (in minutes) you want between each exposure refresh. 4. Set the variable to **1440** minutes (24 hours) by default. This means that downstream exposures won’t refresh Tableau extracts more often than this set interval, even if the related models run more frequently. [![Set the environment variable \`DBT\_ACTIVE\_EXPOSURES\` to \`1\`.](/img/docs/platform-integrations/auto-exposures/active-exposures-env-var.jpg?v=2 "Set the environment variable `DBT_ACTIVE_EXPOSURES` to `1`.")](#)Set the environment variable \`DBT\_ACTIVE\_EXPOSURES\` to \`1\`. 5. Run a production job on [Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with dbt v1. Each run can trigger a downstream exposure refresh; if a job runs before the configured interval has passed, dbt skips the downstream exposure refresh and marks it as `skipped` in the job logs. 6. View downstream exposure entries in your run job logs. [![View the downstream exposure logs in the dbt run job logs.](/img/docs/platform-integrations/auto-exposures/active-exposure-log.jpg?v=2 "View the downstream exposure logs in the dbt run job logs.")](#)View the downstream exposure logs in the dbt run job logs. * View more details in the debug logs for any troubleshooting. --- ### Organize your outputs [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-schemas.md) ###### [Custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) [Learn how to use the `schema` configuration key to specify a custom schema.](https://docs.getdbt.com/docs/build/custom-schemas.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-databases.md) ###### [Custom databases](https://docs.getdbt.com/docs/build/custom-databases.md) [Learn how to use the `database` configuration key to specify a custom database.](https://docs.getdbt.com/docs/build/custom-databases.md) <br /> [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-aliases.md) ###### [Custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) [Learn how to use the `alias` model configuration to change the name of a model's identifier in the database.](https://docs.getdbt.com/docs/build/custom-aliases.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-target-names.md) ###### [Custom target names](https://docs.getdbt.com/docs/build/custom-target-names.md) [Learn how to define a custom target name for a dbt job.](https://docs.getdbt.com/docs/build/custom-target-names.md) --- ### Packages Software engineers frequently modularize code into libraries. These libraries help programmers operate with leverage: they can spend more time focusing on their unique business logic, and less time implementing code that someone else has already spent the time perfecting. In dbt, libraries like these are called *packages*. dbt's packages are so powerful because so many of the analytic problems we encountered are shared across organizations, for example: * transforming data from a consistently structured SaaS dataset, for example: * turning [Snowplow](https://hub.getdbt.com/dbt-labs/snowplow/latest/) or [Segment](https://hub.getdbt.com/dbt-labs/segment/latest/) pageviews into sessions * transforming [AdWords](https://hub.getdbt.com/dbt-labs/adwords/latest/) or [Facebook Ads](https://hub.getdbt.com/dbt-labs/facebook_ads/latest/) spend data into a consistent format. * writing dbt macros that perform similar functions, for example: * [generating SQL](https://github.com/dbt-labs/dbt-utils#sql-helpers) to union together two relations, pivot columns, or construct a surrogate key * creating [custom schema tests](https://github.com/dbt-labs/dbt-utils#schema-tests) * writing [audit queries](https://hub.getdbt.com/dbt-labs/audit_helper/latest/) * building models and macros for a particular tool used in your data stack, for example: * Models to understand [Redshift](https://hub.getdbt.com/dbt-labs/redshift/latest/) privileges. * Macros to work with data loaded by [Stitch](https://hub.getdbt.com/dbt-labs/stitch_utils/latest/). * shaping unstructured data so AI agents can retrieve, trust, and cite it, for example: * chunking, embedding, and semantically searching call transcripts or support tickets with the [dbt\_context\_engineering](https://hub.getdbt.com/dbt-labs/dbt_context_engineering/latest/) package. Refer to the [dbt\_context\_engineering blog](https://docs.getdbt.com/blog/dbt-context-engineering). dbt *packages* are in fact standalone dbt projects, with models, macros, and other resources that tackle a specific problem area. As a dbt user, by adding a package to your project, all of the package's resources will become part of your own project. This means: * Models in the package will be materialized when you `dbt run`. * You can use `ref` in your own models to refer to models from the package. * You can use `source` to refer to sources in the package. * You can use macros in the package in your own project. * It's important to note that defining and installing dbt packages is different from [defining and installing Python packages](https://docs.getdbt.com/docs/build/python-models.md#using-pypi-packages) #### Use cases The following setup will work for every dbt project: * Add [any package dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-project-dependencies) to `packages.yml` * Add [any project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-package-dependencies) to `dependencies.yml` However, you may be able to consolidate both into a single `dependencies.yml` file. Read the following section to learn more. ###### About packages.yml and dependencies.yml The `dependencies.yml` file can contain both types of dependencies: "package" and "project" dependencies. * [Package dependencies](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) lets you add source code from someone else's dbt project into your own, like a library. * Project dependencies provide a different way to build on top of someone else's work in dbt. * For private Git packages that need Jinja (for example the [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method)), use `packages.yml` on dbt v1, which supports Jinja in package specs. dbt v2 also supports Jinja in `dependencies.yml`, but use `packages.yml` if the project must also run on dbt v1. Refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). For private packages without Jinja tokens, you can use [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) in `packages.yml` or `dependencies.yml`. If your dbt project doesn't require Jinja in package specifications, you can rename `packages.yml` to `dependencies.yml`. If you need Jinja in package specs (for example an environment variable or the [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method)): * On dbt v1, keep using `packages.yml`. dbt v1 does not render Jinja in `dependencies.yml`. * On the dbt v2, you can use Jinja in `dependencies.yml`. It is recommended to use `packages.yml` if the project must also run on dbt v1. Refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md) for the full matrix. Use the following toggles to understand the differences and determine when to use `dependencies.yml` or `packages.yml` (or both). Refer to the [FAQs](#faqs) for more info.  When to use Project dependencies Project dependencies are designed for the [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) and [cross-project reference](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) workflow: * Use `dependencies.yml` when you need to set up cross-project references between different dbt projects, especially in a dbt Mesh setup. * Use `dependencies.yml` when you want to include both projects and non-private dbt packages in your project's dependencies. * Use `dependencies.yml` for organization and maintainability if you're using both [cross-project refs](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) and [dbt Hub packages](https://hub.getdbt.com/). This reduces the need for multiple YAML files to manage dependencies.  When to use Package dependencies Package dependencies allow you to add source code from someone else's dbt project into your own, like a library: * If you only use packages like those from the [dbt Hub](https://hub.getdbt.com/), remain with `packages.yml`. * Use `packages.yml` when you want to download dbt packages, such as dbt projects, into your root or parent dbt project. Something to note is that it doesn't contribute to the dbt Mesh workflow. * Use `packages.yml` to include packages in your project's dependencies. This includes both public packages, such as those from the [dbt Hub](https://hub.getdbt.com/), and private packages. dbt now supports [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages). * [`packages.yml` supports Jinja rendering](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for historical reasons, allowing dynamic configurations. This can be useful if you need to insert values, like a [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method) from an environment variable, into your package specifications. dbt v2 also supports Jinja in `dependencies.yml`; refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). Previously, to use private Git repositories in dbt, you needed to use a workaround that involved embedding a Git token with Jinja. This is not ideal as it requires extra steps like creating a user and sharing a Git token. We’ve introduced support for [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages-) to address this. #### How do I create a package? Creating packages is an advanced use of dbt, but it can be a relatively simple task. The only strict requirement is the presence of a [`dbt_project.yml` file](https://docs.getdbt.com/reference/dbt_project.yml.md). The most common use-cases for packages are: * Sharing [models](https://docs.getdbt.com/docs/build/models.md) to share across multiple projects. * Sharing [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) to share across multiple projects. Note that packages can be [private](#private-packages) — they don't need to be shared publicly. Private packages can be hosted on your own Git provider (for example, GitHub or GitLab). For instructions on creating dbt packages and additional information, refer to our guide [Building dbt packages](https://docs.getdbt.com/guides/building-packages.md?step=1). #### How do I add a package to my project? 1. Add a file named `dependencies.yml` or `packages.yml` to your dbt project. This should be at the same level as your `dbt_project.yml` file. 2. Specify the package(s) you wish to add using one of the supported syntaxes, for example: ```yaml packages: - package: dbt-labs/snowplow version: 0.7.0 - git: "https://github.com/dbt-labs/dbt-utils.git" revision: 0.9.2 - local: /opt/dbt/redshift ``` The default [`packages-install-path`](https://docs.getdbt.com/reference/project-configs/packages-install-path.md) is `dbt_packages`. 3. Run `dbt deps` to install the package(s). Packages get installed in the `dbt_packages` directory – by default this directory is ignored by git, to avoid duplicating the source code for the package. #### How do I specify a package? You can specify a package using one of the following methods, depending on where your package is stored. ##### Hub packages (recommended) dbt Labs hosts the [Package hub](https://hub.getdbt.com), registry for dbt packages, as a courtesy to the dbt Community, but does not certify or confirm the integrity, operability, effectiveness, or security of any Packages. Please read the [dbt Labs Package Disclaimer](https://hub.getdbt.com/disclaimer/) before installing Hub packages. You can install available hub packages in the following way: packages.yml ```yaml packages: - package: dbt-labs/snowplow version: 0.7.3 # version number ``` Hub packages require a version to be specified – you can find the latest release number on dbt Hub. Since Hub packages use [semantic versioning](https://semver.org/), we recommend pinning your package to the latest patch version from a specific minor release, like so: ```yaml packages: - package: dbt-labs/snowplow version: [">=0.7.0", "<0.8.0"] ``` `dbt deps` "pins" each package by default. See ["Pinning packages"](#pinning-packages) for details. Where possible, we recommend installing packages via dbt Hub, since this allows dbt to handle duplicate dependencies. This is helpful in situations such as: * Your project uses both the dbt-utils and Snowplow packages, and the Snowplow package *also* uses the dbt-utils package. * Your project uses both the Snowplow and Stripe packages, both of which use the dbt-utils package. In comparison, other package installation methods are unable to handle the duplicate dbt-utils package. Advanced users can choose to host an internal version of the package hub based on [this repository](https://github.com/dbt-labs/hub.getdbt.com) and setting the (Applies to dbt v1.11 and later) `DBT_ENGINE_PACKAGE_HUB_URL` environment variable. ###### Prerelease versions Some package maintainers may wish to push prerelease versions of packages to the dbt Hub, in order to test out new functionality or compatibility with a new version of dbt. A prerelease version is demarcated by a suffix, such as `a1` (first alpha), `b2` (second beta), or `rc3` (third release candidate). By default, `dbt deps` will not include prerelease versions when resolving package dependencies. You can enable the installation of prereleases in one of two ways: * Explicitly specifying a prerelease version in your `version` criteria * Setting `install_prerelease` to `true`, and providing a compatible version range For example, both of the following configurations would successfully install `0.4.5-a2` for the [`dbt_artifacts` package](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/): ```yaml packages: - package: brooklyn-data/dbt_artifacts version: 0.4.5-a2 ``` ```yaml packages: - package: brooklyn-data/dbt_artifacts version: [">=0.4.4", "<0.4.6"] install_prerelease: true ``` ##### Git packages Packages stored on a Git server can be installed using the `git` syntax, like so: packages.yml ```yaml packages: - git: "https://github.com/dbt-labs/dbt-utils.git" # git URL revision: 0.9.2 # tag or branch name ``` Add the Git URL for the package, and optionally specify a revision. The revision can be: * a branch name * a tagged release * a specific commit (full 40-character hash) Example of a revision specifying a 40-character hash: ```yaml packages: - git: "https://github.com/dbt-labs/dbt-utils.git" revision: 4e28d6da126e2940d17f697de783a717f2503188 ``` By default, `dbt deps` "pins" each package. See ["Pinning packages"](#pinning-packages) for details. ##### Internally hosted tarball URL Some organizations have security requirements to pull resources only from internal services. To address the need to install packages from hosted environments such as Artifactory or cloud storage buckets, dbt enables you to install packages from internally-hosted tarball URLs. ```yaml packages: - tarball: https://codeload.github.com/dbt-labs/dbt-utils/tar.gz/0.9.6 name: 'dbt_utils' ``` Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within. #### Private packages ##### Native private packages Native private packages let you install packages from [supported](#prerequisites) private Git repos using the `private` key, without having to configure a [token](#git-token-method) or write out a full Git URL. This simplifies setup and reduces credential management. * dbt platform: Uses your existing Git [integration](https://docs.getdbt.com/docs/platform/git/configure-git.md) for authentication. * Locally using dbt v2 or dbt v1.12+: Uses your system's SSH configuration. Requires the [`provider` key](#using-the-provider-key). ###### Prerequisites * **dbt platform**: You must have one of the following Git providers configured in the **Integrations** section of your **Account settings**: * **[GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md)** * **[Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md)** * Use the `org/project/repo` path with the `ado` provider. * **[GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md)** * Every GitLab repo with private packages must also be a dbt platform project. * **Locally using dbt v2 or dbt v1.12+**: You must have an SSH key configured on your machine for the relevant Git provider and include the [`provider` key](#using-the-provider-key) in your package configuration. ###### Configuration Use the `private` key in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Git integration without having to provision an access token or create a dbt environment variable. packages.yml ```yaml packages: - private: dbt-labs/awesome_repo # your-org/your-repo path provider: "github" # Supported values: "github", "gitlab", "ado" - package: normal packages [...] ``` dependencies.yml ```yaml packages: - private: dbt-labs/awesome_repo # your-org/your-repo path provider: "github" # Supported values: "github", "gitlab", "ado" ``` Azure DevOps considerations and limitations There are some considerations and limitations when using native private packages from Azure DevOps. Open the expandable section to learn more.  Native private packages and Azure DevOps limitations 1. Use the `ado` provider and specify the `org/project/repo` path in the `private` key. packages.yml ```yaml packages: - private: my-org/my-project/my-repo provider: "ado" ``` 2. On dbt platform, native private packages from Azure DevOps can fail when the package is in a different Azure DevOps project than the job that installs it, especially if your account is connected to many Azure DevOps projects: * This happens because dbt platform has a 32 KB limit for the Azure DevOps authentication details it can use at job runtime. If your connected Azure DevOps projects exceed that limit, dbt platform can only use the current project's connection. As a result, packages in other Azure DevOps projects can't be accessed and `dbt deps` fails. * As a workaround, reduce the number of Azure DevOps projects connected to your account, then rerun `dbt deps`. The number of projects you can connect depends on your Azure DevOps organization structure and repo count. This limitation doesn't affect local development with dbt v1 or dbt v2 when cloning private packages over SSH. We're currently working to address this, and if you're running into issues, please contact your dbt Labs account team. You can pin private packages similar to regular dbt packages: ```yaml packages: - private: dbt-labs/awesome_repo revision: "0.9.5" # Pin to a tag, branch, or complete 40-character commit hash ``` ###### Using the `provider` key Add the `provider` key when: * You are using multiple Git integrations or using dbt v2. * You are using dbt v2 locally (with the [dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) or the [VS Code extension](https://docs.getdbt.com/docs/local/install-dbt.md?version=2)) (required). * You are using dbt v1.12 or later for SSH-based cloning (required). ```yaml packages: - private: dbt-labs/awesome_repo provider: "github" # Supported values: "github", "gitlab", "ado" ``` dbt v1 and dbt v2 use the `provider` value to construct the correct SSH URL for cloning, based on the provider: | Provider | SSH URL format | | -------- | ------------------------------------------- | | `github` | `git@github.com:org/repo.git` | | `gitlab` | `git@gitlab.com:org/repo.git` | | `ado` | `git@ssh.dev.azure.com:v3/org/project/repo` | dbt v1 and dbt v2 rely on your system's SSH configuration to authenticate and clone the private repository. If `git clone` works on your system for the private package repo, the private package install should work too. ##### SSH key method (CLI only) note This method uses the `git:` key with a full SSH URL, which is different from [native private packages](#native-private-packages) that use the `private:` key. For most use cases, native private packages is the recommended approach as it simplifies setup. If you're using the Command Line, private packages can be cloned via SSH and an SSH key. When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html). packages.yml ```yaml packages: - git: "git@github.com:dbt-labs/dbt-utils.git" # git SSH URL ``` If you're using the dbt platform, the SSH key method will not work, but you can use [native private packages](#native-private-packages) or the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages.md#git-token-method). ##### Git token method note [Native private packages](#native-private-packages) is the recommended approach for GitHub, GitLab, and Azure DevOps. The git token method is still functional in dbt v1, dbt v2, and the dbt platform, but requires provisioning a personal access token. It remains the supported path for dbt v1 users who need HTTPS-based cloning. This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo. dbt usage If you are using dbt, you must adhere to the naming conventions for environment variables. Environment variables in dbt must be prefixed with either `DBT_` or `DBT_ENV_SECRET`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt's UI. In GitHub: packages.yml ```yaml packages: # use this format when accessing your repository via a github application token - git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL # use this format when accessing your repository via a classical personal access token - git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL # use this format when accessing your repository via a fine-grained personal access token (username sometimes required) - git: "https://GITHUB_USERNAME:{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL ``` Read more about creating a GitHub Personal Access token [here](https://docs.github.com/en/enterprise-server@3.1/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token). You can also use a GitHub App installation [token](https://docs.github.com/en/rest/reference/apps#create-an-installation-access-token-for-an-app). In GitLab: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_USER_NAME')}}:{{env_var('DBT_ENV_SECRET_DEPLOY_TOKEN')}}@gitlab.example.com/dbt-labs/awesome_project.git" # git HTTPS URL ``` Read more about creating a GitLab Deploy Token [here](https://docs.gitlab.com/ee/user/project/deploy_tokens/#creating-a-deploy-token) and how to properly construct your HTTPS URL [here](https://docs.gitlab.com/ee/user/project/deploy_tokens/#git-clone-a-repository). Deploy tokens can be managed by Maintainers only. In Azure DevOps: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_ENV_SECRET_PERSONAL_ACCESS_TOKEN')}}@dev.azure.com/dbt-labs/awesome_project/_git/awesome_repo" # git HTTPS URL ``` Read more about creating a Personal Access Token [here](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops\&tabs=preview-page#create-a-pat). In Bitbucket: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_USER_NAME')}}:{{env_var('DBT_ENV_SECRET_PERSONAL_ACCESS_TOKEN')}}@bitbucketserver.com/scm/awesome_project/awesome_repo.git" # for Bitbucket Server ``` Read more about creating a Personal Access Token [here](https://confluence.atlassian.com/bitbucketserver/personal-access-tokens-939515499.html). #### Configure subdirectory for packaged projects In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`). packages.yml ```yaml packages: - git: "https://github.com/dbt-labs/dbt-labs-experimental-features" # git URL subdirectory: "materialized-views" # name of subdirectory containing `dbt_project.yml` ``` ##### Local packages A "local" package is a dbt project accessible from your local file system. They're best suited for when there is a common collection of models and macros that you want to share across multiple downstream dbt projects (but each downstream project still has its own unique models, macros, etc). You can install local packages by specifying the project's path. It works best when you nest the project within a subdirectory relative to your current project's directory. packages.yml ```yaml packages: - local: relative/path/to/subdirectory ``` Other patterns may work in some cases, but not always. For example, if you install this project as a package elsewhere, or try running it on a different system, the relative and absolute paths will yield the same results. packages.yml ```yaml packages: # not recommended - support for these patterns vary - local: /../../redshift # relative path to a parent directory - local: /opt/dbt/redshift # absolute path on the system ``` There are a few specific use cases where we recommend using a "local" package: 1. **Monorepo** — When you have multiple projects, each nested in a subdirectory, within a monorepo. "Local" packages allow you to combine projects for coordinated development and deployment. 2. **Testing changes** — To test changes in one project or package within the context of a downstream project or package that uses it. By temporarily switching the installation to a "local" package, you can make changes to the former and immediately test them in the latter for quicker iteration. This is similar to [editable installs](https://pip.pypa.io/en/stable/topics/local-project-installs/) in Python. 3. **Nested project** — When you have a nested project that defines fixtures and tests for a project of utility macros, like [the integration tests within the `dbt-utils` package](https://github.com/dbt-labs/dbt-utils/tree/main/integration_tests). #### What packages are available? To see the library of published dbt packages, check out the [dbt package hub](https://hub.getdbt.com)! #### dbt v2 package compatibility To determine if a package is compatible with dbt v2, visit the [dbt package hub](https://hub.getdbt.com/) and look for the dbt v2-compatible badge, or review the package's [`require-dbt-version` configuration](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md#pin-to-a-range). * Packages with a `require-dbt-version` that equals or contains `2.0.0` are compatible with dbt v2. For example, `require-dbt-version: ">=1.10.0,<3.0.0"`. Even if a package doesn't reflect compatibility in the package hub, it may still work with v2. Work with package maintainers to track updates, and [thoroughly test packages](https://docs.getdbt.com/guides/dbt-package-compat?step=5) that aren't clearly compatible before deploying. * Package maintainers who would like to make their package compatible with v2 can refer to the [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) for instructions. Fivetran package considerations: * The Fivetran `source` and `transformation` packages have been combined into a single package. * If you manually installed source packages like `fivetran/github_source`, you need to ensure `fivetran/github` is installed and deactivate the transformation models. ###### Package compatibility messages Inconsistent v2 warnings and `dbt-autofix` logs dbt v2 warnings and `dbt-autofix` logs may show different messages about package compatibility. If you use [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) while upgrading to v2 in the Studio IDE or dbt VS Code extension, you may see different messages about package compatibility between `dbt-autofix` and v2 warnings. Here's why: * dbt v2 warnings are emitted based on a package's `require-dbt-version` and whether `require-dbt-version` contains `2.0.0`. * Some packages are already v2-compatible even though package maintainers haven't yet updated `require-dbt-version`. * `dbt-autofix` knows about these compatible packages and will not try to upgrade a package that it knows is already compatible. This means that even if you see a v2 warning for a package that `dbt-autofix` identifies as compatible, you don't need to change the package. The message discrepancy is temporary while we implement and roll out `dbt-autofix`'s enhanced compatibility detection to v2 warnings. Here's an example of a v2 warning in the Studio IDE that says a package isn't compatible with v2 but `dbt-autofix` indicates it is compatible: ```text dbt1065: Package 'dbt_utils' requires dbt version [>=1.30,<2.0.0], but current version is 2.0.0. This package may not be compatible with your dbt version. dbt(1065) [Ln 1, Col 1] ``` (Applies to dbt v2.0 and later) #### Ship agent skills in a package A package can share more than models and macros — it can also ship [agent skills](https://docs.getdbt.com/docs/dbt-ai/package-skills.md), which are reusable instructions that a coding agent reads from a `SKILL.md` file. An agent doesn't know that your staging models take a `stg_` prefix, so teams end up copying those conventions between repos, where they drift. Using a package makes agent skills a versioned dependency like everything else. Add a `skills` directory to the package, and every project that installs it gets those skills with `dbt deps`, as long as the installing project sets the `ai_provider` flag. Refer to [Installing agent skills from dbt packages](https://docs.getdbt.com/docs/dbt-ai/package-skills.md) for the setup. #### Advanced package configuration ##### Updating a package When you update a version or revision in your `packages.yml` file, it isn't automatically updated in your dbt project. You should run `dbt deps` to update the package. You may also need to run a [full refresh](https://docs.getdbt.com/reference/commands/run.md) of the models in this package. ##### Uninstalling a package When you remove a package from your `packages.yml` file, it isn't automatically deleted from your dbt project, as it still exists in your `dbt_packages/` directory. If you want to completely uninstall a package, you should either: * delete the package directory in `dbt_packages/`; or * run `dbt clean` to delete *all* packages (and any compiled models), followed by `dbt deps`. ##### Pinning packages Running [`dbt deps`](https://docs.getdbt.com/reference/commands/deps.md) "pins" each package by creating or updating the `package-lock.yml` file in the *project\_root* where `packages.yml` is recorded. * The `package-lock.yml` file contains a record of all packages installed. * If subsequent `dbt deps` runs contain no changes to `dependencies.yml` or `packages.yml`, dbt installs from `package-lock.yml`. For example, if you use a branch name, the `package-lock.yml` file pins to the head commit. If you use a version range, it pins to the latest release. In either case, subsequent commits or versions will **not** be installed. To get new commits or versions, run `dbt deps --upgrade` or add `package-lock.yml` to your .gitignore file. dbt will warn you if you install a package using the `git` syntax without specifying a revision (see below). ##### Configuring packages You can configure the models and seeds in a package from the `dbt_project.yml` file, like so: dbt\_project.yml ```yml vars: snowplow: 'snowplow:timezone': 'America/New_York' 'snowplow:page_ping_frequency': 10 'snowplow:events': "{{ ref('sp_base_events') }}" 'snowplow:context:web_page': "{{ ref('sp_base_web_page_context') }}" 'snowplow:context:performance_timing': false 'snowplow:context:useragent': false 'snowplow:pass_through_columns': [] models: snowplow: +schema: snowplow seeds: snowplow: +schema: snowplow_seeds ``` For example, when using a dataset specific package, you may need to configure variables for the names of the tables that contain your raw data. Configurations made in your project YAML file (`dbt_project.yml`) will override any configurations in a package (either in the project YAML file of the package, or in config blocks). ##### Specifying unpinned Git packages If your project specifies an "unpinned" Git package, you may see a warning like: ```text The git package "https://github.com/dbt-labs/dbt-utils.git" is not pinned. This can introduce breaking changes into your project without warning! ``` This warning can be silenced by setting `warn-unpinned: false` in the package specification. **Note:** This is not recommended. packages.yml ```yaml packages: - git: https://github.com/dbt-labs/dbt-utils.git warn-unpinned: false ``` #### Troubleshooting If you encounter errors while working with dbt packages, see the following FAQs: Why am I receiving a Runtime Error in my packages? If you're receiving the runtime error below in your packages.yml folder, it may be due to an old version of your dbt\_utils package that isn't compatible with your current dbt version. ```shell Running with dbt=xxx Runtime Error Failed to read package: Runtime Error Invalid config version: 1, expected 2 Error encountered in dbt_utils/dbt_project.yml ``` Try updating the old version of the dbt\_utils package in your packages.yml to the latest version found in the [dbt hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/): ```shell packages: - package: dbt-labs/dbt_utils version: xxx ``` If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! \[Error] Could not find my\_project package If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain *any* macros, dbt will raise an error like: ```shell Compilation Error In dispatch: Could not find package 'my_project' ``` This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Parallel microbatch execution Use parallel batch execution to process your microbatch models faster. The microbatch strategy offers the benefit of updating a model in smaller, more manageable batches. Depending on your use case, configuring your microbatch models to run in parallel offers faster processing, in comparison to running batches sequentially. Parallel batch execution means that multiple batches are processed at the same time, instead of one after the other (sequentially) for faster processing of your microbatch models. dbt automatically detects whether a batch can be run in parallel in most cases, which means you don’t need to configure this setting. However, the [`concurrent_batches` config](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md) is available as an override (not a gate), allowing you to specify whether batches should or shouldn’t be run in parallel in specific cases. For example, if you have a microbatch model with 12 batches, you can execute those batches to run in parallel. Specifically they'll run in parallel limited by the number of [available threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). #### Prerequisites To use parallel execution, you must meet the following prerequisites: * Use a supported adapter: Snowflake or BigQuery. * We'll continue to test and add concurrency support for more adapters in the future. * A batch can only be run in parallel if: * The batch is *not* the first batch. * The batch is *not* the last batch. #### How parallel batch execution works After checking for the conditions in the [prerequisites](#prerequisites), and if `concurrent_batches` value isn't set, dbt will intelligently auto-detect if the model invokes the [`{{ this }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md) Jinja function. If it references `{{ this }}`, the batches will run sequentially since `{{ this }}` represents the database of the current model and referencing the same relation causes conflict. Otherwise, if `{{ this }}` isn't detected (and other conditions are met), the batches will run in parallel, which can be overridden when you [set a value for `concurrent_batches`](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md). #### Parallel or sequential execution Choosing between parallel batch execution and sequential processing depends on the specific requirements of your use case. * Parallel batch execution is faster but requires logic independent of batch execution order. For example, if you're developing a data pipeline for a system that processes user transactions in batches, each batch is executed in parallel for better performance. However, the logic used to process each transaction shouldn't depend on the order of how batches are executed or completed. * Sequential processing is slower but essential for calculations like [cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md) in microbatch models. It processes data in the correct order, allowing each step to build on the previous one. #### Configure `concurrent_batches` By default, dbt auto-detects whether batches can run in parallel for microbatch models, and this works correctly in most cases. However, you can override dbt's detection by setting the [`concurrent_batches` config](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md) in your `dbt_project.yml` or model `.sql` file to specify parallel or sequential execution, given you meet all the [conditions](#prerequisites): ##### dbt\_project.yml dbt\_project.yml ```yaml models: +concurrent_batches: true # value set to true to run batches in parallel ``` ##### my\_model.sql models/my\_model.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', event_time='session_start', begin='2020-01-01', batch_size='day', concurrent_batches=true, # value set to true to run batches in parallel ... ) }} select ... ``` --- ### Power BI Preview ### Power BI [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Starter, Enterprise, Enterprise+ The Power BI integration enables you to query the Semantic Layer directly, allowing you to build dashboards with trusted, live data in Power BI. It provides a live connection to the Semantic Layer through Power BI Desktop or Power BI Service. info This integration is currently in preview. Production readiness depends on Microsoft's participation in the Apache Ossie standard. Using Power BI? You can help shape the future of this integration. Ask your Microsoft representative about Ossie support! #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * You are on a supported [dbt release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or on dbt v1.6 or higher. * You installed [Power BI Desktop or Power BI On-premises Data Gateway](https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-custom-connectors). * Power BI Service doesn't natively support custom connectors. To use the connector in Power BI Service, you must install and configure it on an On-premises Data Gateway. * You need your [dbt host](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#3-view-connection-detail), [Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer), and a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to log in. This account should be set up with the Semantic Layer. * You must have a dbt Starter or Enterprise-tier [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Install the connector power bi versions The Power BI connector may be incompatible with older versions of Power BI desktop. For the best results, we recommend installing the most recent version directly from the [Microsoft Store](https://apps.microsoft.com/detail/9ntxr16hnw1t?hl=en-US\&gl=US) or [Download Center](https://www.microsoft.com/en-us/download/details.aspx?id=58494). The Semantic Layer Power BI connector consists of a custom `.pqx` Power BI connector and an ODBC driver. Install both using our Windows installer by following these steps: 1. Download and install the [`.msi` installer](https://github.com/dbt-labs/semantic-layer-powerbi-connector/releases/download/v1.0.0/dbt.Semantic.Layer.for.Power.BI.zip) 2. Run the installer and follow the on-screen instructions to install the ODBC driver and connector onto your Power BI Desktop. ##### Verify installation Note that users on older versions of Power BI may have to [configure the connector](#configure-the-connector) before they can verify the installation. To verify the installation: 1. Open **ODBC Data Sources (64-bit)** file on your computer. 2. Navigate to **System DSN** and verify that the `dbt Labs ODBC DSN` is registered. 3. Navigate to **Drivers** and verify that the `dbt Labs ODBC Driver` is installed. 4. Open Power BI Desktop, navigate to **Settings**, then **Data Source Settings**. Verify that the `dbt Semantic Layer` connector is properly loaded. To allow published reports in Power BI Service to use the connector. An IT admin in your organization needs to install and configure the connector on an On-premises Data Gateway. #### For IT admins This section is for IT admins trying to install the ODBC driver and connector into an On-premises Data Gateway. To allow published reports to use the connector in Power BI Service, an IT Admin must install and configure the connector: 1. Install the ODBC driver and connector into an On-premises Data Gateway. Run the same `.msi` installer used for Power BI Desktop and install it on the machine where your gateway is hosted. 2. Copy connector file to Gateway directory: 1. Locate that `.pqx` file: `C:\Users\<YourUser>\Documents\Power BI Desktop\Custom Connectors\dbtSemanticLayer.pqx`. 2. Copy it to the Power BI On-premises Data Gateway custom connectors directory: `C:\Windows\ServiceProfiles\PBIEgwService\Documents\Power BI Desktop\Custom Connectors`. 3. Verify installation by following the steps from the [install the connector](#verify-installation) section. 4. Enable connector in Power BI Enterprise Gateway: 1. Open the `EnterpriseGatewayConfigurator.exe`. 2. Navigate to **Connectors**. 3. Verify that the `dbt Semantic Layer` connector is installed and active. For more information on how to set up custom connectors in the Power BI On-premises Data Gateway, refer to Power BI’s [official documentation](https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-custom-connectors). #### Configure the connector After installing the connector, you’ll have to configure your project credentials to connect to the Semantic Layer from a report. To configure project credentials in Power BI Desktop: 1. Create a blank report. 2. On the top-left, click on **Get data**. 3. Search for Semantic Layer, then click **Connect**. 4. Fill in your connection details. You can find your Host and Environment ID under the Semantic Layer configuration for your dbt project. tip Make sure you select **DirectQuery** under **Data Connectivity mode** since the Semantic Layer connector does not support **Import** mode. See [Considerations](#considerations) for more details. 5. Click **OK** to proceed. [![Select DirectQuery mode](/img/docs/platform-integrations/sl-pbi/pbi-directquery.jpg?v=2 "Select DirectQuery mode")](#)Select DirectQuery mode 6. On the next screen, paste your service or personal token and then click **Connect**. 7. You should see a side pane with a few "virtual" tables. `ALL` represents all of your defined semantic layer objects. The other tables represent each of your saved queries. Select the one you want to load into your dashboard. Then click **Load**. [![Select tables in the side panel](/img/docs/platform-integrations/sl-pbi/pbi-sidepanel.jpg?v=2 "Select tables in the side panel")](#)Select tables in the side panel Now that you've configured the connector, you can configure published reports in the next section to use the connector. #### Configure published reports After publishing a report and the first time you hit **Publish** on a given report, configure Power BI Service to use your organization’s On-premises Data Gateway to access data from the Semantic Layer: 1. On the top right, click on **Settings > Power BI settings**. [![Navigate to Settings > Power BI Settings](/img/docs/platform-integrations/sl-pbi/pbi-settings.jpg?v=2 "Navigate to Settings > Power BI Settings")](#)Navigate to Settings > Power BI Settings 2. Navigate to the **Semantic models** tab and select your report on the sidebar on the left. 3. Under **Gateway and cloud connections**, select the **On-premises Data Gateway** where your IT admin has installed the Semantic Layer connector. * If the Status is **Not configured correctly**, you’ll have to configure it. [![Configure the gateway connection](/img/docs/platform-integrations/sl-pbi/pbi-gateway-cloud-connections.jpg?v=2 "Configure the gateway connection")](#)Configure the gateway connection 4. Click on the arrow under **Actions** and then, click on **Manually add to gateway**. [![Manually add to gateway](/img/docs/platform-integrations/sl-pbi/pbi-manual-gateway.jpg?v=2 "Manually add to gateway")](#)Manually add to gateway 5. Provide a name for your connection and enter your connection details. * Set the connection as **Encrypted** (Required). Failing to do so will result in the Semantic Layer servers rejecting the connection. [![Set the connection as Encrypted](/img/docs/platform-integrations/sl-pbi/pbi-encrypted.jpg?v=2 "Set the connection as Encrypted")](#)Set the connection as Encrypted 6. Click **Create**. This will run a connection test (unless you choose to skip it). If the connection succeeds, the connection will be saved. You can now go back to your published report on Power BI Service to assert data loads as expected. #### Use the connector This section describes how to use the Semantic Layer connector in Power BI. The Semantic Layer connector creates: * A virtual table for each saved query. * A `METRICS.ALL` table containing all metrics, and dimensions and entities appear as regular dimension columns. These tables do not actually map to an underlying table in your data warehouse. Instead, Power BI sends queries to these tables and (before actually executing on the warehouse) the Semantic Layer servers: * Parse the SQL. * Extract all the queried columns, group bys and filters. * Generates SQL to query your existing tables. * Returns data back to Power BI, which doesn’t know any of this happened. [![Power BI integration diagram](/img/docs/platform-integrations/sl-pbi/sl-pbi.jpg?v=2 "Power BI integration diagram")](#)Power BI integration diagram This allows for very flexible analytics workflows, like drag and drop metrics and slice by dimensions and entities — the Semantic Layer will generate the appropriate SQL to actually query your data source for you. ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). #### Considerations  Not every “column” of METRICS.ALL are compatible with every other column * `METRICS.ALL` combines all your existing metrics, entities and dimensions. Queries must be valid Semantic Layer queries, otherwise they'll fail with MetricFlow query compilation errors. * For saved query tables, all “columns” will be compatible with every other “column” since, by definition, saved queries are valid queries that can be sliced by any of the dimensions present in the query.  The dbt Semantic Layer connector does not support Import mode natively * Use `DirectQuery` mode to ensure compatibility. * `Import` mode tries to select an entire table to import into Power BI, which means it'll likely generate SQL that translates to an invalid Semantic Layer query which will try to query all metrics, dimensions and entities at the same time. * To import data into a PowerBI report, select a valid combination of columns to import, (something that will generate a valid Semantic Layer query). * You can use `Table.SelectColumns` for this: `= Table.SelectColumns(Source{[Item="ALL",Schema="METRICS",Catalog=null]}[Data], {"Total Profit", "Metric Time (Day)"})` * Be aware that all calculations will happen inside of Power BI and won’t pass through Semantic Layer servers. This could lead to incorrect or diverging results. * For example, the Semantic Layer is usually responsible for rolling up cumulative metrics to coarser time granularities. Doing a sum over all the weeks in a year to get a yearly granularity out of a weekly Semantic Layer query will most likely generate incorrect results. Instead, you should query the Semantic Layer directly to get accurate results.  The dbt Semantic Layer connector ignores aggregations defined in Power BI * If you change the aggregation type of a metric from `SUM()` to `COUNT()` or anything else, nothing will change. This is because aggregation functions are defined in the Semantic Layer and we ignore them when translating Power BI generated SQL into Semantic Layer queries. * Aggregations like `Count (Distinct)`, `Standard Deviation`, `Variance`, and `Median` in Power BI may return an error and not work at all.  What actions aren't supported? The following are not supported: * Custom modeling * Joining tables * Creating custom columns within a table * Custom Data Analysis Expressions (DAX) or Power Query (PQ) --- ### Preview new and experimental features in the dbt platform dbt platform dbt Labs often tests experimental features before deciding to continue on the [Product lifecycle](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). You can access experimental features to preview beta features that haven’t yet been released to dbt. You can toggle on or off all experimental features by clicking your account name in the bottom left, selecting **Your profile**, and finding the toggle under **User profile**. Experimental features: * May not be feature-complete or fully stable as we’re actively developing them. * Could be discontinued at any time. * May require feedback from you to understand their limitations or impact. Each experimental feature collects feedback directly in dbt, which may impact dbt Labs' decisions to implement. * May have limited technical support and be excluded from our Support SLAs. * May not have public documentation available. To enable or disable experimental features: 1. Click your account name in the bottom left and select **Your profile**. 2. Under **User profile**, find **Experimental features**. 3. Click **Beta** to toggle the features on or off as shown in the following image. [![Experimental features](/img/docs/dbt-versions/experimental-feats.png?v=2 "Experimental features")](#)Experimental features #### Beta terms and conditions By using or enabling features that are not yet in general release ("Beta Features"), you agree to the [Beta Features Terms and Conditions](https://docs.getdbt.com/assets/files/beta-tc-740ff696113c89c38a96bb70b968775e.pdf). --- ### Product lifecycles dbt Labs manages the lifecycle of features across the cloud-hosted dbt platform and self-hosted v1 and v2. Each feature is assigned a lifecycle status that describes its stability, support level, and availability. Use the tabs below to find the lifecycle stages for the product you're using. Service level objective (SLO) support varies between products and lifecycles. ##### dbt platform dbt platform features adhere to the following lifecycle path: **Beta** In active development. May not be fully stable and breaking changes can occur. Documentation may be incomplete, technical support is limited, and SLOs may not apply. Download the [Beta Terms and Conditions](https://docs.getdbt.com/assets/files/beta-tc-740ff696113c89c38a96bb70b968775e.pdf) for details. If marked `Private`, access must be enabled by dbt Labs. **Preview** Stable and functionally ready for production. Planned additions or non-backward-compatible changes may still occur before GA. Includes documentation, technical support, and SLOs. If marked `Private`, access must be enabled by dbt Labs. **Generally available (GA)** Stable features available to all qualified dbt accounts. SLOs, documentation, and technical support apply. Pricing changes may change or apply. Feature availability may depend on your environment's dbt version. Use a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive the latest GA features. **Deprecated** No longer being actively developed or enhanced. Features continue to function as-is and documentation remains available until the removal date. Technical support no longer applies. **Removed** No longer available on the platform in any capacity. ##### dbt v1 and v2 Self-hosted dbt releases follow semantic versioning. Read more in [About dbt versions](https://docs.getdbt.com/docs/dbt-versions.md). Both v1 and v2 releases adhere to the following lifecycle path: **Undocumented** Open source dbt codebases may have visibility into internal, non-contracted, or intentionally undocumented functionality. Not considered part of the release's product surface area. **Unreleased** Planned for the next minor version prerelease. No commitments on behavior or implementation. Maintainers reserve the right to change or remove it entirely. **Alpha** No commitments on behavior or implementation and not intended for any production work. Use at your own discretion. Maintainers reserve the right to change or remove it entirely. **Beta** First glimpse of net-new features in an upcoming release. Code should work without regressions, but new features may be incomplete or have known edge cases. Changes are not locked and maintainers may still alter or remove them. **Release Candidate** A 2-week window for production-level testing before final release. Features are expected to ship as-is, though maintainers may still address significant bugs before the final release. **Generally Available** Ready for use in production. **Deprecated** No longer actively developed or enhanced. Continues to function as-is until its removal date. --- ### Project dependencies dbt platform | Enterprise, Enterprise+ Available on dbt [Enterprise or Enterprise+](https://www.getdbt.com/pricing) plans. For a long time, dbt has supported code reuse and extension by installing other projects as [packages](https://docs.getdbt.com/docs/build/packages.md). When you install another project as a package, you are pulling in its full source code, and adding it to your own. This enables you to call macros and run models defined in that other project. While this is a great way to reuse code, share utility macros, and establish a starting point for common transformations, it's not a great way to enable collaboration across teams and at scale, especially in larger organizations. dbt Labs supports an expanded notion of `dependencies` across multiple dbt projects: * **Packages** — Familiar and pre-existing type of dependency. You take this dependency by installing the package's full source code (like a software library). * **Projects** — The dbt method to take a dependency on another project. Using a metadata service that runs behind the scenes, dbt resolves references on-the-fly to public models defined in other projects. You don't need to parse or run those upstream models yourself. Instead, you treat your dependency on those models as an API that returns a dataset. The maintainer of the public model is responsible for guaranteeing its quality and stability. #### Prerequisites * Available in [dbt Enterprise or Enterprise+](https://www.getdbt.com/pricing). To use it, designate a [public model](https://docs.getdbt.com/docs/mesh/govern/model-access.md) and add a [cross-project ref](#how-to-write-cross-project-ref). * For the upstream ("producer") project setup: * Configure models in upstream project with [`access: public`](https://docs.getdbt.com/reference/resource-configs/access.md) and have at least one successful job run after defining `access`. * Define a [Production deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) in the upstream project and make sure at least *one deployment job* has run successfully there. This job should generate a [`manifest.json` file](https://docs.getdbt.com/reference/artifacts/manifest-json.md) — it includes the metadata needed for downstream projects. * If the upstream project has a Staging environment, run at least one successful deployment job there to ensure downstream cross-project references resolve correctly. * Each project `name` must be unique in your dbt account. For example, if you have a dbt project (codebase) for the `jaffle_marketing` team, avoid creating projects for `Jaffle Marketing - Dev` and `Jaffle Marketing - Prod`; use [environment-level isolation](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments) instead. * dbt supports [Connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management), available to all dbt users. Connections allows different data platform connections per environment, eliminating the need to duplicate projects. Projects can use multiple connections of the same warehouse type. Connections are reusable across projects and environments. * The `dbt_project.yml` file is case-sensitive, which means the project name must exactly match the name in your `dependencies.yml`. For example, `jaffle_marketing`, not `JAFFLE_MARKETING`. #### Use cases The following setup will work for every dbt project: * Add [any package dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-project-dependencies) to `packages.yml` * Add [any project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-package-dependencies) to `dependencies.yml` However, you may be able to consolidate both into a single `dependencies.yml` file. Read the following section to learn more. ###### About packages.yml and dependencies.yml The `dependencies.yml` file can contain both types of dependencies: "package" and "project" dependencies. * [Package dependencies](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) lets you add source code from someone else's dbt project into your own, like a library. * Project dependencies provide a different way to build on top of someone else's work in dbt. * For private Git packages that need Jinja (for example the [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method)), use `packages.yml` on dbt v1, which supports Jinja in package specs. dbt v2 also supports Jinja in `dependencies.yml`, but use `packages.yml` if the project must also run on dbt v1. Refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). For private packages without Jinja tokens, you can use [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) in `packages.yml` or `dependencies.yml`. If your dbt project doesn't require Jinja in package specifications, you can rename `packages.yml` to `dependencies.yml`. If you need Jinja in package specs (for example an environment variable or the [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method)): * On dbt v1, keep using `packages.yml`. dbt v1 does not render Jinja in `dependencies.yml`. * On the dbt v2, you can use Jinja in `dependencies.yml`. It is recommended to use `packages.yml` if the project must also run on dbt v1. Refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md) for the full matrix. Use the following toggles to understand the differences and determine when to use `dependencies.yml` or `packages.yml` (or both). Refer to the [FAQs](#faqs) for more info.  When to use Project dependencies Project dependencies are designed for the [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) and [cross-project reference](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) workflow: * Use `dependencies.yml` when you need to set up cross-project references between different dbt projects, especially in a dbt Mesh setup. * Use `dependencies.yml` when you want to include both projects and non-private dbt packages in your project's dependencies. * Use `dependencies.yml` for organization and maintainability if you're using both [cross-project refs](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) and [dbt Hub packages](https://hub.getdbt.com/). This reduces the need for multiple YAML files to manage dependencies.  When to use Package dependencies Package dependencies allow you to add source code from someone else's dbt project into your own, like a library: * If you only use packages like those from the [dbt Hub](https://hub.getdbt.com/), remain with `packages.yml`. * Use `packages.yml` when you want to download dbt packages, such as dbt projects, into your root or parent dbt project. Something to note is that it doesn't contribute to the dbt Mesh workflow. * Use `packages.yml` to include packages in your project's dependencies. This includes both public packages, such as those from the [dbt Hub](https://hub.getdbt.com/), and private packages. dbt now supports [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages). * [`packages.yml` supports Jinja rendering](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for historical reasons, allowing dynamic configurations. This can be useful if you need to insert values, like a [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method) from an environment variable, into your package specifications. dbt v2 also supports Jinja in `dependencies.yml`; refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). Previously, to use private Git repositories in dbt, you needed to use a workaround that involved embedding a Git token with Jinja. This is not ideal as it requires extra steps like creating a user and sharing a Git token. We’ve introduced support for [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages-) to address this. #### Define project dependencies If your dbt project relies on models from another project, you can define that relationship using project dependencies. The following steps walk you through specifying project dependencies in dbt: 1. Create a file called `dependencies.yml` at the root of your dbt project. 2. In the `dependencies.yml`, list the upstream dbt project your project depends on as they appear in the `dbt_projects.yml` file. 3. (Optional) Define the specific models you expect from that upstream project to make the dependency explicit. 4. Use [`ref()`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) with the project name to reference upstream models in your SQL. 5. Commit the changes and ensure the dependency is configured in dbt. 6. dbt resolves the dependency using public model metadata from the selected upstream environment and surfaces cross-project lineage in the lineage and DAG (Directed Acyclic Graph) views. Run upstream deployment jobs separately, or configure [job chaining](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion) if you need coordinated execution. ##### Example As an example, let's say you work on the Marketing team at the Jaffle Shop. The name of your team's project is `jaffle_marketing`: dbt\_project.yml ```yml name: jaffle_marketing ``` As part of your modeling of marketing data, you need to take a dependency on two other projects: * `dbt_utils` as a package: A collection of utility macros you can use while writing the SQL for your own models. This package is open-source public and maintained by dbt Labs. * `jaffle_finance` as a project use case: Data models about the Jaffle Shop's revenue. This project is private and maintained by your colleagues on the Finance team. You want to select from some of this project's final models, as a starting point for your own work. Refer to [Use cases](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#use-cases) for information on package and project dependencies. dependencies.yml ```yml packages: - package: dbt-labs/dbt_utils version: 1.1.1 projects: - name: jaffle_finance # case sensitive and matches the 'name' in the 'dbt_project.yml' ``` What's happening here? The `dbt_utils` package — When you run `dbt deps`, dbt will pull down this package's full contents (100+ macros) as source code and add them to your environment. You can then call any macro from the package, just as you can call macros defined in your own project. The `jaffle_finance` projects — This is a new scenario. Unlike installing a package, the models in the `jaffle_finance` project will *not* be pulled down as source code and parsed into your project. Instead, dbt provides a metadata service that resolves references to [**public models**](https://docs.getdbt.com/docs/mesh/govern/model-access.md) defined in the `jaffle_finance` project. ##### Advantages When you're building on top of another team's work, resolving the references in this way has several advantages: * You're using an intentional interface designated by the model's maintainer with `access: public`. * You're keeping the scope of your project narrow, and avoiding unnecessary resources and complexity. This is faster for you and faster for dbt. * You don't need to mirror any conditional configuration of the upstream project such as `vars`, environment variables, or `target.name`. You can reference the Finance team's public models in the upstream environment selected for this dependency. Even if the Finance team makes changes like renaming the model, changing the name of its schema, or [bumping its version](https://docs.getdbt.com/docs/mesh/govern/model-versions.md), your `ref` would still resolve successfully. * You eliminate the risk of accidentally building those models with `dbt run` or `dbt build`. While you can select those models, you can't actually build them. This prevents unexpected warehouse costs and permissions issues. This also ensures proper ownership and cost allocation for each team's models. ##### How to write cross-project ref **Writing `ref`:** Models referenced from a `project`-type dependency must use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models), including the project name: models/marts/roi\_by\_channel.sql ```sql with monthly_revenue as ( select * from {{ ref('jaffle_finance', 'monthly_revenue') }} ), ... ``` (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ###### Cycle detection You can enable bidirectional dependencies across projects so these relationships can go in either direction, meaning that the `jaffle_finance` project can add a new model that depends on any public models produced by the `jaffle_marketing` project, so long as the new dependency doesn't introduce any node-level cycles. dbt checks for cycles across projects and raises errors if any are detected. When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: 1. The `project_a` project runs in a deployment environment and produces public models. 2. The `project_b` project adds `project_a` as a dependency. 3. The `project_b` project runs in a deployment environment and produces public models. 4. The `project_a` project adds `project_b` as a dependency. For more guidance on how to use Mesh, refer to the dedicated [Mesh guide](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) and also our freely available [Mesh learning course](https://learn.getdbt.com/courses/dbt-mesh). ##### Safeguarding production data with staging environments To protect production data, set up a [Staging deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) within your projects. When the upstream project has a Staging environment, cross-project `ref`s from downstream Development and Staging environments resolve to the upstream Staging environment. When the upstream project has no Staging environment, they resolve to the upstream Production environment. By default, references from downstream Production environments resolve to the upstream Production environment. This ensures consistency between environments and adds a layer of security by preventing access to production data during development workflows. Read [Why use a staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#why-use-a-staging-environment) for more information about the benefits. To override this automatic selection, refer to [Route environments across projects](#route-environments-across-projects). ###### Staging with downstream dependencies dbt begins using the Staging environment to resolve cross-project references from downstream projects as soon as it exists in a project without "fail-over" to Production. This means that dbt will consistently use metadata from the Staging environment to resolve references in downstream projects, even if there haven't been any successful runs in the configured Staging environment. To avoid causing downtime for downstream developers, you should define and trigger a job before marking the environment as Staging: 1. Create a new environment, but do NOT mark it as **Staging**. 2. Define a job in that environment. 3. Trigger the job to run, and ensure it completes successfully. 4. Update the environment to mark it as **Staging**. ##### Route environments across projects By default, dbt picks the upstream environment for a cross-project `ref` using the production and staging rules detailed in the previous section. To customize which upstream environment is used, edit the `dependencies.yml` file in the *downstream* project and add the `mesh_environment_routing` field under each upstream project you want to route. You don't change anything in the upstream project. The `ref` in your model code remains unchanged. Availability Environment routing is available on the dbt platform dbt v1 Latest and Compatible, and on dbt v2 Stable and Nightly [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). It ships to the Extended tracks in November 2026. ###### How routing works You set routes per upstream project in the downstream `dependencies.yml`, so one downstream run can point at different environments for different projects. The following table compares the upstream environment dbt uses for each downstream environment: | Downstream environment | Default without routing | With routing | | ---------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------- | | Production | Upstream production | Upstream environment you choose | | Staging | Upstream staging if configured; otherwise production | Upstream environment you choose, such as production | | General | Upstream staging if configured; otherwise production | Upstream environment you choose, such as a specific general environment by ID | | Development | Upstream staging if configured; otherwise production | Upstream environment you choose | If a downstream environment doesn't match a route, dbt uses that project's `default_upstream_environment` (if you set one). With no matching route and no default, dbt keeps the default behavior for that project. You don't need to route every environment. ###### Configure routes In the following example: * `orders_project` sends staging runs to its production environment, sends development runs to whichever environment the `DBT_ROUTE_TO_ID` environment variable names, and sends the downstream general environment to a specific upstream general environment. Any other environment uses production because of the default. * `finance_project` has no default, so downstream environments other than `12001` keep the automatic behavior. dependencies.yml ```yml projects: - name: orders_project default_upstream_environment: production mesh_environment_routing: - in_this_project_environment: staging use_upstream_environment: production - in_this_project_environment: development use_upstream_environment: DBT_ROUTE_TO_ID - in_this_project_environment: 12001 # Downstream general environment ID use_upstream_environment: 34002 # orders_project general environment ID - name: finance_project mesh_environment_routing: - in_this_project_environment: 12001 use_upstream_environment: production ``` The following table describes each key: | Key | Description | Valid values | | ------------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `name` | The upstream project. Must match the `name` in its `dbt_project.yml`. | An existing project name (case-sensitive). | | `mesh_environment_routing` | Ordered, explicit routes for this upstream project. Optional. | A YAML list of route pairs. | | `in_this_project_environment` | The downstream environment where dbt runs. | A numeric environment ID; `production`, `staging`, or `development`; or an environment variable name. | | `use_upstream_environment` | The environment in the upstream project to resolve the `ref` against. | A numeric *deployment* environment ID; `production` or `staging`; or an environment variable name. | | `default_upstream_environment` | The upstream environment to use when no route matches. Set it beside `mesh_environment_routing`. Optional. | Same values as `use_upstream_environment`. | Keep the following rules in mind when configuring routes: * The first matching route in a project entry takes precedence. Avoid duplicate selectors. * A matching route takes precedence over the project's `default_upstream_environment`. * To target a general or other custom deployment environment, use its environment ID. `general` isn't a keyword. * Write the bare variable name (`DBT_ROUTE_TO_ID`), not Jinja delimiters such as `{{ env_var('DBT_ROUTE_TO_ID') }}`. * Define `DBT_ROUTE_TO_ID` in the downstream run or development environment. * Set its value to `production`, `staging`, or the ID of a deployment environment in `orders_project`. ###### Validate routes Routing changes which upstream environment resolves the `ref`. It doesn't copy data or grant access. Before you rely on a route, check the following: * The upstream model is [public](https://docs.getdbt.com/docs/mesh/govern/model-access.md). * The selected upstream environment has public model metadata from a successful job. * Warehouse permissions allow the downstream query. Run `dbt compile --select YOUR_DOWNSTREAM_MODEL` and check the database and schema in the compiled SQL or `target/manifest.json`. Then run `dbt build --select YOUR_DOWNSTREAM_MODEL` to validate warehouse access. Routing isn't deferral Mesh routing chooses an environment in *another project* for a cross-project `ref`. The Studio IDE **Defer** menu and [`--defer`](https://docs.getdbt.com/reference/node-selection/defer.md) use state to resolve unbuilt or unselected nodes in the *current project*. You can use both in the same development workflow. ##### Comparison If you were to instead install the `jaffle_finance` project as a `package` dependency, you would instead be pulling down its full source code and adding it to your runtime environment. This means: * dbt needs to parse and resolve more inputs (which is slower) * dbt expects you to configure these models as if they were your own (with `vars`, env vars, etc) * dbt will run these models as your own unless you explicitly `--exclude` them * You could be using the project's models in a way that their maintainer (the Finance team) hasn't intended There are a few cases where installing another internal project as a package can be a useful pattern: * Unified deployments — In a production environment, if the central data platform team of Jaffle Shop wanted to schedule the deployment of models across both `jaffle_finance` and `jaffle_marketing`, they could use dbt's [selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) to create a new "passthrough" project that installed both projects as packages. * Coordinated changes — In development, if you wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`) on a downstream model (`jaffle_marketing.roi_by_channel`) *before* introducing changes to a staging or production environment, you can install the `jaffle_finance` package as a package within `jaffle_marketing`. The installation can point to a specific git branch, however, if you find yourself frequently needing to perform end-to-end testing across both projects, we recommend you re-examine if this represents a stable interface boundary. These are the exceptions, rather than the rule. Installing another team's project as a package adds complexity, latency, and risk of unnecessary costs. By defining clear interface boundaries across teams, by serving one team's public models as "APIs" to another, and by enabling practitioners to develop with a more narrowly defined scope, we can enable more people to contribute, with more confidence, while requiring less context upfront. #### FAQs Can I define private packages in the dependencies.yml file? It depends on how you're accessing your private packages: * If you're using [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages), you can define them in the `dependencies.yml` file. * If you're using the [git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method), use `packages.yml` on dbt v1 because Jinja isn't supported in `dependencies.yml`. On dbt v2, you can use `dependencies.yml`, but we recommend using `packages.yml` if your project must also run on dbt v1. Refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). Why doesn’t an indirectly referenced upstream public model appear in Explorer? For [project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) in Mesh, [Catalog](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) only displays directly referenced [public models](https://docs.getdbt.com/docs/mesh/govern/model-access.md) from upstream projects, even if an upstream model indirectly depends on another public model. So for example, if: * `project_b` adds `project_a` as a dependency * `project_b`'s model `downstream_c` references `project_a.upstream_b` * `project_a.upstream_b` references another public model, `project_a.upstream_a` Then: * In Explorer, only directly referenced public models (`upstream_b` in this case) appear. * In the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) lineage view, however, `upstream_a` (the indirect dependency) *will* appear because dbt dynamically resolves the full dependency graph. This behavior makes sure that Catalog only shows the immediate dependencies available to that specific project. #### Related docs * Refer to the [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) guide for more guidance on how to use Mesh. * [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) * [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) — manage credentials, env vars, deferral, and dbt v2 versions when developing locally against a Mesh project --- ### Project recommendations dbt platform | Enterprise, Enterprise+ Catalog provides recommendations about your project from the `dbt_project_evaluator` [package](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) using metadata from the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). * Catalog also offers a global view, showing all the recommendations across the project for easy sorting and summarizing. * These recommendations provide insight into how you can create a better-documented, better-tested, and better-built dbt project, creating more trust and less confusion. * For a seamless and consistent experience, recommendations use `dbt_project_evaluator`'s pre-defined settings and don't import customizations applied to your package or project. On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### Recommendations page The Recommendations overview page includes two top-level metrics measuring the test and documentation coverage of the models in your project. * **Model test coverage** — The percent of models in your project (models not from a package or imported via Mesh) with at least one dbt test configured on them. * **Model documentation coverage** — The percent of models in your project (models not from a package or imported via Mesh) with a description. [![Example of the Recommendations overview page with project metrics and the recommendations for all resources in the project](/img/docs/collaborate/dbt-explorer/example-recommendations-overview.png?v=2 "Example of the Recommendations overview page with project metrics and the recommendations for all resources in the project")](#)Example of the Recommendations overview page with project metrics and the recommendations for all resources in the project #### List of rules The following table lists the rules currently defined in the `dbt_project_evaluator` [package](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/). | Category | Name | Description | Package Docs Link | | ------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | Modeling | Direct Join to Source | Model that joins both a model and source, indicating a missing staging model | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#direct-join-to-source) | | Modeling | Duplicate Sources | More than one source node corresponds to the same data warehouse relation | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#duplicate-sources) | | Modeling | Multiple Sources Joined | Models with more than one source parent, indicating lack of staging models | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#multiple-sources-joined) | | Modeling | Root Model | Models with no parents, indicating potential hardcoded references and need for sources | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#root-models) | | Modeling | Source Fanout | Sources with more than one model child, indicating a need for staging models | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#source-fanout) | | Modeling | Unused Source | Sources that are not referenced by any resource | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#unused-sources) | | Performance | Exposure Dependent on View | Exposures with at least one model parent materialized as a view, indicating potential query performance issues | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/performance/#exposure-parents-materializations) | | Testing | Missing Primary Key Test | Models with insufficient testing on the grain of the model. | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/testing/#missing-primary-key-tests) | | Documentation | Undocumented Models | Models without a model-level description | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-models) | | Documentation | Undocumented Source | Sources (collections of source tables) without descriptions | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-sources) | | Documentation | Undocumented Source Tables | Source tables without descriptions | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-source-tables) | | Governance | Public Model Missing Contract | Models with public access that do not have a model contract to ensure the data types | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/governance/#public-models-without-contracts) | #### The Recommendations tab Models, sources, and exposures each also have a **Recommendations** tab on their resource details page, with the specific recommendations that correspond to that resource: [![Example of the Recommendations tab ](/img/docs/collaborate/dbt-explorer/example-recommendations-tab.png?v=2 "Example of the Recommendations tab ")](#)Example of the Recommendations tab --- ### Project variables dbt provides a mechanism called [variables](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) to provide data to models for compilation. Variables allow you to define configurable values for your project instead of hardcoding them in SQL. (Applies to dbt v1.12 and later) You can define variables in three ways: * In the `vars.yml` file (recommended for projects with several variables, or when you need to reference variables from within [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md)) * In the `dbt_project.yml` file * On the command line Variables defined in the `vars.yml` or `dbt_project.yml` act as project-wide defaults. These defaults apply anywhere the variable is referenced. You can override them at runtime using the `--vars` command-line argument, which is useful when you want to change a value for a specific run. For example, when testing with a different date range or running models with environment-specific settings. You might use variables to [configure timezones](https://github.com/dbt-labs/snowplow/blob/0.3.9/dbt_project.yml#L22), set reporting date ranges, [avoid hardcoding table names](https://github.com/dbt-labs/quickbooks/blob/v0.1.0/dbt_project.yml#L23), or otherwise control how models are compiled. To use a variable in a model, hook, or macro, use the `{{ var('...') }}` function. The `var()` function retrieves the value defined in your project or passed using `--vars`. For more information, see [About var function](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md). Note, refer to [YAML tips](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for more YAML information. (Applies to dbt v1.12 and later) ##### Defining variables in `vars.yml` `vars.yml` is a dedicated file for project variables located at the project root. Because dbt parses it *before* `dbt_project.yml`, variables from `vars.yml` are available when `dbt_project.yml` is evaluated, so you can reference them in `dbt_project.yml` using `{{ var('...') }}`. For more information, refer to [Using vars.yml variables in dbt\_project.yml](#using-varsyml-variables-in-dbt_projectyml). To define variables in a `vars.yml` file, add a `vars` config. You can scope these variables globally, or to a specific package imported in your project. The following example shows variables in a `vars.yml` file: vars.yml ```yaml vars: # The `start_date` variable is accessible to all resources start_date: '2016-06-01' # The `platforms` variable is only accessible to resources in the my_dbt_project project my_dbt_project: platforms: ['web', 'mobile'] # The `app_ids` variable is only accessible to resources in the snowplow package snowplow: app_ids: ['marketing', 'app', 'landing-page'] ``` You cannot define variables in both `vars.yml` and `dbt_project.yml`. If both files contain a `vars` block with definitions, dbt raises an error. If `vars.yml` is empty or doesn't exist, variables defined in `dbt_project.yml` are used instead. ###### Using `vars.yml` variables in `dbt_project.yml` Because `vars.yml` is parsed before `dbt_project.yml`, you can reference variables from `vars.yml` within `dbt_project.yml` using `{{ var('...') }}`. Variables referenced in `dbt_project.yml` are resolved only from `vars.yml` or the `--vars` command-line argument, not from the `vars` block in `dbt_project.yml` itself. If `dbt_project.yml` references a variable that is defined only in its own `vars` block, dbt raises an error. For example, you have the following variables defined in `vars.yml`: vars.yml ```yaml vars: schema_name: analytics materialization: table ``` You can reference them in `dbt_project.yml`: dbt\_project.yml ```yaml models: my_dbt_project: +schema: "{{ var('schema_name') }}" +materialized: "{{ var('materialization') }}" ``` ##### Defining variables in `dbt_project.yml` info Jinja is not supported within the `vars` config, and all values will be interpreted literally. To define variables in a dbt project, add a `vars` config to your `dbt_project.yml` file. These `vars` can be scoped globally, or to a specific package imported in your project. dbt\_project.yml ```yaml name: my_dbt_project version: 1.0.0 config-version: 2 vars: # The `start_date` variable will be accessible in all resources start_date: '2016-06-01' # The `platforms` variable is only accessible to resources in the my_dbt_project project my_dbt_project: platforms: ['web', 'mobile'] # The `app_ids` variable is only accessible to resources in the snowplow package snowplow: app_ids: ['marketing', 'app', 'landing-page'] models: ... ``` (Applies to dbt v1.12 and later) You cannot define variables in both `vars.yml` and `dbt_project.yml`. If both files contain a `vars` block with definitions, dbt raises an error. ##### Defining variables on the command line The `dbt_project.yml` file is a great place to define variables that rarely change. When you need to override a variable for a specific run, use the `--vars` command line option. For example, when you want to test with a different date range, run models with environment-specific settings, or adjust behavior dynamically. Use `--vars` to pass one or more variables to a dbt command. Provide the argument as a YAML dictionary string. For example: ```shell $ dbt run --vars '{"event_type": "signup"}' ``` You can use the same `--vars` syntax with other dbt commands, such as `dbt snapshot`: ```shell $ dbt snapshot --select my_snapshot --vars '{"cutoff_date": "2026-01-01"}' ``` Inside a model, snapshot, or macro, access the value using the `var()` function: ```shell select '{{ var("event_type") }}' as event_type ``` When you pass variables using `--vars`, you can access them anywhere you use the `var()` function in your project. You can pass multiple variables at once: ```shell $ dbt run --vars '{event_type: signup, region: us}' ``` If only one variable is being set, the brackets are optional: ```shell $ dbt run --vars 'event_type: signup' ``` The `--vars` argument accepts a YAML dictionary as a string on the command line. YAML is convenient because it does not require strict quoting as with JSON. Both of the following are valid and equivalent: ```shell $ dbt run --vars '{"key": "value", "date": 20180101}' $ dbt run --vars '{key: value, date: 20180101}' ``` Variables defined using `--var`, override values defined in `dbt_project.yml`. This makes `--vars` useful for temporarily overriding configuration without changing your committed project files. For the complete order of precedence (including package-scoped variables and default values defined in `var()`), see [Variable precedence](https://docs.getdbt.com/docs/build/project-variables.md#variable-precedence). You can find more information on defining dictionaries with YAML [here](https://github.com/Animosity/CraftIRC/wiki/Complete-idiot%27s-introduction-to-yaml). ##### Variable precedence (Applies to dbt v1.12 and later) Variables defined with the `--vars` command line argument override variables defined in `vars.yml` or `dbt_project.yml`. They are globally scoped and accessible to the root project and all installed packages. The order of precedence for variable declaration is as follows (highest priority first): 1. The variables defined on the command line with `--vars` 2. The package-scoped variable declaration in the root `vars.yml` file (if present) 3. The global variable declaration in the root `vars.yml` file (if present) 4. The package-scoped variable declaration in the root `dbt_project.yml` file 5. The global variable declaration in the root `dbt_project.yml` file 6. If this node is defined in a package: variable declarations in that package's `dbt_project.yml` file 7. The variable's default argument (if one is provided) `vars.yml` is parsed *before* `dbt_project.yml`, so variables defined in `vars.yml` are available when `dbt_project.yml` is parsed. If `vars.yml` is empty or doesn't exist, variables from `dbt_project.yml` are used instead. When any variable (in `vars.yml`, `dbt_project.yml`, or from the command line) changes, partial parsing is disabled for the next run. If dbt is unable to find a definition for a variable after checking all possible variable declaration places, then a compilation error will be raised. **Note:** Variable scope is based on the node ultimately using that variable. Imagine the case where a model defined in the root project is calling a macro defined in an installed package. That macro, in turn, uses the value of a variable. The variable will be resolved based on the *root project's* scope, rather than the package's scope. --- ### Python models Note that only specific data platforms support `dbt-py` models. Check the [platform configuration pages](https://docs.getdbt.com/reference/resource-configs.md) to confirm if Python models are supported. Python models for Snowflake, BigQuery, and Databricks are supported in [dbt v2](https://docs.getdbt.com/docs/introduction.md). Please refer to the [supported features](https://docs.getdbt.com/docs/dbt/supported-features.md) page to learn more about dbt v2. We encourage you to: * Read [the original discussion](https://github.com/dbt-labs/dbt/discussions/5261) that proposed this feature. * Share your thoughts and ideas on [next steps for Python models](https://github.com/dbt-labs/dbt/discussions/5742). * Join the **#dbt-python-models** channel in the [dbt Community Slack](https://www.getdbt.com/community/join-the-community/). #### Overview dbt Python (`dbt-py`) models can help you solve use cases that can't be solved with SQL. You can perform analyses using tools available in the open-source Python ecosystem, including state-of-the-art packages for data science and statistics. Before, you would have needed separate infrastructure and orchestration to run Python transformations in production. Python transformations defined in dbt are models in your project with all the same capabilities around testing, documentation, and lineage. models/my\_python\_model.py ```python import ... def model(dbt, session): my_sql_model_df = dbt.ref("my_sql_model") final_df = ... # stuff you can't write in SQL! return final_df ``` models/config.yml ```yml models: - name: my_python_model # Document within the same codebase description: My transformation written in Python # Configure in ways that feel intuitive and familiar config: materialized: table tags: ['python'] # Test the results of my Python transformation columns: - name: id # Standard validation for 'grain' of Python results data_tests: - unique - not_null data_tests: # Write your own validation logic (in SQL) for Python results - custom_generic_test ``` [![SQL + Python, together at last](/img/docs/building-a-dbt-project/building-models/python-models/python-model-dag.png?v=2 "SQL + Python, together at last")](#)SQL + Python, together at last The prerequisites for dbt Python models include using an adapter for a data platform that supports a fully featured Python runtime when using dbt v1 or dbt v2 engine. In a dbt Python model, all Python code is executed remotely on the platform. None of it is run by dbt locally. We believe in clearly separating *model definition* from *model execution*. In this and many other ways, you'll find that dbt's approach to Python models mirrors its longstanding approach to modeling data in SQL. We've written this guide assuming that you have some familiarity with dbt. If you've never before written a dbt model, we encourage you to start by first reading [dbt Models](https://docs.getdbt.com/docs/build/models.md). Throughout, we'll be drawing connections between Python models and SQL models, as well as making clear their differences. ##### What is a Python model? A dbt Python model is a function that reads in dbt sources or other models, applies a series of transformations, and returns a transformed dataset. DataFrame operations define the starting points, the end state, and each step along the way. This is similar to the role of CTEs in dbt SQL models. We use CTEs to pull in upstream datasets, define (and name) a series of meaningful transformations, and end with a final `select` statement. You can run the compiled version of a dbt SQL model to see the data included in the resulting view or table. When you `dbt run`, dbt wraps that query in `create view`, `create table`, or more complex DDL to save its results in the database. Instead of a final `select` statement, each Python model returns a final DataFrame. Each DataFrame operation is "lazily evaluated." In development, you can preview its data, using methods like `.show()` or `.head()`. When you run a Python model, the full result of the final DataFrame will be saved as a table in your data warehouse. dbt Python models have access to almost all of the same configuration options as SQL models. You can test and document them, add `tags` and `meta` properties, and grant access to their results to other users. You can select them by their name, file path, configurations, whether they are upstream or downstream of another model, or if they have been modified compared to a previous project state. ##### Defining a Python model Each Python model lives in a `.py` file in your `models/` folder. It defines a function named **`model()`**, which takes two parameters: * **`dbt`**: A class compiled by dbt unique to each model, enables you to run your Python code in the context of your dbt project and DAG. * **`session`**: A class representing your data platform’s connection to the Python backend. The session is needed to read in tables as DataFrames, and to write DataFrames back to tables. In PySpark, by convention, the `SparkSession` is named `spark`, and available globally. For consistency across platforms, we always pass it into the `model` function as an explicit argument called `session`. The `model()` function must return a single DataFrame. On Snowpark (Snowflake), this can be a Snowpark or pandas DataFrame. On BigQuery this can be BigFrames, pandas or Spark datafame. Via PySpark (Databricks), this can be a Spark, pandas, or pandas-on-Spark DataFrame. For more information about choosing between pandas and native DataFrames, see [DataFrame API + syntax](#dataframe-api-and-syntax). When you `dbt run --select python_model`, dbt will prepare and pass in both arguments (`dbt` and `session`). All you have to do is define the function. This is how every single Python model should look: models/my\_python\_model.py ```python def model(dbt, session): ... return final_df ``` ##### Referencing other models Python models participate fully in dbt's directed acyclic graph (DAG) of transformations. Use the `dbt.ref()` method within a Python model to read data from other models (SQL or Python). If you want to read directly from a raw source table, use `dbt.source()`. These methods return DataFrames pointing to the upstream source, model, seed, or snapshot. models/my\_python\_model.py ```python def model(dbt, session): # DataFrame representing an upstream model upstream_model = dbt.ref("upstream_model_name") # DataFrame representing an upstream source upstream_source = dbt.source("upstream_source_name", "table_name") ... ``` Of course, you can `ref()` your Python model in downstream SQL models, too: models/downstream\_model.sql ```sql with upstream_python_model as ( select * from {{ ref('my_python_model') }} ), ... ``` caution Referencing [ephemeral](https://docs.getdbt.com/docs/build/materializations.md#ephemeral) models is currently not supported (see [feature request](https://github.com/dbt-labs/dbt/issues/7288)) From dbt version 1.8, Python models also support dynamic configurations within Python f-strings. This allows for more nuanced and dynamic model configurations directly within your Python code. For example: models/my\_python\_model.py ```python # Previously, attempting to access a configuration value like this would result in None print(f"{dbt.config.get('my_var')}") # Output before change: None # Now you can access the actual configuration value # Assuming 'my_var' is configured to 5 for the current model print(f"{dbt.config.get('my_var')}") # Output after change: 5 ``` This also means you can use `dbt.config.get()` within Python models to ensure that configuration values are effectively retrievable and usable within Python f-strings. #### Configuring Python models Just like SQL models, there are three ways to configure Python models: 1. In `dbt_project.yml`, where you can configure many models at once 2. In a dedicated `.yml` file, within the `models/` directory 3. Within the model's `.py` file, using the `dbt.config()` method Calling the `dbt.config()` method will set configurations for your model within your `.py` file, similar to the `{{ config() }}` macro in `.sql` model files: models/my\_python\_model.py ```python def model(dbt, session): # setting configuration dbt.config(materialized="table") ``` There's a limit to how complex you can get with the `dbt.config()` method. It accepts *only* literal values (strings, booleans, and numeric types) and dynamic configuration. Passing another function or a more complex data structure is not possible. The reason is that dbt statically analyzes the arguments to `config()` while parsing your model without executing your Python code. If you need to set a more complex configuration, we recommend you define it using the [`config` property](https://docs.getdbt.com/reference/resource-properties/config.md) in a properties YAML file. Platform-specific parameters are documented on each adapter's page: * [BigQuery](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#additional-parameters) * [Databricks](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#python-submission-methods) * [Snowflake](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#additional-parameters) ###### Accessing project context dbt Python models don't use Jinja to render compiled code. Python models have limited access to global project contexts compared to SQL models. That context is made available from the `dbt` class, passed in as an argument to the `model()` function. Out of the box, the `dbt` class supports: * Returning DataFrames referencing the locations of other resources: `dbt.ref()` + `dbt.source()` * Accessing the database location of the current model: `dbt.this()` (also: `dbt.this.database`, `.schema`, `.identifier`) * Determining if the current model's run is incremental: `dbt.is_incremental` * Accessing custom values stored in `meta`: `dbt.config.meta_get()` It is possible to extend this context by "getting" them with `dbt.config.get()` after they are configured in the [model's config](https://docs.getdbt.com/reference/model-configs.md). The `dbt.config.get()` method supports dynamic access to configurations within Python models, enhancing flexibility in model logic. This includes inputs such as `var`, `env_var`, and `target`. If you want to use those values for the conditional logic in your model, we require setting them through a dedicated properties YAML file config: models/config.yml ```yml models: - name: my_python_model config: materialized: table target_name: "{{ target.name }}" specific_var: "{{ var('SPECIFIC_VAR') }}" specific_env_var: "{{ env_var('SPECIFIC_ENV_VAR') }}" ``` Then, within the model's Python code, use the `dbt.config.get()` function to *access* values of configurations that have been set: models/my\_python\_model.py ```python def model(dbt, session): target_name = dbt.config.get("target_name") specific_var = dbt.config.get("specific_var") specific_env_var = dbt.config.get("specific_env_var") orders_df = dbt.ref("fct_orders") # limit data in dev if target_name == "dev": orders_df = orders_df.limit(500) ``` ###### Accessing custom meta values To store custom values, use the [`meta` config](https://docs.getdbt.com/reference/resource-configs/meta.md). For example, if you have a model named `my_python_model` and you want to store custom values, you can do the following: models/schema.yml ```yml models: - name: my_python_model config: meta: custom_value: "111" another_value: "abc" ``` Then access them in your Python model using the `dbt.config.meta_get()` method: models/my\_python\_model.py ```python def model(dbt, session): # Access custom values stored in meta directly custom_value = dbt.config.meta_get("custom_value") another_value = dbt.config.meta_get("another_value") # Use your custom values in your model logic orders_df = dbt.ref("fct_orders") ... ``` Alternative approach You can also retrieve meta values using `dbt.config.get("meta")`, which returns the entire meta dictionary. When using this approach, handle the case where `meta` might not be configured: ```python custom_value = dbt.config.get("meta", {}).get("custom_value") ``` ###### Dynamic configurations In addition to the existing methods of configuring Python models, you also have dynamic access to configuration values set with `dbt.config()` within Python models using f-strings. This increases the possibilities for custom logic and configuration management. models/my\_python\_model.py ```python def model(dbt, session): dbt.config(materialized="table") # Dynamic configuration access within Python f-strings, # which allows for real-time retrieval and use of configuration values. # Assuming 'my_var' is set to 5, this will print: Dynamic config value: 5 print(f"Dynamic config value: {dbt.config.get('my_var')}") ``` ##### Materializations Python models support these materializations: * `table` (default) * `incremental` Incremental Python models support all the same [incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) as their SQL counterparts. The specific strategies supported depend on your adapter. As an example, incremental models are supported on BigQuery with Dataproc for the `merge` incremental strategy; the `insert_overwrite` strategy is not yet supported. Python models can't be materialized as `view` or `ephemeral`. Python isn't supported for non-model resource types (like tests and snapshots). For incremental models, like SQL models, you need to filter incoming tables to only new rows of data: ##### Snowpark models/my\_python\_model.py ```python import snowflake.snowpark.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.dateadd("day", F.lit(-3), F.current_timestamp())) ... return df ``` ##### BigQuery DataFrames models/my\_python\_model.py ```python import datetime def model(dbt, session): dbt.config(materialized = "incremental") bdf = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" bdf = bdf[bdf['updated_at'] >= bpd.read_gbq(max_from_this).values[0][0]] # or only rows from the past 3 days bdf = bdf[bdf['updated_at'] >= datetime.date.today() - datetime.timedelta(days=3)] ... return bdf ``` ##### PySpark models/my\_python\_model.py ```python import pyspark.sql.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.date_add(F.current_timestamp(), F.lit(-3))) ... return df ``` #### Python-specific functionality ##### Defining functions In addition to defining a `model` function, the Python model can import other functions or define its own. Here's an example on Snowpark, defining a custom `add_one` function: models/my\_python\_model.py ```python def add_one(x): return x + 1 def model(dbt, session): dbt.config(materialized="table") temps_df = dbt.ref("temperatures") # warm things up just a little df = temps_df.withColumn("degree_plus_one", add_one(temps_df["degree"])) return df ``` Currently, Python functions defined in one dbt model can't be imported and reused in other models. Refer to [Code reuse](#code-reuse) for the potential patterns being considered. ##### Using PyPI packages You can also define functions that depend on third-party packages so long as those packages are installed and available to the Python runtime on your data platform. In this example, we use the `holidays` package to determine if a given date is a holiday in France. The code below uses the pandas API for simplicity and consistency across platforms. The exact syntax, and the need to refactor for multi-node processing, still vary. ##### Snowpark models/my\_python\_model.py ```python import holidays def is_holiday(date_col): # Chez Jaffle french_holidays = holidays.France() is_holiday = (date_col in french_holidays) return is_holiday def model(dbt, session): dbt.config( materialized = "table", packages = ["holidays"] ) orders_df = dbt.ref("stg_orders") df = orders_df.to_pandas() # apply our function # (columns need to be in uppercase on Snowpark) df["IS_HOLIDAY"] = df["ORDER_DATE"].apply(is_holiday) df["ORDER_DATE"].dt.tz_localize('UTC') # convert from Number/Long to tz-aware Datetime # return final dataset (Pandas DataFrame) return df ``` ##### BigQuery DataFrames models/my\_python\_model.py ```python import holidays def model(dbt, session): dbt.config(submission_method="bigframes") data = { 'id': [0, 1, 2], 'name': ['Brian Davis', 'Isaac Smith', 'Marie White'], 'birthday': ['2024-03-14', '2024-01-01', '2024-11-07'] } bdf = bpd.DataFrame(data) bdf['birthday'] = bpd.to_datetime(bdf['birthday']) bdf['birthday'] = bdf['birthday'].dt.date us_holidays = holidays.US(years=2024) return bdf[bdf['birthday'].isin(us_holidays)] ``` ##### PySpark models/my\_python\_model.py ```python import holidays def is_holiday(date_col): # Chez Jaffle french_holidays = holidays.France() is_holiday = (date_col in french_holidays) return is_holiday def model(dbt, session): dbt.config( materialized = "table", packages = ["holidays"] ) orders_df = dbt.ref("stg_orders") df = orders_df.to_pandas_on_spark() # Spark 3.2+ # df = orders_df.toPandas() in earlier versions # apply our function df["is_holiday"] = df["order_date"].apply(is_holiday) # convert back to PySpark df = df.to_spark() # Spark 3.2+ # df = session.createDataFrame(df) in earlier versions # return final dataset (PySpark DataFrame) return df ``` ###### Configuring packages We encourage you to configure required packages and versions so dbt can track them in project metadata. This configuration is required for the implementation on some platforms. If you need specific versions of packages, specify them. models/my\_python\_model.py ```python def model(dbt, session): dbt.config( packages = ["numpy==1.23.1", "scikit-learn"] ) ``` models/config.yml ```yml models: - name: my_python_model config: packages: - "numpy==1.23.1" - scikit-learn ``` ###### User-defined functions (UDFs) You can use the `@udf` decorator or `udf` function to define an "anonymous" function and call it within your `model` function's DataFrame transformation. This is a typical pattern for applying more complex functions as DataFrame operations, especially if those functions require inputs from third-party packages. * [Snowpark Python: Creating UDFs](https://docs.snowflake.com/en/developer-guide/snowpark/python/creating-udfs.html) * [BigQuery DataFrames UDFs](https://cloud.google.com/bigquery/docs/use-bigquery-dataframes#custom-python-functions) * [PySpark functions: udf](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.udf.html) tip You can also define [SQL or Python UDFs](https://docs.getdbt.com/docs/build/udfs.md) as first-class resources under `/functions` with a matching `YAML` file. dbt builds them as part of the DAG, and you reference them from SQL using `{{ function('my_udf') }}`. These UDFs are reusable across tools (BI, notebooks, SQL clients) because they live in your warehouse. ##### Snowpark models/my\_python\_model.py ```python import snowflake.snowpark.types as T import snowflake.snowpark.functions as F import numpy def register_udf_add_random(): add_random = F.udf( # use 'lambda' syntax, for simple functional behavior lambda x: x + numpy.random.normal(), return_type=T.FloatType(), input_types=[T.FloatType()] ) return add_random def model(dbt, session): dbt.config( materialized = "table", packages = ["numpy"] ) temps_df = dbt.ref("temperatures") add_random = register_udf_add_random() # warm things up, who knows by how much df = temps_df.withColumn("degree_plus_random", add_random("degree")) return df ``` **Note:** Due to a Snowpark limitation, it is not currently possible to register complex named UDFs within stored procedures and, therefore, dbt Python models. We are looking to add native support for Python UDFs as a project/DAG resource type in a future release. For the time being, if you want to create a "vectorized" Python UDF via the Batch API, we recommend either: * Writing [`create function`](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-batch.html) inside a SQL macro, to run as a hook or run-operation * [Registering from a staged file](https://docs.snowflake.com/en/developer-guide/snowpark/python/creating-udfs#creating-a-udf-from-a-python-source-file) within your Python model code ##### BigQuery DataFrames models/my\_python\_model.py ```python def model(dbt, session): dbt.config(submission_method="bigframes") # You can also use @bpd.udf @bpd.remote_function(dataset='jialuo_test_us') def my_func(x: int) -> int: return x * 1100 data = {"int": [1, 2], "str": ['a', 'b']} bdf = bpd.DataFrame(data=data) bdf['int'] = bdf['int'].apply(my_func) return bdf ``` ##### PySpark models/my\_python\_model.py ```python import pyspark.sql.types as T import pyspark.sql.functions as F import numpy # use a 'decorator' for more readable code @F.udf(returnType=T.DoubleType()) def add_random(x): random_number = numpy.random.normal() return x + random_number def model(dbt, session): dbt.config( materialized = "table", packages = ["numpy"] ) temps_df = dbt.ref("temperatures") # warm things up, who knows by how much df = temps_df.withColumn("degree_plus_random", add_random("degree")) return df ``` ###### Code reuse To re-use a Python function across multiple dbt models, you can define [Python UDFs](https://docs.getdbt.com/docs/build/udfs.md) under `/functions` with a matching YAML file. These UDFs live in your warehouse and can be reused across tools (BI, notebooks, SQL clients). In the future, we're considering also adding support for Private Python packages. In addition to importing reusable functions from public PyPI packages, many data platforms support uploading custom Python assets and registering them as packages. The upload process looks different across platforms, but your code’s actual `import` looks the same. ❓ dbt questions * How can dbt help users when uploading or initializing private Python assets? Is this a new form of `dbt deps`? * How can dbt support users who want to test custom functions? If defined as UDFs: "unit testing" in the database? If "pure" functions in packages: encourage adoption of `pytest`? 💬 Discussion: ["Python models: package, artifact/object storage, and UDF management in dbt"](https://github.com/dbt-labs/dbt/discussions/5741) ##### DataFrame API and syntax Over the past decade, most people writing [data transformations](https://www.getdbt.com/analytics-engineering/transformation/) in Python have adopted DataFrame as their common abstraction. dbt follows this convention by returning `ref()` and `source()` as DataFrames, and it expects all Python models to return a DataFrame. A DataFrame is a two-dimensional data structure (rows and columns). It supports convenient methods for transforming that data and creating new columns from calculations performed on existing columns. It also offers convenient ways for previewing data while developing locally or in a notebook. That's about where the agreement ends. There are numerous frameworks with their own syntaxes and APIs for DataFrames. The [pandas](https://pandas.pydata.org/docs/) library offered one of the original DataFrame APIs, and its syntax is the most common to learn for new data professionals. Most newer DataFrame APIs are compatible with pandas-style syntax, though few can offer perfect interoperability. This is true for BigQuery DataFrames, Snowpark, and PySpark, which have their own DataFrame APIs. When developing a Python model, you will find yourself asking these questions: **Why pandas?** — It's the most common API for DataFrames. It makes it easy to explore sampled data and develop transformations locally. You can “promote” your code as-is into dbt models and run it in production for small datasets. **Why *not* pandas?** — Performance. pandas runs "single-node" transformations, which cannot benefit from the parallelism and distributed computing offered by modern data warehouses. This quickly becomes a problem as you operate on larger datasets. Some data platforms support optimizations for code written using pandas DataFrame API, preventing the need for major refactors. For example, [pandas on PySpark](https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_ps.html) offers support for 95% of pandas functionality, using the same API while still leveraging parallel processing. ❓ dbt questions * When developing a new dbt Python model, should we recommend pandas-style syntax for rapid iteration and then refactor? * Which open source libraries provide compelling abstractions across different data engines and vendor-specific APIs? * Should dbt attempt to play a longer-term role in standardizing across them? 💬 Discussion: ["Python models: the pandas problem (and a possible solution)"](https://github.com/dbt-labs/dbt/discussions/5738) #### Limitations Python models have capabilities that SQL models do not. They also have some drawbacks compared to SQL models: * **Time and cost.** Python models are slower to run than SQL models, and the cloud resources that run them can be more expensive. Running Python requires more general-purpose compute. That compute might sometimes live on a separate service or architecture from your SQL models. **However:** We believe that deploying Python models via dbt—with unified lineage, testing, and documentation—is, from a human standpoint, **dramatically** faster and cheaper. By comparison, spinning up separate infrastructure to orchestrate Python transformations in production and different tooling to integrate with dbt is much more time-consuming and expensive. * **Syntax differences** are even more pronounced. Over the years, dbt has done a lot, via dispatch patterns and packages such as `dbt_utils`, to abstract over differences in SQL dialects across popular data warehouses. Python offers a **much** wider field of play. If there are five ways to do something in SQL, there are 500 ways to write it in Python, all with varying performance and adherence to standards. Those options can be overwhelming. As the maintainers of dbt, we will be learning from state-of-the-art projects tackling this problem and sharing guidance as we develop it. * **These capabilities are very new.** As data warehouses develop new features, we expect them to offer cheaper, faster, and more intuitive mechanisms for deploying Python transformations. **We reserve the right to change the underlying implementation for executing Python models in future releases.** Our commitment to you is around the code in your model `.py` files, following the documented capabilities and guidance we're providing here. * **Lack of `print()` support.** The data platform runs and compiles your Python model without dbt's oversight. This means it doesn't display the output of commands such as Python's built-in [`print()`](https://docs.python.org/3/library/functions.html#print) function in dbt's logs. *  Alternatives to using print() in Python models The following explains other methods you can use for debugging, such as writing messages to a dataframe column: * Using platform logs: Use your data platform's logs to debug your Python models. * Return logs as a dataframe: Create a dataframe containing your logs and build it into the warehouse. * Develop locally with DuckDB: Test and debug your models locally using DuckDB before deploying them. Here's an example of debugging in a Python model: ```python def model(dbt, session): dbt.config( materialized = "table" ) df = dbt.ref("my_source_table").df() # One option for debugging: write messages to temporary table column # Pros: visibility # Cons: won't work if table isn't building for some reason msg = "something" df["debugging"] = f"My debug message here: {msg}" return df ``` As a general rule, if there's a transformation you could write equally well in SQL or Python, we believe that well-written SQL is preferable: it's more accessible to a greater number of colleagues, and it's easier to write code that's performant at scale. If there's a transformation you *can't* write in SQL, or where ten lines of elegant and well-annotated Python could save you 1000 lines of hard-to-read Jinja-SQL, Python is the way to go. --- ### Ratio metrics Ratio metrics allow you to create a ratio between two metrics. You specify a numerator and a denominator metric. You can optionally apply filters, names, and aliases to both the numerator and denominator when computing the metric. The parameters for ratio metrics are as follows: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------- | | `name` | The name of the metric. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of the metric (cumulative, derived, ratio, or simple). | Required | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `numerator` | The name of the metric used for the numerator. Can be a string (metric name) or a dict with `name`, `filter`, and `alias` properties. | Required | String or Dict | | `denominator` | The name of the metric used for the denominator. Can be a string (metric name) or a dict with `name`, `filter`, and `alias` properties. | Required | String or Dict | ###### Numerator/Denominator dictionary properties The following properties are available for the numerator and denominator dictionary: | Property | Description | Required | Type | | -------- | ------------------------------ | -------- | ------ | | `name` | Name of the metric. | Required | String | | `filter` | Filter to apply to the metric. | Optional | String | | `alias` | Alias for the metric. | Optional | String | The complete specification for ratio metrics is as follows: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: file_name semantic_model: - enabled: true # required - name: my_semantic_model ... rest of config... metrics: - name: my_advanced_ratio_metric type: ratio numerator: name: my_simple_metric filter: "{{ Dimension('my_primary_entity__my_categorical_dimension_column') }} > 10" alias: joel_loves_data denominator: name: my_simple_metric_that_uses_the_other_time_dimension_but_is_also_from_another_semantic_model filter: "{{ Dimension('my_primary_entity__my_categorical_dimension_column') }} < 10" alias: joel_hates_data ``` For advanced data modeling, you can use `fill_nulls_with` and `join_to_timespine` to [set null metric values to zero](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md), ensuring numeric values for every data row. #### Ratio metrics example These examples demonstrate how to create ratio metrics in your model. They cover basic and advanced use cases, including applying filters to the numerator and denominator metrics. ###### Example 1 This example is a basic ratio metric that calculates the ratio of food orders to total orders: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml metrics: - name: food_order_pct description: "The food order count as a ratio of the total order count" label: Food order ratio type: ratio numerator: food_orders denominator: orders ``` ###### Example 2 This example is a ratio metric that calculates the ratio of food orders to total orders, with a filter and alias applied to the numerator. Note that in order to add these attributes, you'll need to use an explicit key for the name attribute too. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml metrics: - name: food_order_pct description: "The food order count as a ratio of the total order count" label: Food order ratio by location type: ratio numerator: name: food_orders filter: location = 'New York' alias: ny_food_orders denominator: name: orders filter: location = 'New York' alias: ny_orders ``` #### Ratio metrics using different semantic models The system will simplify and turn the numerator and denominator into a ratio metric from different semantic models by computing their values in sub-queries. It will then join the result set based on common dimensions to calculate the final ratio. Here's an example of the SQL generated for such a ratio metric. ```sql select subq_15577.metric_time as metric_time, cast(subq_15577.mql_queries_created_test as double) / cast(nullif(subq_15582.distinct_query_users, 0) as double) as mql_queries_per_active_user from ( select metric_time, sum(mql_queries_created_test) as mql_queries_created_test from ( select cast(query_created_at as date) as metric_time, case when query_status in ('PENDING','MODE') then 1 else 0 end as mql_queries_created_test from prod_dbt.mql_query_base mql_queries_test_src_2552 ) subq_15576 group by metric_time ) subq_15577 inner join ( select metric_time, count(distinct distinct_query_users) as distinct_query_users from ( select cast(query_created_at as date) as metric_time, case when query_status in ('MODE','PENDING') then email else null end as distinct_query_users from prod_dbt.mql_query_base mql_queries_src_2585 ) subq_15581 group by metric_time ) subq_15582 on ( ( subq_15577.metric_time = subq_15582.metric_time ) or ( ( subq_15577.metric_time is null ) and ( subq_15582.metric_time is null ) ) ) ``` #### Add filter Users can define constraints on input metrics for a ratio metric by applying a filter directly to the input metric, like so: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml metrics: - name: frequent_purchaser_ratio description: Fraction of active users who qualify as frequent purchasers type: ratio numerator: name: distinct_purchasers filter: | "{{ Dimension('customer__is_frequent_purchaser') }}" alias: frequent_purchasers denominator: name: distinct_purchasers ``` Note the `filter` and `alias` parameters for the metric referenced in the numerator. * Use the `filter` parameter to apply a filter to the metric it's attached to. * The `alias` parameter is used to avoid naming conflicts in the rendered SQL queries when the same metric is used with different filters. * If there are no naming conflicts, the `alias` parameter can be left out. #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Release tracks in dbt platform dbt platform New capabilities in the dbt framework are delivered continuously to dbt platform. Your projects and environments are upgraded automatically on a cadence that you choose, depending on your dbt plan. #### Which release tracks are available? The release tracks available for your project vary depending on the major version of dbt. ##### dbt v2 release tracks The dbt platform has four dbt v2 release tracks available to give you control over stability vs. velocity. *`v2 Stable` is the default for all new dbt v2-powered projects.* | Release track | Cadence | Description | [Plan availability](https://www.getdbt.com/pricing) | API value | | ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- | ----------------- | | **v2 Nightly** | Nightly | The latest nightly build. Includes early access to new features. | All plans | `fusion-nightly` | | **v2 Stable** (default) | Weekly | A weekly release that balances stability and feature access.<br /> | All plans | `fusion-stable` | | **v2 Extended** | Monthly | The previous month's final v2 Stable release. Designed for those who want maximum stability and additional testing time. | Enterprise, Enterprise+ | `fusion-extended` | | **v2 Fallback** | Monthly | The previous month's v2 Extended release. Emergency rollback option for account admins. | Enterprise+ | `fusion-fallback` | ##### dbt v1 release tracks | Release track | Description | Plan availability | API value | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ------------ | | **v1 Latest** | Provides a continuous release of the latest functionality in the dbt platform .<br /><br />Includes early access to new features of the dbt framework before they're available in dbt v1. | All plans | `latest` | | **v1 Compatible** | Provides a monthly release aligned with the most recent open source versions of dbt v1 and adapters, plus functionality exclusively available in the dbt platform.<br /><br />See [v1 Compatible track changelog](https://docs.getdbt.com/docs/dbt-versions/compatible-track-changelog.md) for more information. | Starter, Enterprise, Enterprise+ | `compatible` | | **v1 Extended** | The previous month's **v1 Compatible** release. | Enterprise, Enterprise+ | `extended` | | **v1 Fallback** | The previous month's **v1 Extended** release. | Enterprise+ | `fallback` | ##### API release tracks API naming differences API release track names differ from the names in the UI. They still use legacy names. We plan to add aliases for the new names soon. We'll notify customers when those are available and update this page. Names prefixed with `fusion` are for v2. Names without that prefix are for v1. To configure an environment in the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) to use a release track, set `dbt_version` to the release track name: * `fusion-nightly` * `fusion-stable` (formerly `latest-fusion`) * `fusion-extended` * `fusion-fallback` * `latest` * `compatible` * `extended` #### Which release track should I choose? We recommend that all new projects run dbt v2. Projects still on v1 release tracks are eligible for upgrade to v2 and projects on legacy versions of v1.x should [migrate to release tracks](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md). ##### Choosing a dbt v2 release track Choose **v2 Stable** (the default for new projects) for a weekly release cadence that balances stability and feature access. This is the recommended starting point for most customers migrating to dbt v2. Choose **v2 Nightly** if you want the latest features and are comfortable with a daily release cadence. This track ideal for teams prioritizing the most recent features. Choose **v2 Extended** if your team needs maximum stability and more time to test before updates roll out. This track generally lags v2 Stable by one month and is available for Enterprise and Enterprise+ accounts. Choose **v2 Fallback** as an emergency measure only. This is a temporary rollback option for Enterprise+ accounts, not for ongoing use. Account admins can set the org-wide default release track for new projects via a toggle in **Account Settings**. Individual project owners can then override that default per environment. ##### Choosing a dbt v1 release track Choose the **v1 Latest** release track to continuously receive new features, fixes, performance improvements — latest & greatest dbt. This is the default for all customers on dbt. Choose the **v1 Compatible** and **v1 Extended** release tracks if you need a less-frequent release cadence, the ability to test new dbt releases before they go live in production, and/or ongoing compatibility with the latest open source releases of dbt v1. ##### Using the v2 Fallback release track The **v2 Fallback** release track provides an emergency rollback option for account admins if you suspect a regression in the "v2 Extended" track. This is available only for dbt v2 in Enterprise+ accounts. 1. Go to **Account settings** 2. Click the **Revert to Fallback** release track button in the dbt platform interface. This is not accessible through environment settings. 3. Fill in the details in the **Revert to Fallback** pop-up to confirm and share information with dbt Support. Switching to **v2 Fallback** alerts the dbt Support team, who may reach out to help resolve the issue. This track is meant only as a temporary safety option to unblock you and not for ongoing use. Return to "v2 Extended" or "v2 Stable" (dbt v2) / "v1 Compatible" (dbt v1) once the issue is resolved. [![v2 Fallback release track button in dbt platform](/img/docs/dbt-versions/rollback.png?v=2 "v2 Fallback release track button in dbt platform")](#)v2 Fallback release track button in dbt platform [![v2 Fallback release track popup in dbt platform](/img/docs/dbt-versions/rollback-popup.png?v=2 "v2 Fallback release track popup in dbt platform")](#)v2 Fallback release track popup in dbt platform ##### Common architectures ###### dbt v2 track architectures **Default** — All plans * Prioritize a weekly release cadence that balances stability and new features * Leave all environments on the **v2 Stable** release track (default for new projects at GA) **Velocity-focused** — All plans * Prioritize immediate access to the latest v2 features and fixes * Configure all environments to use the **v2 Nightly** release track **Cautious** — Enterprise, Enterprise+ * Prioritize maximum stability and additional testing time * Configure production environments to use **v2 Extended** (one month behind v2 Stable) * Configure development environments to use **v2 Stable** so developers get access to upcoming changes before they reach production ###### dbt v1 track architectures **Default** — Majority of customers on all plans * Prioritize immediate access to fixes and features * Leave all environments on the **v1 Latest** release track (default configuration) **Hybrid** — Starter, Enterprise, Enterprise+ * Prioritize ongoing compatibility between dbt and dbt v1 for development & deployment using both products in the same dbt projects * Configure all environments to use the **v1 Compatible** release track * Understand that new features will not be available until they are first released in dbt v1 OSS (several months after the **v1 Latest** release track) **Cautious** — Enterprise, Enterprise+, Business Critical * Prioritize "bake in" time for new features & fixes * Configure development & test environments to use the **v1 Compatible** release track * Configure pre-production & production environments to use the **v1 Extended** release track * Understand that new features will not be available until *a month after* they are first released in dbt v1 OSS and the v1 Compatible track. Developers (on **v1 Compatible**) will get access to new features before they can leverage those capabilities in production (on **v1 Extended**), and must be mindful of the additional delay. **Virtual Private dbt or Single Tenant** * Changes to all release tracks roll out as part of dbt instance upgrades once per week #### Upgrading from older versions ##### How to upgrade If you're running a deprecated version of dbt, we recommend that you upgrade or delete your project. For more information about the right path for you, refer to [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md). We recommend you start upgrading projects that are smaller, newer, or more familiar for your team first. They'll be faster and easier to troubleshoot and enable you to more efficiently migrate complex projects. ##### Considerations To learn more about how dbt Labs deploys stable dbt upgrades in a safe manner to dbt, we recommend that you read our blog post: [How we're making sure you can confidently switch to the "Latest" release track in dbt](https://docs.getdbt.com/blog/latest-dbt-stability).  I'm using an older version of dbt in the dbt platform. What should I do? What happens if I do nothing? If you're on version v1.3-v1.7, you need to follow the path for your project(s) as outlined in [Migrate off legacy versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md). If you have not migrated by January 31, 2027, you will lose access to these projects. If you have any questions or concerns, please [contact support](https://docs.getdbt.com/docs/dbt-support.md)  What are other known issues when upgrading from older v1 versions? If you are upgrading from a very old unsupported version of dbt v1, you may run into one of these edge cases after the upgrade to a newer version: * \[v1.3] Customers with non-dbt `.py` files defined within their project directories, such as `models/`. Since v1.3, dbt expects these files be valid [Python models](https://docs.getdbt.com/docs/build/python-models.md). The customer needs to move these files out of their `models/` directory, or ignore them via `.dbtignore` * \[v1.5] Customers who have `--m` in their job definitions, instead of `-m` or `--models`. This autocompletion (`--m[odels]` for `--models`) has never been officially documented or supported. It was an implicit behavior of argparse (CLI library used in dbt-core v1.0-1.4) that is not supported by `click` (the CLI library used in dbt-core since v1.5+). * \[v1.5] Empty invalid `tests` config start raising a validation error]\(/docs/dbt-versions/dbt-upgrade/Older%20versions/upgrading-to-v1.5). Replace empty `tests` config with `tests: []` or remove it altogether. * \[v1.6] Performance optimization to `load_result` means you cannot call it on the same query result multiple times. Instead, save it to a local variable once, and reuse that variable (context: [dbt-labs/dbt#7371](https://github.com/dbt-labs/dbt/pull/7371) You should [contact dbt support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) to request an extension, during which you will need to make those updates.  I see that my account was migrated to v1 Latest. What should I do For the vast majority of customers, there is no further action needed. If you see new failures in your scheduled jobs now that they are running on a newer version of dbt, you may need to update your project code to account for one of the edge cases described on this page. You should [contact dbt support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) to request an extension, during which you will need to make those updates.  What about breaking changes to packages (maintained by dbt Labs or by others)? When we talk about *latest version*, we’re referring to the underlying runtime for dbt, not the versions of packages you’re installing. Our continuous release for dbt includes testing against several popular dbt packages. This ensures that updates we make to `dbt`, adapters, or anywhere else are compatible with the code in those packages. If a new version of a dbt package includes a breaking change (for example, a change to one of the macros in `dbt_utils`), you don’t have to immediately use the new version. In your `packages` configuration (in `dependencies.yml` or `packages.yml`), you can still specify which versions or version ranges of packages you want dbt to install. If you're not already doing so, we strongly recommend [checking `package-lock.yml` into version control](https://docs.getdbt.com/reference/commands/deps.md#predictable-package-installs) for predictable package installs in deployment environments and a clear change history whenever you install upgrades. If you upgrade to the **v1 Latest** release track, and immediately see something that breaks, please [contact support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) and, in the meantime, downgrade back to v1.7. If you’re already on the **v1 Latest** release track, and you observe a breaking change (like something worked yesterday, but today it isn't working, or works in a surprising/different way), please [contact support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) immediately. Depending on your contracted support agreement, the dbt Labs team will respond within our SLA time and we would seek to roll back the change and/or roll out a fix (just as we would for any other part of dbt). This is the same whether or not the root cause of the breaking change is in the project code or in the code of a package. If the package you’ve installed relies on *undocumented* functionality of dbt, it doesn't have the same guarantees as functionality that we’ve documented and tested. However, we will still do our best to avoid breaking them. --- ### Retry your dbt jobs dbt platform If your dbt job run completed with a status of **Error**, you can rerun it from start or from the point of failure in dbt. #### Prerequisites * You have a [dbt account](https://www.getdbt.com/signup). * You must be using [dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) 1.6 or newer. * dbt can successfully parse the project and generate a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) * The most recent run of the job hasn't completed successfully. The latest status of the run is **Error**. * The job command that failed in the run must be one that supports the [retry command](https://docs.getdbt.com/reference/commands/retry.md). #### Rerun an errored job 1. Select **Deploy** from the top navigation bar and choose **Run History.** 2. Choose the job run that has errored. 3. In the **Run Summary** tab on the job’s **Run** page, expand the run step that failed. An ❌ denotes the failed step. 4. Examine the error message and determine how to fix it. After you have made your changes, save and commit them to your [Git repo](https://docs.getdbt.com/docs/platform/git/git-version-control.md). 5. Return to your job’s **Run** page. In the upper right corner, click **Rerun** and choose **Rerun from start** or **Rerun from failure**. If you chose to rerun from the failure point, a **Rerun failed steps** modal opens. The modal lists the run steps that will be invoked: the failed step and any skipped steps. To confirm these run steps, click **Rerun from failure**. The job reruns from the failed command in the previously failed run. A banner at the top of the **Run Summary** tab captures this with the message, "This run resumed execution from last failed step". [![Example of the Rerun options in dbt](/img/docs/deploy/native-retry.gif?v=2 "Example of the Rerun options in dbt")](#)Example of the Rerun options in dbt #### Related content * [Retry a failed run for a job](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retry%20Failed%20Job) API endpoint * [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) * [Jobs](https://docs.getdbt.com/docs/deploy/jobs.md) * [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) --- ### Run the self-hosted dbt MCP server Local development This quickstart walks you through connecting dbt MCP server to your local dbt project. This setup gives you dbt command tools (`run`, `build`, `test`, `compile`, and more) inside your AI assistant. If you'd like to connect to dbt platform with the CLI, see the [OAuth quickstart](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md). To connect to dbt Wizard, see the [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). No clone required You don't need to clone the dbt-mcp repository. Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` — it fetches and runs dbt-mcp for you. #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) * A local dbt project (the folder containing your `dbt_project.yml` file) * dbt installed and available locally For the full list of environment variables and how to enable or disable toolsets, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Step 1: Find your paths You need two values before configuring your MCP client: * `DBT_PROJECT_DIR` — the full path to your dbt project folder (where `dbt_project.yml` lives). For example, if your project name is `jaffle_shop`, the path should be `/Users/yourname/dbt-projects/jaffle_shop`. * `DBT_PATH` — the full path to your dbt executable. ##### macOS/Linux ```bash # Find DBT_PATH which dbt # Example output: /opt/homebrew/bin/dbt # Find DBT_PROJECT_DIR — run from inside your project folder pwd # Example output: /Users/yourname/projects/my_dbt_project ``` ##### Windows ```bash # Find DBT_PATH where dbt # Example output: C:\Python39\Scripts\dbt.exe # Find DBT_PROJECT_DIR — run from inside your project folder cd # Example output: C:\Users\yourname\projects\my_dbt_project ``` Note: Use forward slashes in your configuration: `C:/Python39/Scripts/dbt.exe` #### Step 2: Add to your MCP client Replace the paths below with the values from Step 1: ##### Claude Desktop 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing the paths with your actual values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing the paths with your actual values: ```shell claude mcp add dbt \ -e DBT_PROJECT_DIR=/path/to/your/dbt/project \ -e DBT_PATH=/path/to/your/dbt/executable \ -- uvx dbt-mcp ``` ##### Cursor Click the link below with Cursor open to auto-configure: [Add dbt v1 or dbt v2 to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX1BST0pFQ1RfRElSIjoiL3BhdGgvdG8veW91ci9kYnQvcHJvamVjdCIsIkRCVF9QQVRIIjoiL3BhdGgvdG8veW91ci9kYnQvZXhlY3V0YWJsZSJ9LCJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJkYnQtbWNwIl19) After clicking: 1. Update `DBT_PROJECT_DIR` with the full path to your dbt project. 2. Update `DBT_PATH` with the full path to your dbt executable (from Step 1). 3. Save the configuration. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`, replacing the paths with your actual values: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` 4. Save the file. #### Step 3: Test your setup Ask your AI assistant to run a dbt command (for example, *"Run `dbt compile` on my project"* or *"List all models in my project"*). If dbt MCP is working, the assistant will execute the command against your local project.  Optional: verify from the command line ```bash uvx dbt-mcp ``` If there are no errors, your configuration is correct. Press `Ctrl+C` to stop the server. #### What's available With CLI-only setup, your AI assistant can use: * `dbt run`, `dbt build`, `dbt test`, `dbt compile`, `dbt list`, `dbt parse`, `dbt show` * Model lineage and node details from your local project * Codegen tools (when enabled — see [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md)) Platform features like Semantic Layer, Discovery API, and metadata queries require a dbt platform account. To add them, see [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md). #### Troubleshooting Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). For all troubleshooting topics, see [dbt MCP FAQs and troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). #### Next steps * Add dbt platform features: see [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) * Configure toolsets or disable specific tools: see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) --- ### Run visibility dbt platform You can view the history of your runs and the model timing dashboard to help identify where improvements can be made to jobs. #### Run history The **Run history** dashboard in dbt helps you monitor the health of your dbt project. It provides a detailed overview of all your project's job runs and empowers you with a variety of filters that enable you to focus on specific aspects. You can also use it to review recent runs, find errored runs, and track the progress of runs in progress. You can access it from the top navigation menu by clicking **Deploy** and then **Run history**. The dashboard displays your full run history, including job name, status, associated environment, job trigger, commit SHA, schema, and timing info. dbt developers can access their run history for the last 365 days through the dbt user interface (UI) and API. dbt Labs limits self-service retrieval of run history metadata to 365 days to improve dbt's performance. [![Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.](/img/docs/dbt-platform/deployment/run-history.png?v=2 "Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.")](#)Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more. #### Job run details From the **Run history** dashboard, select a run to view complete details about it. The job run details page displays job trigger, commit SHA, time spent in the scheduler queue, all the run steps and their [logs](#access-logs), [model timing](#model-timing), and more. Click **Rerun now** to rerun the job immediately. An example of a completed run with a configuration for a [job completion trigger](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion): [![Example of run details](/img/docs/dbt-platform/deployment/example-job-details.png?v=2 "Example of run details")](#)Example of run details ##### Run summary tab You can view and download in-progress and historical logs for your dbt runs. This makes it easier for you to debug errors more efficiently. For in-progress steps, dbt platform only displays the tail of the log output — up to the last 1,000 lines or 0.5 MB, whichever comes first. This applies to both console and debug logs. * When logs are truncated, a notice appears at the top of the log. Because only the tail is displayed, a resource that is still running may not appear in the logs until it completes and its output reaches the tail. * When a step is complete, the full log is available. (Applies to dbt v2.0 and later) When a job on dbt v2 finishes, selecting a step displays a structured logs view showing the status of each resource. Nodes are classified into the following categories, and you can expand each node to view its log details: * **Success** * **Reused** * **Failed** * **Warning** * **Running** * **Skipped** * **No-op** For more information about each status, refer to [dbt v2 telemetry and observability](https://docs.getdbt.com/reference/telemetry-observability.md#node-outcome). [![Structured logs in v2](/img/docs/dbt-platform/deployment/v2-logs.png?v=2 "Structured logs in v2")](#)Structured logs in v2 ###### Downloading logs * To download logs for an individual step, select the step in the **Run summary** tab and click **Download** > **Download logs**. * Note that when viewing debug logs, the log output is truncated. To view and export all debug logs for an individual step, click **Download** > **Download all debug logs**. * You can download OpenTelemetry (OTel) logs for dbt v2 job command steps as a Parquet file. The file contains structured step-level log data that you can query or inspect outside dbt. To download it, go to the **Run summary** tab in the job, select a step and click **Download** > **Download OTel log** to export a Parquet file This option only appears when the step emitted an OTel log artifact. Some steps, such as `dbt deps`, don't produce one. For analysis examples, refer to [Querying telemetry data](https://docs.getdbt.com/reference/telemetry-observability.md#querying-telemetry-data). [![Download logs](/img/docs/dbt-platform/deployment/download-logs.png?v=2 "Download logs")](#)Download logs ###### Log size limits dbt enforces cumulative log size limits on run endpoints. If a single step's logs or the total run logs exceed this limit, dbt omits the logs. When dbt omits logs due to size, it displays a **Run logs are too large** banner and shows a message where the logs would usually appear. The run step also displays an **Unknown** status. You can still download omitted logs. If the log file is too large, the download may fail. If that happens, you can [reach out to support](mailto:support@getdbt.com). ##### Lineage tab View the lineage graph associated with the job run so you can better understand the dependencies and relationships of the resources in your project. To view a node's metadata directly in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), select it (double-click) from the graph. [![Example of accessing dbt Catalog from the Lineage tab](/img/docs/collaborate/dbt-explorer/explorer-from-lineage.gif?v=2 "Example of accessing dbt Catalog from the Lineage tab")](#)Example of accessing dbt Catalog from the Lineage tab ##### Model timing tab [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") The **Model timing** tab displays the composition, order, and time each model takes in a job run. This helps you identify bottlenecks in your runs so you can investigate them and potentially make changes to improve performance. You can find it on the [job's run details](#job-run-details). The tab includes the following sections: * [Metric tiles](#metric-tiles) * [Execution timeline](#execution-timeline) * [Concurrency over time](#concurrency-over-time) * [Resource details](#resource-details) ###### Metric tiles Six metric tiles appear at the top of the tab: * **Est. critical path**: The estimated duration of the longest chain of dependent models, and what percentage of total run time it represents. * **Peak concurrency**: The maximum number of models running simultaneously, and at what point in the run it occurred. * **Avg active models**: The average number of models active at any point over the run duration. * **Longest model**: The duration and name of the slowest model in the run. * **Wall clock**: The total elapsed time of the run. * **Latest start**: The name and start time of the last model to start in the run. [![Metric tiles showing key run statistics like estimated critical path, peak concurrency, longest model, and more.](/img/docs/dbt-platform/deployment/model-timing-metric-tiles.png?v=2 "Metric tiles showing key run statistics like estimated critical path, peak concurrency, longest model, and more.")](#)Metric tiles showing key run statistics like estimated critical path, peak concurrency, longest model, and more. ###### Execution timeline A Gantt-style timeline of all resources in the run. Hover over bars to see details. You can customize the view using: * **Group by**: Controls how resources are grouped in the timeline: * **Resource type**: Groups by node type: Model, Test, Snapshot, or Exposure. * **Folder**: Groups by the folder path of the resource in your project. * **Execution phase**: Groups resources into phases 0 through 4 based on execution start time — Phase 0 started earliest, while Phase 4 started latest. * **Thread**: Groups by the dbt execution thread that ran each resource. dbt uses multiple threads to run models in parallel; this view shows which thread handled which resources. * **No grouping**: Lists all resources without any grouping. * **Highlight**: Changes how bars are colored to help you focus on what matters: * **Est. critical path**: Highlights resources on the estimated critical path by graying out all others. * **All equal**: Shows all bars in their resource type color with no additional emphasis. The legend shows the color for each type: Model, Test, Snapshot, and Exposure. * **By duration**: Grays out shorter-running resources and shows longer-running ones in color. * **Search resources**: Filters the timeline to resources matching your search term. [![Execution timeline showing a Gantt-style view of all resources in the run](/img/docs/dbt-platform/deployment/model-timing-timeline.png?v=2 "Execution timeline showing a Gantt-style view of all resources in the run")](#)Execution timeline showing a Gantt-style view of all resources in the run ###### Concurrency over time A stacked bar chart showing model activity over the run duration. Each bar is split into **Active models** and **Queued / ready**, so you can see how many models were running versus waiting at any point in time. It also displays the peak concurrency reached during the run. [![Concurrency over time chart showing active models and queued/ready models throughout the run](/img/docs/dbt-platform/deployment/model-timing-concurrency.png?v=2 "Concurrency over time chart showing active models and queued/ready models throughout the run")](#)Concurrency over time chart showing active models and queued/ready models throughout the run ###### Resource details A paginated, searchable table listing all resources in the run. It includes the following columns: * **Model**: The resource name. * **Start**: The time the resource started. * **End**: The time the resource finished. * **Duration**: How long the resource took to run. * **Execution phase**: A number from 0 to 4 indicating when the resource started relative to others in the run — Phase 0 started earliest, while Phase 4 started latest. * **Est. critical path**: Whether the resource is on the estimated critical path. * **Type**: The resource type (Model, Test, Snapshot, or Exposure), displayed as a color-coded badge. * **Folder**: The folder path of the resource. [![Resource details table showing each model's start time, end time, duration, execution phase, critical path status, type, and folder](/img/docs/dbt-platform/deployment/model-timing-resource-details.png?v=2 "Resource details table showing each model's start time, end time, duration, execution phase, critical path status, type, and folder")](#)Resource details table showing each model's start time, end time, duration, execution phase, critical path status, type, and folder ##### Artifacts tab This provides a list of the artifacts generated by the job run. The files are saved and available for download. [![Example of the Artifacts tab](/img/docs/dbt-platform/example-artifacts-tab.png?v=2 "Example of the Artifacts tab")](#)Example of the Artifacts tab ##### Explain tab The **Explain** tab appears on job runs that used [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). It shows why dbt State rebuilt, reused, or cloned each resource, so you can investigate unexpected behavior or verify that State is working as expected. The tab is available while a run is in progress and updates as resources finish. The tab displays an **Explain results** table with one row per resource. You can search by resource name and download the full results as a text file. Expand a row to see the full decision details. Not all analyses apply to every resource type: | Field | Description | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Resource name** | The name of the resource. | | **Resource type** | The resource type: model, seed, snapshot, test, and so on. | | **Decision** | The reason dbt State rebuilt, reused, or cloned this resource. | | **Run step** | The job command that ran this resource (for example, `dbt build --exclude tag:ml_pipeline`). | | **Table analysis** | Whether the target table already exists in the schema. | | **Query analysis** | Whether the resource query or its upstream queries have changed. | | **Data freshness analysis** | Whether upstream data is fresh or within the configured [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). | [![Explain tab showing the decision breakdown](/img/docs/dbt-platform/deployment/explain-tab.png?v=2 "Explain tab showing the decision breakdown")](#)Explain tab showing the decision breakdown ##### Compare tab [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") The **Compare** tab is shown for [CI job runs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) with the **Run compare changes** setting enabled. It displays details about [the changes from the comparison dbt performed](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) between what's in your production environment and the pull request. To help you better visualize the differences, dbt highlights changes to your models in red (deletions) and green (inserts). From the **Modified** section, you can view the following: * **Overview**: High-level summary about the changes to the models such as the number of primary keys that were added or removed. * **Primary keys**: Details about the changes to the records. * **Modified rows**: Details about the modified rows. Click **Show full preview** to display all columns. * **Columns**: Details about the changes to the columns. To view the dependencies and relationships of the resources in your project more closely, click **View in Catalog** to launch [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). [![Example of the Compare tab](/img/docs/dbt-platform/example-ci-compare-changes-tab.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab [![Example of Modified rows tab which shows you details of the modified rows.](/img/docs/dbt-platform/advanced-ci-modified-rows.png?v=2 "Example of Modified rows tab which shows you details of the modified rows.")](#)Example of Modified rows tab which shows you details of the modified rows. --- ### Saved queries Saved queries are a way to save commonly used queries in MetricFlow. You can group metrics, dimensions, and filters that are logically related into a saved query. Saved queries are nodes and visible in the dbt DAG. Saved queries serve as the foundational building block, allowing you to [configure exports](#configure-exports) in your saved query configuration. Exports takes this functionality a step further by enabling you to [schedule and write saved queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) directly within your data platform using [dbt's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). #### Parameters To create a saved query, refer to the following table parameters. tip Note that we use dot notation (`.`) to indicate whether a parameter is nested within another parameter. For example, `query_params.metrics` means the `metrics` parameter is nested under `query_params`. (Applies to dbt v1.9 and later) | Parameter | Type | Required | Description | | -------------------------- | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` | String | Required | Name of the saved query object. | | `description` | String | Required | A description of the saved query. | | `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | | `config` | String | Optional | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your saved query. Supports `cache`, [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md), `export_as`, [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md), and [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) configurations. | | `config.cache.enabled` | Object | Optional | An object with a sub-key used to specify if a saved query should populate the [cache](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md). Accepts sub-key `true` or `false`. Defaults to `false` | | `limit` | Integer | Optional | The maximum number of rows to return. | | `order_by` | String | Optional | The metrics and group bys to order the query by. | | `query_params` | Structure | Required | Contains the query parameters. | | `query_params.metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | | `query_params.group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | | `query_params.where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | | `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | `exports.name` | String | Required | Name of the export object. | | `exports.config` | List or Structure | Required | A [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property for any parameters specifying the export. | | `exports.config.export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | `exports.config.schema` | String | Optional | The [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) for creating the table or view. This option cannot be used for caching. | | `exports.config.alias` | String | Optional | The table [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) used to write to the table or view. This option cannot be used for caching. | If you use multiple metrics in a saved query, then you will only be able to reference the common dimensions these metrics share in the `group_by` or `where` clauses. Use the entity name prefix with the Dimension object, like `Dimension('user__ds')`. #### Configure saved query Use saved queries to define and manage common Semantic Layer queries in YAML, including metrics and dimensions. Saved queries enable you to organize and reuse common MetricFlow queries within dbt projects. For example, you can group related metrics together for better organization, and include commonly used dimensions and filters. In your saved query config, you can also leverage [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) with the dbt job scheduler to cache common queries, speed up performance, and reduce compute costs. In the following example, you can set the saved query in the `semantic_model.yml` file: semantic\_model.yml (Applies to dbt v1.9 and later) ```yaml saved_queries: - name: test_saved_query description: "{{ doc('saved_query_description') }}" label: Test saved query config: cache: enabled: true | false tags: 'my_tag' query_params: metrics: - simple_metric group_by: - "Dimension('user__ds')" where: - "{{ Dimension('user__ds', 'DAY') }} <= now()" - "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" exports: - name: my_export config: export_as: table alias: my_export_alias schema: my_export_schema_name ``` Note that you can set `export_as` to both the saved query and the exports [config](https://docs.getdbt.com/reference/resource-properties/config.md), with the exports config value taking precedence. If a key isn't set in the exports config, it will inherit the saved query config value. ###### Where clause Use the following syntax to reference entities, dimensions, time dimensions, or metrics in filters and refer to [Metrics as dimensions](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) for details on how to use metrics as dimensions with metric filters: ```yaml filter: | {{ Entity('entity_name') }} filter: | {{ Dimension('primary_entity__dimension_name') }} filter: | {{ TimeDimension('time_dimension', 'granularity') }} filter: | {{ Metric('metric_name', group_by=['entity_name']) }} ``` ###### Project-level saved queries To enable saved queries at the project level, you can set the `saved-queries` configuration in the [`dbt_project.yml` file](https://docs.getdbt.com/reference/dbt_project.yml.md). This saves you time in configuring saved queries in each file: dbt\_project.yml ```yaml saved-queries: my_saved_query: +cache: enabled: true ``` For more information on `dbt_project.yml` and config naming conventions, see the [dbt\_project.yml reference page](https://docs.getdbt.com/reference/dbt_project.yml.md#naming-convention). To build `saved_queries`: * Make sure you set the right [environment variable](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#set-environment-variable) in your environment. * Run the command `dbt build --resource-type saved_query` using the [`--resource-type` flag](https://docs.getdbt.com/reference/global-configs/resource-type.md). #### Configure exports Exports are an additional configuration added to a saved query. They define *how* to write a saved query, along with the schema and table name. Once you've configured your saved query and set the foundation block, you can now configure exports in the `saved_queries` YAML configuration file (the same file as your metric definitions). This will also allow you to [run exports](#run-exports) automatically within your data platform using [dbt's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). The following is an example of a saved query with an export: semantic\_model.yml (Applies to dbt v1.9 and later) ```yaml saved_queries: - name: order_metrics description: Relevant order metrics config: tags: - order_metrics query_params: metrics: - orders - large_order - food_orders - order_total group_by: - Entity('order_id') - TimeDimension('metric_time', 'day') - Dimension('customer__customer_name') - ... # Additional group_by where: - "{{TimeDimension('metric_time')}} > current_timestamp - interval '1 week'" - ... # Additional where clauses exports: - name: order_metrics config: export_as: table # Options available: table, view alias: my_export_alias # Optional - defaults to Export name schema: my_export_schema_name # Optional - defaults to deployment schema ``` #### Run exports Once you've configured exports, you can now take things a step further by running exports to automatically write saved queries within your data platform using [dbt's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). This feature is only available with the [dbt's Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). For more information on how to run exports, refer to the [Exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) documentation. #### FAQs  Can I have multiple exports in a single saved query? Yes, this is possible. However, the difference would be the name, schema, and materialization strategy of the export.  How can I select saved\_queries by their resource type? To include all saved queries in the dbt build run, use the [`--resource-type` flag](https://docs.getdbt.com/reference/global-configs/resource-type.md) and run the command `dbt build --resource-type saved_query`. #### Related docs * [Validate semantic nodes in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) * Configure [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) --- ### Semantic models Tip Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to generate semantic models in the dbt platform or locally in the CLI. Semantic models are the foundation for data definition in MetricFlow, which powers the Semantic Layer: (Applies to dbt v1.12 and later) * Think of semantic models as nodes connected by entities in a semantic graph. You can configure this graph by making metadata annotations on your dbt models to describe their usage in metric calculations. * MetricFlow uses YAML configuration files to create this graph for querying metrics. * Each semantic model corresponds to a dbt model in your DAG, requiring a unique YAML configuration for each semantic model. * Each dbt model can define one semantic model via a `semantic_model` block. Use the optional `name` field if you need a different display name. * Configure semantic models in a YAML file within your dbt project directory, embedded within your model definitions rather than as separate configurations. * You can also define semantic models using [Apache Ossie](https://github.com/apache/ossie) documents, an alternative to dbt's native YAML configuration. For more information, refer to [Ossie semantic layer documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md). 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). Here we describe the Semantic model components with examples: (Applies to dbt v1.12 and later) | Component | Description | Required | Type | | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------ | | [Name](#name) | Choose a name for the semantic model. Avoid using double underscores (\_\_) in the name as they're not supported. Defaults to value of the model name. | Optional | String | | [Time dimension](#time-dimension) | Only `agg_time_dimension` is supported. | Required | Dict | | [Entities](#entities) | Define entities at the column level. For any column that serves as a join key, add an entity block under the column with `type` set to primary, foreign, unique, or natural. Optionally include name, description, label, config, or use expr when the join key differs from the column. | Required | List | | [Primary Entity](#primary-entity) | If a column declares an entity with `type: primary`, you don’t need `primary_entity`. If no column is primary, set the top-level `primary_entity` to name the model’s primary entity. | Optional | String | | [Dimensions](#dimensions) | Different ways to group or slice data for a metric, they can be `time` or `categorical`. | Required | List | | [Derived semantics](#derived_semantics-in-dimensions-and-entities) | Use to create dimensions and entities that don’t match a single, physical column. The `expr` field is required. | Optional | List | | [Simple metrics](#measures) | Aggregations applied to columns in your data model. They can be the final metric or used as building blocks for more complex metrics. | Optional | List | | [Label](#label) | The display name for your semantic model `node`, `dimension`, `entity`, and/or simple metrics. | Optional | String | | `config` | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your metric. Supports [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), and [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) configs. | Optional | Dict | #### Semantic models components The complete spec for semantic models is below: (Applies to dbt v1.12 and later) ```yaml models: - name: the_name_of_the_model # Model name semantic_model: enabled: true name: the_name_of_the_semantic_model # Optional: semantic model name, defaults to model name agg_time_dimension: dimension_name # Required # Entities, dimensions defined at column level columns: - name: entity_column_name entity: type: primary # or foreign name: entity_name description: entity description - name: dimension_column_name dimension: type: categorical # or time name: dimension_name description: dimension description # Simple metrics replace measures metrics: - name: metric_name description: metric description type: simple label: Metric Label agg: sum # Required expr: column_name # Optional, defaults to metric name # Other properties like fill_nulls_with, join_to_timespine can go here ``` The following example displays a complete configuration and detailed descriptions of each field: (Applies to dbt v1.12 and later) ```yaml models: - name: fact_transactions # Model name description: "Transaction fact table at the transaction level. This table contains one row per transaction and includes the transaction timestamp." semantic_model: enabled: true agg_time_dimension: transaction_date columns: - name: transaction_id entity: # Entities included in the table are defined here. MetricFlow will use these columns as join keys. type: primary name: transaction - name: customer_id entity: type: foreign name: customer - name: transaction_date granularity: day dimension: # Dimensions are qualitative values such as names, dates, or geographical data. They provide context to metrics and allow "metric by group" data slicing. type: time - name: order_country dimension: type: categorical name: transaction_location metrics: # Simple metrics are columns we perform an aggregation over. These are inputs to metrics. - name: transaction_total description: "The total value of the transaction." type: simple label: Transaction Total agg: sum expr: transaction_total - name: average_transaction_total description: "The average total sale of the transaction." type: simple label: Average Transaction Total agg: average expr: average_transaction_total - name: median_sales description: "The median sale of the transaction." type: simple label: Median Sales agg: median expr: transaction_total - name: dim_customers # Another model description: "A customer dimension table." semantic_model: enabled: true columns: - name: customer_id entity: type: primary name: customer - name: first_name dimension: type: categorical name: first_name ``` (Applies to dbt v1.12 and later) Semantic models support [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), and [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property in the schema file: ```yml models: - name: orders semantic_model: enabled: true | false group: some_group config: meta: some_key: some_value ``` (Applies to dbt v1.12 and later) ##### Name Define the name of the semantic model. If not provided, this defaults to the value of the model name. Avoid using double underscores (\_\_) in the name as they're not supported. (Applies to dbt v1.12 and later) ##### Time dimension `agg_time_dimension` represents the default time dimensions for simple metrics. This can be overridden by adding the `agg_time_dimension` key directly to a simple metric - see [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) for examples. (Applies to dbt v1.12 and later) ##### Entities To specify the [entities](https://docs.getdbt.com/docs/build/entities.md) in your model, add an `entity` block at the column level with `type` set to primary, foreign, unique, or natural. Optionally, provide `name` and `expr` when the join key’s name differs from the column. ##### Primary entity (Applies to dbt v1.12 and later) MetricFlow requires that all dimensions be tied to an entity to guarantee unique dimension names. If your data source doesn't have a primary entity, you need to assign the entity under the column that serves as the key by adding `type: primary`. If no column is marked as primary, set a top-level `primary_entity` to name the model’s primary entity. The primary entity doesn't necessarily have to map to a column in that table and assigning the name doesn't affect query generation. You can define a primary entity using the following configs: (Applies to dbt v1.12 and later) ```yaml models: - name: bookings_monthly_source semantic_model: enabled: true agg_time_dimension: ds columns: - name: booking_id entity: type: primary name: booking_id - name: ds dimension: type: time name: ds metrics: - name: bookings_monthly description: "Sum of bookings monthly" type: simple label: "Bookings Monthly" agg: sum expr: bookings_monthly ``` ##### Entity types Here are the types of keys: * **Primary** — Only one record per row in the table, and it includes every record in the data platform. * **Unique** — Only one record per row in the table, but it may have a subset of records in the data platform. Null values may also be present. * **Foreign** — Can have zero, one, or multiple instances of the same record. Null values may also be present. * **Natural** — A column or combination of columns in a table that uniquely identifies a record based on real-world data. For example, the `sales_person_id` can serve as a natural key in a `sales_person_department` dimension table. ##### Sample config This example shows a semantic model with three entities and their entity types: `transaction` (primary), `order` (foreign), and `user` (foreign). To reference a desired column, use the actual column name from the model in the `name` parameter. You can also use `name` as an alias to rename the column, and the `expr` parameter to refer to the original column name or a SQL expression of the column. ```yaml entity: - name: transaction type: primary - name: order type: foreign expr: id_order - name: user type: foreign expr: substring(id_order FROM 2) ``` You can refer to entities (join keys) in a semantic model using the `name` parameter. Entity names must be unique within a semantic model, and identifier names can be non-unique across semantic models since MetricFlow uses them for [joins](https://docs.getdbt.com/docs/build/join-logic.md). ##### Dimensions [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) are different ways to organize or look at data. They are effectively the group by parameters for metrics. For example, you might group data by things like region, country, or job title. (Applies to dbt v1.12 and later) MetricFlow takes a dynamic approach when making dimensions available for metrics. Instead of trying to figure out all the possible groupings ahead of time, MetricFlow lets you ask for the dimensions you need and constructs any joins necessary to reach the requested dimensions at query time. The advantage of this approach is that you don't need to set up a system that pre-materializes every possible way to group data, which can be time-consuming and prone to errors. Instead, you define dimensions within the semantic model by nesting them under columns, and they will automatically be made available for valid metrics. Dimensions have the following characteristics: * There are two types of dimensions: categorical and time. Categorical dimensions are for things you can't measure in numbers, while time dimensions represent dates and timestamps. Time dimensions require a column-level granularity. * Dimensions are bound to the primary entity of the semantic model in which they are defined. For example, if a dimension called `full_name` is defined in a model with `user` as a primary entity, then `full_name` is scoped to the `user` entity. To reference this dimension, you would use the fully qualified dimension name `user__full_name`. * The naming of dimensions must be unique in each semantic model with the same primary entity. Dimension names can be repeated if defined in semantic models with a different primary entity. For time groups For semantic models that define metrics, include at least one time dimension column with a `granularity` and set the model’s `agg_time_dimension`. You can override the aggregation time dimension on individual metrics if needed. (Applies to dbt v1.12 and later) ##### `derived_semantics` in dimensions and entities Use the `derived_semantics` key in the model YAML entry when you want to create dimensions and entities that don’t match a single, physical column. The `expr` field is required when using `derived_semantics`. For more information, see [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md#derived_semantics-in-dimensions) and [Entities](https://docs.getdbt.com/docs/build/entities.md#derived_semantics-in-entities). ##### Simple metrics Simple metrics are direct aggregations over columns in your data warehouse using different aggregation types. They serve as building blocks for more complex metrics and can be filtered by dimensions. Simple metrics have various parameters which are listed in a table along with their descriptions and types. For more information, see [Simple metrics](https://docs.getdbt.com/docs/build/simple.md). #### Dependencies (Applies to dbt v1.12 and later) Metric nodes will reflect dependencies on semantic models based on their simple metrics. However, dependencies based on filters should not be reflected in: * [dbt selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) * Visualization of the DAG in dbt-docs and the [integrated development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (IDE). This is because metrics need to source nodes for their `depends_on` attribute from a few different places: * `RATIO` metrics depend on `numerator` and `denominator` metrics, while `DERIVED` metrics depend on `input_metrics`. * `SIMPLE` type metrics depend on their semantic\_model. For example, when you run the command `dbt list --select my_semantic_model+`, it will show you the metrics that belong to the specified semantic model. But there's a condition: Only the metrics that actually use simple metrics or derived metrics from that semantic model will be included in the list. In other words, if a metric only uses a dimension from the semantic model in its filters, it won't be considered as part of that semantic model. #### Related docs (Applies to dbt v1.12 and later) * [About MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) * [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) * [Entities](https://docs.getdbt.com/docs/build/entities.md) * [Simple metrics](https://docs.getdbt.com/docs/build/simple.md) * [Semantic Layer best practices guide](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md) --- ### Set up automatic exposures in Tableau dbt platform | Enterprise, Enterprise+ Set up and automatically populate downstream exposures for supported BI tool integrations, like Tableau. Visualize and orchestrate them through [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects) and the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) for a richer experience. As a data team, it’s critical that you have context into the downstream use cases and users of your data products. By leveraging automatic downstream [exposures](https://docs.getdbt.com/docs/build/exposures.md), you can: * Gain a better understanding of how models are used in downstream analytics, improving governance and decision-making. * Reduce incidents and optimize workflows by linking upstream models to downstream dependencies. * Automate exposure tracking for supported BI tools, ensuring lineage is always up to date. * [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) to refresh the underlying data sources during scheduled dbt jobs, improving timeliness and reducing costs. Orchestrating exposures is a way to ensure that your BI tools are updated regularly using the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). See the [previous page](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) for more information. In dbt, you can configure downstream exposures in two ways: * Manually — Declared [explicitly](https://docs.getdbt.com/docs/build/exposures.md#declaring-an-exposure) in your project’s YAML files. * Automatic — dbt [creates and visualizes downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) automatically for supported integrations, removing the need for manual YAML definitions. These downstream exposures are stored in dbt’s metadata system, appear in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), and behave like manual exposures. However, they don’t exist in YAML files. Tableau Server If you're using Tableau Server, you need to add the [dbt IP addresses for your region](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) to your allowlist. #### Prerequisites To configure automatic downstream exposures, you should meet the following: 1. Your environment and jobs are on a supported [dbt release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). 2. You have a dbt account on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/). 3. You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run. 4. You have [proper permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to edit dbt project or production environment settings. 5. Use Tableau as your BI tool and enable metadata permissions or work with an admin to do so. Compatible with Tableau Cloud or Tableau Server with the Metadata API enabled. 6. You have configured a [Tableau personal access token (PAT)](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) whose creator has permission to view data sources. The PAT inherits the permissions of its creator, so ensure the Tableau user who created the token has [Connect permissions](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_permissions.htm). ##### Considerations Configuring automatic downstream exposures with Tableau have the following considerations: * You can only connect to a single Tableau site on the same server. * If you're using Tableau Server, you need to [allowlist dbt's IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your dbt region. * Tableau dashboards built using custom SQL queries aren't supported. * Downstream exposures sync automatically *once per day* or when a user updates the selected collections. *  The database fully qualified names (FQNs) in Tableau must match those in the dbt build. Tableau's database FQNs (fully qualified names) must match those in the dbt build. To view all expected dependencies in your exposure, the FQNs must match but aren't case-sensitive. For example: | Tableau FQN | dbt FQN | Result | | ---------------------------------- | --------------------------------------- | ---------------------------------------------------------------- | | `analytics.dbt_data_team.my_model` | `analytics.dbt_data_team.my_model` | ✅ Matches and dependencies will display as expected. | | `analytics.dbt_data_team.my_model` | `prod_analytics.dbt_data_team.my_model` | ❌ Doesn't match and not all expected dependencies will display. | To troubleshoot this: 1. In dbt, download the `manifest.json` from the most recent production run that includes the missing dependencies by clicking on the **Artifacts** tab and scrolling to `manifest.json`. 2. Run the following [GraphiQl](https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html#explore-the-metadata-api-schema-using-graphiql) query. Make sure to run the query at `your_tableau_server/metadata/graphiql`, where `your_tableau_server` is the value you provided for the Server URL when [setting up your Tableau integration](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-tableau): ```jsx query { workbooks { name uri id luid projectLuid projectName upstreamTables { id name schema database { name connectionType } } } } ``` 3. Compare database FQNs between `manifest.json` and the GraphiQL response. Make sure that `{database}.{schema}.{name}` matches in both. The following images are examples of FQNs that *match* in both `manifest.json` and the GraphiQL response and aren't case-sensitive: [![manifest.json example with lowercase FQNs.](/img/docs/platform-integrations/auto-exposures/manifest-json-example.png?v=2 "manifest.json example with lowercase FQNs.")](#)manifest.json example with lowercase FQNs. [![GraphiQl response example with uppercase FQNs.](/img/docs/platform-integrations/auto-exposures/graphiql-example.png?v=2 "GraphiQl response example with uppercase FQNs.")](#)GraphiQl response example with uppercase FQNs. 4. If the FQNs don't match, update your Tableau FQNs to match the dbt FQNs. 5. If you're still experiencing issues, please contact [dbt Support](mailto:support@getdbt.com) and share the results with them. #### Set up downstream exposures Set up downstream exposures in [Tableau](#set-up-in-tableau) and [dbt](#set-up-in-dbt-cloud) to ensure that your BI tool's extracts are updated automatically. ##### Set up in Tableau This section explains the steps to configure the integration in Tableau. A Tableau site admin must complete these steps. Once configured in both Tableau and [dbt](#set-up-in-dbt-cloud), you can [view downstream exposures](#view-downstream-exposures) in Catalog. 1. Enable [personal access tokens (PATs)](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) for your Tableau account. [![Enable PATs for the account in Tableau](/img/docs/platform-integrations/auto-exposures/tableau-enable-pat.jpg?v=2 "Enable PATs for the account in Tableau")](#)Enable PATs for the account in Tableau 2. Create a PAT to add to dbt to pull in Tableau metadata for the downstream exposures. When creating the token, you must have permission to access collections/folders, as the PAT only grants access matching the creator's existing privileges. [![Create PATs for the account in Tableau](/img/docs/platform-integrations/auto-exposures/tableau-create-pat.jpg?v=2 "Create PATs for the account in Tableau")](#)Create PATs for the account in Tableau 3. Copy the **Secret** and the **Token name** for use in a later step in dbt. The secret is only displayed once, so store it in a safe location (like a password manager). [![Copy the secret and token name to enter them in dbt](/img/docs/platform-integrations/auto-exposures/tableau-copy-token.jpg?v=2 "Copy the secret and token name to enter them in dbt")](#)Copy the secret and token name to enter them in dbt 4. Copy the **Server URL** and **Sitename**. You can find these in the URL while logged into Tableau. [![Locate the Server URL and Sitename in Tableau](/img/docs/platform-integrations/auto-exposures/tablueau-serverurl.jpg?v=2 "Locate the Server URL and Sitename in Tableau")](#)Locate the Server URL and Sitename in Tableau For example, if the full URL is: `10az.online.tableau.com/#/site/dbtlabspartner/explore`: * The **Server URL** is the fully qualified domain name, in this case: `10az.online.tableau.com` * The **Sitename** is the path fragment right after `site` in the URL, in this case: `dbtlabspartner` 5. With the following items copied, you are now ready to set up downstream exposures in dbt: * ServerURL * Sitename * Token name * Secret ##### Set up in dbt 1. In dbt, navigate to the **Dashboard** of the project you want to add the downstream exposure to and then select **Settings**. 2. Under the **Exposures** section, select **Add lineage integration** to add the Tableau connection. [![Select Add lineage integration to add the Tableau connection.](/img/docs/platform-integrations/auto-exposures/cloud-add-integration.png?v=2 "Select Add lineage integration to add the Tableau connection.")](#)Select Add lineage integration to add the Tableau connection. 3. Enter the details for the exposure connection you collected from Tableau in the [previous step](#set-up-in-tableau) and click **Continue**. Note that all fields are case-sensitive. [![Enter the details for the exposure connection.](/img/docs/platform-integrations/auto-exposures/cloud-integration-details.png?v=2 "Enter the details for the exposure connection.")](#)Enter the details for the exposure connection. 4. Select the collections you want to include for the downstream exposures and click **Save**. [![Select the collections you want to include for the downstream exposures.](/img/docs/platform-integrations/auto-exposures/cloud-select-collections.png?v=2 "Select the collections you want to include for the downstream exposures.")](#)Select the collections you want to include for the downstream exposures. info dbt automatically imports and syncs any workbook within the selected collections. New additions to the collections will appear in the lineage in dbt once per day — after the daily sync and a job run. dbt immediately starts a sync when you update the selected collections list, capturing new workbooks and removing irrelevant ones. 5. dbt imports everything in the collection(s) and you can continue to [view them](#view-downstream-exposures) in Catalog. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. #### View downstream exposures After setting up downstream exposures in dbt, you can view them in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for a richer experience. Navigate to Catalog by selecting **Catalog** from the top-level navigation. You can view downstream exposures from a couple of places: * [Exposures menu](#exposures-menu) * [Exposure detail page](#exposure-detail-page) * [Project lineage](#project-lineage) ##### Exposures menu View all downstream exposures for a project from the Catalog sidebar: 1. In the sidebar, select your project. 2. Under the project, select **Exposure**. You will only see this option if you set up downstream exposures in [Tableau](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-tableau) and [dbt](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-dbt-cloud). The **Exposures** table lists every exposure in the project so you can quickly access and manage them: * **Name**: The name of the exposure. * **Health**: The [data health signal](https://docs.getdbt.com/docs/explore/data-health-signals.md) of the exposure. * **Type**: The type of exposure, such as `dashboard` or `notebook`. * **Owner**: The owner of the exposure. * **Owner email**: The email address of the owner of the exposure. * **Integration**: The BI tool that the exposure is integrated with. * **Exposure mode**: The type of exposure defined: **Auto** or **Manual**. Auto exposures from Tableau appear alongside manual exposures [![View the Exposures table from the Catalog sidebar.](/img/docs/platform-integrations/auto-exposures/explorer-view-resources.png?v=2 "View the Exposures table from the Catalog sidebar.")](#)View the Exposures table from the Catalog sidebar. ##### Exposure detail page After you open the **Exposures** table ([previous section](#exposures-menu)), select an exposure name to open its detail page. On the exposure detail page, you can review metadata, [data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md), description, and lineage. For Tableau auto exposures, use **Open in Dashboard** to open the workbook in Tableau, or **Modify integration** to update your Tableau connection settings. [![View an exposure detail page in Catalog.](/img/docs/platform-integrations/auto-exposures/explorer-view-exposure-detail.png?v=2 "View an exposure detail page in Catalog.")](#)View an exposure detail page in Catalog. ##### Project lineage You can also view exposures from the **Project lineage** view, separate from the **Exposures** table: 1. In the sidebar, select your project. 2. Click **View lineage**. 3. Select an exposure node with the Tableau icon to view its details in the side panel. This view visualizes the dependencies and relationships in your project. For Tableau auto exposures, use **View in Tableau** or **Modify integration** from the side panel. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. #### Orchestrate exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) using the dbt [Cloud job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to proactively refresh the underlying data sources (extracts) that power your Tableau Workbooks. This beta capability requires deployment jobs on [**v1 Latest**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with dbt v1 and *isn't* available on [**v2 Stable**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). For setup details, refer to [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md). * Orchestrating exposures with a `dbt build` job ensures that downstream exposures, like Tableau extracts, are updated regularly and automatically. * You can control the frequency of these refreshes by configuring environment variables. To set up and proactively run exposures with the dbt job scheduler, refer to [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md). --- ### Set up Cost Insights dbt platform | Enterprise, Enterprise+ This guide walks you through setting up Cost Insights to track warehouse compute costs and cost reductions from dbt State and state-aware orchestration across your dbt projects and models. #### Prerequisites Before setting up Cost Insights, ensure you have: * An administrator role. * A supported data warehouse: * Snowflake * BigQuery * Databricks * Amazon Redshift [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") To set up Cost Insights, follow these steps: 1. [Assign required permissions.](#assign-required-permissions) 2. [Configure platform metadata credentials.](#configure-platform-metadata-credentials) 3. [(Optional) Configure Cost Insights settings.](#configure-cost-insights-settings-optional) 4. [(Optional, Snowflake only) Configure custom relation overrides.](#custom-relation-overrides) 5. [(Optional) Enable dbt State or state-aware orchestration in your job settings.](#enable-dbt-state-or-state-aware-orchestration-optional) After completing these setup steps, you can view cost and optimization data across multiple areas of the dbt platform. Refer to [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md) to learn more about the Cost Insights section and how to use it. #### Assign required permissions Users with the following [permission sets](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) can view cost data by default: * Account Admin * Account Viewer * Cost Insights Admin * Cost Insights Viewer * Database Admin * Git Admin * Job Admin * Project Creator * Team Admin For more information on how to assign permissions to users, refer to [About user access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md). #### Configure platform metadata credentials 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Under **Settings**, go to **Connections**. 3. Select an existing connection or create a new connection for the project where you want to enable Cost Insights. 4. Configure platform metadata credentials for your connection. 1. Go to the **Platform metadata credentials** section. Depending on your current configuration, you may see one of the following: * If no platform metadata credentials are configured yet, the credentials form is immediately in edit mode. * If you previously canceled the form and see **Add credentials**, click the button to reopen the form. * If you have multiple connections that reference the same account identifier, you will only be prompted to add platform metadata credentials to one of them. Other connections using the same account identifier will display a message indicating that platform metadata credentials are already configured. 2. Add credentials with permissions to the warehouse tables. Expand each connection to see the permissions required.  Snowflake * `read` permissions to the [`ORGANIZATION_USAGE`](https://docs.snowflake.com/en/sql-reference/organization-usage) and [`ACCOUNT_USAGE`](https://docs.snowflake.com/en/sql-reference/account-usage) schemas * A Snowflake database role assigned the following access: * `ACCOUNT_USAGE.QUERY_HISTORY` * `ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY` * `ORGANIZATION_USAGE.USAGE_IN_CURRENCY_DAILY` (Optional) * `ACCOUNT_USAGE.QUERY_METERING_HISTORY` (Optional; required for [Adaptive Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-adaptive) cost attribution) If `QUERY_METERING_HISTORY` access is not granted, Adaptive Warehouse queries appear as $0 in Cost Insights and a warning is shown in the connection test. For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/account-usage/query_metering_history). note If you don't have access to the `SNOWFLAKE` system database, you can [configure custom relation overrides](#custom-relation-overrides) to point Cost Insights to your own tables or views.  BigQuery * `bigquery.datasets.get` * `bigquery.jobs.create` * `bigquery.jobs.listAll`  Databricks * Access to a [Unity Catalog workspace](https://docs.databricks.com/aws/en/admin/system-tables/#requirements) * `USE` permissions on the catalog and schema * `SELECT` permissions on the following system tables: * [`system.billing.usage`](https://docs.databricks.com/aws/en/admin/system-tables/billing) * [`system.billing.list_prices`](https://docs.databricks.com/aws/en/admin/system-tables/pricing) * [`system.query.history`](https://docs.databricks.com/aws/en/admin/system-tables/query-history) For more information, refer to the Databricks documentation on [granting access to system tables](https://docs.databricks.com/aws/en/admin/system-tables/#grant-access-to-system-tables).  Amazon Redshift[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") By default, Redshift users can only view their own queries. dbt must be able to query all users' queries in `SYS_QUERY_HISTORY` to attribute costs across your dbt runs. Grant one of the following permissions to the platform metadata credentials user: * **`sys:monitor` role** (recommended): ```sql GRANT ROLE sys:monitor TO <user>; ``` * **Unrestricted syslog access**: ```sql ALTER USER <user> SYSLOG ACCESS UNRESTRICTED; ``` For more information, refer to the [Amazon Redshift documentation on enhanced query monitoring permissions](https://docs.aws.amazon.com/redshift/latest/mgmt/metrics-enhanced-query-monitoring.html#metrics-enhanced-query-monitoring-permissions). dbt verifies cross-user visibility during the connection test. If the credentials can only see their own queries, the test fails and cost data will not be processed for that environment. 5. Verify that **Cost insights** is enabled under **Features**. This feature is enabled by default when you configure platform metadata credentials. 6. Click **Save**. #### Configure Cost Insights settings (optional) **Note:** This step is required for Amazon Redshift users. Without a configured price, costs will appear as $0. By default, dbt uses standard warehouse pricing, which you can override if you have custom pricing agreements. Databricks and Amazon Redshift do not have default values. The default values vary by warehouse: | Warehouse | Default values | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | [Snowflake](https://www.snowflake.com/en/pricing-options/) | `price_per_credit` = $3 | | [BigQuery](https://cloud.google.com/bigquery/pricing) | `price_per_slot_hour` = $0.04, `price_per_tib` = $6.25 | | [Databricks](https://docs.databricks.com/aws/en/admin/system-tables/pricing) | dbt queries the `list_prices` system table directly, so there is no default value. | | [Amazon Redshift Serverless](https://aws.amazon.com/redshift/pricing/) | `rpu_price_per_hour` — no default value; costs appear as $0 until configured. | | [Amazon Redshift Provisioned](https://aws.amazon.com/redshift/pricing/) | `node_price_per_hour` — no default value; costs appear as $0 until configured. | <br /> To change the default value: 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Under **Settings**, go to **Connections**. 3. Select the connection where you want to configure Cost Insights settings. 4. Go to the **Cost Insights settings** section. 5. Enter your custom value in the **Price per credit** field. 6. Click **Save**. These custom values will apply to all future cost calculations for this connection. If you clear these values, they will reset to the default warehouse pricing. #### Configure custom relation overrides (optional) [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Private beta feature This feature is available only for Snowflake connections with Cost Insights enabled. To join the private beta, contact your account representative. If you're a Snowflake user whose credentials don't have access to the `SNOWFLAKE` system database, you can use your own tables or views in place of the default [Snowflake system tables](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#snowflake) that Cost Insights queries. You don't need to override all three — any table you don't configure falls back to the Snowflake system default. To configure custom relation overrides: 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Under **Settings**, go to **Connections**. 3. Select the Snowflake connection where you want to configure custom relation overrides. 4. Go to the **Cost Insights settings** section and scroll to **Custom relation overrides**. 5. Enter the fully-qualified name for one or more of the following fields. Each value must be a fully-qualified `database.schema.table` identifier (unquoted or double-quoted). * **Custom query history relation**: Replaces `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY`. * **Custom query attribution history relation**: Replaces `SNOWFLAKE.ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY`. * **Custom usage in currency daily relation**: Replaces `SNOWFLAKE.ORGANIZATION_USAGE.USAGE_IN_CURRENCY_DAILY`. 6. Click **Save**. #### Enable dbt State (optional) State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). Cost Insights displays cost data for your dbt models and jobs without dbt State. However, to understand the impact of optimizations and see cost reductions from model and test reuse, you must enable dbt State in your jobs. For steps on how to enable it, see [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). Cost Insights also reflects cost reductions and efficiency gains from state-aware orchestration if you have it enabled. note For accounts already using dbt State or state-aware orchestration before Cost Insights is enabled, at least one full model build must occur within the last 10 days to establish a baseline for cost reduction calculations. If you don't see cost reduction data, try running a full build to establish the baseline. #### Disable Cost Insights To disable Cost Insights, you must have an administrator role. 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Under **Settings**, go to **Connections**. 3. Select the connection where you want to disable Cost Insights. 4. Go to **Platform metadata credentials** and click **Edit**. 5. Go to the **Features** section and clear the **Cost Insights** option. 6. Click **Save**. --- ### Set up remote MCP dbt platform The remote MCP server uses an HTTP connection and makes calls to dbt-mcp hosted on the cloud-based dbt platform. The self-hosted installation is not required for remote MCP use and is ideal for data consumption use cases. [![Remote dbt MCP server architecture](/img/mcp/remote-dbt-mcp.jpg?v=2 "Remote dbt MCP server architecture")](#)Remote dbt MCP server architecture #### When to use remote MCP The remote MCP server is the ideal choice when: * You don't want to or are restricted from installing additional software (`uvx`, `dbt-mcp`) on your system. * Your primary use case is *consumption-based*: querying metrics, exploring metadata, viewing lineage. * You need access to Semantic Layer, Administrative, and Discovery APIs without maintaining a local dbt project. * You don't need to execute CLI commands. Remote MCP does not support self-hosted dbt CLI commands (`dbt run`, `dbt build`, `dbt test`, and more). If you need to execute dbt commands, use the [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) instead. Remote MCP doesn't require AI features You don't need [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled to use remote MCP. If an admin turns AI features off, [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features) is the only tool hidden and every other tool keeps working. `text_to_sql` is also the only tool that consumes your dbt Copilot action allotment. Other MCP tools don't. If your account runs out of dbt Copilot actions, `text_to_sql` will not work until the limit resets. Other remote MCP tools keep working. If you need help, contact your account manager. #### Choose your auth method | If you need... | Use... | | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | | Fastest first-time setup and your MCP client supports OAuth for HTTP servers (including `execute_sql`) | **[OAuth (remote)](#oauth-remote-mcp)**<br />Available in public beta for Starter, Enterprise, and Enterprise+ accounts | | `execute_sql` with token-based auth, automation, shared setup, or clients without OAuth | **Token-based** with a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) | | Shared or team setup | **Service token** (token-based) | | CI or automation | **Service token** (token-based) | Service tokens do not work for `execute_sql` If you connect with a token instead of OAuth, `execute_sql` only works with a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. To run SQL without creating a PAT, use [OAuth (remote MCP)](#oauth-remote-mcp) instead. #### OAuth (remote MCP) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") OAuth lets you connect to the remote MCP server without copying API tokens into your MCP client, when your client supports OAuth for HTTP-based MCP servers. ##### Prerequisites * Starter, Enterprise, or Enterprise+ account * An MCP client that supports OAuth for remote (HTTP) MCP servers. * Your **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. Check out the next section [MCP URL](#mcp-url) for more information. AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Turning them off doesn't disable remote MCP — it only hides the [`text_to_sql` tool](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features). ##### MCP URL You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). ##### How it works 1. In your MCP client, navigate to the connector or integrations settings and add your MCP URL (refer to [MCP URL](#mcp-url)). For example, if you used Claude or ChatGPT, you would go to: * **Claude (web)**: **Customize** → **Connectors** → **+** → **Add custom connector** * **ChatGPT**: **Settings** → **Apps** → **Create App** * For Claude Desktop and Claude Code, refer to [Integrate Claude with dbt MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md). 2. When prompted in your MCP client, complete sign-in in the browser and approve the requested scopes on the consent screen. 3. Return to your MCP client; subsequent requests use the OAuth session according to your client's behavior. You can use `execute_sql` with OAuth. Add your MCP URL to your client and sign in when your MCP client prompts you. You don't need a personal access token or extra headers. ##### Register your MCP client OAuth requires every client to be registered with dbt platform. There are two paths: * **Dynamic registration (default)** — Clients that implement [Dynamic Client Registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration) self-register the first time a user connects. No admin action required. Most modern MCP clients (such as Claude Desktop, Claude Code, Cursor, and VS Code) support this. * **Manual registration** — For clients that don't support dynamic registration, an account admin registers the client in **Account settings → Integrations → App integrations**. Manually registered clients use [PKCE (RFC 7636)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration) instead of a client secret. Refer to [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration) for the admin walkthrough. Both types appear in **App integrations** in dbt platform, where admins can review and audit each connected client. ##### Scopes you'll consent to The first time you connect, dbt shows a consent screen listing the scopes (the specific permissions the client is allowed to use) the MCP client is requesting. Scopes act as a **filter on your existing permissions** — they don't grant new access. You can also choose whether the client gets access to all projects or only selected projects. For the full list of scopes and what each one allows, refer to [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent). For sessions, refresh tokens, revoking access, and audit logging, refer to [Connect apps with OAuth](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md). ##### Limitations * Remote MCP doesn't support self-hosted dbt CLI commands (like `dbt run`, `dbt build`, `dbt test`, and more) or local project access; use the [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) for those workflows. For client-specific steps, refer to [Integrate Claude with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md), [Integrate Cursor with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md), [Integrate Snowflake Cortex with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-snowflake-cortex.md), or [Integrate VS Code with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md). #### Token-based authentication Token-based authentication lets you connect to the remote MCP server without OAuth by passing a PAT or service token in your MCP client config. Use it when your client doesn't support OAuth for HTTP-based MCP servers, when you need a shared or CI setup, or when you need `execute_sql`, which requires a PAT. Currently, token-based access is all-or-nothing: a token that passes the permission check gets every tool. If you want individual tools filtered to a narrower set of permissions, use [OAuth](#oauth-remote-mcp) instead. This also differs from the [self-hosted server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md), which authorizes each toolset independently as tools are called. Required permission set Token-based connections currently require the `Developer` permission set, which can only be assigned to a service token on Enterprise and Enterprise+ plans. Starter and legacy Team plans don't support it and must use [OAuth](#oauth-remote-mcp) instead. For the permission sets available on each plan, refer to [Service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). ##### Setup instructions AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Turning them off doesn't disable remote MCP — it only hides the [`text_to_sql` tool](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features). 1. Obtain the following information from dbt platform: * **dbt platform host**: Use this to form the full URL. For example, replace `YOUR_DBT_HOST_URL` here: `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/`. It may look like: `https://cloud.getdbt.com/api/ai/v1/mcp/`. If you have a multi-cell account, the host URL will be in the `ACCOUNT_PREFIX.us1.dbt.com` format. For more information, refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). * **Production environment ID**: From **Orchestration** → **Environments** in dbt platform. Use this to set an `x-dbt-prod-environment-id` header. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **Token**: Generate either a personal access token (PAT) or a service token. The token must have Developer access on the project that owns the environment you set in `x-dbt-prod-environment-id`. A token without it is rejected with a `401` when the MCP client connects, rather than connecting with reduced functionality. For a service token, assign the `Developer` permission set, scoped either to that project or to all projects — `Developer` on its own is sufficient. For a PAT, create the token as a user who has Developer access on that project; PATs don't take permission sets, they inherit the permissions of the user who created them. * If you plan to use `execute_sql`, you must use a [Personal Access Token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens *do not* work for this tool. For other tools that require `x-dbt-user-id`, a PAT is also required. 2. For the remote MCP, you will pass on headers through the JSON blob to configure required fields: ###### Configuration for APIs and SQL tools | Header | Required | Description | | ------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Authorization | Required | Your [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) from the dbt platform.<br />**Note**: The token must have Developer access on the project that owns the environment in `x-dbt-prod-environment-id`, or it's rejected with a `401` at connection. Service tokens need the `Developer` permission set, which is sufficient on its own; PATs inherit the permissions of the user who created them, so create the PAT as a user with Developer access on that project. When using the Semantic Layer, we recommend using a PAT.<br /><br />The value must be in the format `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`, replacing `YOUR_DBT_ACCESS_TOKEN` with your actual token. | | x-dbt-prod-environment-id | Required | Your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. | ###### Additional configuration for SQL tools | Header | Required | Description | | ------------------------ | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | x-dbt-dev-environment-id | Required for `execute_sql` | Your dbt platform development environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. | | x-dbt-user-id | Required for `execute_sql` | Your dbt platform user ID. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. | ###### Additional configuration for dbt v2 tools By default, dbt v2 tools use the environment you set in `x-dbt-prod-environment-id` for model and table metadata. | Header | Required | Description | | -------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | x-dbt-dev-environment-id | Required | Your dbt platform development environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. | | x-dbt-user-id | Required | Your dbt platform user ID. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. | | x-dbt-fusion-disable-defer | Optional | Default: `false`. When set to `true`, dbt v2 tools will not defer to the production environment and use the models and table metadata from the development environment (`x-dbt-dev-environment-id`) instead. | ###### Configuration to disable tools | Header | Required | Description | | ---------------------- | -------- | ---------------------------------------------------------------------------------------------------- | | x-dbt-disable-tools | Optional | A comma-separated list of tools to disable. For instance: `get_all_models,text_to_sql,list_entities` | | x-dbt-disable-toolsets | Optional | A comma-separated list of toolsets to disable. For instance: `semantic_layer,sql,discovery` | 3. After establishing which headers you need, you can follow the [examples](https://github.com/dbt-labs/dbt-mcp/tree/main/examples) to create your own agent. #### Examples The MCP protocol is programming language and framework agnostic, so use whatever helps you build agents. If you use [OAuth (remote MCP)](#oauth-remote-mcp) in Beta, you only need your MCP URL. If you use token-based authentication, add the headers in the examples below. Configuration varies by client — select your tool in the following tabs and replace the placeholder values with your own: ##### OAuth ##### Claude Code ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Cursor ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Token-based ##### Claude Code ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Cursor ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Gemini ```json { "mcpServers": { "dbt": { "httpUrl": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` Use numeric IDs, not full URLs Header values like `x-dbt-prod-environment-id` and `x-dbt-user-id` expect numeric IDs, not full URLs. The host in the `url` field should include `https://`, but ID headers must be integers only: ```bash # ✅ Correct "url": "https://cloud.getdbt.com/api/ai/v1/mcp" "x-dbt-prod-environment-id": "54321" "x-dbt-user-id": "123" # ❌ Wrong — don't paste full URLs into ID headers "x-dbt-prod-environment-id": "https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321" "x-dbt-user-id": "https://cloud.getdbt.com/settings/profile" ``` For other MCP clients (Codex, Windsurf, and so on), refer to your client's MCP configuration docs for the correct key format. For self-hosted MCP, use environment variables to configure your setup; refer to the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Related docs Step-by-step client setup (including Cursor, VS Code, and Claude) is in: * [Integrate Cursor with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md) * [Integrate VS Code with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md) * [Integrate Claude with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md) * [Integrate Snowflake Cortex with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-snowflake-cortex.md) --- ### Set up self-hosted MCP Local development [The self-hosted dbt MCP server](https://github.com/dbt-labs/dbt-mcp) runs on your machine and supports dbt v1, dbt v2, and dbt platform CLI. You can use it with or without a dbt platform account. No clone required You don't need to clone the dbt-mcp repository to use self-hosted MCP. [Install uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp`, which fetches and runs dbt-mcp for you. If you'd like to contribute to dbt MCP, clone the [dbt-mcp repo](https://github.com/dbt-labs/dbt-mcp) and contribute away! #### Tool requirements at a glance Use this table to understand what each toolset needs and whether it works with or without a dbt platform account: | Toolset | Required variables | Works with dbt platform | Works without dbt platform | | ----------------------------- | -------------------------------------------------------------- | ----------------------- | -------------------------- | | dbt CLI | `DBT_PROJECT_DIR`, `DBT_PATH` | Yes | Yes | | Semantic Layer | `DBT_HOST`, `DBT_TOKEN`, `DBT_PROD_ENV_ID` | Yes | No | | Discovery API | `DBT_HOST`, `DBT_TOKEN`, `DBT_PROD_ENV_ID` | Yes | No | | Admin API | `DBT_HOST`, `DBT_TOKEN`, `DBT_ACCOUNT_ID` | Yes | No | | SQL execution (`execute_sql`) | Personal access token, `DBT_DEV_ENV_ID`, `DBT_USER_ID` | Yes | No | | Codegen | `DBT_PROJECT_DIR`, `DBT_PATH`, and `DISABLE_DBT_CODEGEN=false` | Yes | Yes | | LSP / dbt v2 | `DBT_PROJECT_DIR`, `DBT_PATH`, and the dbt VS Code extension | Yes | Yes | Toolsets auto-disable when required variables are missing If a required variable is not set, dbt-mcp will automatically disable that toolset rather than error. For example, if `DBT_HOST` is not configured, the Semantic Layer, Discovery, and Admin API toolsets won't be available. To confirm which toolsets are active, set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment and check the [server logs](#debug-configurations). #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) to be able to run `dbt-mcp` and [related dependencies](https://github.com/dbt-labs/dbt-mcp/blob/main/pyproject.toml) into an isolated virtual environment. * Have a local dbt project (if you want to use dbt CLI commands). #### Choose your auth method If you're connecting to dbt platform features (Semantic Layer, Discovery API, Admin API, or SQL execution), you need to authenticate. Use this table to choose the right method: | If you need... | Use... | | ------------------------ | ------------------------------------------------------------------------------- | | Fastest first-time setup | **OAuth** | | `execute_sql` tool | **Personal Access Token (PAT)**. Service tokens *do not* work for `execute_sql` | | Shared or team setup | **Service token** | | CI or automation | **Service token** | `execute_sql` requires a PAT The `execute_sql` tool does *not* work with service tokens. You must use a [Personal Access Token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) for `DBT_TOKEN` when using this tool. #### Setup options Choose the setup method that best fits your workflow: ##### OAuth authentication with dbt platform [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") This method uses OAuth to authenticate with your dbt platform account. It's the simplest setup and doesn't require managing tokens or environment variables manually. Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. ###### Configuration options ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication. Once configured, your session connects to the dbt platform account, starts the OAuth authentication workflow, and then opens your account where you can select the project you want to reference. [![Select your dbt platform project](/img/mcp/select-project.png?v=2 "Select your dbt platform project")](#)Select your dbt platform project After completing OAuth setup, skip to [Test your configuration](#optional-test-your-configuration). ##### CLI only (no dbt platform) This option runs the MCP server locally and connects it to your local dbt project using `DBT_PROJECT_DIR` and `DBT_PATH`. If you're using dbt v1 or dbt v2 and don't need access to dbt platform features (Discovery API, Semantic Layer, Administrative API), you can set up local MCP with just your dbt project information. Add this configuration to your MCP client (refer to the specific [integration guides](#set-up-your-mcp-client) for exact file locations): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` ###### Locating your paths Follow the appropriate instructions for your OS to locate your path:  macOS/Linux * **DBT\_PROJECT\_DIR**: The full path to your dbt project folder * Example: `/Users/yourname/dbt-projects/my_project` * This is the folder containing your `dbt_project.yml` file. * **DBT\_PATH**: Find your dbt executable path by running in terminal: ```bash which dbt ``` * Example output: `/opt/homebrew/bin/dbt` * Use this exact path in your configuration.  Windows * **DBT\_PROJECT\_DIR**: The full path to your dbt project folder * Example: `C:\Users\yourname\dbt-projects\my_project` * This is the folder containing your `dbt_project.yml` file. * Use forward slashes or escaped backslashes: `C:/Users/yourname/dbt-projects/my_project` * **DBT\_PATH**: Find your dbt executable path by running in Command Prompt or PowerShell: ```bash where dbt ``` * Example output: `C:\Python39\Scripts\dbt.exe` * Use forward slashes or escaped backslashes: `C:/Python39/Scripts/dbt.exe` After completing this setup, skip to [Test your configuration](#optional-test-your-configuration). ##### Environment variable configuration If you need to configure multiple environment variables or prefer to manage them separately, you can use an `.env` file or pass them inline. You only need to supply the variables relevant to your setup — dbt CLI variables for CLI-only use, or dbt platform variables for platform features. For the complete list of variables, enabling or disabling toolsets, and logging options, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). Where to put the `.env` file Create the `.env` file in your *dbt project root* (the same folder as `dbt_project.yml`). When referencing it with `--env-file`, always *use an absolute path* so your MCP client can find it reliably. For example, `/absolute/path/to/your-dbt-project/.env`. Pick the `.env` example that matches your setup. Only include the variables you need: ##### CLI only Use this if you're running dbt commands locally and don't need dbt platform features (Discovery API, Semantic Layer, etc.): ```code DBT_PROJECT_DIR=/path/to/your/dbt/project DBT_PATH=/path/to/your/dbt/executable ``` ##### dbt platform only Use this if you only need dbt platform features and won't run dbt commands: ```code DBT_HOST=cloud.getdbt.com DBT_TOKEN=dbtc_your_token DBT_PROD_ENV_ID=12345 ``` ##### CLI and dbt platform (most common) Use this if you want both dbt CLI commands and dbt platform features: ```code DBT_PROJECT_DIR=/path/to/your/dbt/project DBT_PATH=/path/to/your/dbt/executable DBT_HOST=cloud.getdbt.com DBT_TOKEN=dbtc_your_token DBT_PROD_ENV_ID=12345 ``` ##### All variables A complete reference of all available variables. Most setups only need a subset of these — refer to the [API and SQL tool settings](#api-and-sql-tool-settings) and [dbt CLI settings](#dbt-cli-settings) tables for details on each variable. ```code DBT_HOST=cloud.getdbt.com DBT_PROD_ENV_ID=your-production-environment-id DBT_DEV_ENV_ID=your-development-environment-id DBT_USER_ID=your-user-id DBT_ACCOUNT_ID=your-account-id DBT_TOKEN=your-service-token DBT_PROJECT_DIR=/path/to/your/dbt/project DBT_PATH=/path/to/your/dbt/executable ``` ###### How to pass environment variables to dbt-mcp Here are some of the ways to pass environment variables. We recommend using an `.env` file with `--env-file` for most setups: ##### .env file with \`--env-file\` **1. `.env` file with `--env-file` (recommended)** (make sure to use an absolute path in `args`): Reference the file using an absolute path in your MCP client config: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` Replace `/absolute/path/to/your-dbt-project` with the full path to the folder containing your `dbt_project.yml`. ##### Inline in the MCP client config **2. Inline in the MCP client config**: Pass variables directly in the `env` field, replacing the values with your actual ones. This keeps everything in one file but means tokens are stored in your client config: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` ##### Shell environment variables **3. Shell environment variables** Export variables in your terminal before starting the MCP client, replacing the values with your actual ones: ```bash export DBT_HOST=cloud.getdbt.com export DBT_TOKEN=your-token-here export DBT_PROJECT_DIR=/path/to/project uvx dbt-mcp ``` #### API and SQL tool settings | Environment variable | Required | Description | | -------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_HOST` | Required | Your dbt platform [instance hostname](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). The default is `cloud.getdbt.com`. For multi-cell and multi-tenant accounts with a static subdomain, use the full hostname — for example, `abc123.us1.dbt.com`. | | `DBT_TOKEN` | Required | Your personal access token or service token from the dbt platform.<br />**Note**: If you're using a service token, assign the permission sets for the toolsets you enable: `Semantic Layer Only` for Semantic Layer tools and `Metadata Only` for Discovery tools. The self-hosted server doesn't check permissions when it starts. Instead, each API authorizes the token when a tool calls it. A token that covers only some toolsets works for those toolsets, and calls to the other toolsets fail individually. In contrast, the [remote MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) checks the token once when you connect.<br /><br />The `execute_sql` tool requires a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) and doesn't work with service tokens. We also recommend a PAT for Semantic Layer tools. | | `DBT_ACCOUNT_ID` | Required for Administrative API tools and PAT-based auth | Your [dbt account ID](https://docs.getdbt.com/faqs/Accounts/find-user-id.md). Also required when using a Personal Access Token (PAT) as your `DBT_TOKEN`. | | `DBT_PROD_ENV_ID` | Required | Your dbt platform production environment ID | | `DBT_DEV_ENV_ID` | Optional | Your dbt platform development environment ID | | `DBT_USER_ID` | Optional | Your dbt platform user ID ([docs](https://docs.getdbt.com/faqs/Accounts/find-user-id.md)) | Use values only, not full URLs A common mistake is pasting a full URL instead of the value. These variables expect hostnames or numeric IDs: ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` **Subdomain prefix configuration example:** ✅ **Correct configuration:** ```bash DBT_HOST=abc123.us1.dbt.com # Use the full hostname including the prefix DBT_ACCOUNT_ID=12345 # Required when using PAT-based auth ``` You don't need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. #### Environment variables The self-hosted dbt-mcp supports all flavors of dbt, including dbt v1 and dbt v2. | Environment variable | Required | Description | Example | | -------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `DBT_PROJECT_DIR` | Required | The full path to where the repository of your dbt project is hosted locally. This is the folder containing your `dbt_project.yml` file. | macOS/Linux: `/Users/myname/reponame`<br />Windows: `C:/Users/myname/reponame` | | `DBT_PATH` | Required | The full path to your dbt executable (dbt v1/dbt v2/dbt platform CLI). See the next section for how to find this. | macOS/Linux: `/opt/homebrew/bin/dbt`<br />Windows: `C:/Python39/Scripts/dbt.exe` | | `DBT_CLI_TIMEOUT` | Optional | Configure the number of seconds before your agent will timeout dbt commands. | Defaults to 60 seconds. | ##### Locating your `DBT_PATH` Follow the instructions for your OS to locate your `DBT_PATH`:  macOS/Linux Run this command in your Terminal: ```bash which dbt ``` Example output: `/opt/homebrew/bin/dbt`  Windows Run this command in Command Prompt or PowerShell: ```bash where dbt ``` Example output: `C:\Python39\Scripts\dbt.exe` **Note:** Use forward slashes in your configuration: `C:/Python39/Scripts/dbt.exe` **Additional notes:** * You can set any environment variable supported by your dbt executable, like [the ones supported in dbt v1](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). * dbt MCP respects the standard environment variables and flags for usage tracking mentioned [here](https://docs.getdbt.com/reference/global-configs/usage-stats.md). * `DBT_WARN_ERROR_OPTIONS='{"error": ["NoNodesForSelectionCriteria"]}'` is automatically set so that the MCP server knows if no node is selected when running a dbt command. You can overwrite it if needed, but it provides a better experience when calling dbt from the MCP server, ensuring the tool selects valid nodes. #### Controlling tool access dbt-mcp has two modes for controlling which tools are available. Pick one approach: * disable mode (turn off what you don't want) * enable mode (turn on only what you want) Which mode should I use? * Disable mode — Use when you want *most* tools available and just need to turn off a few. This is the default behavior. * Enable mode — Use when you want *only* a small set of tools available (allowlist approach). **Do not mix both modes** for the same toolset. For example, don't set both `DISABLE_SEMANTIC_LAYER=true` and `DBT_MCP_ENABLE_SEMANTIC_LAYER=true` together — the behavior may be unpredictable. ##### Disable mode (default) All tools are available by default. Set any of these to `true` to turn off a toolset: | Name | Default | Description | | ----------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DISABLE_DBT_CLI` | `false` | Disable dbt v1, dbt platform CLI, and dbt v2 MCP tools. | | `DISABLE_SEMANTIC_LAYER` | `false` | Disable dbt Semantic Layer MCP tools. | | `DISABLE_DISCOVERY` | `false` | Disable dbt Discovery API MCP tools. | | `DISABLE_ADMIN_API` | `false` | Disable dbt Administrative API MCP tools. | | `DISABLE_SQL` | `true` | SQL MCP tools are disabled by default. Set to `false` to enable. | | `DISABLE_DBT_CODEGEN` | `true` | [dbt codegen MCP tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#codegen-tools) are disabled by default. Set to `false` to enable (requires dbt-codegen package). | | `DISABLE_LSP` | `false` | Disable dbt LSP/dbt v2 MCP tools. | | `DISABLE_MCP_SERVER_METADATA` | `true` | MCP server metadata tools (like `get_mcp_server_version`) are disabled by default. Set to `false` to enable. | | `DISABLE_TOOLS` | `""` | A comma-separated list of specific tool names to disable. | ##### Enable mode Use `DBT_MCP_ENABLE_*` variables when you want to explicitly allowlist which toolsets are available. When an enable variable is set, only the enabled toolsets will be active: | Name | Default | Description | | ------------------------------- | ------- | -------------------------------------------------------- | | `DBT_MCP_ENABLE_DBT_CLI` | Not set | Set to `true` to enable dbt CLI tools. | | `DBT_MCP_ENABLE_SEMANTIC_LAYER` | Not set | Set to `true` to enable Semantic Layer tools. | | `DBT_MCP_ENABLE_DISCOVERY` | Not set | Set to `true` to enable Discovery API tools. | | `DBT_MCP_ENABLE_ADMIN_API` | Not set | Set to `true` to enable Administrative API tools. | | `DBT_MCP_ENABLE_SQL` | Not set | Set to `true` to enable SQL tools. | | `DBT_MCP_ENABLE_DBT_CODEGEN` | Not set | Set to `true` to enable dbt codegen tools. | | `DBT_MCP_ENABLE_LSP` | Not set | Set to `true` to enable LSP/dbt v2 tools. | | `DBT_MCP_ENABLE_TOOLS` | Not set | A comma-separated list of specific tool names to enable. | ##### Precedence When multiple variables are set, they are evaluated in this order (highest priority first): 1. `DBT_MCP_ENABLE_TOOLS` (enable specific tools by name) 2. `DISABLE_TOOLS` (disable specific tools by name) 3. Toolset enable (`DBT_MCP_ENABLE_*=true`) 4. Toolset disable (`DISABLE_*=true`) 5. Default behavior #### (Optional) Test your configuration In your command line tool, run the following to test your setup: **If using the `env` field in JSON:** ```bash export DBT_PROJECT_DIR=/path/to/project export DBT_PATH=/path/to/dbt uvx dbt-mcp ``` **If using an `.env` file:** (use an absolute path, for example to `.env` in your dbt project root) ```bash uvx --env-file /absolute/path/to/your-dbt-project/.env dbt-mcp ``` If there are no errors, your configuration is correct. #### Set up your MCP client After completing your configuration, follow the specific integration guide for your chosen tool: * [Claude](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md) * [Cursor](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md) * [VS Code](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md) #### Debug configurations These settings allow you to customize the MCP server’s logging level to help with diagnosing and troubleshooting. | Name | Default | Description | | ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ | | `DBT_MCP_LOG_LEVEL` | `INFO` | Environment variable to override the MCP server log level. Options are: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. | To see more detail about what’s happening inside the MCP server and help debug issues, you can temporarily set the log level to `DEBUG`. We recommend setting it temporarily to avoid filling up disk space with logs. #### Troubleshooting Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). For all troubleshooting topics, see [dbt MCP FAQs and troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). --- ### Setting up authentication for non-interactive environments Login required | Usage-based In a non-interactive environment, dbt runs without a person available to complete authentication manually — for example, CI/CD pipelines (such as GitHub Actions, GitLab CI, and Jenkins) and production orchestration tools (such as Airflow and Prefect). Browser-based authentication isn't possible in these environments. Instead, dbt State authenticates using credentials provided through environment variables, allowing it to continue caching state and optimizing your builds. dbt State automatically detects when it's running in a non-interactive environment. If valid credentials are not provided, dbt State disables itself and displays a warning, allowing your dbt commands to continue without caching. Running dbt in a non-interactive environment requires a service account token for authentication. A dbt platform account is required to generate a service token; if you don't have one yet, refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) to create a free account before proceeding. If you previously signed up for dbt State through the standalone app (`app.state.dbt.com`) and haven't migrated to dbt platform yet, refer to [OAuth client credentials](https://docs.getdbt.com/docs/deploy/dbt-state-oauth.md). #### Prerequisites Before you begin, make sure you have: * A dbt platform account. * **Owner** or **Account Admin** permissions to create a service token. * dbt State installed and configured. Refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) for more information. #### Creating a service token To create a service account token in dbt platform, refer to [Generate service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#generate-service-account-tokens). When adding permissions for the token, assign at least one of the following: * **Account Admin** The following are available on Enterprise plans only: * **Owner** * **Job Admin** * **Job Creator** * **Job Runner** (Recommended; provides the minimum access required for dbt State) * **Developer** #### Configuring authentication Set the following environment variables in your orchestration environment: ```bash DBT_CLOUD_TOKEN=YOUR_SERVICE_TOKEN DBT_CLOUD_ACCOUNT_HOST=YOUR_ACCOUNT_HOST DBT_CLOUD_ACCOUNT_ID=YOUR_ACCOUNT_ID ``` Replace `YOUR_SERVICE_TOKEN` with your service token, `YOUR_ACCOUNT_HOST` with your [account host](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) (for example, `abc123.us1.dbt.com`), and `YOUR_ACCOUNT_ID` with your numeric account ID. Go to **Account settings** > **Account** to find your account ID and account host (the hostname from the **Access URL** field). ##### Network requirements If your network or environment restricts outbound traffic, allow the following URLs so dbt State can authenticate and communicate with the dbt platform: | URL | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `auth.state.dbt.com` | dbt State authentication | | `api.state.dbt.com` | dbt State API | | `abc123.us1.dbt.com` | Your dbt platform [access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) (Replace with your account's access URL) | Once authentication is configured, [verify dbt State is active](#verifying-dbt-state-is-active). #### Verifying dbt State is active 1. Run any dbt transformation job in your orchestrated environment. 2. Check the log output. You should see a message like this, then the specific dbt State step status: ```text dbt State adapter: dbt-state v2.10.1 is enabled ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### Setting up dbt State Login required | Usage-based This page walks you through setting up dbt State across dbt v1, dbt v2, and dbt platform. #### Prerequisites Before you set up dbt State, make sure you have: * **A supported dbt version**: dbt State is natively available in dbt platform, dbt v2, and dbt v1.13. It's also available as a plugin for dbt v1.7–1.12. * **A supported data platform**: Snowflake, Databricks, BigQuery, or Redshift. More warehouses are on the roadmap. * **A dbt account**: dbt State uses your dbt account for authentication, even if you're running dbt locally or in a non-interactive environment like CI/CD pipelines or an external orchestrator. Refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md#signing-up-for-dbt-state) for sign-up details, and [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md) for pricing details. #### Setting up dbt State ##### Self-hosted dbt dbt State requires a dbt account, even when running dbt locally, to authenticate to the dbt State service. No need to migrate your existing workflows or purchase any dbt seats. 1. Log in to your dbt account and start your dbt State trial. (Applies to dbt v1.13 and later) 1. Navigate to your project: ```bash cd to/your/project ``` 2. Log in to your dbt account: ```bash dbt login ``` This opens a browser window where you can log in to or create a free dbt account. If you're creating a new account: 1. Enter your name and password. 2. Select **Enable a trial of dbt State**. 3. Agree to the Terms of Service and click **Continue**. 4. Sign in to dbt platform. 5. Set up an authentication method. 6. Verify your email address. 3. Agree to the dbt State Terms of Service and click **Start 30-day trial**. Once started, you cannot pause the trial. After the trial ends, your account receives a free monthly DATT allowance, which varies by plan. For details, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). You can also enable or disable dbt State per run using [CLI flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md): `--manage-state` or `--no-manage-state`, or set the `DBT_ENGINE_MANAGE_STATE` environment variable. To enable dbt State for everyone on your project, add [`manage_state: true`](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) to the `flags:` block in `dbt_project.yml`: ```yaml flags: manage_state: true ``` 2. Run `dbt run` or `dbt build`. * **First run:** With no prior state to compare against, dbt builds every node from scratch. dbt State captures metadata from this run for future comparisons. * **Later runs:** dbt State compares each node's logic and upstream data against previous builds across all environments. It rebuilds a node when its logic has changed, or when its upstream data has changed and its last build is older than the [`lag_tolerance`](#configuring-lag-tolerance) window. Otherwise, it skips or clones an existing node. To see how dbt State optimizes your runs, refer to [dbt State usage examples](https://docs.getdbt.com/docs/deploy/dbt-state-examples.md). 3. Set up your orchestration environment. * **Own orchestrator**: Before you can use dbt State in your own orchestrator, create a service token in dbt platform and configure your orchestration environment to use it. Refer to [Setting up authentication for non-interactive environments](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) for steps. Once configured, you can use your orchestrator to run dbt. * **dbt platform**: Go to **Orchestration** to [create your environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md) and [jobs](https://docs.getdbt.com/docs/deploy/jobs.md), then [enable dbt State on them](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md). 4. [Verify dbt State is active](#verify). 5. [Invite your team to use dbt State](#invite). ##### Existing dbt platform account **Prerequisite**: You must be an admin in your dbt platform account. 1. Start your dbt State trial. 1. In your dbt platform account, click your account name in the lower-left corner above your username and click **Account settings**. 2. Under **Settings**, go to **Billing & Usage** > **Usage-based features**. 3. Under the **State** tab, click **Start free trial**. Once started, you cannot pause the trial. After the trial ends, your account receives a free monthly DATT allowance, which varies by plan. For details, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). Extended trial for state-aware orchestration users If you were using state-aware orchestration prior to June 1, 2026, your dbt State trial will be extended beyond 30 days. If the extension isn't applied to your account, contact your account team. 4. Review and agree to the terms of service. 5. Click **Start 30-day trial**. 2. Click **Enable dbt State**. 3. Select where to enable dbt State. You can either enable: * **By environment**: Enables dbt State on the selected deployment or development environment. * For deployment environments, new jobs automatically default to **Inherited from environment** and have dbt State enabled without additional configuration. Existing jobs are not automatically updated — you must [configure each job manually](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md#enabling-dbt-state-on-individual-jobs). * For development environments, this [enables dbt State in the Studio IDE](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md). * **By specific jobs**: Enables dbt State on individual jobs. To enable it on additional jobs, refer to [Enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-env-jobs.md#enabling-dbt-state-on-individual-jobs). 4. Click **Enable dbt State**. 5. [Verify dbt State is active](#verify). 6. [Invite your team to use dbt State](#invite). #### Verifying dbt State is active After a run completes, confirm that dbt State is active: * **From the CLI:** Check the log output for a message like `dbt State adapter: dbt-state v2.10.1 is enabled`. * **From dbt platform:** Check the [logs view of built resources](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#logs-view-of-built-resources) to see if they were built or reused, or open the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the State decision for each resource. If dbt State isn't behaving as expected, refer to [Debugging dbt State](#debugging-dbt-state). #### Inviting team members The more team members you have using dbt State, the better it gets; more team members means more opportunities to clone existing nodes rather than rebuilding them. To invite colleagues: * **From the CLI (dbt v1.13+):** Have your colleagues run [`dbt login`](https://docs.getdbt.com/reference/commands/login.md) after dbt State is enabled on the account. * **From dbt platform:** Go to **Account settings** > **Users** and click **Invite users**. For more information, refer to [Invite new users](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md#invite-new-users). #### Configuring lag tolerance Lag tolerance controls how long dbt State waits before rebuilding a node once its upstream data changes. A node rebuilds only when its last build is older than the tolerance window and its upstream data has changed. You can set it at the project, environment, or model level. If not configured, `lag_tolerance` defaults to `45m`. We recommend starting with the following Jinja expression: dbt\_project.yml ```yaml models: +state: lag_tolerance: "{{ '4h' if target.name == 'prod' else '7d' }}" ``` In this example, models in the `prod` target rebuild once their last build is more than 4 hours old and their upstream data has changed. In all other environments, models rebuild once their last build is more than 7 days old and their upstream data has changed. For more details, refer to the [`lag_tolerance` config reference](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). tip To help you tune `lag_tolerance` values, the **dbt State** page on the dbt platform provides [lag tolerance recommendations](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#lag-tolerance-recommendations) based on your models' 30-day build history, so you can see which models would benefit from a higher tolerance. #### Debugging dbt State If dbt State is behaving unexpectedly, you can prepend your run command with the `DBT_ENGINE_MANAGE_STATE` environment variable to isolate the issue: ```bash DBT_ENGINE_MANAGE_STATE=0 dbt run --target dev --select "customers" ``` To see which decision dbt State made for each node after a run and why, you can run the (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) command. If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. #### Next steps * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) * [`dbt login` with dbt State](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-with-dbt-state) * [Configure deferral](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md) * [Setting up authentication for non-interactive environments](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) --- ### Setting up state-aware orchestration Private preview ### Setting up state-aware orchestration [Private preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Set up state-aware orchestration to automatically determine which models to build by detecting changes in code or data and only building the changed models each time a job is run. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). important dbt v2 is available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our dbt Community Slack channel [`#dbt-v2`](https://getdbt.slack.com/archives/C088YCAB6GH). #### Prerequisites To use state-aware orchestration, make sure you meet these prerequisites: * You must have a dbt [Enterprise and Enterprise+ accounts](https://www.getdbt.com/signup/) and a [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * You have updated the environment that will run state-aware orchestration to dbt v2. For more information, refer to [Upgrading to dbt v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). * Your account must have access to state-aware orchestration. Contact your account manager to request access. * You must have a dbt project connected to a [data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * You must have [access permission](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to view, create, modify, or run jobs. * You must set up a [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) that is production or staging only. * You must use a deploy job. Continuous integration (CI) and merge jobs currently do not support state-aware orchestration. * (Optional) To customize behavior, you have configured your model or source data with [advanced configurations](#advanced-configurations). info State-aware orchestration is available for SQL models only. Python models are not supported. #### Default settings Once your account has access to state-aware orchestration, any new deploy job you create in a dbt v2 environment is automatically state-aware. Without additional configuration, the job only builds models when code has changed or new data exists in a source. #### Create a job New jobs are state-aware by default For existing jobs, select **Enable dbt v2 cost optimization features** in the **Job settings** page to enable state-aware orchestration. To create a state-aware job: 1. From your deployment environment page, click **Create job** and select **Deploy job**. 2. Options in the **Job settings** section: * **Job name**: Specify the name, for example, `Daily build`. * (Optional) **Description**: Provide a description of what the job does (for example, what the job consumes and what the job produces). * **Environment**: By default, it’s set to the deployment environment you created the state-aware job from. 3. Options in the **Execution settings** and **Triggers** sections. * **Execution settings** section: * **Commands**: By default, it includes the `dbt build` command. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when the job runs. * **Enable dbt v2 cost optimization features**: Select this option to enable **State-aware orchestration**. **Efficient testing** is disabled by default. You can expand **More options** to enable or disable individual settings. * **Triggers** section: * **Run on schedule**: Run the deploy job on a set schedule. * **Timing**: Specify whether to [schedule](#schedule-days) the deploy job using **Intervals** that run the job every specified number of hours, **Specific hours** that run the job at specific times of day, or **Cron schedule** that run the job specified using [cron syntax](#cron-schedule). * **Days of the week**: By default, it’s set to every day when **Intervals** or **Specific hours** is chosen for **Timing**. * **Run when another job finishes**: Run the deploy job when another *upstream* deploy [job completes](#trigger-on-job-completion). * **Project**: Specify the parent project that has that upstream deploy job. * **Job**: Specify the upstream deploy job. * **Completes on**: Select the job run status(es) that will [enqueue](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) the deploy job. 6. (Optional) Options in the **Advanced settings** section: * **Environment variables**: Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when the deploy job runs. * **Target name**: Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably. * **Run timeout**: Cancel the deploy job if the run time exceeds the timeout value. * **Compare changes against**: By default, it’s set to **No deferral**. Select either **Environment** or **This Job** to let dbt know what it should compare the changes against. 7. Click **Save**. You can see which models dbt builds in the run summary logs. Models that weren't rebuilt during the run are tagged as **Reused** with context about why dbt skipped rebuilding them (and saving you unnecessary compute!). You can also see the reused models under the **Reused** tab. [![Example logs for state-aware orchestration](/img/docs/dbt-platform/using-dbt-platform/SAO_logs_view.png?v=2 "Example logs for state-aware orchestration")](#)Example logs for state-aware orchestration #### Delete a job To delete a job or multiple jobs in dbt: 1. Click **Deploy** on the navigation header. 2. Click **Jobs** and select the job you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete job** to delete the job. <br /> [![Delete a job](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-job.png?v=2 "Delete a job")](#)Delete a job 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. 6. Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Advanced configurations By default, we use the warehouse metadata to check if sources (or upstream models in the case of Mesh) are fresh. For more advanced use cases, dbt provides other options that enable you to specify what gets run by state-aware orchestration. You can use the following optional parameters to customize your state-aware orchestration: | Parameter | Description | Allowed values | Supports Jinja | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | `loaded_at_field` | Specifies a specific column to use from the data. | Name of timestamp column. For example, `created_at`, `"CAST(created_at AS TIMESTAMP)"`. | ✅ | | `loaded_at_query` | Defines a custom freshness condition in SQL to account for partial loading or streaming data. | SQL string. For example, `"select {{ current_timestamp() }}"`. For a multi-line query, see the example after this table. | ✅ | | `build_after.count` | Determines how many units of time must pass before a model can be rebuilt to help reduce build frequency. | A positive integer or a Jinja expression. For example, `4` or `"{{ var('build_after_count', 4) }}"`. | ✅ | | `build_after.period` | The time unit for the count to define the build interval. | `minute`, `hour`, `day`, or a Jinja expression (for example, `"{{ var('build_after_period', 'day') }}"`). | ✅ | | `build_after.updates_on` | Determines whether a model rebuild is triggered when any upstream dependency has fresh data or only when all upstream dependencies are fresh. | \* `any` (default) — Use this value when you want a downstream model to rebuild if *any* of its upstream dependencies receives fresh data, even if others haven’t.\* `all` — Use this value when you want to trigger a rebuild only when *all* upstream dependencies are fresh — minimizing unnecessary builds and reducing compute cost. Recommended to use in state-aware orchestration. | ❌ | Some notes when using `loaded_at_field` or `loaded_at_query`: * You can either define `loaded_at_field` or `loaded_at_query` but not both. * To use a multi-line SQL query for a `loaded_at_query` configuration, include your query as a YAML block so dbt can execute it as the custom freshness query. For example: ```yaml loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` * If a source is a view in the data warehouse, the available metadata is usually insufficient to discern freshness, and dbt emits a warning during freshness checks. To determine freshness for sources that are views, add a `loaded_at_field` or `loaded_at_query` to your configuration. To learn more about model freshness and `build_after`, refer to [model `freshness` config](https://docs.getdbt.com/reference/resource-configs/freshness.md). To learn more about source and upstream model freshness configs, refer to [resource `freshness` config](https://docs.getdbt.com/reference/resource-configs/freshness.md). ##### Customizing behavior You can optionally configure state-aware orchestration when you want to fine-tune orchestration behavior for these reasons: * **Defining source freshness:** By default, dbt uses metadata from the data warehouse to automatically detect when source data changes. Freshness configuration is not required for state-aware orchestration to work. You can optionally configure source freshness if you want to: * Receive alerts when sources don't update within your expected Service Level Agreement (SLA) using `warn_after`/`error_after`. * Specify a custom column using `loaded_at_field`. * Specify a custom SQL statement using `loaded_at_query` to define what freshness means. Not all source freshness is equal — especially with partial ingestion pipelines. You may want to delay a model build until your sources have received a larger volume of data or until a specific time window has passed. You can define what "fresh" means on a source-by-source basis using a custom freshness query. This lets you: * Add a time difference to account for late-arriving data * Delay freshness detection until a threshold is reached (for example, number of records or hours of data) The following examples show how to configure a source so that state-aware orchestration detects new upstream data only when your custom condition is met. ##### loaded\_at\_field State-aware orchestration treats the source as fresh when the maximum value of the `loaded_at_field` column changes since the previous run: models/sources.yml ```yaml sources: - name: jaffle_shop config: freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at ``` ##### loaded\_at\_query To define freshness with custom SQL, use `loaded_at_query`. State-aware orchestration runs the query to get a single timestamp. When that value changes compared to the previous run, the source is considered fresh. models/sources.yml ```yaml sources: - name: raw_orders tables: - name: orders loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` In this example, dbt runs the custom `loaded_at_query` to get a single timestamp — the latest `ingested_at` within the last three days. On each run, dbt compares this new maximum timestamp to the value from the previous run. If the maximum timestamp is newer, state-aware orchestration considers the source to have fresh data and may trigger rebuilds. * **Reducing model build frequency** Some models don’t need to be rebuilt every time their source data is updated. To control this: * Set a refresh interval on models, folders, or the project to define how often they should be rebuilt at most * This helps avoid overbuilding and reduces costs by only running what's really needed * **Changing the default from `any` to `all`** Based on what a model depends on upstream, you may want to wait until all upstream models have been refreshed rather than going as soon as there is any new data. * Change what orchestration waits on from any to all for models, folders, or the project to wait until all upstream models have new data * This helps avoid overbuilding and reduces costs by building models once everything has been refreshed To configure and customize behavior, you can do so in the following places using the `build_after` config: * `dbt_project.yml` at the project level in YAML * `model/properties.yml` at the model level in YAML * `model/model.sql` at the model level in SQL These configurations are powerful because you can define a sensible default at the project level or for specific model folders, and override it for individual models or model groups that require more frequent updates. ##### Handling late-arriving data If your incremental models use a lookback window to capture [late-arriving data](https://docs.getdbt.com/best-practices/materializations/4-incremental-models.md#late-arriving-facts), make sure your freshness logic aligns with that window. When you use a `loaded_at_field` or `loaded_at_query`, state-aware orchestration uses that value to determine whether new data has arrived. When the `loaded_at` value reflects an event timestamp (for example, `event_date`), late-arriving records may not update this value if the event occurred in the past. In these cases, state-aware orchestration may not trigger a rebuild, even though your incremental model’s lookback window would normally include those rows. To ensure late-arriving data is detected by state-aware orchestration, use `loaded_at_query` and make sure it aligns with the same lookback window used in your incremental filter. See the following samples of a lookback window and its corresponding `loaded_at_query` value: ##### Lookback window ```sql {{ config( materialized='incremental', unique_key='order_id' ) }} select * from {{ source('raw_orders', 'orders') }} {% if is_incremental() %} where ingested_at > (select max(ingested_at) from {{ this }}) - interval '3 days' {% endif %} ``` ##### loaded\_at\_query ```yaml loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` #### Example Let's use an example to illustrate how to customize our project so a model and its parent model are rebuilt only if they haven't been refreshed in the past 4 hours — even if a job runs more frequently than that. A Jaffle shop has recently expanded globally and wanted to make savings. To reduce spend, they found out about dbt's state-aware orchestration and want to rebuild models only when needed. Maggie — the analytics engineer — wants to configure her dbt `jaffle_shop` project to only rebuild certain models if they haven't been refreshed in the last 4 hours, even if a job runs more often than that. To do this, she uses the model `freshness` config. This config helps state-aware orchestration decide *when* a model should be rebuilt. Note that for every `freshness` config, you're required to set values for both `count` and `period`. This applies to all `freshness` types: `freshness.warn_after`, `freshness.error_after`, and `freshness.build_after`. Refer to the following examples for using the `freshness` config in the model file, in the project YAML file, and in the config block of the `model.sql` file: ##### Model YAML models/model.yml ```yaml models: - name: dim_wizards config: freshness: build_after: count: 4 # how long to wait before rebuilding period: hour # unit of time updates_on: all # only rebuild if all upstream dependencies have new data - name: dim_worlds config: freshness: build_after: count: 4 period: hour updates_on: all ``` ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +freshness: build_after: count: 4 period: hour updates_on: all ``` ##### SQL file config models/\<filename>.sql ```jinja {{ config( freshness={ "build_after": { "count": 4, "period": "hour", "updates_on": "all" } } ) }} ``` With this config, dbt: * Checks if there's new data in the upstream sources * Checks when `dim_wizards` and `dim_worlds` were last built If any new data is available *and* at least 4 hours have passed, dbt rebuilds the models. You can override freshness rules set at higher levels in your dbt project. For example, in the project YAML file, you set: dbt\_project.yml ```yml models: +freshness: build_after: count: 4 period: hour jaffle_shop: # this needs to match your project `name:` in dbt_project.yml staging: +materialized: view marts: +materialized: table ``` This configuration means that every model in the project has a `build_after` of 4 hours. To change this for specific models or groups of models, you could set: dbt\_project.yml ```yml models: +freshness: build_after: count: 4 period: hour marts: # only applies to models inside the marts folder +freshness: build_after: count: 1 period: hour ``` If you want to exclude a model from the freshness rule set at a higher level, set `freshness: null` for that model. With freshness disabled, state-aware orchestration falls back to its default behavior and builds the model whenever there’s an upstream code or data change. ##### Differences between `all` and `any` * Since Maggie configured `updates_on: all`, this means *both* models must have new upstream data to trigger a rebuild. If only one model has fresh data and the other doesn't, nothing is built -- which will massively reduce unnecessary compute costs and save time. * If Maggie wanted these models to rebuild more often (for example, if *any* upstream source has new data), she would then use `updates_on: any` instead: models/model.yml ```yaml freshness: build_after: count: 1 period: hour updates_on: any ``` This way, if either `dim_wizards` or `dim_worlds` has fresh upstream data and enough time passed, dbt rebuilds the models. This method helps when the need for fresher data outweighs the costs. #### Related docs * [State-aware orchestration configuration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) * [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [`freshness`](https://docs.getdbt.com/reference/resource-configs/freshness.md) * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### Sign in Local development Sign in to connect the dbt VS Code extension to your dbt platform account. Signing in is what lets the extension reach your account for dbt platform capabilities, such as the [**Catalog** tab](https://docs.getdbt.com/docs/dbt-extension-features.md#explore-your-catalog) and [compare changes](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md) with dbt platform deferral. Refer to [VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability) for the full list of features and their availability. (Applies to dbt v1.13 and later) You can sign in from the command line using [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0), available with v1.13 and later. Your login state can be shared across dbt features, including the dbt VS Code extension and, in supported versions, dbt State. Run [`dbt login status`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-status) to view your currently authenticated status. #### Key points * The extension is available to all users. Refer to the [dbt Product License Agreement](https://www.getdbt.com/dbt-product-license-agreement) for more information. * Signing in links your editor to your dbt platform account so the extension can use dbt platform capabilities. * You can authenticate with `dbt login` or sign in from inside the extension. If you don't have a dbt platform account, you can create a free account during authentication. * If a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file exists on your machine, the extension can use it automatically. If you've signed in before, you won't need to authenticate again unless your session expired. #### Session expiry and re-authorization Your sign-in session stays active across editor restarts. dbt automatically renews your session while you're using the extension or the CLI. You may need to re-authorize in a few specific cases: * **Session expired after 7 days of inactivity.** The extension shows the message: "Your dbt extension session expired. Sign in again to continue using the extension." Click the prompt or run **dbt: Register dbt extension** from the command palette to sign in again. * **A feature needs broader access than your initial sign-in granted.** [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) and the dbt VS Code extension may request different sets of permissions. If you signed in via `dbt login` and later use an extension feature that needs more access (for example, the catalog tab or running a job), the extension prompts you to re-authorize. You'll re-authorize with the same dbt platform account — you don't need a second account. * **You authenticated with `dbt_cloud.yml` instead of OAuth.** File-based credentials don't expire on the 7-day inactivity rule. You only re-authenticate if you regenerate the file or it's removed. When re-authorizing, the **Authorize dbt login** consent screen lists the access being requested. Click **Allow access** to continue. #### Choose a sign-in path There are a couple of different ways to sign in to a dbt platform account. Choose the best path for you: | If you... | Use this path | | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Starting authentication from the terminal | [`dbt login`](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=dbt-login#choose-a-sign-in-path) from the command line. | | Are new to dbt and don't have a dbt platform account | [First-time sign-in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=first-time#choose-a-sign-in-path) from the dbt VS Code extension. | | Can't sign in or need to reset access | [Recover your login](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=recover-login#choose-a-sign-in-path) from the dbt VS Code extension. | | Prefer file-based authentication to remove need for re-authenticating | [`dbt_cloud.yml`](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=dbt-cloud-yml#choose-a-sign-in-path) from the dbt VS Code extension. | ##### dbt login ##### dbt login Use this path if you prefer the terminal or want a single auth flow that works across the CLI and the dbt VS Code extension. Running `dbt login` opens the same browser flow that the extension uses when you select **dbt: Register dbt extension** from the command palette, so completing it once authenticates both. 1. From your terminal, run: ```shell dbt login ``` 2. Complete the sign-in or sign-up flow in your browser. * Refer to [First-time registration](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=first-time#first-time-registration) if you've never registered * Refer to [Existing account sign-in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=existing-account#existing-dbt-account) if you already have a dbt account. * Refer to [Recover your login](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=recover-login#recover-your-login) if you've forgotten your password or your account is locked. 3. On the **Authorize dbt login** screen, click **Allow access**. dbt saves your credentials to your local dbt configuration directory and uses them for future CLI commands: * macOS and Linux: `~/.dbt/` * Windows: `C:\Users\[username]\.dbt\` [![The Authorize VS Code Extension consent screen.](/img/docs/extension/signin-authorize-extension.png?v=2 "The Authorize VS Code Extension consent screen.")](#)The Authorize VS Code Extension consent screen. 4. Return to your editor and restart the dbt VS Code extension, or open a new editor window. 5. The extension detects the credentials and shows **Registered**. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). The extension can now reach your dbt platform account. For details on what `dbt login` does across tools, refer to the [`dbt login` reference](https://docs.getdbt.com/reference/commands/login.md?version=2.0). ##### First-time registration ##### First-time registration Create a dbt platform account and connect it to the extension. Use this path if you've never signed in before. The extension and the CLI open the same browser flow, so you can start from either entry point: * **From the extension:** Click the registration prompt, or open the command palette (Windows/Linux: Ctrl + Shift + P, macOS: Cmd + Shift + P) and run **dbt: Register dbt extension**. [![The extension registration prompt in VS Code.](/img/docs/extension/registration-prompt.png?v=2 "The extension registration prompt in VS Code.")](#)The extension registration prompt in VS Code. * **From the CLI:** Run `dbt login` in your terminal. After you start the flow, complete the browser steps: 1. On the **Log in to dbt platform** page, enter your email and click **Continue**. 2. On the **Create your dbt platform** page, enter your name, create a password, then click **Continue**. dbt creates a free account for you. 3. On the **Authorize dbt login** screen, review the requested access. Leave **Keep session alive** on if you want dbt to renew your session automatically, then click **Allow access**. [![The Authorize dbt login consent screen.](/img/docs/extension/signin-authorize-extension.png?v=2 "The Authorize dbt login consent screen.")](#)The Authorize dbt login consent screen. 4. Return to your editor. The extension shows **Registered**. If you started from the CLI, your credentials are also saved for future CLI commands. 5. Continue with the [Get started](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started) workflow. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). ##### Existing account sign-in ##### Existing account sign-in Use this path if you've registered before, including with an older or inactive registration. The extension and the CLI open the same browser flow, so you can start from either entry point: * **From the extension:** [Update the VS Code extension](https://code.visualstudio.com/docs/setup/setup-overview#_update-cadence) to the latest version, restart your editor, then click the registration prompt or run **dbt: Register dbt extension** from the command palette. * **From the CLI:** Run `dbt login` in your terminal. After you start the flow, complete the browser steps: 1. On the **Sign in to dbt** page, enter the email you used to register, agree to the terms of service, complete the "I'm not a robot" check, then click **Continue**. 2. On the **Verify your email** page, enter the verification code dbt sent to your email and click **Continue**. 3. If your email is linked to more than one account, select the account you want to use on the account selection page. 4. If your account uses an authenticator app, choose a method to verify your identity, then complete the prompt. [![The verify-your-identity screen for accounts that use an authenticator app.](/img/docs/extension/signin-verify-identity.png?v=2 "The verify-your-identity screen for accounts that use an authenticator app.")](#)The verify-your-identity screen for accounts that use an authenticator app. 5. On the **Authorize dbt login** screen, review the requested access and click **Allow access**. **When you might still need a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md):** * You want a file-based credential for automations or defer. * You use platform features that require `dbt_cloud.yml`, such as Mesh or auto-deferral. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). ##### Recover your login ##### Recover your login Use this path if the login flow says you have an account but you don't remember your password. 1. On the **Sign in to dbt** page, click **Reset password**. 2. On the reset-password screen, enter the email you used to register. 3. Check your inbox for the password-reset email and follow the link. 4. Set a new password. 5. Return to your terminal and run `dbt login` again. The password reset opens a separate browser session from the original terminal login session, so you need to restart the login flow. 6. On the **Sign in to dbt** page, sign in with the email and new password you just set, agree to the terms of service, complete the "I'm not a robot" check, then click **Continue**. 7. If your email is linked to more than one account, select the account you want to use. 8. On the **Authorize dbt login** screen, click **Allow access**. When authorization completes, the flow returns to your terminal and the dbt VS Code extension. If you still can't sign in, your account may be locked. Contact [dbt Support](mailto:support@getdbt.com) to unlock your account, then continue with the **Existing account sign-in** flow. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). ##### dbt\_cloud.yml ##### Register with `dbt_cloud.yml` Use this path if your workflow requires a credential file or if you want to remove the need to re-authenticate when sessions expire. `dbt_cloud.yml` is a separate, complementary authentication path. You may still need `dbt_cloud.yml` for dbt platform features such as Mesh and auto-deferral. 1. Log in to dbt platform. 2. Go to **Account settings** → **VS Code extension**. 3. In the **Set up your credentials** section, click **Download credentials** to get your [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file. [![Download the dbt\_cloud.yml file from your dbt platform account.](/img/docs/extension/download-registration-2.png?v=2 "Download the dbt_cloud.yml file from your dbt platform account.")](#)Download the dbt\_cloud.yml file from your dbt platform account. 4. Move the file into your `.dbt` directory: * macOS and Linux: `~/.dbt/dbt_cloud.yml` * Windows: `C:\Users\[username]\.dbt\dbt_cloud.yml` 5. Return to your editor, open the command palette, and run **dbt: Register dbt extension**. 6. The extension detects the credential file and continues the registration flow. **Behavior details:** * If a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file exists, the extension uses it automatically. * If the file is missing, the extension prompts you to sign in or add the file. If your dbt project uses environment variables, configure them locally so the extension can use the same values as your dbt environment. For more information, refer to [Configure environment variables](https://docs.getdbt.com/docs/configure-dbt-extension.md). #### Troubleshooting If you run into any issues, check out the troubleshooting section below.  How to create a .dbt directory in root and move dbt\_cloud.yml file If you've never had a `.dbt` directory, you should perform the following recommended steps to create one. If you already have a `.dbt` directory, move the `dbt_cloud.yml` file into it. Some information about the `.dbt` directory: * A `.dbt` directory is a hidden folder in the root of your filesystem. It's used to store your dbt configuration files. The `.` prefix is used to create a hidden folder, which means it's not visible in Finder or File Explorer by default. * To view hidden files and folders, press Command + Shift + G on macOS or Ctrl + Shift + G on Windows. This opens the "Go to Folder" dialog where you can search for the `.dbt` directory. ##### Create a .dbt directory 1. Clone your dbt project repository locally. 2. Use the `mkdir` command followed by the name of the folder you want to create. * If using macOS, add the `~` prefix to create a `.dbt` folder in the root of your filesystem: * macOS: `mkdir ~/.dbt` * Windows: `mkdir %USERPROFILE%\.dbt` ##### Move the dbt\_cloud.yml file You can move the `dbt_cloud.yml` file into the `.dbt` directory using the `mv` command or by dragging and dropping the file into the `.dbt` directory by opening the Downloads folder using the "Go to Folder" dialog and then using drag-and-drop in the UI. To move the file using the terminal, use the `mv/move` command. This command moves the `dbt_cloud.yml` from the `Downloads` folder to the `.dbt` folder. If your `dbt_cloud.yml` file is located elsewhere, adjust the path accordingly. ###### Mac or Linux In your command line, use the `mv` command to move your `dbt_cloud.yml` file into the `.dbt` directory. If you've just downloaded the `dbt_cloud.yml` file and it's in your Downloads folder, the command might look something like this: ```bash mv ~/Downloads/dbt_cloud.yml ~/.dbt/dbt_cloud.yml ``` ###### Windows In your command line, use the move command. Assuming your file is in the Downloads folder, the command might look like this: ```bash move %USERPROFILE%\Downloads\dbt_cloud.yml %USERPROFILE%\.dbt\dbt_cloud.yml ```  I can't see the lineage tab in Cursor If you're using the dbt VS Code extension in Cursor, the lineage tab works best in Editor mode and doesn't render in Agent mode. If you're in Agent mode and the lineage tab isn't rendering, just switch to Editor mode to view your project's table and column lineage.  The extension gets stuck in a loading state If the extension is attempting to activate during startup and locks into a permanent loading state, check that: * Your dbt VS Code extension is on the latest version. * Your IDE is on the latest version. * You have a valid `dbt_cloud.yml` file configured and in the [correct location](#register-with-dbt_cloudyml). If you're still experiencing issues, try these steps before contacting dbt Support: * Delete and download a new copy of your `dbt_cloud.yml` file. * Delete and reinstall the dbt VS Code extension.  dbt platform configurations If you're a cloud-based dbt platform user who has the `dbt-cloud:` config in the `dbt_project.yml` file and are also using dbt Mesh, you must have the project ID configured: ```yaml dbt-cloud: project-id: 12345 # Required ``` If you don’t configure this correctly, cross-platform references will not resolve properly, and you will encounter errors executing dbt commands.  dbt extension not activating If the dbt extension has activated successfully, you will see the **dbt Extension** label in the status bar at the bottom left of your editor. You can view diagnostic information about the dbt extension by clicking the **dbt Extension** button. If the **dbt Extension** label is not present, then it is likely that the dbt extension was not installed successfully. If this happens, try uninstalling the extension, restarting your editor, and then reinstalling the extension. **Note:** It is possible to "hide" status bar items in VS Code. Double-check if the dbt Extension status bar label is hidden by right-clicking on the status bar in your editor. If you see dbt Extension in the right-click menu, then the extension has installed successfully.  Missing dbt LSP features If you receive a `no active LSP for this workspace` error message or aren't seeing dbt Language Server (LSP) features in your editor (like autocomplete, go-to-definition, or hover text), start by first following the general troubleshooting steps mentioned earlier. If you've confirmed the dbt extension is installed correctly but don't see LSP features, try the following: 1. Check extension version — Ensure that you're using the latest available version of the dbt extension by: * Opening the **Extensions** page in your editor, or * Going to the **Output** tab and looking for the version number, or * Running `dbtf --version` in the terminal. 2. Reinstall the LSP — If the version is correct, reinstall the LSP: 1. Open the Command Palette: Command + Shift + P (macOS) or Ctrl + Shift + P (Windows/Linux). 2. Paste `dbt: Reinstall dbt LSP` and enter. This command downloads the LSP and re-activates the extension to resolve the error.  Unsupported dbt version If you see an error message indicating that your version of dbt is unsupported, then there is likely a problem with your environment. Check the dbt Path setting in your VS Code settings. If this path is set, ensure that it is pointing to a valid dbt v2 executable. If necessary, you can also install dbt v2 directly using these instructions: [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2)  dbt v2 binary not found at the configured path If the extension reports that the dbt v2 binary can't be found at the configured path (for example, `dbt-fusion binary not found at [path]`), the `dbt.fusionPath` setting is pointing to a location that doesn't contain a valid binary. * Verify that [`dbt.fusionPath`](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings) points to a valid dbt v2 binary. * If you haven't installed dbt v2 manually, clear the setting and let the extension download and manage it for you. * To install manually, follow [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2).  dbt v2 version is not compatible with this extension If the extension reports that the installed dbt v2 version isn't compatible with your dbt VS Code extension version, the two are outside the supported range. 1. Run `dbt --version` to check your installed dbt v2 version. 2. Compare it against the [version compatibility matrix](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#compatibility-matrix) for your extension version. 3. Update dbt v2 or the extension so both fall within the supported range. Use the **Download compatible version** action in the notification if it appears.  dbt v2 crashes on startup If the extension reports that dbt v2 crashed on startup, confirm the binary runs on its own: 1. Run `dbt --version` in your terminal. If this fails, reinstall dbt v2 using [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2). 2. Use the **Show Logs** action in the notification (or open the **Output** tab) to review the startup error.  A known-bad dbt v2 version is installed If the extension warns that your installed dbt v2 version has a known regression, dbt Labs has flagged that release as [known-bad](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases). Update to the version named in the notification. For standalone installations: ```shell dbt system update ``` The warning persists across restarts until you update. If you work in an air-gapped environment, refer to [known-bad releases](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases) for how to distribute the manifest locally.  Addressing the 'dbt language server is not running in this workspace' error To resolve the `dbt language server is not running in this workspace` error, you need to add your dbt project folder to a workspace: 1. In VS Code, click **File** in the toolbar then select **Add Folder to Workspace**. 2. Select the dbt project file you want to add to a workspace. 3. To save your workspace, click **File** then select **Save Workspace As**. 4. Navigate to the location you want to save your workspace. This should resolve the error and open your dbt project by opening the workspace it belongs to. For more information on workspaces, refer to [What is a VS Code workspace?](https://code.visualstudio.com/docs/editing/workspaces/workspaces).  Manifest cannot be downloaded from the dbt platform If the dbt VS Code extension cannot download the manifest from the dbt platform or you get `warning: dbt1200: Failed to download manifest` using dbt v2 locally, you are probably having DNS-related issues. To confirm this, do a DNS lookup for the host dbt v2 is trying to download from (for example, prodeu2.blob.core.windows.net) by using `dig` on Linux/Mac or `nslookup` on Windows. If this doesn't return an IP address, the likely reason is that your company uses the same cloud provider with private endpoints for cloud resources, and DNS requests for these are forwarded to private DNS zones. This situation can be remedied by setting up an internet fallback, which will then return a public IP to any cloud storage that does not have a private IP registered with the private DNS zone. For Azure refer to [Fallback to internet for Azure Private DNS zones](https://learn.microsoft.com/en-us/azure/dns/private-dns-fallback). #### More information about dbt v2 * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ### Simple metrics (Applies to dbt v1.12 and later) Simple metrics are metrics that directly reference a single column expression within a semantic model, without any additional columns involved. They are aggregations over a column in your data platform and can be filtered by one or multiple dimensions. The parameters, description, and type for simple metrics are: | Parameter | Description | Required | Type | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | | `name` | The name of the metric. It must be unique within your project and can include lowercase letters, numbers, and underscores. Use this name to reference the metric from the Semantic Layer API. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of the metric (cumulative, derived, ratio, or simple). | Required | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `agg` | The aggregation function to use. Use `sum`, `max`, `min`, `average`, `median`, `count`, `count_distinct`, `percentile`, and `sum_boolean`. | Required | String | | `expr` | The expression to use, like a column name. Defaults to the metric name. | Optional | String | | `percentile` | The percentile to use. Required if `agg` is `percentile`. | Optional | Integer | | `percentile_type` | The percentile type to use. Use `discrete` or `continuous`. Required for `percentile` metrics. | Optional | String | | `non_additive_dimension` | The non-additive dimension to use. | Optional | String | | `agg_time_dimension` | The time dimension to use. | Optional | String | | `join_to_timespine` | Indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | Boolean | | `fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | Integer | The following displays the complete specification for simple metrics, along with an example. (Applies to dbt v1.12 and later) Note that you must define simple metrics within a semantic model's YAML entry. ```yaml models: - name: my_model semantic_model: enabled: true ... metrics: - name: my_simple_metric # Required description: The metric description # Optional label: My simple metric label # Optional type: simple # Required agg: count_distinct # Required sum | max | min | average | median | count_distinct | percentile, and sum_boolean expr: case when is_a then 1 else 0 end # Optional for simple metric, defaults to name of metric join_to_timespine: true fill_nulls_with: 0 - name: my_simple_metric_that_uses_the_other_time_dimension description: The metric description label: My simple metric that uses the other time dimension label type: simple agg: count_distinct agg_time_dimension: my_other_time_dimension_column # Optional, if not using the default time dimension ``` For advanced data modeling, you can use `fill_nulls_with` and `join_to_timespine` to [set null metric values to zero](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md), ensuring numeric values for every data row. #### Simple metrics example (Applies to dbt v1.12 and later) ```yaml metrics: - name: customers description: Count of customers type: simple label: Count of customers agg: count expr: customers fill_nulls_with: 0 join_to_timespine: true filter: "{{ Dimension('customer__customer_total') }} >= 20" # alias is not supported on simple metrics # use alias in input_metrics when referencing this metric in derived, ratio, or conversion metrics - name: large_orders description: "Order with order values over 20." type: simple label: Large orders agg: count expr: orders filter: "{{ Dimension('customer__order_total_dim') }} >= 20" ``` #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Snowflake and Apache Iceberg dbt supports materializing models in the Iceberg table format in two ways: * **Simplest:** The model config `table_format = 'iceberg'` instructs dbt to materialize this model as an Iceberg table in Snowflake Horizon (managed catalog). Whether dbt writes to Snowflake-managed storage by default depends on the version you're running — refer to [External volume defaults](#external-volume-defaults). * **Extensible:** Define an Iceberg catalog in `catalogs.yml` and configure this model with `catalog_name` #### Creating Iceberg tables dbt supports creating Iceberg tables for three of the Snowflake materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) * [Dynamic Table](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) #### Iceberg catalogs Snowflake supports writing Iceberg tables to Snowflake Horizon (its managed catalog), and to external catalogs through [catalog-linked databases](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database). Those external catalogs include Polaris (self-hosted), Open Catalog (Snowflake's managed Polaris), AWS Glue, GCP BigLake, Databricks Unity, and (in theory) any other catalog that implements Iceberg REST compatibility. ##### Snowflake Horizon (Snowflake-managed) ###### Simplest: Create a single Iceberg table models/MODEL\_NAME.sql ```sql {{ config( materialized = "table", table_format="iceberg", ) }} select * from {{ ref('raw_orders') }} ``` The following configurations are supported. For more information, check out the Snowflake reference for [`CREATE ICEBERG TABLE` (Snowflake as the catalog)](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake). | Parameter | Type | Required | Description | Sample input | Note | | ----------------------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `table_format` | String | Yes | Configures the objects table format. | `iceberg` | `iceberg` is the only accepted value. | | `external_volume` | String | No | Specifies the identifier (name) of the external volume where Snowflake writes the Iceberg table's metadata and data files. Set it to `SNOWFLAKE_MANAGED` to write to [Snowflake-managed storage](https://docs.snowflake.com/en/user-guide/tables-iceberg-internal-storage). | `my_s3_bucket`<br />`SNOWFLAKE_MANAGED` | Refer to [External volume defaults](#external-volume-defaults). | | `base_location_root` | String | No | If provided, the input overrides the default dbt `base_location` value of `_dbt` | `foo` | | | `base_location_subpath` | String | No | An optional suffix to add to the `base_location` path that dbt automatically specifies. | `jaffle_marketing_folder` | We recommend that you don't specify this. Modifying this parameter results in a new Iceberg table. Refer to [Base location](#base-location) for more information. | | `iceberg_version` | Integer | No | Specifies the Iceberg format version for the table. Defaults to `2`. Cannot be changed after table creation. | `3` | Set to `3` for improved `VARIANT` type support and better incremental/snapshot performance through deletion vectors. | ###### External volume defaults When a model sets `table_format: iceberg` without an `external_volume`, dbt's behavior depends on the version you're running. (Applies to dbt v2.0 and later) dbt emits `external_volume = 'SNOWFLAKE_MANAGED'` in the `CREATE ICEBERG TABLE` DDL and omits `base_location`, so the table uses Snowflake-managed storage. When you set `external_volume` to `SNOWFLAKE_MANAGED`, dbt omits `base_location` in every version. dbt resolves `external_volume` from the model config first and then from the catalog definition in `catalogs.yml`, so setting it in either place omits `base_location`. When `external_volume` names a user-defined volume, such as an Amazon S3 bucket, dbt emits both `external_volume` and `base_location`. ###### Extensible: Configure `horizon` catalog First, configure a catalog with `type: horizon` in `catalogs.yml`: ##### (Applies to dbt v2.0 and later) New spec catalogs.yml ```yaml catalogs: - name: my_horizon_catalog type: horizon table_format: iceberg config: snowflake: # optional - specify additional Snowflake-specific configurations change_tracking: true iceberg_version: 3 # available in v1.12+ ``` ##### Old spec catalogs.yml ```yml catalogs: - name: my_horizon_catalog active_write_integration: snowflake_write_integration write_integrations: - name: snowflake_write_integration external_volume: dbt_external_volume table_format: iceberg catalog_type: built_in adapter_properties: change_tracking: True iceberg_version: 3 # available in v1.12+ ``` Next, configure a dbt model with the name of your Horizon catalog. models/my\_iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_horizon_catalog', iceberg_version=3, # available in v1.12+ ) }} select * from {{ ref('jaffle_shop_customers') }} ``` Finally, run the model: `dbt run -s my_iceberg_model`. Because dbt understands that `type: horizon` refers to Snowflake's managed catalog, dbt templates the appropriate Snowflake DDL/DML for creating and updating managed Iceberg tables. ##### External catalogs dbt can also template Snowflake DDL/DML for creating and updating Iceberg tables managed by external catalogs. First, you need to set up a catalog integration and (recommended) catalog-linked database within Snowflake. See Snowflake docs for how to [create a catalog integration](https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration) and [catalog-linked database](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked). Caveats: * For some external catalogs (for example, AWS Glue), table and column identifiers must use only alphanumeric characters (letters and numbers), be lowercase, and surrounded by double quotes. (Applies to dbt v2.0 and later) * dbt supports table and incremental materializations (the `append`, `merge`, and `delete+insert` strategies) on Iceberg tables registered in a Glue catalog through a catalog-linked database. For incremental models: * They must resolve to a `type: glue` catalog in `catalogs.yml` (using `catalog_name`) whose `config.snowflake.catalog_database` names the catalog-linked database. * Their identifiers must follow Glue's lowercase, double-quoted rule (for example, set `quoting: {schema: true, identifier: true}` at the project level). After you create the external catalog integration, you can do two things: * **Query an externally managed table:** Snowflake can query Iceberg tables whose metadata lives in the external catalog. In this scenario, Snowflake is a "reader" of the external catalog. The table’s data remains in external cloud storage (AWS S3 or GCP Bucket) as defined in the catalog storage configuration. Snowflake uses the catalog integration to fetch metadata using the REST API. Snowflake then reads the data files from cloud storage. * **Write tables to the external catalog, using Snowflake compute:** You can materialize a dbt model as an Iceberg table using Snowflake's compute, and Snowflake registers and syncs that table to the external catalog (for example, AWS Glue or Databricks Unity). The dbt model appears in that catalog, and other query engines can read its data there. Now, we can configure that external catalog in `catalogs.yml`. Here is an example for an AWS Glue catalog: ##### (Applies to dbt v2.0 and later) New spec catalogs.yml ```yaml catalogs: - name: my_glue_catalog type: glue table_format: iceberg config: snowflake: catalog_database: catalog_linked_db_glue # name of catalog-linked database in Snowflake ``` ##### Old spec catalogs.yml ```yml catalogs: - name: my_glue_catalog active_write_integration: glue_rest write_integrations: - name: glue_rest catalog_type: iceberg_rest table_format: iceberg adapter_properties: catalog_linked_database: catalog_linked_db_glue # name of catalog-linked database in Snowflake catalog_linked_database_type: glue ``` #### Snowflake-specific configs for Iceberg catalogs These are the additional configurations, specific to Snowflake, that can be supplied and nested under `config.snowflake` (in the new catalog spec) or `adapter_properties` (in the old catalog spec). Available configurations are different when writing dbt models as Snowflake-managed Iceberg tables (Snowflake Horizon catalog) versus writing to external catalogs. ###### Snowflake-managed (Horizon) (Applies to dbt v1.12 and later) | Field | Required | Accepted values | | --------------------------------- | -------- | --------------------------------------------------------------------------------------- | | `change_tracking` | Optional | `True` or `False` | | `data_retention_time_in_days` | Optional | Standard Account: `1`, Enterprise or higher: `0` to `90`, default `1` | | `max_data_extension_time_in_days` | Optional | `0` to `90` with a default of `14` | | `storage_serialization_policy` | Optional | `COMPATIBLE` or `OPTIMIZED` | | `base_location_root` | Optional | Relative path segment (for example, `'subpath1/subpath2'`) | | `base_location_subpath` | Optional | Relative path segment (for example, `'subpath1/subpath2'`), only configurable per-model | | `iceberg_version` (v1.12+) | Optional | `2` (default) or `3` | ###### External catalogs (Applies to dbt v1.12 and later) | Field | Required | Accepted values | | --------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------- | | `auto_refresh` | Optional | `True` or `False` | | `catalog_linked_database` | Required for `catalog type: iceberg_rest` | Catalog-linked database name | | `catalog_linked_database_type` | Optional | Catalog-linked database type. For example, `glue` | | `max_data_extension_time_in_days` | Optional | `0` to `90` (default: `14`) | | `target_file_size` | Optional | Values like `'AUTO'`, `'16MB'`, `'32MB'`, `'64MB'`, `'128MB'`. Case-insensitive | | `iceberg_version` (v1.12+) | Optional | `2` (default) or `3` | * **storage\_serialization\_policy:** The serialization policy tells Snowflake what kind of encoding and compression to perform on the table data files. If not specified at table creation, the table inherits the value set at the schema, database, or account level. If the value isn’t specified at any level, the table uses the default value. You can’t change the value of this parameter after table creation. * **max\_data\_extension\_time\_in\_days:** The maximum number of days Snowflake can extend the data retention period for tables to prevent streams on the tables from becoming stale. The `MAX_DATA_EXTENSION_TIME_IN_DAYS` parameter enables you to limit this automatic extension period to control storage costs for data retention, or for compliance reasons. * **data\_retention\_time\_in\_days:** For managed Iceberg tables, you can set a retention period for Snowflake Time Travel and undropping the table over the default account values. For tables that use an external catalog, Snowflake uses the value of the DATA\_RETENTION\_TIME\_IN\_DAYS parameter to set a retention period for Snowflake Time Travel and undropping the table. When the retention period expires, Snowflake doesn't delete the Iceberg metadata or snapshots from your external cloud storage. * **change\_tracking:** Specifies whether to enable change tracking on the table. * **catalog\_linked\_database:** [Catalog-linked databases](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database) (CLD) in Snowflake ensure that Snowflake can automatically sync metadata (including namespaces and Iceberg tables) from the external Iceberg catalog and registers them as remote tables in the catalog-linked database. The reason we require the usage of Catalog-linked databases for building Iceberg tables with external catalogs is that without it, dbt is unable to truly manage the table end-to-end. Snowflake doesn't support dropping the Iceberg table on non-CLDs in the external catalog; instead, it only allows unlinking the Snowflake table, which creates a discrepancy with how dbt expects to manage the materialized object. * **auto\_refresh:** Specifies whether Snowflake should automatically poll the external Iceberg catalog for metadata updates. If `REFRESH_INTERVAL_SECONDS` isn’t set on the catalog integration, the default refresh interval is 30 seconds. * **target\_file\_size:** Specifies a target Parquet file size. Default is `AUTO`. (Applies to dbt v1.12 and later) * **iceberg\_version:** Specifies the Iceberg format version for the table. Default value is `2`. Set to `3` for improved support for `VARIANT` data types and faster incremental operations. Version 3 uses [deletion vectors](https://docs.snowflake.com/en/user-guide/tables-iceberg-manage#tables-iceberg-deletion-vectors), which let Snowflake mark rows as deleted without rewriting the underlying data files, making incremental runs faster. Note that you can't change the Iceberg version after table creation. As an alternative, you can [configure the default Iceberg version](https://docs.snowflake.com/en/user-guide/tables-iceberg-v3-specification-support#configure-the-default-iceberg-version) at the account, database, or schema level in Snowflake. - **base\_location\_root:** Specifies the prefix of the [`BASE_LOCATION`](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#optional-parameters), the write path for the Iceberg table. - **base\_location\_subpath:** Specifies the suffix of the [`BASE_LOCATION`](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#optional-parameters), the write path for the Iceberg table. This property can only be set in model configurations, not in `catalogs.yml`. ##### Base location Snowflake's `CREATE ICEBERG TABLE` DDL requires a `base_location` when the table uses a custom external volume. dbt defines this parameter on your behalf to streamline usage and enforce basic isolation of table data within the `EXTERNAL VOLUME`. By default, dbt provides a `base_location` string of the form `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`. Snowflake rejects `BASE_LOCATION` for Iceberg tables that use Snowflake-managed storage, so dbt omits it when you set `external_volume` to `SNOWFLAKE_MANAGED`. dbt also omits it when you leave `external_volume` unset, depending on the version you're running — refer to [External volume defaults](#external-volume-defaults). We recommend using the default behavior, but if you need to customize the resulting `base_location`, you can configure the `base_location` with the model configuration fields `base_location_root` and `base_location_subpath`. (Applies to dbt v2.0 and later) `base_location_subpath` is only accepted in model configurations. * If no inputs are provided, dbt outputs for base\_location `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}` * If base\_location\_root = `foo`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}` * If base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}/bar` * If base\_location\_root = `foo` and base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}/bar` While you can customize paths with `base_location_root` and `base_location_subpath`, we don't recommend you rely on these for environment isolation (such as separating development and production environments). These configuration values can be easily modified by anyone with repository access. For true environment isolation, use separate `EXTERNAL VOLUME`s with infrastructure-level access controls. An example model with a customized `base_location`: iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_horizon_catalog', base_location_root='foo', base_location_subpath='bar', ) }} select * from {{ ref('jaffle_shop_customers') }} ``` note While you can customize paths with `base_location_root` and `base_location_subpath`, we don't recommend relying on them for environment isolation (such as separating development and production environments). Anyone with repository access can easily modify these configuration values. For true environment isolation, use separate `external_volume` values with infrastructure-level access controls. ###### Rationale By default, dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog keeps track of your Iceberg table regardless of where the data lives within the `external volume` declared and the `base_location` provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This behavior could result in future technical debt because it limits the ability to: * Navigate the underlying object store (S3/Azure blob) * Read Iceberg tables through an object-store integration * Grant schema-specific access to tables through object store * Use a crawler pointed at the tables within the external volume to build a new catalog with another tool To maintain best practices, dbt enforces an input and, by default, writes your tables within a `_dbt/{SCHEMA_NAME}/{TABLE_NAME}` prefix to ensure easier object-store observability and auditability. ##### Limitations * When you use Iceberg tables with dbt, dbt materializes your query in Iceberg. However, dbt often creates intermediary objects as temporary and transient tables for certain materializations, such as incremental ones. You can't configure these temporary objects to be Iceberg-formatted. You may see non-Iceberg tables created in the logs to support specific materializations, but they are dropped after usage. * You can't incrementally update a pre-existing incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag. * As of Snowflake change bundle `2025-01`, the `SHOW TABLES` command doesn't include the `is_iceberg` column in its output. This forced dbt v1.9 to run a command similar to the following query for all the models in the dbt project (regardless of whether they're configured as `iceberg` models): ```sql select all_objects.*, is_iceberg from table(result_scan(last_query_id(-1))) all_objects left join INFORMATION_SCHEMA.tables as all_tables on all_tables.table_name = all_objects."name" and all_tables.table_schema = all_objects."schema_name" and all_tables.table_catalog = all_objects."database_name" ``` This query may be relatively inefficient and potentially expensive, depending on the size of your Snowflake warehouse. Thus, the ability to run Iceberg models is gated behind the `enable_iceberg_materializations` flag. --- ### Source freshness dbt platform dbt provides a helpful interface around dbt's [source data freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) calculations. When a dbt job is configured to run source freshness checks, dbt will render a user interface showing you the state of the most recent check. This interface is intended to help you determine if your source data freshness is meeting the service level agreement (SLA) that you've defined for your organization. [![Data Sources in dbt](/img/docs/dbt-platform/using-dbt-platform/data-sources-next.png?v=2 "Data Sources in dbt")](#)Data Sources in dbt ##### Enabling source freshness checks [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) does *not* include source freshness checks when building and testing resources in your DAG. Instead, you can use one of these common patterns for defining jobs: * Add `dbt build` to the run step to run models, tests, and so on. * (Applies to dbt v2.0 and later) Every job run refreshes [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) metadata automatically, so you don't need a docs step. * Select the **Run source freshness** checkbox to enable [source freshness](#checkbox) as the first step of the job. To enable source freshness checks, first make sure to configure your sources with [source freshness information](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness). You can add source freshness to the list of commands in the job run steps or enable the checkbox. However, you can expect different outcomes when you configure a job by selecting the **Run source freshness** checkbox compared to adding the command to the run steps. Review the following options and outcomes: | Options | Outcomes | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Select checkbox[]()** | The **Run source freshness** checkbox in your **Execution Settings** will run `dbt source freshness` as the first step in your job and won't break subsequent steps if it fails. If you wanted your job dedicated *exclusively* to running freshness checks, you still need to include at least one placeholder step, such as `dbt compile`. | | **Add as a run step** | Add the `dbt source freshness` command to a job anywhere in your list of run steps. However, if your source data is out of date — this step will "fail", and subsequent steps will not run. dbt will trigger email notifications (if configured) based on the end state of this step.<br /><br />You can create a new job to check source freshness.<br /><br />If you *do not* want your models to run if your source data is out of date, then it could be a good idea to run `dbt source freshness` as the first step in your job. Otherwise, we recommend adding `dbt source freshness` as the last step in the job, or creating a separate job just for this task. | [![Adding a step to check source freshness](/img/docs/dbt-platform/using-dbt-platform/job-step-source-freshness.png?v=2 "Adding a step to check source freshness")](#)Adding a step to check source freshness ##### Source freshness check frequency It's important that your freshness jobs run frequently enough to measure data latency in accordance with your SLAs. You can imagine that if you have a 1 hour SLA on a particular dataset, checking the freshness of that table once daily would not be appropriate. As a good rule of thumb, you should run your source freshness jobs with at least double the frequency of your lowest SLA. Here's an example table of some reasonable check frequencies given typical SLAs: | SLA | Check frequency | | ------ | --------------- | | 1 hour | 30 mins | | 1 day | 12 hours | | 1 week | About daily | #### Further reading * Refer to [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) for more info on how to create dbt artifacts, share links to the latest documentation, and share source freshness reports with your team. * Source freshness for Snowflake is calculated using the `LAST_ALTERED` column. Read about the limitations in [Snowflake configs](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#source-freshness-known-limitation). --- ### SQL models #### Related reference docs * [Model configurations](https://docs.getdbt.com/reference/model-configs.md) * [Model properties](https://docs.getdbt.com/reference/model-properties.md) * [`run` command](https://docs.getdbt.com/reference/commands/run.md) * [`ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) #### Getting started Building your first models If you're new to dbt, we recommend that you read a [quickstart guide](https://docs.getdbt.com/guides.md) to build your first dbt project with models. dbt's Python capabilities are an extension of its capabilities with SQL models. If you're new to dbt, we recommend that you read this page first, before reading: ["Python Models"](https://docs.getdbt.com/docs/build/python-models.md) A SQL model is a `select` statement. Models are defined in `.sql` files (typically in your `models` directory): * Each `.sql` file contains one model / `select` statement * The model name is inherited from the filename and must match the *filename* of a model — including case sensitivity. Any mismatched casing can prevent dbt from applying configurations correctly and may affect metadata in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). * We strongly recommend using underscores for model names, not dots. For example, use `models/my_model.sql` instead of `models/my.model.sql`. * Models can be nested in subdirectories within the `models` directory. * Starting in dbt v1.12, you can use Jinja-style suffixes (`.j2`, `.jinja`, `.jinja2`) on `.sql` files (for example, `my_model.sql.j2`) by enabling the [`allow_jinja_file_extensions`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) flag. Refer to [How we style our dbt models](https://docs.getdbt.com/best-practices/how-we-style/1-how-we-style-our-dbt-models.md) for details on how we recommend you name your models. When you execute the [`dbt run` command](https://docs.getdbt.com/reference/commands/run.md), dbt will build this model data warehouse by wrapping it in a `create view as` or `create table as` statement. For example, consider this `customers` model: models/customers.sql ```sql with customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from jaffle_shop.orders group by 1 ) select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from jaffle_shop.customers left join customer_orders using (customer_id) ``` When you execute `dbt run`, dbt will build this as a *view* named `customers` in your target schema: ```sql create view dbt_alice.customers as ( with customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from jaffle_shop.orders group by 1 ) select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from jaffle_shop.customers left join customer_orders using (customer_id) ) ``` Why a *view* named `dbt_alice.customers`? By default dbt will: * Create models as views * Build models in a target schema you define * Use your file name as the view or table name in the database You can use *configurations* to change any of these behaviors — more on that later. ##### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. Which SQL dialect should I write my models in? Or which SQL dialect does dbt use? dbt can feel like magic, but it isn't actually magic. Under the hood, it's running SQL in your own warehouse — your data is not processed outside of your warehouse. As such, your models should just use the **SQL dialect of your own database**. Then, when dbt wraps your `select` statements in the appropriate DDL or DML, it will use the correct DML for your warehouse — all of this logic is written in to dbt. You can find more information about the databases, platforms, and query engines that dbt supports in the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) docs. Want to go a little deeper on how this works? Consider a snippet of SQL that works on each warehouse: models/test\_model.sql ```sql select 1 as my_column ``` To replace an existing table, here's an *illustrative* example of the SQL dbt will run on different warehouses (the actual SQL can get much more complicated than this!) ##### Redshift ```sql -- you can't create or replace on redshift, so use a transaction to do this in an atomic way begin; create table "dbt_alice"."test_model__dbt_tmp" as ( select 1 as my_column ); alter table "dbt_alice"."test_model" rename to "test_model__dbt_backup"; alter table "dbt_alice"."test_model__dbt_tmp" rename to "test_model" commit; begin; drop table if exists "dbt_alice"."test_model__dbt_backup" cascade; commit; ``` ##### BigQuery ```sql -- Make an API call to create a dataset (no DDL interface for this)!!; create or replace table `dbt-dev-87681`.`dbt_alice`.`test_model` as ( select 1 as my_column ); ``` ##### Snowflake ```sql create schema if not exists analytics.dbt_alice; create or replace table analytics.dbt_alice.test_model as ( select 1 as my_column ); ``` #### Configuring models Configurations are "model settings" that you can set in your `dbt_project.yml` file, *and* in your model file using a `config` block. Some example configurations include: * Changing the materialization that a model uses — a [materialization](https://docs.getdbt.com/docs/build/materializations.md) determines the SQL that dbt uses to create the model in your warehouse. * Build models into separate [schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). * Apply [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to a model. The following diagram shows an example directory structure of a models folder: ```text models ├── staging └── marts └── marketing ``` Here's an example of a model configuration: dbt\_project.yml ```yaml name: jaffle_shop config-version: 2 ... models: jaffle_shop: # this matches the `name:`` config +materialized: view # this applies to all models in the current project marts: +materialized: table # this applies to all models in the `marts/` directory marketing: +schema: marketing # this applies to all models in the `marts/marketing/`` directory ``` models/customers.sql ```sql {{ config( materialized="view", schema="marketing" ) }} with customer_orders as ... ``` It is important to note that configurations are applied hierarchically — a configuration applied to a subdirectory will override any general configurations. You can learn more about configurations in the [reference docs](https://docs.getdbt.com/reference/model-configs.md). ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Building dependencies between models You can build dependencies between models by using the [`ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) in place of table names in a query. Use the name of another model as the argument for `ref`. ##### Model models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), ... ``` ##### Compiled code in dev ```sql create view dbt_alice.customers as ( with customers as ( select * from dbt_alice.stg_customers ), orders as ( select * from dbt_alice.stg_orders ), ... ) ... ``` ##### Compiled code in prod ```sql create view analytics.customers as ( with customers as ( select * from analytics.stg_customers ), orders as ( select * from analytics.stg_orders ), ... ) ... ``` dbt uses the `ref` function to: * Determine the order to run the models by creating a dependent acyclic graph (DAG). [![The DAG for our dbt project](/img/dbt-dag.png?v=2 "The DAG for our dbt project")](#)The DAG for our dbt project * Manage separate environments — dbt will replace the model specified in the `ref` function with the database name for the table (or view). Importantly, this is environment-aware — if you're running dbt with a target schema named `dbt_alice`, it will select from an upstream table in the same schema. Check out the tabs above to see this in action. Additionally, the `ref` function encourages you to write modular transformations, so that you can re-use models, and reduce repeated code. #### Testing and documenting models You can also document and test models — skip ahead to the section on [testing](https://docs.getdbt.com/docs/build/data-tests.md) and [documentation](https://docs.getdbt.com/docs/build/documentation.md) for more information. #### Additional FAQs Are there any example dbt models? Yes! * **Quickstart Tutorial:** You can build your own example dbt project in the [quickstart guide](https://docs.getdbt.com/docs/get-started-dbt.md) * **Jaffle Shop:** A demonstration project (closely related to the tutorial) for a fictional e-commerce store. Refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) for clone instructions, or view the [main source code](https://github.com/dbt-labs/jaffle-shop) and [source code using DuckDB](https://github.com/dbt-labs/jaffle_shop_duckdb) on GitHub. * **GitLab:** Gitlab's internal dbt project is open source and is a great example of how to use dbt at scale ([source code](https://gitlab.com/gitlab-com/content-sites/handbook/blob/main/content/handbook/enterprise-data/platform/dbt-guide.md)) * **dummy-dbt:** A containerized dbt project that populates the Sakila database in Postgres and populates dbt seeds, models, snapshots, and tests. The project can be used for testing and experimentation purposes ([source code](https://github.com/gmyrianthous/dbt-dummy)) * **Google Analytics 4:** A demonstration project that transforms the Google Analytics 4 BigQuery exports to various models ([source code](https://github.com/stacktonic-com/stacktonic-dbt-example-project), [docs](https://stacktonic.com/article/google-analytics-big-query-and-dbt-a-dbt-example-project)) * **Make Open Data:** A production-grade ELT with tests, documentation, and CI/CD (GHA) about French open data (housing, demography, geography, etc). It can be used to learn with voluminous and ambiguous data. Contributions are welcome ([source code](https://github.com/make-open-data/make-open-data). If you have an example project to add to this list, suggest an edit by clicking **Edit this page** below. Can I store my models in a directory other than the \`models\` directory in my project? By default, dbt expects the files defining your models to be located in the `models` subdirectory of your project. To change this, update the [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml model-paths: ["transformations"] ``` Can I build my models in a schema other than my target schema or split my models across multiple schemas? Yes! Use the [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) configuration in your `dbt_project.yml` file, or using a `config` block: dbt\_project.yml ```yml name: jaffle_shop ... models: jaffle_shop: marketing: +schema: marketing # models in the `models/marketing/` subdirectory will use the marketing schema ``` models/customers.sql ```sql {{ config( schema='core' ) }} ``` Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). If models can only be \`select\` statements, how do I insert records? For those coming from an ETL (Extract Transform Load) paradigm, there's often a desire to write transformations as `insert` and `update` statements. In comparison, dbt will wrap your `select` query in a `create table as` statement, which can feel counter-productive. * If you wish to use `insert` statements for performance reasons (i.e. to reduce data that is processed), consider [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * If you wish to use `insert` statements since your source data is constantly changing (e.g. to create "Type 2 Slowly Changing Dimensions"), consider [snapshotting your source data](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness), and building models on top of your snapshots. Why can't I just write DML in my transformations? ###### `select` statements make transformations accessible More people know how to write `select` statements, than DML, making the transformation layer accessible to more people! ###### Writing good DML is hard If you write the DDL / DML yourself you can end up getting yourself tangled in problems like: * What happens if the table already exists? Or this table already exists as a view, but now I want it to be a table? * What if the schema already exists? Or, should I check if the schema already exists? * How do I replace a model atomically (such that there's no down-time for someone querying the table) * What if I want to parameterize my schema so I can run these transformations in a development environment? * What order do I need to run these statements in? If I run a `cascade` does it break other things? Each of these problems *can* be solved, but they are unlikely to be the best use of your time. ###### dbt does more than generate SQL You can test your models, generate documentation, create snapshots, and more! ###### You reduce your vendor lock in SQL dialects tend to diverge the most in DML and DDL (rather than in `select` statements) — check out the example [here](https://docs.getdbt.com/faqs/Models/sql-dialect.md). By writing less SQL, it can make a migration to a new database technology easier. If you do need to write custom DML, there are ways to do this in dbt using [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). How do I specify column types? Simply cast the column to the correct type in your model: ```sql select id, created::timestamp as created from some_other_table ``` You might have this question if you're used to running statements like this: ```sql create table dbt_alice.my_table id integer, created timestamp; insert into dbt_alice.my_table ( select id, created from some_other_table ) ``` In comparison, dbt would build this table using a `create table as` statement: ```sql create table dbt_alice.my_table as ( select id, created from some_other_table ) ``` So long as your model queries return the correct column type, the table you create will also have the correct column type. To define additional column options: * Rather than enforcing uniqueness and not-null constraints on your column, use dbt's [data testing](https://docs.getdbt.com/docs/build/data-tests.md) functionality to check that your assertions about your model hold true. * Rather than creating default values for a column, use SQL to express defaults (e.g. `coalesce(updated_at, current_timestamp()) as updated_at`) * In edge-cases where you *do* need to alter a column (e.g. column-level encoding on Redshift), consider implementing this via a [post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). --- ### Supported data platforms dbt connects to and runs SQL against your database, warehouse, lake, or query engine. These SQL-speaking platforms are collectively referred to as *data platforms*. dbt connects with data platforms by using a dedicated adapter plugin for each. Plugins are built as Python modules that dbt v1 discovers if they are installed on your system. Refer to the [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) guide for details. Adapters are maintained by dbt Labs, partners, and community members. Refer to [community adapters](https://docs.getdbt.com/docs/community-adapters.md) for the combined list. ##### Install an adapter To use dbt from the command line, install the adapter for your data platform and set up a `profiles.yml` file. Refer to [installing dbt](https://docs.getdbt.com/docs/local/install-dbt.md). (Applies to dbt v2.0 and later) Adapters ship with dbt v2. When you [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md), the supported data platforms are available with no separate `pip install` for each adapter. Refer to [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md?step=1) for more information. (Applies to dbt v2.0 and later) #### Adapter lifecycle dbt v2 is available across adapters (data warehouse connectors). Track status by adapter using the following table: | Adapter | Lifecycle | | ----------------------- | ------------------- | | Snowflake | Generally available | | BigQuery | Generally available | | Databricks | Generally available | | Redshift | Generally available | | Apache Spark (CLI only) | Beta | | DuckDB (CLI only) | Generally available | | ClickHouse | Private beta | *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* Considerations for depending on an open-source project 1. Does it work? 2. Does anyone "own" the code, or is anyone liable for ensuring it works? 3. Do bugs get fixed quickly? 4. Does it stay up-to-date with new dbt v1 features? 5. Is the usage substantial enough to self-sustain? 6. Do other known projects depend on this library? #### Footnotes 1. Use the PyPI package name when installing with `pip`. [↩](#user-content-fnref-1) --- ### Tableau dbt platform | Starter, Enterprise, Enterprise+ The Tableau integration allows you to use worksheets to query the Semantic Layer directly and produce your dashboards with trusted data. It provides a live connection to the Semantic Layer through Tableau Desktop or Tableau Server. #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and are using dbt v1.6 or higher. * You must have [Tableau Desktop](https://www.tableau.com/en-gb/products/desktop) version 2021.1 and greater, Tableau Server, or [Tableau Cloud](https://www.tableau.com/products/cloud-bi). * Log in to Tableau Desktop (with Cloud or Server credentials) or Tableau Cloud. You can also use a licensed Tableau Server deployment. * You need your [dbt host](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#3-view-connection-detail), [Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer), and a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to log in. This account should be set up with the Semantic Layer. * You must have a dbt Starter or Enterprise-tier [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Installing the connector The Semantic Layer Tableau connector is available to download directly on [Tableau Exchange](https://exchange.tableau.com/products/1020). The connector is supported in Tableau Desktop, Tableau Server, and Tableau Cloud. Alternatively, you can follow these steps to install the connector. Note that these steps only apply to Tableau Desktop and Tableau Server. The connector for Tableau Cloud is managed by Tableau. 1. Download the GitHub [connector file](https://github.com/dbt-labs/semantic-layer-tableau-connector/releases/latest/download/dbt_semantic_layer.taco) locally and add it to your default folder: | Operating system | Tableau Desktop | Tableau Server | | ---------------- | --------------------------------------------------------------------- | ------------------------------------- | | Windows | `C:\Users\\[Windows User]\Documents\My Tableau Repository\Connectors` | `C:\Program Files\Tableau\Connectors` | | Mac | `/Users/[user]/Documents/My Tableau Repository/Connectors` | Not applicable | | Linux | `/opt/tableau/connectors` | `/opt/tableau/connectors` | 2. Install the [JDBC driver](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) to the folder based on your operating system: * Windows: `C:\Program Files\Tableau\Drivers` * Mac: `~/Library/Tableau/Drivers` or `/Library/JDBC` or `~/Library/JDBC` * Linux: `/opt/tableau/tableau_driver/jdbc` 3. Open Tableau Desktop or Tableau Server and find the **Semantic Layer by dbt Labs** connector on the left-hand side. You may need to restart these applications for the connector to be available. 4. Connect with your Host, Environment ID, and service or personal token information dbt provides during the [Semantic Layer configuration](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * In Tableau Server, the authentication screen may show "User" & "Password" instead, in which case the User is the Environment ID and the password is the Service Token. #### Using the integration 1. **Authentication** — Once you authenticate, the system will direct you to the data source page. 2. **Access all Semantic Layer Objects** — Use the "ALL" data source to access all the metrics, dimensions, and entities configured in your Semantic Layer. Note that the "METRICS\_AND\_DIMENSIONS" data source has been deprecated and replaced by "ALL". Be sure to use a live connection since extracts are not supported at this time. 3. **Access saved queries** — You can optionally access individual [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) that you've defined. These will also show up as unique data sources when you log in. 4. **Access worksheet** — From your data source selection, go directly to a worksheet in the bottom left-hand corner. 5. **Query metrics and dimensions** — Then, you'll find all the metrics, dimensions, and entities that are available to query on the left side of your window based on your selection. Visit the [Tableau documentation](https://help.tableau.com/current/pro/desktop/en-us/gettingstarted_overview.htm) to learn more about how to use Tableau worksheets and dashboards. ##### Publish from Tableau Desktop to Tableau Server * **From Desktop to Server** — Like any Tableau workflow, you can publish your workbook from Tableau Desktop to Tableau Server. For step-by-step instructions, visit Tableau's [publishing guide](https://help.tableau.com/current/pro/desktop/en-us/publish_workbooks_share.htm). ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). #### Things to note **Aggregation**<br /> * All metrics are shown as using the "SUM" aggregation type in Tableau's UI, and this cannot be altered using Tableau's interface. * The Semantic Layer controls the aggregation type in code and it is intentionally fixed. Keep in mind that the underlying aggregation in the Semantic Layer might not be "SUM" ("SUM" is Tableau's default). **Data sources and display**<br /> * In the "ALL" data source, Tableau surfaces all metrics and dimensions from the Semantic Layer on the left-hand side. Note, that not all metrics and dimensions can be combined. You will receive an error message if a particular dimension cannot be sliced with a metric (or vice versa). You can use saved queries for smaller pieces of data that you want to combine. * To display available metrics and dimensions, Semantic Layer returns metadata for a fake table with the dimensions and metrics as 'columns' on this table. Because of this, you can't actually query this table for previews or extracts. **Calculations and querying**<br /> * Certain Table calculations like "Totals" and "Percent Of" may not be accurate when using metrics aggregated in a non-additive way (such as count distinct) * In any of our Semantic Layer interfaces (not only Tableau), you must include a [time dimension](https://docs.getdbt.com/docs/build/cumulative.md#limitations) when working with any cumulative metric that has a time window or granularity. * We can support calculated fields for creating parameter filters or dynamically selecting metrics and dimensions. However, other uses of calculated fields are not supported. * *Note: For calculated field use cases that are not currently covered, please reach out to [dbt Support](<mailto:support@getdbt.com?subject=dbt Semantic Layer feedback>) and share them so we can further understand.* * When using saved queries that include filters, we will automatically apply any filters that the query has. #### Unsupported functionality The following Tableau features aren't supported at this time, however, the Semantic Layer may support some of this functionality in a future release: * Updating the data source page * Using "Extract" mode to view your data * Unioning Tables * Writing Custom SQL / Initial SQL * Table Extensions * Cross-Database Joins * Some functions in Analysis --> Create Calculated Field * Filtering on a Date Part time dimension for a Cumulative metric type * Changing your date dimension to use "Week Number" * Performing joins between tables that the Semantic Layer creates. It handles joins for you, so there's no need to join components in the Semantic Layer. Note, that you *can* join tables from the Semantic Layer to ones outside your data platform. * The Tableau integration doesn't currently display descriptive labels defined in your `metrics` configuration, meaning custom labels won't be visible when those metrics are imported/queried into Tableau. #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### Trial and billing Start a dbt Wizard trial, set a spend limit, and manage paid access for both the dbt platform and local CLI. This page covers how to get dbt Wizard access and pay for it. For the models dbt Wizard can use and how tokens and credits work, refer to [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md). Billing and spend controls are shared across the dbt platform and local CLI. Both surfaces draw from the same account-level balance. #### Prerequisites * Account admin or billing admin permissions to start a trial or change a spend limit. * A dbt account to manage usage, billing, spend limits, and more. * If you don't have one, you can create one during setup. No paid dbt platform plan required. This is generally useful for users on self-hosted dbt running the CLI. * A business email address. Personal domains such as Gmail aren't eligible for a trial. #### What you get by plan Every new account gets free dbt Wizard usage credits to start. What you get, and how you keep going, depends on your plan. | Plan | What you get | How it renews | When it runs out | | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------ | | Developer and Starter, or self-hosted dbt with a free dbt account | 30-day trial with $100 in usage credits per account | One-time | Add a credit card and set a monthly spend limit or contact your account team. | | Enterprise, including [legacy Enterprise](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) | $100/month in usage credits per account | Resets each billing month | [Contact your account team](https://www.getdbt.com/contact) to add committed spend | | Enterprise+ | $200/month in usage credits per account | Resets each billing month | [Contact your account team](https://www.getdbt.com/contact) to add committed spend | | [Legacy Team](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) | No access to dbt Wizard | — | Move to a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing) | * On Developer and Starter plans, the trial is opt-in, so it won't start automatically. It ends when you use up the $100 in credits or after 30 days, whichever comes first, and unused credits don't carry over. * Legacy Enterprise plans get the same dbt Wizard access and monthly usage credits as current Enterprise. Only legacy Team has no access. * Enterprise and Enterprise+ usage credits are granted automatically — there's no trial to start and no credit card required. Credits don't roll over or get prorated, and if you downgrade out of Enterprise or Enterprise+, any unused credits are removed at the plan change. * Enterprise and Enterprise+ accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. Running dbt Wizard from the CLI against a self-hosted dbt project? The trial is opt-in here too — running `dbt login` (or `wizard login`) is how you opt in. The command creates your free dbt account and starts the same 30-day trial, and that account is where you manage usage and spend limits. Everything on this page describes dbt managed billing — usage that dbt Labs bills through your dbt account. If you bring your own key, your AI provider bills you directly and none of this applies. Refer to [BYOK for dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) instead. #### Start your trial Start your trial from anywhere in the dbt platform or from the dbt Wizard CLI. Note, Enterprise-tiered plans automatically have a [spend limit set](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md#what-does-the-dbt-wizard-enterprise-plan-monthly-usage-credits-include) ##### dbt platform Start from **Billing & Usage**, or from the dbt Wizard prompt in Studio IDE or the home tab as they all take you to the same flow. 1. Click your account name, then select **Account settings**. (Or click on any button that says "Start trial" to start your Wizard trial.) 2. Under **Settings**, click **Billing & Usage**. 3. On the **Overview** tab, find the **dbt Wizard** card and click **Start trial**. No credit card is required. Your 30-day trial with $100 in usage credits starts right away. The $100 is per account and shared by everyone on it, and you can track how much you've used anytime in **Billing & Usage**. [![The Billing & Usage Overview page, showing dbt State and dbt Wizard cards with Start trial buttons, plus a usage-by-month chart](/img/docs/dbt-platform/wizard-billing-overview.png?v=2 "The Billing & Usage Overview page, showing dbt State and dbt Wizard cards with Start trial buttons, plus a usage-by-month chart")](#)The Billing & Usage Overview page, showing dbt State and dbt Wizard cards with Start trial buttons, plus a usage-by-month chart ##### Wizard CLI There's no **Start trial** button in the CLI. Logging in is what starts your trial — one command creates your free dbt account, if you don't have one, and provisions the 30-day trial with $100 in usage credits for your account at the same time. The $100 is per account, not per user, so it's shared with anyone else on your account. 1. [Install dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). 2. Run `dbt login` and complete the browser sign-in, or create a new account to manage your dbt Wizard spend limits: ```shell dbt login ``` 3. Run `wizard` in your project and choose **dbt-managed** when onboarding asks how AI usage is billed. You don't need an AI provider key. You don't need a paid dbt platform plan, so this path is the same whether you're on a dbt platform plan or running against a self-hosted dbt project. Refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) for the full onboarding walkthrough. #### Set up paid usage What you do next depends on your plan. ##### Developer, Starter, or self hosted 1. Go to **Billing & Usage > Usage-based features**. 2. Click **Set up billing**. 3. Add a credit card and fill in your payment details. Click **Save card**. 4. Complete your setup to choose a preset monthly spend limit, or set your own with the **Custom** option. 5. Optionally, turn on automatic increases for when you're close to your limit. 6. Click **Continue**. 7. Review and confirm your spend limit. Click **Activate dbt Wizard** to start your billing. You won't be charged today and will be billed monthly for actual usage, up to the limit you set. Usage then pauses if you reach that limit. [![The Add billing page, showing a credit card form and a complete billing setup flow](/img/docs/dbt-platform/wizard-add-billing.png?v=2 "The Add billing page, showing a credit card form and a complete billing setup flow")](#)The Add billing page, showing a credit card form and a complete billing setup flow [![The Set your dbt Wizard spend limit page, showing pre-set monthly options, a Custom option, and an auto-raise toggle](/img/docs/dbt-platform/wizard-manage-spend.png?v=2 "The Set your dbt Wizard spend limit page, showing pre-set monthly options, a Custom option, and an auto-raise toggle")](#)The Set your dbt Wizard spend limit page, showing pre-set monthly options, a Custom option, and an auto-raise toggle [![The Activate dbt Wizard page where you can review and confirm your spend limit and a button to activate dbt Wizard](/img/docs/dbt-platform/wizard-activate.png?v=2 "The Activate dbt Wizard page where you can review and confirm your spend limit and a button to activate dbt Wizard")](#)The Activate dbt Wizard page where you can review and confirm your spend limit and a button to activate dbt Wizard ##### Enterprise and Enterprise+ plans There's no trial to start and no self-serve credit card flow. Your monthly usage credits — $100/month on Enterprise and $200/month on Enterprise+ — are granted automatically per account, not per user, and shared by everyone on the account. [Contact your account team](https://www.getdbt.com/contact) to set up or adjust committed spend. Enterprise and Enterprise+ accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. #### Manage your spend limit Your spend limit caps how much dbt managed dbt Wizard usage your account can consume in a billing period, across both the dbt platform and local development. * You only pay for actual usage, up to the limit you choose. The limit is a cap, not a prepaid charge. * If you reach your limit, dbt Wizard usage pauses until you raise it or the next billing cycle starts. This applies on every plan, including Enterprise-tiered plans that set the optional limit. Enterprise-tiered accounts that don't have a committed spend amount will be prompted to connect with their account rep about adding a spend amount. * Limits are set separately for dbt Wizard and [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), but both draw from your account's overall usage-based spend. To view or update your limit, go to **Billing & Usage > Usage-based features > Wizard**. Enterprise-tiered plans can [contact their account team](https://www.getdbt.com/contact) to adjust their limit. [![The Manage your dbt Wizard spend limit page, showing pre-set monthly options and a Custom option.](/img/docs/dbt-platform/wizard-manage-spend.png?v=2 "The Manage your dbt Wizard spend limit page, showing pre-set monthly options and a Custom option.")](#)The Manage your dbt Wizard spend limit page, showing pre-set monthly options and a Custom option. #### View your usage and costs To see what you've spent, go to **Account settings > Billing & Usage > Usage-based features** and open the **Wizard** tab. From there you can check: * How much of your included monthly usage you've used, what's left, and when it resets. * Your current dbt Wizard spend limit, with an **Edit** button to change it. * **dbt Wizard usage by model**, which breaks down your usage (in UTC) so you can see which models are driving your costs. #### How usage is measured dbt Wizard usage is measured in tokens, then converted into dollar-based usage based on the model and the token type, such as input, cached read, cache write, or output. Refer to [Key terms](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#key-terms) for what a token is, and the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for current rates. #### Related docs * [dbt Wizard billing FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) for common billing questions * [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md) for model options and token pricing * [BYOK for dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) for bring-your-own-key setup * [Billing](https://docs.getdbt.com/docs/platform/billing.md) for general dbt platform billing --- ### Unit tests Did you know... Available from dbt v1.8 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Historically, dbt's test coverage was confined to [“data” tests](https://docs.getdbt.com/docs/build/data-tests.md), assessing the quality of input data or resulting datasets' structure. However, these tests could only be executed *after* building a model. There is an additional type of test in dbt: unit tests. In software programming, unit tests validate small portions of your functional code, and they work much the same way here. Unit tests allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production. Unit tests enable test-driven development, benefiting developer efficiency and code reliability. #### Prerequisites * We currently only support unit testing SQL models. * We currently only support adding unit tests to models in your *current* project. * We currently *don't* support unit testing models that use the [`materialized view`](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) materialization. * We currently *don't* support unit testing models that use recursive SQL. * We currently *don't* support unit testing models that use introspective queries. * If your model has multiple versions, by default the unit test will run on *all* versions of your model. Read [unit testing versioned models](https://docs.getdbt.com/reference/resource-properties/unit-testing-versions.md) for more information. * Unit tests must be defined in a YML file in your [`models/` directory](https://docs.getdbt.com/reference/project-configs/model-paths.md). * Table names must be aliased in order to unit test `join` logic. * Include all [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) or [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) model references in the unit test configuration as `input`s to avoid "node not found" errors during compilation. Unit tests are discovered from `model-paths` (by default, the `models/` directory), so define them alongside your models in a `.yml` file under your `model-paths`. Don't define unit test YAML in the `tests/` directory, which is reserved for [data tests](https://docs.getdbt.com/docs/build/data-tests.md). ###### Adapter-specific caveats * You must specify all fields in a BigQuery `STRUCT` in a unit test. You cannot use only a subset of fields in a `STRUCT`. * Redshift customers need to be aware of a [limitation when building unit tests](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#unit-test-limitations) that requires a workaround. * Redshift sources need to be in the same database as the models. tip Check out our [Unit tests on-demand course](https://learn.getdbt.com/learn/course/unit-testing/welcome-to-unit-testing-5min/introduction-to-unit-testing) to learn how to add unit tests and more! Read the [reference doc](https://docs.getdbt.com/reference/resource-properties/unit-tests.md) for more details about formatting your unit tests. ##### When to add a unit test to your model You should unit test a model: * When your SQL contains complex logic: * Regex * Date math * Window functions * `case when` statements when there are many `when`s * Truncation * When you're writing custom logic to process input data, similar to creating a function. * We don't recommend conducting unit testing for functions like `min()` since these functions are tested extensively by the warehouse. If an unexpected issue arises, it's more likely a result of issues in the underlying data rather than the function itself. Therefore, fixture data in the unit test won't provide valuable information. * Logic for which you had bugs reported before. * Edge cases not yet seen in your actual data that you want to handle. * Prior to refactoring the transformation logic (especially if the refactor is significant). * Models with high "criticality" (public, contracted models or models directly upstream of an exposure). ##### When to run unit tests dbt Labs strongly recommends only running unit tests in development or CI environments. Since the inputs of the unit tests are static, there's no need to use additional compute cycles running them in production. Use them in development for a test-driven approach and CI to ensure changes don't break them. Use the [resource type](https://docs.getdbt.com/reference/global-configs/resource-type.md) flag `--exclude-resource-type` or the (Applies to dbt v1.11 and later) `DBT_ENGINE_EXCLUDE_RESOURCE_TYPES` environment variable to exclude unit tests from your production builds and save compute. To run only unit tests on demand, use the `test_type` selector — this works across all engines (dbt v1 and dbt v2): ```bash dbt test --select "test_type:unit" ``` (Applies to dbt v2.0 and later) #### Run unit tests locally Opt in with an environment variable Running an individual unit test on local compute is currently an experimental feature. Set `DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS=true` in the environment where dbt runs before you use `compute: local`. You can run unit tests locally when you're working through tricky SQL and you want to know right away whether your logic works. By default, each unit test sends a query to your data platform and waits for the result. This can slow down testing and use warehouse compute. Because unit tests use static fixtures instead of real data, they don’t need to necessarily run on your data platform. Use the [`compute: local` config](https://docs.getdbt.com/reference/resource-configs/compute.md) to run them locally with DuckDB for faster feedback *without* the warehouse compute cost. That gives you: * A test loop that keeps up with you. Change your SQL, rerun the test, and repeat without waiting on a warehouse queue. * Room to test as you go, so you catch broken logic while you're still writing it instead of finding it in CI (or worse, after it reaches production) * Queries that execute with the same functions and runtime semantics as the remote warehouse * Your data platform's compute left for building models. ##### Prerequisites * The `DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS` environment variable set to `true`. This config is experimental and you need to opt in. Without it, dbt fails with an invalid configuration error that names the variable. * Snowflake or BigQuery. Local execution isn't available for other data platforms, and it only applies to unit tests. * The direct upstream models of the model you're testing already exist in your data platform. dbt fetches their schemas to translate your SQL, so if they don't exist, the test fails with an error about fetching the upstream relation schema. * `static_analysis` isn't set to `off` on the test. Local execution needs static analysis to translate your SQL, so `compute: local` promotes [static analysis](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) to `strict` for that test. If you set `static_analysis: off`, the test can't run locally and fails with `ExecutorFailed (dbt1401)`. ##### What to know before you use it * dbt fetches upstream schemas the first time you run the test, then caches them. Later runs reuse the cache, so they don't re-fetch. * Local execution only works if dbt can compile your model's SQL and translate it to DuckDB. Complex SQL and functions specific to your data platform might have no DuckDB equivalent so Snowflake's `AI_CLASSIFY` and `haversine` are two examples. So for example, a model that calls `haversine` fails with `failed in db_runner: Internal: Catalog Error: Scalar Function with name haversine does not exist!`. * A translation failure is a test failure. `local` doesn't fall back to your data platform, so the test fails and dbt exits with a non-zero code. ##### How to configure First, opt in to the config in the environment where dbt runs: ```bash export DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS=true ``` Then you can configure it on a single unit test: models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address model: dim_customers config: compute: local ``` Or on every unit test in your project: dbt\_project.yml ```yaml unit_tests: my_project: +compute: local ``` [`compute`](https://docs.getdbt.com/reference/resource-configs/compute.md) accepts two values: | Value | What it does | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `remote` | Sends the test to your data platform to run, using warehouse compute like any other query. This is the default, so you only need to set it explicitly to opt a test out of a project-level `+compute: local`. | | `local` | Runs the test with DuckDB, wherever dbt itself is running. Nothing is sent to your data platform, so the test returns quickly and uses no warehouse compute. | You might also see `sidecar` in error messages but it means the same thing as `local`. #### Unit testing a model This example creates a new `dim_customers` model with a field `is_valid_email_address` that calculates whether or not the customer’s email is valid: ```sql with customers as ( select * from {{ ref('stg_customers') }} ), accepted_email_domains as ( select * from {{ ref('top_level_email_domains') }} ), check_valid_emails as ( select customers.customer_id, customers.first_name, customers.last_name, customers.email, coalesce (regexp_like( customers.email, '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$' ) = true and accepted_email_domains.tld is not null, false) as is_valid_email_address from customers left join accepted_email_domains on customers.email_top_level_domain = lower(accepted_email_domains.tld) ) select * from check_valid_emails ``` The logic posed in this example can be challenging to validate. You can add a unit test to this model to ensure the `is_valid_email_address` logic captures all known edge cases: emails without `.`, emails without `@`, and emails from invalid domains. ```yaml unit_tests: - name: test_is_valid_email_address description: "Check my is_valid_email_address logic captures all known edge cases - emails without ., emails without @, and emails from invalid domains." model: dim_customers given: - input: ref('stg_customers') format: dict rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: dict rows: - {tld: example.com} - {tld: gmail.com} expect: format: dict rows: - {email: cool@example.com, is_valid_email_address: true} - {email: cool@unknown.com, is_valid_email_address: false} - {email: badgmail.com, is_valid_email_address: false} - {email: missingdot@gmailcom, is_valid_email_address: false} ``` The previous example defines the mock data using the inline `dict` format, but you can also use `csv` or `sql` either inline or in a separate fixture file. Store your fixture files in a `fixtures` subdirectory in any of your [test paths](https://docs.getdbt.com/reference/project-configs/test-paths.md). For example, `tests/fixtures/my_unit_test_fixture.sql`. The following examples show how to define mock data and expected output using `csv` and `sql`. models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address__csv model: dim_customers given: - input: ref('stg_customers') format: dict rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: csv rows: | tld example.com gmail.com expect: format: csv fixture: valid_email_address_fixture_output ``` models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address__sql model: dim_customers given: - input: ref('stg_customers') format: dict rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: sql rows: | select 'example.com' as tld union all select 'gmail.com' as tld expect: format: sql fixture: valid_email_address_fixture_output ``` When using the `dict` or `csv` format, you only have to define the mock data for the columns relevant to you. This enables you to write succinct and *specific* unit tests. note The direct parents of the model that you’re unit testing (in this example, `stg_customers` and `top_level_email_domains`) need to exist in the warehouse before you can execute the unit test. Use the [`--empty`](https://docs.getdbt.com/reference/commands/build.md#the---empty-flag) flag to build an empty version of the models to save warehouse spend. ```bash dbt run --select "stg_customers top_level_email_domains" --empty ``` Alternatively, use `dbt build` to, in lineage order: * Run the unit tests on your model. * Materialize your model in the warehouse. * Run the data tests on your model. Now you’re ready to run this unit test. You have a couple of options for commands depending on how specific you want to be: * `dbt test --select dim_customers` runs *all* of the tests on `dim_customers`. * `dbt test --select "dim_customers,test_type:unit"` runs all of the *unit* tests on `dim_customers`. * `dbt test --select test_is_valid_email_address` runs the test named `test_is_valid_email_address`. ```shell dbt test --select test_is_valid_email_address 16:03:49 Running with dbt=1.8.0-a1 16:03:49 Registered adapter: postgres=1.8.0-a1 16:03:50 Found 6 models, 5 seeds, 4 data tests, 0 sources, 0 exposures, 0 metrics, 410 macros, 0 groups, 0 semantic models, 1 unit test 16:03:50 16:03:50 Concurrency: 5 threads (target='postgres') 16:03:50 16:03:50 1 of 1 START unit_test dim_customers::test_is_valid_email_address ................... [RUN] 16:03:51 1 of 1 FAIL 1 dim_customers::test_is_valid_email_address ............................ [FAIL 1 in 0.26s] 16:03:51 16:03:51 Finished running 1 unit_test in 0 hours 0 minutes and 0.67 seconds (0.67s). 16:03:51 16:03:51 Completed with 1 error and 0 warnings: 16:03:51 16:03:51 Failure in unit_test test_is_valid_email_address (models/marts/unit_tests.yml) 16:03:51 actual differs from expected: @@ ,email ,is_valid_email_address → ,cool@example.com,True→False ,cool@unknown.com,False ...,... ,... 16:03:51 16:03:51 compiled Code at models/marts/unit_tests.yml 16:03:51 16:03:51 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` The clever regex statement wasn’t as clever as initially thought, as the model incorrectly flagged `cool@example.com` as an invalid email address. Updating the regex logic to `'^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$'` (those pesky escape characters) and rerunning the unit test solves the problem: ```shell dbt test --select test_is_valid_email_address 16:09:11 Running with dbt=1.8.0-a1 16:09:12 Registered adapter: postgres=1.8.0-a1 16:09:12 Found 6 models, 5 seeds, 4 data tests, 0 sources, 0 exposures, 0 metrics, 410 macros, 0 groups, 0 semantic models, 1 unit test 16:09:12 16:09:13 Concurrency: 5 threads (target='postgres') 16:09:13 16:09:13 1 of 1 START unit_test dim_customers::test_is_valid_email_address ................... [RUN] 16:09:13 1 of 1 PASS dim_customers::test_is_valid_email_address .............................. [PASS in 0.26s] 16:09:13 16:09:13 Finished running 1 unit_test in 0 hours 0 minutes and 0.75 seconds (0.75s). 16:09:13 16:09:13 Completed successfully 16:09:13 16:09:13 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1 ``` Your model is now ready for production! Adding this unit test helped catch an issue with the SQL logic *before* you materialized `dim_customers` in your warehouse and will better ensure the reliability of this model in the future. #### Unit testing incremental models When configuring your unit test, you can override the output of macros, vars, or environment variables. This enables you to unit test your incremental models in "full refresh" and "incremental" modes. note Incremental models need to exist in the database before running unit tests or doing a `dbt build`. Use the [`--empty` flag](https://docs.getdbt.com/reference/commands/build.md#the---empty-flag) to build an empty version of the models to save warehouse spend. You can also optionally select only your incremental models using the [`--select` flag](https://docs.getdbt.com/reference/node-selection/syntax.md#shorthand). ```shell dbt run --select "config.materialized:incremental" --empty ``` After running the command, you can then perform a regular `dbt build` for that model and then run your unit test. When testing an incremental model, the expected output is the **result of the materialization** (what will be merged/inserted), not the resulting model itself (what the final table will look like after the merge/insert). For example, say you have an incremental model in your project: my\_incremental\_model.sql ```sql {{ config( materialized='incremental' ) }} select * from {{ ref('events') }} {% if is_incremental() %} where event_time > (select max(event_time) from {{ this }}) {% endif %} ``` You can define unit tests on `my_incremental_model` to ensure your incremental logic is working as expected: ```yaml unit_tests: - name: my_incremental_model_full_refresh_mode model: my_incremental_model overrides: macros: # unit test this model in "full refresh" mode is_incremental: false given: - input: ref('events') rows: - {event_id: 1, event_time: 2020-01-01} expect: rows: - {event_id: 1, event_time: 2020-01-01} - name: my_incremental_model_incremental_mode model: my_incremental_model overrides: macros: # unit test this model in "incremental" mode is_incremental: true given: - input: ref('events') rows: - {event_id: 1, event_time: 2020-01-01} - {event_id: 2, event_time: 2020-01-02} - {event_id: 3, event_time: 2020-01-03} - input: this # contents of current my_incremental_model rows: - {event_id: 1, event_time: 2020-01-01} expect: # what will be inserted/merged into my_incremental_model rows: - {event_id: 2, event_time: 2020-01-02} - {event_id: 3, event_time: 2020-01-03} ``` There is currently no way to unit test whether the dbt framework inserted/merged the records into your existing model correctly, but [we're investigating support for this in the future](https://github.com/dbt-labs/dbt/issues/8664). #### Unit testing a model that depends on ephemeral model(s) If you want to unit test a model that depends on an ephemeral model, you must use `format: sql` for that input. ```yaml unit_tests: - name: my_unit_test model: dim_customers given: - input: ref('ephemeral_model') format: sql rows: | select 1 as id, 'emily' as first_name expect: rows: - {id: 1, first_name: emily} ``` #### Unit test exit codes Unit test successes and failures are represented by two exit codes: * Pass (0) * Fail (1) Exit codes differ from data test success and failure outputs because they don't directly reflect failing data tests. Data tests are queries designed to check specific conditions in your data, and they return one row per failed test case (for example, the number of values with duplicates for the `unique` test). dbt reports the number of failing records as failures. Whereas, each unit test represents one 'test case', so results are always 0 (pass) or 1 (fail) regardless of how many records failed within that test case. Learn about [exit codes](https://docs.getdbt.com/reference/exit-codes.md) for more information. #### Additional resources * [Unit testing reference page](https://docs.getdbt.com/reference/resource-properties/unit-tests.md) * [Supported data formats for mock data](https://docs.getdbt.com/reference/resource-properties/data-formats.md) * [Unit testing versioned models](https://docs.getdbt.com/reference/resource-properties/unit-testing-versions.md) * [Unit test inputs](https://docs.getdbt.com/reference/resource-properties/unit-test-input.md) * [Unit test overrides](https://docs.getdbt.com/reference/resource-properties/unit-test-overrides.md) * [Platform-specific data types](https://docs.getdbt.com/reference/resource-properties/data-types.md) --- ### Upgrade to dbt v2 Local development After [installing the dbt VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md), use the **Get started** panel to upgrade your project to dbt v2 if you haven't already done so. note If you are already running dbt v2, you must be on version `2.0.0-beta.66` or higher to use the upgrade tool. The dbt extension provides two ways to upgrade your project to dbt v2 from the **Get started** panel: * [**Agentic migration**:](#agentic-migration) Runs the dbt v1-to-dbt v2 migration with or AI agents. Select this option in the **Check dbt v2 compatibility** step. Requires or AI agents. * [**Manual CLI onboarding**:](#manual-cli-onboarding) Walks you through the upgrade in your terminal. Use this if you prefer the CLI or don't use or AI agents. #### Agentic migration 1. Select **Start with an agent** from the **Check dbt v2 compatibility** step. 2. The extension installs the **Migrate dbt v1 to dbt v2** agent skill to your editor's skill folder, then opens your AI chat with a migration prompt. 3. Click **Install & open chat** to continue. The chat opens with the migration prompt already loaded, so you can run it directly from the editor. No CLI commands are required. #### Manual CLI onboarding The **Get started** panel has an upgrade assistant that guides you through the upgrade process in your terminal. [![The dbt extension Get started panel and upgrade assistant.](/img/docs/extension/vsce-manual-upgrade.png?v=2 "The dbt extension Get started panel and upgrade assistant.")](#)The dbt extension Get started panel and upgrade assistant. You can start manual onboarding in either of the following ways: ##### From the Get started panel * From the **Get started** panel, select **Start manually in CLI** in the **Check dbt v2 compatibility** step. * Follow the prompts in the upgrade assistant to complete the upgrade. * Run `dbtf compile` to verify your project is ready for dbt v2. [![The message received when you have completed upgrading your project to v2.](/img/docs/extension/v2-onboarding-complete.png?v=2 "The message received when you have completed upgrading your project to v2.")](#)The message received when you have completed upgrading your project to v2. Once the upgrade is completed, you're ready to dive into all the features that dbt v2 has to offer! Next, [sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to a dbt platform account if you want to use the extension's dbt platform capabilities. ##### From your terminal * From your terminal, run: ```shell dbt init --v2-upgrade ``` The upgrade tool guides you through a series of prompts:  Do you have an existing dbt platform account? If you answer `Y`, the tool provides instructions for downloading your dbt platform profile or authenticating. If you answer `N`, the flow continues to local project setup.  Ready to run dbt init? If no `profiles.yml` file is present, the tool guides you through creating one and connecting to your data warehouse.  Ready to run dbt debug? If a `profiles.yml` file exists, the tool validates that your project is configured correctly and can connect to your data warehouse.  Ready to run dbt parse? The tool parses your dbt project to check compatibility with dbt v2. * If parsing fails, the tool can run [dbt-autofix](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation) to help resolve errors. * If you skip dbt-autofix, you can run it later or manually fix errors. The upgrade tool can't continue until parsing errors are resolved.  Ready to run dbt compile with static analysis off? After parsing succeeds, the tool compiles your project without static analysis. This mimics dbt v1 behavior by rendering Jinja into SQL while temporarily disabling dbt v2's advanced SQL comprehension. AI Agents There are cases where dbt-autofix may not resolve all errors and requires manual intervention. For those cases, the dbt-autofix tool provides an [AI Agents.md](https://github.com/dbt-labs/dbt-autofix/blob/main/AGENTS.md) file to help AI agents continue the migration work after dbt-autofix has completed its part.  Ready to run dbt compile? The tool compiles your project with full dbt v2 static analysis. This checks that your SQL code is valid in the context of your warehouse's tables and columns. [![The message received when you have completed upgrading your project to v2.](/img/docs/extension/v2-onboarding-complete.png?v=2 "The message received when you have completed upgrading your project to v2.")](#)The message received when you have completed upgrading your project to v2. Once the upgrade is completed, you're ready to dive into all the features that dbt v2 has to offer! Next, [sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to a dbt platform account if you want to use the extension's dbt platform capabilities. #### Next steps Once you've upgraded your project to dbt v2, you can: * [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to a dbt platform account if you want to use the extension's dbt platform capabilities, such as the **Catalog** tab. * [Configure your local environment](https://docs.getdbt.com/docs/configure-dbt-extension.md) to mirror your dbt platform environment and [set environment variables](https://docs.getdbt.com/docs/configure-dbt-extension.md#configure-environment-variables) required by your project. * [Compare changes locally](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md) to preview data changes caused by your local edits. * [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md) for stronger local validation without slowing deployment jobs. * Review the [limitations and unsupported features](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations). --- ### Upgrade versions in dbt platform dbt platform In dbt, both [jobs](https://docs.getdbt.com/docs/deploy/jobs.md) and [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) are configured to use a specific version of dbt. The version can be upgraded at any time. #### Environments Navigate to the settings page of an environment, then click **Edit**. Click the **dbt version** dropdown bar and make your selection. You can select a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing updates (recommended), or a legacy version of dbt v1. Be sure to save your changes before navigating away. [![Example environment settings in dbt](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Example environment settings in dbt")](#)Example environment settings in dbt ##### Release tracks Release tracks determine the version of dbt and release cadence for your projects. dbt supports two major versions — v1 and v2 — and automatically delivers [minor version updates](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) for you. For more information, refer to [About release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). As a best practice, dbt Labs recommends that you test the upgrade in development first; use the [Override dbt version](#override-dbt-version) setting to test *your* project on the latest dbt version before upgrading your deployment environments and the default development environment for all your colleagues. ##### Override dbt version Configure your project to use a different dbt version than what's configured in your [development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments). This *override* only affects your user account, no one else's. Use this to safely test new dbt features before upgrading the dbt version for your projects. 1. Click your account name from the left side panel and select **Account settings**. 2. Choose **Credentials** from the sidebar and select a project. This opens a side panel. 3. In the side panel, click **Edit** and scroll to the **User development settings** section. 4. Choose a version from the **dbt version** dropdown and click **Save**. An example of overriding the configured version to [**v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for the selected project: [![Example of overriding the dbt version on your user account](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-override-version.png?v=2 "Example of overriding the dbt version on your user account")](#)Example of overriding the dbt version on your user account 5. (Optional) Verify that dbt will use your override setting to build the project by invoking a `dbt build` command in the Studio IDE's command bar. Expand the **System Logs** section and find the output's first line. It should begin with `Running with dbt=` and list the version dbt is using. <br /> <br /> For users on Release tracks, the output will display `Running dbt...` instead of a specific version, reflecting the flexibility and continuous automatic updates provided by the release track functionality. #### dbt v2 dbt Labs has introduced the new [dbt v2](https://docs.getdbt.com/docs/introduction.md), a ground-up rebuild of dbt. This is currently generally available for Snowflake, BigQuery, Redshift, and Databricks projects. It is in preview for other supported adapters on the dbt platform. Eligible customers can update environments to dbt v2 using the same workflows as v1.x, but remember: * If you don't see the **v2 Stable** release track as an option, you should check with your dbt Labs account team about eligibility. * To increase the compatibility of your project, update all jobs and environments to the **v1 Latest** release track and read more about the changes in our [upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). * Make sure you're using a [supported adapter](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md?version=2.0) and authentication method:  BigQuery * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks * Service Account / User Token * Native OAuth  Redshift * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  ClickHouse[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password *Adapter lifecycle can differ between the dbt platform and local development — an adapter can reach GA in the dbt platform before it reaches GA for local use.* <br /> * Once you upgrade your development environment(s) to `v2 Stable`, every user will have to restart the IDE. [![Upgrade to v2 in your environment settings.](/img/docs/dbt-platform/platform-configuring-dbt-platform/platform-upgrading-dbt-versions/upgrade-v2.png?v=2 "Upgrade to v2 in your environment settings.")](#)Upgrade to v2 in your environment settings. ##### Upgrading environments to dbt v2 When you're ready to upgrade your project(s) to dbt v2, there are some tools available to you in the dbt platform UI to help you get started. The dbt v2 upgrade assistant will step you through the process of preparing and upgrading your projects. [![The v2 upgrade assistant.](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/v2-upgrade-gui.png?v=2 "The v2 upgrade assistant.")](#)The v2 upgrade assistant. ###### Prerequisites To take advantage of the upgrade assistant and other upgrade tools, you'll need to meet the following prerequisites: * Your dbt project must be updated to use the **v1 Latest** release track. * You must have a `developer` license. * You must have the proper [permissions set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to execute individual upgrade tasks. Migrating to dbt v2 is a multi-step process and some of these steps may be repeated across projects by different users: | Upgrade task | Required permission(s) | Supported permission sets | | ----------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | Enable dbt v2 access (triggers migration flows) | dbt v2 readiness: `write`<br />Projects: `write` | Admin, Account Admin, and anyone assigned the v2 Migration Admin set, provided their base role also has `write` access to projects | | View dbt v2 readiness and job eligibility | dbt v2 readiness: `read` | Developer, Admin, Member, Account Admin, v2 Migration Admin | | **Run once on dbt v2** job action | Runs: `write` | Job Admin, Job Runner, Admin, Member, Project Creator, Account Admin | | Enable **v2 Stable** for an environment | Environments: `write` | Job Admin, Admin, Member, Project Creator, Account Admin | ###### Assign access to upgrade The dbt v2 readiness & upgrade flow are controlled by two account-level settings that an admin must configure. Step 1: Enable dbt v2 readiness features (required) The upgrade assistant and readiness panel only appear after enabling this setting. From your **Account settings**: 1. Navigate to the **Account** screen. 2. Click **Edit** and scroll to the **Settings** section. 3. Click the box next to **Enable dbt v2 readiness & upgrade features**. 4. Click **Save**. Once enabled, all admins and developers can see each project's v2 readiness status and which jobs are eligible or ineligible for dbt v2. Admins can also initiate the dbt v2 upgrade from development environments, environment settings, and job settings (subject to existing user permissions). Developer-licensed users will have access to debug tools to help make projects dbt v2 eligible in both development and production environments. Step 2: Restrict upgrade access (optional, Enterprise/Enterprise+ only) By default, all admins and developer-licensed users can access the dbt v2 readiness & upgrade flow. To restrict upgrade execution to users with the `v2 Migration Admin` permission set, enable this additional setting. From your **Account settings**: 1. Navigate to the **Account** screen. 2. Click **Edit** and scroll to the **Settings** section. 3. Click the box next to **Enable restricted dbt v2 upgrade permissions**. 4. Click **Save**. This hides the dbt v2 upgrade workflows from users who don't have the [`v2 Migration Admin`](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin) permission set. To grant access to the upgrade workflows to specific projects and/or specific users: 1. Navigate to an existing group in your **Account settings** and click **Edit**, or click [**Create group**](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#create-new-groups) to create a new one. 2. Scroll to the **Access and permissions** section and click **Add permission**. 3. Select the **v2 Migration Admin** permission set from the dropdown, then select the project(s) you want the users to access. 4. Click **Save**. The dbt v2 upgrade workflows helps identify areas of the project that need to be updated and provides tools for manually resolving and autofixing any errors. ###### Upgrade your development environment To begin the process of upgrading to dbt v2 with the assistant: 1. From the project homepage, click the **Start dbt v2 upgrade** or **Upgrade** button. You will be redirected to the Studio IDE. 2. At the top of the Studio IDE click **Check deprecation warnings**. [![Begin the process of parsing for deprecation warnings.](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/check-deprecations.png?v=2 "Begin the process of parsing for deprecation warnings.")](#)Begin the process of parsing for deprecation warnings. 3. dbt parses your project for the deprecations and presents a list of all deprecation warnings along with the option to **Autofix warnings**. Autofixing attempts to correct all syntax errors automatically. See [Fix deprecation warnings](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md) for more information. 4. Once the deprecation warnings have been resolved, click the **Upgrade to dbt v2** button. This upgrades your development environment to dbt v2! [![You're now ready to upgrade to v2 in your development environment!](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/autofix-success.png?v=2 "You're now ready to upgrade to v2 in your development environment!")](#)You're now ready to upgrade to v2 in your development environment! Now that you've upgraded your development environment to dbt v2, you're ready to start the process of upgrading your Production, Staging, and General environments. Follow your organization's standard procedures and use the [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to upgrade. Enable the dbt v2 readiness panel The dbt v2 readiness panel shows each project's eligibility status and blockers in the dbt platform. It's rolling out in phases — if it's not enabled for your account yet, an [account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) can turn it on in **Account settings** → **Account**. Refer to [Enable dbt v2 readiness features](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=2) for setup steps. If you have access to dbt Wizard, use the [dbt Wizard's dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) to help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed! ###### Upgrade considerations Keep in mind the following considerations during the upgrade process: * **Manifest compatibility** — dbt v2 produces a `v12` [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) that's compatible with dbt v1. The only differences are optional dbt v2-specific fields that only dbt v2 writes, which dbt v1 safely ignores. As a result, you can run dbt v2 and dbt v1 side by side. State-dependent features such as `state:modified`, `--defer`, and cross-environment `dbt docs generate` work across mixed dbt v2 and dbt v1 environments, so you can migrate to dbt v2 incrementally without breaking existing dbt v1 jobs. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). #### Jobs Each job in dbt can be configured to inherit parameters from the environment it belongs to. [![Settings of a dbt job](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/job-settings.png?v=2 "Settings of a dbt job")](#)Settings of a dbt job The example job seen in the screenshot above belongs to the environment "Prod". It inherits the dbt version of its environment as shown by the **Inherited from ENVIRONMENT\_NAME (DBT\_VERSION)** selection. You may also manually override the dbt version of a specific job to be any of the current dbt v1 releases supported by Cloud by selecting another option from the dropdown. #### Supported versions dbt Labs has always encouraged users to upgrade dbt v1 versions whenever a new minor version is released. We released our first major version of dbt - `dbt 1.0` - in December 2021. Alongside this release, we updated our policy on which versions of dbt v1 we will support in the dbt platform. > **Starting with v1.0, all subsequent minor versions are available in dbt. Versions are actively supported, with patches and bug fixes, for 1 year after their initial release. At the end of the 1-year window, we encourage all users to upgrade to a newer version for better ongoing maintenance and support.** We provide different support levels for different versions, which may include new features, bug fixes, or security patches: * **[Active](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: In the first few months after a minor version's initial release, we patch it with bugfix releases. These include fixes for regressions, new bugs, and older bugs / quality-of-life improvements. We implement these changes when we have high confidence that they're narrowly scoped and won't cause unintended side effects. * **[Critical](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: When a newer minor version ships, the previous one transitions to "Critical Support" for the remainder of its one-year window. Patches during this period are limited to critical security and installation fixes. After the one-year window ends, the version reaches end of life. * **[End of Life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions)**: Minor versions that have reached EOL no longer receive new patch releases. * **Deprecated**: dbt v1 versions that are no longer maintained by dbt Labs, nor supported in the dbt platform. We'll continue to update the following release table so that users know when we plan to stop supporting different versions of dbt v1 in dbt. ##### Latest releases | dbt v1 | Initial release | Support level and end date | | ------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------ | | [**v1.13**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.13.md) | (coming soon) | Beta | | [**v1.12**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.12.md) | Jul 16, 2026 | **Active support — July 15, 2027** | | [**v1.11**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.11.md) | Dec 19, 2025 | **Critical support — Dec 18, 2026** | | [**v1.10**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md) | Jun 16, 2025 | Deprecated ⛔️ | | [**v1.9**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md) | Dec 9, 2024 | Deprecated ⛔️ | | [**v1.8**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.8.md) | May 9, 2024 | Deprecated ⛔️ | | [**v1.7**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.7.md) | Nov 2, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.6**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.6.md>) | Jul 31, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.5**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) | Apr 27, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.4**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.4.md>) | Jan 25, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.3**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.3.md>) | Oct 12, 2022 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.2**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.2.md>) | Jul 26, 2022 | Deprecated ⛔️ | | [**v1.1**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.1.md>) | Apr 28, 2022 | Deprecated ⛔️ | | [**v1.0**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md>) | Dec 3, 2021 | Deprecated ⛔️ | | **v0.X** ⛔️ | (Various dates) | Deprecated ⛔️ | All functionality in v1 since the v1.7 release is available in [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), which provide automated upgrades at a cadence appropriate for your team. 1 Release tracks are required for the Developer and Starter plans on dbt. Accounts using older dbt versions will be migrated to the **v1 Latest** release track. For customers of dbt: dbt Labs strongly recommends migrating environments on older and unsupported versions to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or a supported version. On January 31, 2027, dbt Labs will deprecate v1 versions v1.3 through v1.7, removing them from availability in dbt platform. Starting with v1.0, dbt will ensure that you're always using the latest compatible patch release of `dbt-core` and plugins, including all the latest fixes. You may also choose to try prereleases of those patch releases before they are generally available. For more on version support and future releases, see [Understanding dbt v1 versions](https://docs.getdbt.com/docs/dbt-versions.md). ##### Need help upgrading? If you want more advice on how to upgrade your dbt projects, check out our [migration guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) and our [upgrading Q\&A page](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#upgrading-legacy-versions-under-10). ##### Testing your changes before upgrading Once you know what code changes you'll need to make, you can start implementing them. We recommend you: * Create a separate dbt project, "Upgrade project", to test your changes before making them live in your main dbt project. * In your "Upgrade project", connect to the same repository you use for your production project. * Set the development environment [settings](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to run the latest version of dbt v1. * Check out a branch `dbt-version-upgrade`, make the appropriate updates to your project, and verify your dbt project compiles and runs with the new version in the Studio IDE. * If upgrading directly to the latest version results in too many issues, try testing your project iteratively on successive minor versions. There are years of development and a few breaking changes between distant versions of dbt v1 (for example, 1.0 --> 1.10). The likelihood of experiencing problems upgrading between successive minor versions is much lower, which is why upgrading regularly is recommended. * Once you have your project compiling and running on the latest version of dbt in the development environment for your `dbt-version-upgrade` branch, try replicating one of your production jobs to run off your branch's code. * You can do this by creating a new deployment environment for testing, setting the custom branch to 'ON' and referencing your `dbt-version-upgrade` branch. You'll also need to set the dbt version in this environment to the latest dbt v1 version. [![Setting your testing environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/platform-upgrading-dbt-versions/upgrade-environment.png?v=2 "Setting your testing environment")](#)Setting your testing environment * Then add a job to the new testing environment that replicates one of the production jobs your team relies on. * If that job runs smoothly, you should be all set to merge your branch into main. * Then change your development and deployment environments in your main dbt project to run off the newest version of dbt v1. --- ### Upgrading to dbt utils v1.0 For the first time, [dbt utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) is crossing the major version boundary. From [last month’s blog post](https://www.getdbt.com/blog/announcing-dbt-v1.3-and-utils/): > It’s time to formalize what was already unofficial policy: you can rely on dbt utils in the same way as you do dbt v1, with stable interfaces and consistent and intuitive naming. Just like the switch to dbt 1.0 last year, there are some breaking changes as we standardized and prepared for the future. Most changes can be handled with find-and-replace. If you need help, post on the [Community Forum](https://discourse.getdbt.com) or in [#package-ecosystem](https://getdbt.slack.com/archives/CU4MRJ7QB) channel on Slack. #### New features * `get_single_value()` — An easy way to pull a single value from a SQL query, instead of having to access the `[0][0]`th element of a `run_query` result. * `safe_divide()` — Returns null when the denominator is 0, instead of throwing a divide-by-zero error. * New `not_empty_string` test — An easier wrapper than using `expression_is_true` to check the length of a column. #### Enhancements * Many tests are more meaningful when you run them against subgroups of a table. For example, you may need to validate that recent data exists for every turnstile instead of a single data source being sufficient. Add the new `group_by_columns` argument to your tests to do so. Review [this article](https://www.emilyriederer.com/post/grouping-data-quality/) by the test's author for more information. * With the addition of an on-by-default `quote_identifiers` argument in the `star()` macro, you can now disable quoting if necessary. * The `recency` test now has an optional `ignore_time_component` argument which can be used when testing against a date column. This prevents the time of day the test runs from causing false negatives/positives. #### Fixes * `union()` now includes/excludes columns case-insensitively * `slugify()` prefixes an underscore when the first char is a digit * The `expression_is_true` test doesn’t output `*` unless storing failures, a cost improvement for BigQuery. #### Breaking Changes ##### Changes to `surrogate_key()`: * `surrogate_key()` has been replaced by `generate_surrogate_key()`. The original treated null values and blank strings the same, which could lead to duplicate keys being created. `generate_surrogate_key()` does not have this flaw. Compare the [surrogate keys calculated for these columns](https://docs.google.com/spreadsheets/d/1qWfdbieUOSgkzdY0kmJ9iCgdqyWccA0R-6EW0EgaMQc/edit#gid=0): [![A table comparing the behavior of surrogate\_key and generate\_surrogate\_key](/img/guides/migration/versions/surrogate_key_behaviour.png?v=2 "A table comparing the behavior of surrogate_key and generate_surrogate_key")](#)A table comparing the behavior of surrogate\_key and generate\_surrogate\_key Changing the calculation method for surrogate keys, even for the better, could have significant consequences in downstream uses (such as snapshots and incremental models which use this column as their `unique_key`). As a result, it's possible to opt into the legacy behavior by setting the following variable in your dbt project: ```yaml #dbt_project.yml vars: surrogate_key_treat_nulls_as_empty_strings: true #turn on legacy behavior ``` By creating a new macro instead of updating the behavior of the old one, we are requiring all projects who use this macro to make an explicit decision about which approach is better for their context. **Our recommendation is that existing users should opt into the legacy behavior** unless you are confident that either: * your surrogate keys never contained nulls, or * your surrogate keys are not used for incremental models, snapshots or other stateful artifacts and so can be regenerated with new values without issue. Warning to package maintainers You can not assume one behavior or the other, as each project can customize its behavior. ##### Functionality now native to dbt v1: * The `expression_is_true` test no longer has a dedicated `condition` argument. Instead, use `where` which is [now available natively to all tests](https://docs.getdbt.com/reference/resource-configs/where.md): ```yaml models: - name: old_syntax tests: - dbt_utils.expression_is_true: expression: "col_a + col_b = total" #replace this... condition: "created_at > '2018-12-31'" - name: new_syntax tests: - dbt_utils.expression_is_true: expression: "col_a + col_b = total" # ...with this... where: "created_at > '2018-12-31'" ``` **Note** — This may cause some tests to get the same autogenerated names. To resolve this, you can [define a custom name for a test](https://docs.getdbt.com/reference/resource-properties/data-tests.md#define-a-custom-name-for-one-test). * The deprecated `unique_where` and `not_null_where` tests have been removed, because [where is now available natively to all tests](https://docs.getdbt.com/reference/resource-configs/where.md). To migrate, find and replace `dbt_utils.unique_where` with `unique` and `dbt_utils.not_null_where` with `not_null`. * `dbt_utils.current_timestamp()` is replaced by `dbt.current_timestamp()`. * Note that Postgres and Snowflake’s implementation of `dbt.current_timestamp()` differs from the old `dbt_utils` one ([full details here](https://github.com/dbt-labs/dbt-utils/pull/597#issuecomment-1231074577)). If you use Postgres or Snowflake and need identical backwards-compatible behavior, use `dbt.current_timestamp_backcompat()`. This discrepancy will hopefully be reconciled in a future version of dbt. * All other cross-db macros have moved to the dbt namespace, with no changes necessary other than replacing `dbt_utils.` with `dbt.`. Review the [cross database macros documentation](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md) for the full list. * In your code editor, you can do a global find and replace with regex: `\{\{\s*dbt_utils\.(any_value|bool_or|cast_bool_to_text|concat|dateadd|datediff|date_trunc|escape_single_quotes|except|hash|intersect|last_day|length|listagg|position|replace|right|safe_cast|split_part|string_literal|type_bigint|type_float|type_int|type_numeric|type_string|type_timestamp|type_bigint|type_float|type_int|type_numeric|type_string|type_timestamp|except|intersect|concat|hash|length|position|replace|right|split_part|escape_single_quotes|string_literal|any_value|bool_or|listagg|cast_bool_to_text|safe_cast|dateadd|datediff|date_trunc|last_day)` → `{{ dbt.$1` ##### Removal of `insert_by_period` materialization * The `insert_by_period` materialization has been moved to the [experimental-features repo](https://github.com/dbt-labs/dbt-labs-experimental-features/tree/main/insert_by_period). To continue to use it, add the below to your packages.yml file: ```yaml packages: - git: https://github.com/dbt-labs/dbt-labs-experimental-features subdirectory: insert_by_period revision: XXXX #optional but highly recommended. Provide a full git sha hash, e.g. 1c0bfacc49551b2e67d8579cf8ed459d68546e00. If not provided, uses the current HEAD. ``` ##### Removal of deprecated legacy behavior: * `safe_add()` only works with a list of arguments; use `{{ dbt_utils.safe_add(['column_1', 'column_2']) }}` instead of varargs `{{ dbt_utils.safe_add('column_1', 'column_2') }}`. * Several long-promised deprecations to `deduplicate()` have been applied: * The `group_by` argument is replaced by `partition_by`. * `relation_alias` is removed. If you need an alias, you can pass it directly to the `relation` argument. * `order_by` is now mandatory. Pass a static value like `1` if you don’t care how they are deduplicated. * The deprecated `table` argument has been removed from `unpivot()`. Use `relation` instead. #### Resolving error messages After upgrading, these are common error messages you may encounter, along with their resolutions. `dict object has no attribute MACRO_NAME` **Cause**: No macro called `MACRO_NAME` exists. This is most likely because the macro has moved to the `dbt` namespace (see above). It could also be because you haven't run dbt deps or have misspelled a macro's name. **Resolution**: For [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md), change `dbt_utils.MACRO_NAME()` to `dbt.MACRO_NAME()`. `macro 'dbt_macro__generate_surrogate_key' takes not more than 1 argument(s)` **Cause**: `generate_surrogate_key()` requires a single argument containing a list of columns, not a set of varargs. **Resolution**: Change to `dbt_utils.generate_surrogate_key(['column_1', 'column_2'])` - note the square brackets. `The dbt_utils.surrogate_key has been replaced by dbt_utils.generate_surrogate_key` **Cause**: `surrogate_key()` has been replaced. **Resolution**: 1. Decide whether you need to enable backwards compatibility [as detailed above](#changes-to-surrogate_key). 2. Find and replace `dbt_utils.surrogate_key` with `dbt_utils.generate_surrogate_key`. `macro dbt_macro__test_expression_is_true takes no keyword argument condition` **Cause**: `condition` has been removed from the `expression_is_true` test, now that `where` is available on all tests automatically. **Resolution**: Replace `condition` with `where`. `No materialization insert_by_period was found for adapter` **Cause**: `insert_by_period` has moved to the experimental features repo (see above). **Resolution**: Install the package as [described above](#removal-of-insert_by_period-materialization). `dbt found two tests with the name "XXX".` **Cause**: Changing from `condition` to `where` in the `expression_is_true` test, as configs are not part of a test's unique name. **Resolution**: Define a [custom name for your test](https://docs.getdbt.com/reference/resource-properties/tests#define-a-custom-name-for-one-test). --- ### Upgrading to v1.0 Available in v1 ##### Resources * [Discourse](https://discourse.getdbt.com/t/3180) * [Changelog](https://github.com/dbt-labs/dbt/blob/1.0.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt v1 major version 1.0 includes a number of breaking changes! Wherever possible, we have offered backwards compatibility for old behavior, and (where necessary) made migration simple. ##### Renamed fields in `dbt_project.yml` **These affect everyone:** * [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) have replaced `source-paths` in `dbt-project.yml`. * [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) have replaced `data-paths` in `dbt-project.yml` with a default value of `seeds`. * The [packages-install-path](https://docs.getdbt.com/reference/project-configs/packages-install-path.md) was updated from `modules-path`. Additionally the default value is now `dbt_packages` instead of `dbt_modules`. You may need to update this value in [`clean-targets`](https://docs.getdbt.com/reference/project-configs/clean-targets.md). * Default for `quote_columns` is now `True` for all adapters other than Snowflake. **These probably don't:** * The default value of [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) has been updated to be the plural `tests`. * The default value of [analysis-paths](https://docs.getdbt.com/reference/project-configs/analysis-paths.md) has been updated to be the plural `analyses`. ##### Tests The two **test types** are now "singular" and "generic" (instead of "data" and "schema", respectively). The `test_type:` selection method accepts `test_type:singular` and `test_type:generic`. (It will also accept `test_type:schema` and `test_type:data` for backwards compatibility.) **Not backwards compatible:** The `--data` and `--schema` flags to dbt test are no longer supported, and tests no longer have the tags `'data'` and `'schema'` automatically applied. Updated docs: [data tests](https://docs.getdbt.com/docs/build/data-tests.md), [test selection](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md), [selection methods](https://docs.getdbt.com/reference/node-selection/methods.md). The `greedy` flag/property has been renamed to **`indirect_selection`**, which is now eager by default. **Note:** This reverts test selection to its pre-v0.20 behavior by default. `dbt test -s my_model` *will* select multi-parent tests, such as `relationships`, that depend on unselected resources. To achieve the behavior change in v0.20 + v0.21, set `--indirect-selection=cautious` on the CLI or `indirect_selection: cautious` in YAML selectors. Updated docs: [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md), [yaml selectors](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md). ##### Global macros Global project macros have been reorganized, and some old unused macros have been removed: `column_list`, `column_list_for_create_table`, `incremental_upsert`. This is unlikely to affect your project. ##### Installation * [Installation docs](https://docs.getdbt.com/docs/supported-data-platforms.md) reflects adapter-specific installations * `python -m pip install dbt` is no longer supported, and will raise an explicit error. Install the specific adapter plugin you need as `python -m pip install dbt-<adapter>`. * `brew install dbt` is no longer supported. Install the specific adapter plugin you need (among Postgres, Redshift, Snowflake, or BigQuery) as `brew install dbt-<adapter>`. * Removed official support for python 3.6, which is reaching end of life on December 23, 2021 ##### For users of adapter plugins * **BigQuery:** Support for ingestion-time-partitioned tables has been officially deprecated in favor of modern approaches. Use `partition_by` and incremental modeling strategies instead. For more information, refer to [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md). ##### For maintainers of plugins + other integrations We've introduced a new [**structured event interface**](https://docs.getdbt.com/reference/events-logging.md), and we've transitioned all dbt logging to use this new system. **This includes a breaking change for adapter plugins**, requiring a very simple migration. For more details, see the [`events` module README](https://github.com/dbt-labs/dbt/blob/HEAD/core/dbt/events/README.md#adapter-maintainers). If you maintain a different kind of plugin that *needs* legacy logging, for the time being, you can re-enable it with an env var (`DBT_ENABLE_LEGACY_LOGGER=True`); be advised that we will remove this capability in a future version of dbt. The [**dbt RPC Server**](https://docs.getdbt.com/reference/commands/rpc.md) has been split out from `dbt-core` and is now packaged separately. Its functionality will be fully deprecated by the end of 2022, in favor of a new dbt Server. Instead of `dbt rpc`, use `dbt-rpc serve`. **Artifacts:** New schemas (manifest v4, run results v4, sources v3). Notable changes: add `metrics` nodes; schema test + data test nodes are renamed to generic test + singular test nodes; freshness threshold default values look slightly different. ##### Deprecations from long ago Several under-the-hood changes from past minor versions, tagged with deprecation warnings, have now been fully deprecated. * The `packages` argument of [dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) has been deprecated and will raise an exception when used. * The "adapter\_macro" macro has been deprecated. Instead, use the [dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) method to find a macro and call the result. * The `release` arg has been removed from the `execute_macro` method. #### New features and changed documentation * Add [metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md), a new node type * [Generic tests](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) can be defined in `tests/generic` (new), in addition to `macros/` (as before) * [Parsing](https://docs.getdbt.com/reference/parsing.md): partial parsing and static parsing have been turned on by default. * [Global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) have been standardized. Related updates to [global CLI flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) and [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md). * [The `init` command](https://docs.getdbt.com/reference/commands/init.md) has a whole new look and feel. It's no longer just for first-time users. * Add `result:<status>` subselectors for smarter reruns when dbt models have errors and tests fail. See examples: [Pro-tips for Workflows](https://docs.getdbt.com/best-practices/best-practice-workflows.md#pro-tips-for-workflows) * Secret-prefixed [env vars](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) are now allowed only in `profiles.yml` + `packages.yml` --- ### Upgrading to v1.1 Available in v1 ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.1.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading There are no breaking changes for code in dbt projects and packages. We are committed to providing backwards compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### For maintainers of adapter plugins We have reworked the testing suite for adapter plugin functionality. For details on the new testing suite, refer to the "Test your adapter" step in the [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) guide. The abstract methods `get_response` and `execute` now only return `connection.AdapterReponse` in type hints. Previously, they could return a string. We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter. This also gives you the opportunity to add fields specific to your adapter's query execution, such as `rows_affected` or `bytes_processed`. ##### For consumers of dbt artifacts (metadata) The manifest schema version will be updated to v5. The only change is to the default value of `config` for parsed nodes. For users of [state-based functionality](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection), such as the `state:modified` selector, recall that: > The `--state` artifacts must be of schema versions that are compatible with the currently running dbt version. If you have two jobs, whereby one job compares or defers to artifacts produced by the other, you'll need to upgrade both at the same time. If there's a mismatch, dbt will alert you with this error message: ```text Expected a schema version of "https://schemas.getdbt.com/dbt/manifest/v5.json" in <state-path>/manifest.json, but found "https://schemas.getdbt.com/dbt/manifest/v4.json". Are you running with a different version of dbt? ``` #### New and changed documentation [**Incremental models**](https://docs.getdbt.com/docs/build/incremental-models.md) can now accept a list of multiple columns as their `unique_key`, for models that need a combination of columns to uniquely identify each row. This is supported by the most common data warehouses, for incremental strategies that make use of the `unique_key` config (`merge` and `delete+insert`). [**Generic tests**](https://docs.getdbt.com/reference/resource-properties/data-tests.md) can define custom names. This is useful to "prettify" the synthetic name that dbt applies automatically. It's needed to disambiguate the case when the same generic test is defined multiple times with different configurations. [**Sources**](https://docs.getdbt.com/reference/source-properties.md) can define configuration inline with other `.yml` properties, just like other resource types. The only supported config is `enabled`; you can use this to dynamically enable/disable sources based on environment or package variables. ##### Advanced and experimental functionality **Fresh Rebuilds.** There's a new *experimental* selection method in town: [`source_status:fresher`](https://docs.getdbt.com/reference/node-selection/methods.md#source_status). Much like the `state:` and `result` methods, the goal is to use dbt metadata to run your DAG more efficiently. If dbt has access to previous and current results of `dbt source freshness` (the `sources.json` artifact), dbt can compare them to determine which sources have loaded new data, and select only resources downstream of "fresher" sources. Read more in [Understanding State](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection) and [CI/CD in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md). [**dbt-Jinja functions**](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) have a new landing page, and two new members: * [`print`](https://docs.getdbt.com/reference/dbt-jinja-functions/print.md) exposes the Python `print()` function. It can be used as an alternative to `log()`, and together with the `QUIET` config, for advanced macro-driven workflows. * [`selected_resources`](https://docs.getdbt.com/reference/dbt-jinja-functions/selected_resources.md) exposes, at runtime, the list of DAG nodes selected by the current task. [**Global configs**](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) include some new additions: * `QUIET` and `NO_PRINT`, to control which log messages dbt prints to terminal output. For use in advanced macro-driven workflows, such as [codegen](https://hub.getdbt.com/dbt-labs/codegen/latest/). * `CACHE_SELECTED_ONLY` is an *experimental* config that can significantly speed up dbt's start-of-run preparations, in cases where you're running only a few models from a large project that manages many schemas. ##### For users of specific adapters **dbt-bigquery** added Support for finer-grained configuration of query timeout and retry when defining your [connection profile](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md). **dbt-spark** added support for a [`session` connection method](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md#session), for use with a pySpark session, to support rapid iteration when developing advanced or experimental functionality. This connection method is not recommended for new users, and it is not supported in dbt. ##### Dependencies [Python compatibility](https://docs.getdbt.com/faqs/Core/install-python-compatibility.md): dbt v1 officially supports Python 3.10 --- ### Upgrading to v1.10 Available in v1 #### Resources * dbt [v1.10 changelog](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality that is included in dbt v1.10! If you have selected the **v1 Compatible** release track, you will have access in the next monthly **v1 Compatible** release after the dbt v1.10 final release. #### New and changed features and functionality New features and functionality available in dbt v1.10 ##### The `--sample` flag Large data sets can slow down dbt build times, making it harder for developers to test new code efficiently. The [`--sample` flag](https://docs.getdbt.com/docs/build/sample-flag.md), available for the `run` and `build` commands, helps reduce build times and warehouse costs by running dbt in sample mode. It generates filtered refs and sources using time-based sampling, allowing developers to validate outputs without building entire models. ##### Move standalone anchors under `anchors:` key As part of the ongoing process of making the dbt authoring language more precise, dbt v1.10 raises a warning when it sees an unexpected top-level key in a properties YAML file. A common use case behind these unexpected keys is standalone anchor definitions at the top level of a properties YAML file. You can use the new top-level `anchors:` key as a container for these reusable configuration blocks. For example, rather than using this configuration: models/\_models.yml ```yml id_column: &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models. - name: my_second_model columns: - *id_column_alias ``` Move the anchor under the `anchors:` key instead: models/\_models.yml ```yml anchors: - &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models - name: my_second_model columns: - *id_column_alias ``` This move is only necessary for fragments defined outside of the main YAML structure. For more information about this new key, see [anchors](https://docs.getdbt.com/reference/resource-properties/anchors.md). ##### Parsing `catalogs.yml` dbt v1 can now parse the `catalogs.yml` file. This is an important milestone in the journey to supporting external catalogs for Iceberg tables, as it enables write integrations. You'll be able to provide a config specifying a catalog integration for your producer model: For example: ```yml catalogs: - name: catalog_dave # materializing the data to an external location, and metadata to that data catalog write_integrations: - name: databricks_glue_write_integration external_volume: databricks_external_volume_prod table_format: iceberg catalog_type: unity ``` The implementation for the model would look like this: models/schemas.yml ```yaml models: - name: my_second_public_model config: catalog_name: catalog_dave ``` Check out our [docs on external catalog support](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) today! We'll have more information about this in the coming weeks, but this is an exciting step in journey to cross-platform support. ##### Integrating dbt v1 artifacts with dbt projects With [hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md), dbt v1 users working in the command line interface (CLI) can execute runs that seamlessly upload [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) into dbt. This enhances hybrid dbt v1/dbt deployments by: * Fostering collaboration between dbt + dbt v1 users by enabling them to visualize and perform [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) to models defined in dbt v1 projects. This feature unifies dbt + dbt v1 workflows for a more connected dbt experience. * Giving dbt and dbt v1 users insights into their models and assets in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). To view Catalog, you must have a [developer or read-only license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * (Coming soon) Enabling users working in the [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to build off of models already created by a central data team in dbt v1 rather than having to start from scratch. Hybrid projects are available as a private beta to [dbt Enterprise accounts](https://www.getdbt.com/pricing). Contact your account representative to register your interest in the beta. ##### Managing changes to legacy behaviors dbt v1.10 introduces new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Introduced, disabled by default) [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). If set to `true`, dbt raises a warning if the argument names or types in your macro YAML don't match the macro's Jinja definition or use unsupported type values. Note that projects using `--warn-error` may experience build failures at parse time if macro argument declarations in YAML are out of sync with the macro's Jinja signature. * (Introduced, disabled by default) [`require_all_warnings_handled_by_warn_error`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_all_warnings_handled_by_warn_error.md). If this flag is set to `true`, all warnings raised during a run will be routed through the `--warn-error` / `--warn-error-options` handler. This ensures consistent behavior when promoting warnings to errors or silencing them. When the flag is `false` (which is the current default), only some warnings are processed by the handler — others may bypass it. Turning it on for projects that use `--warn-error` (or `--warn-error-options='{"error":"all"}'`) may cause build failures on warnings that were previously ignored to fail so we recommend enabling it gradually, one a project at a time. ##### Deprecation warnings Starting in `v1.10`, you will receive deprecation warnings for dbt code that will become invalid in the future, including: * Custom inputs (for example, unrecognized resource properties, configurations, and top-level keys) * Duplicate YAML keys in the same file * Unexpected Jinja blocks (for example, `{% endmacro %}` tags without a corresponding `{% macro %}` tag) * Some `properties` are moving to `configs` * And more dbt will start raising these warnings in version `1.10`, but making these changes will not be a prerequisite for using it. We at dbt Labs understand that it will take existing users time to migrate their projects, and it is not our goal to disrupt anyone with this update. The goal is to enable you to work with more safety, feedback, and confidence going forward. What does this mean for you? 1. If your project (or dbt package) encounters a new deprecation warning in `v1.10`, plan to update your invalid code soon. Although it’s just a warning for now, in a future version, dbt will enforce stricter validation of the inputs in your project. Check out the [`dbt-autofix` tool](https://github.com/dbt-labs/dbt-autofix) to autofix many of these! 2. In the future, the [`meta` config](https://docs.getdbt.com/reference/resource-configs/meta.md) will be the only place to put custom user-defined attributes. Everything else will be strongly typed and strictly validated. If you have an extra attribute you want to include in your project, or a model config you want to access in a custom materialization, you must nest it under `meta` moving forward. 3. If you are using the [`—-warn-error` flag](https://docs.getdbt.com/reference/global-configs/warnings.md) (or `--warn-error-options '{"error": "all"}'`) to promote all warnings to errors, this will include new deprecation warnings coming to dbt v1. If you don’t want these to be promoted to errors, the `--warn-error-options` flag gives you more granular control over exactly which types of warnings are treated as errors. You can set `"warn": ["Deprecations"]` (new as of `v1.10`) to continue treating the deprecation warnings as warnings. 4. The `--models` / `--model` / `-m` flag was renamed to `--select` / `--s` way back in dbt v0.21 (Oct 2021). Silently skipping this flag means ignoring your command's selection criteria, which could mean building your entire DAG when you only meant to select a small subset. For this reason, the `--models` / `--model` / `-m` flag **will raise a warning** in dbt v1.10, and an error in dbt v2. Please update your job definitions accordingly. ###### Custom inputs Historically, dbt has allowed you to configure inputs largely unconstrained. A common example of this is setting custom YAML properties: ```yml models: - name: my_model description: A model in my project. dbt_is_awesome: true # a custom property ``` dbt detects the unrecognized custom property (`dbt_is_awesome`) and silently continues. Without a set of strictly defined inputs, it becomes challenging to validate your project's configuration. This creates unintended issues such as: * Silently ignoring misspelled properties and configurations (for example, `description:` instead of `description:`). * Unintended collisions with user code when dbt introduces a new “reserved” property or configuration. If you have an unrecognized custom property, you will receive a warning, and in a future version, dbt will cease to support custom properties. Moving forward, these should be nested under the [`meta` config](https://docs.getdbt.com/reference/resource-configs/meta.md), which will be the only place to put custom user-defined attributes: ```yml models: - name: my_model description: A model in my project. config: meta: dbt_is_awesome: true ``` ###### Custom keys not nested under meta Previously, when you could define any additional fields directly under `config`, it could lead to collisions between pre-existing user-defined configurations and official configurations of the dbt framework. In the future, the `meta` config will be the sole location for custom user-defined attributes. Everything else will be strongly typed and strictly validated. If you have an extra attribute you want to include in your project, or a model config you want to access in a custom materialization, you must nest it under `meta` moving forward: ```yaml models: - name: my_model config: meta: custom_config_key: value columns: - name: my_column config: meta: some_key: some_value ``` ###### Duplicate keys in the same yaml file If two identical keys exist in the same properties YAML file, you will get a warning, and in a future version, dbt will stop supporting duplicate keys. Previously, if identical keys existed in the same properties YAML file, dbt silently overwrite, using the last configuration listed in the file. profiles.yml ```yml my_profile: target: my_target outputs: ... my_profile: # dbt would use only this profile key target: my_other_target outputs: ... ``` Moving forward, you should delete unused keys or move them to a separate properties YAML file. ###### Unexpected Jinja blocks If you have an orphaned Jinja block, you will receive a warning, and in a future version, dbt will stop supporting unexpected Jinja blocks. Previously, these orphaned Jinja blocks were silently ignored. macros/my\_macro.sql ```sql {% endmacro %} # orphaned endmacro jinja block {% macro hello() %} hello! {% endmacro %} ``` Moving forward, you should delete these orphaned Jinja blocks. ###### Properties moving to configs Some historical properties are moving entirely to configs. This will include: `freshness`, `meta`, `tags`, `docs`, `group`, and `access` If you previously set one of the impacted properties, such as `freshness`: ```yaml sources: - name: ecom schema: raw description: E-commerce data for the Jaffle Shop freshness: warn_after: count: 24 period: hour ``` You should now set it under `config`: ```yaml sources: - name: ecom schema: raw description: E-commerce data for the Jaffle Shop config: freshness: warn_after: count: 24 period: hour ``` ###### Custom output path for source freshness The ability to override the default path for `sources.json` via the `--output` or `-o` flags has been deprecated. You can still set the path for all artifacts in the step with `--target-path`, but will receive a warning if trying to set the path for just source freshness. ###### Warn error options The `warn_error_option` options for `include` and `exclude` have been deprecated and replaced with `error` and `warn`, respectively. ```yaml ... flags: warn_error_options: error: # Previously called "include" warn: # Previously called "exclude" silence: # To silence or ignore warnings - NoNodesForSelectionCriteria ``` #### Adapter-specific features and functionalities Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Snowflake * You can use the `platform_detection_timeout_seconds` parameter to control how long the Snowflake connector waits when detecting the cloud platform where the connection is being made. For more information, see [Snowflake setup](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md#platform_detection_timeout_seconds). ##### BigQuery * `dbt-bigquery` cancels BigQuery jobs that exceed their configured timeout by sending a cancellation request. If the request succeeds, dbt stops the job. If the request fails, the BigQuery job may keep running in the background until it finishes or you cancel it manually. For more information, see [Timeout and retries](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#timeouts-and-retries). #### Quick hits * Use the [`--use-fast-test-edges`](https://docs.getdbt.com/reference/global-configs/fast-test-edges.md) flag with `dbt build` to reduce the number of test edges dbt adds to the execution graph. In large projects, this can significantly reduce run times and memory usage. * Provide the [`loaded_at_query`](https://docs.getdbt.com/reference/resource-configs/freshness.md) property for source freshness to specify custom SQL to generate the `maxLoadedAt` time stamp on the source (versus the [built-in query](https://github.com/dbt-labs/dbt-adapters/blob/6c41bedf27063eda64375845db6ce5f7535ef6aa/dbt/include/global_project/macros/adapters/freshness.sql#L4-L16), which uses the `loaded_at_field`). You cannot define `loaded_at_query` if the `loaded_at_field` config is also provided. * Provide validation for macro arguments using the [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md) flag, which is disabled by default. When enabled, this flag checks that documented macro argument names match those in the macro definition and validates their types against a supported format. Previously, dbt did not enforce standard argument types, treating the type field as documentation-only. If no arguments are documented, dbt infers them from the macro and includes them in the manifest.json file. Learn more about [supported types](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). * You can use the [`config.meta_get()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_get) and [`config.meta_require()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_require) functions to access custom configurations stored under `meta`. --- ### Upgrading to v1.11 Available in v1 #### Resources * [dbt v1.11 changelog](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality included in the latest dbt v1 version! If you have selected the **v1 Compatible** release track, you will have access in the next monthly **v1 Compatible** release after the dbt v1.11 final release. #### New and changed features and functionality New features and functionality available in dbt v1.11 ##### User-defined functions (UDFs) dbt v1.11 introduces support for user-defined functions (UDFs), which enable you to define and register custom functions in your warehouse. Like macros, UDFs promote code reuse, but they are objects in the warehouse so you can reuse the same logic in tools outside dbt. Key features include: * **Define UDFs as first-class dbt resources**: Create UDF files in a `functions/` directory with corresponding YAML configuration. * **Execution**: Create, update, and rename UDFs as part of DAG execution using `dbt build --select "resource_type:function"` * **DAG integration**: When executing `dbt build`, UDFs are built before models that reference them, ensuring proper dependency management. * **New `function()` macro**: Reference UDFs in your models using the `{{ function('function_name') }}` Jinja macro. * **Deferral**: When you run a dbt command with `--defer` and `--state`, `function()` calls resolve to the UDF in the state manifest, so you can run models that depend on UDFs without building those UDFs first. Read more about UDFs, including prerequisites and how to define and use them in the [UDF documentation](https://docs.getdbt.com/docs/build/udfs.md). ##### `DBT_ENGINE_` prefix for environment variables Engine configuration environment variables use the `DBT_ENGINE_` prefix. For example, `DBT_STATE` becomes `DBT_ENGINE_STATE`, `DBT_PROJECT_DIR` becomes `DBT_ENGINE_PROJECT_DIR`, and so on. Refer to [About flags (global configs)](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) for the full mapping. ##### Managing changes to legacy behaviors dbt v1.11 introduces new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Introduced, disabled by default) [`require_unique_project_resource_names`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_unique_project_resource_names.md). This flag is set to `false` by default. With this setting, if two unversioned resources in the same package share the same name, dbt continues to run and raises a [`DuplicateNameDistinctNodeTypesDeprecation`](https://docs.getdbt.com/reference/deprecations.md#duplicatenamedistinctnodetypesdeprecation) warning. When set to `true`, dbt raises a `DuplicateResourceNameError` error. * (Introduced, disabled by default) [`require_ref_searches_node_package_before_root`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_ref_searches_node_package_before_root.md). This flag is set to `false` by default. With this setting, when dbt resolves a `ref()` in a package model, it searches for the referenced model in the root project *first*, then in the package where the model is defined. When set to `true`, dbt searches the package where the model is defined *before* searching the root project. ##### Deprecation warnings enabled by default Deprecation warnings from JSON schema validation are now enabled by default when validating your YAML configuration files (such as `schema.yml` and `dbt_project.yml`) for projects running using the Snowflake, Databricks, BigQuery, and Redshift adapters. These warnings help you proactively identify and update deprecated configurations (such as misspelled config keys, deprecated properties, or incorrect data types). You'll see the following deprecation warnings by default: * [CustomKeyInConfigDeprecation](https://docs.getdbt.com/reference/deprecations.md#customkeyinconfigdeprecation) * [CustomKeyInObjectDeprecation](https://docs.getdbt.com/reference/deprecations.md#customkeyinobjectdeprecation) * [CustomTopLevelKeyDeprecation](https://docs.getdbt.com/reference/deprecations.md#customtoplevelkeydeprecation) * [MissingPlusPrefixDeprecation](https://docs.getdbt.com/reference/deprecations.md#missingplusprefixdeprecation) * [SourceOverrideDeprecation](https://docs.getdbt.com/reference/deprecations.md#sourceoverridedeprecation) Each deprecation type can be silenced using the [warn-error-options](https://docs.getdbt.com/reference/global-configs/warnings.md#configuration) project configuration. For example, to silence all of the above deprecations within `dbt_project.yml`: dbt\_project.yml ```yml flags: warn_error_options: silence: - CustomTopLevelKeyDeprecation - CustomKeyInConfigDeprecation - CustomKeyInObjectDeprecation - MissingPlusPrefixDeprecation - SourceOverrideDeprecation ``` Alternatively, the `--warn-error-options` flag can be used to silence specific deprecations from the command line: ```sh dbt parse --warn-error-options '{"silence": ["CustomTopLevelKeyDeprecation", "CustomKeyInConfigDeprecation", "CustomKeyInObjectDeprecation", "MissingPlusPrefixDeprecation", "SourceOverrideDeprecation"]}' ``` To silence *all* deprecation warnings within `dbt_project.yml`: dbt\_project.yml ```yml flags: warn_error_options: silence: - Deprecations ``` Similarly, all deprecation warnings can be silenced via the `--warn-error-options` command line flag: ```sh dbt parse --warn-error-options '{"silence": ["Deprecations"]}' ``` #### Adapter-specific features and functionalities ##### Snowflake * The Snowflake adapter supports basic table materialization on Iceberg tables registered in a Glue catalog through a [catalog-linked database](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database#label-catalog-linked-db-create). For more information, see [Glue Data Catalog](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md#external-catalogs). * The `cluster_by` configuration is supported in dynamic tables. For more information, see [Dynamic table clustering](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-table-clustering). * The `immutable_where` configuration is supported in dynamic tables. For more information, see [Snowflake configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#immutable-where). * You can set [`copy_grants: true`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#copy-grants-dynamic-tables) on a dynamic table to preserve existing object-level privileges when the table is recreated during a `--full-refresh`. When set to `false` (default), all previously granted permissions are dropped on recreation, requiring manual re-grants. * Set the [`refresh_warehouse`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#refresh-warehouse) parameter to choose which Snowflake warehouse runs a dynamic table's automatic refreshes. This is separate from `snowflake_warehouse`, which is used for DDL execution. For example, you might use a smaller warehouse for refreshes and a larger one for DDL. If `refresh_warehouse` is not set, `snowflake_warehouse` is used for both DDL execution and automatic refreshes. ##### BigQuery * To improve performance, dbt can issue a single batch query when calculating source freshness through metadata, instead of executing one query per source. To enable this feature, set [bigquery\_use\_batch\_source\_freshness](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_use_batch_source_freshness-flag) to `true`. ##### Redshift * The new `datasharing` profile credential enables `dbt-redshift` to use Redshift-native metadata commands (`SHOW` commands such as `SHOW TABLES` and `SHOW COLUMNS`) instead of PostgreSQL catalog tables such as `pg_*` and `information_schema`. This supports cross-database and cross-cluster access with [Redshift Datasharing](https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html). For configuration details, refer to [Redshift setup](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md#datasharing).[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * The `drop_without_cascade` profile credential emits `DROP TABLE/VIEW/MATERIALIZED VIEW` statements without `CASCADE`. Redshift resolves the `CASCADE` dependency graph on every `DROP`, which adds overhead on large clusters. If your project has no downstream dependents (for example, it uses only unbound views) you can set `drop_without_cascade: true` to skip that cost. When enabled and a dependent object exists, Redshift raises an error. For configuration details, refer to [Redshift setup](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md). ##### Spark * New profile configurations have been added to enhance [retry handling for PyHive connections](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#retry-handling-for-pyhive-connections): * `poll_interval`: Controls how frequently the adapter polls the Thrift server to check if an async query has completed. * `query_timeout`: Adds an overall timeout (in seconds) for query execution. If a query exceeds the set duration during polling, it raises a `DbtRuntimeError`. This helps prevent indefinitely hanging queries. * `query_retries`: Handles connection loss during query polling by automatically retrying. #### Quick hits You will find these quick hits in dbt v1.11: * The [`--sqlparse`](https://docs.getdbt.com/reference/global-configs/sqlparse.md) flag sets [`sqlparse`](https://sqlparse.readthedocs.io/en/latest/api.html#security-and-performance-considerations) `MAX_GROUPING_DEPTH` and `MAX_GROUPING_TOKENS` when dbt parses SQL during compilation. * The `dbt ls` command can now write out nested keys. This makes it easier to debug and troubleshoot your project. Example: `dbt ls --output json --output-keys config.materialized` * Manifest metadata now includes `run_started_at`, providing better tracking of when dbt runs were initiated. * When a model is disabled, unit tests for that model are automatically disabled as well. * You can use the new [`config.meta_get()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_get) and [`config.meta_require()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_require) functions to access custom configurations stored under `meta`. These functions have been backported to dbt v1.10. --- ### Upgrading to v1.12 Available in v1 #### Resources * [dbt v1.12 changelog](https://github.com/dbt-labs/dbt/blob/1.12.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality included in the latest dbt v1 version! If you have selected the **v1 Compatible** release track, you will have access to the next monthly **v1 Compatible** release after the dbt v1.12 final release. #### New and changed features and functionality ##### Opt-in v2 parser dbt v1.12 introduces the `--use-v2-parser` flag that delegates parsing to the new v2 Rust parser instead of the v1 Python parser. The Rust parser is significantly faster than the Python parser — especially on larger projects, where it can be 5–10× quicker. If you're looking to speed up your development workflow or cut down on job startup times. Using the Rust parser is a natural first step toward v2 compatibility, so you can catch and fix any project issues gradually rather than all at once. This is an opt-in flag that changes no behavior unless explicitly set, making it a low-risk way to test v2 parser compatibility from within dbt v1.12. note The Rust parser is beta. Its output manifest may differ from the Python parser's in edge cases, which can affect downstream behavior. Fall back by removing `--use-v2-parser`, and [report issues](https://github.com/dbt-labs/dbt/issues) to help us close the gap. For more information on how to enable the flag, related behaviors, and parser error types, refer to [Opt-in v2 parser](https://docs.getdbt.com/reference/global-configs/parsing.md#opt-in-v2-parser). ##### New Iceberg `catalogs` spec We reworked the [`catalogs.yml` spec](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) to make it simpler to use, and to bring it up to speed with latest developments in Iceberg support among ecosystem vendors. The new spec can also power [cross-platform dbt Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh.md). ##### Native private packages dbt v1 now supports [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) in `packages.yml` and `dependencies.yml`. You can install packages from private GitHub, GitLab, or Azure DevOps repos using the `private` key without configuring a token or full Git URL. dbt uses your system's SSH configuration for authentication. Use the [`provider` key](https://docs.getdbt.com/docs/build/packages.md#using-the-provider-key) to specify your Git provider and tell dbt which SSH URL format to construct. For example: ```yaml packages: - private: your-org/your-internal-repo provider: "github" # Supported values: "github", "gitlab", "ado" ``` ##### Extensions to UDFs * dbt v1.12 adds support for JavaScript user-defined functions (UDFs) on Snowflake and BigQuery. Define a JavaScript UDF by creating a `.js` file in your `functions/` directory and a corresponding YAML file with the function's arguments and return type. For more information, refer to [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md). * You can define multiple argument signatures for the same user-defined function (UDF) using the `overloads` property. This lets you call the same function name with different input types, without creating separate UDFs for each variant. This is supported for SQL UDFs in Snowflake and Postgres, and Python UDFs in Snowflake. Each overload references a separate file using `defined_in`, with optional `arguments` and `returns`. All overloads are grouped into one DAG node, so they're built and selected together. On retry, dbt skips overloads that succeeded and reruns only those that failed. For more information, refer to [Defining overloaded UDFs](https://docs.getdbt.com/docs/build/udfs.md#defining-udfs-in-dbt#defining-overloaded-UDFs) and [`overloads`](https://docs.getdbt.com/reference/resource-properties/overloads.md). * You can specify public third-party PyPI packages for your Python UDF with the optional `packages` config. The warehouse installs these packages when it creates the UDF, which lets your UDF use functionality from external Python libraries. For more information, refer to [Defining UDFs in dbt](https://docs.getdbt.com/docs/build/udfs.md#defining-udfs-in-dbt) and the [packages](https://docs.getdbt.com/reference/resource-configs/packages.md) config reference. * Python UDFs are now supported in Databricks, in addition to Snowflake and BigQuery. For Databricks, Python UDFs require [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/), and the `runtime_version` and `entry_point` configs are ignored. For more information, refer to [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md). ##### `latest_version_pointer` for versioned models For versioned models, you can configure dbt to automatically create a pointer view named after a model's base name (for example, `dim_customers`) once the latest version materializes successfully. This lets you query the current version without maintaining a view manually. Enable this feature in your project with the [`latest_version_pointer_enabled_by_default: true`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) flag in `dbt_project.yml`, or per model using the [`latest_version_pointer.enabled`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) config. You can customize the pointer name per model with `latest_version_pointer.alias`, or globally by overriding the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro. For more information, refer to [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md#pointing-to-the-latest-version). ##### `--sql` flag for `dbt run-operation` You can now use the `--sql` flag with `dbt run-operation` to execute ad hoc database statements directly against your warehouse, without defining a macro. This is useful for one-off operations like dropping or altering a table, applying grants, or running a data fix. The statement runs through dbt's full Jinja compilation pipeline, so you have access to `ref()`, `source()`, `var()`, `target`, and all other context variables. For more information, refer to [About dbt run-operation](https://docs.getdbt.com/reference/commands/run-operation.md). ##### `on_error` model config You can configure whether downstream models run when an upstream model fails using the [`on_error`](https://docs.getdbt.com/reference/resource-configs/on_error.md) config. Set `on_error: continue` on a model to allow its downstream models to still attempt to run even when it fails. By default (`skip_children`), dbt skips all downstream models on failure. Note that [`--fail-fast`](https://docs.getdbt.com/reference/global-configs/failing-fast.md) takes precedence — runs with `--fail-fast` stop at the first failure, even if a model is configured with `on_error: continue`. ##### Apache Ossie semantic layer support * dbt v1.12 supports the [Apache Ossie](https://github.com/apache/ossie) standard for defining semantic models and metrics. You can place Ossie-format `.json` files in an `osi/` directory at the root of your project, and dbt parses them into the manifest alongside any native dbt semantic models. To use a different directory, configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml`. Ossie versions `0.1.0` and `0.1.1` are supported; any other version raises a parse error. For more information, refer to [Ossie semantic layer documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md). * dbt writes an `osi_document.json` file to your `target/` directory alongside `semantic_manifest.json` at parse time. This artifact provides an Ossie representation of your project's Semantic Layer. For more information, refer to [Semantic manifest](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). This is separate from the new Semantic Layer YAML spec below, which changes how you define semantic models and metrics in native dbt YAML. ##### New Semantic Layer YAML spec dbt v1.12 adds support for the latest Semantic Layer YAML specification, which simplifies how you define metrics and dimensions by embedding semantic annotations directly alongside each model. Key changes in the new spec: * `semantic_model` is nested directly under each model instead of being a standalone top-level key. * Entities and dimensions are defined at the column level. * Measures are replaced with `type: simple` metrics defined within the model. * `type_params` is deprecated; its parameters are now top-level keys within each metric definition. For migration guidance and a comparison between the latest spec and the legacy spec, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). For the semantic model reference, refer to [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md). ##### `selector` method for named YAML selectors You can reference a named selector from `selectors.yml` inside `--select` or `--exclude` using the [`selector` method](https://docs.getdbt.com/reference/node-selection/methods.md#selector) (for example, `selector:my_selector`). This makes it easier to compose reusable YAML selectors with other [selection methods](https://docs.getdbt.com/reference/node-selection/methods.md), [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md), and [set operators](https://docs.getdbt.com/reference/node-selection/set-operators.md) on the command line without duplicating logic. When you use the legacy `--selector` flag together with `--select` or `--exclude`, dbt only uses `--selector` for node selection and ignores `--select` and `--exclude`. Starting in dbt v1.12, dbt also raises a warning when these flags are combined. If you want to combine a selector with `--select` or `--exclude`, use the new `selector:` method instead. ##### Support for `vars.yml` You can use the [`vars.yml`](https://docs.getdbt.com/docs/build/project-variables.md#defining-variables-in-varsyml) file, located at the project root, to define project variables. This keeps variable definitions in one place and helps simplify `dbt_project.yml`. Variables defined in `vars.yml` are parsed *before* `dbt_project.yml`, so you can reference them in `dbt_project.yml` using `{{ var('...') }}`. You can continue to define variables in `dbt_project.yml` as before, but you cannot define variables in both files. For details and precedence, refer to [Project variables](https://docs.getdbt.com/docs/build/project-variables.md). ##### Improved exception handling and error messages dbt v1.12 improves exception handling so error messages are clearer and stack traces are easier to interpret. Previously, some internal failures surfaced as Python errors (for example, `AttributeError`, `KeyError`, `IndexError`, `RuntimeError`), which could be difficult to understand. In dbt v1.12, these are replaced with dbt errors (such as `CompilationError` and `ParsingError`) that include a clear error message. When you need the full Python error output for debugging, use `--debug` or check the logs. Key improvements: * **Cleaner default output**: Built-in Python exceptions (`Exception`, `ValueError`, `RuntimeError`) are replaced with dbt errors, so dbt no longer treats them as internal errors or displays unnecessary stack traces. * **Parsing and config validation**: Invalid field values raise a `ParsingError` instead of a raw `InvalidFieldValue` exception when applying `dbt_project.yml` configs to resources. In a generic data test, a `config` value that is a string or a number instead of a set of key-value pairs raises a `TestConfigNotDictError`. * **Snapshot validation**: When snapshot validation fails, dbt shows the relevant error message and omits the long Python error output. * **`dbt run-operation`**: When a `run-operation` call fails, the exception message is included in `run_results.json`, which makes failures easier to inspect. * **Cycle detection**: Dependency graph cycles raise a `CompilationError` instead of the built-in `RuntimeError`. * **Semantic model dependencies**: When a semantic model references a disabled or missing model, dbt raises a `CompilationError` instead of an `IndexError`.  More scenarios with exception handling improvements * A string concatenation in a `doc()` argument (such as `doc('foo' ~ 'bar')`) is skipped during doc block resolution instead of crashing with an `AttributeError`. * A Jinja variable (such as `doc(my_variable)`) raises a `DocTargetNotFoundError`. * When a `meta` value in `schema.yml` references an undefined Jinja variable, dbt converts it to `None` instead of raising a `TypeError` during partial parse. * When `sources`, `tables`, `exposure` tags, or `packages` are set to `null`, dbt treats them as an empty list instead of raising a `TypeError`. * When a model with custom contract constraints is evaluated during `state:modified` selection, dbt returns `None` for unknown constraint types instead of raising a `KeyError`. ##### Managing changes to legacy behaviors dbt v1.12 introduces new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Mature, enabled by default) [`require_all_warnings_handled_by_warn_error`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_all_warnings_handled_by_warn_error.md). All warnings raised during a run are now routed through the `--warn-error` / `--warn-error-options` handler. This only affects projects using `warn_error: true` or `--warn-error`. * (Mature, enabled by default) [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md). dbt now executes custom microbatch strategies in batches. If you have a custom microbatch macro and set the flag to `false`, dbt issues a deprecation warning. * (Mature, enabled by default) [`require_nested_cumulative_type_params`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md). dbt now raises an error (instead of a warning) if cumulative metrics use the un-nested syntax. Cumulative-type metrics must be nested under the `cumulative_type_params` field, as required since dbt v1.9. * (Mature, enabled by default) [`require_yaml_configuration_for_mf_time_spines`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_yaml_configuration_for_mf_time_spines.md). dbt suppresses the `MFTimespineWithoutYamlConfigurationDeprecation` warning. The legacy SQL file configuration continues to work. This flag has no functional impact. * (Mature, enabled by default) [`skip_nodes_if_on_run_start_fails`](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md). dbt skips all selected resources if there is a failure on an `on-run-start` hook. * (Mature, enabled by default) [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md). dbt uses unrendered values in `state:modified` comparisons, reducing false positives when configs differ by target environment (such as `prod` vs. `dev`). * (Mature, enabled by default) [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). dbt validates macro arguments by default. * (Introduced, disabled by default) [`require_valid_schema_from_generate_schema_name`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_valid_schema_from_generate_schema_name.md). This flag is set to `false` by default. With this setting, dbt raises the [`GenerateSchemaNameNullValueDeprecation`](https://docs.getdbt.com/reference/deprecations.md#generateschemanamenullvaluedeprecation) warning when a custom `generate_schema_name` macro returns a `null` value. When set to `true`, dbt enforces stricter validation and raises a parsing error instead of a warning. * (Introduced, disabled by default) [`require_sql_header_in_test_configs`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_sql_header_in_test_configs.md). When set to `true`, you can set [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) in the `config` of a generic data test at the model or column level in your `properties.yml` file. For more information, refer to [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md). * (Introduced, disabled by default) [`require_corrected_analysis_fqns`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md). When set to `true`, dbt applies project-level analysis configuration from `dbt_project.yml`. Previously, dbt silently ignored this configuration. This flag also corrects fully qualified names (FQNs) of analyses by removing the extra path segment, making them consistent with other resource types (for example, `your_project.my_analysis` instead of `your_project.analyses.my_analysis`). For more information, refer to [Analyses](https://docs.getdbt.com/docs/build/analyses.md). * (Introduced, disabled by default) [`require_source_and_semantic_model_names_without_spaces`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_source_and_semantic_model_names_without_spaces.md). By default, dbt raises a [`ResourceNamesWithSpacesDeprecation`](https://docs.getdbt.com/reference/deprecations.md#resourcenameswithspacesdeprecation) warning if it detects a space in a source name or semantic model name. When the flag is set to `true`, dbt raises an error. * (Introduced, disabled by default) [`allow_jinja_file_extensions`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md). When set to `True`, dbt recognizes Jinja-style extension suffixes (`.j2`, `.jinja`, `.jinja2`) on `.sql` and `.md` files. This enables Jinja-aware syntax highlighting in IDEs that associate these suffixes with Jinja templating. * (Introduced, disabled by default) [`latest_version_pointer_enabled_by_default`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md). When set to `true`, dbt automatically creates a latest version pointer view for every versioned model in your project, without requiring per-model configuration. #### Adapter-specific features and functionalities ##### Snowflake * You can materialize a model as a Snowflake [interactive table](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#interactive-tables) by setting `materialized: interactive_table`. Interactive tables are optimized for low-latency queries and support both static and dynamic (auto-refreshing) forms, depending on whether you set [`target_lag`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#target-lag-interactive-tables). Unlike dynamic tables, [`cluster_by`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#cluster-by-interactive-tables) is required. [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * You can use the [`snowflake.quote_args`](https://docs.getdbt.com/reference/resource-configs/quote_args.md) config on JavaScript UDFs to control whether argument names are quoted in the generated `CREATE FUNCTION` statement. When `true` (default), Snowflake quotes argument names, preserving their exact casing, so you reference arguments using the same case as defined in the YAML inside the function body. When `false`, argument names are unquoted and Snowflake uppercases them, so you must reference them in uppercase inside the function body. * You can set the [`iceberg_version`](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) config on Snowflake Iceberg tables to control which Iceberg format version Snowflake uses. Set it to `3` to use Iceberg V3, which improves `VARIANT` type support and makes row-level changes more efficient by tracking deletions separately instead of rewriting data. The default value is `2`. Note that you cannot change the value of `iceberg_version` after table creation. * You can configure the [`scheduler`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#scheduler) parameter on Snowflake dynamic tables to control how refreshes are managed. Setting it to `ENABLE` lets Snowflake automatically refresh the dynamic table, while `DISABLE` means dbt manages refreshes during model execution. When `scheduler` is set to `ENABLE`, you must also specify [`target_lag`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#target-lag). By default, dbt sets `scheduler` to `DISABLE` (unlike Snowflake’s native default of `ENABLE`), so dbt controls the refresh schedule unless you explicitly opt in to Snowflake’s scheduler. If you specify `target_lag` without setting `scheduler`, dbt automatically sets `scheduler` to `ENABLE`. * You can use the [`snowflake_initialization_warehouse`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#initialization-warehouse) parameter to specify a warehouse for the initial build and reinitialization of a dynamic table, separate from `snowflake_warehouse` which is used for regular incremental refreshes. * You can create Snowflake dynamic tables as transient (no [Fail-safe period](https://docs.snowflake.com/en/user-guide/data-failsafe)) by setting the [`transient`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#transient-dynamic-tables) config on models. When `transient` is not set on a model, the [`snowflake_default_transient_dynamic_tables`](https://docs.getdbt.com/reference/global-configs/snowflake-changes.md#the-snowflake_default_transient_dynamic_tables-flag) flag controls the default. Set this flag to `true` to make all dynamic tables transient by default. ##### BigQuery * BigQuery now supports [parallel microbatch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md), in addition to Snowflake. * Added the [`bigquery_use_standard_sql_for_partitions`](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_use_standard_sql_for_partitions-flag) flag, which controls whether `get_partitions_metadata()` uses standard SQL (`INFORMATION_SCHEMA.PARTITIONS`) or legacy SQL (`$__PARTITIONS_SUMMARY__`). The flag defaulted to `false` when first introduced in this release, but has been flipped to `true` by default ahead of BigQuery's [legacy SQL deprecation on June 1, 2026](https://docs.cloud.google.com/bigquery/docs/release-notes#February_25_2026). To revert to legacy SQL, set the flag to `false` in `dbt_project.yml`. * Added the [`bigquery_reject_wildcard_metadata_source_freshness`](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_reject_wildcard_metadata_source_freshness-flag) flag. When you set this flag to `true`, dbt raises a `DbtRuntimeError` if you run metadata-based source freshness checks with wildcard table identifiers (for example, `events_*`), preventing incorrect freshness results. * You can configure BigQuery job link logging with `job_link_info_level_log`. By default, dbt logs job links at the debug level. To log job links at the info level, set `job_link_info_level_log: true` in your BigQuery profile. This makes job links visible in dbt logs for easier access to the BigQuery console. For more information, see [BigQuery setup](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#job_link_info_level_log). * You can set `job_execution_timeout_seconds` per model, snapshot, seed, or test, in addition to the profile-level configuration. The per-resource value takes precedence over the default value set in the profile level. For more information, refer to [BigQuery setup](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#job_execution_timeout_seconds). ##### Redshift * The [`redshift_skip_autocommit_transaction_statements`](https://docs.getdbt.com/reference/global-configs/redshift-changes.md#redshift_skip_autocommit_transaction_statements-flag) flag defaults to `false`, preserving legacy behavior of sending `BEGIN`/`COMMIT`/`ROLLBACK` statements even when autocommit is enabled. To skip unnecessary transaction statements and improve performance, set the flag to `true`. * Added support for the `query_group` session parameter, allowing dbt to tag queries for Redshift Workload Manager routing and query logging. When configured in a profile, dbt sets `query_group` when opening a connection and the value applies for the duration of that session. You can also configure `query_group` at the model level to temporarily override the default value for a specific model, and dbt reverts the value at the end of model materialization. For more information, see [Redshift configurations](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#session-configuration). ##### Databricks * You can use the [`row_filter`](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#setting-row-filters) config to apply a [Unity Catalog row filter](https://docs.databricks.com/aws/en/tables/row-and-column-filters) to a model, restricting which rows a query returns based on a SQL UDF. Row filters are supported on `table`, `incremental`, `materialized_view`, and `streaming_table` materializations. They are not supported on regular views or Hive Metastore relations. * [`databricks_tags`](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#databricks_tags) has two updates in v1.12: * You can now set a tag's value to `''` or `None` to apply a key-only tag with no value. * Configs now merge additively across hierarchy levels (for example, project-level and model-level) instead of lower-level configs completely replacing higher-level ones. When the same tag key is defined at multiple levels, the lower-level value takes precedence. #### Quick hits * dbt now surfaces occasional, non-blocking hints that suggest ways to optimize your project. Hints are enabled by default. Disable them with the [`hints_enabled` flag](https://docs.getdbt.com/reference/global-configs/about-global-configs.md). * dbt v1.12 now tolerates dbt v2-specific names in [`warn_error_options`](https://docs.getdbt.com/reference/global-configs/warnings.md) instead of raising an error. If your config includes a v2-specific name (for example, `StaticAnalysis`, `PackageParsingCompatibility`), dbt v1 ignores it and emits a note: `<name> is not being used because it's specific to the dbt Fusion engine.` This lets you share `warn_error_options` configs across dbt v1 and dbt v2. * Macros invoked with the [`dbt run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) command can now `ref()` models with `private` or `protected` [access](https://docs.getdbt.com/reference/resource-configs/access.md) without raising a `DbtReferenceError`. Because macros are not part of the group and access control system, dbt doesn't enforce group membership when a macro called by `run-operation` references a model. * `dbt seed` now supports the [`--empty`](https://docs.getdbt.com/reference/commands/seed.md#the---empty-flag) flag. Use it to create seed tables with the correct schema but without loading any data. * dbt v1 now automatically loads environment variables from a `.env` file in your current working directory. Shell environment variables take precedence over `.env` values. New projects created with `dbt init` include `.env` in the default `.gitignore`. For more information, refer to [About env\_var function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md#using-the-env-file). * `dbt compile` writes compiled SQL for [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) to `target/compiled/`, consistent with models, tests, analyses, and functions. Each snapshot gets its own output file, named from the snapshot identifier, so multiple snapshot blocks in the same source file do not share one compiled path. For more information, refer to [About dbt compile](https://docs.getdbt.com/reference/commands/compile.md). --- ### Upgrading to v1.13 Beta ### Upgrading to v1.13 [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Available in v1 #### Resources * [dbt v1.13 changelog](https://github.com/dbt-labs/dbt/blob/1.13.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality included in the latest dbt v1 version! If you have selected the **v1 Compatible** release track, you will have access to the next monthly **v1 Compatible** release after the dbt v1.13 final release. #### New and changed features and functionality ##### dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) makes dbt smarter about what to build — instead of rebuilding every node on every run, dbt reuses nodes by cloning from another location or skipping a rebuild when the logic and data haven't changed. dbt State is natively available in dbt v1.13. To get started, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). #### Adapter-specific features and functionalities **Coming soon** --- ### Upgrading to v1.2 Available in v1 ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.2.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading There are no breaking changes for code in dbt projects and packages. We are committed to providing backwards compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### For consumers of dbt artifacts (metadata) The manifest schema version has been updated to `v6`. The relevant changes are: * Change to `config` default, which includes a new `grants` property with default value `{}` * Addition of a `metrics` property, to any node which could reference metrics using the `metric()` function For users of [state-based selection](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection): This release also includes new logic declaring forwards compatibility for older manifest versions. While running dbt v1.2, it should be possible to use `state:modified --state ...` selection against a manifest produced by dbt v1.0 or v1.1. #### For maintainers of adapter plugins See GitHub discussion [dbt-labs/dbt#5468](https://github.com/dbt-labs/dbt/discussions/5468) for detailed information #### New and changed functionality * **[Grants](https://docs.getdbt.com/reference/resource-configs/grants.md)** are natively supported in `dbt-core` for the first time. That support extends to all standard materializations, and the most popular adapters. If you already use hooks to apply simple grants, we encourage you to use built-in `grants` to configure your models, seeds, and snapshots instead. This will enable you to [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) up your duplicated or boilerplate code. * **[Metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md)** now support an `expression` type (metrics-on-metrics), as well as a `metric()` function to use when referencing metrics from within models, macros, or `expression`-type metrics. For more information on how to use expression metrics, check out the [**`dbt_metrics` package**](https://github.com/dbt-labs/dbt_metrics) * **[dbt-Jinja functions](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md)** now include the [`itertools` Python module](https://docs.getdbt.com/reference/dbt-jinja-functions/modules.md#itertools), as well as the [set](https://docs.getdbt.com/reference/dbt-jinja-functions/set.md) and [zip](https://docs.getdbt.com/reference/dbt-jinja-functions/zip.md) functions. * **[Node selection](https://docs.getdbt.com/reference/node-selection/syntax.md)** includes a [file selection method](https://docs.getdbt.com/reference/node-selection/methods.md#file) (`-s model.sql`), and [yaml selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) inheritance. * **[Global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md)** now include CLI flag and environment variable settings for [`target-path`](https://docs.getdbt.com/reference/global-configs/json-artifacts.md) and [`log-path`](https://docs.getdbt.com/reference/global-configs/logs.md), which can be used to override the values set in `dbt_project.yml` ##### Specific adapters * [Postgres](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md) and [Redshift](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) profiles support a `retries` config, if dbt encounters an operational error or timeout when opening a connection. The default is 1 retry. --- ### Upgrading to v1.3 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.3.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading We are committed to providing backward compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). There are three changes in dbt v1.3 that may require action from some users: 1. If you have a `profiles.yml` file located in the root directory where you run dbt, dbt will start preferring that profiles file over the default location on your machine. [You can read more details here](https://docs.getdbt.com/docs/local/profiles.yml.md#advanced-customizing-a-profile-directory). 2. If you already have `.py` files defined in the `model-paths` of your dbt project, dbt will start trying to read them as Python models. You can use [the new `.dbtignore` file](https://docs.getdbt.com/reference/dbtignore.md) to tell dbt to ignore those files. 3. If you have custom code accessing the `raw_sql` property of models (with the [model](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md) or [graph](https://docs.getdbt.com/reference/dbt-jinja-functions/graph.md) objects), it has been renamed to `raw_code`. This is a change to the manifest contract, described in more detail below. ##### For users of dbt Metrics The names of metric properties have changed, with backward compatibility. Those changes are: * Renamed `type` to `calculation_method` * Renamed `sql` to `expression` * Renamed `expression` calculation method metrics to `derived` calculation method metrics We plan to keep backward compatibility for a full minor version. Defining metrics with the old names will raise an error in dbt v1.4. ##### For consumers of dbt artifacts (metadata) We have updated the manifest schema version to `v7`. This includes the changes to metrics described above and a few other changes related to the addition of Python models: * Renamed `raw_sql` to `raw_code` * Renamed `compiled_sql` to `compiled_code` * A new top-level node property, `language` (`'sql'` or `'python'`) For users of [state-based selection](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection): This release includes logic providing backward and forward compatibility for older manifest versions. While running dbt v1.3, it should be possible to use `state:modified --state ...` selection against a manifest produced by dbt v1.0 and higher. ##### For maintainers of adapter plugins GitHub discussion with details: [dbt-labs/dbt#6011](https://github.com/dbt-labs/dbt/discussions/6011) #### New and changed documentation * **[Python models](https://docs.getdbt.com/docs/build/python-models.md)** are natively supported in `dbt-core` for the first time, on data warehouses that support Python runtimes. * Updates made to **[Metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md)** reflect their new syntax for definition, as well as additional properties that are now available. * Plus, a few related updates to **[exposure properties](https://docs.getdbt.com/reference/exposure-properties.md)**: `config`, `label`, and `name` validation. * **[Custom `node_color`](https://docs.getdbt.com/reference/resource-configs/docs.md)** in `dbt-docs`. For the first time, you can control the colors displayed in dbt's DAG. Want bronze, silver, and gold layers? It's at your fingertips. * **[`Profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md#advanced-customizing-a-profile-directory)** search order now looks in the current working directory before `~/.dbt`. ##### Quick hits * **["Full refresh"](https://docs.getdbt.com/reference/resource-configs/full_refresh.md)** flag supports a short name, `-f`. * **[The "config" selection method](https://docs.getdbt.com/reference/node-selection/methods.md#config)** supports boolean and list config values, in addition to strings. * Two new dbt-Jinja context variables for accessing invocation metadata: [`invocation_args_dict`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#invocation_args_dict) and [`dbt_metadata_envs`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md#custom-metadata). --- ### Upgrading to v1.4 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.4.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) **Final release:** January 25, 2023 dbt v1.4 is a "behind-the-scenes" release. We've been hard at work rebuilding `dbt-core` internals on top of more-solid foundations, to enable an exciting year of new feature development. Check out the [v1.5 milestone](https://github.com/dbt-labs/dbt/milestone/82) in GitHub for a preview of what's planned for April. #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### For consumers of dbt artifacts (metadata) The manifest schema version has updated to `v8`. These changes are relevant for people who parse or analyze the contents of the `manifest.json` file, or who have custom code accessing the [`model`](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md) or [`graph`](https://docs.getdbt.com/reference/dbt-jinja-functions/graph.md) variables, for example, `{{ model.root_path }}`. Relevant changes: * The `root_path` attribute has been removed for non-seed nodes to reduce duplicative information. * Unused attributes have been removed from seed nodes (including `depends_on.nodes`), and from `macros` (including `tags`). * The `unique_id` of docs blocks now start with `doc` for consistency with other resource types. ##### For maintainers of adapter plugins > **TL;DR** Not much heavy lifting for this minor version. We anticipate more work for `1.5.0`. We plan to release betas early & often, and provide guidance on upgrading. The high-level changes are: * Add support for Python 3.11 * Rename/replace deprecated exception functions * Add support for Incremental Predicates (if applicable) * Make use of new adapter-zone tests For more detailed information and to ask any questions, please visit [dbt v1/discussions/6624](https://github.com/dbt-labs/dbt/discussions/6624). #### New and changed documentation * [**Events and structured logging**](https://docs.getdbt.com/reference/events-logging.md): dbt's event system got a makeover. Expect more consistency in the availability and structure of information, backed by type-safe event schemas. * [**Python support**](https://docs.getdbt.com/faqs/Core/install-python-compatibility.md): Python 3.11 was released in October 2022. It is officially supported in dbt-core v1.4, although full support depends also on the adapter plugin for your data platform. According to the Python maintainers, "Python 3.11 is between 10-60% faster than Python 3.10." We encourage you to try [`dbt parse`](https://docs.getdbt.com/reference/commands/parse.md) with dbt v1.4 + Python 3.11, and compare the timing with dbt v1.3 + Python 3.10. Let us know what you find! * [**Metrics**](https://docs.getdbt.com/docs/build/build-metrics-intro.md): `time_grain` is optional, to provide better ergonomics around metrics that aren't time-bound. * **dbt-Jinja context:** The [local\_md5](https://docs.getdbt.com/reference/dbt-jinja-functions/local_md5.md) context method will calculate an [MD5 hash](https://en.wikipedia.org/wiki/MD5) for use *within* dbt. (Not to be confused with SQL md5!) * [**Exposures**](https://docs.getdbt.com/docs/build/exposures.md) can now depend on `metrics`. * [**"Tarball" packages**](https://docs.getdbt.com/docs/build/packages.md#internally-hosted-tarball-URL): Some organizations have security requirements to pull resources only from internal services. To address the need to install packages from hosted environments (such as Artifactory or cloud storage buckets), it's possible to specify any accessible URL where a compressed dbt package can be downloaded. * [**Granular "warn error" configuration**](https://docs.getdbt.com/reference/global-configs/warnings.md): Thanks to a full cleanup and consolidation of warning and exception classes within `dbt-core`, it is now possible to define a more granular `--warn-error-options` configuration that specifies the exact warnings you do (or don't) want dbt to treat as errors. * [**Deferral**](https://docs.getdbt.com/reference/node-selection/defer.md#favor-state) supports an optional configuration, `--favor-state`. ##### Advanced configurations for incremental models * [**`incremental_predicates`** config](https://docs.getdbt.com/docs/build/incremental-strategy.md#about-incremental_predicates) is now supported on the most popular adapters, enabling greater flexibility when tuning performance in `merge` and `delete` statements against large datasets. * **BigQuery:** The `insert_overwrite` incremental strategy supports a new (old) mechanism, [`time_ingestion_partitioning`](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#partitioning-by-an-ingestion-date-or-timestamp) + [`copy_partitions`](#copying-ingestion-time-partitions), that can yield significant savings in cost + time for large datasets. ##### Updates to Python models * Python models are [configured to materialize](https://docs.getdbt.com/docs/build/python-models.md) as `table` by default. * Python models [running on Snowpark](https://docs.getdbt.com/docs/build/python-models.md) will use "anonymous" stored procedures by default, enabling a small speedup and a cleaner query history. --- ### Upgrading to v1.5 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. dbt v1.5 is a feature release, with two significant additions: 1. [**Model governance**](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) — access, contracts, versions — the first phase of [multi-project deployments](https://github.com/dbt-labs/dbt/discussions/6725) 2. A Python entry point for [**programmatic invocations**](https://docs.getdbt.com/reference/programmatic-invocations.md), at parity with the CLI #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.5.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Release schedule](https://github.com/dbt-labs/dbt/issues/6715) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### Behavior changes Why changes to previous behavior? This release includes significant new features, and rework to `dbt-core`'s CLI and initialization flow. As part of refactoring its internals from [`argparse`](https://docs.python.org/3/library/argparse.html) to [`click`](https://click.palletsprojects.com), we made a handful of changes to runtime configuration. The net result of these changes is more consistent and practical configuration options, and a more legible codebase. ***Wherever possible, we will provide backward compatibility and deprecation warnings for at least one minor version before actually removing the old functionality.*** In those cases, we still reserve the right to fully remove backwards compatibility for deprecated functionality in a future v1.x minor version of `dbt-core`. Setting `log-path` and `target-path` in `dbt_project.yml` has been deprecated for consistency with other invocation-specific runtime configs ([dbt-labs/dbt#6882](https://github.com/dbt-labs/dbt/issues/6882)). We recommend setting via env var or CLI flag instead. The `dbt list` command will now include `INFO` level logs by default. Previously, the `list` command (and *only* the `list` command) had `WARN`-level stdout logging, to support piping its results to [`jq`](https://jqlang.github.io/jq/manual/), a file, or another process. To achieve that goal, you can use either of the following parameters: * `dbt list --log-level warn` (recommended; equivalent to previous default) * `dbt list --quiet` (suppresses all logging less than ERROR level, except for "printed" messages and `list` output) The following env vars have been renamed, for consistency with the convention followed by all other parameters: * `DBT_DEFER_TO_STATE` → `DBT_DEFER` * `DBT_FAVOR_STATE_MODE` → `DBT_FAVOR_STATE` * `DBT_NO_PRINT` → `DBT_PRINT` * `DBT_ARTIFACT_STATE_PATH` → `DBT_STATE` As described in [dbt-labs/dbt#7169](https://github.com/dbt-labs/dbt/pull/7169), command-line parameters that could be silent before will no longer be silent. See [dbt-labs/dbt#7158](https://github.com/dbt-labs/dbt/issues/7158) and [dbt-labs/dbt#6800](https://github.com/dbt-labs/dbt/issues/6800) for more examples of the behavior we are fixing. An empty `tests:` key in a yaml file will now raise a validation error, instead of being silently skipped. You can resolve this by removing the empty `tests:` key, or by setting it to an empty list explicitly: ```yml # ❌ this will raise an error models: - name: my_model tests: config: ... # ✅ this is fine models: - name: my_model tests: [] # todo! add tests later config: ... ``` Some options that could previously be specified *after* a subcommand can now only be specified *before*. This includes the inverse of the option, `--write-json` and `--no-write-json`, for example. The list of affected options are: List of affected options ```bash --cache-selected-only | --no-cache-selected-only --debug, -d | --no-debug --deprecated-print | --deprecated-no-print --enable-legacy-logger | --no-enable-legacy-logger --fail-fast, -x | --no-fail-fast --log-cache-events | --no-log-cache-events --log-format --log-format-file --log-level --log-level-file --log-path --macro-debugging | --no-macro-debugging --partial-parse | --no-partial-parse --partial-parse-file-path --populate-cache | --no-populate-cache --print | --no-print --printer-width --quiet, -q | --no-quiet --record-timing-info, -r --send-anonymous-usage-stats | --no-send-anonymous-usage-stats --single-threaded | --no-single-threaded --static-parser | --no-static-parser --use-colors | --no-use-colors --use-colors-file | --no-use-colors-file --use-experimental-parser | --no-use-experimental-parser --version, -V, -v --version-check | --no-version-check --warn-error --warn-error-options --write-json | --no-write-json ``` Additionally, some options that could be previously specified *before* a subcommand can now only be specified *after*. Any option *not* in the above list must appear *after* the subcommand from v1.5 and later. For example, `--profiles-dir`. The built-in [collect\_freshness](https://github.com/dbt-labs/dbt/blob/1.5.latest/core/dbt/include/global_project/macros/adapters/freshness.sql) macro now returns the entire `response` object, instead of just the `table` result. If you're using a custom override for `collect_freshness`, make sure you're also returning the `response` object; otherwise, some of your dbt commands will never finish. For example: ```sql {{ return(load_result('collect_freshness')) }} ``` Finally: The [built-in `generate_alias_name` macro](https://github.com/dbt-labs/dbt/blob/1.5.latest/core/dbt/include/global_project/macros/get_custom_name/get_custom_alias.sql) now includes logic to handle versioned models. If your project has reimplemented the `generate_alias_name` macro with custom logic, and you want to start using [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md), you will need to update the logic in your macro. Note that, while this is **not** a prerequisite for upgrading to v1.5—only for using the new feature—we recommend that you do this during your upgrade, whether you're planning to use model versions tomorrow or far in the future. Likewise, if your project has reimplemented the `ref` macro with custom logic, you will need to update the logic in your macro as described [here](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md). ##### For consumers of dbt artifacts (metadata) The [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) schema version will be updated to `v9`. Specific changes: * Addition of `groups` as a top-level key * Addition of `access`, `constraints`, `version`, `latest_version` as a top-level node attributes for models * Addition of `constraints` as a column-level attribute * Addition of `group` and `contract` as node configs * To support model versions, the type of `refs` has changed from `List[List[str]]` to `List[RefArgs]`, with nested keys `name: str`, `package: Optional[str] = None`, and `version: Union[str, float, NoneType] = None)`. ##### For maintainers of adapter plugins For more detailed information and to ask questions, please read and comment on the GH discussion: [dbt-labs/dbt#7213](https://github.com/dbt-labs/dbt/discussions/7213). #### New and changed documentation ##### Model governance The first phase of supporting dbt deployments at scale, across multiple projects with clearly defined ownership and interface boundaries. [Read about model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md), all of which is new in v1.5. ##### Revamped CLI Compile and preview dbt models and `--inline` dbt-SQL queries on the CLI using: * [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) * [`dbt show`](https://docs.getdbt.com/reference/commands/show.md) (new!) [Node selection methods](https://docs.getdbt.com/reference/node-selection/methods.md) can use Unix-style wildcards to glob nodes matching a pattern: ```text dbt ls --select "tag:team_*" ``` And (!): a first-ever entry point for [programmatic invocations](https://docs.getdbt.com/reference/programmatic-invocations.md), at parity with CLI commands. Run `dbt --help` to see new & improved help documentation :) ##### Quick hits * The [`version: 2` top-level key](https://docs.getdbt.com/reference/project-configs/version.md) is now **optional** in all YAML files. Also, the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and `version:` top-level keys are now optional in `dbt_project.yml` files. * [Events and logging](https://docs.getdbt.com/reference/events-logging.md): Added `node_relation` (`database`, `schema`, `identifier`) to the `node_info` dictionary, available on node-specific events * Support setting `--project-dir` via environment variable: [`DBT_PROJECT_DIR`](https://docs.getdbt.com/reference/dbt_project.yml.md) * More granular configurations for logging (to set [log format](https://docs.getdbt.com/reference/global-configs/logs.md#log-formatting), [log levels](https://docs.getdbt.com/reference/global-configs/logs.md#log-level), and [colorization](https://docs.getdbt.com/reference/global-configs/logs.md#color)) and [cache population](https://docs.getdbt.com/reference/global-configs/cache.md#cache-population) * [dbt overwrites the `manifest.json` file](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md#overwrites-the-manifestjson) during parsing, which means when you reference `--state` from the `target/ directory`, you may encounter a warning indicating that the saved manifest wasn't found. --- ### Upgrading to v1.6 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. dbt v1.6 has three significant areas of focus: 1. Next milestone of [multi-project deployments](https://github.com/dbt-labs/dbt/discussions/6725): improvements to contracts, groups/access, versions; and building blocks for cross-project `ref` 2. Semantic layer re-launch: dbt v1 and [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) integration 3. Mechanisms to support mature deployment at scale (`dbt clone` and `dbt retry`) #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.6.latest/CHANGELOG.md) * [dbt v1 installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Release schedule](https://github.com/dbt-labs/dbt/issues/7481) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### Behavior changes Action required if your project defines `metrics` The [spec for metrics](https://github.com/dbt-labs/dbt/discussions/7456) has changed and now uses [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md). If your dbt project defines metrics, you must migrate to dbt v1.6 because the YAML spec has moved from dbt\_metrics to MetricFlow. Any tests you have won't compile on v1.5 or older. * dbt v1.6 does not support Python 3.7, which reached End Of Life on June 23. Support Python versions are 3.8, 3.9, 3.10, and 3.11. * As part of the [dbt Semantic layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) re-launch, the spec for `metrics` has changed significantly. * The manifest schema version is now v10. * dbt Labs is ending support for Homebrew installation of dbt v1 and adapters. See [the discussion](https://github.com/dbt-labs/dbt/discussions/8277) for more details. ##### For consumers of dbt artifacts (metadata) The [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) schema version has been updated to `v10`. Specific changes: * Addition of `semantic_models` and changes to `metrics` attributes * Addition of `deprecation_date` as a model property * Addition of `on_configuration_change` as default node configuration (to support materialized views) * Small type changes to `contracts` and `constraints` * Manifest `metadata` includes `project_name` ##### For maintainers of adapter plugins For more detailed information and to ask questions, please read and comment on the GH discussion: [dbt-labs/dbt-core#7958](https://github.com/dbt-labs/dbt/discussions/7958). #### New and changed documentation ##### MetricFlow * [**Build your metrics**](https://docs.getdbt.com/docs/build/build-metrics-intro.md) with MetricFlow, a key component of the Semantic Layer. You can define your metrics and build semantic models with MetricFlow, available on the command line (CLI) for dbt v1.6 beta or higher. ##### Materialized views Supported on: * [Postgres](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#materialized-view) * [Redshift](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#materialized-view) * [Snowflake](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) * [Databricks](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#materialized-views-and-streaming-tables) ##### New commands for mature deployment [`dbt retry`](https://docs.getdbt.com/reference/commands/retry.md) executes the previously run command from the point of failure. Rebuild just the nodes that errored or skipped in a previous run/build/test, rather than starting over from scratch. [`dbt clone`](https://docs.getdbt.com/reference/commands/clone.md) leverages each data platform's functionality for creating lightweight copies of dbt models from one environment into another. Useful when quickly spinning up a new development environment, or promoting specific models from a staging environment into production. ##### Multi-project collaboration [**Deprecation date**](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md): Models can declare a deprecation date that will warn model producers and downstream consumers. This enables clear migration windows for versioned models, and provides a mechanism to facilitate removal of immature or little-used models, helping to avoid project bloat. [Model names](https://docs.getdbt.com/faqs/Project/unique-resource-names.md) can be duplicated across different namespaces (projects/packages), so long as they are unique within each project/package. We strongly encourage using [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) when referencing a model from a different package/project. More consistency and flexibility around packages. Resources defined in a package will respect variable and global macro definitions within the scope of that package. * `vars` defined in a package's `dbt_project.yml` are now available in the resolution order when compiling nodes in that package, though CLI `--vars` and the root project's `vars` will still take precedence. See ["Variable Precedence"](https://docs.getdbt.com/docs/build/project-variables.md#variable-precedence) for details. * `generate_x_name` macros (defining custom rules for database, schema, alias naming) follow the same pattern as other "global" macros for package-scoped overrides. See [macro dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) for an overview of the patterns that are possible. Closed Beta - dbt Enterprise [**Project dependencies**](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md): Introduces `dependencies.yml` and dependent `projects` as a feature of dbt Enterprise. Allows enforcing model access (public vs. protected/private) across project/package boundaries. Enables cross-project `ref` of public models, without requiring the installation of upstream source code. ##### Deprecated functionality The ability for installed packages to override built-in materializations without explicit opt-in from the user is being deprecated. * Overriding a built-in materialization from an installed package raises a deprecation warning. * Using a custom materialization from an installed package does not raise a deprecation warning. * Using a built-in materialization package override from the root project via a wrapping materialization is still supported. For example: ```text {% materialization view, default %} {{ return(my_cool_package.materialization_view_default()) }} {% endmaterialization %} ``` ##### Quick hits * [`state:unmodified` and `state:old`](https://docs.getdbt.com/reference/node-selection/methods.md#state) for [MECE](https://en.wikipedia.org/wiki/MECE_principle) stateful selection * [`invocation_args_dict`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#invocation_args_dict) includes full `invocation_command` as string * [`dbt debug --connection`](https://docs.getdbt.com/reference/commands/debug.md) to test just the data platform connection specified in a profile * [`dbt docs generate --empty-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md) to skip catalog population while generating docs * [`--defer-state`](https://docs.getdbt.com/reference/node-selection/defer.md) enables more-granular control * [`dbt ls`](https://docs.getdbt.com/reference/commands/list.md) adds the Semantic model selection method to allow for `dbt ls -s "semantic_model:*"` and the ability to execute `dbt ls --resource-type semantic_model`. * Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. --- ### Upgrading to v1.7 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.7.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Release schedule](https://github.com/dbt-labs/dbt/issues/8260) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Behavior changes dbt v1.7 expands the amount of sources you can configure freshness for. Previously, freshness was limited to sources with a `loaded_at_field`; now, freshness can be generated from warehouse metadata tables when available. As part of this change, the `loaded_at_field` is no longer required to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: * If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). * If a `loaded_at_field` is *not* provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). This is a relatively small behavior change, but worth calling out in case you notice that dbt is calculating freshness for *more* sources than before. To exclude a source from freshness calculations, explicitly set `freshness: null`. Beginning with v1.7, running [`dbt deps`](https://docs.getdbt.com/reference/commands/deps.md) creates or updates the `package-lock.yml` file in the *project\_root* where `packages.yml` is recorded. The `package-lock.yml` file contains a record of all packages installed and, if subsequent `dbt deps` runs contain no updated packages in `dependencies.yml` or `packages.yml`, dbt-core installs from `package-lock.yml`. To retain the behavior prior to v1.7, there are two main options: 1. Use `dbt deps --upgrade` everywhere `dbt deps` was used previously. 2. Add `package-lock.yml` to your `.gitignore` file. #### New and changed features and functionality * [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) now supports `--select` to generate [catalog metadata](https://docs.getdbt.com/reference/artifacts/catalog-json.md) for a subset of your project. * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) can now be generated from warehouse metadata tables. ##### MetricFlow enhancements * Automatically create metrics on measures with [`create_metric: true`](https://docs.getdbt.com/docs/build/semantic-models.md). * Optional [`label`](https://docs.getdbt.com/docs/build/semantic-models.md) in semantic\_models, measures, dimensions, and entities. * New configurations for semantic models - [enable/disable](https://docs.getdbt.com/reference/resource-configs/enabled.md), [group](https://docs.getdbt.com/reference/resource-configs/group.md), and [meta](https://docs.getdbt.com/reference/resource-configs/meta.md). * Support `fill_nulls_with` and `join_to_timespine` for metric nodes. * `saved_queries` extends governance beyond the semantic objects to their consumption. ##### For consumers of dbt artifacts (metadata) * The [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) schema version has been updated to v11. * The [run\_results](https://docs.getdbt.com/reference/artifacts/run-results-json.md) schema version has been updated to v5. * There are a few specific changes to the [catalog.json](https://docs.getdbt.com/reference/artifacts/catalog-json.md): * Added [node attributes](https://docs.getdbt.com/reference/artifacts/run-results-json.md) related to compilation (`compiled`, `compiled_code`, `relation_name`) to the `catalog.json`. * The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. ##### Model governance dbt v1.5 introduced model governance which we're continuing to refine. v1.7 includes these additional features and functionality: * **[Breaking change detection](https://docs.getdbt.com/reference/resource-properties/versions.md#detecting-breaking-changes) for models with contracts enforced:** When dbt detects a breaking change to a model with an enforced contract during state comparison, it will now raise an error for versioned models and a warning for models that are not versioned. * **[Set `access` as a config](https://docs.getdbt.com/reference/resource-configs/access.md):** You can now set a model's `access` within config blocks in the model's SQL file or in the project YAML file (`dbt_project.yml`) for an entire subfolder at once. * **[Type aliasing for model contracts](https://docs.getdbt.com/reference/resource-configs/contract.md):** dbt will use each adapter's built-in type aliasing for user-provided data types—meaning you can now write `string` always, and dbt will translate to `text` on Postgres/Redshift. This is "on" by default, but you can opt-out. * **[Raise warning for numeric types](https://docs.getdbt.com/reference/resource-configs/contract.md):** Because of issues when putting `numeric` in model contracts without considering that default values such as `numeric(38,0)` might round decimals accordingly. dbt will now warn you if it finds a numeric type without specified precision/scale. ##### dbt clean [dbt clean](https://docs.getdbt.com/reference/commands/clean.md) only cleans paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in the `clean-targets` section of `dbt_project.yml`, even if they're outside the dbt project. Supported flags: * `--clean-project-files-only` (default) * `--no-clean-project-files-only` ##### Additional attributes in run\_results.json The run\_results.json now includes three attributes related to the `applied` state that complement `unique_id`: * `compiled`: Boolean entry of the node compilation status (`False` after parsing, but `True` after compiling). * `compiled_code`: Rendered string of the code that was compiled (empty after parsing, but full string after compiling). * `relation_name`: The fully-qualified name of the object that was (or will be) created/updated within the database. ##### Deprecated functionality The ability for installed packages to override built-in materializations without explicit opt-in from the user is being deprecated. * Overriding a built-in materialization from an installed package raises a deprecation warning. * Using a custom materialization from an installed package does not raise a deprecation warning. * Using a built-in materialization package override from the root project via a wrapping materialization is still supported. For example: ```text {% materialization view, default %} {{ return(my_cool_package.materialization_view_default()) }} {% endmaterialization %} ``` ##### Quick hits With these quick hits, you can now: * Configure a [`delimiter`](https://docs.getdbt.com/reference/resource-configs/delimiter.md) for a seed file. * Use packages with the same git repo and unique subdirectory. * Access the `date_spine` macro directly from dbt-core (moved over from dbt-utils). * Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. --- ### Upgrading to v1.8 Available in v1 #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.8.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, except for any changes explicitly mentioned on this page. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). #### Release tracks dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. Select a release track in your development, staging, and production [environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md) to access everything in dbt v1.8+ and more. To upgrade an environment in the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest), set `dbt_version` to the string `latest`. #### New and changed features and functionality Features and functionality new in dbt v1.8. Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Unit Tests Historically, dbt's test coverage was confined to [“data” tests](https://docs.getdbt.com/docs/build/data-tests.md), assessing the quality of input data or resulting datasets' structure. In v1.8, we're introducing native support for [unit testing](https://docs.getdbt.com/docs/build/unit-tests.md). Unit tests validate your SQL modeling logic on a small set of static inputs **before** you materialize your full model in production. They support a test-driven development approach, improving both the efficiency of developers and the reliability of code. Starting from v1.8, when you execute the `dbt test` command, it will run both unit and data tests. Use the [`test_type`](https://docs.getdbt.com/reference/node-selection/methods.md#test_type) method to run only unit or data tests: ```shell dbt test --select "test_type:unit" # run all unit tests dbt test --select "test_type:data" # run all data tests ``` Unit tests are defined in YML files in your `models/` directory and are currently only supported on SQL models. To distinguish between the two, the `tests:` config has been renamed to `data_tests:`. Both are currently supported for backward compatibility. ###### New `data_tests:` syntax The `tests:` syntax is changing to reflect the addition of unit tests. Start migrating your [data test](https://docs.getdbt.com/docs/build/data-tests.md#new-data_tests-syntax) YML to use `data_tests:` after you upgrade to v1.8 to prevent issues in the future. ```yml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` ###### The `--empty` flag The [`run`](https://docs.getdbt.com/reference/commands/run.md#the-%60--empty%60-flag) and [`build`](https://docs.getdbt.com/reference/commands/build.md#the---empty-flag) commands now support the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. ##### dbt-core and adapters are decoupled Before v1.8, dbt adapters directly depended on components of `dbt-core`, and `dbt-core` depended on the adapter for execution. This bidirectional dependency made it difficult to develop adapters independently. Starting in dbt v1.8, [`dbt-core` and adapters are decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87), making it easier to maintain and evolve them independent of each other. For backward compatibility, adapter packages continue to depend on `dbt-core` at install time. Running a dbt project still requires both `dbt-core` and an adapter, and since v1.0, many users have installed both together using [`pip install dbt-<adapter>`](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md#installation>). This behavior remains unchanged. For example, `pip install dbt-snowflake` installs the latest versions of both `dbt-core` and `dbt-snowflake`. ##### Deprecated functionality The ability for installed packages to override built-in materializations without explicit opt-in from the user is being deprecated. * Overriding a built-in materialization from an installed package raises a deprecation warning. * Using a custom materialization from an installed package does not raise a deprecation warning. * Using a built-in materialization package override from the root project via a wrapping materialization is still supported. For example: ```sql {% materialization view, default %} {{ return(my_cool_package.materialization_view_default()) }} {% endmaterialization %} ``` ##### Managing changes to legacy behaviors dbt v1.8 has introduced flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Mature, enabled by default) [Require explicit package overrides for builtin materializations](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_explicit_package_overrides_for_builtin_materializations.md) * (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_spaces.md) * (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md) #### Quick hits * Custom defaults of [global config flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) should be set in the `flags` dictionary in [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md), instead of in [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md). Support for `profiles.yml` has been deprecated. * New CLI flag [`--resource-type`/`--exclude-resource-type`](https://docs.getdbt.com/reference/global-configs/resource-type.md) for including/excluding resources from dbt `build`, `run`, and `clone`. * To improve performance, dbt now issues a single (batch) query when calculating `source freshness` through metadata, instead of executing a query per source. * Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. * The `--select` flag now reliably limits `catalog.json` to the selected nodes in all projects, regardless of project size. In earlier versions, node selectors were ignored for projects with 100 or more nodes, and sources were omitted from `catalog.json` entirely when more than 100 nodes were selected. --- ### Upgrading to v1.9 Available in v1 #### Resources * [dbt 1.9 changelog](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality that is included in dbt v1.9! If you have selected the **v1 Compatible** release track, you will have access in the next monthly **v1 Compatible** release after the dbt v1.9 final release. #### New and changed features and functionality Features and functionality new in dbt v1.9. ##### Microbatch `incremental_strategy` info If you use a custom microbatch macro, set the [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) behavior flag in your `dbt_project.yml` to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the microbatch strategy. Incremental models are, and have always been, a *performance optimization* — for datasets that are too large to be dropped and recreated from scratch every time you do a `dbt run`. Learn more about [incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md). Historically, managing incremental models involved several manual steps and responsibilities, including: * Add a snippet of dbt code (in an `is_incremental()` block) that uses the already-existing table (`this`) as a rough bookmark, so that only new data gets processed. * Pick one of the strategies for smushing old and new data together (`append`, `delete+insert`, or `merge`). * If anything goes wrong, or your schema changes, you can always "full-refresh", by running the same simple query that rebuilds the whole table from scratch. While this works for many use-cases, there’s a clear limitation with this approach: *Some datasets are just too big to fit into one query.* Starting in dbt 1.9, you can use the new [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md#what-is-microbatch-in-dbt) to optimize your largest datasets -- **process your event data in discrete periods with their own SQL queries, rather than all at once.** The benefits include: * Simplified query design: Write your model query for a single batch of data. dbt will use your `event_time`, `lookback`, and `batch_size` configurations to automatically generate the necessary filters for you, making the process more streamlined and reducing the need for you to manage these details. * Independent batch processing: dbt automatically breaks down the data to load into smaller batches based on the specified `batch_size` and processes each batch independently, improving efficiency and reducing the risk of query timeouts. If some of your batches fail, you can use `dbt retry` to load only the failed batches. * Targeted reprocessing: To load a *specific* batch or batches, you can use the CLI arguments `--event-time-start` and `--event-time-end`. * [Automatic parallel batch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md): Process multiple batches at the same time, instead of one after the other (sequentially) for faster processing of your microbatch models. dbt intelligently auto-detects if your batches can run in parallel, while also allowing you to manually override parallel execution with the [`concurrent_batches` config](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md). Currently microbatch is supported on these adapters with more to come: * postgres * redshift * snowflake * bigquery * spark * databricks ##### Snapshots improvements Beginning in dbt 1.9, we've streamlined snapshot configuration and added a handful of new configurations to make dbt **snapshots easier to configure, run, and customize.** These improvements include: * New snapshot specification: Snapshots can now be configured in a YAML file, which provides a cleaner and more consistent set up. * New `snapshot_meta_column_names` config: Allows you to customize the names of meta fields (for example, `dbt_valid_from`, `dbt_valid_to`, etc.) that dbt automatically adds to snapshots. This increases flexibility to tailor metadata to your needs. * `target_schema` is now optional for snapshots: When omitted, snapshots will use the schema defined for the current environment. * Standard `schema` and `database` configs supported: Snapshots will now be consistent with other dbt resource types. You can specify where environment-aware snapshots should be stored. * Warning for incorrect `updated_at` data type: To ensure data integrity, you'll see a warning if the `updated_at` field specified in the snapshot configuration is not the proper data type or timestamp. * Set a custom current indicator for the value of `dbt_valid_to`: Use the [`dbt_valid_to_current` config](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. * Use the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) configuration to get more control on how to handle deleted rows from the source. Supported methods are `ignore` (default), `invalidate` (replaces legacy `invalidate_hard_deletes=true`), and `new_record`. Setting `hard_deletes='new_record'` allows you to track hard deletes by adding a new record when row becomes "deleted" in source. Read more about [Snapshots meta fields](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields). To learn how to safely migrate existing snapshots, refer to [Snapshot configuration migration](https://docs.getdbt.com/reference/snapshot-configs.md#snapshot-configuration-migration) for more information. ##### Some `properties` moved to `configs` The following `properties` were moved to `configs` in [dbt v1.10](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md) and backported to dbt v1.9: * [`freshness`](https://docs.getdbt.com/reference/resource-configs/freshness.md) for sources * dbt v2 [expands freshness checks](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md?version=2#model-freshness-and-the-dbt-freshness-command) to support models. * [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) under `columns` * [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md) under `columns` ##### `state:modified` improvements We’ve made improvements to `state:modified` behaviors to help reduce the risk of false positives and negatives. Read more about [the `state:modified` behavior flag](#managing-changes-to-legacy-behaviors) that unlocks this improvement: * Added environment-aware enhancements for environments where the logic purposefully differs (for example, materializing as a table in `prod` but a `view` in dev). ##### Managing changes to legacy behaviors dbt v1.9 has a handful of new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Introduced, disabled by default) [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md). Set to `true` to persist `unrendered_config` during model parsing and `unrendered_database`/`unrendered_schema` during source parsing, then compare on unrendered values during `state:modified` checks. This reduces false positives from environment-aware logic, but requires rebuilding your state directory; without a rebuild, `state:modified` may select nodes with Jinja in YAML configs that haven't actually changed. * (Introduced, disabled by default) [`skip_nodes_if_on_run_start_fails` project config flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md). If the flag is set and **any** `on-run-start` hook fails, mark all selected nodes as skipped. * `on-run-start/end` hooks are **always** run, regardless of whether they passed or failed last time. - (Introduced, disabled by default) [`require_nested_cumulative_type_params`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md). If the flag is set to `true`, users will receive an error instead of a warning if they're not properly formatting cumulative metrics using the new [`cumulative_type_params`](https://docs.getdbt.com/docs/build/cumulative.md#parameters) nesting. - (Introduced, disabled by default) [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md). Set to `true` if you use a custom microbatch macro to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the microbatch strategy. #### Adapter-specific features and functionalities Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Redshift * Support IAM Role auth ##### Snowflake * Iceberg Table Format — Support will be available on three out-of-the-box materializations: table, incremental, dynamic tables. * Breaking change — When upgrading from dbt 1.8 to 1.9 `{{ target.account }}` replaces underscores with dashes. For example, if the `target.account` is set to `sample_company`, then the compiled code now generates `sample-company`. [Refer to the `dbt-snowflake` issue](https://github.com/dbt-labs/dbt-snowflake/issues/1286) for more information. ##### Bigquery * Can cancel running queries on keyboard interrupt * Auto-drop intermediate tables created by incremental models to save resources ##### Spark * Support overriding the ODBC driver connection string which now enables you to provide custom connections #### Quick hits We also made some quality-of-life improvements in dbt 1.9, enabling you to: * Maintain data quality now that dbt returns an error (versioned models) or warning (unversioned models) when someone [removes a contracted model by deleting, renaming, or disabling](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md#how-are-breaking-changes-handled) it. * Document [data tests](https://docs.getdbt.com/reference/resource-properties/description.md). * Use `ref` and `source` in [foreign key constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md). * Use `dbt test` with the `--resource-type` / `--exclude-resource-type` flag, making it possible to include or exclude data tests (`test`) or unit tests (`unit_test`). * The [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) config is now available for unit tests. Defaults to `true` if not defined. --- ### Upgrading to v2 Available in v2 dbt v2 is the latest major version of the dbt framework. When you install dbt, you get dbt v2 by default. This guide walks you through upgrading a v1 project to v2. v2 is faster and stricter, but your existing project language and DAG semantics carry over, so once you upgrade, your project works as before — just faster. important dbt v2 is available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our dbt Community Slack channel [`#dbt-v2`](https://getdbt.slack.com/archives/C088YCAB6GH). #### More information about dbt v2 * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) #### Install dbt dbt v2 has a new PyPI name: `dbt`. When installing with a package manager like `pip`, use the `dbt` namespace to get the free, fully-featured distribution: ```shell python -m pip install dbt ``` For full instructions, including Homebrew, winget, and additional options, refer to [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). #### What to know before upgrading If you have an older project that isn't ready to move to v2, or you need compatibility with existing tooling, packages, or workflows that haven't moved to v2 yet, you can stay on dbt v1.x, which remains fully supported. Over time, new capabilities will land in v2 only, so most people will eventually want to upgrade. To install or continue using v1.x, refer to [Install dbt v1.x](https://docs.getdbt.com/docs/local/install-dbt.md?version=1). This new major version is an opportunity to *strengthen the framework* by removing deprecated functionality, rationalizing confusing behavior, and providing more rigorous validation on erroneous inputs. This means that there is some work involved in preparing an existing dbt project for v2. That work is documented below — it should be simple, straightforward, and in many cases, auto-fixable with the [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) helper or the [agent skill](https://github.com/dbt-labs/dbt-agent-skills/tree/main/skills/dbt-migration/skills/migrating-dbt-core-to-v2). Test v2 parser compatibility from dbt v1.12 If you're on dbt v1.12, you can test the rust parser compatibility before fully migrating by using the opt-in [`--use-v2-parser`](https://docs.getdbt.com/reference/global-configs/parsing.md#opt-in-v2-parser) flag. This delegates parsing to the v2 parser without changing any other behavior, making it a low-risk way to catch compatibility issues early. ###### Upgrade considerations Keep in mind the following considerations during the upgrade process: * **Manifest compatibility** — dbt v2 produces a `v12` [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) that's compatible with dbt v1. The only differences are optional dbt v2-specific fields that only dbt v2 writes, which dbt v1 safely ignores. As a result, you can run dbt v2 and dbt v1 side by side. State-dependent features such as `state:modified`, `--defer`, and cross-environment `dbt docs generate` work across mixed dbt v2 and dbt v1 environments, so you can migrate to dbt v2 incrementally without breaking existing dbt v1 jobs. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). ##### Supported adapters The following adapters are supported in v2:  BigQuery * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks * Service Account / User Token * Native OAuth  Redshift * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  ClickHouse[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password  Apache Spark (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Thrift * Simple Authentication and Security Layer (SASL) PLAIN * No SASL (NOSASL) * Livy * Basic authentication (username and password) * When deployed on Amazon Web Services (AWS): AWS Signature Version 4 * Supports authentication using single sign-on, service accounts, or user tokens  DuckDB (CLI only) DuckDB does not require authentication — it runs locally on your machine. *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* ##### A clean slate v2 will not support any deprecated functionality (see the [Changes overview](https://docs.getdbt.com/reference/changes-overview.md) for details): * All [deprecation warnings](https://docs.getdbt.com/reference/deprecations.md) must be resolved before upgrading to the new engine. This includes historic deprecations and [new ones as of dbt v1.10](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md#deprecation-warnings). * Some [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) will be removed (generally enabled). Flags marked `2.0` in the **Removed** column in that table are removed: the new behavior is always enabled and you can no longer opt out with `flags:` in your `dbt_project.yml`. All other flags without a removal version (marked `-`) remain configurable, including v2-specific flags listed later on that page. ##### Ecosystem packages The most popular `dbt-labs` packages (`dbt_utils`, `audit_helper`, `dbt_external_tables`, `dbt_project_evaluator`) are already compatible with v2. External packages published by organizations outside of dbt may use outdated code or incompatible features that fail to parse in v2. We're working with those package maintainers to make packages available for v2. Packages requiring an upgrade to a new release for v2 compatibility, will be documented in this upgrade guide. #### New and changed features and functionality ##### Strict validation In v1, misspelled configs, unexpected YAML keys, and invalid flags were silently ignored. In v2, dbt enforces a tightly-defined language specification at parse time and raises explicit errors for any violation, including unused config paths in `dbt_project.yml`, unknown CLI options, and duplicate config keys. ##### Faster Rust parser The v2 engine is a complete rewrite in Rust, delivering faster parse and compile times, especially on large projects. No configuration is needed; the performance improvement is automatic. ##### dbt Information Schema Similar to a database's `INFORMATION_SCHEMA`, the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) is a contracted interface into the metadata for all of the resources in your dbt project. When you use the `--generate-info-schema` flag or set `DBT_ENGINE_GENERATE_INFO_SCHEMA=true`, dbt writes the Information Schema to `target/info_schema/` in a versioned subdirectory (for example, `target/info_schema/v1/`) as standard Parquet files. The metadata available in the schema grows with each step: parsing produces basic metadata, compiling adds model column types and column-level lineage (with `--static-analysis strict`), and running or building populates runtime results. For more information, refer to [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md). ##### Checks In dbt v2, you can create [checks](https://docs.getdbt.com/docs/build/checks.md) to enforce project standards (for example, all models must have a description, a public model must have an owner, and so on) at parse time, before any warehouse work runs. Write a SQL rule under the `checks/` directory, then run checks on demand with `dbt check`. Checks also run automatically with every `dbt build`. Use `--skip-checks` to bypass checks on a build. For more information, refer to [Checks](https://docs.getdbt.com/docs/build/checks.md). ##### dbt Docs v2 v2 introduces [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2), a fast, modern self-hosted catalog experience built to help you understand and trust your production data. You get column-level lineage, Semantic Layer metadata, and a beautifully refreshed interface, all working smoothly on the largest projects. Under the hood, your metadata lives in efficient Parquet artifacts for faster load times and a catalog that scales as your project grows. `dbt docs generate` compiles your project, produces the v2 Parquet artifacts, and exports a static site in a single command. `dbt docs serve` previews that site locally. Because the browser queries those artifacts directly with DuckDB-WASM, you can also host the generated files on any static file host. Column-level lineage is visible when you build with `--static-analysis strict`. dbt Docs v2 runs locally only. If you use dbt platform, your jobs refresh Catalog metadata automatically, so explore your project in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) instead. To hydrate catalog metadata (`catalog.json`) for Catalog without building the site, use the [`--write-catalog` flag](https://docs.getdbt.com/reference/commands/cmd-docs.md#--write-catalog-flag) instead. For full usage, refer to [About dbt docs commands](https://docs.getdbt.com/reference/commands/cmd-docs.md?version=2). ##### Batch tests Data tests can now run in batches! When you use `--batch-tests`, compatible data tests applied to a single model are batched into one query per test type. Internal testing shows batch tests reduce test command runtime by 40%, queries issued by 65%, and warehouse cost of running tests by 15%! Enable with `--batch-tests` on [`dbt test`](https://docs.getdbt.com/reference/commands/test.md) or [`dbt build`](https://docs.getdbt.com/reference/commands/build.md), or using the `DBT_ENGINE_BATCH_TESTS=true` environment variable. For more information on batching, refer to [batch tests](https://docs.getdbt.com/reference/global-configs/batch-tests.md). ##### Model freshness and the `dbt freshness` command v2 expands freshness checks to models, building on the existing support for sources. You can configure freshness thresholds on models to receive warnings or errors when the data is stale. For config options and materialization requirements, refer to [freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md). Use the new [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) command to check all sources and models with freshness configured in a single invocation and, and to write results to a [`target/freshness.json` file](https://docs.getdbt.com/reference/artifacts/freshness-json.md). The [`dbt source freshness`](https://docs.getdbt.com/reference/commands/source.md?version=2#dbt-source-freshness) command remains supported for backward compatibility, checks sources only, and continues to produce a `sources.json` file. We recommend using `dbt freshness` going forward. ##### Adapters built on ADBC drivers All v2 adapters connect to data warehouses via the [Arrow Database Connectivity (ADBC)](https://arrow.apache.org/adbc/) standard instead of Python-based adapter libraries. As a result, dbt ships as a single self-contained binary with no Python runtime required. On first run, dbt downloads adapter drivers from the dbt Labs CDN and caches them locally. Subsequent runs work offline. For supported adapters, refer to [Supported data platforms](https://docs.getdbt.com/docs/supported-data-platforms.md). ##### `dbt lint` v2 introduces [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md), a high-performance SQL linter built into dbt. It is SQLFluff-compatible: you keep your existing .sqlfluff config and rule codes (for example, `CP01`, `RF03`). Run `dbt lint` to lint all models, or `dbt lint [FILE]` to target a specific file. Use `--fix` to auto-apply fixable violations. ##### Static analysis Unlike dbt v1.x, which only rendered Jinja-templated SQL strings into queries, v2 adds a second phase: static analysis. After rendering, the engine produces and validates a logical plan for every query in your project — without executing anything against the warehouse. This enables dialect-aware validation, column-level lineage, and precise type checking. The [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) config controls how strictly this is applied. ###### Baseline static analysis Baseline mode is the default in dbt v2. It parses each model's SQL at compile time to understand its structure without a warehouse connection, catching most SQL errors while providing a smooth migration experience. In baseline mode, all findings are warnings rather than errors, so your project continues running even when the compiler flags issues. Enable it explicitly per-model or project-wide: ```yaml # dbt_project.yml models: your_project: +static_analysis: baseline ``` Or pass `--static-analysis baseline` on the CLI. For details, refer to [About static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md). ###### Strict static analysis Strict mode fully resolves column types and validates references across your entire project before execution begins; nothing runs until the project is proven valid. It is required to produce [column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) and unlocks additional LSP features like column go-to-definition and type checking. Enable it per-model or project-wide: ```yaml # dbt_project.yml models: your_project: +static_analysis: strict ``` Or pass `--static-analysis strict` on the CLI (or set `DBT_STATIC_ANALYSIS=strict`). For details, refer to [static\_analysis](https://docs.getdbt.com/reference/resource-configs/static-analysis.md). Deprecated values `static_analysis: on` and `static_analysis: unsafe` are deprecated synonyms for `strict`. Update these to `strict`; they will be removed in a future release. ##### Column-level lineage v2 tracks which source columns flow into which output columns across your entire DAG. To generate column lineage, build or compile with `--static-analysis strict`: ```shell dbt build --static-analysis strict ``` The lineage is then visible in [dbt-docs](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2). No additional configuration is needed; the site detects the presence of the lineage artifact automatically. For details, refer to [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md). ##### Language server protocol (LSP) v2 includes a built-in language server that enables IDE features for dbt SQL and YAML files, including hover information, diagnostics, go-to-definition, and column-level completions. The standalone `dbt-lsp` package is no longer published; LSP is now bundled in the `dbt` binary and used automatically by the dbt VS Code extension and Studio IDE. Features available depend on your `static_analysis` setting: `baseline` adds syntax error detection and CTE preview; `strict` adds column go-to-definition, column lineage, and type checking. For details, refer to [About dbt LSP](https://docs.getdbt.com/docs/about-dbt-lsp.md). ##### Agent skills v2 introduces [agent skills](https://docs.getdbt.com/docs/dbt-ai/package-skills.md), which are reusable instructions your coding agent reads from a `SKILL.md` file. You and your team can ship skills from your own project or from a package, so everyone works from one set of conventions instead of copying files between repos. To use agent skills, you need to: * Include skills in your project's `skills/` directory, use a package that ships skills, or both * Set the `ai_provider` flag in your root project to tell dbt which coding agent you use. Supported values are `wizard`, `claude`, `openai`, `codex`, `cursor`, or `gemini` (case-insensitive). Once both are in place, `dbt deps` installs those skills into the directory your agent reads from (such as `.claude/skills` or `.agents/skills`), and `dbt clean` removes them. If you're missing either piece, `dbt deps` installs your packages as usual and skips the skills. For full usage info, including how to disable a skill you don't want, refer to [Installing agent skills from dbt packages](https://docs.getdbt.com/docs/dbt-ai/package-skills.md). ##### Experimental features ###### Local execution of unit tests v2 introduces the [`compute`](https://docs.getdbt.com/reference/resource-configs/compute.md) config for unit tests. Set your unit tests with `compute: local` and dbt runs the test with DuckDB instead of sending it to your data platform, which takes the warehouse round trip out of your development loop. This config is experimental and requires opt-in: set `DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS=true` in the environment where dbt runs before you use `compute: local`. For details, refer to [Run unit tests locally](https://docs.getdbt.com/docs/build/unit-tests.md#run-unit-tests-locally). ###### Multi-adapter invocations v2 supports running a single project against multiple adapters simultaneously as part of our cross-platform [dbt Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh.md?version=2). This is experimental and requires opt-in: ```bash export DBT_ENGINE_EXPERIMENTAL_MULTI_ADAPTER=true ``` Without this flag, dbt fails at parse time if any node in the project has an `adapter` config, even for runs that don't select that node. The gate reads config as written, not as selected, so the entire project needs the env var to parse once any model uses a non-default adapter. ##### Changed functionality When developing v2, there were opportunities to improve the dbt framework — failing earlier (when possible), fixing bugs, optimizing run order, and deprecating flags that are no longer relevant. The result is a handful of specific and nuanced changes to existing behavior. When upgrading to v2, you should expect the following changes in functionality: ###### Calling project macros in YAML files You can call project macros directly in YAML files such as `sources.yml` and `schema.yml`: ```yaml sources: - name: raw_orders database: "{{ get_environment() }}" ``` This is not supported in dbt v1, because the two engines use different Jinja contexts when rendering YAML: * dbt v2 registers project macros before rendering, so they're available to Jinja in the YAML. * In dbt v1, YAML files are rendered using a separate, restricted context that doesn't include project macros — only built-in Jinja functions like `env_var()` and `doc()`. For more information, refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). ###### Parse time printing of relations will print out the full qualified name, instead of an empty string In dbt v1, when printing the result of `get_relation()`, the parse time output for that Jinja would print `None` (the undefined object coerces to the string "None"). In v2, to help with intelligent batching of `get_relation()` calls (and significantly speed up `dbt compile`), dbt needs to construct a relation object with the fully qualified name resolved at parse time for the `get_relation()` adapter call. Constructing a relation object with the fully qualified name in v2 produces different behavior than v1 in `print()`, `log()`, or any Jinja macro that outputs to `stdout` or `stderr` at parse time. Example: ```jinja {% set relation = adapter.get_relation( database=db_name, schema=db_schema, identifier='a') %} {{ print('relation: ' ~ relation) }} {% set relation_via_api = api.Relation.create( database=db_name, schema=db_schema, identifier='a' ) %} {{ print('relation_via_api: ' ~ relation_via_api) }} ``` The output after `dbt parse` in dbt v1: ```text relation: None relation_via_api: my_db.my_schema.my_table ``` The output after `dbt parse` in v2: ```text relation: my_db.my_schema.my_table relation_via_api: my_db.my_schema.my_table ``` ###### Deprecated flags Deprecated flags are command-line flags (like `--models`, `--print`) that you pass to dbt commands. These are being removed in v2. This is different from: * [Deprecation warnings](https://docs.getdbt.com/reference/deprecations.md) — Features in your project code (models, YAML, macros) that need to be updated * [Behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) — Flags in `dbt_project.yml` that let you opt in/out of new behaviors See the [Changes overview](https://docs.getdbt.com/reference/changes-overview.md) for a full comparison. Some historic CLI flags from v1 will no longer do anything in v2. If you pass them into a dbt command in v2, the command will not error, but the flag will do nothing (and warn accordingly). | flag name | remediation | | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | `--models` / `--model` / `-m` | Refer to [CLI flags that need changes](#cli-flags-that-need-changes). | | `dbt seed` [`--show`](https://docs.getdbt.com/reference/commands/seed.md) | N/A | | [`--print` / `--no-print`](https://docs.getdbt.com/reference/global-configs/print-output.md) | No action required | | [`--printer-width`](https://docs.getdbt.com/reference/global-configs/print-output.md#printer-width) | No action required | | [`--source`](https://docs.getdbt.com/reference/commands/deps.md#non-hub-packages) | No action required | | [`--record-timing-info` / `-r`](https://docs.getdbt.com/reference/global-configs/record-timing-info.md) | No action required | | [`--cache-selected-only` / `--no-cache-selected-only`](https://docs.getdbt.com/reference/global-configs/cache.md) | No action required | | [`--clean-project-files-only` / `--no-clean-project-files-only`](https://docs.getdbt.com/reference/commands/clean.md#--clean-project-files-only) | No action required | | `--single-threaded` / `--no-single-threaded` | No action required | | `dbt source freshness` [`--output` / `-o`](https://docs.getdbt.com/reference/commands/source.md?version=1.12#source-freshness-commands) | | | [`--config-dir`](https://docs.getdbt.com/reference/commands/debug.md) | No action required | | [`--resource-type` / `--exclude-resource-type`](https://docs.getdbt.com/reference/global-configs/resource-type.md) | Refer to [CLI flags that need changes](#cli-flags-that-need-changes). | | `--show-resource-report` / `--no-show-resource-report` | No action required | | [`--log-cache-events` / `--no-log-cache-events`](https://docs.getdbt.com/reference/global-configs/logs.md#logging-relational-cache-events) | No action required | | `--use-experimental-parser` / `--no-use-experimental-parser` | No action required | | [`--empty-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate) | | | [`--compile` / `--no-compile`](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate) | | | `--inline-direct` | No action required | | `--partial-parse-file-diff` / `--no-partial-parse-file-diff` | No action required | | `--partial-parse-file-path` | No action required | | `--populate-cache` / `--no-populate-cache` | No action required | | `--static-parser` / `--no-static-parser` | No action required | | `--use-fast-test-edges` / `--no-use-fast-test-edges` | No action required | | `--inject-ephemeral-ctes` / `--no-inject-ephemeral-ctes` | | | [`--partial-parse` / `--no-partial-parse`](https://docs.getdbt.com/reference/parsing.md#partial-parsing) | Refer to [CLI flags that need changes](#cli-flags-that-need-changes). | ###### CLI flags that need changes The following deprecated flag requires updates in your job definitions or scripts: * **`--models` / `--model` / `-m`:** Use `--select` / `-s` instead (renamed in dbt v0.21). dbt raises an error in v2 if you use the old flags. Do not pass `--models` as the value to `-s` (for example, `dbt run -s --models`); v1 treated that as a model name, but v2 requires a valid selector. dbt v2 job runs no longer support the `--partial-parse` and `--no-partial-parse` CLI flags. If you pass them (for example, from a dbt v1 command or script), dbt logs deprecation warning `dbt1700`. Remove these flags from your dbt v2 job commands. For more information, refer to [Deprecated flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) in the guide to upgrading to dbt v2. ###### Conflicting package versions when a local package depends on a hub package which the root package also wants will error If a local package depends on a hub package that the root package also wants, `dbt deps` doesn't resolve conflicting versions in dbt v1; it will install whatever the root project requests. v2 will present an error: ```bash error: dbt8999: Cannot combine non-exact versions: =0.8.3 and =1.1.1 ``` ###### Parse will fail on nonexistent macro invocations and adapter methods When you call a nonexistent macro in dbt: ```sql select id as payment_id, # my_nonexistent_macro is a macro that DOES NOT EXIST {{ my_nonexistent_macro('amount') }} as amount_usd, from app_data.payments ``` Or a nonexistent adapter method: ```sql {{ adapter.does_not_exist() }} ``` In v1, `dbt parse` passes, but `dbt compile` fails. In v2, dbt will error out during `parse`. ###### Parse will fail on missing generic test When you have an undefined generic test in your project: ```yaml models: - name: dim_wizards data_tests: - does_not_exist ``` In v1, `dbt parse` passes, but `dbt compile` fails. In v2, dbt will error out during `parse`. ###### Parse will fail on missing variable When you have an undefined variable in your project: ```sql select {{ var('does_not_exist') }} as my_column ``` In v1, `dbt parse` passes, but `dbt compile` fails. In v2, dbt will error out during `parse`. ###### Stricter evaluation of duplicate docs blocks In v1, it was possible to create scenarios with duplicate [docs blocks](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks). For example, you can have two packages with identical docs blocks referenced by an unqualified name in your dbt project. In this case, v1 would use whichever docs block is referenced without any warnings or errors. v2 adds stricter evaluation of names of docs blocks to prevent such ambiguity. It will present an error if it detects duplicate names: ```bash dbt found two docs with the same name: 'docs_block_title' in files: 'models/crm/_crm.md' and 'docs/crm/business_class_marketing.md' ``` To resolve this error, rename any duplicate docs blocks. ###### `dbt clean` will not delete any files in configured resource paths or files outside the project directory In dbt v1, `dbt clean` deletes: * Any files outside the project directory if `clean-targets` is configured with an absolute path or relative path containing `../`, though there is an opt-in config to disable this (`--clean-project-files-only` / `--no-clean-project-files-only`). * Any files in the `asset-paths` or `doc-paths` (even though other resource paths, like `model-paths` and `seed-paths`, are restricted). In v2, `dbt clean` will not delete any files in configured resource paths or files outside the project directory. ###### All unit tests are run first in `dbt build` In dbt v1, the direct parents of the model being unit tested needed to exist in the warehouse to retrieve the needed column name and type information. `dbt build` runs the unit tests (and their dependent models) *in lineage order*. In v2, `dbt build` runs *all* of the unit tests *first*, and then builds the rest of the DAG, due to built-in column name and type awareness. ###### Configuring `--threads` dbt v1 runs with `--threads 1` by default. You can increase this number to run more nodes in parallel on the remote data platform, up to the max parallelism enabled by the DAG. v2 handles threading differently depending on your data platform: | Adapter | Behavior | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Snowflake** | dbt v2 automatically manages connection parallelism based on platform limits and backpressure. The `threads` setting acts as a maximum connection cap if set, but dbt v2 is designed to work optimally without it configured. If you're experiencing timeout or rate limit issues, setting `threads` to a lower value can help. | | **Databricks** | dbt v2 automatically manages connection parallelism based on platform limits and backpressure. The `threads` setting acts as a maximum connection cap if set, but dbt v2 is designed to work optimally without it configured. If you're experiencing timeout or rate limit issues, setting `threads` to a lower value can help. | | **BigQuery** | dbt v2 respects user-set threads to manage API rate limits.<br />Setting `--threads 0` (or omitting the setting) allows dbt v2 to dynamically optimize parallelism. | | **Redshift** | dbt v2 respects user-set threads to manage concurrency limits.<br />Setting `--threads 0` (or omitting the setting) allows dbt v2 to dynamically optimize parallelism. | For more information, refer to [Using threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md#dbt-v2-thread-optimization). ###### Continue to compile unrelated nodes after hitting a compile error As soon as v1's `compile` encounters an error compiling one of your models, dbt stops and doesn't compile anything else. When v2's `compile` encounters an error, it will skip nodes downstream of the one that failed to compile, but it will keep compiling the rest of the DAG (in parallel, up to the number of configured / optimal threads). ###### Seeds with extra commas don't result in extra columns In dbt v1, if you have an additional comma on your seed, dbt creates a seed with an additional empty column. For example, the following seed file (with an extra comma): ```text animal, dog, cat, bear, ``` Will produce this table when `dbt seed` is executed: | animal | b | | ------ | - | | dog | | | cat | | | bear | | In v2, it will not produce this extra column in the table resulting from `dbt seed`: | animal | | ------ | | dog | | cat | | bear | ###### Move standalone anchors under `anchors:` key As part of the ongoing process of making the dbt authoring language more precise, unexpected top-level keys in a YAML file will result in errors. A common use case behind these unexpected keys is standalone anchor definitions at the top level of a YAML file. You can use the new top-level `anchors:` key as a container for these reusable configuration blocks. For example, rather than using this configuration: models/\_models.yml ```yml # id_column is not a valid name for a top-level key in the dbt authoring spec, and will raise an error id_column: &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models. - name: my_second_model columns: - *id_column_alias ``` Move the anchor under the `anchors:` key instead: models/\_models.yml ```yml anchors: - &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models - name: my_second_model columns: - *id_column_alias ``` This move is only necessary for fragments defined outside of the main YAML structure. For more information about this new key, see [anchors](https://docs.getdbt.com/reference/resource-properties/anchors.md). ###### Self-referential (recursive) YAML anchors are not supported In v1, dbt could parse a YAML anchor that merges into an element of the same sequence it's defined on, creating a self-referential (cyclic) anchor. For example, anchoring a full `tables:` sequence and then merging that anchor into one of the sequence's own elements: ```yml sources: - name: catalogue tables: &tables - name: anchor_item description: The first table in the sequence. - <<: *tables name: merged_item ``` This parsed successfully in v1 only because PyYAML (the YAML library dbt v1 depends on) incidentally allows self-referential anchors, a side effect of Python's own support for cyclic data structures, not an intentional YAML feature. No other major YAML implementation allows this pattern. In v2, parsing this pattern hits a recursion limit and raises an error, so the entire properties file fails to parse. This is a deliberate limitation, not a bug. v2 does not plan to support self-referential anchors. To resolve this, remove the self-reference. Anchor only the parts of the document that don't merge back into themselves, for example, anchor a single table mapping instead of the whole sequence: ```yml sources: - name: catalogue tables: - &anchor_item_alias name: anchor_item description: The first table in the sequence. - <<: *anchor_item_alias name: merged_item ``` ###### Algebraic operations in Jinja macros In v1, you can set algebraic functions in the return function of a Jinja macro: ```jinja {% macro my_macro() %} return('xyz') + 'abc' {% endmacro %} ``` This is no longer supported in v2 and will emit a warning: ```bash [warning] [JinjaTopLevelReturn (dbt1508)]: return is not at the top level of the block. Its value is final and cannot be modified by surrounding expressions. Example: return(0) + 1. The + 1 is ignored and the macro returns 0. ``` This is not a common use case and there is no deprecation warning for this behavior in v1. The supported format is: ```jinja {% macro my_macro() %} return('xyzabc') {% endmacro %} ``` ###### Accessing custom configurations in meta `config.get()` and `config.require()` don't return values from the `meta` dictionary. If you try to access a key that only exists in `meta`, dbt emits a warning: ```bash warning: The key 'my_key' was not found using config.get('my_key'), but was detected as a custom config under 'meta'. Please use config.meta_get('my_key') or config.meta_require('my_key') instead. ``` Behavior when a key exists only in meta: | Method | Behavior | | -------------------------- | ---------------------------------------------- | | `config.get('my_key')` | Returns the default value and emits a warning. | | `config.require('my_key')` | Raises an error and emits a warning. | In addition, accessing the `meta` dictionary itself (rather than a specific key) behaves differently across engines: | Engine | `config.get('meta', default={})` | | ------ | ----------------------------------- | | dbt v1 | Does not work. | | dbt v2 | Returns the full `meta` dictionary. | To access custom configurations stored under `meta`, use the explicit methods: ```jinja {% set owner = config.meta_get('owner') %} {% set has_pii = config.meta_require('pii') %} ``` For more information, see [config.meta\_get()](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_get) and [config.meta\_require()](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_require). #### Quick hits * v2 supports exporting traces and logs in JSONL, Parquet, and OTLP formats. For details, refer to [dbt v2 telemetry and observability](https://docs.getdbt.com/reference/telemetry-observability.md). * The v2 compiler parses and type-checks [Snowflake model function](https://docs.snowflake.com/en/guides-overview-ml-functions) calls (`model!method(...)`), accepting any arguments and treating results as `VARIANT`. Cast the result to the type you need (for example, `model!predict(col)::float`). * In dbt v2, [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) opens your browser to sign in to dbt platform or create a free account. Run [`dbt login status`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-status) to view your current authentication status. #### Package support To determine if a package is compatible with dbt v2, visit the [dbt package hub](https://hub.getdbt.com/) and look for the dbt v2-compatible badge, or review the package's [`require-dbt-version` configuration](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md#pin-to-a-range). * Packages with a `require-dbt-version` that equals or contains `2.0.0` are compatible with dbt v2. For example, `require-dbt-version: ">=1.10.0,<3.0.0"`. Even if a package doesn't reflect compatibility in the package hub, it may still work with v2. Work with package maintainers to track updates, and [thoroughly test packages](https://docs.getdbt.com/guides/dbt-package-compat?step=5) that aren't clearly compatible before deploying. * Package maintainers who would like to make their package compatible with v2 can refer to the [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) for instructions. Fivetran package considerations: * The Fivetran `source` and `transformation` packages have been combined into a single package. * If you manually installed source packages like `fivetran/github_source`, you need to ensure `fivetran/github` is installed and deactivate the transformation models. ###### Package compatibility messages Inconsistent v2 warnings and `dbt-autofix` logs dbt v2 warnings and `dbt-autofix` logs may show different messages about package compatibility. If you use [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) while upgrading to v2 in the Studio IDE or dbt VS Code extension, you may see different messages about package compatibility between `dbt-autofix` and v2 warnings. Here's why: * dbt v2 warnings are emitted based on a package's `require-dbt-version` and whether `require-dbt-version` contains `2.0.0`. * Some packages are already v2-compatible even though package maintainers haven't yet updated `require-dbt-version`. * `dbt-autofix` knows about these compatible packages and will not try to upgrade a package that it knows is already compatible. This means that even if you see a v2 warning for a package that `dbt-autofix` identifies as compatible, you don't need to change the package. The message discrepancy is temporary while we implement and roll out `dbt-autofix`'s enhanced compatibility detection to v2 warnings. Here's an example of a v2 warning in the Studio IDE that says a package isn't compatible with v2 but `dbt-autofix` indicates it is compatible: ```text dbt1065: Package 'dbt_utils' requires dbt version [>=1.30,<2.0.0], but current version is 2.0.0. This package may not be compatible with your dbt version. dbt(1065) [Ln 1, Col 1] ``` #### Distributions v2 is available in two distributions. For more information, refer to [dbt licensing](https://docs.getdbt.com/docs/dbt-licensing.md). | Distribution | Package | Use it when | | ------------ | --------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | dbt v2 | `dbt` | The recommended v2 experience. | | dbt OSS | `dbt-oss` | Your organization has a strict requirement to use the Apache 2.0 [open-source runtime](https://docs.getdbt.com/docs/local/install-dbt-v2.md). | If you have a older project that isn’t ready to move to v2, continue using v1.x for compatibility. For new or upgraded projects, we recommend v2. --- ### Use dbt Wizard locally Beta ### Use dbt Wizard locally [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Install dbt Wizard locally and start an agentic dbt development session from your terminal. You can run the dbt Wizard CLI locally from any project on any dbt engine. Be warned, the wizard has been known to cast spells . Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. #### Prerequisites * Access to a [supported AI provider](#supported-ai-providers). Use a dbt managed provider or configure [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) with your own provider credentials. * A dbt project with a built `target/` directory (run `dbt parse`, `dbt compile`, or `dbt build`) dbt Wizard is data warehouse agnostic and works on both v1 and v2. #### Supported AI providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). #### Install and set up dbt Wizard 12345 View all stepsNext 1 Install the dbt Wizard CLI Run the install script for your operating system: macOS/Linux: ```bash curl -fsSL https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.sh | sh ``` Windows (PowerShell): ```powershell irm https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.ps1 | iex ``` This installs dbt Wizard to `/usr/local/bin/wizard`, along with the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) that powers dbt Wizard's project-aware answers. For install and update details, refer to [Install dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md); to remove them, refer to [Uninstall](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md#uninstall). Best practices for using dbt Wizard Once you're set up, refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows on real project tasks. #### Useful terminal commands Use the following commands to get started: | Command | Description | Example | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | | `wizard "[prompt]"` | Start an interactive session seeded with a prompt. Once you activate the session, you don't need to pass your prompt in quotes. | `wizard "summarize what this project does"` | | `wizard exec "[prompt]"` | Run a single prompt non-interactively and exit | `wizard exec "list all models with no tests"` | | `wizard review --uncommitted` | Non-interactive code review of uncommitted changes | `wizard review --uncommitted` | | `wizard review --base BRANCH` | Review diff against a base branch | `wizard review --base main` | | `wizard resume` | Resume a previous session | `wizard resume --last` | | `wizard apply` | Apply the latest Wizard diff to your working directory | `wizard apply TASK_ID` | | `wizard login` / `logout` | Authenticate with your dbt account | `wizard login` | | `wizard mcp` | Manage MCP server connections | `wizard mcp add dbt` | | `wizard update` | Update Wizard to the latest version | `wizard update` | Need to re-run setup? If you want to re-run onboarding — re-authenticate, reset project config, or retrigger the trusted folder prompt — refer to [Re-trigger onboarding flows](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#re-trigger-onboarding-flows). #### Next steps * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) for realistic analytics engineering scenarios * [Install and update reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) for full install, update, and uninstall details * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) for managing your API key and choosing an AI model * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows * [Use skills locally](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) for giving Wizard reusable instructions for your project * [Use MCP servers](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) to connect dbt Wizard CLI to more tools and context * [Migrate from Claude Code](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) for bringing existing Claude Code project context into dbt Wizard See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### Use MCP servers with dbt Wizard in dbt platform dbt platform | Usage-based dbt Wizard in dbt platform includes built-in dbt context. Custom MCP server connections are not yet supported in the platform experience. In dbt platform, dbt Wizard can use dbt project context, dbt metadata, and built-in dbt documentation tooling without requiring you to configure an MCP server. Refer to [Available MCP tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md?version=2.0) for more info. Although custom MCP server connections aren't supported yet for dbt Wizard in dbt platform, you can use the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) if you need custom MCP servers for development workflows. To discuss dbt platform MCP support, contact your dbt Labs account representative. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md) * [Use MCP servers with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) * [About dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) --- ### Use MCP servers with the dbt Wizard CLI Beta ### Use MCP servers with the dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development The Model Context Protocol (MCP) connects dbt Wizard to external tools and context. Add an MCP server and dbt Wizard can call its tools mid-session — query the dbt MCP server for governed project metadata, open a pull request through the GitHub MCP server, or pull in any other MCP-compatible service. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! For background on MCP itself, refer to the [Model Context Protocol introduction](https://modelcontextprotocol.io/introduction). For the dbt-maintained server specifically, refer to the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md). #### Locations and precedence MCP servers are configured under `[mcp_servers.NAME]` in `config.toml`. Use user-level config for servers you want in every local project, and project-level config for servers that should travel with a trusted repo. The following table summarizes where dbt Wizard looks for MCP server configuration. Use the intended locations for new MCP servers, and keep compatibility locations only when migrating existing setup. Project-level MCP config loads only for trusted projects. Within project-level locations, the current working directory wins over parent directories. Project-level servers win over user-level servers with the same name. | Location | Level | Use for new MCP servers? | Precedence | | --------------------------- | ------- | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `.dbt/wizard/config.toml` | Project | Yes. Use `[mcp_servers.NAME]` here for repo-shared MCP servers. | Highest project MCP config location; closer to the current working directory wins over parent directories. | | `~/.dbt/wizard/config.toml` | User | Yes. Use this for MCP servers you want across all local projects. The `wizard mcp add` command writes here. | Below project config, above compatibility imports. | | `.mcp.json` | Project | No. Compatibility import for existing MCP setup. | Imported with the project config for the same directory. | | `~/.dbt/wizard/.mcp.json` | User | No. Compatibility import for existing MCP setup. | Imported with user config; takes precedence over `~/.mcp.json` if both exist. | | `~/.mcp.json` | User | No. Compatibility import for existing MCP setup. | User-level fallback. | Avoid defining the same MCP server name in more than one location unless you intentionally want the higher-precedence location to override it. If a compatibility `.mcp.json` file and `config.toml` in the same directory define the same server name, remove the duplicate and keep the intended `config.toml` entry. #### Why use an MCP server dbt Wizard natively understands your dbt project. An MCP server extends that reach to the other tools and systems your work depends on, so you can do more without leaving your [session](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#sessions). Each server you add gives dbt Wizard a new set of tools it can call on your behalf. For example: * dbt MCP server for governed access to your models, metrics, and lineage. * GitHub server to read and review pull requests. * Data warehouse server, or another server, to pull in context that lives outside dbt. The dbt Wizard CLI lets you add, remove, authenticate, and customize MCP servers, including per-tool approvals, through the `config.toml` file. MCP servers are a CLI feature You can configure MCP servers only in the dbt Wizard CLI. You can't add your own MCP servers in dbt platform (Studio IDE and the home app), but dbt Wizard includes built-in dbt tools, such as [dbt Agent skills](https://github.com/dbt-labs/dbt-agent-skills) and product documentation fetching through the dbt MCP server. #### Supported MCP server types dbt Wizard supports two transports: | Type | Description | | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | STDIO server (standard input/output, a server that runs as a program on your own computer, instead of one you connect to over the internet) | Runs as a local process that dbt Wizard starts with a command (for example, `npx` or `uvx`). Supports environment variables. | | Streamable HTTP server | A server you reach at a URL. Supports bearer-token and OAuth authentication. | Server instructions For either transport, dbt Wizard reads the `instructions` field the server returns during initialization and uses it as cross-tool guidance. #### Add an MCP server Use the `wizard mcp add` command, or edit `~/.dbt/wizard/config.toml` directly. Either one writes user-level `[mcp_servers.NAME]` configuration. ##### Add a STDIO server ```bash wizard mcp add SERVER_NAME --env VAR1=value1 -- COMMAND ARGS ``` Where: * `SERVER_NAME` is a name you choose for the server (for example, `filesystem`). * `--env VAR1=value1` is optional and repeatable, and applies only to STDIO servers. * Everything after `--` is the command dbt Wizard runs to launch the server, so the space after `--` is intentional. For example, add a filesystem MCP server that runs locally through `npx`. This server needs no environment variables, so omit `--env`: ```bash wizard mcp add filesystem -- npx -y @modelcontextprotocol/server-filesystem /Users/you/my-project ``` To connect the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md), use the streamable HTTP form below — refer to [dbt MCP server](#dbt-mcp-server) under Examples. ##### Add a streamable HTTP server ```bash wizard mcp add SERVER_NAME --url https://example.com/mcp --bearer-token-env-var MY_TOKEN ``` To see all MCP subcommands, run `wizard mcp --help`. For the full list of flags, refer to the [CLI command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md). ##### Edit config.toml directly Instead of the `wizard mcp add` command, you can edit `config.toml` yourself. dbt Wizard stores user-level MCP configuration in `~/.dbt/wizard/config.toml` alongside its other settings: \~/.dbt/wizard/config.toml ```toml # STDIO server (runs locally) [mcp_servers.filesystem] command = "npx" args = ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/my-project"] # Streamable HTTP server [mcp_servers.github] url = "https://api.githubcopilot.com/mcp/" bearer_token_env_var = "GITHUB_MCP_TOKEN" http_headers = { "X-Region" = "us-east-1" } ``` Restart `wizard` after editing `config.toml` — MCP servers are loaded at session start. For how settings resolve, refer to [Config precedence](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#config-precedence). #### Configuration keys These keys can be set under an `[mcp_servers.NAME]` block in `config.toml`. | Key | Applies to | Description | | ----------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `command` | STDIO | The command that launches the server (for example, `uvx` or `npx`). | | `args` | STDIO | Array of arguments passed to `command`. | | `env` | STDIO | Table of environment variables set when launching the server. | | `env_vars` | STDIO | Names of existing environment variables to pass through to the server. | | `url` | HTTP | The server endpoint for a streamable HTTP server. | | `bearer_token_env_var` | HTTP | Name of the environment variable to read a bearer token from. Sent as `Authorization: Bearer TOKEN`. | | `http_headers` | HTTP | Table of static HTTP headers to send with each request. | | `env_http_headers` | HTTP | HTTP headers whose values are read from environment variables. | | `scopes` | HTTP | Array of OAuth scopes to request during login. Overrides the scopes the server advertises. | | `oauth.client_id` | HTTP | OAuth client identifier presented during authorization and token exchange. | | `oauth_resource` | HTTP | OAuth resource parameter to include in login requests ([RFC 8707](https://datatracker.ietf.org/doc/html/rfc8707)). | | `enabled` | Both | Whether the server is active. Defaults to `true`. | | `required` | Both | When `true`, `wizard exec` errors if this server fails to initialize. | | `enabled_tools` | Both | Allowlist of tool names to expose from the server. | | `disabled_tools` | Both | Blocklist of tool names to hide from the server. | | `default_tools_approval_mode` | Both | Default approval mode for this server's tools: `prompt` (ask before each call) or `auto`/`approve` (run without asking, which behave the same). | | `startup_timeout_sec` | Both | How long to wait for the server to start and list its tools. | | `tool_timeout_sec` | Both | How long to wait for an individual tool call. | Set per-tool approvals with a `[mcp_servers.NAME.tools.TOOL_NAME]` block and an `approval_mode` of `auto`, `prompt`, or `approve`: ```toml [mcp_servers.github.tools.create_pull_request] approval_mode = "approve" ``` #### Authenticate a server If a streamable HTTP server uses OAuth, you must authenticate from the CLI before dbt Wizard can use it. Run: ```bash wizard mcp login SERVER_NAME wizard mcp logout SERVER_NAME ``` To request specific scopes at login, pass the `--scopes` CLI flag with a comma-separated list. This requests the same scopes as the `scopes` key in `config.toml`, but only for that login: ```bash wizard mcp login SERVER_NAME --scopes read,write ``` For servers that use a static token, set `bearer_token_env_var` to the name of an environment variable holding the token, and export that variable before starting `wizard`. #### Manage MCP servers (CLI) Manage your configured servers through the dbt Wizard CLI, or by editing `config.toml` directly. The CLI provides these commands: | Command | What it does | | ------------------------- | ---------------------------------------------------------------------- | | `wizard mcp list` | List configured MCP servers. Add `--json` for machine-readable output. | | `wizard mcp get NAME` | Show the configuration for one server. | | `wizard mcp add NAME ...` | Add a STDIO or streamable HTTP server. | | `wizard mcp remove NAME` | Remove a server's configuration. | | `wizard mcp login NAME` | Authenticate with an OAuth server. | | `wizard mcp logout NAME` | Sign out of an OAuth server. | #### Approvals and tool permissions MCP tool calls follow the same [approval and sandboxing](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing) rules as the rest of dbt Wizard. Set `enabled_tools` and `disabled_tools` in `config.toml` to control which tools a server exposes (there's no dedicated CLI flag). That way, dbt Wizard calls only the tools you intend. #### Examples The following examples show common scenarios for adding an MCP server and how to configure them. ##### dbt MCP server The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) gives dbt Wizard governed access to your project's models, metrics, lineage, freshness, and platform APIs. You can connect it two ways: ##### Self-hosted (no account required) Runs on your machine through `uvx` and works with or without a dbt platform account — the best fit for development: ```bash wizard mcp add dbt -- uvx dbt-mcp ``` The self-hosted server reads its connection settings (such as `DBT_HOST`, `DBT_TOKEN`, and `DBT_PROJECT_DIR`) from environment variables, typically a `.env` file in your dbt project root. You don't need a URL. For setup, refer to [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) and [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md). ##### Remote (dbt platform account) Hosted on dbt platform. Build the URL from your platform host (`https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/`, for example `https://cloud.getdbt.com/api/ai/v1/mcp/`), then authenticate: ```bash wizard mcp add dbt --url https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/ wizard mcp login dbt ``` For finding your host and token, refer to [Connect to the remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md) and [Connections and authentication (MCP)](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#connections-and-authentication-mcp). Then prompt dbt Wizard: ```text Use the dbt MCP server to find the most recent failed run for the nightly job and summarize the error. ``` ##### Fivetran Agent Context MCP server Connect dbt Wizard to the [Fivetran Agent Context MCP server](https://fivetran.com/docs/context-layer/agent-context-mcp) which gives Wizard access to your organization's Fivetran context layer, so it can answer questions using your business definitions, metrics, and operational data. Add it with the following command: ```bash wizard mcp add AgentContext \ --url "https://api.fivetran.ai/mcp" ``` Wizard then prompts you to authorize the connection. Follow the prompts to finish connecting. If your organization uses system keys, set your API key as an environment variable first, then include it in the command: ```bash export FIVETRAN_AI_API_KEY=YOUR_API_KEY wizard mcp add AgentContext --url "https://api.fivetran.ai/mcp" --bearer-token-env-var FIVETRAN_AI_API_KEY ``` ##### GitHub MCP server for pull request review Connect a GitHub MCP server so dbt Wizard can read a pull request and post review comments: \~/.dbt/wizard/config.toml ```toml [mcp_servers.github] url = "https://api.githubcopilot.com/mcp/" # This is the NAME of an environment variable, not the token itself. # Keep your real token out of this file. bearer_token_env_var = "GITHUB_MCP_TOKEN" ``` Then set that environment variable to your actual token before starting dbt Wizard: ```bash export GITHUB_MCP_TOKEN="your-real-token-here" ``` At runtime, dbt Wizard reads the token from the environment and sends it as `Authorization: Bearer <the token>`. Store only the variable name in `config.toml` to keep the secret out of your committed config. ```text Review the dbt model changes in PR #482 — check for missing tests on new columns and confirm downstream refs still resolve. ``` #### Related docs * [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md): The dbt-maintained server and its available tools * [Use subagents with dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md): Delegate work to specialized agents * [dbt Wizard CLI config](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md): `config.toml` keys and precedence * [dbt Wizard CLI command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md): `wizard mcp` flags and subcommands * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md): Approvals and sandboxing --- ### Use skills with dbt Wizard CLI Beta ### Use skills with dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Skills are reusable instructions that help dbt Wizard follow your team's SQL conventions, naming rules, modeling patterns, and workflows without repeating them in every prompt. dbt Wizard also includes built-in skills from [dbt Agent skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs. These are always available, no setup needed. When you initialize dbt Wizard in a project, it can also detect existing Claude Code skills in the current directory. For general project context (`AGENTS.md`, `CLAUDE.md`), refer to [Migrate to dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) — dbt Wizard reads those instruction files out of the box. #### Locations and precedence At the start of every session, dbt Wizard scans for project-level skills in `.agents/skills/` from your dbt project root down to your current working directory: ```text .agents/ skills/ your-skill-name/ SKILL.md ``` The CLI also discovers user-level skills in `~/.agents/skills/`, user-level Claude Code skills under `~/.claude/skills/`, and project-level Claude Code skills under `.claude/skills/`. ```text .claude/ skills/ your-skill-name/ SKILL.md ``` Custom skills use the [Agent Skills](https://agentskills.io/specification) format, the same format used by [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills) on GitHub. A skill file contains YAML frontmatter (`name`, `description`) and a Markdown body with instructions. The following table summarizes where dbt Wizard looks for skills. Use the intended locations for new skills, and keep compatibility locations only when migrating existing setup. Within project-level locations, the current working directory wins over parent directories. Project-level skills win over user-level and built-in skills. | Location | Level | Use for new skills? | Precedence | | ------------------------------------------------------------------------- | -------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `.agents/skills/NAME/SKILL.md` | Project | Yes. Use this for skills shared with a repo or subdirectory. | Highest project skill location; closer to the current working directory wins over parent directories. | | `~/.agents/skills/NAME/SKILL.md` | User | Yes. Use this for skills you want across all local projects. | Below project skills, above compatibility imports and built-in skills. | | `.claude/skills/NAME/SKILL.md` | Project | No. Compatibility import for existing Claude Code skills. | Below intended project and user skill locations. | | `~/.claude/skills/NAME/SKILL.md` | User | No. Compatibility import for existing Claude Code skills. | Below intended project and user skill locations. | | Built-in [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills) | Built-in | No. Shipped with dbt Wizard. | Lowest. | Avoid duplicate skill names across locations. If you need to replace a built-in or imported skill, create the replacement in an intended project or user location and remove or rename the older copy. #### Skills from dbt packages (Applies to dbt v2.0 and later) On dbt v2, a dbt package can ship skills, and `dbt deps` installs them into the directory your coding agent reads from. To opt in, set `ai_provider` to one of the [supported providers list](https://docs.getdbt.com/docs/dbt-ai/package-skills.md?version=2#set-the-ai_provider-flag). If you use Wizard, set `ai_provider` to `wizard` in your root project: dbt\_project.yml ```yml flags: ai_provider: wizard ``` Use `wizard` whether you bring your own key or use managed AI. For the full setup, including how to ship skills in a package and disable ones you don't want, examples, and more — refer to [Installing agent skills from dbt packages](https://docs.getdbt.com/docs/dbt-ai/package-skills.md). #### Create a skill From your terminal, use the following commands to create a skill folder and file: ```bash mkdir -p .agents/skills/my-team-style touch .agents/skills/my-team-style/SKILL.md ``` Then write your skill file. Refer to [Skill file format](#skill-file-format) in the next section for more information. Start a new `wizard` session to apply the changes. Skills are discovered at session start and changes made mid-session won't be picked up until you restart. #### Skill file format A skill file has two required parts: YAML frontmatter and a Markdown body. * **Frontmatter**: Includes at minimum, `name` and `description`. The `description` is especially important: it's how dbt Wizard decides when to apply the skill automatically. * **Body**: Markdown sections explaining when to use the skill, what it does, and any conventions or constraints. * **References** (optional): Supporting `.md` files in a `references/` subfolder for detail that would make `SKILL.md` too long. ##### Example skill file If you're new to skills, start with a small `SKILL.md` like the following example, then grow it over time: ```markdown --- name: my-team-style description: Apply Santi Corp's modeling conventions when editing or creating dbt models. Use when the user asks for refactors, new models, or YAML in this project. --- # My team style ## When to use Use this skill whenever you are changing SQL or YAML under `models/` and the user did not override these rules. ## Conventions - Staging models use prefix `stg_` and live in `models/staging/`. - Facts and dimensions use `fct_` and `dim_` prefixes respectively. - Document new columns in the same PR as the model change. ## Optional detail For edge cases, read `references/naming-conventions.md` in this skill folder before proposing renames. ``` For a full production-style example, refer to dbt's [`adding-dbt-unit-test` skill](https://github.com/dbt-labs/dbt-agent-skills/blob/main/skills/dbt/skills/adding-dbt-unit-test/SKILL.md). #### Folder layout ```text .agents/ skills/ my-team-style/ SKILL.md ← required references/ ← optional: extra detail Wizard loads when needed naming-conventions.md sql-patterns.md ``` The `references/` subfolder is for supporting material that would make `SKILL.md` too long. dbt Wizard reads reference files when the skill is active and the task is relevant. #### Use a skill dbt Wizard applies skills automatically when it detects a match between the skill's `description` and your task. You can also invoke a skill explicitly: Reference the skill by name in your prompt: ```text Use my-team-style to refactor fct_orders. ``` Or point to the file directly with `@`: ```text @.agents/skills/my-team-style/SKILL.md — apply these conventions to the new model. ``` #### Global skills (CLI only) If you use the CLI, you can store skills in `~/.agents/skills/` to make them available across all your dbt projects. This is useful for personal conventions you want everywhere: \~/.agents/skills/ ```text personal-defaults/ SKILL.md ``` If a project skill and a global skill use the same name, the project skill takes precedence. #### Built-in dbt skills dbt Wizard ships with skills from [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs and the community. These capture analytics engineering knowledge for common workflows and are always available. The agent loads the relevant skill automatically when your prompt matches its use case. The following skills all ship bundled with dbt Wizard. You don't call them directly as the agent loads whichever one best matches your prompt. Refer to the [dbt Agent Skills repository](https://github.com/dbt-labs/dbt-agent-skills) for installation in other AI clients, contributing new skills, and the latest catalog. | Skill | What it does | | ----------------------------------------------- | ---------------------------------------------------------------------------- | | `using-dbt-for-analytics-engineering` | Build and modify dbt models, debug errors, explore data sources, write tests | | `adding-dbt-unit-test` | Add unit tests for dbt models; practice test-driven development | | `building-dbt-semantic-layer` | Create semantic models, metrics, and dimensions with MetricFlow | | `answering-natural-language-questions-with-dbt` | Answer business questions by querying the semantic layer | | `working-with-dbt-mesh` | Implement dbt Mesh governance and cross-project collaboration | | `troubleshooting-dbt-job-errors` | Diagnose and resolve dbt platform job failures | | `configuring-dbt-mcp-server` | Set up the dbt MCP server for Claude, Cursor, or VS Code | | `fetching-dbt-docs` | Look up dbt documentation efficiently | | `running-dbt-commands` | Run dbt CLI commands with correct flags, selectors, and parameter formats | ##### Migration (one-off use) These skills are for migration projects rather than everyday sessions: | Skill | What it does | | ---------------------------------------- | ------------------------------------------ | | `migrating-dbt-core-to-fusion` | Migrate dbt projects from dbt v1 to dbt v2 | | `migrating-dbt-project-across-platforms` | Migrate dbt projects across data platforms | Built-in skills are updated with each dbt Wizard release. Custom skills with the same name take precedence. #### Tips * **Keep `SKILL.md` focused.** One skill per concern (style guide, testing conventions, deployment workflow). Smaller skills are loaded more reliably than large monolithic ones. * **Use `description` to control when dbt Wizard activates the skill.** A precise description ("Use when creating or editing models in models/marts/") means the skill fires when relevant, not on every prompt. * **Start a new session after adding or editing a skill.** Skills are discovered at session start — mid-session changes aren't picked up until you start a new chat. * **Share skills across projects**: To share skills across projects in v2, [ship them in a dbt package](https://docs.getdbt.com/docs/dbt-ai/package-skills.md?version=2) and install it with `dbt deps`. Refer to [Skills from dbt packages](https://docs.getdbt.com/docs/dbt-ai/package-skills.md?version=2). For personal skills you want across all your local projects, use ~/.agents/skills/. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Use skills in the dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-skills.md) * [Installing agent skills from dbt packages](https://docs.getdbt.com/docs/dbt-ai/package-skills.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [dbt Wizard CLI overview](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) --- ### Use skills with dbt Wizard in dbt platform Beta ### Use skills with dbt Wizard in dbt platform [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based Skills are reusable instructions that help dbt Wizard follow your team's SQL conventions, naming rules, modeling patterns, and workflows without repeating them in every prompt. dbt Wizard also includes built-in skills from [dbt Agent skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs. These are always available, no setup needed. For general project context (`AGENTS.md`, `CLAUDE.md`), refer to [Migrate to dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) — dbt Wizard reads those instruction files out of the box. #### How skills work At the start of every session, dbt Wizard scans for repo-level skills in the `.agents/skills/` directory in your dbt project root: ```text .agents/ skills/ your-skill-name/ SKILL.md ``` Custom skills use the [Agent Skills](https://agentskills.io/specification) format, the same format used by [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills) on GitHub. A skill file contains YAML frontmatter (`name`, `description`) and a Markdown body with instructions. If a custom skill and a built-in skill share the same name, the custom skill takes precedence. #### Create a skill In [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md), use the file explorer to create the skill directory and file: 1. Create a folder at `.agents/skills/SKILL_NAME/` in your project root. 2. Add a `SKILL.md` file inside it. A typical `SKILL.md` includes: * YAML frontmatter with at minimum `name` and `description`. * A Markdown body with sections for when to use the skill, workflow steps, and conventions. * Optional `references/` files for extra detail the agent can load when needed. 3. Optionally add supporting files under `.agents/skills/SKILL_NAME/references/` (for example, `references/naming-conventions.md`) that the agent can read when the skill is active. 4. Start a new dbt Wizard chat — skills are discovered at session start, so mid-session changes won't be picked up until you start a new chat. #### Skill file format A skill file has two required parts: YAML frontmatter and a Markdown body. * **Frontmatter**: Includes at minimum, `name` and `description`. The `description` is especially important: it's how dbt Wizard decides when to apply the skill automatically. * **Body**: Markdown sections explaining when to use the skill, what it does, and any conventions or constraints. * **References** (optional): Supporting `.md` files in a `references/` subfolder for detail that would make `SKILL.md` too long. ##### Example skill file If you're new to skills, start with a small `SKILL.md` like the following example, then grow it over time: ```markdown --- name: my-team-style description: Apply Santi Corp's modeling conventions when editing or creating dbt models. Use when the user asks for refactors, new models, or YAML in this project. --- # My team style ## When to use Use this skill whenever you are changing SQL or YAML under `models/` and the user did not override these rules. ## Conventions - Staging models use prefix `stg_` and live in `models/staging/`. - Facts and dimensions use `fct_` and `dim_` prefixes respectively. - Document new columns in the same PR as the model change. ``` For a full production-style example, refer to dbt's [`adding-dbt-unit-test` skill](https://github.com/dbt-labs/dbt-agent-skills/blob/main/skills/dbt/skills/adding-dbt-unit-test/SKILL.md). #### Use a skill In the dbt Wizard chat panel: 1. Prompt the agent and reference the skill by name — for example: `Use my-team-style to refactor this model and update related YAML`. 2. If needed, use `@` mentions to point to the skill file or supporting `.md` files directly. 3. Review and approve proposed changes as usual in the dbt Wizard panel. #### Built-in dbt skills dbt Wizard ships with skills from [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs and the community. These capture analytics engineering knowledge for common workflows and all ship bundled with dbt Wizard. You don't call them directly as the agent loads whichever one matches your prompt. For the latest catalog, refer to the [dbt Agent Skills repository](https://github.com/dbt-labs/dbt-agent-skills). #### Tips * **Keep `SKILL.md` focused.** One skill per concern (style guide, testing conventions, deployment workflow). Smaller skills are loaded more reliably than large monolithic ones. * **Use `description` to control when dbt Wizard activates the skill.** A precise description ("Use when creating or editing models in models/marts/") means the skill fires when relevant, not on every prompt. * **Start a new session after adding or editing a skill.** Skills are discovered at session start — mid-session changes aren't picked up until you start a new chat. * **Cross-project sharing isn't supported yet.** To reuse a skill in another repo, copy the skill files manually. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Use skills locally](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) --- ### Use subagents with dbt Wizard CLI Beta ### Use subagents with dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Subagents let dbt Wizard spin up focused, parallel agents to handle parts of a larger task — one to explore your project, one to make changes, one to review them. dbt Wizard orchestrates them and consolidates the results back into your session. Use subagents when you can split a task into independent pieces of work. They're useful for reviewing a large pull request, debugging a failed job, adding tests across multiple models, or researching documentation while another agent inspects your project. For smaller, direct tasks, you usually don't need subagents. Ask a single question, like "What does this model do?" or "Fix this failing test," without splitting the work across agents. Subagents run work in parallel, which uses more tokens than handling the same task in a single-agent session. Use them when the task benefits from dividing the work. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Locations and precedence You can define custom agent roles by adding standalone TOML files at the project level or user level. Use project-level agents for roles that should travel with a repo, and user-level agents for roles you want across all local projects. The following table summarizes where dbt Wizard looks for custom agents. Use the intended locations for new agents, and keep compatibility locations only when migrating existing setup. Project-level agent config loads only for trusted projects. Within project-level locations, the current working directory wins over parent directories. Project-level agents win over user-level agents with the same name. | Location | Level | Use for new agents? | Precedence | | -------------------------------- | ------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `.dbt/wizard/agents/NAME.toml` | Project | Yes. Use this for agents shared with a repo or subdirectory. | Highest project agent location; closer to the current working directory wins over parent directories. | | `~/.dbt/wizard/agents/NAME.toml` | User | Yes. Use this for agents you want across all local projects. | Below project agents, above compatibility imports. | | `.claude/agents/NAME.md` | Project | No. Compatibility import for existing Claude Code agents. | Below `.dbt/wizard/agents/` in the same project location. | | `~/.claude/agents/NAME.md` | User | No. Compatibility import for existing Claude Code agents. | Below `~/.dbt/wizard/agents/`. | If two custom agents use the same name, dbt Wizard uses the higher-precedence location and fills in any missing fields from lower-precedence locations when possible. Avoid duplicate names unless you intentionally want to override an existing role. #### Where you can use subagents Subagents work in the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md). You can define custom agent roles, set display nicknames, and configure global limits through the `config.toml` file. #### How subagents work An agent is a role that describes a type of work, like `explorer`, `worker`, or `test_writer`. A subagent is a running instance of one of those roles. For example, if you start two explorer agents to answer two different questions, you have two subagents that share the same agent role. dbt Wizard handles orchestration for you. It starts subagents, routes work to them, waits for their results, and consolidates their output into your session. You can use subagents in the following ways: | How subagents start | Example | | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | You ask to split up the work. | `Review PR #123. Use one agent to map what changed, one to check tests and downstream impact, and one to look up the relevant dbt docs.` | | dbt Wizard automatically uses a configured agent because your task matches that agent's description. | `Add tests for stg_customers and check whether similar staging models are missing tests.` | dbt Wizard then spawns the agents, lets them work in parallel, and brings their findings back into one response. #### Built-in agents dbt Wizard ships with built-in agents that it can spawn without any configuration. Several are purpose-built for dbt work: | Agent | What it's for | Example | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | `explorer` | Answers specific, well-scoped questions about your project. Fast and read-only — spawn several in parallel for independent questions. | `Use explorer to explain what depends on fct_orders.` | | `worker` | Performs execution and production work, such as implementing part of a feature, fixing tests or bugs, or splitting a large refactor into independent chunks. | `Use worker to update these staging models to follow our naming convention.` | | `validation` | Provides dbt validation. After model edits, runs structured validation — SQL check, `dbt run` with `--defer`, prod vs. dev comparison, and impact analysis — to validate changes before you merge. | `Use validation to check whether my changes to int_payments are safe to merge.` | | `test_writer` | Improves dbt test coverage. Analyzes project metadata and warehouse data to find coverage gaps, validates assumptions with queries, and writes `schema.yml` tests for models with low or no coverage. | `Use test_writer to add tests to stg_customers.` | You don't need to declare these — dbt Wizard routes to them automatically when a task fits, or you can ask for one by name. For example, both of these prompts can use the `test_writer` agent: | Prompt style | Example | | ------------------------- | ------------------------------------------------ | | Ask naturally | Add useful tests for stg\_customers. | | Ask for the agent by name | Use test\_writer to add tests to stg\_customers. | To add your own roles, refer to [Custom agents](#custom-agents-cli). #### Manage subagents Use slash commands inside an interactive session to inspect and steer agent threads: | Command | What it does | Example | | ------------ | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `/agent` | Switch the active agent thread to inspect or steer ongoing work. | Use `/agent` when you want to jump from the main session into a specific agent thread to give it more direction. | | `/subagents` | View and switch between running subagent threads. | Use `/subagents` to see which helper agents are active during a delegated task, then select one to inspect its progress. | You can also tell dbt Wizard in plain language to steer, stop, or close an agent thread. For example, `Stop the docs researcher agent and continue with the reviewer findings.` For the full list of session commands, refer to the [slash command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-slash-commands.md). #### Approvals and sandbox Subagents inherit the parent session's [approval and sandbox policy](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing). Any runtime overrides you set for a turn (permissions, sandbox mode) apply to the subagents it spawns. When a subagent requests approval, your session shows the request with its source so you know which agent is asking. A custom agent can override sandbox settings for itself — useful when, for example, an exploration agent should stay read-only while a build agent needs workspace write access. #### Custom agents A custom agent role is a reusable role for a particular type of work you want dbt Wizard to perform, such as writing UDFs, exploring a project, or debugging an issue. You can create any role name that fits your workflow; it does not have to be one of the built-in roles. Each file defines one custom agent role. The file name must match the agent's `name` field. For example, an agent with `name = "udf_helper"` must be defined in an `agents/udf_helper.toml` file. dbt Wizard loads each custom agent file as a configuration layer for spawned agent sessions. This means a custom agent can override the same settings as a normal dbt Wizard session config, such as the model, instructions, sandbox mode, and MCP servers. The best custom agents are narrow and opinionated. Give each one a clear job, the tools it needs for that job, and instructions that keep it from drifting into adjacent work. Every custom agent file must define: | Field | Required | Description | | ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Yes | Agent name dbt Wizard uses when spawning or referring to this agent. Must match the file name without `.toml`. For example, `name = "udf_helper"` must be in `udf_helper.toml`. | | `description` | Yes | Explains when to use the agent. dbt Wizard reads this description to decide whether the agent fits a task. | | `developer_instructions` | Yes | Self-hosted dbt instructions that define the agent's behavior. | | `nickname_candidates` | No | Display-only labels for spawned instances of this agent in the UI, such as `UDF helper` or `UDF queen`. The nickname does not identify the agent. | | `model` | No | Model this agent should use. Inherits from the parent session when omitted. | | `sandbox_mode` | No | Sandbox mode for this agent. Inherits from the parent session when omitted. | | `mcp_servers` | No | MCP servers available to this agent. Inherits from the parent session when omitted. | You can also include other supported [`config.toml`](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#configtoml) keys in a custom agent file. Any setting you don't define in the custom agent file inherits from the parent session. ##### Example custom agent Create a standalone custom agent file. The file name and `name` value must match: \~/.dbt/wizard/agents/udf\_helper.toml ```toml name = "udf_helper" # matches the udf_helper.toml file name description = "Helps design and implement dbt UDFs and models that use them." developer_instructions = """ You are an expert in dbt UDFs. Focus on: - choosing the right UDF pattern - writing clear function definitions - configuring function paths - using {{ function() }} references correctly - adding focused models and tests that demonstrate the UDF - keeping changes small and additive Before editing files, explain the proposed UDF design. When editing files, avoid unrelated changes. """ sandbox_mode = "workspace-write" nickname_candidates = ["UDF helper", "UDF queen"] ``` Then ask dbt Wizard to use the agent by name: ```text Use udf_helper to create a UDF model that checks whether customer IDs are positive integers. ``` You can also let dbt Wizard choose the agent automatically by describing the task: ```text Create a dbt UDF that checks whether customer IDs are positive integers, then add a small model that uses it. ``` ##### Display nicknames When several instances of the same agent run at once, use `nickname_candidates` to give each instance a readable label in the UI. For example, two spawned `udf_helper` agents might appear as `UDF helper` and `UDF queen`. Nicknames are display-only. dbt Wizard identifies the agent by its `name` field, not by the nickname shown in the UI. ##### Example with an MCP server Custom agent files can include MCP server configuration when the agent needs additional tools or context. Again, the file name and `name` value must match: \~/.dbt/wizard/agents/dbt\_docs\_helper.toml ```toml name = "dbt_docs_helper" description = "Looks up relevant dbt documentation for patterns referenced in a change." developer_instructions = """ Use the dbt docs MCP server to confirm APIs, options, and version-specific behavior. Return concise answers with links or exact references when available. Do not edit files. """ sandbox_mode = "read-only" [mcp_servers.dbt] command = "uvx" args = ["dbt-mcp"] ``` This setup works well for prompts like: ```text Have udf_helper design the UDF change and dbt_docs_helper verify the dbt function configuration. ``` ##### Global settings Global subagent settings live under the `[agents]` section in `~/.dbt/wizard/config.toml`: \~/.dbt/wizard/config.toml ```toml [agents] max_depth = 2 job_max_runtime_seconds = 1800 interrupt_message = true ``` | Key | Description | | ------------------------- | ------------------------------------------------------------------------------------------------ | | `max_depth` | Maximum nesting depth for spawned agent threads. Root sessions start at depth 0. | | `job_max_runtime_seconds` | Default maximum runtime, in seconds, for agent job workers. | | `interrupt_message` | Whether to record a model-visible message when an agent turn is interrupted. Defaults to `true`. | #### Examples ##### UDF implementation Define a custom agent that specializes in dbt UDF work. Each custom agent is a standalone TOML file under `~/.dbt/wizard/agents/`. \~/.dbt/wizard/agents/udf\_helper.toml ```toml name = "udf_helper" description = "Helps design and implement dbt UDFs and models that use them." developer_instructions = """ You are an expert in dbt UDFs. Focus on: - choosing the right UDF pattern - writing clear function definitions - configuring function paths - using {{ function() }} references correctly - adding focused models and tests that demonstrate the UDF - keeping changes small and additive Before editing files, explain the proposed UDF design. When editing files, avoid unrelated changes. """ sandbox_mode = "workspace-write" nickname_candidates = ["UDF helper", "UDF queen"] ``` Optionally, configure global subagent settings in your main `config.toml`: \~/.dbt/wizard/config.toml ```toml [agents] max_depth = 2 ``` Then prompt dbt Wizard: ```text Use udf_helper to create a UDF that checks whether customer IDs are positive integers, then add a small model that uses it. ``` dbt Wizard starts the requested agent, lets it work on the UDF implementation, and consolidates the result into your session. ##### Debug a failed job run (home app) In the dbt platform home app, ask dbt Wizard to investigate a failed job by delegating to focused agents: ```text The nightly job failed. Use one agent to pull the run error and logs, one to trace the failing model's lineage and find the root cause, and one to propose a fix. Summarize what each found. ``` dbt Wizard spawns the agents, each one works on its part of your connected project, and dbt Wizard consolidates the diagnosis and proposed fix into one response. #### Related docs * [Use MCP servers with the dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) to give agents more tools and context * [Use skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) for reusable, project-specific instructions * [dbt Wizard slash commands](https://docs.getdbt.com/docs/dbt-ai/wizard-slash-commands.md) for `/agent` and `/subagents` * [dbt Wizard CLI config](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) for `config.toml` keys and precedence * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) for approvals and sandboxing --- ### Use subagents with dbt Wizard in dbt platform Beta ### Use subagents with dbt Wizard in dbt platform [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based Subagents let dbt Wizard spin up focused agents to handle parts of a larger task, such as exploring your project, making changes, or reviewing results. dbt Wizard orchestrates them and consolidates the results back into your session. Use subagents when you can split a task into independent pieces of work. They're useful for reviewing a large pull request, debugging a failed job, adding tests across multiple models, or researching documentation while another agent inspects your project. For smaller, direct tasks, you usually don't need subagents. Ask a single question, like "What does this model do?" or "Fix this failing test," without splitting the work across agents. Subagents use more tokens than handling the same task in a single-agent session. Use them when the task benefits from dividing the work. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### How subagents work An agent is a role that describes a type of work, like `explorer`, `worker`, or `test_writer`. A subagent is a running instance of one of those roles. For example, if you start two explorer agents to answer two different questions, you have two subagents that share the same agent role. dbt Wizard handles orchestration for you. It starts subagents, routes work to them, waits for their results, and consolidates their output into your session. You can use subagents in the following ways: | How subagents start | Example | | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | You ask to split up the work. | `Review PR #123. Use one agent to map what changed, one to check tests and downstream impact, and one to look up the relevant dbt docs.` | | dbt Wizard automatically uses a configured agent because your task matches that agent's description. | `Add tests for stg_customers and check whether similar staging models are missing tests.` | #### Built-in agents dbt Wizard ships with built-in agents that it can spawn without any configuration. Several are purpose-built for dbt work: | Agent | What it's for | Example | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | `explorer` | Answers specific, well-scoped questions about your project. Fast and read-only — spawn several in parallel for independent questions. | `Use explorer to explain what depends on fct_orders.` | | `worker` | Performs execution and production work, such as implementing part of a feature, fixing tests or bugs, or splitting a large refactor into independent chunks. | `Use worker to update these staging models to follow our naming convention.` | | `validation` | Provides dbt validation. After model edits, runs structured validation — SQL check, `dbt run` with `--defer`, prod vs. dev comparison, and impact analysis — to validate changes before you merge. | `Use validation to check whether my changes to int_payments are safe to merge.` | | `test_writer` | Improves dbt test coverage. Analyzes project metadata and warehouse data to find coverage gaps, validates assumptions with queries, and writes `schema.yml` tests for models with low or no coverage. | `Use test_writer to add tests to stg_customers.` | You don't need to declare these — dbt Wizard routes to them automatically when a task fits, or you can ask for one by name. #### Related docs * [About dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Use subagents locally](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) --- ### Use Wizard Desktop Beta ### Use Wizard Desktop [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Navigate Wizard Desktop and learn how to work in a session with dbt Wizard — with rich data visualizations, data diffs, lineage, and more rendered right in the app. Start with [Get started](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md). This page is your map of the app: where everything lives, how a session works, what to do when dbt Wizard proposes a change, and how it conjures its magic . #### Tour the app When you open Wizard Desktop, everything you need is in one window — you'll have your projects and sessions on the left, the chat in the middle, and panes on the right that show your project, your lineage, your changes, and your results as rendered charts and tables. [![A new session, with the sidebar, chat composer, and Explorer pane](/img/wizard/desktop/wizard-desktop-ui.png?v=2 "A new session, with the sidebar, chat composer, and Explorer pane")](#)A new session, with the sidebar, chat composer, and Explorer pane [![The same layout mid-task, with a chart in the chat and validation results in the side pane](/img/wizard/desktop/wizard-desktop-validation.png?v=2 "The same layout mid-task, with a chart in the chat and validation results in the side pane")](#)The same layout mid-task, with a chart in the chat and validation results in the side pane | What you see | What it's for | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Sidebar (left) | **+ New** starts a session. **Pinned** keeps the chats and environments you pin at the top, and you can reorder them. **Projects** lists your projects and the sessions in each one, **Chats** holds conversations that aren't tied to a project, and **Settings** sits at the bottom | | Session header | The session name, the branch it's working on, and **Open in** to jump to the same branch in Finder, VS Code, Cursor, iTerm, or Terminal | | Chat area (middle) | Your conversation. Replies, charts, tables, and diffs all render here | | Wayfinder bar | Above the chat composer box: your project, branch, connection · environment, plus **Validation** and **Create PR** | | Chat composer | Where you ask. Press `⌘` + `L` to jump to it. **+** attaches files, and the two pickers set your permission mode and AI model | | Side pane (right) | Whichever pane you've opened, with a terminal underneath. The Explorer is read-only, so browse your files here but edit them in your own editor | | Pane rail (far right) | Switch panes: Explorer, Lineage, Changes, Queries, Commands, Validation, Summary, Checks, Metrics, and Insights. A badge means something needs a look | To change or remove a project, hover its name in the sidebar and select the ellipsis (**⋮**) → **Project settings** or **Remove project**. [![The project row ellipsis menu with Project settings and Remove project](/img/wizard/desktop/wizard-desktop-project-settings.png?v=2 "The project row ellipsis menu with Project settings and Remove project")](#)The project row ellipsis menu with Project settings and Remove project ##### Use the panes Select an icon in the right hand pane rail to open a pane in the side panel: | Pane | What's in it | | ---------- | ------------------------------------------------------------------------------------------------------------------------------- | | Summary | A summary of the work in this chat | | Changes | Diffs waiting for review | | Validation | Your validation plan and its results — what passed, what failed, and what needs a look. Refer to [Validation](#validation) | | Explorer | Your project files. Select a file path in the chat to open it here. The Explorer is read-only, so edit files in your own editor | | Lineage | How your dbt models connect | | Queries | Write and run ad hoc SQL against your connection. Refer to [Explore your data](#explore-your-data) | | Commands | Queries you run and commands dbt Wizard executes, scoped to this chat | | Checks | The checks dbt Wizard ran and their results | | Insights | Analysis of your project and data | | Metrics | The metrics in your project | | Terminal | Run commands without leaving the app. Sits underneath whichever pane you've opened | [![Select an icon in the right hand pane rail to open a pane in the side panel](/img/wizard/desktop/wizard-desktop-right-pane.png?v=2 "Select an icon in the right hand pane rail to open a pane in the side panel")](#)Select an icon in the right hand pane rail to open a pane in the side panel Prefer your own editor? Select **Open in** to open dbt Wizard and your branch in Finder, VS Code, Cursor, iTerm, or Terminal, or copy the file path. #### Start a session Set these in the prompt area before you send your first message. You can change the AI model and permission mode mid-session; the rest are set for the chat. | Setting | What it does | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Project | The dbt project dbt Wizard works in. The app finds projects in Git repos in your home folder, and you can add a folder yourself. You can also skip adding a project — dbt Wizard can work across projects | | Connection · environment | Which warehouse connection and environment to run against, like `default · dev`. Comes from your local [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) | | Schema suffix | Gets appended onto your target schema so this chat builds into its own tables. Refer to [Schema suffix](#schema-suffix) | | AI model | The AI model and how hard it thinks. Refer to [dbt managed inference](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) for more info. | | Permission mode | How much dbt Wizard does before checking with you. Refer to [Permission modes](#permission-modes) | Type your task, hit **Enter**, and you're off. Each chat keeps its own context, branch, and changes. You don't have to pick a project to get started. Skip it and dbt Wizard can work across your projects — useful when you're asking a question that spans more than one, or you're not sure which project holds the answer yet. Project-less conversations live under **Chats** in the sidebar. ##### Permission modes Pick a mode from the selector next to the send button. Switch any time, even mid-run. | Mode | What dbt Wizard does | | ---------------- | ----------------------------------------------------------------------------- | | Plan changes | Looks around your project and tells you what it would do. Nothing gets edited | | Ask for approval | Asks first. You get a diff and decide, change by change | | Auto | Just does it. Edits land and you review them after | Start big jobs in **Plan changes** so you can sanity-check the approach, then switch to **Auto** to let it run. ##### Schema suffix Each chat builds into its own schema, so two chats (or you and a chat) never overwrite each other's tables. The suffix is what makes that schema unique — your target schema plus the suffix. #### Work in a session This section goes over what you can do when you're working in a chat session. ##### Prompt Wizard * Type your prompt and hit **Enter**. dbt Wizard reads your files, lineage, tests, and metadata, then edits and runs things based on your permission mode. * Attach files and images, like a screenshot of the error you hit, a chart that looks wrong, or a mockup of what you want. Select **+** next to the chat composer, drag files in, paste them, or copy them from Finder. You can preview attachments before you send them, and sent files show up as readable cards in the chat. * Changed your mind halfway through? No worries, just stop the run, or just type the correction and send it — dbt Wizard then picks it up after the current step and adjusts. * When dbt Wizard mentions a dbt model, select its name to: * **Copy name** * **Add to chat** so dbt Wizard uses it as context * **Open code** to read the SQL in the Explorer * **Open lineage** to see what it's connected to [![Selecting a dbt model name in a Wizard response opens Copy name, Add to chat, Open code, and Open lineage](/img/wizard/desktop/wizard-desktop-model-menu.png?v=2 "Selecting a dbt model name in a Wizard response opens Copy name, Add to chat, Open code, and Open lineage")](#)Selecting a dbt model name in a Wizard response opens Copy name, Add to chat, Open code, and Open lineage ##### Review and validate changes When using Wizard, nothing gets committed without you seeing it: * **Changes** shows the diffs, file by file. * **Inline impact** shows which dbt models a change touches, with diffs you can open right there. * **Comments** let you hover a line, diff, table cell, or chart and say what's wrong with that exact thing. Mark up a few and send them together. Way faster than describing it in a paragraph. Once the changes look right, [validation](#validation) helps you preview the output of dbt Wizard's changes and check for problems before you open a pull request. It becomes available after dbt Wizard creates the change summary. ###### Validation Generate a validation plan to see which affected resources and checks dbt Wizard proposes to validate. Review the plan, then run the checks you approve. The results show what passed, what failed, and what needs your attention. Validation is broader than previewing or compiling: * **Preview** lets you inspect the output of a particular change. It helps you determine whether the result looks right, but doesn't check the rest of the affected project. * [**Compile**](https://docs.getdbt.com/reference/commands/compile.md) generates executable SQL from your dbt code. It can catch compilation problems, but it doesn't build the changed dbt models or compare their data. * **Validation** assesses the affected resources and creates a plan that can combine compilation, builds, tests, previews, and data comparisons. The checks depend on your changes and on the **Validation level** you select: * **Small** compiles and lints the modified dbt models, runs their schema and data tests, and reviews them for dbt anti-patterns, with no warehouse writes * **Medium** also materializes those dbt models and builds their downstream dependents * **Large** adds a comparison of prod against dev row counts, schema, and sample records. <br /> <br /> Each level lists its checks and its warehouse impact before you select **Run validation**. If the plan includes `dbt-compare`, you can use its results to understand how the changes affect your data. Expand a check to see what ran, or select **Re-run validation** after addressing a failure. When the checks pass, commit from the app and select **Create PR**. Each chat works on its own branch, so the pull request only carries that chat's work. ##### Explore your data For exploratory analysis, you can use Wizard to ask it questions about your data. For ad-hoc production SQL, [Semantic Layer](https://docs.getdbt.com/docs/build/about-metricflow.md) queries, and business-user analytics, use [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md). Sometimes you just want to poke at the data yourself: 1. Select **Queries** in the pane rail on the right. 2. Write your ad hoc SQL in the query editor. 3. Select **Run**. Your results render in **Results** underneath the editor. To run SQL that dbt Wizard wrote, select **Open in Queries** on the SQL in the chat. It opens in the same pane. The pane sits alongside your chat, so you can check something yourself without losing your place in the conversation. To start another conversation instead, select **+ New** in the left sidebar. The app uses the same config as the CLI, so the skills, subagents, and MCP servers you've already set up just work: * [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Subagents](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) * [MCP servers](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) * [Config files](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) #### Chats and worktrees Every chat on a project gets its own Git worktree. A worktree is a second copy of your repo, sitting in its own folder, on its own branch. Same repo and same history — just a separate place to work. So when a chat edits `stg_orders`, it's editing its own copy, not the files in your other tasks so you can have Wizard work on multiple bugs in the same project at the same time. That's what lets you run several chats at once. Refactor a model in one, chase a failing test in another, answer a data question in a third. Nothing collides, and you don't have to stash anything. Because each project chat has an isolated worktree, start a new chat with **+ New** for each discrete task rather than reusing one long conversation. Go back to an existing chat only when you're continuing that same task. For more info on how worktrees work, refer to [git worktree](https://git-scm.com/docs/git-worktree). ##### Worktree and schema names dbt Wizard names each chat's worktree for you by pairing a random adjective with a random animal, like `docs-fox`. The name has nothing to do with your project or your data, so you won't recognize it. The branch and schema for that chat are then built from the worktree name: | What you see | Example | Where it comes from | | ------------------------------------------ | ------------- | ------------------------------------------------------------------------ | | Worktree name, which is also the chat name | `docs-fox` | Generated when the chat starts | | Branch | `oz/docs-fox` | Your `github.user` git config value, or `feature` if you haven't set one | | Target in the worktree's `profiles.yml` | `docs-fox` | The worktree name, with hyphens swapped for underscores | | Schema | `dbt_oz_a1b2` | Your project's schema plus a random four-character suffix | [![You can find the dbt project, git worktree name, branch, target, and schema above the chat composer](/img/wizard/desktop/wizard-desktop-git-worktree.png?v=2 "You can find the dbt project, git worktree name, branch, target, and schema above the chat composer")](#)You can find the dbt project, git worktree name, branch, target, and schema above the chat composer The suffix on the schema is what keeps each chat in its own schema, so two chats never write to the same tables. dbt Wizard writes these targets to a `profiles.yml` inside the worktree folder and adds it to `.gitignore`. Your own `profiles.yml` is left alone. #### Troubleshooting  No connections found in the connection picker The picker lists targets from your local [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md). If you've only ever run this project in [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md), you probably don't have one yet. You don't need to write one by hand. A banner on the project opens a form where you pick your warehouse, enter your credentials, and let the app write the profile for you — then it runs `dbt debug` so you see the warehouse's own error instead of a silent empty picker. If you prefer to use platform-managed warehouse credentials, the same dialog walks you through connecting the project through the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) instead. Refer to [Set up your warehouse connection](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md#set-up-your-warehouse-connection).  My project isn't listed The app only scans Git repos in your home folder. If your project lives somewhere else, add the folder from **Settings**. #### Related docs * [Get started with Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) * [Wizard Desktop settings](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-settings.md) * [Use dbt Wizard locally (CLI)](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * [dbt Wizard config reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * [Data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md) See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### User-defined functions User-defined functions (UDFs) enable you to define and register custom functions in your warehouse. Like [macros](https://docs.getdbt.com/docs/build/jinja-macros.md), UDFs promote code reuse, but they are objects in the warehouse so you can reuse the same logic in tools outside dbt, such as BI tools, data science notebooks, and more. UDFs are particularly valuable for sharing logic across multiple tools, standardizing complex business calculations, improving performance for compute-intensive operations (since they're compiled and optimized by your warehouse's query engine), and version controlling custom logic within your dbt project. dbt creates, updates, and renames UDFs as part of DAG execution. The UDF is built in the warehouse before the model that references it. Refer to [listing and building UDFs](https://docs.getdbt.com/docs/build/udfs.md#listing-and-building-udfs) for more info on how to build UDFs in your project. Refer to [Function properties](https://docs.getdbt.com/reference/function-properties.md) or [Function configurations](https://docs.getdbt.com/reference/function-configs.md) for more information on the configs/properties for UDFs. #### Prerequisites * Make sure you're using dbt platform's **v2 Stable** or **v1 Latest** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or dbt v1.11+. * Use one of the following adapters: ##### dbt v1 * BigQuery * Snowflake * Redshift * Postgres * Databricks ##### dbt v2 * BigQuery * Snowflake * Redshift * Databricks UDF support JavaScript UDFs are supported in dbt v1.12+ on Snowflake and BigQuery. Additional languages (for example, Java, Scala) aren't currently supported for UDFs. See the [Limitations](#limitations) section below for the full list of currently supported UDF capabilities. #### Defining UDFs in dbt You can define SQL, Python, and JavaScript (available in dbt v1.12+) UDFs in dbt. Python UDFs are supported in Snowflake, BigQuery, and Databricks ([Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/) required). JavaScript UDFs are supported in Snowflake and BigQuery. Follow these steps to define UDFs in dbt: 1. Create a SQL, Python, or JavaScript file under the `functions` directory. For example, this UDF checks if a string represents a positive integer: ##### SQL Define a SQL UDF in a SQL file. functions/is\_positive\_int.sql ```sql # syntax for BigQuery, Snowflake, and Databricks REGEXP_INSTR(a_string, '^[0-9]+$') # syntax for Redshift and Postgres SELECT REGEXP_INSTR(a_string, '^[0-9]+$') ``` ##### Python Define a Python UDF in a Python file. functions/is\_positive\_int.py ```py import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 ``` For Databricks, the contents of the `.py` file become the UDF body verbatim, and Databricks evaluates that body directly instead of calling a named entry point. Write the body so its last statement is a top-level `return` that produces the result. Because of that top-level `return`, the Databricks source is a function *body*, not a runnable `.py` module. For example: functions/is\_positive\_int.py ```py import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 return main(a_string) ``` **Note:** Python UDFs on Databricks require [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/). ##### JavaScript Define a JavaScript UDF in a JavaScript file. functions/is\_positive\_int.js ```js return /^[0-9]+$/.test(a_string) ? 1 : 0; ``` **Note**: You can specify configs in a config block in the SQL file or in the corresponding properties YAML file in step 2. 2. Specify the function name and define the config, properties, return type, and optional arguments in a corresponding properties YAML file. ##### SQL functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int # required description: My UDF that returns 1 if a string represents a naked positive integer (like "10", "+8" is not allowed). # optional config: schema: udf_schema database: udf_db volatility: deterministic arguments: # optional - name: a_string # required if arguments is specified data_type: string # required if arguments is specified description: The string that I want to check if it's representing a positive integer (like "10") default_value: "'1'" # optional, available in Snowflake and Postgres returns: # required data_type: integer # required ``` ##### Python The following configs are required when defining a Python UDF on Snowflake and BigQuery: * [`runtime_version`](https://docs.getdbt.com/reference/resource-configs/runtime-version.md) — Specify the Python version to run. Supported values are: * [Snowflake](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-introduction): `3.10`, `3.11`, `3.12`, and `3.13` * [BigQuery](https://cloud.google.com/bigquery/docs/user-defined-functions-python): `3.11` * [`entry_point`](https://docs.getdbt.com/reference/resource-configs/entry-point.md) — Specify the Python function to be called. <br /> On Databricks, `runtime_version` and `entry_point` are accepted for cross-adapter compatibility but have no effect. Databricks manages the Python runtime internally and uses the function body directly, so dbt displays a warning if you set them. You can specify public third-party PyPI packages for your Python UDF with the optional `packages` config. List package names, such as `numpy` and `pandas`, and optionally pin versions, such as `pandas==1.5.0`. The warehouse installs these packages when it creates the UDF, so your UDF can use functionality from external Python libraries. On Snowflake, some packages are installed from the Anaconda repository, and you may need to [accept Anaconda's Terms of Service](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages#using-third-party-packages-from-anaconda) before you can use them. The following example shows a Python UDF with the required configs (`runtime_version`, `entry_point`), the optional `packages` config, and other common configs: functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int # required description: My UDF that returns 1 if a string represents a naked positive integer (like "10", "+8" is not allowed). # optional config: runtime_version: "3.11" # required for Snowflake and BigQuery; optional and ignored on Databricks entry_point: main # required for Snowflake and BigQuery; optional and ignored on Databricks packages: # optional, Python UDFs only - numpy - pandas==1.5.0 schema: udf_schema database: udf_db volatility: deterministic arguments: # optional - name: a_string # required if arguments is specified data_type: string # required if arguments is specified description: The string that I want to check if it's representing a positive integer (like "10") default_value: "'1'" # optional, available in Snowflake and Postgres returns: # required data_type: integer # required ``` ##### JavaScript You can optionally set [`snowflake.quote_args`](https://docs.getdbt.com/reference/resource-configs/quote_args.md) to control whether argument names are quoted when creating a JavaScript UDF on Snowflake. functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int # required description: My UDF that returns 1 if a string represents a naked positive integer (like "10", "+8" is not allowed). # optional config: snowflake: # optional quote_args: true # optional, JavaScript UDFs on Snowflake only arguments: # optional - name: a_string # required if arguments is specified data_type: string # required if arguments is specified description: The string to check # optional returns: # required data_type: integer # required ``` volatility warehouse-specific `volatility` is accepted in dbt for SQL, Python, and JavaScript UDFs, but the handling of it is warehouse-specific. For SQL and Python UDFs on BigQuery, `volatility` is ignored and dbt displays a warning. For JavaScript UDFs on BigQuery, `deterministic` and `non-deterministic` are applied when creating the UDF; `stable` is not supported. In Snowflake, all supported volatility values are applied when creating the UDF. Refer to [volatility](https://docs.getdbt.com/reference/resource-configs/volatility.md) for more information. 3. Run one of the following `dbt build` commands to build your UDFs and create them in the warehouse: Build all UDFs: ```bash dbt build --select "resource_type:function" ``` Or build a specific UDF: ```bash dbt build --select is_positive_int ``` When you run `dbt build`, the property file (`functions/is_positive_int.yml`) and the corresponding SQL, Python, or JavaScript file work together to generate the `CREATE FUNCTION` statement. The rendered `CREATE FUNCTION` statement depends on which adapter you're using. For example: ##### SQL ##### Snowflake ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING DEFAULT '1') RETURNS INTEGER LANGUAGE SQL IMMUTABLE AS $$ REGEXP_INSTR(a_string, '^[0-9]+$') $$; ``` ##### Redshift ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string VARCHAR) RETURNS INTEGER IMMUTABLE AS $$ SELECT REGEXP_INSTR(a_string, '^[0-9]+$') $$ LANGUAGE SQL; ``` ##### BigQuery ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT64 AS ( REGEXP_INSTR(a_string, r'^[0-9]+$') ); ``` ##### Databricks ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT DETERMINISTIC RETURN REGEXP_INSTR(a_string, '^[0-9]+$'); ``` ##### Postgres ```sql CREATE OR REPLACE FUNCTION udf_schema.is_positive_int(a_string text DEFAULT '1') RETURNS int LANGUAGE sql IMMUTABLE AS $$ SELECT regexp_instr(a_string, '^[0-9]+$') $$; ``` ##### Python ##### Snowflake ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING DEFAULT '1') RETURNS INTEGER LANGUAGE PYTHON RUNTIME_VERSION = '3.11' HANDLER = 'main' PACKAGES = ('numpy', 'pandas==1.5.0') AS $$ import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 $$; ``` ##### BigQuery ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT64 LANGUAGE python OPTIONS( runtime_version = "python-3.11", entry_point = "main", packages = ['numpy', 'pandas==1.5.0'] ) AS r''' import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 '''; ``` ##### Databricks ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT LANGUAGE PYTHON AS $$ import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 return main(a_string) $$; ``` Databricks omits the `RUNTIME_VERSION` and `HANDLER` clauses. The runtime is managed internally, and the contents of your `.py` file become the function body verbatim — including the trailing `return main(a_string)` that produces the result. ##### JavaScript ##### Snowflake ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int("a_string" STRING) RETURNS INTEGER LANGUAGE JAVASCRIPT AS $$ return /^[0-9]+$/.test(a_string) ? 1 : 0; $$; ``` ##### BigQuery ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT64 LANGUAGE js AS r''' return /^[0-9]+$/.test(a_string) ? 1 : 0; '''; ``` 4. Reference the UDF in a model using the `{{ function(...) }}` macro. For example: models/my\_model.sql ```sql select maybe_positive_int_column, {{ function('is_positive_int') }}(maybe_positive_int_column) as is_positive_int from {{ ref('a_model_i_like') }} ``` When using [`--defer`](https://docs.getdbt.com/reference/node-selection/defer.md), `function()` resolves to the existing UDF in the deferred environment (for example, production) if the function is not selected or not yet built in your target environment. This requires a state manifest specified using `--state` or an equivalent environment variable (such as `DBT_ENGINE_STATE`), which dbt uses to determine where to defer. This allows models that depend on UDFs to run successfully in [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) and development workflows. For more information, refer to [Configure state selection](https://docs.getdbt.com/reference/node-selection/configure-state.md). 5. Run `dbt compile` to see how the UDF is referenced. In the following example, the `{{ function('is_positive_int') }}` is replaced by the UDF name `udf_db.udf_schema.is_positive_int`. models/my\_model.sql ```sql select maybe_positive_int_column, udf_db.udf_schema.is_positive_int(maybe_positive_int_column) as is_positive from analytics.dbt_schema.a_model_i_like ``` In your DAG, a UDF node is created from the SQL/Python and YAML definitions, and there will be a dependency between `is_positive_int` → `my_model`. [![The DAG for the UDF node](/img/docs/building-a-dbt-project/UDF-DAG.png?v=2 "The DAG for the UDF node")](#)The DAG for the UDF node After defining a UDF, your changes are applied to the UDF in the warehouse the next time you run `dbt build` when you update any of the following: * The SQL, Python, or JavaScript file that contains its function body (`is_positive_int.sql`, `is_positive_int.py`, or `is_positive_int.js` in these examples) * Its configurations * Its properties defined in the `.yml` file (such as `arguments` or `returns`) dbt detects all of these changes when using [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state). ##### Defining overloaded UDFs Use the [`overloads`](https://docs.getdbt.com/reference/resource-properties/overloads.md) property (available in dbt v1.12+) to define multiple argument signatures for the same function. This lets you call the same function name with different input types, without creating separate UDFs for each variant. `overloads` is supported for SQL UDFs in Snowflake and Postgres, and Python and JavaScript UDFs in Snowflake. To define overloaded UDFs: 1. Add an `overloads` list to the function definition in your properties YAML file. Each entry uses `defined_in` to reference a separate file, with optional `arguments` and `returns`: functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int arguments: - name: a_string data_type: string returns: data_type: integer overloads: - defined_in: is_positive_int_numeric # references functions/is_positive_int_numeric.sql arguments: - name: a_num data_type: numeric returns: # optional, inherits from root function if omitted data_type: integer ``` 2. Create a separate file for each overload body. For example, the body for the root function, which accepts a `string` argument: functions/is\_positive\_int.sql ```sql # Snowflake syntax REGEXP_INSTR(a_string, '^[0-9]+$') ``` And the body for the overload, which accepts a `numeric` argument: functions/is\_positive\_int\_numeric.sql ```sql # Snowflake syntax CASE WHEN a_num > 0 THEN 1 ELSE 0 END ``` All overloads are grouped into one DAG node (the root function), so they're built and selected together. On retry, dbt skips overloads that succeeded and reruns only those that failed. When dbt builds the function, it renders a separate `CREATE FUNCTION` statement for each overload using the same function name but different argument types. For more information, refer to [`overloads`](https://docs.getdbt.com/reference/resource-properties/overloads.md). #### Using UDFs in unit tests You can use [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) to validate models that reference UDFs. Before running unit tests, make sure the function exists in your warehouse. To ensure that the function exists for a unit test, run: ```bash dbt build --select "+my_model_to_test" --empty ``` Following the example in [Defining UDFs in dbt](#defining-udfs-in-dbt), here's an example of a unit test that validates a model that calls a UDF: tests/test\_is\_positive\_int.yml ```yml unit_tests: - name: test_is_positive_int description: "Check my is_positive_int logic captures edge cases" model: my_model given: - input: ref('a_model_i_like') rows: - { maybe_positive_int_column: 10 } - { maybe_positive_int_column: -4 } - { maybe_positive_int_column: +8 } - { maybe_positive_int_column: 1.0 } expect: rows: - { maybe_positive_int_column: 10, is_positive: true } - { maybe_positive_int_column: -4, is_positive: false } - { maybe_positive_int_column: +8, is_positive: true } - { maybe_positive_int_column: 1.0, is_positive: true } ``` #### Listing and building UDFs Use the [`list` command](https://docs.getdbt.com/reference/commands/list.md#listing-functions) to list UDFs in your project: `dbt list --select "resource_type:function"` or `dbt list --resource-type function`. Use the [`build` command](https://docs.getdbt.com/reference/commands/build.md#functions) to select UDFs when building a project: `dbt build --select "resource_type:function"`. For more information about selecting UDFs, see the examples in [Node selector methods](https://docs.getdbt.com/reference/node-selection/methods.md#file). #### Limitations * UDFs in other languages (for example, Java or Scala) are not yet supported. * JavaScript UDFs are supported in Snowflake and BigQuery only. Using JavaScript UDFs on an unsupported adapter raises a parsing error. * Python UDFs are supported in Snowflake, BigQuery, and Databricks only (when using dbt v1 or dbt v2). Other warehouses aren't yet supported for Python UDFs. * Only scalar and aggregate functions are currently supported. For more information, see [Supported function types](https://docs.getdbt.com/reference/resource-configs/type.md#supported-function-types). * The `overloads` property is supported for SQL UDFs in Snowflake and Postgres, and Python and JavaScript UDFs in Snowflake. #### Related FAQs When should I use a UDF instead of a macro? Both user-defined functions (UDFs) and macros let you reuse logic across your dbt project, but they work in fundamentally different ways. Here's when to use each: ###### Use UDFs when:  You need logic accessible outside dbt UDFs are created in your warehouse and can be used by BI tools, data science notebooks, SQL clients, or any other tool that connects to your warehouse. Macros only work within dbt.  You want to standardize warehouse-native functions UDFs let you create reusable warehouse functions for data validation, custom formatting, or business-specific calculations that need to be consistent across all your data tools. Once created, they become part of your warehouse's function catalog.  You want dbt to manage the function lifecycle dbt manages UDFs as part of your DAG execution, ensuring they're created before models that reference them. You can version control UDF definitions alongside your models, test changes in development environments, and deploy them together through CI/CD pipelines.  Jinja compiles at creation time, not on each function call You can use Jinja (loops, conditionals, macros, `ref`, `source`, `var`) inside a UDF configuration. dbt resolves that Jinja **when the UDF is created**, and the resulting SQL body is what gets stored in your warehouse. Jinja influences the function when it’s created, whereas arguments influence it when it runs in the warehouse: * ✅ **Allowed:** Jinja that depends on project or build-time state — for example, `var(“can_do_things”)`, static `ref(‘orders’)`, or environment-specific logic. These are all evaluated once at creation time. * ❌ **Not allowed:** Jinja that depends on **function arguments** passed at runtime. The compiler can’t see those, so dynamic `ref(ref_name)` or conditional Jinja based on argument values won’t work.  You need Python logic that runs in your warehouse A Python UDF creates a Python function directly within your data warehouse, which you can invoke using SQL.<br />This makes it easier to apply complex transformations, calculations, or logic that would be difficult or verbose to express in SQL. Python UDFs support conditionals and looping within the function logic itself (using Python syntax), and execute at runtime, not at compile time like macros. Python UDFs are currently supported in Snowflake and BigQuery. ###### Use macros when:  You need to generate SQL at compile time Macros generate SQL dynamically **before** it's sent to the warehouse (at compile time). This is essential for: * Building different SQL for different warehouses * Generating repetitive SQL patterns (like creating dozens of similar columns) * Creating entire model definitions or DDL statements * Dynamically referencing models based on project structure UDFs execute **at query runtime** in the warehouse. While they can use Jinja templating in their definitions, they don't generate new SQL queries—they're pre-defined functions that get called by your SQL. Expanding UDFs Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You want to generate DDL or DML statements Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You need to adapt SQL across different warehouses Macros can use Jinja conditional logic to generate warehouse-specific SQL (see [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md)), making your dbt project portable across platforms. UDFs are warehouse-specific objects. Even though UDFs can include Jinja templating in their definitions, each warehouse has different syntax for creating functions, different supported data types, and different SQL dialects. You would need to define separate UDF files for each warehouse you support.  Your logic needs access to dbt context Both macros and UDFs can use Jinja, which means they can access dbt context variables like `{{ ref() }},` `{{ source() }}`, environment variables, and project configurations. You can even call a macro from within a UDF (and vice versa) to combine dynamic SQL generation with runtime execution. However, the difference between the two is *when* the logic runs: * Macros run at compile time, generating SQL before it’s sent to the warehouse. * UDFs run inside the warehouse at query time.  You want to avoid creating warehouse objects Macros don't create anything in your warehouse; they just generate SQL at compile time. UDFs create actual function objects in your warehouse that need to be managed. ###### Can I use both together? Yes! You can use a macro to call a UDF or call a macro from within a UDF, combining the benefits of both. So the following example shows how to use a macro to define default values for arguments alongside your logic, for your UDF ```sql {% macro cents_to_dollars(column_name, scale=2) %} {{ function('cents_to_dollars') }}({{ column_name }}, {{scale}}) {% endmacro %} ``` ###### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Jinja macros](https://docs.getdbt.com/docs/build/jinja-macros.md) --- ### Using catalogs.yml Now that you understand [what an Iceberg catalog is](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md), let's talk about how to use them within dbt. #### Getting started: Managed catalogs Several data platforms offer their own "managed" catalogs that support the Iceberg table format out-of-the-box. These include Snowflake Horizon, Databricks Unity, AWS Glue (for Athena + Redshift), and BigLake (for BigQuery). If you're using dbt with a data platform that offers a managed Iceberg catalog, then the simplest way to materialize your first dbt model as an Iceberg table is to set the `table_format` configuration: models/hello\_iceberg.sql ```sql {{ config( materialized = 'table', table_format = 'iceberg' ) }} select 'hello_iceberg' as message ``` That's it. This model is materialized as an Iceberg table, with all the default configurations for this adapter, and stored in the default (managed) catalog offered by this data platform. You can now connect to that catalog using another engine (such as DuckDB) to read this table. Congratulations, you're using Iceberg! note Most open source query engines, including DuckDB and Apache Spark, can operate well with external catalogs, but they don't come with a "managed" catalog. Instead, their default behavior for materializing Iceberg tables is to write Parquet files and Iceberg metadata to the local filesystem (wherever the query engine is running). #### Next step: Using catalogs You should start using `catalogs` when: * You want a single place to define custom configurations for how dbt should materialize Iceberg tables * You want to write to multiple catalogs ("external" as well as built-in / managed) * You want to access the same catalog across multiple data platforms / dbt projects (Applies to dbt v2.0 and later) dbt defines `catalogs` in a single top-level file, `catalogs.yml`, that lives in the root of your project directory. We first introduced `catalogs.yml` in dbt v1.10, and dbt v1.12 added a new simpler spec (recommended). In dbt v2, the new spec is the default. To keep using the old spec, set `use_catalogs_v2: false` under `flags:` in `dbt_project.yml`. ##### New spec (recommended) *Available in dbt v1.12+ (including v2). See GitHub discussion [dbt-labs/dbt#12723](https://github.com/dbt-labs/dbt/discussions/12723) for an explanation of the motivations behind the new spec, and an overview of what's changed.* Each entry in `catalogs` refers to a specific catalog containing Iceberg tables. Each catalog **should** map to a top-level logical namespace (often called "database" in dbt). Each catalog may be managed or external for this data platform. Each catalog may be accessed (read from and written to) by one or multiple data platforms. For this reason, each catalog's adapter-specific configuration is nested under `ADAPTER` keys (such as `snowflake:` and `databricks:`). If you run the same dbt project, with the same `catalogs.yml`, using different adapters, dbt always uses the catalog configuration for the current active adapter. That said, one "catalog" **must** always point to the same actual data (Iceberg tables in object storage), regardless of whether that catalog is external to or managed by the current active adapter. (Applies to dbt v1.12 and later) catalogs.yml ```yml catalogs: - name: my_iceberg_catalog type: CATALOG_TYPE # see below table_format: iceberg # required config: ADAPTER: # Configuration for a specific adapter to integrate with this catalog. # See available configs for each adapter + catalog combination. ``` ###### Catalog types | catalog `type` | default for | supported by | Notes | | ------------------ | ----------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | horizon | snowflake | snowflake, duckdb | | | glue | athena | athena, snowflake, duckdb | Default catalog type for Athena. Refer to [Iceberg catalogs](https://docs.getdbt.com/reference/resource-configs/athena-configs.md#iceberg-catalogs). | | s3\_tables | | athena | Iceberg catalogs managed by Amazon S3 Tables (`dbt-athena` 1.11.1 and later). Refer to [AWS S3 Tables](https://docs.getdbt.com/reference/resource-configs/athena-configs.md#aws-s3-tables). | | biglake\_metastore | bigquery | bigquery, snowflake | Supports BigQuery [Lakehouse Runtime Catalog tables](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md#lakehouse-runtime-catalog-lrc) through the `lakehouse_catalog` config | | unity | databricks | databricks, snowflake, duckdb | Supports Iceberg (native), Delta, "Uniform" formats | | hive\_metastore | | databricks | Supports Hudi format, in addition to Iceberg + Delta | | ducklake | | duckdb | | | local\_filesystem | | duckdb | | | iceberg\_rest | | snowflake, duckdb | In theory, this option can support any catalog that implements an Iceberg-compatible REST API | ###### Config inheritance Configurations defined in `catalogs.yml` are lowest in the [model-config precedence](https://docs.getdbt.com/reference/model-configs.md). This means that if you set a more-specific config for one model (within its `.sql` or `.py` file), or set a project-level config, those take precedence. For example, you set a default `base_location_root` for all models in the `finance_db` catalog: catalogs.yml ```yml catalogs: - name: finance_db type: unity config: snowflake: base_location_root: 's3://my-bucket/finance_db' ``` But then you override that config for one particular model: models/finance/my\_special\_model.sql ```sql {{ config( catalog_name = 'finance_db', base_location_root = 's3://my-bucket/somewhere_else' ) }} ``` Some Iceberg-related configurations are only available at the model configuration level, so they can't be set in `catalogs.yml`. For example, the related config `base_location_subpath` determines the exact write path for a single Iceberg table, so it only makes sense to configure per-model, rather than setting a default for all models in the catalog. ###### The `catalog_database` config There is a dedicated configuration for the name of the database mapped to this catalog in each adapter. Unlike other catalog-level configurations, the `catalog_database` applies to *all* models configured with this `catalog_name`, it can't be changed for specific models, and it doesn't follow the usual rules about the `database` config or the `generate_database_name` macro. This means that if a model has both a `database` config and a `catalog_name` config, the catalog’s `catalog_database` takes precedence over the model’s database config, to tell dbt where to materialize this model. Why? We strongly recommend a 1:1 mapping between each Iceberg catalog and the top-level namespace (logical "database") to which it is linked/synced. The namespaces don't have to be identical, but if you can make them the same everywhere, it's simpler to reason about and debug. If you don't specify a `catalog_database`, then dbt materializes models based on their `database` config. In this case, the catalog serves as a collection of shared Iceberg configs, but it doesn't map to a consistent namespace containing all its Iceberg tables. ##### Old spec *Available in dbt v1.10+* (Applies to dbt v2.0 and later) To use the old spec, first set this behavior flag: dbt\_project.yml ```yml flags: use_catalogs_v2: false ``` Each catalog configures one or more `write_integrations`, and then specifies an "active" write integration to use for the current invocation / data warehouse. catalogs.yml ```yml catalogs: - name: my_glue_catalog active_write_integration: glue_rest write_integrations: - name: glue_rest catalog_type: iceberg_rest table_format: iceberg adapter_properties: catalog_linked_database: catalog_linked_db_glue catalog_linked_database_type: glue ``` --- ### Validations Validations refer to the process of checking whether a system or configuration meets the expected requirements or constraints. In the case of the Semantic Layer, powered by MetricFlow, there are three built-in validations — [parsing](#parsing), [semantic](#semantic), and [data platform](#data-platform). These validations ensure that configuration files follow the expected schema, the semantic graph doesn't violate any constraints, and semantic definitions in the graph exist in the physical table — providing effective data governance support. These three validation steps occur sequentially and must succeed before proceeding to the next step. The code that handles validation [can be found here](https://github.com/dbt-labs/dbt-semantic-interfaces/tree/main/dbt_semantic_interfaces/validations) for those who want to dive deeper into this topic. #### Validations command You can run validations from the dbt platform or the command line with the following [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md). In dbt, you need user credentials configured in your account settings to run `dbt sl validate` in the IDE or CLI, and deployment credentials to run it in CI. * For dbt v2 and dbt users in the dbt platform CLI or locally with a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md): ```bash dbt sl validate ``` This runs parsing, semantic, and (where supported) data platform validations. When using `dbt sl validate` locally, the command validates your local semantic manifest, and not the platform's manifest. This means your uncommitted local changes are included in the validation. * For dbt v1 (open source) users or v2 CLI users not connected to dbt platform and using local MetricFlow: ```bash mf validate-configs ``` This runs parsing and semantic validations. #### Availability by environment Validation behavior and availability differ depending on your environment and setup: | Environment | Who can use | Parsing | Semantic syntax | Data platform | How to run | | ---------------- | --------------------------------------------------- | ------- | --------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | dbt v2 | dbt platform users for full Semantic Layer features | ✅ | ✅ \* | ✅ | - Parsing validations run automatically while generating the semantic manifest.<br />- When running in development, semantic syntax validations run automatically on dbt platform if [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) is configured. If not, run manually using `mf validate-configs`.<br />- Data platform validations don't run automatically for dbt v2. You must run `dbt sl validate` to run data platform validations. | | dbt platform CLI | dbt platform users | ✅ | ✅ | ✅ | Run any dbt platform CLI command; validations execute automatically except data platform validations. You must run `dbt sl validate` to run data platform validations. | | dbt v1 | Open source users | ✅ | ✅ | ❌ | Use dbt v1 for parsing/builds. Run additional validation manually with the MetricFlow CLI. | | MetricFlow CLI | Open source users | ✅ | ✅ | ✅ | Run `mf validate-configs` locally to validate and test metrics. | \*Jobs run in **Orchestration** or **Studio IDE** run this validation automatically. #### Parsing In this validation step, we ensure your config files follow the defined schema for each semantic graph object and can be parsed successfully. It validates the schema for the following core objects: (Applies to dbt v1.12 and later) * Semantic models * Identifiers * Simple metrics * Dimensions * Metrics #### Semantic syntax This syntactic validation step occurs after we've built your semantic graph. The Semantic Layer, powered by MetricFlow, runs a suite of tests to ensure that your semantic graph doesn't violate any constraints. For example, we check to see if (Applies to dbt v1.12 and later) simple metric names are unique, or if metrics referenced in materialization exist. The current semantic rules we check for are: (Applies to dbt v1.12 and later) 1. Check those semantic models with simple metrics have a valid time dimension 2. Check that there is only one primary identifier defined in each semantic model 3. Dimension consistency 4. Unique simple metrics in semantic models 5. Simple metrics are valid 6. Cumulative metrics are configured properly #### Data platform This type of validation checks to see if the semantic definitions in your semantic graph exist in the underlying physical table. To test this, we run queries against your data platform to ensure the generated SQL for semantic models, dimensions, and metrics will execute. We run the following checks: (Applies to dbt v1.12 and later) * Simple metrics and dimensions exist * Underlying tables for data sources exist * Generated SQL for metrics will execute You can run semantic validations (against your semantic layer) in a CI job to guarantee any code changes made to dbt models don't break these metrics. For more information, refer to [semantic validation in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci). --- ### Version compatibility dbt v2, its language server (LSP), and the dbt VS Code extension work together to power local development. This page explains how their versions relate, which combinations are compatible, and how to verify you have matching binaries. #### How versioning works dbt v2 ships as a single binary that provides both the command-line interface (CLI) and the language server (LSP). Because the CLI and LSP come from the *same* binary, they always share the same version and can't be mismatched. The dbt VS Code extension keeps its own release version because it follows the VS Code Marketplace release lifecycle. To make compatibility explicit, each extension release declares the range of dbt v2 versions it supports. When the extension starts the LSP, it checks the installed dbt v2 version against that range and surfaces an [actionable error](#troubleshooting-version-issues) if they don't match. | Component | Example versioning | | ------------------------- | ----------------------------------------------------------------------------------- | | dbt v2 binary (CLI + LSP) | `2.0.0` | | dbt VS Code extension | The extension version (for example, `0.36.0`), with compatible dbt v2 version range | #### Compatibility matrix The following table maps each dbt VS Code extension version to the dbt versions it supports. If you let the extension download and manage dbt v2 for you, it always installs a compatible version automatically — you only need this table if you install or pin binaries manually. | dbt VS Code extension | Min dbt v2 version | Max dbt v2 version | Checksums | Notes | | --------------------- | ------------------ | ------------------ | -------------------- | ------- | | 0.36.x | 2.0.0-preview.90 | — | `checksums-0.36.txt` | Current | | 0.35.x | 2.0.0-preview.80 | 2.0.0-preview.89 | `checksums-0.35.txt` | - | A dash (-) in the **Max** column means there is no upper bound yet. Any dbt v2 release at or above the minimum is supported. #### Check your installed versions To confirm your setup is within the supported range, check the version of dbt v2 you have installed: ```shell dbt --version ``` For a machine-readable summary you can paste into a bug report or parse in tooling, add the `--format json` flag: ```shell dbt --version --format json ``` For details on the full output, refer to [About dbt --version](https://docs.getdbt.com/reference/commands/version.md). #### Verify binaries for manual and air-gapped installs If you install dbt v2 manually in an air-gapped or firewall-restricted environment where the extension can't download binaries for you, use the [compatibility matrix](#compatibility-matrix) to pick a version that matches your dbt VS Code extension, then verify the download before you distribute it. 1. From the compatibility matrix, note the dbt v2 version range for your extension version. 2. Download the matching dbt v2 binary and its checksum file. 3. Verify the checksum before installing or distributing the binary: ##### Mac / Linux ```shell shasum -a 256 -c checksums-0.36.txt ``` ##### Windows (PowerShell) ```powershell Get-FileHash .\dbt.exe -Algorithm SHA256 ``` Compare the printed hash against the value in the checksum file. 4. Point the dbt VS Code extension at the verified binary using the `dbt.fusionPath` setting. Refer to [dbt extension settings](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings). For network and proxy requirements, and how to pre-build an offline bundle, refer to [Networking requirements](https://docs.getdbt.com/docs/local/dbt-networking-requirements.md#restricted-network-installation). #### Known-bad releases If a shipped dbt v2 release is later found to contain a regression, dbt Labs flags it as a known-bad release. When you have a flagged version installed, the dbt VS Code extension shows a warning notification telling you which version to update to, even if that version is otherwise within the supported range. Air-gapped users who don't have outbound network access can distribute the known-bad releases manifest alongside their binary bundle and point the extension at the local copy with the `dbt.badReleasesManifestPath` setting. Refer to [dbt extension settings](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings). #### Troubleshooting version issues If the extension can't find, start, or verify a compatible dbt v2 binary, it surfaces an actionable error with a link or button to resolve it. Refer to the [dbt VS Code extension troubleshooting](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#troubleshooting) section for the full list of messages and fixes. --- ### Version upgrade guides #### [v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) [Your to-do list for preparing for dbt v2](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) --- ### View documentation dbt provides intuitive and scalable tools for viewing your dbt documentation. Detailed documentation is essential for your developers and other stakeholders to gain shared context for your dbt project. You can view documentation in three complementary ways, depending on your needs: | Option | Description | Availability | | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | | [**dbt Docs (Legacy)**](#dbt-docs) | Generates a static website with model lineage, metadata, and documentation that can be hosted on your web server (like S3 or Netlify). | dbt v1 or dbt Developer plans | | [**dbt Docs v2**](#dbt-docs-v2) | A modern, performant open-source catalog built for data consumers. Includes a redesigned UI, large-project performance, Semantic Layer metadata, and column-level lineage (dbt v2), served as a static site you can host anywhere. | dbt v2 (local only) | | [**Catalog**](https://docs.getdbt.com/docs/explore/explore-projects.md) | The premier documentation experience in dbt. Builds on dbt Docs to provide a dynamic, real-time interface with rich [metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata), customizable views, deep insight into your project and resources, and collaborative tools. | dbt Starter, Enterprise, or Enterprise+ plans | #### Navigating your documentation The following sections describe how to navigate your documentation in Catalog and dbt Docs. ##### Catalog [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) offers a dynamic, interactive way to explore your models, sources, and lineage. To access Catalog, navigate to the **Catalog** option in the dbt navigation menu. [![Example of Catalog's resource details page and its lineage.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "Example of Catalog's resource details page and its lineage.")](#)Example of Catalog's resource details page and its lineage. [![Access Catalog from the lineage tab in a job by double-clicking a lineage node.](/img/docs/collaborate/dbt-explorer/explorer-main-page.gif?v=2 "Access Catalog from the lineage tab in a job by double-clicking a lineage node.")](#)Access Catalog from the lineage tab in a job by double-clicking a lineage node. Catalog offers users a comprehensive suite of features to enhance data project navigation and understanding, like: * Interactive lineage visualization for your project's DAG to understand relationships between resources. * Resource search bar with comprehensive filters to help find project resources efficiently and quickly. * Model performance insights to access metadata on dbt runs for in-depth analysis of model performance and quality. * Project recommendations with suggestions to improve test coverage and documentation across your data estate. * Data health signals to monitor the health and performance of each resource through data health indicators. * Model query history to track consumption queries on your models to gain deeper insights into data usage. * Downstream exposures to automatically expose relevant data models from tools like Tableau to enhance visibility. Sharing docs with stakeholders? Anyone with a developer or read-only seat can explore your project(s) in Catalog. Add as many read-only seats as you need to share docs with stakeholders, no separate docs site required. For additional details and instructions on how to explore your lineage, navigate your resources, view model query history and data health signals, feature availability, and more — refer to [Discover data with Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). ##### dbt Docs v2 dbt Docs v2 is the next-generation open-source catalog experience, available when using dbt v2. It is designed for data consumers (analysts, BI users, data scientists, and stakeholders) who need to understand what data exists, how it was built, and whether they can trust it. dbt Docs v2 availability dbt Docs v2 is available on [self-hosted dbt v2 installations](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) only. In dbt platform, jobs on dbt v2 refresh Catalog metadata automatically, so use [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) to explore your project, and give all your applicable users read-only access to it without restrictions. Refer to [platform behavior](https://docs.getdbt.com/reference/commands/cmd-docs.md?version=2#platform-behavior) for more info. Key improvements over dbt Docs: * **Performance:** Better handling for large dbt projects. The browser queries compact, pre-built index files with DuckDB-WASM (WebAssembly) instead of loading the full `manifest.json`. * **Modernized UI:** Visually aligned with the dbt platform, with better navigation and resource discovery. * **Semantic Layer metadata:** Surfaces compiled SQL logic, queryable dimensions, and metric definitions from your dbt Semantic Layer. * **Column-level lineage:** Available when using dbt v2. * **Statically hostable:** `dbt docs generate` writes a self-contained static site — no server or live warehouse connection — that you can host on any file host, such as S3, GitHub Pages, or Netlify. To generate and serve dbt Docs v2 with dbt v2, run `dbt docs generate` to build the site, then `dbt docs serve` to preview it locally. `dbt docs generate` compiles your project and writes the index for you in a single command. To include model column types inferred from your SQL and column-level lineage, run `dbt build`, `dbt run`, or `dbt compile` with [`--generate-info-schema`](https://docs.getdbt.com/docs/build/dbt-information-schema.md) and [`--static-analysis strict`](https://docs.getdbt.com/docs/build/about-static-analysis.md), then export the site: ```shell dbt build --generate-info-schema --static-analysis strict dbt docs generate --no-compile ``` `--no-compile` skips the default compile step in `dbt docs generate` and exports the existing index. Refer to [dbt docs commands](https://docs.getdbt.com/reference/commands/cmd-docs.md) for full usage and [Commands for generating metadata](https://docs.getdbt.com/reference/commands/cmd-docs.md#commands-for-generating-metadata) for a breakdown of flags and the metadata they generate. ###### Self-hosting dbt Docs v2 Because `dbt docs generate` writes a self-contained static site — the single-page app plus the Parquet index files, with no server-side query engine — you can host it on any static file host instead of running `dbt docs serve` locally. The browser loads DuckDB-WASM from a content delivery network (CDN) at runtime and queries the Parquet directly, so viewers only need a static file server and outbound internet access to reach the CDN. Generate a self-contained directory and publish it to your host of choice: ```shell dbt docs generate --output-dir site ``` This writes a `site/` directory (the app, hashed assets, and a copy of the index) that you can host on S3, GitHub Pages, Netlify, GitLab Pages, or any similar static file host. To refresh the docs, regenerate the site and re-publish. ##### dbt Docs (Legacy) dbt Docs provides valuable insights into your dbt v1 or dbt Developer plan projects. The interface enables you to navigate to the documentation for specific models. That might look something like this: [![Auto-generated documentation for a dbt model](/img/docs/building-a-dbt-project/testing-and-documentation/f2221dc-Screen_Shot_2018-08-14_at_6.29.55_PM.png?v=2 "Auto-generated documentation for a dbt model")](#)Auto-generated documentation for a dbt model Here, you can see a representation of the project structure, a markdown description for a model, and a list of all of the columns (with documentation) in the model. From the dbt Docs page, click the green button in the bottom-right corner of the webpage to expand a "mini-map" of your DAG. This pane displays the immediate parents and children of the model that you're exploring. [![Opening the DAG mini-map](/img/docs/building-a-dbt-project/testing-and-documentation/ec77c45-Screen_Shot_2018-08-14_at_6.31.56_PM.png?v=2 "Opening the DAG mini-map")](#)Opening the DAG mini-map In this example, the `fct_subscription_transactions` model only has one direct parent. By clicking the "Expand" button in the top-right corner of the window, we can pivot the graph horizontally and view the full lineage for our model. This lineage is filterable using the `--select` and `--exclude` flags, which are consistent with the semantics of [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). Further, you can right-click to interact with the DAG, jump to documentation, or share links to your graph visualization with your coworkers. [![The full lineage for a dbt model](/img/docs/building-a-dbt-project/testing-and-documentation/ac97fba-Screen_Shot_2018-08-14_at_6.35.14_PM.png?v=2 "The full lineage for a dbt model")](#)The full lineage for a dbt model #### Deploy the documentation site Effortlessly deploy documentation in Catalog or dbt Docs to make it available to your teams. Security The `dbt docs serve` command is only intended for local/development hosting of the documentation site. Please use one of the methods listed in the next section (or similar) to ensure that your documentation site is hosted securely! ##### Catalog [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Catalog automatically updates documentation after each production or staging job run using the metadata generated. This means it always has the latest results for your project with no manual deployment required. For details on how Catalog uses metadata to automatically update documentation, refer to [Generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata). To learn how to deploy your documentation site, see [Build and view your docs with dbt](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md). ##### dbt Docs (Legacy) dbt Docs was built to make it easy to host on the web. The site is "static," meaning you don't need any "dynamic" servers to serve the docs. You can host your documentation in several ways: * Host on [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) (optionally [with IP access restrictions](https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-3)) * Publish with [Netlify](https://discourse.getdbt.com/t/publishing-dbt-docs-to-netlify/121) * Use your own web server like Apache/Nginx * If you're on a dbt Developer plan, see [Build and view your docs with dbt](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs) to learn how to deploy your documentation site. Interested in using Catalog for the complete dbt documentation experience, sign up for a free [dbt trial](https://www.getdbt.com/signup) or [contact us](https://www.getdbt.com/contact). --- ### Visualize and orchestrate downstream exposures dbt platform | Enterprise, Enterprise+ The following table summarizes the differences between visualizing and orchestrating downstream exposures: | Info | Set up and visualize downstream exposures | Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") | | ----------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Automatically brings downstream assets into your dbt lineage. | Proactively refreshes the underlying data sources during scheduled dbt jobs. | | Benefits | Provides visibility into data flow and dependencies. | Ensures BI tools always have up-to-date data without manual intervention. | | Location | Exposed in dbt [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) | Exposed in [dbt scheduler](https://docs.getdbt.com/docs/deploy/deployments.md) | | Supported BI tool | Tableau | Tableau | | Use case | Helps users understand how models are used and reduces incidents. | Optimizes timeliness and reduces costs by running models when needed. | Check out the following sections for more information on visualizing and orchestrating downstream exposures: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) ###### [Set up and visualize downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [Set up downstream exposures automatically from dashboards to understand how models are used in downstream tools for a richer downstream lineage.](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) ###### [Orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) [Proactively refreshes the underlying data sources (like Tableau extracts) using the dbt scheduler during scheduled dbt jobs.](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) --- ### Visualize and orchestrate downstream exposures dbt platform | Enterprise, Enterprise+ Visualize and orchestrate downstream exposures in dbt to automatically generate exposures from dashboards and proactively refresh the underlying data sources (like Tableau extracts) during scheduled dbt jobs. The following table summarizes the differences between visualizing and orchestrating downstream exposures: | Info | Set up and visualize downstream exposures | Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") | | ----------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Automatically brings downstream assets into your dbt lineage. | Proactively refreshes the underlying data sources during scheduled dbt jobs. | | Benefits | Provides visibility into data flow and dependencies. | Ensures BI tools always have up-to-date data without manual intervention. | | Location | Exposed in dbt [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) | Exposed in [dbt scheduler](https://docs.getdbt.com/docs/deploy/deployments.md) | | Supported BI tool | Tableau | Tableau | | Use case | Helps users understand how models are used and reduces incidents. | Optimizes timeliness and reduces costs by running models when needed. | Check out the following sections for more information on visualizing and orchestrating downstream exposures: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) ###### [Set up and visualize downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [Set up downstream exposures automatically from dashboards to understand how models are used in downstream tools for a richer downstream lineage.](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) ###### [Orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) [Proactively refreshes the underlying data sources (like Tableau extracts) using the dbt scheduler during scheduled dbt jobs.](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) --- ### Visualize downstream exposures dbt platform | Enterprise, Enterprise+ Downstream exposures integrate natively with Tableau and auto-generate downstream lineage in Catalog for a richer experience. As a data team, it’s critical that you have context into the downstream use cases and users of your data products. By leveraging downstream [exposures](https://docs.getdbt.com/docs/build/exposures.md) automatically, data teams can: * Gain a better understanding of how models are used in downstream analytics, improving governance and decision-making. * Reduce incidents and optimize workflows by linking upstream models to downstream dependencies. * Automate exposure tracking for supported BI tools, ensuring lineage is always up to date. * [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) to refresh the underlying data sources during scheduled dbt jobs, improving timeliness and reducing costs. Orchestrating exposures is essentially a way to ensure that your BI tools are updated regularly by using the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/deployments.md). * For more info on the differences between visualizing and orchestrating exposures, see [Visualize and orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md). To configure downstream exposures automatically from dashboards in Tableau, prerequisites, and more — refer to [Configure downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md). ##### Supported plans Downstream exposures is available on all dbt [Enterprise-tier plans](https://www.getdbt.com/pricing/). Currently, you can only connect to a single Tableau site on the same server. Tableau Server If you're using Tableau Server, you need to [allowlist dbt's IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your dbt region. #### View downstream exposures After setting up downstream exposures in dbt, you can view them in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for a richer experience. Navigate to Catalog by selecting **Catalog** from the top-level navigation. You can view downstream exposures from a couple of places: * [Exposures menu](#exposures-menu) * [Exposure detail page](#exposure-detail-page) * [Project lineage](#project-lineage) ##### Exposures menu View all downstream exposures for a project from the Catalog sidebar: 1. In the sidebar, select your project. 2. Under the project, select **Exposure**. You will only see this option if you set up downstream exposures in [Tableau](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-tableau) and [dbt](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-dbt-cloud). The **Exposures** table lists every exposure in the project so you can quickly access and manage them: * **Name**: The name of the exposure. * **Health**: The [data health signal](https://docs.getdbt.com/docs/explore/data-health-signals.md) of the exposure. * **Type**: The type of exposure, such as `dashboard` or `notebook`. * **Owner**: The owner of the exposure. * **Owner email**: The email address of the owner of the exposure. * **Integration**: The BI tool that the exposure is integrated with. * **Exposure mode**: The type of exposure defined: **Auto** or **Manual**. Auto exposures from Tableau appear alongside manual exposures [![View the Exposures table from the Catalog sidebar.](/img/docs/platform-integrations/auto-exposures/explorer-view-resources.png?v=2 "View the Exposures table from the Catalog sidebar.")](#)View the Exposures table from the Catalog sidebar. ##### Exposure detail page After you open the **Exposures** table ([previous section](#exposures-menu)), select an exposure name to open its detail page. On the exposure detail page, you can review metadata, [data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md), description, and lineage. For Tableau auto exposures, use **Open in Dashboard** to open the workbook in Tableau, or **Modify integration** to update your Tableau connection settings. [![View an exposure detail page in Catalog.](/img/docs/platform-integrations/auto-exposures/explorer-view-exposure-detail.png?v=2 "View an exposure detail page in Catalog.")](#)View an exposure detail page in Catalog. ##### Project lineage You can also view exposures from the **Project lineage** view, separate from the **Exposures** table: 1. In the sidebar, select your project. 2. Click **View lineage**. 3. Select an exposure node with the Tableau icon to view its details in the side panel. This view visualizes the dependencies and relationships in your project. For Tableau auto exposures, use **View in Tableau** or **Modify integration** from the side panel. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. --- ### Webhooks for your jobs dbt platform | Starter, Enterprise, Enterprise+ With dbt, you can create outbound webhooks to send events (notifications) about your dbt jobs to your other systems. Your other systems can listen for (subscribe to) these events to further automate your workflows or to help trigger automation flows you have set up. A webhook is an HTTP-based callback function that allows event-driven communication between two different web applications. This allows you to get the latest information on your dbt jobs in real time. Without it, you would need to make API calls repeatedly to check if there are any updates that you need to account for (polling). Because of this, webhooks are also called *push APIs* or *reverse APIs* and are often used for infrastructure development. dbt sends a JSON payload to your application's endpoint URL when your webhook is triggered. You can send a [Slack](https://docs.getdbt.com/guides/zapier-slack.md) notification, a [Microsoft Teams](https://docs.getdbt.com/guides/zapier-ms-teams.md) notification, [open a PagerDuty incident](https://docs.getdbt.com/guides/serverless-pagerduty.md) when a dbt job fails. You can create webhooks for these events from the [dbt web-based UI](#create-a-webhook-subscription) and by using the [dbt API](#api-for-webhooks): * `job.run.started` — Run started. * `job.run.completed` — Run completed. This can be a run that has failed or succeeded. * `job.run.errored` — Run errored. dbt retries sending each event five times. dbt keeps a log of each webhook delivery for 7 days. Every webhook has its own **Recent Deliveries** section, which lists whether a delivery was successful or failed at a glance. A webhook in dbt has a timeout of 10 seconds. This means that if the endpoint doesn't respond within 10 seconds, the webhook processor will time out. This can result in a situation where the client responds successfully after the 10 second timeout and records a success status while the dbt webhooks system will interpret this as a failure. Videos If you're interested in course learning with videos, check out the [Webhooks on-demand course](https://learn.getdbt.com/courses/webhooks) from dbt Labs. You can also check out the free [dbt Fundamentals course](https://learn.getdbt.com/courses/dbt-fundamentals). #### Prerequisites * You have a dbt account that is on the [Starter or Enterprise-tier](https://www.getdbt.com/pricing/) plan. * For `write` access to webhooks: * **Enterprise-tier plans** — Permission sets are the same for both API service tokens and the dbt UI. You, or the API service token, must have the Account Admin, Admin, or Developer [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md). * **Starter plan accounts** — For the dbt user interface (UI), you need to have a [Developer license](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md). * You have a multi-tenant or an AWS single-tenant deployment model in dbt. For more information, refer to [Tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). * Your destination system supports [Authorization headers](#troubleshooting). #### Create a webhook subscription 1. Navigate to **Account settings** in dbt (by clicking your account name from the left side panel) 2. Go to the **Webhooks** section and click **Create webhook**. 3. To configure your new webhook: * **Webhook name** — Enter a name for your outbound webhook. * **Description** — Enter a description of the webhook. * **Events** — Choose the event you want to trigger this webhook. You can subscribe to more than one event. * **Jobs** — Specify the job(s) you want the webhook to trigger on. Or, you can leave this field empty for the webhook to trigger on all jobs in your account. By default, dbt configures your webhook at the account level. * **Endpoint** — Enter your application's endpoint URL, where dbt can send the event(s) to. 4. When done, click **Save**. dbt provides a secret token that you can use to [check for the authenticity of a webhook](#validate-a-webhook). It’s strongly recommended that you perform this check on your server to protect yourself from fake (spoofed) requests. #### Archived webhooks dbt automatically archives a webhook after 1,000 consecutive failed deliveries or 7 consecutive days of failed deliveries, whichever comes first. * On the **Account settings → Webhooks** page in dbt platform, a dismissible warning banner appears when one or more subscriptions are archived. * Archived subscriptions show an **Archived** badge in the status column instead of an HTTP status. * Hover over the badge for details on why dbt archived it and how to reactivate it. [![Webhooks page showing the archived subscription warning banner with the 'Archived; status badge](/img/docs/deploy/webhooks-archived.png?v=2 "Webhooks page showing the archived subscription warning banner with the 'Archived; status badge")](#)Webhooks page showing the archived subscription warning banner with the 'Archived; status badge To reactivate a webhook, use one of the following methods: * **dbt platform UI**: Update the webhook's endpoint URL. dbt automatically reactivates the webhook when the URL changes. * **REST API**: Send a `PUT` request to [Update a webhook](#update-a-webhook) and set `active` to `true`, or update `client_url` to a new endpoint URL. * **Terraform provider**: Set `active = true` and update `client_url` in your webhook resource. To find the appropriate dbt access URL for your region and plan, refer to [Regions & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). ##### Differences between completed and errored webhook events The `job.run.errored` event is a subset of the `job.run.completed` events. If you subscribe to both, you will receive two notifications when your job encounters an error. However, dbt triggers the two events at different times: * `job.run.completed` — This event only fires once the job’s metadata and artifacts have been ingested and are available from the dbt Admin and Discovery APIs. * `job.run.errored` — This event fires immediately so the job’s metadata and artifacts might not have been ingested. This means that information might not be available for you to use. If your integration depends on data from the Admin API (such as accessing the logs from the run) or Discovery API (accessing model-by-model statuses), use the `job.run.completed` event and filter on `runStatus` or `runStatusCode`. If your integration doesn’t depend on additional data or if improved delivery performance is more important for you, use `job.run.errored` and build your integration to handle API calls that might not return data a short period at first. #### Validate a webhook You can use the secret token provided by dbt to validate that webhooks received by your endpoint were actually sent by dbt. Official webhooks will include the `Authorization` header that contains a SHA256 hash of the request body and uses the secret token as a key. An example for verifying the authenticity of the webhook in Python: ```python auth_header = request.headers.get('authorization', None) app_secret = os.environ['MY_DBT_CLOUD_AUTH_TOKEN'].encode('utf-8') signature = hmac.new(app_secret, request_body, hashlib.sha256).hexdigest() return signature == auth_header ``` Note that the destination system must support [Authorization headers](#troubleshooting) for the webhook to work correctly. You can test your endpoint's support by sending a request with curl and an Authorization header, like this: ```shell curl -H 'Authorization: 123' -X POST https://<your-webhook-endpoint> ``` #### Inspect HTTP requests When working with webhooks, it’s good practice to use tools like [RequestBin](https://requestbin.com/) and [Requestly](https://requestly.io/). These tools allow you to inspect your HTML requests, response payloads, and response headers so you can debug and test webhooks before incorporating them into your systems. #### Examples of JSON payloads An example of a webhook payload for a run that's started: ```json { "accountId": 1, "webhookId": "wsu_12345abcde", "eventId": "wev_2L6Z3l8uPedXKPq9D2nWbPIip7Z", "timestamp": "2023-01-31T19:28:15.742843678Z", "eventType": "job.run.started", "webhookName": "test", "data": { "jobId": "123", "jobName": "Daily Job (dbt build)", "runId": "12345", "environmentId": "1234", "environmentName": "Production", "dbtVersion": "1.0.0", "projectName": "Snowflake Github Demo", "projectId": "167194", "runStatus": "Running", "runStatusCode": 3, "runStatusMessage": "None", "runReason": "Kicked off from the UI by test@test.com", "runStartedAt": "2023-01-31T19:28:07Z" } } ``` An example of a webhook payload for a completed run: ```json { "accountId": 1, "webhookId": "wsu_12345abcde", "eventId": "wev_2L6ZDoilyiWzKkSA59Gmc2d7FDD", "timestamp": "2023-01-31T19:29:35.789265936Z", "eventType": "job.run.completed", "webhookName": "test", "data": { "jobId": "123", "jobName": "Daily Job (dbt build)", "runId": "12345", "environmentId": "1234", "environmentName": "Production", "dbtVersion": "1.0.0", "projectName": "Snowflake Github Demo", "projectId": "167194", "runStatus": "Success", "runStatusCode": 10, "runStatusMessage": "None", "runReason": "Kicked off from the UI by test@test.com", "runStartedAt": "2023-01-31T19:28:07Z", "runFinishedAt": "2023-01-31T19:29:32Z" } } ``` An example of a webhook payload for an errored run: ```json { "accountId": 1, "webhookId": "wsu_12345abcde", "eventId": "wev_2L6m5BggBw9uPNuSmtg4MUiW4Re", "timestamp": "2023-01-31T21:15:20.419714619Z", "eventType": "job.run.errored", "webhookName": "test", "data": { "jobId": "123", "jobName": "dbt Vault", "runId": "12345", "environmentId": "1234", "environmentName": "dbt Vault Demo", "dbtVersion": "1.0.0", "projectName": "Snowflake Github Demo", "projectId": "167194", "runStatus": "Errored", "runStatusCode": 20, "runStatusMessage": "None", "runReason": "Kicked off from the UI by test@test.com", "runStartedAt": "2023-01-31T21:14:41Z", "runErroredAt": "2023-01-31T21:15:20Z" } } ``` #### API for webhooks You can use the dbt API to create new webhooks that you want to subscribe to, get detailed information about your webhooks, and to manage the webhooks that are associated with your account. The following sections describe the API endpoints you can use for this. Access URLs dbt is hosted in multiple regions in the world and each region has a different access URL. People on Enterprise-tier plans can choose to have their account hosted in any one of these regions. For a complete list of available dbt access URLs, refer to [Regions & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). ##### List all webhook subscriptions List all webhooks that are available from a specific dbt account. ###### Request ```shell GET https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscriptions ``` ###### Path parameters | Name | Description | | ----------------- | ------------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhooks are associated with. | ###### Response sample ```json { "data": [ { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "job_ids": [ "123", "321" ], "event_types": [ "job.run.started" ], "client_url": "https://test.com", "active": true, "created_at": "1675735768491774", "updated_at": "1675787482826757", "account_id": "123", "http_status_code": "0" }, { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Notification Webhook", "description": "Webhook used to trigger notifications in Slack", "job_ids": [], "event_types": [ "job.run.completed", "job.run.started", "job.run.errored" ], "client_url": "https://test.com", "active": true, "created_at": "1674645300282836", "updated_at": "1675786085557224", "http_status_code": "410", "dispatched_at": "1675786085548538", "account_id": "123" } ], "status": { "code": 200 }, "extra": { "pagination": { "total_count": 2, "count": 2 }, "filters": { "offset": 0, "limit": 10 } } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `data` | List of available webhooks for the specified dbt account ID. | | | `id` | The webhook ID. This is a universally unique identifier (UUID) that's unique across all regions, including multi-tenant and single-tenant | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Description of the webhook. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | - Empty list<br />- List of job IDs | | `event_types` | The event type(s) the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | | `dispatched_at` | Timestamp of when the webhook was last dispatched to the specified endpoint URL. | | | `account_id` | The dbt account ID. | | ##### Get details about a webhook Get detailed information about a specific webhook. ###### Request ```shell GET https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id} ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want detailed information on. | ###### Response sample ```json { "data": { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "event_types": [ "job.run.started" ], "client_url": "https://test.com", "active": true, "created_at": "1675789619690830", "updated_at": "1675793192536729", "dispatched_at": "1675793192533160", "account_id": "123", "job_ids": [], "http_status_code": "0" }, "status": { "code": 200 } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | The webhook ID. | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Complete description of the webhook. | | | `event_types` | The event type the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `dispatched_at` | Timestamp of when the webhook was last dispatched to the specified endpoint URL. | | | `account_id` | The dbt account ID. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | ##### Create a new webhook subscription Create a new outbound webhook and specify the endpoint URL that will be subscribing (listening) to the webhook's events. ###### Request sample ```shell POST https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscriptions ``` ```json { "event_types": [ "job.run.started" ], "name": "Webhook for jobs", "client_url": "https://test.com", "active": true, "description": "A webhook for when jobs are started", "job_ids": [ 123, 321 ] } ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | ###### Request parameters | Name | Description | Possible Values | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | `event_types` | Enter the event you want to trigger this webhook. You can subscribe to more than one event. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `name` | Enter the name of your webhook. | | | `client_url` | Enter your application's endpoint URL, where dbt can send the event(s) to. | | | `active` | Enter a Boolean value to indicate whether your webhook is active or not. | One of these: - `true`<br />- `false` | | `description` | Enter a description of your webhook. | | | `job_ids` | Enter the specific jobs you want the webhook to trigger on or you can leave this parameter as an empty list. If this is an empty list, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | ###### Response sample ```json { "data": { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "job_ids": [ "123", "321" ], "event_types": [ "job.run.started" ], "client_url": "https://test.com", "hmac_secret": "12345abcde", "active": true, "created_at": "1675795644808877", "updated_at": "1675795644808877", "account_id": "123", "http_status_code": "0" }, "status": { "code": 201 } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | The webhook ID. | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Complete description of the webhook. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | | `event_types` | The event type the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `hmac_secret` | The secret key for your new webhook. You can use this key to [validate the authenticity of this webhook](#validate-a-webhook). | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `account_id` | The dbt account ID. | | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | ##### Update a webhook Update the configuration details for a specific webhook. ###### Request sample ```shell PUT https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id} ``` ```json { "event_types": [ "job.run.started" ], "name": "Webhook for jobs", "client_url": "https://test.com", "active": true, "description": "A webhook for when jobs are started", "job_ids": [ 123, 321 ] } ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want to update. | ###### Request parameters | Name | Description | Possible Values | | ------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `event_types` | Update the event type the webhook is set to trigger on. You can subscribe to more than one. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `name` | Change the name of your webhook. | | | `client_url` | Update the endpoint URL for an application where dbt can send event(s) to. | | | `active` | Change the Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `description` | Update the webhook's description. | | | `job_ids` | Change which jobs you want the webhook to trigger for. Or, you can use an empty list to trigger it for all jobs in your account. | One of these: - Empty list<br />- List of job IDs | ###### Response sample ```json { "data": { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "job_ids": [ "123" ], "event_types": [ "job.run.started" ], "client_url": "https://test.com", "active": true, "created_at": "1675798888416144", "updated_at": "1675804719037018", "http_status_code": "200", "account_id": "123" }, "status": { "code": 200 } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | The webhook ID. | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Complete description of the webhook. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | | `event_types` | The event type the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | | `account_id` | The dbt account ID. | | ##### Test a webhook Test a specific webhook. ###### Request ```shell GET https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id}/test ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want to test. | ###### Response sample ```json { "data": { "verification_error": null, "verification_status_code": "200" }, "status": { "code": 200 } } ``` ##### Delete a webhook Delete a specific webhook. ###### Request ```shell DELETE https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id} ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want to delete. | ###### Response sample ```json { "data": { "id": "wsu_12345abcde" }, "status": { "code": 200, "is_success": true } } ``` #### Related docs * [dbt CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Use dbt's webhooks with other SaaS apps](https://docs.getdbt.com/guides.md?tags=Webhooks) #### Troubleshooting If your destination system isn't receiving dbt webhooks, ensure it allows Authorization headers. dbt webhooks send an Authorization header, and if your endpoint doesn't support this, it may be incompatible. Services like Azure Logic Apps and Power Automate may not accept Authorization headers, so they won't work with dbt webhooks. You can test your endpoint's support by sending a request with curl and an Authorization header, like this: ```shell curl -H 'Authorization: 123' -X POST https://<your-webhook-endpoint> ``` --- ### Weekly dbt single-tenant release notes dbt platform | Free dbt Single-tenant release notes for weekly updates. Release notes fall into one of these categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by date for single-tenant environments. ![RSS](/img/fontawesome/rss.svg)Subscribe to release note updates via [RSS](https://docs.getdbt.com/feeds/release-notes-st-rss.xml), [Atom](https://docs.getdbt.com/feeds/release-notes-st-atom.xml), or [JSON Feed](https://docs.getdbt.com/feeds/release-notes-st-rss.json). #### October 28, 2026 #### New ##### dbt platform * **Private connection for a GitHub Enterprise Server app you manage**: If your GitHub Enterprise Server can only be reached over a private connection (PrivateLink), you can now save that connection on the GitHub app. Set `private_link_endpoint_id` when you create or update the app through the API, and dbt routes the connection over your private network. ##### Orchestration and run status * **Filter the Model timing table:** On the Model timing tab, you can now filter [Resource details](https://docs.getdbt.com/docs/deploy/run-visibility.md#resource-details) to find what's slowing a run down. Use the **Est. critical path** filter to show only the resources on the longest chain of dependent resources, which sets the minimum run time (Yes), or everything else (No). Use the resource type filter to narrow the list to **Model**, **Test**, **Snapshot**, or **Exposure**. You can still search by name. ##### dbt v2 * **Set the environment for a Mesh dependency**: You can now route cross-project refs to a specific upstream environment, such as Staging, instead of only the default Production or Staging environment. * **Files that end with a Jinja suffix**: Set [`allow_jinja_file_extensions`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) in your project config so dbt reads files with an extra Jinja ending, such as `model.sql.j2` or `doc.md.j2`. Jinja is the templating language dbt already uses inside SQL and Markdown files. ##### dbt AI and agents * **Bookmarkable dbt Wizard home conversations**: In the dbt Wizard [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), the conversation you have open appears in the browser URL. You can bookmark that conversation and open it again from the URL. #### Enhancements ##### dbt platform * **Clearer message when you cannot select dbt v2**: If a project can use dbt v2 but you do not have permission to select it, the dbt v2 choices stay visible and disabled in the environment, job, and development version menus. The tooltip says that an Account Admin or a [v2 Migration Admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin) has to select dbt v2. ##### Orchestration and run status * **Definitions on the Model timing tiles**: The six [summary tiles](https://docs.getdbt.com/docs/deploy/run-visibility.md#metric-tiles) on the Model timing tab now have a question-mark icon: **Est. critical path**, **Peak concurrency**, **Avg active models**, **Longest model**, **Wall clock**, and **Latest start**. Select the icon to read what that number means. * **Retry when a usage chart does not load**: If a [usage chart](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#viewing-usage-in-the-product) fails to load or refresh, you see a message and a **Retry** button. This applies to the monthly models built chart on **Project home** and the top jobs chart on an **Environment** page. A chart that loaded earlier keeps its last numbers and shows when they were updated while the retry runs. ##### Studio IDE * **dbt v2 Stable no longer shows Preview**: In Studio [User development settings](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#override-dbt-version), the **v2 Stable** option no longer shows a **Preview** label. dbt v2 is generally available. ##### dbt v2 * **Resize columns in query results**: In the dbt VS Code extension, drag the edge of a column header in the query results panel to make that column wider or narrower. #### Fixes ##### dbt platform * **Signup tells you when an email is already taken**: If a signup uses an email address that already has an account, including when two signups use that email at the same time, dbt shows a page that says the email is already taken. You no longer get a generic error page. ##### Identity, APIs, and Administration * **Clearer errors for an invalid API request**: A request with a bad value now returns a clear error instead of a server error. An unsupported `order_by` value on version 3 list endpoints, and a negative or too-large `limit` or `offset` on version 2 list endpoints, return a 400 error (the request was invalid). Creating a project with `POST /api/v2/accounts/{id}/projects/` also returns a 400 error when `type` is not text. Getting run timing with `GET /api/v3/accounts/:id/projects/:id/runs/:id/timing/` returns a 404 (not found) when that run is not in the account or project in the URL. ##### Studio IDE * **Ephemeral models show in command results**: In Studio IDE, an [ephemeral](https://docs.getdbt.com/docs/build/materializations.md#ephemeral) model is a model that is not built as its own table or view. Those models were missing from the command result list, so a tab such as **Error**, **Warn**, **Skip**, **Reused**, or **Pass** could show a count and no rows. They now appear on the matching tab with the other models. ##### Orchestration and run status * **Model logs on the System logs tab**: Open a job run, select a step, and open **System logs**. That tab now includes the logs from your models, so you can troubleshoot a failed run in one place. ##### dbt v2 * **`dbt fmt` keeps a stable indentation**: [`dbt fmt`](https://docs.getdbt.com/reference/commands/lint.md#dbt-format) no longer shifts a Jinja tag to the left when that same line ends with a SQL comment. It also settles on one indentation for Jinja tags inside a `for` loop, and for an `{% if %}...{%- else -%}` block whose condition looks up warehouse information, such as the columns on a table. Running `dbt fmt` again does not keep changing those lines. #### Behavior changes ##### Orchestration and run status * **System logs now include model logs**: The **System logs** tab on a run step used to show only system messages. It now shows every log for that step, including logs from your models. If you used this tab for system messages only, you will see more lines there. ##### dbt v2 * **New names for three environment variables**: dbt v2 renamed the variable that selects an AI provider, and the two variables that write the [information schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md#generating-the-information-schema) (a local copy of project metadata you can query). If you set `DBT_AI_PROVIDER`, change it to [`DBT_ENGINE_AI_PROVIDER`](https://docs.getdbt.com/docs/dbt-ai/package-skills.md#set-the-ai_provider-flag). If you set `DBT_GENERATE_INFO_SCHEMA` or `DBT_INFO_SCHEMA_DIR`, change them to `DBT_ENGINE_GENERATE_INFO_SCHEMA` and `DBT_ENGINE_INFO_SCHEMA_DIR`. The command-line flags for these options are unchanged. Update the variables anywhere you set them, including continuous integration (CI). #### October 21, 2026 #### New ##### dbt platform * **dbt State allowance and trial email notifications**: Billing admins now receive proactive email notifications about dbt State usage, including a warning at 80% of the monthly allowance (with a projected run-out date), a notification when the allowance is fully used and State is paused, and a reminder 7 days before a trial expires. All emails include plan-specific messaging and calls to action. ##### Insights * **Cost Insights ingestion failure email alerts**: Account admins now receive an email when Cost Insights ingestion has been failing for 3 or more cycles, with follow-up emails at 14 and 28 days if it continues. #### Enhancements ##### dbt platform * **dbt State usage graph period selector**: The dbt State usage graph now includes a dropdown to select the reporting period: this month, previous month, last 30 days, last 60 days, or last 90 days. The chart x-axis also shows month abbreviations when the selected range spans multiple months. ##### dbt AI and agents * **Turn duration visible in Wizard conversations**: dbt Wizard now displays how long each turn took at the bottom of every completed turn. For previously loaded conversations, the completion timestamp is also shown. #### Fixes ##### APIs, Identity, and Administration * **SCIM `licenseType` updates applied to pending invites**: If you manage access through SCIM (your identity provider, like Okta or Microsoft Entra ID), you can change the user's license type before they even log in. Previously this only worked for active users, but now it works for pending and not-yet-fully-signed-in users too. dbt still checks that you have an available seat of that license type before applying any change. * **Clearer API validation errors**: API requests with an invalid permission value or a malformed body now return a descriptive error instead of a server error. #### October 19, 2026 #### New ##### APIs, Identity, and Administration * **Default license for new users**: Admins can now set the [default license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#default-license-for-new-users) for new users who aren't assigned a license through SSO, SCIM, or license mapping. Choose Developer, Read-Only, or Analyst. Admins should check their settings and update the default as needed. This is available for all single-tenant accounts. #### October 14, 2026 #### Enhancements ##### APIs, Identity, and Administration * **Account connections page**: You can now sort columns, search by connection name, and filter by connection type, credentials, and connection method (private link or OAuth). ##### dbt State * **dbt State trial email**: When a dbt State trial ends, you'll get an email that communicates your Consumption or Enterprise plan and includes the allowance end date when your plan has one. #### Fixes ##### Insights * **Dates on Insights charts**: Charts now show dates such as `2026-01` or `2026-01-15` from oldest to newest. Tables keep your query's row order until you sort a column. #### Behavior change ##### APIs, Identity, and Administration * **SCIM tokens require a signed-in user**: You can no longer create a System for Cross-domain Identity Management (SCIM) token with a personal access token or a service token. Sign in to dbt and [create a SCIM token](https://docs.getdbt.com/docs/platform/manage-access/scim.md#set-up-dbt). #### October 7, 2026 #### New ##### dbt AI and agents * **Explore mode**: [Explore mode](https://docs.getdbt.com/docs/platform/wizard-home.md#ask-questions-in-explore-mode) is now available in preview in the dbt Wizard [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md#ask-questions-in-explore-mode) and [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#ask-questions-in-explore-mode). Ask questions of governed production data in plain language, with the SQL or metric definition behind every answer. * **Invite read-only users to the Wizard home tab**: You can now [invite](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) read-only users to ask questions about your data in the Wizard home tab without a developer license. ##### APIs, Identity, and Administration * **Filter jobs by ID list**: The jobs list API endpoint (`GET /api/v2/accounts/{account_id}/jobs/`) now accepts a `pk__in` query parameter, letting you retrieve up to 500 specific jobs by ID in a single request. * **ClickHouse available in the Administrative API (private beta)**: You can now specify `clickhouse_v0` as the adapter\_version when creating connections and credentials, filtering connections, and retrieving adapter schemas in the Administrative API v3. Contact your account representative to enable. ##### Semantic Layer * **Multi-term metric search**: The `metrics` and `metricsPaginated` GraphQL queries now accept a `searchTerms` argument, letting you filter metrics by multiple terms in a single request with correct pagination. Note: `search` and `searchTerms` cannot be combined, and a maximum of 20 terms is allowed. #### Enhancements ##### dbt AI and agents * **Better file search for Wizard @-mentions**: Workspace @-mention queries now use server-side fuzzy matching instead of filtering a local list, returning better-ranked results for large repositories. The context file picker also prevents selecting stale results while a search is still in flight. * **Table blocks in reports**: You can now add a `table` block type to `compose_report` reports to render query results as formatted, scrollable tables. Columns can be narrowed and reordered using the `columns` field, and the display caps at 50 rows on screen with a truncation notice for clipped results. ##### Catalog * **Column transformation type in lineage drawer**: You can use the new transformation type badge on column cards to see how each column is derived. ##### Webhooks * **Faster webhook job picker on large accounts**: The webhook subscription job picker now searches and paginates server-side. You see a search box with debounced filtering and scroll-to-load pagination instead of a slow or frozen dropdown when your account has many jobs. ##### dbt platform * **dbt State no longer in preview**: The dbt State navigation option no longer carries a "Preview" badge. Incomplete projects (those without a repository or skipped setup) are also excluded from automatic project selection on the dbt State home page. * **Microsoft Entra Application API exposes `client_id` and `tenant_id`**: The GET, POST, and PATCH endpoints for Microsoft Entra applications now return `client_id` and `tenant_id` in their responses, giving you visibility into your Azure DevOps (ADO) integration configuration. #### Fixes ##### dbt AI and agents * **Wizard diff card no longer crashes on malformed patches**: When the Wizard receives a malformed diff patch, it now displays a "Couldn't parse diff" error badge instead of crashing. * **Correct chart axis labels for decimal data**: Charts with decimal data values (for example, 25.43 minutes) no longer push the y-axis name outside the visible canvas area. The axis gap is now calculated from rounded tick values. * **Horizontal bar chart axis name no longer overlaps the legend**: The x-axis name, grid bottom margin, and legend centering on horizontal bar charts are now computed correctly so the axis label no longer overlaps the legend. ##### Cost Insights * **Cost Insights no longer stuck on "Preparing"**: The Cost Insights dashboard no longer shows a perpetual loading spinner when partial cost data exists. You now see available cost data even when some rows have not been fully processed. #### September 30, 2026 #### New ##### Orchestration and run status * **State Explain tab available for all dbt State runs**: The job run details page now shows an [**Explain** tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#explain-tab) for every dbt State run. The tab is available while a run is in progress and updates as resources finish. #### Enhancements ##### Catalog * **SQL transformation expressions in column lineage**: The [column lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md#column-lens) panel now shows SQL transformation expressions for all users. You can see how columns are derived across your lineage without any additional configuration. ##### Insights * **UTC timestamps in account Insights table**: The "last run finished at" column in the account Insights table now displays times in UTC, giving you a consistent reference regardless of your local timezone. **Accurate Cost Insights build counts**: Cost Insights no longer counts nodes that never executed as builds (for example, from a cancelled or failed run), aligning that count with dbt State. You may see lower build counts for affected run dates after this change. * **Snowflake Adaptive warehouse cost tracking for all users**: Per-query Snowflake Adaptive warehouse costs are now tracked automatically for all accounts in [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) when `QUERY_METERING_HISTORY` is accessible, with no additional configuration required. ##### Orchestration and run status * **dbt State mode in job settings**: Job settings now always show a dbt State mode dropdown with inherit, on, or off. This replaces the previous per-job checkbox. You can also set dbt State as the default for a deployment environment. ##### dbt AI and agents * **Correct reasoning block order after conversation reload**: AI reasoning blocks in agent chat now render in the correct order relative to answer text, both during live streaming and after you reload the conversation. ##### Semantic Layer * **More actionable BigQuery connection errors**: BigQuery authentication failures now surface as distinct 401 errors, and access-denied failures surface as 403 permission errors, instead of generic data platform errors. You get a more specific message to help diagnose credential and permission issues. * **Clearer Redshift connection errors**: When a Redshift connection fails because the credentials are wrong, or a role does not have the required permissions, you now see a specific error message instead of a generic failure. ##### dbt platform * **External metadata ingestion toggle shown only when enabled**: The external metadata ingestion checkbox in connection settings is now hidden unless it is already enabled for that connection. The toggle has also been removed from the account-level settings page. #### Fixes ##### Orchestration and run status * **Warned models included in Model Timing tab**: Models with a "Warned" status now appear correctly in the [Execution timeline](https://docs.getdbt.com/docs/deploy/run-visibility.md#execution-timeline) Gantt chart. The [critical path, peak concurrency, average active, and longest model](https://docs.getdbt.com/docs/deploy/run-visibility.md#metric-tiles) summary tiles now include warned models in their calculations. #### September 23, 2026 #### Enhancements ##### dbt AI and agents * **Monthly spend limit of $0 for dbt Wizard**: You can now set a $0 monthly [spend limit](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md#manage-your-spend-limit) for dbt Wizard, which stops all paid usage account-wide immediately. Selecting a limit below your current period's accrued spend no longer blocks submission. dbt platform displays a warning instead, and the limit takes effect right away. ##### dbt platform * **Committed spend breakdown by product**: The committed spend card on the dbt State and AI Usage tabs now shows a per-product breakdown of how much committed spend has been drawn by dbt State and dbt Wizard respectively. * **Billing period day counter**: The billing period card in workspace settings now shows a "Day X of Y" counter instead of "X days left." The total days are calculated from the actual billing window length, so the counter stays accurate for billing periods shorter or longer than 30 days. * **Rolling 12-month window in Models built chart**: The Models built (Enterprise Usage by Month) chart now always displays a rolling 12-month window ending at the current month. Expired or completed billing plans no longer anchor the window to the plan end date. * **Clearer GitHub outage errors**: GitHub integration flows now return a consistent HTTP 503 response with a human-readable message ("GitHub is temporarily unavailable. Please try again in a few minutes.") during GitHub outages, timeouts, and upstream 5xx errors, instead of raw or generic failures. ##### APIs, Identity, and Administration * **Pagination for List Service Tokens API**: The `GET /api/v3/accounts/{id}/service-tokens/` endpoint now accepts `limit` and `offset` query parameters. Query with no parameters specified will continue to return all service tokens to preserve backwards-compatibility for existing callers. #### Fixes ##### dbt platform * **Email addresses stay in the invite panel**: Email addresses you type in the invite panel stay in place when the list of available groups reloads. * **Actionable error for adapter version incompatibility**: When a connection test fails because an adapter requires a higher dbt version than the environment provides, you now see a clear HTTP 400 error message instead of a generic HTTP 500 server error. * **Actionable error for managed repository download failures**: When a managed repository cannot be downloaded due to a GitHub App permissions issue, you now see the message "This managed repository couldn't be downloaded due to a dbt Labs configuration issue. Please contact support." instead of a silent or generic failure. * **Correct dbt version when an adapter does not support dbt v2**: When you select an adapter that is not compatible with dbt v2 during new project setup, a warning banner appears and the development environment is created with the latest compatible dbt version. #### September 9, 2026 #### New ##### Catalog * **Column counts in the models table**: The models table in Catalog navigation now shows a column count for each model. #### Enhancements ##### Studio IDE * **Clearer missing credentials prompt**: Studio IDE now detects missing development credentials at startup and shows a **Development Credentials Required** modal with a direct link to add credentials, instead of failing silently. ##### Catalog * **Upstream sources sorted by freshness severity**: The **Upstream Sources** table on the model detail pages now sorts by freshness severity by default (Error first, Pass last). Click **Name** or **Status** to change the sort order. The **Status** column also no longer overflows on wide screens. ##### Orchestration and run status * **Full model timing view for large runs**: The model timing Gantt chart no longer limits groups to 2,000 rows. You can now see all models in the timing view for large runs. ##### dbt platform * **Analytics connection visible to read-only users**: Read-only users can now see their assigned analytics connection in project settings instead of **Not configured**. * **dbt Wizard overage email notifications**: Account admins now receive email when dbt Wizard usage credits are exhausted, with separate notices for accounts that have no usage commitment and accounts that have used their full commitment. ##### APIs, Identity, and Administration * **Clearer GitLab unavailability errors**: When a GitLab host is unreachable, affected API endpoints now return HTTP 503 with the message "GitLab is unavailable, please try again." instead of an unclear failure. This applies when you create a repository or list GitLab groups. #### Fixes ##### dbt AI and agents * **Pinned model names in context card**: Pinned models in the dbt Wizard context card now display the model name (for example, `customers`) instead of the raw metadata unique ID (for example, `model.jaffle_shop.customers`). ##### APIs, Identity, and Administration * **Correct error for cross-account write attempts**: Cross-account write attempts now return HTTP 403 Forbidden instead of a generic 500 Internal Server Error, so you get a clear rejection when the account doesn't match. * **Corrected Azure DevOps account linking error**: The error for a missing Azure DevOps account link now shows the correct message: "Missing Azure user; link your Azure DevOps account in your personal profile." #### Behavior change ##### APIs, Identity, and Administration * **Account creation blocked for service tokens and OAuth tokens**: Creating accounts via `POST /api/v2/accounts/` is now blocked for service tokens, account-scoped personal access tokens (PATs), and OAuth access tokens. Service tokens and account-scoped user API tokens receive HTTP 400; OAuth access tokens receive HTTP 403, with explicit error messages in each case. ##### Insights * **Cost Insights test counts now include unit tests**: Cost Insights aggregates now include unit tests when you filter by the Test resource type. Previously, unit tests were silently excluded, causing lower-than-expected execution counts. #### September 2, 2026 #### Enhancements ##### APIs, Identity, and Administration * **Analyst Read permission set available to all accounts**: The [Analyst Read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set is now available to all accounts without requiring a feature flag. You can assign it to groups so read-only users can view Catalog and project configuration such as connections, environments, and Semantic Layer settings. ##### Semantic Layer * **Clearer Snowflake authentication and permission errors**: When you connect to Snowflake through the Semantic Layer, authentication failures and permission errors now return distinct messages prefixed with `[WAREHOUSE_AUTHENTICATION_FAILED]` or `[WAREHOUSE_PERMISSION_DENIED]`, so you can tell credential issues apart from missing grants. #### Fixes ##### Orchestration and run status * **Clearer invalid cron expression errors**: Saving a job with an invalid day-of-month value such as `*,L` now shows a validation error and prevents the broken schedule from being saved. Use either `*` or `L` in the day-of-month field, not both. #### Behavior change ##### APIs, Identity, and Administration * **Unscoped group and service token permissions rejected**: When you assign a project-scoped permission set to a group or service token, you must now specify either all projects or a specific project. Requests that leave project scope unset return a `400` error. Account-level permission sets such as Billing Admin and Notification Manager are not affected. Existing legacy assignments continue to work until you change them. #### August 26, 2026 #### New ##### Orchestration and run status * **dbt State explain tab on run details**: A new "State explain" tab on the run details page shows dbt State's decision for each model in a run (rebuilt, reused, or cloned), with expandable details, search, and Comma-Separated Values (CSV) download. You can use this tab to investigate why each model was rebuilt or reused. Contact your account manager to enable. #### Enhancements ##### dbt platform * **Clearer billing admin guidance for non-admins**: Non-admin users now see an "Ask an admin to enable" message on the dbt Wizard and dbt State cards in Billing & Usage when a trial is available but they lack permission to start it, instead of a blank space. The same message appears if a non-admin tries to start a trial from a dbt State or dbt Wizard link. ##### Orchestration and run status * **dbt State available on more release tracks**: dbt State is now available for jobs running on the Compatible, Fusion Extended, and Fusion Fallback release tracks, in addition to previously supported tracks. * **Corrected Fusion release track names**: The display names for dbt Fusion release tracks are now "Fusion Stable" and "Fusion Nightly" instead of the previous reversed labels "Stable Fusion" and "Nightly Fusion". * **Large compare results in pull request comments**: When compare results are larger than 50 MB, pull request comments now show a "too large to summarize" notice with a link to the full compare report, instead of failing with no message. ##### Studio IDE * **Correct browser tab title**: The Studio IDE browser tab now displays "dbt Studio" instead of a generic editor title. ##### Integrations * **More reliable MCP OAuth sign-in**: When you connect an MCP client with OAuth, more clients can now complete sign-in successfully. #### August 19, 2026 #### Enhancements ##### Studio IDE * **Console tab persists across sessions**: New sessions open on the Wizard tab when available, and the Studio IDE remembers your last-used tab for each project so you can pick up where you left off. ##### Catalog * **Exact model relation name in the Discovery API**: A new `relationName` field on the `ModelAppliedStateNode` and `ModelAppliedStateNestedNode` GraphQL types exposes the fully-qualified, adapter-rendered relation name (for example, `"database"."schema"."model_name"`) from the last successful model build. #### August 12, 2026 #### Enhancements ##### dbt AI and agents * **Auto-expanding Wizard chat input**: The Wizard chat input grows vertically as you type or paste text, and shrinks back when content is removed. ##### APIs, Identity, and Administration * **Job read access is now included in the `account:read` OAuth scope**: Applications authorized with `account:read` can now read job data without also requesting the `jobs:run` scope. #### August 5, 2026 #### New ##### dbt AI and agents * **dbt Wizard home tab**: [The dbt Wizard home tab in dbt platform](https://docs.getdbt.com/docs/platform/wizard-home.md) is now available in public preview. You can build and change dbt projects through natural language, with inline diffs, DAG previews, and validation built in. #### Enhancements ##### dbt AI and agents * **Auto-open diff preview in edit-automatically mode**: When dbt Wizard edits files automatically, the diff preview side pane now opens immediately so you can review changes without an extra click. When the side pane is not available, for example in non-fullscreen Studio IDE, the diff falls back to an inline card. ##### Orchestration and run status * **Run history refreshes automatically after a trigger**: After you trigger a run or rerun, the run history list now polls every 2.5 seconds until the new run appears, eliminating the need for a manual page reload. Polling stops automatically once the run is visible or after 45 seconds. #### Behavior change ##### Catalog * **30-day model staleness removed from health criteria**: Models are no longer flagged as unhealthy solely because they have not been rebuilt in the past 30 days. The "Stale state" warning banner no longer appears on resource detail pages, and models whose only health issue was staleness now show as healthy in the Trust Signals badge. Source staleness is unchanged and continues to surface as a health issue for sources. #### July 29, 2026 #### New ##### Semantic Layer * **External OAuth for Redshift with AWS IAM Identity Center**: Semantic Layer development connections to Redshift now support external OAuth using Okta or Microsoft Entra with AWS IAM Identity Center, in addition to username and password. #### Enhancements ##### APIs, Identity, and Administration * **Clearer SCIM error messages**: System for Cross-domain Identity Management (SCIM) API errors now include the user email addresses that caused seat or license failures, so you can identify which users blocked provisioning. #### Behavior change ##### APIs, Identity, and Administration * **GraphQL complexity limit now enforced**: Semantic Layer GraphQL queries that exceed the complexity limit of 200,000 now return an error instead of completing with a warning. If you hit this error, request fewer fields, use pagination, narrow your filters, or split the query into smaller ones. #### July 22, 2026 #### New ##### dbt AI and agents * **Unified `get_node_details` MCP tool**: The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#discovery) now uses one `get_node_details` tool for all resource types. The older type-specific tools are deprecated and will be removed in a future release. ##### Orchestration and run status * **Job deactivation reason in banners**: When a job is deactivated, the banner now shows a specific reason — repeated run failures, account inactivity, or a generic fallback — with tailored reactivation instructions for each case. #### Enhancements ##### dbt AI and agents * **Reliability improvements for wizard platform**: The agent now automatically retries transient LLM provider failures — network timeouts, rate limits, and server errors — with exponential backoff, so brief provider blips are less likely to surface as errors during your session. * **Unlimited client tool loop iterations**: Client tool loops now run until the agent finishes rather than stopping after 50 iterations, eliminating premature termination of long-running agentic workflows. * **Copy button on code blocks in dbt Wizard**: Code blocks in dbt Wizard responses now include a Copy button on hover, so you can reuse generated SQL or YAML more easily. * **Clearer error for Bring-Your-Own-Key models that don't support embeddings**: When a Bring-Your-Own-Key (BYOK) OpenAI model is configured with a deployment that does not support embeddings (for example, a `gpt-4o` Azure deployment), the similar models feature now returns an actionable error message prompting you to use a text-embedding model instead of a generic internal error. ##### Catalog * **Directional lineage filtering in `get_lineage`**: The `get_lineage` tool now accepts a `direction` parameter (`upstream`, `downstream`, or `both`) to narrow results to only ancestors or only descendants of a target node, reducing response size for large graphs. The response also now includes a `description` field on each returned node. * **Metric filtering by metadata in `list_metrics`**: The `list_metrics` tool now accepts a `meta_filter` parameter to restrict results to metrics whose `config.meta` contains specified key-value pairs (for example, `{"agent_accessible": true}`), keeping result sets small enough to preserve description and metadata in the response. * **Health status filtering for applied models**: The `ModelAppliedFilter` input type now includes a `health` field, letting you filter applied models by health status (`unknown`, `degraded`, `caution`, or `healthy`) directly in the Discovery API. * **`warn` run status surfaced in model execution info**: The `RunStatus` enum and the `lastRunStatus` field on model execution information now include `warn`, so models whose last run completed with warnings correctly reflect that status. * **Warn run status available as a filter**: The run status filter panel now includes a **Warn** option alongside **Success**, **Error**, **Skipped**, and **Reused**. * **Default search environment matches page context**: When searching from within a project environment route (for example, a Staging page), the Catalog search now defaults the environment filter to that environment type rather than always defaulting to Production. * **Redesigned search result cards**: A redesigned search result card replaces tooltip-based match pills with inline expandable snippets for columns, tags, descriptions, and code matches. Please contact your account manager to enable. ##### Insights * **Snowflake Adaptive warehouse cost support**: Cost Insights can now attribute query costs to models run on Snowflake Adaptive warehouses using the `QUERY_METERING_HISTORY` table. Without access to this table, Adaptive warehouse queries were previously recorded as $0. The connection test now also checks and reports on `QUERY_METERING_HISTORY` access so you can diagnose missing attribution before it affects cost data. Please contact your account manager to enable. ##### Studio IDE * **Fusion Stable is now the default track**: The `latest-fusion` release track is now Fusion Stable across all settings. Existing configurations have been updated automatically. No action is needed. * **Environments already on Fusion no longer see upgrade checkbox**: On the Enable Fusion Environments page, environments already running Fusion now show a disabled checkbox, preventing unnecessary saves. * **More specific error messages on failed Fusion environment upgrades**: When saving a Fusion upgrade fails, the platform now displays the top-level user message from the API instead of internal field-level error details. * **Faster command status updates in Studio IDE**: The command panel now shows live status updates as commands run, so you see progress sooner without waiting for a refresh. ##### dbt platform * **Consumption pool card renamed and repositioned**: The "Committed spend" card is now labeled "Consumption pool" with copy explaining that usage-based features like dbt State draw from it. The card now appears between the current plan metric tiles and the product-specific sections on billing Overview and usage tab pages. * **dbt State DATT chart shows billable and free usage separately**: The Daily Active Target Tables (DATTs) chart now stacks billable and free series, so trial users whose usage is entirely free see real bars instead of an empty chart. ##### Orchestration and run status * **Reduced out-of-memory rates in Fusion**: Memory-tuning optimizations are now applied automatically to all Fusion runs, reducing out-of-memory kill rates and improving overall uptime. #### Fixes * **More reliable Claude responses**: Claude-backed agents can return longer answers and handle some previously broken interactions more reliably. #### July 15, 2026 #### New ##### Orchestration and run status * **Hybrid job type**: You can now create hybrid jobs to track runs triggered by an external orchestrator. Hybrid jobs have a simplified setup that omits execution steps, triggers, advanced settings, and cost-optimization controls. They display **Externally triggered** as their next-run schedule and are available only for projects configured as [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md). #### Enhancements ##### Orchestration and run status * **Faster linting for Fusion-version runs**: Runs using a Fusion dbt version now invoke the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. Fusion virtual environments do not include SQLFluff, so linting now works for all Fusion-version runs and runs faster. ##### dbt AI and agents * **Compaction indicator during context optimization**: When the agent compresses conversation context in the background, a spinner labeled **Optimizing conversation context…** now appears in the chat area. Submitting new messages and stopping the agent are disabled while compaction is in progress to prevent conflicts. * **Wizard unavailable screen replaces generic "not enabled" message**: When [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is unavailable (not activated, trial expired, or spend limit reached), Studio IDE now shows a dedicated screen with the specific reason and an appropriate action instead of a generic message. ##### APIs, Identity, and Administration * **Server-side user search and pagination in account settings**: The users table, group member lists, and user edit drawer now search, filter, sort, and paginate server-side. On large accounts, all users are findable by name, email, or license type, group member search no longer misses results beyond the first page, and users beyond the first page can be opened and edited in the user edit drawer. ##### dbt platform * **Global account discovery generally available**: The **Enable global account discovery** setting on the **Account settings** page is now visible to all entitled accounts without requiring a feature flag. You can allow or restrict account discovery from [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). * **Connection overrides visible in profile view mode**: Credential-level [connection overrides](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) (such as Databricks catalog, Snowflake warehouse, role, and database) are now surfaced as a read-only **Connection overrides** section in the profile details view, without requiring you to open the edit form. #### Fixes ##### Orchestration and run status * **Runs no longer stuck in "running" after OOM kill**: When a run pod is Out of Memory (OOM)-killed and restarted, the platform now passes the correct status code and message to the config API so the run transitions to a failed state in the dbt platform UI instead of remaining **running** indefinitely. ##### Semantic Layer * **More reliable SSH tunnel connections**: The Secure Shell (SSH) connection and authentication timeouts for Semantic Layer data platform connections are now 30 seconds (previously 1 second). If your bastion host or network path has higher latency, you will no longer experience deterministic connection failures. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information. * **Compile SQL rejects oversized query shapes**: Some types of Compile SQL queries are now rejected if they are too complex. If a request fails with a validation error, try reducing the number of metrics or group-by dimensions in the query. #### Behavior change ##### APIs, Identity, and Administration * **Service token creation now requires service token authentication**: You can no longer create a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) using an account-scoped [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) (PAT). Requests authenticated with a PAT now return a `400` error. Use a service token to create new ones instead. #### July 13, 2026 #### Behavior change ##### Billing * **Improved billing email alerts**: You’ll now receive email alerts when you’re drawing down on your consumption — whether that’s from models built or newer consumption features like dbt State. #### July 8, 2026 #### Enhancements ##### dbt AI and agents * **User list search and group filtering**: The user listing API now supports a `search` parameter (case-insensitive substring match across email and name) and a `group_id` parameter to retrieve only members of a specific group, enabling paginated group-member lookups for large accounts. * **Expanded Analyst Read permission set**: The [Analyst Read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set now includes the project and account read permissions analysts need to browse catalog and configuration without write access. #### Fixes ##### Orchestration and run status * **Clearer errors for malformed dbt commands**: A dbt step with invalid command syntax, such as an unclosed quote, now returns a user-facing syntax error instead of a generic unhandled exception. #### July 1, 2026 #### Enhancements ##### APIs, Identity, and Administration * **SCIM user responses include standard group fields**: User responses from SCIM endpoints now include the standard `value` (group ID) and `display` (group display name) fields alongside the existing `id` and `displayName` fields, improving compatibility with SCIM spec consumers. * **Credentials page accessible with `user_credential_write` permission**: The credentials list and detail pages are now accessible to users who have `user_credential_write` permission even without `credentials_read`, supporting credential self-service flows. #### Fixes ##### dbt platform * **Email notifications accept addresses with special characters**: External email notification addresses with valid but non-standard local-part characters (for example, ampersands in `ops&alerts@example.com`) are now accepted instead of being rejected by the validator. ##### Orchestration and run status * **Clearer errors for oversized environment variable payloads**: Oversized custom environment variables now fail at run start with a clear error that names the largest offender, instead of crashing mid-run. Reduce the variable size and retry. ##### dbt AI and agents * **More reliable dbt Wizard conversations**: dbt Wizard now recovers automatically from transient errors that could previously interrupt a conversation, so long threads keep working. * **Accurate job and run scoping in dbt Wizard**: dbt Wizard now correctly scopes job and run investigations to your current project instead of returning results across your entire account. You no longer see unrelated jobs from other projects when asking dbt Wizard to investigate a run. #### Behavior change ##### dbt platform * **"Development credentials" renamed to "User credentials"**: All user-facing labels, section headings, tooltip text, and in-app messages that previously referred to "development credentials" now use "user credentials." * **Versionless migration banner removed**: The banner notifying users about migration to versionless dbt has been removed from the notification stack. #### June 24, 2026 #### New ##### Insights * **Cost breakdown by job**: Cost Insights now includes a Jobs table view alongside the existing all-models view. Use the **All**/**Jobs** toggle to switch between a per-model breakdown and a per-job summary, and select **Download** to export the active view as a comma-separated values (CSV) file. ##### Run Logs * **Download OpenTelemetry logs**: You can download OpenTelemetry (OTel) logs for individual dbt command steps in Fusion job runs. ##### APIs, Identity, and Administration * **Run history now scoped to projects**: You can now view live run, step, and log data scoped to a specific project. Results support filtering by step status and log type, with consistent pagination across all views. * **Presigned log download URLs**: Logs for completed run steps are now downloaded directly from storage rather than streamed through the service, improving download reliability and performance. Download links expire after 15 minutes. #### Enhancements ##### Studio IDE * **Find in files**: The [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#search-your-project) now includes search and replace functionality and a command palette, enabling you to quickly find and replace text across your project, navigate files, jump to symbols, and run IDE configuration commands. ##### Catalog * **Filter assets by run status**: You can now filter Catalog search results by the most recent run status of an asset — success, error, or skipped — making it easier to spot and investigate assets that may need attention. * **Model metadata preserved after failed or skipped runs:** Lineage, tests, descriptions, and other model metadata now persist correctly even when a run fails, is skipped, or reuses a prior result. Previously, these runs could overwrite stored metadata, causing tests to detach from their models and lineage to disappear. ##### dbt State * **dbt State now works on CI and merge jobs**: You can now enable the dbt State cost-optimization option on CI and merge jobs, not just deploy jobs. Previously this returned a validation error. * **dbt State credential management simplified**: Studio IDE now uses account-level dbt State credentials, removing the per-user provisioning step that previously ran on first use. ##### Semantic Layer * **Longer query timeout for Semantic Layer**: The query timeout has been doubled from 10 minutes to 20 minutes, reducing timeout errors for long-running queries. ##### APIs, Identity, and Administration * **[Administrative API v3](https://docs.getdbt.com/dbt-cloud/api-v3) now supports private endpoints**: [`list`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/List%20Private%20Endpoints), [`create`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Create%20Private%20Endpoint), [`retrieve`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Retrieve%20Private%20Endpoint), [`update`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Update%20Private%20Endpoint), and [`delete`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Delete%20Private%20Endpoint). Use these endpoints to manage private connectivity programmatically. * **Clearer error messages for service outages**: When a third-party service, such as a data warehouse, is temporarily unavailable, the dbt platform now returns a descriptive error message instead of a generic one, making it easier to diagnose connection issues. ##### Integrations * **BigQuery Workload Identity Federation falls back to connection impersonation**: When a BigQuery Workload Identity Federation (WIF) credential has no explicit `service_account_impersonation_url`, the platform now derives the URL from the connection-level `impersonate_service_account` field. This supports the Terraform `dbtcloud_global_connection` configuration pattern where impersonation is defined on the connection rather than the credential. * **Snowflake connection hostname normalized for TLS**: Snowflake account identifiers containing underscores (for example, `fdr_apac_dev`) are now hyphenated when building connection hostnames and OAuth URLs, preventing Transport Layer Security (TLS) hostname verification failures against Snowflake's wildcard certificate. * **Google SSO retries on transient token endpoint errors**: Google Workspace Single Sign-On (SSO) group refresh now retries once on 5xx responses from the Google token endpoint before surfacing an authentication error, reducing sign-in failures caused by transient Google API outages. * **Databricks OAuth retries on transient token endpoint errors**: The Databricks OAuth token refresh path now retries once on 5xx responses before surfacing a retryable error, making profile generation for Databricks OAuth connections more resilient to short Databricks outages. ##### dbt AI and agents * **dbt Model Context Protocol (MCP) semantic search for related models**: The `get_related_models` tool is now available in multi-project agent contexts, allowing the agent to search for semantically similar models across projects by resolving each project's production environment automatically. #### Fixes ##### Studio IDE * **Clearer errors for unconfigured development credentials**: Studio IDE now returns an actionable error when development credentials are not configured for an environment. * **More precise error responses for development environment setup**: The development environment endpoint now returns distinct HTTP status codes for missing project configuration (400), unconfigured development credentials (412), permission errors (401), and retrieval timeouts (504), rather than mapping multiple failure modes to the same error response. ##### APIs, Identity, and Administration * **Connection test restricted to authorized environments**: The connection test endpoint now validates that the environment ID in the request belongs to the account and project, returning a 404 for unrecognized environment IDs instead of silently proceeding. * **SSO redirect preserved after session expiry on logout**: When an unauthenticated user visits the logout endpoint on a multi-tenant cell, the platform now resolves the correct Single Sign-On (SSO)-mandatory provider from the request host and redirects to `/enterprise-login/<slug>` instead of stranding them on `/login/`. * **Publications handler handles cloud-config timeouts gracefully**: When a `GetCrossProjectEnvironment` or `ListEnvironments` call to cloud-config exceeds its deadline, the publications handler now returns an `UNAVAILABLE` status to the caller instead of propagating an unhandled error. You can retry the request after a short delay. ##### Semantic Layer * **More reliable Semantic Layer job result retrieval**: Semantic Layer job and paginated query results now deserialize more reliably, reducing failures when retrieving query results. * **Fixed database write errors for cache timestamps**: Resolves an issue where the cache engine could fail to persist timestamps because timezone-aware datetimes cannot be encoded into `TIMESTAMP WITHOUT TIME ZONE` columns by `asyncpg`. The fix ensures timezone-naive UTC timestamps are used for all database writes. * **Result-too-large Semantic Layer errors now return HTTP 400**: Semantic Layer queries that exceed the result size limit now return a bad request error instead of an internal server error. ##### dbt Copilot and agents * **Safer handling of non-JSON OpenAI error responses**: Error handling for OpenAI `BadRequestError` now gracefully handles responses with non-JSON bodies, preventing an unhandled exception when parsing the error code. You should see a proper error rather than an internal server error in these cases. #### Behavior change ##### APIs, Identity, and Administration * **Account-scoped credential reads enforced for Personal Access Tokens**: The user credentials endpoints now enforce account scoping consistently when a request uses an account-scoped Personal Access Token (PAT). Requests that previously returned credentials outside the token's account now return a 404. #### June 17, 2026 #### Enhancements ##### dbt AI and agents * **Live streaming for Wizard dbt command output**: [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) tool calls for dbt command invocations now stream their output live in chat, in both the Studio IDE and [Wizard home](https://docs.getdbt.com/docs/platform/wizard-home.md). * **OAuth scopes declared in Model Context Protocol resource metadata**: The Model Context Protocol (MCP) protected resource metadata endpoint now advertises the OAuth scopes it supports (`offline_access`, `account:read`, `projects:query`, `catalog:read`, `projects:develop`, and `jobs:run`). MCP clients that perform dynamic capability discovery can now request the correct scopes automatically. ##### dbt platform * **dbt State in development**: [Enable dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md#enabling-dbt-state-in-studio) in development environments, or override it per user in **User development settings**. Requires dbt State to be enabled on the account by an admin. * **dbt State disabled for unsupported warehouse adapters on jobs**: The dbt State checkbox in job settings is now disabled when the job's environment uses an unsupported warehouse adapter. A help icon displays a tooltip explaining the limitation, and the feature is automatically removed from the job payload on save. #### June 10, 2026 #### New ##### dbt Copilot and agents * **dbt documentation search and retrieval tools**: The [remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md) now includes `search_product_docs` and `get_product_doc_pages`. Available in and in agent clients connected to remote dbt MCP. Refer to the [product docs toolset](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md?version=2.0#product-docs). * **Dimension values lookup tool**: A new `get_dimension_values` MCP tool lets agents retrieve distinct values for a given Semantic Layer dimension, optionally scoped to specific metrics. Use this to discover valid filter values (for example, available regions or order statuses) before building a `where` clause in a `query_metrics` call. #### Enhancements ##### Studio IDE * **Workspace file operations API**: Adds public Studio file operation endpoints for `stat`, `get`, `put`, `list directory`, `delete`, `mkdir`, and `rename` under `/api/ide/v3/{environment_id}/files/`. File paths are passed as query parameters to avoid user paths appearing in traces. * **Environment status endpoint**: Adds a `/api/ide/v3/{environment_id}/status` endpoint that returns the dbt version and Fusion status for a development environment, allowing Studio to display version information without additional API calls. #### Behavior change ##### dbt platform * **Password management updates**: The in-page password change form has been removed from profile security settings, and the "Forgot password?" link no longer appears on the sign-in page. Password resets are now handled through the email-based reset flow, which can be initiated from your profile settings. #### June 3, 2026 #### New ##### dbt Copilot and agents * **Debug with Copilot from run and job surfaces**: A new "Debug with Copilot" button appears on failed run detail pages, runs lists, job details, environment runs, and the project home activity feed. Clicking it opens dbt Copilot or the full-page Wizard to investigate and debug the failed run. Please contact your account manager to enable. ##### dbt platform * **dbt State trial progress, stats, and usage on settings page**: The dbt State settings page now shows a trial progress bar (days elapsed of 30), monthly model reuse stats (models reused, build reduction percentage, and query run time reduction), and a model build chart. These sections appear when an account has an active dbt State subscription. * **Delete private link endpoint**: You can now delete private link endpoints from the endpoint details page. A confirmation modal requires you to type `DELETE` before the deletion proceeds. Please contact your account manager to enable. ##### APIs, Identity, and Administration * **OAuth consent improvements**: OAuth consent now recognizes the `identity:read` scope, displaying a "Read user details" label and description. Scopes limited to `identity:read` and `offline_access` no longer show the project access selector. #### Enhancements ##### dbt platform * **AI providers settings page consolidated**: The Copilot and Wizard settings pages are unified under a single "AI providers" page at `/settings/accounts/{id}/pages/ai`. The previous `/pages/copilot` URL redirects automatically, and the sidebar item and page title now use "AI providers." * **"Enable dbt State" checked by default on job create**: When creating a new job, the **Enable dbt State** checkbox is now checked by default on all environments when dbt State is available and an active subscription is present. * **dbt State model build chart adds "Reused (cloned)" series**: The dbt State model build chart now tracks three series — Built, Reused (no-op), and Reused (cloned) — giving a more detailed breakdown of model reuse. * **Teams notifications generally available**: Microsoft Teams notifications no longer require a feature flag. The Teams integration now appears in the OAuth integrations card and notification settings for all accounts. * **Private endpoints page shows Beta badge and updated info banner**: The private endpoints list and create pages now display a "Beta" badge in the header. The info banner on the create page is no longer dismissible and has updated copy clarifying that self-serve creation is available only for Snowflake AWS. ##### Orchestration and run status * **Reused node status in run results**: Studio IDE now recognizes and surfaces the `reused` node status in run results and metadata counts, giving you a more accurate picture of what ran during a dbt invocation. #### Fixes ##### Orchestration and Run Status * **Run list action buttons fixed and clickable**: Action buttons on the runs list (for example, "Debug with Copilot") no longer silently navigate to the run detail page instead of triggering the intended action. ##### Catalog * **Accurate health status filtering for stale assets**: The Catalog health filter now correctly classifies assets with a healthy bitmask but a last successful run older than 30 days as "Caution" instead of "Healthy." Assets whose last run was marked `reused` continue to be treated as healthy. #### Behavior change ##### dbt Copilot and agents * **Agent validates autofix with `dbt build` instead of `dbt compile`**: dbt Wizard autofix and model change validation is improved by defaulting to broader `dbt build` commands. #### May 27, 2026 #### New ##### Webhooks * **Disabled webhook subscriptions banner**: The webhooks settings page now shows a dismissible warning banner when one or more webhook subscriptions have been automatically disabled due to repeated failures. Disabled subscriptions now display an "Archived" badge with a tooltip explaining how to reactivate them. ##### Studio IDE * **Directory listing API**: The file browser now supports streaming directory listings in Newline-Delimited JSON (NDJSON) format, returning each file's name and type. The endpoint supports an optional `limit` parameter and `ETag` and `Last-Modified` headers to avoid re-fetching unchanged directory contents. * **File and directory deletion API**: You can now delete individual files or entire directory trees from the workspace. Recursive deletion requires `recursive=true` to be set explicitly, preventing accidental data loss. * **File and directory rename and move API**: You can now move files and directories within the workspace. An `overwrite` parameter controls whether an existing destination is replaced. The endpoint surfaces clear errors for missing sources, path traversal, name-too-long conditions, and directory conflicts. #### Enhancements ##### Webhooks * **Bounded webhook delivery history fetches**: Webhook delivery history reads are now capped at 1,000 records and limited to a 7-day lookback window, preventing unbounded memory growth from high-volume subscriptions. You should see more consistent performance for webhook history lookups on active subscriptions. ##### Integrations * **Smoother token refresh flow in OAuth consent page**: To provide a smoother experience with fewer steps, you only see the project selector for scopes that require project-level access. ##### APIs, Identity, and Administration * **Create account button shown for existing users**: The **Create account** button in the account switcher is now also shown to users who already have accounts, making it easier to create additional accounts. Please contact your account manager to enable. * **Private endpoint edit**: The private endpoint detail page now supports editing the endpoint name and port. An "Edit" button opens an inline form with validation, a confirmation modal, and clear error messaging. Please contact your account manager to enable. * **Private endpoint status badges with icons**: Connectivity status and endpoint state badges on the private endpoint list and detail pages now include status icon variants (success, error, in-progress, waiting, canceled, and health-unknown) for clearer at-a-glance status. #### Fixes ##### dbt Copilot and agents * **Protected edits to generated files**: dbt Copilot is now instructed not to edit files in `dbt_packages/`, `target/`, or `logs/`, directing fixes to source-controlled files instead. * **Build validation after column changes**: dbt Copilot now runs `dbt build` (not just `dbt compile`) after edits that add, rename, or alias columns, or change `ref` or `source` references, catching runtime errors that compilation alone would miss. ##### Webhooks * **Skipped invalid email addresses in model notifications**: Model-level notifications now validates email addresses before attempting delivery, skipping any invalid entries with a warning rather than proceeding with an invalid address. This ensures dispatching model notifications to correct email addresses. #### May 20, 2026 #### New ##### APIs, Identity, and Administration * **Notification Manager permission set**: A new `notification_manager` account-level role grants read and write access to job notification settings, Slack integration status, and member listings without requiring broader admin permissions. Refer to [Notification Manager](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#notification-manager) for more information. #### Enhancements ##### APIs, Identity, and Administration * **Cursor-based pagination for Discovery API job queries**: Job-based Discovery API queries now support cursor pagination with `first` and `after` arguments. Supported queries include `models`, `sources`, `seeds`, `snapshots`, `tests`, `macros`, `metrics`, and `exposures`. Use the returned `paginationCursor` value as `after` to fetch the next page. Page size defaults to and caps at 100. * **Support for the`lastKnownResults` filter accepting `null` values**: The `lastKnownResults` filter on `TestAppliedFilter` now accepts `null` as a value, allowing you to match tests that have no recorded result status. * **OAuth client audit log entries include registration type**: Audit log entries for OAuth client registration now append `(manual)` or `(dynamic)` to the description, making it clear whether a client was registered via dynamic client registration or manually — for example, "ChatGPT was registered for this account (manual)". ##### Orchestration and run status * **Clearer Fusion eligibility message for migrated jobs**: The Fusion eligibility surface now maps the `job_on_fusion` reason code to "This job is already running on Fusion." with no call-to-action, giving clearer feedback for jobs that are already migrated. * **Fusion job warning notifications enabled by default**: Warning notifications for Fusion runs are now always active and no longer require a feature flag, so you will see run-level warning statuses without any additional configuration. * **OpenTelemetry log format always on for Fusion runs**: Fusion runs now always use the OpenTelemetry (OTel) structured log format (except in shadow mode), removing the feature-flag requirement and ensuring consistent log output. ##### Insights * **Builds-only view when Cost Insights connection is disabled**: When your deployment environment's warehouse connection has Cost Insights disabled, the Insights page now shows a "Cost Insights is not enabled for this connection" banner and a Builds tab with the model build chart and environment and period selectors, instead of no chart content. ##### Catalog * **Connection-aware Cost Insights enablement**: Cost Insights now checks the environment connection and platform metadata credentials to confirm `cost_insights_enabled` is configured. If credentials are missing or disabled, cost metrics and tabs are hidden and a banner prompts you to configure platform metadata credentials. ##### Studio IDE * **Faster command history loading**: Command history now loads more quickly because the API returns only the data needed for the history view. ##### Deployment and configuration * **Environment permission elevation for the `job creator` role**: The `job_creator` role is now included in the set of roles that allow environment-level permission elevation, consistent with other development roles such as `developer` and `git_admin`. Refer to [Job creator](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-creator) for more information. #### May 13, 2026 #### New ##### APIs, Identity, and Administration * **Job creator permission set**: Adds a new [Job creator permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-creator) for users who need to create, edit, and run jobs within assigned projects and environments. Job creators have read-only access to environments and environment variables and cannot edit environment settings. ##### Studio IDE * **Hidden file support in file search**: You can optionally include dotfiles in file search results; search remains limited to your project tree. #### Enhancements ##### dbt Copilot and agents * **Admin API tools enabled by default**: The admin API toolset (job management and run operations) is now always available in the dbt Copilot dev agent and no longer requires a feature flag. You no longer need to contact your account manager to enable these tools. * **Improved Studio IDE agent prompt**: The Developer agent in Studio now uses dbt commands more efficiently (including fewer redundant `dbt ls calls` and correct `dbt show` limits), runs independent reads in parallel when appropriate, surfaces clear next-step choices instead of ending mid-action, stays within local git capabilities (no PR/GitHub promises), and formats SQL and YAML in copy-ready code blocks. ##### Orchestration and run status * **Fusion upgrade available banner on environment settings**: A new banner appears on the environment settings page when Fusion is available for a project but not yet enabled. Admins can click **Enable Fusion access** directly from the banner to open the upgrade confirmation modal. Users without the required permissions see a read-only message directing them to contact an admin. ##### Deployment and configuration * **Snowflake password authentication blocked for new credentials**: Password authentication is no longer available when creating new Snowflake development or deployment credentials. New credentials default to key-pair authentication. Existing credentials that still use password authentication display a warning banner and cannot be saved until you migrate to key-pair or OAuth. ##### Integrations * **Paginated OAuth client registration tables**: The App Integrations settings card now loads manually registered and dynamically registered OAuth clients from separate paginated queries (10 rows per page). Each table has independent pagination controls, and deleting the last row on a non-first page automatically returns you to the previous page. * **MCP endpoint URL displayed in account settings and App Integrations**: When the dbt Model Context Protocol (MCP) feature is enabled, your account's MCP endpoint URL is now shown in the Account settings page and in the App Integrations card with a copy button, making it easier to connect external AI tools to your dbt account. Contact your account manager to enable. ##### Catalog * **Multi-value test result status filtering**: The `TestAppliedFilter` input now supports a `lastKnownResults` field that accepts an array of test statuses, allowing you to filter applied tests by multiple result states in a single query. You can include `null` in the array to match tests with no known result. The previous scalar field `lastKnownResult` is deprecated; use `lastKnownResults` instead. For more information, refer to [Filter tests by last-known result](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-tests.md#filter-tests-by-last-known-result) #### Fixes ##### Orchestration and run status * **Repo cache hard reset applied unconditionally**: The git hard reset performed after restoring a repository cache is now always executed, removing a feature flag that previously controlled this behavior and ensuring a clean working directory after every cache restoration. * **Clearer error message for missing repository URL**: When a job cannot clone its repository because no remote URL is configured, the error message now explains the most likely causes — an invalid Git remote URL, a Git provider outage, or a deprecated HTTPS connection — and directs you to verify the URL, confirm your provider is operational, and ensure the repository uses SSH with deploy keys before retrying. ##### Studio IDE * **Accurate node status during runs**: During dbt runs in Studio, node status in run logs now updates correctly when evaluation events arrive, so in-progress state matches what's actually happening. * **Clearer git ref lock errors**: A few failure paths that used to surface as unclear errors now show messages that explain what happened and what to try next. ##### Deployment and configuration * **Clearer connection test error messages**: The connection test failure message now prompts you to verify your connection details and confirm that your credentials have access to the data warehouse, rather than showing a generic failure message. ##### Run logs * **Structured run logs enabled by default**: Structured run step logs are now always used when available, without requiring a feature flag. Runs that already have structured logs continue to display them; runs with only plaintext logs fall back to the legacy view automatically. #### May 6, 2026 #### New ##### dbt Copilot and agents * **Project instruction files in agent system prompt**: dbt Copilot now reads `AGENTS.md` and `CLAUDE.md` files at the root of your dbt project and injects their contents into the agent system prompt on every turn. Combined content must not exceed 64 KB. * **Subdirectory instruction file discovery**: The agent now surfaces a manifest of `AGENTS.md` and `CLAUDE.md` files found in project subdirectories and reads them on demand when working in the relevant subtree, keeping context loading efficient for large projects. ##### dbt platform * **Create account for unlicensed users**: Users with zero accounts can now create a new account directly from the regional account switcher. When enabled and a `create_account_url` is available, a "Create account" button appears on the account switcher for users with no accounts. Contact your account manager to enable. #### Enhancements ##### Packages * **New:** [Native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) are now generally available (GA). ##### Discovery API * **Enhancement:** In the Discovery API [Tests object schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-tests.md), you can now filter `environment.applied.tests` by multiple test result statuses in a single query using the new `lastKnownResults: [TestStatus]` filter field on `TestAppliedFilter`. The single-value `lastKnownResult` filter field is still supported but deprecated. Update your queries to use `lastKnownResults` going forward. ##### dbt Copilot and agents * **Preview**: The [Developer agent](https://docs.getdbt.com/docs/platform/wizard-platform.md) is now in preview. Use natural language prompts to build or refactor models, and generate SQL, tests, documentation, and semantic models from scratch. For more information, refer to the [Developer agent](https://docs.getdbt.com/docs/platform/wizard-platform.md). * **Enhancement:** Delete individual [chat conversations](https://docs.getdbt.com/docs/platform/wizard-platform.md#availability-and-considerations) from the conversation list (**More actions** menu (three dots) > **Delete**). Deleting the open conversation clears the panel. * **Enhancement:** Commands run by and the [](https://docs.getdbt.com/docs/platform/wizard-platform.md)now appear in the Studio IDE **Commands** tab with a icon and **Run by Copilot** tooltip, so you can tell agent-run commands apart from manually run ones. ##### dbt platform * **Clearer message when account creation is disabled**: The `/accounts/new` page now renders an inline "Account creation is disabled" message instead of silently redirecting to the home page, preventing a redirect loop for users with zero accounts. * **Account feature changes emit audit log events**: When an account feature opt-in is toggled (for example, enabling catalog ingestion or advanced CI), an audit log event is now published recording the previous and new values. ##### APIs, Identity, and Administration * **Notification Manager permission set**: A new account-level **Notification Manager** [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) is now available for Enterprise accounts. Assign it to users who need to manage Slack, Microsoft Teams, and email job notifications across all projects without requiring full Account Admin access. * **Credentials page access with `user_credential_write` permission**: Users with the `user_credential_write` permission on any project can now access the Credentials settings page and edit their user credentials, even without a Developer-tier license or `develop_access`. #### Fixes ##### dbt Copilot and agents * **Correct handling of `tool_call_chunk` content blocks**: Fixed a bug in single tenant environments that would occasionally block conversations from being able to be continued. #### Behavior change ##### APIs, Identity, and Administration * **Single Sign-On configuration uses auto-generated slugs only**: The Single Sign-On (SSO) settings card now always uses server-assigned slugs. The manual slug input has been removed. All SSO configurations are created as drafts with an auto-generated slug before you fill in connection details. ##### Deployment and Configuration * **Fusion version detection uses strict catalog membership**: `isFusionDbtVersion` now only returns `true` for version strings that are explicit members of the known Fusion versions catalog (`latest-fusion`, `fusion-stable`, and so on). Previously, any string containing the word "fusion" was treated as a Fusion version. Custom or unknown version strings that happen to include "fusion" are no longer treated as Fusion versions. #### April 29, 2026 #### Enhancements ##### dbt Copilot and agents * **Job investigation support in Studio agent**: The Studio IDE dev agent can now help you investigate and troubleshoot dbt job and run failures using the `troubleshooting-dbt-job-errors` skill. The agent notes when your local project state may differ from the job (for example, a different branch or uncommitted changes). This feature is currently in beta. Refer to [Debug job failures](https://docs.getdbt.com/docs/platform/wizard-platform.md?version=2.0#debug-job-failures) for more information. ##### Semantic Layer * **Semantic Layer MCP request size limit**: Semantic Layer requests through MCP are now capped at 10 MiB (previously unlimited) to improve infrastructure stability. ##### Orchestration and run status * **Finer-grained permissions on the Debug on Fusion menu**: The "Debug in Studio" and "Run once on Fusion" menu items are now independently disabled based on your permissions. If you lack the required permission for an action, that item shows a tooltip explaining why, while the other item remains available. * **Automatic dbt version override before "Debug in Studio"**: When you click "Debug in Studio," dbt platform now automatically sets your user-level `DBT_DEVELOP_CORE_VERSION` environment variable to `latest-fusion` before opening the Studio IDE, so you no longer need to configure this manually. * **Clearer failure state for "Debug in Studio"**: If the preparation step before opening the Studio IDE fails, the button temporarily shows "Debug failed" in a red state for 5 seconds before resetting, so you know to try again rather than seeing a silent failure. * **Fusion upgrade eligibility for projects with no jobs**: Projects with no jobs configured are now treated as eligible for Fusion. The upgrade card no longer requires a successful job run for such projects and instead shows "No jobs configured yet" under job eligibility. * **Confirmation pop-up before overriding Fusion eligibility**: Clicking "Override eligibility status" on the run details page now opens a confirmation pop-up before applying the override, preventing accidental changes. * **Fusion eligibility review button visible to all users**: The review button on the jobs list for jobs with unknown Fusion eligibility is now shown to all users regardless of run write permissions, so anyone can view the eligibility details modal. * **Model execution notifications for dbt Fusion runs**: The orchestrator now publishes model execution events for grouped models and tests when running with dbt Fusion's OpenTelemetry (OTel) log format, extending model notification support to Fusion-based runs. * **Structured logs available for externally ingested runs**: The run ingestion pipeline now detects and signals when structured dbt logs are present for a step, so logs appear correctly in the dbt platform for runs ingested from external executors. ##### Deployment and configuration * **Sticky submit bar on the create private endpoint form**: The Cancel and Save buttons on the create private endpoint page now stick to the bottom of the viewport so they remain accessible when scrolling through the form. * **Docs generation deprecation scoped to Fusion jobs only**: The "Generate docs on run" deprecation notice is now only shown for jobs running on a Fusion dbt version. Non-Fusion jobs continue to show the standard checkbox. #### Fixes ##### Semantic Layer * **Custom metric granularities no longer rejected**: Metric manifest fields `granularity` and `offset_to_grain` now accept arbitrary string values instead of only a fixed enum. Projects using custom granularities such as `fiscal_year` will no longer fail ingestion. ##### Catalog * **Tag search field**: Tags are now a searchable field in the advanced search panel. You can filter results by tag matches. Filtering uses OR logic, returning assets that match any of the specified tags rather than requiring all tags to be present. #### April 22, 2026 #### New ##### Catalog * **Health and run status filters in catalog search**: The catalog search sidebar now includes Health and Last Run Status filter sections. You can filter dbt resources (models, sources, and exposures) by health status (healthy, caution, degraded, unknown) and by last run status (success, error, skipped, reused). * **Tag search field**: Tag is now a searchable field in the advanced search panel. You can filter results by tag matches #### Enhancements ##### Studio IDE * **More reliable dark mode on initial load**: Added additional layers of theme preference fallbacks, including the user's OS theme preferences, to aid in incorrect theming when user-preferences is slow to respond. * **Deep-linking to console tabs**: You can now navigate directly to a specific Studio IDE console tab (for example, Commands or Lineage) using a `consoleTab` URL query parameter. Invalid tab identifiers are removed from the URL automatically. * **Compile button after deprecation autofix in Fusion**: After the deprecation autofix workflow completes in Fusion environments, a **Compile** button now appears in the autofix results panel so you can immediately verify the updated project without manually triggering a compile. ##### Orchestration and run status * **Fusion eligibility toggle replaces dropdown filter**: The jobs list Fusion eligibility dropdown filter has been replaced with a toggle and help icon. When enabled, each job displays its current Fusion eligibility badge, and a persistent info banner explains how eligibility is recalculated. The toggle state is saved per-project in your browser. * **Debug on Fusion menu**: The single "Run once on Fusion" button on the job details page and job list has been replaced with a "Debug on Fusion" menu that offers "Debug in Studio," "Run once on Fusion," and (when dbt Copilot is enabled) "Debug in Studio with Copilot" options. * **Simplified Fusion run error banner**: The Fusion run error banner on run details now uses the same "Debug on Fusion" menu as the jobs page. The banner no longer requires setting a personal dbt version override before navigating to Studio. ##### Webhooks * **Webhook test flow uses receipt polling**: Testing a webhook subscription now triggers a test event and polls for the delivery receipt, showing the actual HTTP status code and error from the endpoint response. A 60-second timeout is applied, with a clear timeout message if the endpoint does not respond in time. * **Webhook receipt endpoint returns 404 for pending events**: The webhook event receipt endpoint now returns a `404` response when a delivery record has not yet been written (for example, when the notification system has not yet processed the event), rather than returning an incomplete record. * **Corrected status code for timed-out webhook deliveries**: Webhook delivery history records now show `504` as the HTTP status code when a delivery timed out (previously stored as `0`), improving accuracy in the delivery history view. * **Webhook event history note always visible**: The note that event history is limited to the past 7 days now appears on the webhook events history page unconditionally. ##### Integrations * **Slack notification settings migration banner**: A migration banner now appears on the Slack notification settings page when you have notification settings from a previous Slack integration. You can migrate them to the new Slack app in one click or dismiss the banner. After migration, you are shown which private channels need the dbt Cloud bot invited for notifications to be delivered. Contact your account manager to enable. ##### dbt platform * **`account:read` scope on OAuth consent page**: The OAuth consent page now displays a "View account information" scope option, which grants view-only access to account details including project and environment information. * **PrivateLink endpoint pending status**: A new `pending` connectivity status is available for PrivateLink endpoints, in addition to the existing `success` and `failed` states. * **`fusion_readiness_read` permission added to Member role**: The Member permission set now includes `fusion_readiness_read`, allowing members to view Fusion readiness information for projects without requiring elevated permissions. #### Fixes ##### dbt Copilot and agents * **Correct model routing for Azure OpenAI Responses API (BYOK customers)**: Azure OpenAI deployments now correctly pass the deployment name as the `model` field when using the Responses API, preventing misrouted requests when the deployment name differs from the model name. #### April 15, 2026 #### Enhancements ##### Catalog * **Health and run status search filters**: The `AccountSearchQueryFilter` input now accepts `health` and `runStatus` filter arrays. Use `health` to narrow results by health status (`healthy`, `caution`, `degraded`, or `unknown`) and `runStatus` to filter by last run outcome (`success`, `error`, `skipped`, or `reused`). Multiple values within each filter are combined with `OR` logic. * **Health-aware search ranking**: Healthy dbt resources (those with no detected issues) now rank higher in search results than resources with unresolved issues when text relevance is otherwise equivalent. ##### Studio IDE * **Keyboard shortcut to open Commands tab**: Press \`Ctrl+\`\` to open the Commands tab directly from the editor. ##### Orchestration and run status * **Clearer Fusion job eligibility messages**: Fusion eligibility reason messages are rewritten to be shorter and more actionable. For example, unsupported adapters now read "This job uses an adapter that's not currently available on the Fusion engine" and jobs not on Latest now read "This job uses a dbt version that's not tested for Fusion eligibility." * **Fusion eligibility confirmation modal**: Clicking "Run once on Fusion" on a job now opens a confirmation modal before triggering the run, showing the environment name and a warning that job commands will execute in that environment. * **Improved `dbt ls` and `dbt list` run log status (dbt Fusion engine only):**: Run steps that execute `dbt ls` or `dbt list` now show node results with a no-op status instead of "unknown," reducing confusion in run logs for list operations. ##### dbt platform * **More descriptive Fusion readiness toggle**: The account-level setting to enable Fusion readiness and upgrade features now has an updated label ("Enable Fusion readiness & upgrade features") and a more detailed description explaining what the setting allows administrators and developers to do. * **Debug on Fusion navigates with version override**: The "Debug on Fusion" button (previously "Debug manually") on failed Fusion run banners now sets your personal `DBT_DEVELOP_CORE_VERSION` override to `latest-fusion` before opening Studio IDE, ensuring you open the IDE on the Fusion engine. A loading state is shown while the override saves, and an inline error is displayed if the save fails. ##### Deployment and configuration * **Private endpoint connectivity status column renamed**: The "Status" column in the private endpoints list is renamed to "Connectivity status" for clarity. * **Snowflake private endpoint validation shows specific missing fields**: When pasting Snowflake Private Link configuration output, the validation error now lists the specific required fields that are missing (for example, `privatelink-account-url`) rather than a generic message. Valid output now also shows a success indicator. * **YAML credential fields now accept array values**: Environment credential and connection forms that accept YAML Extended Attributes (for example, Redshift `db_groups`) now correctly validate arrays as values. Previously, array values were incorrectly rejected during client-side validation. ##### Integrations * **Snowflake PrivateLink supports reusing existing interface endpoints**: When creating a Snowflake PrivateLink connection, you can now supply an optional `interface_endpoint_id` to attach a new profile to an existing interface endpoint rather than always creating a new one. The endpoint must be in `Available` status; a `409 Conflict` is returned otherwise. Contact your account manager to enable. #### Fixes ##### Studio IDE * **New folders in Git Controls now expand correctly**: Files inside a newly created folder are now listed individually in the Git Controls panel. Previously, a new folder appeared as a single unexpanded entry rather than showing the files it contained. * **Parent folder hint shown for all new files**: Files created inside a new folder now always display the parent folder name as a hint in the Git Controls panel, even when the file name is unique across all changed files. #### April 8, 2026 #### New ##### dbt Copilot and agents * **Admin API tools for MCP remote server**: The dbt MCP remote server now includes Admin API tools, including `list_jobs`, `list_projects`, `get_job_details`, `trigger_job_run`, `cancel_job_run`, `retry_job_run`, `get_job_run_details`, `get_job_run_error`, `list_job_run_artifacts`, and `get_job_run_artifact`. These tools let MCP clients list, inspect, trigger, cancel, and retry dbt jobs and runs directly from connected AI assistants. Contact your account manager to enable. ##### dbt platform * **OAuth consent page**: A new OAuth consent page lets you authorize third-party applications (for example, dbt MCP) to access your dbt platform account. You can select which permissions and projects to grant, then approve or deny the request. ##### Catalog * **Performance tab on test and snapshot detail pages**: Test and snapshot detail pages now include a Performance tab showing cost insights data — including cost, usage, build time, and build count charts — matching the existing model performance experience. ##### Orchestration and run status #### Enhancements ##### dbt Copilot and agents * **Smarter validation after file edits**: The Studio DevAgent now selects the lightest appropriate validation check after each change — for example, skipping compilation for description-only edits and running `dbt parse` for project config changes — instead of always running a full `dbt compile`. This reduces unnecessary round-trips and keeps iteration faster. ##### Studio IDE * **Deferral environment selector**: Replaces the simple defer-to-production toggle with a popover that lets you choose between your development environment, dbt's default deferral behavior (staging if available, otherwise production), or a specific custom environment. A badge in the command bar shows your current deferral target at a glance. * **Revert personal dbt version override**: Adds an "Edit / Revert" action to the version override option in the environment popover. Clicking "Revert" opens a confirmation modal that removes your personal dbt version override and restarts the session. * **Improved file context pill in dbt Copilot**: Moves the active-file context pill to above the text input for greater visibility. When you remove the file context, a "Use current file as context" affordance appears so you can restore it without switching tabs. ##### Catalog * **Reused test status in DAG lens**: State-Aware Orchestration (SAO) test runs that reuse prior results now display with a "reused" icon in the DAG test status lens, matching the existing model run status behavior. * **Function resource type support in selectors**: The `function` resource type is now recognized in dbt selectors and the resource node type map, enabling correct filtering and navigation for function resources in Catalog. ##### Insights * **Fusion status column in account insights table**: Look for a "Fusion status" column in your account insights table when the Fusion readiness flow is available for your account. You'll see one of four states: On Fusion, Start upgrade, Partial-Fusion, or Non-Fusion — based on each project's readiness and migration progress. Projects that are ready to upgrade show a "Start upgrade" button that navigates directly to the project home page. Contact your account manager to enable. #### Fixes ##### Studio IDE * **Parse status no longer shows error badge during Fusion compilation**: In Fusion mode, the parse status badge no longer switches to an error state solely because diagnostic errors are present. The badge now correctly reflects compilation progress and completion independent of diagnostic counts. * **Clearer authentication errors for rejected git connections**: Adds "remote rejected authentication" as a recognized, non-retryable git authentication error. You will now see a clear authentication failure message instead of a misleading retry loop when your git provider rejects your credentials. ##### Catalog * **Reused models no longer flagged as stale**: Models with a `last_run_status` of `reused` are no longer marked stale even when their last execution date exceeds 30 days. This prevents false health issue warnings for models that were intentionally reused rather than re-executed. * **Resource counts refresh on environment switch**: Fixes a bug where resource counts on the project landing page were not updated when switching environments. #### April 1, 2026 #### New ##### Studio IDE * **Fuzzy file path search**: Studio IDE now supports fuzzy file path search that finds files in your project using partial name matching. You can filter by glob patterns, set a result limit, and receive ordered results with a total match count, making it faster to navigate large projects. ##### dbt platform * **OAuth consent endpoint for Connected Auth**: A new `/oauth/consent` endpoint enables the Connected Auth OAuth flow, supporting user consent decisions (approve and deny), project-level resource boundaries, and authorization code issuance. This feature is in private beta. To request access, contact your account manager. #### Enhancements ##### dbt Copilot and agents * **Persistent agent mode across sessions**: The Studio agent now remembers your last-used mode (Ask or Code) across browser sessions, so you no longer need to reselect it each time you open the IDE. ##### Studio IDE * **More accurate file search results**: File search now validates each result against the filesystem before returning matches. Files that have been deleted locally but not yet staged are no longer included in search results. * **No unexpected git pulls on the primary branch**: Removes behavior where the IDE server automatically pulled changes from your primary branch during git status checks, which could cause unintended overwrites for projects using trunk-based development. ##### Orchestration and run status * **Teradata column-level lineage support**: Adds Teradata to the SQL dialect adapter map, enabling column-level lineage parsing for dbt projects using the Teradata adapter. ##### APIs, Identity, and Administration * **Fusion status includes readiness and migration availability**: Adds fields indicating availability of readiness and migration features. * **Faster account feature flag propagation**: Account feature flag changes now take effect within 60 seconds instead of up to one hour. You should see feature toggles apply more promptly across your account. ##### Webhooks * **Notification delivery reliability improvements**: Reduced the likelihood of delayed notifications (webhooks, email, Slack, and Teams) in certain third-party/system disruption scenarios. #### Fixes ##### APIs, Identity, and Administration * **GitHub webhook secret null check before signature validation**: The GitHub webhook endpoint now correctly checks for a null webhook secret before attempting to validate the request signature, preventing a crash when a repository's webhook secret is not set. * **`github_installation_id` and `github_webhook_id` support large values**: These repository fields have been promoted from 32-bit to 64-bit integers (`BigIntegerField`) to accommodate GitHub installation and webhook IDs that exceed the 32-bit integer range. #### Behavior change ##### APIs, Identity, and Administration * **Fusion migration gated by API availability**: The Fusion migration checklist, the Enable Fusion Environments page, and the "Enable Fusion" button in Studio IDE now use the `is_migration_available` field from the Fusion status API instead of the legacy `orc2609ShowFusionToggle` feature flag. Fusion migration UI is shown only when the backend has marked the project as ready for migration. #### March 25, 2026 #### New ##### Orchestration and Run Status * **Fusion run error banner**: When a run using the dbt Fusion engine fails, a banner now appears on the run details page with options to debug the failure in Studio IDE. If dbt Copilot is enabled, you can also open a guided fix-with-Copilot workflow directly from the banner. Contact your account manager to enable. #### Enhancements ##### Studio IDE * **Consistent console pane default size**: The bottom console pane now opens at a preferred size of 33% of the available space, providing a more consistent default layout. * **Faster text search results**: File search now reports results incrementally on a per-file basis rather than per-match line, reducing memory pressure and improving perceived responsiveness during large searches. * **Smarter bulk-edit file handling**: When Studio IDE applies multi-file edits (for example, from dbt Copilot agent tasks), it now only updates editor models for files that are already open. Previously, every edited file was opened in a new tab, which cluttered the editor. ##### Deployment and Configuration * **Fusion migration enablement via project API**: You can now set `fusion_migration_enabled` on a project via the project update API. Enabling it requires the `fusion_readiness_write` permission, and the project must meet all readiness prerequisites (supported adapter, supported dbt version, a successful run, and eligible jobs). * **Filter jobs by Fusion readiness**: The jobs list endpoint (`GET /api/v2/accounts/{account_id}/jobs/`) now accepts an `is_fusion_ready` boolean query parameter. When `true`, it returns only conformant or override-ready jobs; when `false`, it returns only non-ready jobs. You can also include `fusion_readiness` in the `include_related` parameter to surface Fusion readiness details alongside the job response. * **Platform metadata credentials form opens immediately**: When adding platform metadata credentials for a connection, the credential form is now shown immediately instead of requiring you to click an "Add credentials" button first. #### Fixes ##### Catalog * **Accurate resource counts on environment switch**: Fixes a bug where resource counts in the navigation tree were not refreshed when switching environments. You should now see up-to-date counts after changing the active environment. ##### Orchestration and Run Status * **Stuck runs are now cancelled**: A new cleanup job detects runs and run steps that have exceeded the maximum allowed duration and marks them as `CANCELLED`, preventing stale in-progress states from accumulating ##### Semantic Layer * **More reliable Snowflake connections after warehouse auto-resume**: The Semantic Layer Gateway now retries the initial connection when a Snowflake warehouse is waking up from auto-suspend, instead of failing immediately. You should see fewer connection errors when querying the Semantic Layer after a period of inactivity. ##### APIs, Identity, and Administration * **Large group permission sync no longer silently truncated**: Fixed an issue where group permission sync could miss updates for groups with many permissions. #### Behavior change ##### Studio IDE * **Fusion OpenTelemetry log rendering always enabled**: Studio IDE now enables Fusion OpenTelemetry (OTel) log rendering for all invocations running on a Fusion core version, removing the previous feature flag requirement. If you are running a Fusion core version, you automatically receive OTel-based log output without any additional configuration. #### March 18, 2026 #### Enhancements ##### Studio IDE * **Faster file search:** Studio IDE now reuses its file-search index across searches, so repeated searches return results faster. * **More responsive Git status decorations:** Studio IDE debounces rapid file change events and avoids applying stale responses, so Git status badges update more reliably during bulk edits and saves. * **Clearer server status details:** The server status popover uses a clearer grouped layout and action buttons to help you troubleshoot user credentials and server health. Please contact your account manager to enable. ##### dbt Copilot and agents * **More accurate product guidance:** Copilot and agents can use a product documentation toolset to answer product and workflow questions more reliably. * **Full-screen Copilot view:** You can open Copilot in a dedicated full-screen view for a more focused chat and coding workflow. Please contact your account manager to enable. * **Lighter default file context:** Copilot now references your active file by path instead of automatically attaching the file contents, which reduces message size and improves chat reliability. * **Run `dbt-autofix` from Copilot and agents:** Copilot can run `dbt-autofix` commands (with confirmation) and stream the output into chat, and Studio IDE agents can run `dbt-autofix` using `run_autofix` for bulk deprecation fixes and migrations. ##### Catalog * **Custom materialization filter:** Catalog search now groups non-standard materializations under a single "Custom" filter, so you can narrow results without picking each materialization type. ##### Insights * **More complete Redshift query attribution:** Insights can resolve missing Redshift query IDs from warehouse query history when artifacts do not include them, improving cost coverage for runs with executions. * **Copilot entry stays available during lockouts:** If dbt Copilot is temporarily locked for your account, you can still open Copilot from Insights to see lock details. ##### Orchestration and Run Status * **Run metadata includes triggering and canceling actors:** Run details now include who triggered or canceled a run (user or service token), which helps you audit run activity. * **Custom branch preserved for runs and reruns:** When an environment uses a custom branch, dbt platform now carries that branch through run triggers, retries, and reruns more consistently. * **Fusion readiness metadata for jobs and environments:** You can now retrieve Fusion readiness signals for projects, environments, and jobs to support Fusion migration planning. Please contact your account manager to enable. * **More accurate command names for dbt Fusion runs:** Orchestration now reads the invocation name from `run_results.json` using `command` when `invocation_command` is missing, so you see the correct dbt command in run details. ##### Run Logs * **More resilient run step history ingestion:** Run step history ingestion now drops invalid events and de-duplicates redundant step-start events before writing step data, improving step-level accuracy. Please contact your account manager to enable. ##### Deployment and Configuration * **Longer project descriptions:** You can now add project descriptions of up to 1,024 characters. * **Connection links in profiles:** You can now open a connection directly from the connection profile table in a new tab. * **Clearer YAML validation for extended attributes:** You now get more consistent validation and clearer error messages for invalid YAML syntax, null values, and non-object YAML content when you edit extended attributes. ##### Semantic Layer * **Improved filtered-query cache matching:** Cached query results can now be matched and reused more reliably when your query includes filters, which can reduce repeated compilation and improve response times. #### Fixes ##### APIs, Identity, and Administration * **Fewer transient Cloud Config failures:** Retries now only apply to transient errors during Cloud Config lookups, so you should see fewer intermittent failures without added delay for permission, authentication, or not-found responses. * **More reliable sign in redirects from `current_email`:** If you are already authenticated and land on `/login` with `current_email`, dbt platform now redirects you to `/api/auth/auth-login/` so the email is forwarded during sign in. * **IP restrictions toggle saves reliably:** Turning IP restrictions on or off now updates form state correctly, so your changes save as expected. * **More consistent audit log date filtering:** The audit log date range defaults no longer shift during re-renders, so your filters stay stable while you review results. * **More reliable Single Sign-On (SSO) migration domain updates:** Domain updates during Single Sign-On (SSO) migration no longer rely on mutating existing provider data, which improves save reliability. ##### Deployment and Configuration * **Clearer Bring Your Own Key (BYOK) credential errors:** If your OpenAI credentials include invalid characters, you now get a clearer error message so you can correct the configuration. * **More reliable credential edits:** Encrypted credential fields now stay optional when you edit credentials, which reduces unexpected validation failures. * **Correct connection details while editing environments:** You now see the correct connection details more consistently when you edit an environment that uses global connections and connection profiles. ##### Orchestration and Run Status * **Run steps are available for ingestion runs:** You can now open and review run steps for ingestion-triggered runs. * **Cleaner run error fields:** Run results no longer populate an error string with `None` when dbt does not provide a message or failure count, so you see clearer run error details. * **Clearer errors for invalid dbt projects:** When Orchestration cannot restore the repository cache because the dbt project is missing or malformed, it now returns an invalid project error so you get a more actionable message in run results. ##### Catalog * **Skipped snapshots show as skipped:** Snapshots selected but not executed in multi-step runs now appear with a skipped status instead of missing run status fields. ##### Insights * **Copilot chat no longer gets stuck loading:** Insights now clears the Copilot chat loading state reliably after responses complete or error, so you can keep chatting without refreshing the page. * **More reliable Copilot handoff starts:** When you arrive in Insights with a Copilot handoff message, Insights now starts the handoff once and clears stale handoff state when you navigate directly. ##### Integrations * **More consistent JDBC typing for Tableau and Power BI:** Semantic Layer now derives explicit string conversions from returned result metadata, so categorical dimensions and entities are more consistently typed as strings in Tableau and Power BI queries. ##### Semantic Layer * **More reliable cache key deletion:** Cache invalidation no longer fails when an in-memory cache key is already missing, which reduces intermittent errors during cache cleanup. * **More accurate run ID validation:** Semantic Layer now requests and caches run details scoped to your account, which reduces incorrect run validation results. ##### dbt Copilot and agents * **More reliable cancellations during tool use:** If you cancel a request while an agent is running tools, the agent now recovers cleanly instead of getting stuck on incomplete tool-call history. * **Cleaner AI diff overlays:** Studio IDE now removes the accept and reject overlay when you leave an artificial intelligence (AI) diff view to prevent stale UI controls. #### Behavior change ##### Studio IDE * **Updated file search and command shortcuts:** Studio IDE now uses VS Code Quick Open for file search (`Cmd+P` or `Ctrl+P`) and the VS Code Command Palette (`Cmd+Shift+P` or `Ctrl+Shift+P`) instead of the legacy Studio dialogs. ##### Integrations * **Disallowed `MIN()` and `MAX()` for metrics and dimensions:** Tableau and Power BI queries can no longer request `MIN()` or `MAX()` for a metric or dimension (except time min-max queries), and you now receive a clear error if you attempt it. #### March 11, 2026 #### New ##### Deployment and Configuration * **Self-serve Snowflake private endpoint requests:** You can request a new Snowflake private endpoint from account settings by pasting the output from `SELECT SYSTEM$GET_PRIVATELINK_CONFIG();`, then track request status in the private endpoints table. This is available for Enterprise Business Critical accounts only, and please contact your account manager to enable. For other connection types, contact <support@dbtlabs.com>. #### Enhancements ##### Orchestration and Run Status * **Run retries support dbt Fusion runs:** You can now retry failed runs as long as your environment is on dbt Core version `1.6` or higher or dbt Fusion. ##### Integrations * **More reliable Slack notifications:** Slack channel discovery and notifications now retry on Slack rate limits to reduce dropped messages during busy periods. ##### APIs, Identity, and Administration * **Improved OpenAPI typing for large integers:** OpenAPI schemas now mark 64-bit integer fields as `format: int64` to improve generated client types. * **Clearer credentials schemas:** Credentials OpenAPI docs now use a `type` discriminator (`postgres`, `redshift`, `snowflake`, `bigquery`, and `adapter`) to improve code generation and request validation. #### Fixes ##### Orchestration and Run Status * **More reliable job search:** Searching jobs with numeric terms (for example, `12`) no longer triggers API validation errors, so you can load job lists reliably. * **Clearer cross-project publication errors:** When dbt platform cannot fetch a publication artifact for an upstream project declared in `dependencies.yml`, you now see which project is missing an artifact and guidance to run the upstream environment at least once. ##### Integrations * **More accurate Microsoft Teams notification triggers:** Microsoft Teams notifications now use the correct trigger event type for each notification, so you see the expected run outcome context in the message. ##### APIs, Identity, and Administration * **More accurate error responses during permission checks:** You now receive more accurate errors from permission checks, and underlying service errors surface instead of being reported as authorization failures. ##### Deployment and Configuration * **Clearer private endpoint validation errors:** Creating a private endpoint now returns a `400` error with a clear message when `snowflake_output` is malformed or not valid JSON. #### Behavior change ##### Orchestration and Run Status * **Model timing unavailable for dbt Fusion runs:** You now see an informational notice instead of the Model timing chart for dbt Fusion runs because dbt Fusion handles threading differently. ##### APIs, Identity, and Administration * **System for Cross-domain Identity Management (SCIM) `id` fields are now strings:** SCIM schema discovery now reports `id` fields as strings for users and groups. #### March 4, 2026 #### Enhancements ##### Orchestration and Run Status * **Clearer SAO description**: Job settings now describe state-aware orchestration (SAO) as only building models when data or code changes are detected. * **Direct links for cost optimization setup**: Fusion cost optimization settings now link to account-level Cost Insights settings and setup documentation so you can validate cost data and savings. ##### APIs, Identity, and Administration * **Confirmation when enabling manual SCIM updates**: When you enable manual updates for System for Cross-domain Identity Management (SCIM), dbt platform now asks you to confirm so you do not accidentally allow changes outside your identity provider. * **More reliable SCIM group provisioning**: SCIM has been updated so that when a SCIM-provisioned user with an expired invite is added to a SCIM-managed group through a SCIM request, the invite is automatically resent during group assignment. This helps prevent errors caused by unaccepted invites. ##### dbt platform * **Project names and descriptions handle empty values better**: Projects with missing names now show as "Untitled Project," and you can save project descriptions as empty. ##### Studio IDE * **Removed non-functional "Open Settings" actions**: Studio IDE no longer shows "Open Settings" buttons in editor notifications because Studio IDE does not expose VS Code settings, and the action would not help you resolve issues. #### Fixes ##### Catalog * **More reliable file tree loading**: Catalog no longer gets stuck loading the file tree on initial page load. * **Clearer trust signals**: Trust signals now suppress less-severe upstream-source issues when a more severe issue is present, so badges and messages are easier to interpret. ##### Integrations * **Clearer deploy key decryption errors**: When dbt platform cannot decrypt a deploy key, you now get a clearer failure instead of a generic git credentials error. ##### Studio IDE * **Cleaner LSP disconnects**: If authentication fails when you connect to the Language Server Protocol (LSP) WebSocket, the connection now closes cleanly instead of failing with an internal server error, so you should see fewer unexpected disconnects. * **Improved timeout handling and authentication stability**: Reduced environment setup timeouts and resolved intermittent authentication failures during busy periods. * **Clearer invalid credentials error**: If your development connection credentials are invalid, you now see a clearer error message to help you diagnose the issue faster. #### Behavior change ##### Orchestration and Run Status * **`versionless` dbt version is no longer accepted**: dbt platform now treats `versionless` as deprecated and updates existing environments and jobs to use `latest`. If you set `dbt_version` in an API integration or automation, update it to send `latest` instead. ##### Webhooks * **Account identifier required for run-based notifications**: If you send events that include a `run_id`, you must also provide an `account_identifier` so the service can validate and resolve the correct account before dispatch. If `account_identifier` is missing, the event fails instead of falling back to a `run_id`-only lookup. #### February 25, 2026 #### New ##### Catalog * **Saved queries now ingested for lineage and governance**: Saved query definitions (including tags, exports, parameters, and lineage relationships) are now captured during ingestion so they can participate in Catalog lineage and governance workflows. #### Enhancements ##### dbt platform * **System logs now surface warnings and errors**: Run step structured logs now show an indicator when system warnings or errors are present, making issues easier to spot during run triage. * **Region labels now use backend display names**: Account Settings now shows the backend-provided region display name for clearer, more accurate region labeling. * **SCIM create group UI change**: Changes to our UI to improve the experience of managing groups with SCIM enabled. * **Updated the post-invite message for SSO accounts**: After a user accepts an invite, the UI now explains that they must log in using SSO to fully redeem the invite and access the account. This replaces the previous "Joined successfully" message and helps avoid confusion when users accept an invite but do not complete the SSO login flow. ##### Studio IDE and Copilot * **Improved crash recovery and not-found routing**: Studio IDE now catches unexpected render failures with a top-level error boundary and shows Not Found more reliably for unknown in-project routes. * **Improved navigation accessibility and semantics in Studio IDE**: The main navigation trigger area is now a navigation element with improved focus and labeling. * **Reduced shortcut conflicts with VS Code search**: When Visual Studio Code (VS Code) search is enabled, Studio IDE avoids unregistering Quick Open and suppresses conflicting command palette shortcuts. ##### Catalog and Insights Data * **More accurate source freshness outdated status in Catalog**: Source freshness Outdated status can now be computed at query time, improving freshness status filtering consistency. * **Improved search and lineage usability in Catalog**: Search results better support column-level navigation and very long queries show a clear validation error, and lineage visuals have improved alignment and reduced edge clutter. * **Improved cross-project lineage and function awareness in Catalog**: Lineage graph building now includes cross-project dependencies and supports function nodes as first-class lineage entities. ##### APIs, Identity, and Administration * **Project deletion now supported in Admin v2 and v3 Projects APIs**: Projects APIs now explicitly support DELETE with stricter permission checks. #### Behavior change ##### Webhooks * **Updated job run event field presence and status normalization**: Webhook payloads now include `runFinishedAt` only for completed events and `runErroredAt` only for errored events; canceled runs no longer include `runCanceledAt`, and run status is normalized from Cancelled to Canceled. Also note that enabling JSON preserve order can change key ordering, so consumers should parse JSON rather than string-compare payloads. ##### Insights APIs * **Optional source freshness expiration windows**: Source freshness expiration windows can optionally derive from each source's freshness criteria rather than a fixed window. You must enable in your deployment. ##### Deployment and Configuration * **Source ingestion may skip sources for extremely large manifests in Catalog**: For very large `manifest.json` files, ingestion may strip sources above a configurable threshold to prevent out of memory failures. Set `SOURCE_INGESTION_THRESHOLD=0` if you must always ingest sources regardless of size. * **Removed deprecated object storage settings in Studio IDE**: Deprecated settings `project_storage_bucket_name` and `project_storage_object_prefix` have been removed. Migrate to `object_storage_bucket_name` and `object_storage_object_prefix`. #### February 18, 2026 #### New ##### Cost Insights * **Estimated warehouse compute costs**: Cost Insights shows estimated warehouse compute costs and run times for your dbt projects and models, directly in the dbt platform. It highlights cost reductions and efficiency gains from optimizations like state-aware orchestration across your project dashboard, model pages, and job details. This feature is in private beta. To request access, contact your account manager. #### Enhancements ##### Studio IDE * **Reduced conflicts across multiple tabs**: Studio IDE can pause the Language Server Protocol (LSP) in background tabs and resume on return to improve stability when the editor is open in more than one tab. * **More informative header and more editor space**: Adds a Visual Studio Code-style header showing a dbt badge and current project name, with an option to hide surrounding chrome for more editor space. Please contact your account manager to enable. * **Clearer file and folder creation errors**: Surfaces more actionable filesystem errors (for example, name too long and file-is-a-directory) instead of generic failures. * **Copy relative path**: Adds a Copy Relative Path action that respects `dbt_project_subdirectory` for quicker navigation and sharing. * **Friendlier lineage error messages**: Improves user-facing errors for lineage failures (including server errors and cases where upstream returns HTML instead of JSON). * **More reliable private connectivity selection**: Improves private endpoint filtering by adapter type and updates Studio IDE to use the correct version 3 private endpoints endpoint. ##### Canvas * **More reliable Add Sources CSV uploads**: Improves Comma-Separated Values (CSV) upload progress, resume behavior, and common error handling during Add Sources. ##### Catalog * **Faster and more usable lineage for large projects**: Improves directed acyclic graph (DAG) performance by rendering only visible elements and improving layout for disconnected nodes. * **Safer search result interactions**: Improves keyboard and hover behavior in the search dropdown and avoids showing stale results while searches are loading. ##### dbt platform * **More informative user invite statuses**: This change shows clearer invite status (invitation sent and invitation accepted) and supports accepted, login pending for Single Sign-On (SSO). * **Unpaid billing banner enabled by default**: The unpaid billing banner is no longer feature-flagged and will display when applicable, while billing link visibility remains permission-based. * **System for cross-domain identity management (SCIM)**: Bug fixes and improvements related to managed invites for easier processing. ##### dbt Copilot and agents * **Streaming control for server-sent events**: Adds Server-Sent Events (SSE) streaming control so clients can choose chunk streaming or message streaming. This enables more responsive Copilot experiences in environments that support streaming. * **More reliable similar models requests**: Improves responsiveness for AI Similar Models and Similar Sources requests by enforcing tighter embedding and database timeouts aligned to request deadlines. Users should see faster, more consistent results when exploring related models. * **dbt Copilot: Improved bring your own key error handling**: Categorizes OpenAI failures with Bring Your Own Key (BYOK) awareness so BYOK failures return the expected 424-class behavior instead of generic 500-series errors. This makes it easier to diagnose and resolve key or configuration issues. * **Expanded dbt Model Context Protocol tooling**: Updates dbt Model Context Protocol (MCP) tooling, including adding `get_all_macros` and improving error categorization, enabling more accurate responses. #### Fixes ##### Studio IDE and Catalog * **More reliable search and replace**: Ensures bulk edits stay in sync after server-side edits to prevent stale content from overwriting changes. * **Correct search preview highlighting**: Fixes preview and match highlighting assembly so match ranges align correctly in multi-line previews. * **Improved startup failure experience**: Shows a proper error layout and notification on unrecoverable initialization failures. ##### Canvas * **Fewer Add Sources UI interruptions**: Prevents incorrect tab closing after uploads complete and avoids showing the floating node panel when not on a file tab. ##### Catalog * **Public model lineage across environments**: Fixes lineage resolution for public model parents when the producer model lives in a non-default environment. ##### dbt Copilot And Agents * **Reduced resource growth under load**: Fixes an OpenAI connection pool leak that could lead to out-of-memory (OOM) conditions under sustained load. Users should see fewer slowdowns during high-traffic periods. * **Fewer related models timeouts**: Reduces intermittent failures when attaching related models by increasing internal timeouts for related-model fetching. Users should experience fewer timeout errors when working with related models. #### Behavior change ##### Studio IDE * **Prevent destructive root operations**: Prevents rename and delete operations on the repository root and shows clearer warnings. * **Resumable dbt command log streaming**: Improves dbt command log streaming reliability by resuming from the last known Command Line Interface (CLI) event offset. Contact your account manager to enable. ##### Admin And APIs * **Job Admin gains write access in Profiles API**: Job Admin now includes `profiles_write`, which can change what Job Admin users can do where Profiles are enabled. * **Search parameter renamed**: Version 3 Private Endpoints query parameter `name_search` is renamed to `search`, and search matches endpoint name and endpoint value. * **Connections: Postgres database name required**: Postgres connection validation now requires a non-empty database name. * **User credentials: Prevent sharing credentials across users**: Prevents associating the same active credentials object to multiple users, returning a conflict instead of silently duplicating associations. ##### Integrations * **GitHub: More flexible repository URL schemes**: GitHub shared webhooks now accept repository URLs using https, git, and Secure Shell (SSH) formats. * **Slack: Tighter permission gating for settings**: Slack linking and notification settings are more strictly gated by the relevant permissions. * **Slack: Permission check aligned to job notification access**: Slack integration listing now uses job notifications read permission, reducing incorrect permission-denied scenarios. ##### CLI Runtime * **Shorter default request timeouts**: Reduces default timeouts from 60 seconds to 5 seconds for Cloud Config and Cloud Artifact calls, causing requests to fail faster in high-latency environments unless overridden. * **OpenTelemetry logs: Corrected JSON field name**: Corrects the OpenTelemetry (OTel) log payload field name to `additional_message` (from the misspelled `addtional_message`), which may require updates to downstream parsing. #### February 11, 2026 #### Enhancements ##### Catalog * **Faster model graph rendering for large projects**: Improved model graph layout performance to reduce load time in larger projects. * **Faster similar models results**: Similar Models lookup now uses an optimized vector search strategy to reduce timeouts on large projects. ##### Studio IDE * **Clearer project root in Catalog file tree**: When your dbt project is in a subdirectory, the project root is highlighted in the Catalog file tree. * **More native rename and delete in Catalog file tree**: Rename and delete actions now use native editor behaviors when using the Catalog file tree. * **More reliable in-browser formatting**: Formatting updates now apply directly to the active editor buffer to reduce prompts and inconsistent results. * **Cleaner code generation workflow**: Code generation no longer creates a temporary file in your repository during generation. ##### dbt platform * **Fusion compatibility validation on environments**: Environment settings now prevent saving a Fusion dbt version with an incompatible connection and surface field level validation errors. * **Smarter Fusion defaults during connection setup**: When setting up a new connection, Fusion eligible adapters now default to the latest Fusion version to reduce misconfiguration during setup. * **Improved Private Link endpoint management**: Private Endpoints can be sorted by status and connections, and endpoint details now show associated connections and environments. ##### Run Logs * **More reliable invocation event streaming**: Invocation event streaming is more reliable for long running jobs by deriving totals from the latest stream event identifier. * **Reduced Redis usage after log streams complete**: Log streaming now cleans up Redis keys after a stream completes, reducing stale keys and Redis memory pressure for high volume runs. #### Fixes ##### dbt Copilot * **Consistent usage limit messaging in Insights and Studio IDE**: When users hit the usage limit, dbt disables Copilot and shows a clear message, including the reset date when available. ##### Studio IDE * **Git status decorations registered once**: Fixed duplicate Git status decorations in the file tree that could cause visual issues and performance impact. * **Avoid automatic pull on primary branch**: Studio IDE no longer runs an automatic pull on the primary branch to reduce unexpected changes during development. * **Clearer file operation validation errors**: File operations now return structured validation errors and explicitly reject names that exceed operating system limits. * **More reliable command log refresh and finalization**: Command logs for the dbt Cloud Command Line Interface (CLI) are refreshed and finalized more reliably. ##### Run Automation * **Correct account attribution for automatically triggered runs**: Scheduler triggered runs now include account context, improving run attribution and preventing some downstream triggers from running without proper context. * **Reject malformed account identifiers for exposure events**: Exposure generated events now validate that account identifiers are numeric before triggering follow on automation. ##### Webhooks * **More compatible run completion payload for canceled and errored runs**: Webhook payloads now include consistent completion and error timestamps, and canceled runs include a canceled timestamp and normalized status. * **Restored dual dispatch for some failure and completion triggers**: When both failure and completion triggers are configured, errored runs may generate two webhook deliveries to match legacy behavior. ##### dbt Project Metadata * **Manifest Ingestion: Accept functions section in manifest.json**: Ingestion now accepts the `functions` section (for example, Snowflake user defined functions (UDF)) to prevent parse failures on newer manifest schemas. * **Macro Metadata: More consistent timestamps and argument comparison**: Macro metadata persistence now uses more consistent Coordinated Universal Time (UTC) timestamps and improves argument comparison to reduce noisy or incorrect macro updates. #### Behavior change ##### dbt platform APIs * **Removed credential configuration fields from responses**: Profiles API responses no longer include credential configuration and extended attributes; use the appropriate credentials and configuration endpoints instead. * **Filter connections by Private Endpoint**: Account Connections list supports filtering by Private Endpoint identifier for easier management. * **Additional ordering options**: Private Endpoints list now supports ordering by endpoint state and connection count. * **Private Link: Updated license permission defaults**: User licenses now include read access for Private Link resources, which may change who can view Private Link related settings. ##### Studio IDE * **Metric generation writes directly to active file**: Generated metrics are now written directly into the active model file instead of using an accept and reject diff flow. #### February 4, 2026 #### New ##### Studio IDE * **Studio IDE: Copilot link in console toolbar**: Adds a link that opens Copilot from the console toolbar. You can use Copilot to read files and list directories for better context. * **Studio IDE: Copy repo-relative path command**: Adds a command to copy a file path relative to your dbt project subdirectory, making it easier to share paths in runbooks and support tickets. #### Enhancements ##### dbt platform * **dbt platform: Fusion eligibility and compatibility indicators in setup flows**: Improves Fusion setup by showing "Fusion compatible" indicators during connection setup. * **dbt platform: Compare Changes shows partial success warnings**: When Compare Changes subqueries fail, the experience now surfaces a partial success state with expandable warning details to make troubleshooting faster. * **dbt platform: In-progress run logs preserve text selection**: Improves log usability during in-progress runs by preserving text selection while logs auto-refresh and rerender. * **dbt platform: Job completion trigger job picker search**: Adds server-side search and clearer loading and empty states to the job picker for job-completion triggers. * **dbt platform: Job artifacts content types and downloads**: Improves artifact handling for job documentation and run artifacts by strengthening HTML detection, defaulting empty paths to `index.html`, and returning clearer `Content-Type` and download filenames. * **dbt platform: Private Endpoints API listing and pagination improvements**: Improves Private Endpoints API v3 list behavior with validated query parameters, filtering, limit and offset pagination, and `connection_count` in responses. ##### Studio IDE * **Studio IDE: Format file more reliable in subdirectories**: Improves formatting reliability by consistently using the active editor content and a stable repo-relative path when invoking formatting. * **Studio IDE: Better stability for tabs and Git operations**: Reduces errors when working with non-file tabs and improves robustness around tab-close and Git checkout flows. * **Studio IDE: Sidebar layout improvements for embedded panels**: Improves embedded panel sizing to reduce clipping and scrolling issues in the sidebar. * **Studio IDE: Fusion prompts reflect actual eligibility**: Improves Fusion banners and prompts by checking project eligibility via a Fusion status endpoint to reduce confusing prompts for ineligible projects. ##### Catalog and Discovery * **Catalog: Improved cross-project lineage for dbt Mesh**: Improves cross-project lineage ("public ancestors") computation to better match expected external lineage boundaries in dbt Mesh experiences. ##### Insights * **Insights: More reliable Copilot Agent requests and context handoff**: Standardizes Copilot Agent requests to the API and includes active tab content as context to improve reliability of agent runs and handoff. #### Fixes ##### dbt platform * **dbt platform: Webhook form editing more resilient**: Improves webhook subscription editing reliability with asynchronous data and fixes a multiselect focus issue that could cause accidental option selection. * **dbt platform: Run warning emails render correctly**: Fixes HTML email markup that could break rendering for run warning notifications. * **dbt platform: Profiles URLs moved under project dashboard**: Profile create and view routes now live under `/dashboard/:accountId/projects/:projectId/profiles/...`, which may affect bookmarks and direct links. ##### Studio IDE * **Studio IDE: Cleaner command history list**: Removes hidden background commands (such as listing and parsing commands) from command history to reduce noise for users. * **Studio IDE: More reliable inline compile and show output**: Improves robustness of inline compile and show output attachment, including cases with tricky quoting and newlines, reducing missing results during interactive use. * **Studio IDE: More reliable log downloads for dbt commands**: Fixes log download behavior so downloads correctly serve either the active `dbt.log` or the finalized compressed log. * **Studio IDE: More reliable artifact uploads to Microsoft Azure Blob Storage**: Fixes edge cases where gzipped artifacts (such as manifests) could fail to upload due to upload stream handling, improving upload reliability. * **Studio IDE: More stable language server protocol (LSP) sessions in workers**: Reduces noisy disconnect and cleanup errors when multiple websocket connections and processes map to the same invocation, improving session stability. ##### Catalog * **Catalog: Search highlighting displays correctly with multiple matches**: Fixes search result highlighting when the backend returns multiple highlights per field, improving readability of matches. Updates search highlights to display as compact badges with counts for easier scanning of results. * **Catalog: Environment filtering more accurate in search results**: Improves environment-scoped Catalog search filtering by using merged environment identifiers and preserving warehouse-only assets via a dedicated sentinel value. * **Catalog: Public models return empty list when none exist**: Improves behavior for environments with no public models by returning an empty list instead of falling into follow-on query logic. ##### Copilot * **Copilot: More reliable model context protocol (MCP) connections during long tool calls**: Improves keep-alive behavior so connections shut down cleanly when the client disconnects, reducing noisy failures. * **Copilot: Semantic Layer tools only offered when available**: Prevents failing tool calls by hiding Semantic Layer tools when the Semantic Layer is not available for the user or environment. * **Copilot: More accurate HTTP error responses**: Improves error reporting by walking wrapped exceptions and exception groups to return the most specific status code and detail available. * **Copilot: Empty Tool Outputs No Longer Cause Failures**: Treats empty tool outputs as valid results (for example, "no matches") to reduce unnecessary "tool call failed" errors. #### Behavior change ##### dbt platform * **dbt platform: Fusion default dbt version selection more restrictive**: During connection setup, the default dbt version now only defaults to `latest-fusion` when the selected adapter is Fusion-compatible and the project and account are eligible. * **dbt platform: dbt version enforcement now project-aware**: dbt version "allowed version" checks now account for `project_id` across jobs and environments, including Application Programming Interface (API)-triggered runs, improving correctness for overrides and automatic mapping to allowed equivalents when possible. * **dbt platform: Connected app refresh tokens now last 7 days**: Refresh token expiration for connected app OAuth flows increased from 8 hours to 7 days, reducing re-authorization frequency. ##### Studio IDE * **Studio IDE: File stat timestamps now milliseconds**: File stat responses now return modified time and created time as integer milliseconds since epoch instead of float seconds; integrations consuming these endpoints may need to adjust. * **Studio IDE: Language Server Protocol deferral controls expanded**: The Language Server Protocol (LSP) websocket now supports `defer_env_id` to defer against a specific environment and `no_defer=true` to explicitly disable deferral. * **Studio IDE: Deferral toggle applied more consistently to Language Server Protocol connections**: When "defer to production" is turned off, the Studio Integrated Development Environment (IDE) now passes `no_defer=true` to align editor intelligence with the selected deferral behavior. (Language Server Protocol (LSP)) ##### Catalog * **Catalog: Source freshness outdated status removed**: The freshness status value `outdated` was removed; unconfigured freshness is now handled explicitly as `unconfigured`, and sources will no longer report `outdated`. * **Catalog: Rows per page selector removed from tables**: The rows-per-page selector was removed, and pagination now uses a fixed page size. ##### Orchestration and Run Status * **Orchestration: Cached and stale outcome status mapping updated**: Cached nodes are now consistently surfaced as Reused with clearer reasons, and stale outcomes are treated as errors, which can change the statuses operators see in run output and telemetry. #### January 28, 2026 ##### New * **Canvas** * **New two-step "upload source" API for more resilient uploads**: Use `POST /v1/workspaces/{workspace_id}/upload-source` to create an upload, then `PATCH /v1/workspaces/{workspace_id}/upload-source/{file_id}/process` to stream processing progress (SSE). ##### Enhancements * **Catalog & Search** * **Improved search relevance and highlighting**: Ranking now boosts results by modeling layer, and highlighting is more consistent (including support for multiple highlight snippets per field). * **dbt platform** * **Private endpoints details page**: The dbt platform now includes a Private Endpoint details view with endpoint properties, connectivity status, and associated projects. * **Fusion-aware default dbt version during setup**: Connection setup and environment creation can now default to `latest-fusion` for eligible projects. * **Studio IDE** * **Search and replace in files**: Adds a dedicated sidebar search experience. Please contact your account manager to enable. * **Autofix now includes package upgrades**: Upgrade flows can proceed from fixing deprecations into package upgrades in the same guided run. * **Editor UI polish**: Fixed multiple layout/styling issues for a more consistent editor experience. ##### Fixes * **dbt platform** * **Run logs render ANSI/structured output more reliably**: Improved rendering and cleanup of escape sequences in step logs. * **More accurate source freshness status in multi-job environments**: Freshness status is preserved when a run lacks freshness results but freshness remains configured. * **More robust seed artifact ingestion**: Ingestion now tolerates missing/null `schema` fields in the manifest to avoid failures. * **Studio IDE** * **CLI project sync no longer fails on broken symlinks**: Sync skips missing symlink targets instead of failing the whole sync. * **IDE abort is clearer when a command is missing**: Aborting a command that no longer exists returns a specific "no-command-found" response. * **More robust inline command results**: Malformed inline commands no longer break result processing; `show --inline` with an empty result returns an empty preview table. * **Canvas** * **Clearer errors for duplicate uploaded-source names**: Creating an uploaded-source model with a duplicate name now returns HTTP 409 with an actionable message. * **Failed uploads are now visible via file state**: Uploaded-source processing records failure state instead of deleting the file record, improving retry/resume workflows. * **Invocation status streaming reliability**: The invocation status SSE endpoint now correctly awaits the status stream. ##### Behavior changes * **Catalog & Search** * **Search highlight fields deprecated and highlights shape expanded**: `AccountSearchHit.highlight` and `AccountSearchHit.matchedField` are deprecated. `AccountSearchHit.highlights` now supports multiple highlight snippets per field (arrays). * **dbt platform** * **Deprecations**: The "Adaptive" job type is deprecated. `last_checked_at` is deprecated and no longer populated in run responses. * **Canvas** * **Existing CSV upload SSE endpoint deprecated**: Migrate to the new two-step [upload source](https://docs.getdbt.com/docs/platform/use-canvas.md#upload-data-to-canvas) flow. #### January 21, 2026 ##### New * **dbt platform** * **Favorites are now available in Catalog**: Add resources to favorites and organize your frequently accessed resources in the Catalog navigation. * **Connectivity / private networking** * **New v3 API endpoint to fetch a specific PrivateLink endpoint**: You can now retrieve individual PrivateLink endpoints by ID, enabling better automation and troubleshooting workflows. ##### Enhancements * **dbt platform** * **Run artifacts are now searchable**: Find specific artifacts faster in run history with the new artifacts search box and improved empty states. * **Webhooks editor is more stable**: The webhook form no longer resets while job options are loading, and server-generated fields now display reliably after creation. * **Fusion onboarding completion card can be dismissed**: After completing the Fusion onboarding checklist, you can now dismiss the card and it will stay dismissed. * **Cross-project lineage is now generally available**: Cross-project lineage is now enabled for all applicable accounts. * **Catalog & Search** * **Improved Catalog search relevance and performance**: Enhanced search scoring and matching provides more accurate results, with better column matching and highlighting for large catalogs. * **Search results are refreshed when column metadata changes**: Column name and description updates now automatically trigger re-indexing, ensuring search results stay current. * **Search typeahead includes "View all results"**: Quickly access full search results from the typeahead dropdown with the new footer link. * **Cleaner environment dropdown behavior**: The environment selector now only shows "Staging" when your account has projects with a staging environment configured. * **Studio IDE** * **Clearer error messages when fetching dev credentials and defer state**: IDE-related endpoints now return more specific and helpful error messages for common configuration issues and timeouts. * **Studio console and command log viewer improvements**: Enhanced command log viewer with improved download capabilities and more consistent error log viewing. ##### Fixes * **AI-assisted workflows** * **Enhancement:** [dbt ](https://docs.getdbt.com/docs/platform/wizard-platform.md)adds missing column descriptions more accurately. generated documentation now correctly detects column names across various `schema.yml` files, adds only missing descriptions, and preserves existing ones. * **Catalog & lineage** * **Fixes missing auto-generated exposures in model lineage**: Auto-generated exposures now appear correctly in lineage views. * **Catalog search no longer errors when a warehouse connection name is missing**: Search now handles missing connection names gracefully without causing errors. * **Improved security: malformed identity headers are rejected cleanly**: Requests with invalid authentication tokens now fail safely with clear error messages. * **Studio IDE** * **Command status is more reliable when Cloud CLI invocation data expires**: Commands that can't be fetched are now properly marked as failed instead of staying in a "running" state. * **APIs** * **Jobs API deferral validation is stricter and clearer**: Job deferral settings are now validated to ensure the deferring job and environment exist within the same account, with improved error messages. ##### Behavior changes * **dbt platform** * **Account Insights default page size changed to 5 rows**: Tables in Account Insights now display 5 rows per page by default (previously 10). * **Webhooks** * **Webhook timestamps are now consistently UTC RFC3339 with `Z`**: All webhook timestamp fields (`run_started_at`, `run_finished_at`, `timestamp`) now use UTC with `Z` suffix and higher precision. Missing/invalid timestamps emit `1970-01-01T00:00:00Z` instead of empty strings. Update webhook consumers if needed. * **Webhook `run_status` string changed from `Error` to `Errored`**: Update webhook consumers that parse this status value strictly. * **Runs / ingestion** * **Very large exposure sets are now limited during ingestion**: Projects with more than 5,000 exposures will skip exposure ingestion to prevent performance issues. All other artifact ingestion continues normally. Contact support if you need to increase this limit. #### January 14, 2026 ##### New * **dbt platform** * **Fusion migration readiness endpoint**: Added an API endpoint to determine whether a project is eligible for Fusion migration. ##### Enhancements * **Copilot and AI** * **More resilient agent runs**: Agent tool execution errors now return structured responses instead of failing the entire run. * **Better project context retrieval**: Agent toolsets include additional retrieval and search capabilities for more relevant responses. * **Improved Azure OpenAI verification**: Azure OpenAI connection verification now uses GPT-5-compatible parameters for GPT-5 deployments. * **BYOK for Azure OpenAI**: Added support for Azure Foundry URLs with automatic endpoint parsing to reduce setup friction. * **Insights and Catalog** * **Semantic Layer querying now generally available (GA)**: Build SQL queries against the Semantic Layer without writing SQL code. * **Improved search relevance**: Search scoring prioritizes exact and multi-term matches more strongly, with better highlighting and column-description matching. * **Catalog UX improvements**: Search labels are more consistent, and the embedded lineage view loads more responsively. * **Studio IDE** * **Unified Studio IDE**: Studio now loads a single unified IDE package. * **Defer-to-production honors `defer-env-id` override**: Studio now respects `dbt-cloud.defer-env-id` settings when Cloud CLI runtime is supported. * **Improved log exporting**: Download and copy behavior for command logs is more consistent, including debug logs. * **Enhanced multi-edit support**: The IDE now supports multiple explicit edits in one request with safer validation. * **Clearer Cloud CLI session errors**: Session creation returns clearer error messages and guidance for setup issues. * **dbt platform** * **Settings detail pages in resizable drawer**: Settings detail experiences now use an improved drawer-based UI. * **More resilient profile creation**: Profile creation now handles dependencies and failures more gracefully. * **Enhanced logging limits for in-progress runs**: Logs for in-progress runs are also limited by memory usage, in addition to the existing 1,000-line limit. ##### Fixes * **dbt platform** * **Profiles API clearing extended attributes**: The Profiles API now allows unsetting extended attributes by setting `extended_attributes_id` to null. * **Recently viewed more reliable**: Recently viewed entries now update atomically and retain the 5 most recent items. * **Run log tailing improvements**: Debug logs for completed runs now consistently fetch only the tail of the log. * **Studio IDE** * **More reliable `show` and `compile`**: CLI flags to disable caching are now positioned correctly to avoid parsing issues. * **Canvas preview improvements**: Fixed argument ordering so `--no-defer` is interpreted consistently. ##### Behavior changes * **dbt platform** * **dbt v1.7 end-of-life**: dbt v1.7 is now labeled as end-of-life in version lifecycle messaging. #### January 7, 2026 No changes of note this week. #### December 24, 2025 ##### New * **AI Codegen** * **File-aware LangGraph agents**: Analysts can now drop `@path` references in the bundled CLI to stream local files into `/private/v1/agents/run`, which are auto-rendered as text inside the run so copilots have the exact config or SQL snippet you referenced. * **dbt platform** * **Slack Copilot feedback loops**: Copilot replies now carry inline "Did that answer your question?" buttons, so you can rate answers without leaving Slack. * **Codex workflows** * **Databricks cost tracking for Model Cost Over Time**: A Databricks history provider and DBU-based cost query now surface daily model cost alongside Snowflake coverage, so Databricks tenants get unified FinOps reporting. * **Canvas** * **CSV upload GA**: The CSV upload endpoint is now generally available. ##### Enhancements * **Cloud artifacts** * **Better similar-model suggestions**: Attachment workflows now only recommend meaningfully related models. * **dbt platform** * **Unified SSO & SCIM admin**: Settings consolidate SSO + SCIM, add an empty state for auto-generated slugs, and render read-only login URLs so admins can start configuration without touching slug fields. * **SCIM token management polish**: Token tables gain fixed pagination, inline search, consistent iconography, and clearer deletion warnings to avoid accidental cuts to live integrations. * **Twice the per-environment custom variables**: The v3 API/UI now allow up to 20 scoped environment variables before enforcing limits, giving larger projects more room for secrets. * **Canvas** * **Dialect-aware projection SQL**: SELECT \* RENAME/EXCEPT support now respects each warehouse's syntax using schema metadata, so SQL previews and column metadata stay accurate across Snowflake, Databricks, BigQuery, and Redshift. ##### Fixes * **dbt platform** * **Webhook editor keeps job selections**: Default values are cached after the first render and stop resetting once the user edits the form, eliminating accidental job-list clearing while tabbing through fields. * **Codex GraphQL** * **Exposure parents mirror the manifest**: `parentsModels` and `parentsSources` now derive from the manifest's `parents` list, so exposures with mixed upstreams display complete lineage in both the GraphQL API and UI. ##### Behavior changes * **dbt platform** * **Legacy Cost Management UI retired**: All cost management pages and hooks were removed, and platform metadata credentials now only expose catalog ingestion and Cost Insights toggles, eliminating dead-end controls. #### December 17, 2025 ##### New * **dbt platform** * **Feature licensing service**: A new `/accounts/<id>/feature-licenses` endpoint issues short-lived JWTs that encode entitled features, and service/PAT authentication now checks that a caller holds an active license on the target account before any Fusion-enabled workflow runs. * **Databricks platform metadata credentials**: Databricks warehouses can register platform metadata credentials (token plus optional catalog), enabling catalog ingestion, metadata sharing, and Cost Insights pipelines without custom adapters. ##### Enhancements * **dbt platform** * **Large list pagination**: Settings's Projects and Credentials now paginate after 25 rows (with search boxes and skeleton states), keeping navigation responsive for large deployments. * **Metadata Explorer** * **Model context & lineage polish**: Model panels now show materialization type, lineage renders metadata strips only when content exists, and upstream public-model columns load automatically for better cross-project visibility. * **Freshness clarity & Studio navigation**: Source tiles respect the `meta5161ExpiredUnconfiguredSources` flag (showing warn/error thresholds) and "Open in IDE" links now point at `/studio/{accountId}/projects/{projectId}` to drop users directly into dbt Studio. * **Insights UI** * **Copilot guardrails**: The Copilot listener now hydrates builder tabs only when a semantic-layer payload arrives, preventing plain-SQL replies from overwriting editor state. * **dbt CLI** * **Improved monorepo support for file sync and the IDE**: * File sync now anchors itself to the invocation directory, making monorepo structures behave more predictably. * Nested `dependencies.yml` files correctly trigger dependency installs. * The IDE's LSP and file sync now recognize dbt subdirectories properly. * Exclusion lists remain accurate even in multi-project repositories. * **Notifications system** * **Webhook auditability**: Outbound calls now persist the exact JSON body in webhook history, making allowlisting and troubleshooting easier. * **Studio** * **Git sidebar & file refresh parity**: The file tree now mirrors Cloud VCS statuses (including conflicts) and automatically invalidates caches after `dbt deps`/`dbt clean`, so new or removed files appear without a reload. * **Log viewers & Autofix UX**: Command and interactive query logs adopt the new accordion-based viewer, and Autofix sessions in Fusion treat plain `parse` commands as the trigger for deprecation summaries, keeping remediation flows consistent. ##### Fixes * **dbt platform** * **Environment variable editor stability**: Editing one variable no longer backfills blank cells with previously edited values, preventing accidental overrides. * **Cost optimization indicator accuracy**: Job pages once again display "Cost optimization features" whenever Fusion actually runs (and gating conditions are met), so users see the right coverage status regardless of feature-flag permutations. ##### Behavior changes * **dbt platform** * **Stronger tenant identity enforcement**: Service/PAT calls without an active license now fail authentication, Slack Copilot sessions build a scoped identity JWT for the invoking user, and SSO providers enforce auto-generated slugs (draft configs can't be targeted), reducing misconfiguration risk. * **dbt CLI** * **User-isolated invocation history**: Every invocation lookup validates the caller's user ID, preventing admins from accidentally reading another developer's runs when multiple accounts share a CLI server. * **IDE server** * **Enhanced security for support-assisted sessions:** Support impersonation sessions now restrict the execution of `show`, `run`, `build`, and `test` commands. Artifacts generated by `dbt show` are also short-lived and will automatically expire after 15 minutes to limit unintended data retention. * **dbt Orchestration** * **Fusion compare support & new dependency**: Fusion tracks now treat `dbt compare` as a supported command (no more target-path hacks). #### December 10, 2025 ##### Enhancements * **AI codegen API**: Streaming middleware enforces request-scoped instrumentation across every AI endpoint, offload warehouse calls via threads, and expose human-readable tool names while gating keyword search behind feature flag for approved tenants. * **dbt platform** * **Operations clarity**: Environment profile drawers link directly to connection settings and treat Snowflake fields as optional, while Compare Changes and run-step drawers now explain whether steps failed or were skipped so troubleshooting is faster. * **Collaboration & notifications**: Slack Copilot mentions are now more reliable, with hardened workers, support for CSV attachments, and improved logging. Webhook channels now accept longer URLs, handle "warning-only" subscriptions correctly, and automatically clean up corrupted job IDs. * **Profile & credential management**: Environment APIs accept `secondary_profile_ids`, run acquisition favors profile-backed credentials, and whoami/auth metrics are scrubbed so cross-platform profiles stay in sync. * **dbt CLI server**: Improved stability and performance for large projects. * **Studio IDE**: For dbt Fusion logging, node start and end times will now properly be displayed in command output. * **Studio IDE**: Copilot Chat automatically appears anywhere AI entitlements exist, preview runs auto-cancel when nodes change, and keyboard shortcuts respect native keymaps with clear UI labels. * **Studio IDE**: Tab view, console pane, and command drawer have been redesigned to enhance efficiency and multitasking. ##### Fixes * **Studio IDE server**: Branch creation now returns explicit feedback for bad branch names/SHAs and detects unauthorized Git errors earlier, making automation failures actionable. #### December 3, 2025 ##### New * **dbt platform** * **Autofix deprecation warnings**: When deprecations are detected, you now see "Autofix deprecation warnings." * **Autofix Packages detailed results**: After running Autofix, you see a results panel with upgraded packages (with links), packages left unchanged and why, and quick access to `packages.yml` to help assess Fusion readiness and next steps. ##### Enhancements * **dbt platform** * **Code Quality tab improvements** * Clearer lint/format actions (SQLFluff, Prettier), better empty states, visible Config button when applicable, and simplified logs retrieval. * Applies to SQL, JSON, YAML, and Markdown workflows. * **Editor experience** * Upgraded editor for stability. * Improved container sizing/overflow. * "Save" overlay only appears when tabs are open. * Minor action‑bar refinements. ##### Fixes * **dbt platform lineage and command pane stability**: Reliability improved by aligning with updated IDE and VS Code command APIs; eliminates intermittent skips. ##### Behavior changes * **dbt platform:** dbt Core "versionless" renamed to "latest" so it's consistent and clear across tenants. --- ### Wizard Desktop settings Beta ### Wizard Desktop settings [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Open **Settings** from the bottom left of the app to change how dbt Wizard behaves. Settings are grouped into General, Account, AI providers, Connectors, Appearance, Shortcuts, and version control. Settings that also exist in the Wizard CLI are shared. Changing them here changes them for `wizard` in your terminal too. Refer to [the config reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) for the underlying files. #### General Controls what appears alongside a chat while you work. | Setting | What it does | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Replay onboarding | Walks you through first-run setup again. Your existing theme and project settings are kept unless you change them during setup. | | Application updates | Shows the version you're on, checks for new versions, and installs them. | [![General settings in Wizard Desktop](/img/wizard/desktop/wizard-desktop-settings-general.png?v=2 "General settings in Wizard Desktop")](#)General settings in Wizard Desktop #### Account Shows the dbt accounts dbt Wizard uses for hosted models and platform access, and gives you the tools to clean up what the app has written to disk. * **Connected dbt accounts** lists each dbt platform account you've connected, with the platform account hostname, such as `abc123.us1.dbt.com`. Connect more than one account by clicking on **+Add account** and switch between them without signing in again. Each account has its own **Sign out**. To pick which account powers AI models, go to [AI providers](#ai-providers). * **Maintenance** reclaims disk space or returns dbt Wizard to a clean state. This will affect the Wizard CLI too: | Action | What it does | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Clean up environments | Deletes archived environment directories from disk. The pane tells you whether any are there to remove | | Clean up app data | Drops orphaned records, and shows what the app is tracking, such as `4 environments, 1 project, 4 journal rows` | | Reset app | Permanently deletes your environments, projects, and chat history, plus cached files. It doesn't clear your configuration file at `$DBT_WIZARD_HOME`, which defaults to `~/.dbt/wizard/config.toml` | Reset app can't be undone When you select **Reset app**, it can't be undone which means chat history goes with it when you delete it. Make sure you export or open a pull request for any work you want to keep first before taking this action. #### AI providers Choose how dbt Wizard reaches a model. Come here if you selected **Skip for now** during onboarding, or to switch approaches later. Select **Refresh** to re-check which providers and models are available. If you've connected more than one dbt platform account, choose which one powers AI models here by clicking **Configure**. | Option | What it means | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Managed by dbt Labs | No setup. dbt Labs [manages](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) the keys and keeps you on the latest models, and usage is metered against your dbt usage credits. The `dbt` provider shows as **Active** when it's in use | | Managed by you | Bring your own key for Amazon Bedrock, Anthropic, Azure OpenAI, Databricks, Google Gemini, OpenAI, or an OpenAI subscription, and more. That provider's API charges apply. Refer to [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) | Each provider lists how many of its models are enabled, such as `6/6 models enabled`. Select **Manage** to add credentials and turn individual models on or off. You then pick the model and reasoning effort per chat from the chat composer, not here. #### Connectors MCP connectors give dbt Wizard context about your pipelines, BI tools, and orchestration. If your project is connected to dbt platform, dbt Wizard connects to the [remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md) by default and asks you to sign in with OAuth. This adds platform tools like job management, run status, and run errors. Use the toggle on this pane to turn it on or off for each dbt account. The remote dbt MCP server is available to dbt platform users only. dbt Wizard still uses its local [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) for project metadata. Browsing and enabling other connectors from this pane is planned for future releases. Until then, add [MCP servers](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) from the terminal: 1. Start a project chat and send your first message, which starts the terminal. 2. Run `wizard` from the terminal. 3. Run `/mcp` to add a connector or manage the ones you've already added. #### Appearance Sets how the app looks and how much detail inline widgets show. | Setting | What it does | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Theme | Switches between **Light**, **Dark**, and **System**. Same choice you made during onboarding. Cycle it with `⌘` + `⌥` + `T` | | Transcript density | Controls how much tool activity shows in chat. Choose **Verbose** to see everything, **Focused** to see the least, or **Balanced** for somewhere in between. Back-to-back tool calls collapse into one card, and you can fine-tune what shows per category | | Inspector rail labels | Shows the surface names beneath the pane rail icons, so you get **Explorer** and **Lineage** as text rather than icons alone | | Terminal font family | The font the integrated terminals use. Enter the name exactly as it's installed, for example `FiraCode Nerd Font`. Your preferred family is used first, with `Geist Mono, monospace` as the fallback | | Terminal font size | A whole number from 8 to 32 | #### Shortcuts Lists the app's keyboard shortcuts, grouped by what they act on. | Shortcut | What it does | | --------------- | --------------------------------------- | | `⌘` + `,` | Open **Settings** | | `⌘` + `K` | Open the command palette | | `⌘` + `/` | Show the keyboard shortcuts | | `⌘` + `L` | Focus the chat | | `Esc` | Close an overlay, or go back to the app | | `⌘` + `T` | Open a new chat | | `⌘` + `⇧` + `\` | Toggle the right sidebar | | `⌘` + `⌥` + `T` | Cycle the theme | #### Version control *Only GitHub is supported at this time. Support for more version control providers coming soon.* Set how dbt Wizard authenticates with GitHub to read repos and open pull requests with **Create PR**. Refer to [Review and validate changes](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-use.md#review-and-validate-changes). | Method | When to use it | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GitHub CLI integration | The default. When the [GitHub CLI](https://cli.github.com/) is installed and authenticated, the pane shows a green status dot and the account you're signed in as | | Personal access token | Optional, and used when the GitHub CLI isn't available. Select **Create one** to generate a token on GitHub, paste it in, then select **Save** | #### Related docs * [Get started with Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) * [Use Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop-use.md) * [dbt Wizard config reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [Use MCP servers](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### Write queries with exports dbt platform | Starter, Enterprise, Enterprise+ Exports enhance [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) by running your saved queries and writing the output to a table or view within your data platform. Saved queries are a way to save and reuse commonly used queries in MetricFlow, exports take this functionality a step further by: * Enabling you to write these queries within your data platform using the dbt job scheduler. * Providing an integration path for tools that don't natively support the Semantic Layer by exposing tables of metrics and dimensions. Essentially, exports are like any other table in your data platform — they enable you to query metric definitions through any SQL interface or connect to downstream tools without a first-class [Semantic Layer integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). Running an export counts towards [queried metrics](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#what-counts-as-a-queried-metric) usage. Querying the resulting table or view from the export does not count toward queried metric usage. #### Prerequisites * You have a dbt account on a [Starter or Enterprise-tier](https://www.getdbt.com/pricing/) plan. * You use one of the following data platforms: Snowflake, BigQuery, Databricks, Redshift, or Postgres. * You are on [dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) 1.7 or newer. * You have the Semantic Layer [configured](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) in your dbt project. * You have a dbt environment with the [job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) enabled. * You have a [saved query](https://docs.getdbt.com/docs/build/saved-queries.md) and [export configured](https://docs.getdbt.com/docs/build/saved-queries.md#configure-exports) in your dbt project. In your configuration, leverage [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) to cache common queries, speed up performance, and reduce compute costs. #### Benefits of exports The following section explains the main benefits of using exports:  DRY representation Currently, creating tables often involves generating tens, hundreds, or even thousands of tables that denormalize data into summary or metric mart tables. The main benefit of exports is creating a "Don't Repeat Yourself (DRY)" representation of the logic to construct each metric, dimension, join, filter, and so on. This allows you to reuse those components for long-term scalability, even if you're replacing manually written SQL models with references to the metrics or dimensions in saved queries.  Easier changes Exports ensure that changes to metrics and dimensions are made in one place and then cascade to those various destinations seamlessly. This prevents the problem of needing to update a metric across every model that references that same concept.  Caching Use exports to pre-populate the cache, so that you're pre-computing what you need to serve users through the dynamic Semantic Layer APIs. ###### Considerations Exports offer many benefits and it's important to note some use cases that fall outside the advantages: * Business users may still struggle to consume from tens, hundreds, or thousands of tables, and choosing the right one can be a challenge. * Business users may also make mistakes when aggregating and filtering from the pre-built tables. For these use cases, use the dynamic [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) instead of exports. #### Run exports Before you're able to run exports in development or production, you'll need to make sure you've [configured saved queries and exports](https://docs.getdbt.com/docs/build/saved-queries.md) in your dbt project. In your saved query config, you can also leverage [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) with the dbt job scheduler to cache common queries, speed up performance, and reduce compute costs. There are two ways to run an export: * [Run exports in development](#exports-in-development) to test the output before production. In Studio IDE or any environment on dbt v2, run `dbt build` instead — enable the [environment variable](#set-environment-variable) first. In the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) outside dbt v2, run `dbt sl export` or `dbt sl export-all`. * [Run exports in production](#exports-in-production) using the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to write these queries within your data platform. #### Exports in development You can run an export in your development environment if you want to test its output before production. This section explains the different commands and options available to run exports in development. If your environment runs on dbt v2 (including Studio IDE), use `dbt build` instead of the following commands. Check out [Run exports](#run-exports) for details. * Use the [`dbt sl export` command](#exports-for-single-saved-query) to test and generate exports in your development environment for a singular saved query. You can also use the `--select` flag to specify particular exports from a saved query. * Use the [`dbt sl export-all` command](#exports-for-multiple-saved-queries) to run exports for multiple saved queries at once. This command provides a convenient way to manage and execute exports for several queries simultaneously, saving time and effort. ##### Exports for single saved query These commands apply when you're using the dbt platform CLI outside Studio IDE or dbt v2. If your environment runs on dbt v2, use `dbt build` instead. For more info, check out [Run exports](#run-exports). Use the following command to run exports in the dbt platform CLI: ```bash dbt sl export ``` The following table lists the options for `dbt sl export` command, using the `--` flag prefix to specify the parameters: | Parameters | Type | Required | Description | | ------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------- | | `name` | String | Required | Name of the `export` object. | | `saved-query` | String | Required | Name of a saved query that could be used. | | `select` | List or String | Optional | Specify the names of exports to select from the saved query. | | `exclude` | String | Optional | Specify the names of exports to exclude from the saved query. | | `export_as` | String | Optional | Type of export to create from the `export_as` types available in the config. Options available are `table` or `view`. | | `schema` | String | Optional | Schema to use for creating the table or view. | | `alias` | String | Optional | Table alias to use to write the table or view. | You can also run any export defined for the saved query and write the table or view in your development environment. Refer to the following command example and output: ```bash dbt sl export --saved-query sq_name ``` The output would look something like this: ```bash Polling for export status - query_id: 2c1W6M6qGklo1LR4QqzsH7ASGFs.. Export completed. ``` ##### Use the select flag You can have multiple exports for a saved query and by default, all exports are run for a saved query. You can use the `select` flag in [development](#exports-in-development) to select specific or multiple exports. You can't sub-select metrics or dimensions from the saved query. You can only change the export configuration, such as the table format or schema. For example, the following command runs `export_1` and `export_2` and doesn't work with the `--alias` or `--export_as` flags: ```bash dbt sl export --saved-query sq_name --select export_1,export2 ``` Overriding export configurations The `--select` flag is mainly used to include or exclude specific exports. If you need to change these settings, you can use the following flags to override export configurations: * `--export-as` — Defines the materialization type (table or view) for the export. This creates a new export with its own settings and is useful for testing in development. * `--schema` — Specifies the schema to use for the written table or view. * `--alias` — Assigns a custom alias to the written table or view. This overrides the default export name. Be careful. The `--select` flag *can't* be used with `alias` or `schema`. For example, you can use the following command to create a new export named `new_export` as a table: ```bash dbt sl export --saved-query sq_number1 --export-as table --alias new_export ``` ##### Exports for multiple saved queries Use the command, `dbt sl export-all`, to run exports for multiple saved queries at once. This is different from the `dbt sl export` command, which only runs exports for a singular saved query. For example, to run exports for multiple saved queries, you can use: ```bash dbt sl export-all ``` The output would look something like this: ```bash Exports completed: - Created TABLE at `DBT_SL_TEST.new_customer_orders` - Created VIEW at `DBT_SL_TEST.new_customer_orders_export_alias` - Created TABLE at `DBT_SL_TEST.order_data_key_metrics` - Created TABLE at `DBT_SL_TEST.weekly_revenue` Polling completed ``` The command `dbt sl export-all` provides the flexibility to manage multiple exports in a single command. #### Exports in production Enabling and executing exports in dbt optimizes data workflows and ensures real-time data access. It enhances efficiency and governance for smarter decisions. Exports use the default credentials of the production environment. To enable exports to run saved queries and write them within your data platform, perform the following steps: 1. [Set an environment variable](#set-environment-variable) in dbt. 2. [Create and execute export](#create-and-execute-exports) job run. ##### Set environment variable 1. Click **Deploy** in the top navigation bar and choose **Environments**. 2. Select **Environment variables**. 3. [Set the environment variable](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables) key to (Applies to dbt v1.11 and later) `DBT_ENGINE_EXPORT_SAVED_QUERIES` and the environment variable's value to `TRUE`. Doing this ensures saved queries and exports are included in your dbt build job. For example, running `dbt build -s sq_name` runs the equivalent of `dbt sl export --saved-query sq_name` in the dbt Job scheduler. If exports aren't needed, you can set the value(s) to `FALSE` ((Applies to dbt v1.11 and later) `DBT_ENGINE_EXPORT_SAVED_QUERIES=FALSE`). [![Add an environment variable to run exports in your production run.](/img/docs/dbt-platform/semantic-layer/env-var-dbt-exports.png?v=2 "Add an environment variable to run exports in your production run.")](#)Add an environment variable to run exports in your production run. When you run a build job, any saved queries downstream of the dbt models in that job will also run. To make sure your export data is up-to-date, run the export as a downstream step (after the model). ##### Create and execute exports 1. Create a [deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) and ensure the (Applies to dbt v1.11 and later) `DBT_ENGINE_EXPORT_SAVED_QUERIES=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable). * This enables you to run any export that needs to be refreshed after a model is built. * Use the [selector syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) `--select` or `-s` option in your build command to specify a particular dbt model or saved query to run. For example, to run all saved queries downstream of the `orders` semantic model, use the following command: ```bash dbt build --select orders+ ``` 2. After dbt finishes building the models, the MetricFlow Server processes the exports, compiles the necessary SQL, and executes this SQL against your data platform. It directly executes a "create table" statement so the data stays within your data platform. 3. Review the exports' execution details in the jobs logs and confirm the export was run successfully. This helps troubleshoot and to ensure accuracy. Since saved queries are integrated into the dbt DAG, all outputs related to exports are available in the job logs. 4. Your data is now available in the data platform for querying! 🎉 #### FAQs  Can I have multiple exports in a single saved query? Yes, this is possible. However, the difference would be the name, schema, and materialization strategy of the export.  How do I run all exports for a saved query? * In production runs, you can build the saved query by calling it directly in the build command, or you build a model and any exports downstream of that model. * In development, run all exports by running `dbt sl export --saved-query sq_name`. If your environment runs on dbt v2 (including Studio IDE), run `dbt build` instead.  Will I run duplicate exports if multiple models are downstream of my saved query? dbt will only run each export once even if it builds multiple models that are downstream of the saved query. For example, you could have a saved query called `order_metrics`, which has metrics from both the `orders` and `order_items` semantic models. You can run a job that includes both models using `dbt build`. This runs both the `orders` and `order_items` models, however, it will only run the `order_metrics` export once.  Can I reference an export as a dbt model using ref() No, you won't be able to reference an export using `ref`. Exports are treated as leaf nodes in your DAG. Modifying an export could lead to inconsistencies with the original metrics from the Semantic Layer.  How can I select saved\_queries by their resource type? To include all saved queries in the dbt build run, use the [`--resource-type` flag](https://docs.getdbt.com/reference/global-configs/resource-type.md) and run the command `dbt build --resource-type saved_query`. #### Related docs * [Validate semantic nodes in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) * Configure [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ## Faqs ### [Error] Could not find my_project package If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain *any* macros, dbt will raise an error like: ```shell Compilation Error In dispatch: Could not find package 'my_project' ``` This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Account-specific features The features in dbt are tailored to each organization's unique configuration, including user permissions, project setup, and subscription level, with guidance provided to help teams make the most of their available capabilities. This document provides a comprehensive overview of account-specific features in dbt according to plan type. #### Copilot [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is an AI-powered assistant designed to accelerate your development workflow and help you focus on delivering high-quality data. dbt Wizard is available to all users in dbt but limits are imposed according to plan type. Have a look at [dbt's pricing](https://www.getdbt.com/pricing) for more information. #### Copilot features ##### Codegen [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") dbt Wizard codegen refers to the code generation capabilities provided by dbt Wizard, an AI-powered assistant integrated into dbt. This feature allows users to generate SQL code, documentation, tests, and semantic models directly from natural language prompts, helping automate and accelerate common analytics engineering workflows.⁠⁠⁠⁠ dbt Wizard codegen uses metadata such as relationships, lineage, and model context from your dbt projects to produce contextually accurate code. This helps avoid mistakes common with generic AI tools by ensuring generated code matches your actual schema and conventions.⁠⁠⁠⁠ The code dbt Wizard generates may include: * Base/staging/semantic models (including SQL for new models) * YAML files for documentation or tests * Inline SQL expressions * Semantic model structures and metrics dbt Wizard codegen is available in the Studio IDE, Canvas, and (soon) Insights, making it possible to generate and edit code directly within these interfaces.⁠ ##### Bring your own key (BYOK) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") BYOK allows users to provide and manage their own encryption or API keys, rather than relying on keys managed by a vendor or third party. This gives organizations greater control over data security, compliance, and contracts. BYOK means users can bring and configure their own OpenAI, Anthropic, or Azure AI Foundry credentials. With BYOK, users have more control over privacy, observability, and security for their data and metadata. Take note of the following when using BYOK: * When you use your own API key, your contract with the LLM provider (not dbt Labs') applies. You are responsible for managing costs, usage limits, and data handling. This means ownership and liability for API use rests with the user, not dbt Labs. * dbt Labs does not impose usage limits on the user's key, as it does with internally managed keys. * OpenAI projects with [data residency controls](https://platform.openai.com/docs/guides/your-data#data-residency-controls) enabled and configured for the United States (project region set to US) don't currently support BYOK. These projects can only use the API key in the dbt platform configuration. Specifying custom endpoints required for data residency isn’t yet supported, and we’re evaluating a solution for this. To use BYOK, ensure your OpenAI project doesn’t have data residency controls enabled. Projects without project region settings will use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. * Currently, BYOK in dbt supports OpenAI and Azure-hosted OpenAI API keys. Users enter their key through the [account settings](https://docs.getdbt.com/docs/platform/account-settings.md), and requests made by dbt Wizard or other AI features are billed directly to the customer by the respective provider.⁠⁠⁠⁠ info The dbt Wizard experience with BYOK and Azure OpenAI will not use metadata information in Insights, Canvas, or the Studio IDE. Without this contextual data, the LLM's responses may be suboptimal compared to those generated by the default dbt AI service. This is a temporary limitation, and we are working on an update that will enable the use of Azure OpenAI APIs. If you choose to BYOK, we don't monitor or collect any data related to your usage. Some of the reasons organizations require BYOK include: * Regulatory and compliance demands (for example, keeping encryption keys or sensitive operations under customer control) * Assurances about how and where data is processed * Ability to negotiate and manage their own vendor contracts * Ability to collect their own observability metrics Note that BYOK is different from bring your own cloud (BYOC). BYOK refers to key or credential management, whereas BYOC refers to running software workloads in your own cloud environment. ##### Natural language in Canvas [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Natural language in Canvas refers to the ability to build data models visually in Canvas using plain language prompts, powered by GenAI (dbt Wizard). You can describe what you want to build or transform, and the tool generates the underlying SQL and transformation steps for you. No SQL expertise is required. It's aimed at making data modeling more accessible to less-technical users or anyone who prefers a drag-and-drop or conversational interface over hand-coding SQL.⁠⁠ Natural language lets users translate business questions or transformation requests directly into data workflows. This accelerates the process of creating governed, production-ready models while maintaining best practices and version control. You can edit Canvas models collaboratively, and you can see both the graphical workflow and the SQL code it produces.⁠⁠⁠⁠ The natural language capability is fully integrated into the Canvas workspace. You can start with a blank model and generate models or transformation steps by specifying requirements in everyday language. dbt Wizard interprets the request, constructs the model in the Canvas, and presents it visually — making it easy to refine, preview, and publish changes.⁠⁠ This approach is especially valuable for analysts and business users, allowing broader participation in data transformation tasks without losing dbt's governance, reproducibility, and code review processes. #### Canvas [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Canvas enables efficient data access and transformation through a visual interface, combining the benefits of code-driven development with AI-assisted code generation for a seamless, flexible experience. #### dbt Insights [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Insights is an interactive feature in dbt designed for writing, running, and analyzing SQL queries within an intuitive interface. It brings together SQL query execution, results visualization, and integration with dbt metadata and documentation — all in one place.⁠⁠ It supports key features such as query history, the ability to export results to CSV, basic charting (for example, line and bar charts), and direct links to Catalog and the Studio IDE for a seamless workflow between exploration and development.⁠⁠⁠⁠ Analysts can quickly analyze metrics across data, while engineers can leverage context, metadata, and dbt lineage details to debug or validate data models.⁠⁠⁠⁠ You can save and share frequently used SQL queries, and explore documentation or data lineage as you work. Each query's results are, for now, limited to 500 rows (with plans to increase this).⁠⁠ The interface supports syntax highlighting, code completion, asset linking (to easily reference dbt models/tables), and connects to the Semantic Layer for querying metrics or columns by name. While Insights supports some light visualizations and query sharing, it is not intended to replace BI tools for reporting or dashboarding. Instead, it's focused on fast ad hoc analysis and insight generation. Integrations allow users to "jump off" into downstream BI tools with their queries if needed.⁠ #### dbt Mesh cross platform [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") dbt Mesh cross-platform (sometimes called "cross-platform Mesh" or "cross-platform dbt Mesh") is a capability in dbt Mesh that allows for referencing models and sharing lineage across multiple dbt projects, even when those projects use different data warehouse platforms. #### SCIM [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") SCIM (System for Cross-Domain Identity Management) automates user identities and groups, enhancing security and simplifying admin tasks. It allows for real-time user provisioning, deprovisioning, and profile updates in dbt, primarily using Okta as the identity provider. #### Hybrid projects [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Hybrid projects refer to a setup where both dbt v1 and dbt are utilized within the same organization, often working on the same codebase or data platform. This approach enables different teams or contributors to work in the environment that aligns best with their preferences or workflows, while still benefiting from shared assets and centralized metadata. #### Enterprise security [Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Enterprise security includes robust capabilities for managing network access and user permissions, designed to safeguard sensitive data. Two widely used features that support these efforts are PrivateLink and IP allowlisting. ##### PrivateLink PrivateLink provides a secure and private connection between your organization's environments, such as databases, version control systems, or data warehouses and dbt. This setup ensures that traffic remains within the AWS network, avoiding exposure to the public internet. ##### IP allowlist IP restrictions (IP allowlist/blocklist) let organizations control which IPs can access their dbt account. #### Projects and run slots The number of projects and run slots available to your organization varies based on your selected plan tier. For detailed information, please refer to our [pricing page](https://www.getdbt.com/pricing). #### Upgrade plan dbt offers a range of plans with varying features to suit different organizational needs. For information on the different plan types and upgrading your plan, refer to our document on [How to upgrade a dbt Cloud account](https://docs.getdbt.com/faqs/Accounts/platform-upgrade-instructions.md). #### Related content * [Billing](https://docs.getdbt.com/docs/platform/billing.md) --- ### Add a seed file 1. Add a seed file: seeds/country\_codes.csv ```text country_code,country_name US,United States CA,Canada GB,United Kingdom ... ``` 2. Run `dbt seed` 3. Ref the model in a downstream model models/something.sql ```sql select * from {{ ref('country_codes') }} ``` --- ### Are the results of freshness stored anywhere? Yes! (Applies to dbt v2.0 and later) The [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) command measures how recently data was loaded and reports a pass/warning/error for each source or model based on your `warn_after` and `error_after` thresholds. dbt writes the model and source freshness results to `target/freshness.json`. When sources are included, it also writes `target/sources.json` for backward compatibility. After enabling source freshness within a job, configure [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) in your **Project Details** page, which you can find by selecting your account name on the left side menu in dbt and clicking **Account settings**. You can see the current status for source freshness by clicking **View Sources** in the job page. --- ### Are there any example dbt projects? Yes! * **Quickstart Tutorial:** You can build your own example dbt project in the [quickstart guide](https://docs.getdbt.com/docs/get-started-dbt.md) * **Jaffle Shop:** A demonstration project (closely related to the tutorial) for a fictional e-commerce store. Refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) for clone instructions, or view the [main source code](https://github.com/dbt-labs/jaffle-shop) and [source code using DuckDB](https://github.com/dbt-labs/jaffle_shop_duckdb) on GitHub. * **GitLab:** Gitlab's internal dbt project is open source and is a great example of how to use dbt at scale ([source code](https://gitlab.com/gitlab-com/content-sites/handbook/blob/main/content/handbook/enterprise-data/platform/dbt-guide.md)) * **dummy-dbt:** A containerized dbt project that populates the Sakila database in Postgres and populates dbt seeds, models, snapshots, and tests. The project can be used for testing and experimentation purposes ([source code](https://github.com/gmyrianthous/dbt-dummy)) * **Google Analytics 4:** A demonstration project that transforms the Google Analytics 4 BigQuery exports to various models ([source code](https://github.com/stacktonic-com/stacktonic-dbt-example-project), [docs](https://stacktonic.com/article/google-analytics-big-query-and-dbt-a-dbt-example-project)) * **Make Open Data:** A production-grade ELT with tests, documentation, and CI/CD (GHA) about French open data (housing, demography, geography, etc). It can be used to learn with voluminous and ambiguous data. Contributions are welcome ([source code](https://github.com/make-open-data/make-open-data). If you have an example project to add to this list, suggest an edit by clicking **Edit this page** below. --- ### Branches when changing Git providers or repositories This FAQ explains how Git branch names and commit history carry over when you switch Git providers (for example, GitHub or GitLab), start using a new repository, or reconnect your project to a different remote. It is for dbt v1 and dbt platform users: general Git behavior about remotes, branches, and history applies to both; sections on the Studio IDE, managed repositories, and disconnecting a project repository apply only to the dbt platform. Your branch names and history only appear on a new Git provider if you copied the entire repository with its history. If you only copied files or created a fresh repository, old branches do not show up. If you use the dbt platform and the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), work you saved there but have not pushed is separate and could be lost. The following sections cover common scenarios, what to expect in each case, and what action to take to keep your branches and history intact:  Git provider migration If you fully migrate the same repository to a new Git provider so all commits and branches are preserved, your branches and history typically carry over. If you only copy files, create a new repository with a single initial commit, or import only the default branch, other branches will not appear on the new remote until you push or import them. For steps to disconnect and reconnect your repository in the dbt platform, refer to [How to migrate git providers](https://docs.getdbt.com/faqs/Git/git-migration.md).  Branches missing after creating a new repository Branches may not appear after you create a new repository. When you create a brand-new repository, it starts with only the commits and branches that exist in that new remote. Branches from your previous repository do not appear automatically, even if the files look similar. If you need earlier branches and commit history, migrate the original repository so the full commit history and branches are retained. If you continue with a newly created repository, push any branches you want to keep from your local clone. If you use the dbt platform and need to disconnect or reconnect your project repository, follow [How to migrate git providers](https://docs.getdbt.com/faqs/Git/git-migration.md).  Project repository connection In the dbt platform, the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) shows branches from the Git remote connected to your project. It does not maintain a second, independent copy of your full commit history and branches. Before you disconnect the repository integration in dbt platform or switch to a different remote, save and push work you need to keep. Uncommitted changes are at risk until you **Commit and sync** (push). Until then they will not appear in your Git provider (for example, GitHub). For disconnect steps, refer to [How to migrate git providers](https://docs.getdbt.com/faqs/Git/git-migration.md). Unsaved edits in the browser can also be lost. **Commit and sync** what you need to keep, or copy it outside the Studio IDE. For information on how work is stored in the Studio IDE, refer to [Work retention](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md?version=1.12#about-the-start-up-process-and-work-retention). Pre-cutover verification To confirm branches and connectivity before switching over, create a separate project in the dbt platform pointed at the new repository, verify the branches you expect, then disconnect the old repository and remove the temporary project when finished.  From managed to self-hosted Exporting a project as a ZIP is a snapshot of files only and does not preserve your Git history or branches. To migrate properly and keep your full history, follow [Move from a managed repository to a self-hosted repository](https://docs.getdbt.com/faqs/Git/managed-repo.md). --- ### Can I add tests and descriptions in a SQL config block? dbt has the ability to define node configs in YAML files, in addition to `config()` blocks and `dbt_project.yml`. But the reverse isn't always true: there are some things in `.yml` files that can *only* be defined there. Certain properties are special, because: * They have a unique Jinja rendering context * They create new project resources * They don't make sense as hierarchical configuration * They're older properties that haven't yet been redefined as configs These properties are: * [`description`](https://docs.getdbt.com/reference/resource-properties/description.md) * [`tests`](https://docs.getdbt.com/reference/resource-properties/data-tests.md) * [`docs`](https://docs.getdbt.com/reference/resource-configs/docs.md) * `columns` * [`quote`](https://docs.getdbt.com/reference/resource-properties/columns.md#quote) * [`source` properties](https://docs.getdbt.com/reference/source-properties.md) (e.g. `loaded_at_field`, `freshness`) * [`exposure` properties](https://docs.getdbt.com/reference/exposure-properties.md) (e.g. `type`, `maturity`) * [`macro` properties](https://docs.getdbt.com/reference/resource-properties/arguments.md) (e.g. `arguments`) --- ### Can I build my models in a schema other than my target schema or split my models across multiple schemas? Yes! Use the [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) configuration in your `dbt_project.yml` file, or using a `config` block: dbt\_project.yml ```yml name: jaffle_shop ... models: jaffle_shop: marketing: +schema: marketing # models in the `models/marketing/` subdirectory will use the marketing schema ``` models/customers.sql ```sql {{ config( schema='core' ) }} ``` --- ### Can I build my seeds in a schema other than my target schema or can I split my seeds across multiple schemas? Yes! Use the [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) configuration in your `dbt_project.yml` file. dbt\_project.yml ```yml name: jaffle_shop ... seeds: jaffle_shop: +schema: mappings # all seeds in this project will use the schema "mappings" by default marketing: +schema: marketing # seeds in the "seeds/marketing/" subdirectory will use the schema "marketing" ``` --- ### Can I connect my dbt project to two databases? The meaning of the term 'database' varies with each major warehouse manager. Hence, the answer to "can a dbt project connect to more than one database?" depends on the warehouse used in your tech stack. * dbt projects connecting to warehouses like Snowflake or Bigquery—these empower one set of credentials to draw from all datasets or 'projects' available to an account—are *sometimes* said to connect to more than one database. * dbt projects connecting to warehouses like Redshift and Postgres—these tie one set of credentials to one database—are said to connect to one database only. Sidestep the 'one database problem' by relying on ELT thinking (i.e. extract -> load -> transform). Remember, dbt is not a loader--with few exceptions, it doesn't move data from sources to a warehouse. dbt is a transformer. It enters the picture after extractors and loaders have funneled sources into a warehouse. It moves and combines data inside the warehouse itself. Hence, instead of thinking "how do I connect my dbt project to two databases", ask "what loader services will best prepare our warehouse for dbt transformations." For more on the modern 'ELT-powered' data stack, see the "dbt and the modern BI stack" section of this [dbt blog post](https://blog.getdbt.com/what-exactly-is-dbt). --- ### Can I define private packages in the dependencies.yml file? It depends on how you're accessing your private packages: * If you're using [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages), you can define them in the `dependencies.yml` file. * If you're using the [git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method), use `packages.yml` on dbt v1 because Jinja isn't supported in `dependencies.yml`. On dbt v2, you can use `dependencies.yml`, but we recommend using `packages.yml` if your project must also run on dbt v1. Refer to [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md). --- ### Can I document things other than models, like sources, seeds, and snapshots? Yes! You can document almost everything in your project using the `description:` key. Check out the reference docs on [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) for more info! --- ### Can I pay via invoice? Currently for Starter plans, self-service dbt payments must be made with a credit card and by default, they will be billed monthly based on the number of [active developer seats and usage](https://docs.getdbt.com/docs/platform/billing.md). We don't have any plans to do invoicing for self-service teams in the near future, but we *do* currently support invoices for companies on the **dbt Enterprise or Enterprise+ plans.** Feel free to [contact us](https://www.getdbt.com/contact) to build your Enterprise pricing. --- ### Can I set a different connection at the environment level? dbt supports [Connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management), available to all dbt users. Connections allows different data platform connections per environment, eliminating the need to duplicate projects. Projects can only use multiple connections of the same warehouse type. Connections are reusable across projects and environments. In dbt v1, you can maintain separate production and development environments through the use of [`targets`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) within a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md). dbt v1 users can define different targets in their profiles.yml, which means you can have targets for different data warehouses for the same profile. --- ### Can I set test failure thresholds? You can use the `error_if` and `warn_if` configs to set custom failure thresholds in your tests. For more details, see [reference](https://docs.getdbt.com/reference/resource-configs/severity.md) for more information. You can also try the following solutions: * Setting the [severity](https://docs.getdbt.com/reference/resource-configs/severity.md) to `warn` or `error` * Writing a [custom generic test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) that accepts a threshold argument ([example](https://discourse.getdbt.com/t/creating-an-error-threshold-for-schema-tests/966)) --- ### Can I store my data tests in a directory other than the `tests` directory in my project? By default, dbt expects your singular data test files to be located in the `tests` subdirectory of your project, and generic data test definitions to be located in `tests/generic` or `macros`. To change this, update the [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml test-paths: ["my_cool_tests"] ``` Then, you can define generic data tests in `my_cool_tests/generic/`, and singular data tests everywhere else in `my_cool_tests/`. --- ### Can I store my models in a directory other than the `models` directory in my project? By default, dbt expects the files defining your models to be located in the `models` subdirectory of your project. To change this, update the [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml model-paths: ["transformations"] ``` --- ### Can I store my seeds in a directory other than the `seeds` directory in my project? By default, dbt expects your seed files to be located in the `seeds` subdirectory of your project. To change this, update the [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml seed-paths: ["custom_seeds"] ``` --- ### Can I store my snapshots in a directory other than the `snapshot` directory in my project? By default, dbt expects your snapshot files to be located in the `snapshots` subdirectory of your project. To change this, update the [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml snapshot-paths: ["snapshots"] ``` Note that you cannot co-locate snapshots and models in the same directory. --- ### Can I test the uniqueness of two columns? Yes, there's a few different options for testing the uniqueness of two columns. Consider an orders table that contains records from multiple countries, and the combination of ID and country code is unique: | order\_id | country\_code | | --------- | ------------- | | 1 | AU | | 2 | AU | | ... | ... | | 1 | US | | 2 | US | | ... | ... | Here are some approaches: ###### 1. Create a unique key in the model and test that models/orders.sql ```sql select country_code || '-' || order_id as surrogate_key, ... ``` models/orders.yml ```yml models: - name: orders columns: - name: surrogate_key data_tests: - unique ``` ###### 2. Test an expression models/orders.yml ```yml models: - name: orders data_tests: - unique: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. column_name: "(country_code || '-' || order_id)" ``` ###### 3. Use the `dbt_utils.unique_combination_of_columns` test This is especially useful for large datasets since it is more performant. Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) for more information. models/orders.yml ```yml models: - name: orders data_tests: - dbt_utils.unique_combination_of_columns: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. combination_of_columns: - country_code - order_id ``` --- ### Can I use a YAML file extension? No. At present, dbt will only search for files with a `.yml` file extension. In a future release of dbt, dbt will also search for files with a `.yaml` file extension. --- ### Can I use environment variables in my profile? Yes! Check out the docs on [environment variables](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) for more information. --- ### Can I use seeds to load raw data? Seeds should **not** be used to load raw data (for example, large CSV exports from a production database). Since seeds are version controlled, they are best suited to files that contain business-specific logic, for example a list of country codes or user IDs of employees. Loading CSVs using dbt's seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your data warehouse. --- ### Can self-hosted GitLab instances only be connected via dbt Enterprise plans? Presently yes, this is only available to Enterprise users. This is because of the way you have to set up the GitLab app redirect URL for auth, which can only be customized if you're a user on an Enterprise plan. Check out our [pricing page](https://www.getdbt.com/pricing/) for more information or feel free to [contact us](https://www.getdbt.com/contact) to build your Enterprise pricing. --- ### Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. --- ### Claude Code can't connect to the dbt MCP server If the dbt MCP server doesn't connect, confirm `.mcp.json` is at the *project root* and that the `dbt` block matches one of the examples on the [Claude setup page](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md#claude-code). For self-hosted installations, apply the same full-path fix for `uvx` (and for `--env-file` paths). For remote setups, verify the URL and headers, and try the equivalent `claude mcp add --transport http` command to compare. --- ### Claude Desktop errors Claude Desktop may return errors such as `Error: spawn uvx ENOENT`, `Error: spawn npx ENOENT`, `Could not connect to MCP server dbt-mcp`, or `Server disconnected`. For self-hosted installations, replace the `command` with the full path to `uvx`: run `which uvx` on Unix systems or `where uvx` on Windows and paste the full path into your JSON (for example, `"command": "/the/full/path/to/uvx"`). For token-based remote setups with `mcp-remote`, apply the same full-path fix for `npx`. If Claude reports that entries in `claude_desktop_config.json` are not valid and were skipped, check that you are not using `"type": "http"` in that file. Use the [token-based `mcp-remote` config](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md#desktop-remote) instead. For remote setups, double-check that the MCP URL ends in `/api/ai/v1/mcp/` (including the trailing slash) and that your `Authorization` header value is `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Logs are at `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). --- ### Configuration not working in WSL (VS Code) **Symptoms:** VS Code MCP config works on other machines or native Windows but not in WSL. **Cause:** Local user settings are not applied in WSL environments. **Solution:** Configure MCP in the WSL-specific settings instead of local user settings: 1. Open the Command Palette → **Preferences: Open Remote Settings**. 2. Or select the **Remote** tab in the Settings editor. 3. Add your MCP server configuration there. --- ### Debug Snapshot target is not a snapshot table errors If you see the following error when you try executing the snapshot command: > Snapshot target is not a snapshot table (missing `dbt_scd_id`, `dbt_valid_from`, `dbt_valid_to`) Double check that you haven't inadvertently caused your snapshot to behave like table materializations by setting its `materialized` config to be `table`. Prior to dbt version 1.4, it was possible to have a snapshot like this: ```sql {% snapshot snappy %} {{ config(materialized = 'table', ...) }} ... {% endsnapshot %} ``` dbt is treating snapshots like tables (issuing `create or replace table ...` statements) **silently** instead of actually snapshotting data (SCD2 via `insert` / `merge` statements). When upgrading to dbt versions 1.4 and higher, dbt now raises a Parsing Error (instead of silently treating snapshots like tables) that reads: ```text A snapshot must have a materialized value of 'snapshot' ``` This tells you to change your `materialized` config to `snapshot`. But when you make that change, you might encounter an error message saying that certain fields like `dbt_scd_id` are missing. This error happens because, previously, when dbt treated snapshots as tables, it didn't include the necessary [snapshot meta-fields](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields) in your target table. Since those meta-fields don't exist, dbt correctly identifies that you're trying to create a snapshot in a table that isn't actually a snapshot. When this happens, you have to start from scratch — re-snapshotting your source data as if it was the first time by dropping your "snapshot" which isn't a real snapshot table. Then dbt snapshot will create a new snapshot and insert the snapshot meta-fields as expected. --- ### Do hooks run with seeds? Yes! The following hooks are available: * [pre-hooks & post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start & on-run-end hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) Configure these in your `dbt_project.yml` file. --- ### Do hooks run with snapshots? Yes! The following hooks are available for snapshots: * [pre-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) * [on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) --- ### Do I need to add a YAML entry for column for it to appear in the docs site? Fortunately, no! dbt will introspect your warehouse to generate a list of columns in each relation, and match it with the list of columns in your `.yml` files. As such, any undocumented columns will still appear in your documentation! --- ### Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. --- ### Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. --- ### Does dbt offer extract and load functionality? dbt is a transformation tool. It is *not* designed for extract or load functionality, and dbt Labs strongly recommends against using dbt in this way. Support is not provided for extract or load functionality. --- ### Does dbt State support incremental models? dbt State works with incremental models. When you make a change to an incremental model and run it in development, dbt State automatically clones the model from production if it exists, then runs the new model logic on top of the clone. If you want to revert to the original dbt behavior and fully refresh the incremental model, pass the [`--full-refresh` flag](https://docs.getdbt.com/reference/commands/run.md#refresh-incremental-models). --- ### Does dbt State support Python models? dbt State builds Python models but does not reuse them. It executes Python models on every run, even when their code and upstream data have not changed. --- ### Does my `.yml` file containing tests and descriptions need to be named `schema.yml`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. --- ### Does my operating system have prerequisites? Your operating system may require pre-installation setup before installing dbt v1 with pip. After downloading and installing any dependencies specific to your development environment, you can proceed with the [pip installation of dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). ##### CentOS CentOS requires Python and some other dependencies to successfully install and run dbt v1. To install Python and other dependencies: ```shell sudo yum install redhat-rpm-config gcc libffi-devel \ python-devel openssl-devel ``` ##### MacOS The MacOS requires Python 3.8 or higher to successfully install and run dbt v1. To check the Python version: ```shell python --version ``` If you need a compatible version, you can download and install [Python version 3.9 or higher for MacOS](https://www.python.org/downloads/macos). If your machine runs on an Apple M1 architecture, we recommend that you install dbt via [Rosetta](https://support.apple.com/en-us/HT211861). This is necessary for certain dependencies that are only supported on Intel processors. ##### Ubuntu/Debian Ubuntu requires Python and other dependencies to successfully install and run dbt v1. To install Python and other dependencies: ```shell sudo apt-get install git libpq-dev python-dev python3-pip sudo apt-get remove python-cffi sudo pip install --upgrade cffi pip install cryptography~=3.4 ``` ##### Windows Windows requires Python and git to successfully install and run dbt v1. Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.9 or higher for Windows](https://www.python.org/downloads/windows/). For further questions, please see the [Python compatibility FAQ](https://docs.getdbt.com/faqs/Core/install-python-compatibility.md) --- ### Does the Cost Insights feature incur warehouse costs? dbt issues lightweight, read-only queries against your warehouse to retrieve metadata and to power features such as Cost Insights. dbt scopes and filters these queries to minimize impact, and most customers see negligible costs (typically on the order of cents). --- ### Errors importing a repository on dbt project set up If you don't see your repository listed, double-check that: * Your repository is in a Gitlab group you have access to. dbt will not read repos associated with a user. If you do see your repository listed, but are unable to import the repository successfully, double-check that: * You are a maintainer of that repository. Only users with maintainer permissions can set up repository connections. If you imported a repository using the dbt native integration with GitLab, you should be able to see if the clone strategy is using a `deploy_token`. If it's relying on an SSH key, this means the repository was not set up using the native GitLab integration, but rather using the generic git clone option. The repository must be reconnected in order to get the benefits described above. --- ### execute_sql tool not working **Symptoms:** The `execute_sql` tool returns an authentication error or is unavailable. **Cause:** How you fix this depends on how you connected: * **Remote MCP with OAuth:** Sign in through your MCP client when prompted. You don't need a personal access token. * **Remote MCP with token-based auth:** You need a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. You also need `x-dbt-dev-environment-id` and `x-dbt-user-id` headers. The user ID must match the user who created the PAT. * **Local MCP:** Set `DBT_TOKEN` to a personal access token (PAT). Service tokens won't work. **Solution:** 1. If you use OAuth for remote MCP, confirm you completed sign-in and consent in your MCP client. 2. If you use token-based auth or local MCP, create a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) in **Account settings** → **API tokens** → **Personal tokens**. 3. For token-based remote MCP, add the PAT to the `Authorization` header as `Token YOUR_PAT`. For local MCP, set the PAT as your `DBT_TOKEN` value. 4. Also ensure `DBT_DEV_ENV_ID` and `DBT_USER_ID` are set (local MCP) or `x-dbt-dev-environment-id` and `x-dbt-user-id` headers are set (remote MCP). Refer to [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. --- ### GitLab token refresh message When you connect dbt to a GitLab repository, GitLab automatically creates a [project access token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) in your GitLab repository in the background. This sends the job run status back to Gitlab using the dbt API for CI jobs. By default, the project access token follows a naming pattern: `dbt token for GitLab project: <project_id>`. If you have multiple tokens in your repository, look for one that follows this pattern to identify the correct token used by dbt. If you're receiving a "Refresh token" message, don't worry — dbt automatically refreshes this project access token for you, which means you never have to manually rotate it. If you still experience any token refresh errors, please try disconnecting and reconnecting the repository in your dbt project to refresh the token. For any issues, please reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### How can I connect dbt to a Google Source repository? Although we don't officially support Google Cloud as a git repository, the below workaround using the SSH URL method should help you to connect: * First: "import" your Repository into dbt using the SSH URL provided to you by GCP. That will look something like: `ssh://drew@fishtownanalytics.com@source.developers.google.com:2022/p/dbt-integration-tests/r/drew-debug` * After importing the repo, you should see a public key generated by dbt for the repository. You'll want to copy that public key into a new SSH Key for your user here: <https://source.cloud.google.com/user/ssh_keys> * After saving this SSH key, dbt should be able to read and write to this repo. If you've tried the workaround above and are still having issues connecting - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### How can I consolidate projects in dbt? Consolidating your dbt projects can be an enormous task, and there is no universal solution. But, there are some common approaches to project consolidation in dbt that you can follow, depending on the scope of the work that needs to be done. If you have multiple projects that contain production-worthy code, there are rarely straightforward solutions to merging them. Let's suppose you have `Main Project` and `Smaller Subset Project`. #### Files and Folders ##### Git and the local directory Reference the [merge git commands](https://gist.github.com/msrose/2feacb303035d11d2d05) to help complete the migration plan. Using the commands will help retain git commit history, but you might result in duplicate folders called `models`, `tests`, etc. You will most likely still have to move files around manually. Another option would be to use an external code editor (for example, VS Code) to move files from the `Smaller Subset Project` to the `Main Project`. This is what internal dbt Labs experts recommend to stay informed about what comes over to the main project and also allows you to be more aware of the incoming files, with the ability to make any minor tweaks to folder hierarchy that you might want to do at the same time. ##### Manual migration with multiple browser tabs If you only have a couple of models or macros that you want to consolidate, copy the raw file contents from your git provider in `Smaller Subset Project`. Then, in the Studio IDE, paste the contents into a new file in your `Main Project`. Alternatively, you can download those files from your git provider (`Smaller Subset Project` repo) and upload them back to your other repository (`Main Project` repo). This doesn’t scale well and could bypass change controls, so it might only be a viable solution for organizations with only a few files. #### Production jobs If you have multiple projects with deployment environments deploying jobs, this poses another challenge. Assuming all the models from `Smaller Subset Project` can be consolidated into `Main Project`, your commands within your jobs will take on a new meaning. In lieu of refactoring your global job strategy at the same time, you can add tags to the incoming project models and utilize that in your job command syntax, with the help of node selection syntax. Main Project job command example: `dbt build --exclude tag:smaller_subset_project` Smaller Subset Project commands: `dbt build --select tag:smaller_subset_project` --- ### How can I fix my .gitignore file? A `.gitignore` file specifies which files git should intentionally ignore or 'untrack'. dbt indicates untracked files in the project file explorer pane by putting the file or folder name in *italics*. If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the Studio IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt to work correctly. The following sections describe how to fix the `.gitignore` file in:  Fix in the Studio IDE To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. 1. Launch the Studio IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. 3. Open the new or existing `gitignore` file, and add the following: ```bash # ✅ Correct target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` * **Note** — You can place these lines anywhere in the file, as long as they're on separate lines. The lines shown are wildcards that will include all nested files and folders. Avoid adding a trailing `'*'` to the lines, such as `target/*`. For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com/docs/gitignore). 4. Save the changes but *don't commit*. 5. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right corner of the IDE screen and select **Restart IDE**. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 6. Once the Studio IDE restarts, go to the **File Catalog** to delete the following files or folders (if they exist). No data will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. **Save** and then **Commit and sync** the changes. 8. Restart the Studio IDE again using the same procedure as step 5. 9. Once the Studio IDE restarts, use the **Create a pull request** (PR) button under the **Version Control** menu to start the process of integrating the changes. 10. When the git provider's website opens to a page with the new PR, follow the necessary steps to complete and merge the PR into the main branch of that repository. * **Note** — The 'main' branch might also be called 'master', 'dev', 'qa', 'prod', or something else depending on the organizational naming conventions. The goal is to merge these changes into the root branch that all other development branches are created from. 11. Return to the Studio IDE and use the **Change Branch** button, to switch to the main branch of the project. 12. Once the branch has changed, click the **Pull from remote** button to pull in all the changes. 13. Verify the changes by making sure the files/folders in the `.gitignore` file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).  Fix in the Git provider Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. There are two options for this approach: editing the main branch directly if allowed, or creating a pull request to implement the changes if required: ##### Edit in main branch When permissions allow it, it's possible to edit the `.gitignore` directly on the main branch of your repo. Here are the following steps: 1. Go to your repository's web interface. 2. Switch to the main branch and the root directory of your dbt project. 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deletions to the main branch. 8. Switch to the Studio IDE , and open the project that you're fixing. 9. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 10. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 11. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 12. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! ##### Unable to edit main branch If you can't edit the `.gitignore` directly on the main branch of your repo, follow these steps: 1. Go to your repository's web interface. 2. Switch to an existing development branch, or create a new branch just for these changes (This is often faster and cleaner). 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deleted folders. 8. Open a merge request using the git provider web interface. The merge request should attempt to merge the changes into the 'main' branch that all development branches are created from. 9. Follow the necessary procedures to get the branch approved and merged into the 'main' branch. You can delete the branch after the merge is complete. 10. Once the merge is complete, go back to the Studio IDE, and open the project that you're fixing. 11. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the Studio IDE by clicking on the three dots next to the **Studio IDE Status** button on the lower right corner of the Studio IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 12. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 13. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 14. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. --- ### How can I fix my .gitignore file? A gitignore file specifies which files Git should intentionally ignore. You can identify these files in your project by their italics formatting. If you can't revert changes, check out a branch, or click commit — this is usually do to your project missing a [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file OR your gitignore file doesn't contain the necessary content inside the folder. To fix this, complete the following steps: 1. In the Studio IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 2. Save your changes but *don't commit* 3. Restart the Studio IDE by clicking on the three dots next to the **Studio IDE Status button** on the lower right of the Studio IDE. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 4. Select **Restart Studio IDE**. 5. Go back to the **File explorer** in the IDE and delete the following files or folders if you have them: * `target`, `dbt_modules`, `dbt_packages`, `logs` 6. **Save** and then **Commit and sync** your changes. 7. Restart the Studio IDE again. 8. Create a pull request (PR) under the **Version Control** menu to integrate your new changes. 9. Merge the PR on your git provider page. 10. Switch to your main branch and click on **Pull from remote** to pull in all the changes you made to your main branch. You can verify the changes by making sure the files/folders in the .gitignore file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics). For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. --- ### How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. --- ### How can I set up the right permissions in BigQuery? To use this functionality, first create the service account you want to impersonate. Then grant users that you want to be able to impersonate this service account the `roles/iam.serviceAccountTokenCreator` role on the service account resource. Then, you also need to grant the service account the same role on itself. This allows it to create short-lived tokens identifying itself, and allows your human users (or other service accounts) to do the same. More information on this scenario is available [here](https://cloud.google.com/iam/docs/understanding-service-accounts#directly_impersonating_a_service_account). Once you've granted the appropriate permissions, you'll need to enable the [IAM Service Account Credentials API](https://console.cloud.google.com/apis/library/iamcredentials.googleapis.com). Enabling the API and granting the role are eventually consistent operations, taking up to 7 minutes to fully complete, but usually fully propagating within 60 seconds. Give it a few minutes, then add the `impersonate_service_account` option to your BigQuery profile configuration. --- ### How can I update my billing information? If you want to change your account's credit card details, go to the left side panel, click **Account settings** → **Billing** → scroll to **Payment information**. Enter the new credit card details on the respective fields then click on **Update payment information**. Only the *account owner* can make this change. To change your billing name or location address, send our Support team a message at <support@getdbt.com> with the newly updated information, and we can make that change for you! --- ### How can we move our project from a managed repository, to a self-hosted repository? dbt Labs can send your managed repository through a ZIP file in its current state for you to push up to a git provider. After that, you'd just need to switch over to the [repo in your project](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to point to the new repository. When you're ready to do this, [contact the dbt Labs Support team](mailto:support@getdbt.com) with your request and your managed repo URL, which you can find by navigating to your project setting. To find project settings: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Click **Projects**, and then select your project. 3. Under **Repository** in the project details page, you can find your managed repo URL. --- ### How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. --- ### How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. --- ### How do I build one seed at a time? You can use a `--select` option with the `dbt seed` command, like so: ```shell $ dbt seed --select country_codes ``` There is also an `--exclude` option. Check out more in the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) documentation. --- ### How do I change a user license type to read-only in dbt? To change the license type for a user from `developer` to `read-only` or `IT` in dbt, you must be an account owner or have admin privileges. You might make this change to free up a billable seat but retain the user’s access to view the information in the dbt account. 1. From dbt, click on your account name in the left side menu and, select **Account settings**. [![Navigate to account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to account settings")](#)Navigate to account settings 2. In **Account Settings**, select **Users** under **Teams**. 3. Select the user you want to remove and click **Edit** in the bottom of their profile. 4. For the **License** option, choose **Read-only** or **IT** (from **Developer**), and click **Save**. [![Change user's license type](/img/docs/dbt-platform/change_user_to_read_only_20221023.gif?v=2 "Change user's license type")](#)Change user's license type License types override group permissions **User license types always override their assigned group permission sets.** For example, a user with a Read-Only license cannot perform administrative actions, even if they belong to an Account Admin group. This ensures that license restrictions are always enforced, regardless of group membership. --- ### How do I create dependencies between models? When you use the `ref` [function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), dbt automatically infers the dependencies between models. For example, consider a model, `customer_orders`, like so: models/customer\_orders.sql ```sql select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from {{ ref('stg_orders') }} group by 1 ``` **There's no need to explicitly define these dependencies.** dbt will understand that the `stg_orders` model needs to be built before the above model (`customer_orders`). When you execute `dbt run`, you will see these being built in order: ```txt $ dbt run Running with dbt=1.9.0 Found 2 models, 28 data tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 0 seed files, 3 sources 11:42:52 | Concurrency: 8 threads (target='dev_snowflake') 11:42:52 | 11:42:52 | 1 of 2 START sql view model dbt_claire.stg_jaffle_shop__orders ....... [RUN] 11:42:55 | 1 of 2 OK created sql view model dbt_claire.stg_jaffle_shop__orders .. [CREATE VIEW in 2.50s] 11:42:55 | 2 of 2 START sql view model dbt_claire.customer_orders .............. [RUN] 11:42:56 | 2 of 2 OK created sql view model dbt_claire.customer_orders ......... [CREATE VIEW in 0.60s] 11:42:56 | Finished running 2 view models in 15.13s. Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2 ``` To learn more about building a dbt project, we recommend you complete the [quickstart guide](https://docs.getdbt.com/guides.md). --- ### How do I debug my Jinja? You should get familiar with checking the compiled SQL in `target/compiled/<your_project>/` and the logs in `logs/dbt.log` to see what dbt is running behind the scenes. You can also use the [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) function to debug Jinja by printing objects to the command line. --- ### How do I define a column type? Your warehouse's SQL engine automatically assigns a [datatype](https://www.w3schools.com/sql/sql_datatypes.asp) to every column, whether it's found in a source or model. To force SQL to treat a columns a certain datatype, use `cast` functions: models/order\_prices.sql ```sql select cast(order_id as integer), cast(order_price as double(6,2)) -- a more generic way of doing type conversion from {{ ref('stg_orders') }} ``` Many modern data warehouses now support `::` syntax as a shorthand for `cast( as )`. models/orders\_prices\_colon\_syntax.sql ```sql select order_id::integer, order_price::numeric(6,2) -- you might find this in Redshift, Snowflake, and Postgres from {{ ref('stg_orders') }} ``` Be warned, reading in data and casting that data may not always yield expected results, and every warehouse has its own subtleties. Certain casts may not be allowed (e.g. on Bigquery, you can't cast a `boolean`-type value to a `float64`). Casts that involve a loss in precision loss (e.g. `float` to `integer`) rely on your SQL engine to make a best guess or follow a specific schema not used by competing services. When performing casts, it's imperative that you are familiar with your warehouse's casting rules to best label fields in your sources and models. Thankfully, popular database services tend to have type docs--[Redshift](https://docs.amazonaws.cn/en_us/redshift/latest/dg/r_CAST_function.html) and [Bigquery](https://cloud.google.com/bigquery/docs/reference/standard-sql/conversion_rules). --- ### How do I delete a project in dbt? To delete a project in dbt, you must be the account owner or have admin privileges. 1. From dbt, click on your account name in the left side menu and select **Account settings**. [![Navigate to account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to account settings")](#)Navigate to account settings 2. In **Account Settings**, select **Projects**. Click the project you want to delete from the **Projects** page. 3. Click the edit icon in the lower right-hand corner of the **Project Details**. A **Delete** option appears on the left side of the same details view. 4. Click **Delete**. Confirm the action to immediately delete the user without additional password prompts. The project is deleted immediately after confirmation. Once a project is deleted, this action cannot be undone. [![Delete projects](/img/docs/dbt-platform/delete_projects_from_dbt_cloud.png?v=2 "Delete projects")](#)Delete projects --- ### How do I delete a user in dbt? To delete a user in dbt, you must be an account owner or have admin privileges. If the user has a `developer` license type, this will open up their seat for another user or allow the admins to lower the total number of seats. 1. From dbt, click on your account name in the left side menu and, select **Account settings**. [![Navigate to account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to account settings")](#)Navigate to account settings 2. In **Account settings**, select **Users** under **Teams**. 3. Select the user you want to delete, then click **Edit**. 4. Click **Delete** in the bottom left. Click **Confirm Delete** to immediately delete the user without additional password prompts. This action cannot be undone. However, you can re-invite the user with the same information if the deletion was made in error. [![Deleting a user](/img/docs/dbt-platform/delete_user.png?v=2 "Deleting a user")](#)Deleting a user If you are on a **Starter** plan and you're deleting users to reduce the number of billable seats, follow these steps to lower the license count to avoid being overcharged: 1. In **Account Settings**, select **Billing**. 2. Under **Billing details**, enter the number of developer seats you want and make sure you fill in all the payment details, including the **Billing address** section. If you leave any field blank, you won't be able to save your changes. 3. Click **Update Payment Information** to save your changes. [![Navigate to Account settings -> Users to modify dbt users](/img/docs/dbt-platform/faq-account-settings-billing.png?v=2 "Navigate to Account settings -> Users to modify dbt users")](#)Navigate to Account settings -> Users to modify dbt users #### Related docs * [dbt licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses) --- ### How do I document macros? To document macros, use a [properties file](https://docs.getdbt.com/reference/macro-properties.md) and nest the configurations under a `macros:` key #### Example macros/properties.yml ```yml macros: - name: cents_to_dollars description: A macro to convert cents to dollars arguments: - name: column_name type: column description: The name of the column you want to convert - name: precision type: integer description: Number of decimal places. Defaults to 2. ``` tip From dbt v1.10, you can opt into validating the arguments you define in macro documentation using the `validate_macro_args` behavior change flag. When enabled, dbt will: * Infer arguments from the macro and includes them in the [manifest.json](https://docs.getdbt.com/reference/artifacts/manifest-json.md) file if no arguments are documented. * Raise a warning if documented argument names don't match the macro definition. * Raise a warning if `type` fields don't follow [supported formats](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). Learn more about [macro argument validation](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). #### Document a custom materialization When you create a [custom materialization](https://docs.getdbt.com/guides/create-new-materializations.md), dbt creates an associated macro with the following format: ```text materialization_{materialization_name}_{adapter} ``` To document a custom materialization, use the previously mentioned format to determine the associated macro name(s) to document. macros/properties.yml ```yaml macros: - name: materialization_my_materialization_name_default description: A custom materialization to insert records into an append-only table and track when they were added. - name: materialization_my_materialization_name_xyz description: A custom materialization to insert records into an append-only table and track when they were added. ``` --- ### How do I exclude a table from a freshness snapshot? Some tables in a data source may be updated infrequently. If you've set a `freshness` property at the source level, this table is likely to fail checks. To work around this, you can set the table's freshness to null (`freshness: null`) to "unset" the freshness for a particular table: models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop config: freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at tables: - name: orders - name: product_skus config: freshness: null # do not check freshness for this table ``` --- ### How do I load data into my warehouse? dbt assumes that you already have a copy of your data, in your data warehouse. We recommend you use an off-the-shelf tool like [Stitch](https://www.stitchdata.com/) or [Fivetran](https://fivetran.com/) to get data into your warehouse. **Can dbt be used to load data?** No, dbt does not extract or load data. It focuses on the transformation step only. --- ### How do I populate the owner column in the generated docs? You cannot change the `owner` column in your generated documentation. dbt pulls the `owner` field in `dbt-docs` from database metadata ([catalog.json](https://docs.getdbt.com/reference/artifacts/catalog-json.md)), meaning the `owner` of that table in the database. With the exception of [exposures](https://docs.getdbt.com/docs/build/exposures.md), dbt does not pull this value from an `owner` field set within dbt. Generally, dbt's database user owns the tables created in the database. The service responsible for ingesting or loading the data usually owns the source tables. If you set `meta.owner`, that field appears under **meta** (pulled from dbt), but still not under the top-level `owner` field. #### Example The following example shows a model with `meta.owner` so it appears under **meta** in the docs. Replace `DATA_TEAM_EMAIL` with your own values. models/stg\_orders.yml ```yaml models: - name: stg_orders description: "Staging table for order events." config: meta: owner: "DATA_TEAM_EMAIL" columns: - name: order_id description: "Primary key for orders." - name: order_date description: "Date when order was placed." ``` --- ### How do I preserve leading zeros in a seed? If you need to preserve leading zeros (for example in a zipcode or mobile number), include leading zeros in your seed file, and use the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) with a varchar datatype of the correct length. --- ### How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. --- ### How do I run data tests on just my sources? To run data tests on all sources, use the following command: ```shell dbt test --select "source:*" ``` (You can also use the `-s` shorthand here instead of `--select`) To run data tests on one source (and all of its tables): ```shell $ dbt test --select source:jaffle_shop ``` And, to run data tests on one source table only: ```shell $ dbt test --select source:jaffle_shop.orders ``` --- ### How do I run models downstream of a seed? You can run models downstream of a seed using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md), and treating the seed like a model. For example, the following would run all models downstream of a seed named `country_codes`: ```shell $ dbt run --select country_codes+ ``` --- ### How do I run models downstream of one source? To run models downstream of a source, use the `source:` selector: ```shell $ dbt run --select source:jaffle_shop+ ``` (You can also use the `-s` shorthand here instead of `--select`) To run models downstream of one source table: ```shell $ dbt run --select source:jaffle_shop.orders+ ``` Check out the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) for more examples! --- ### How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. --- ### How do I run one snapshot at a time? To run one snapshot, use the `--select` flag, followed by the name of the snapshot: ```shell $ dbt snapshot --select order_snapshot ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. --- ### How do I set a datatype for a column in my seed? dbt will infer the datatype for each column based on the data in your CSV. You can also explicitly set a datatype using the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) like so: dbt\_project.yml ```yml seeds: jaffle_shop: # you must include the project name warehouse_locations: +column_types: zipcode: varchar(5) ``` --- ### How do I snapshot freshness for one source only? Use the `--select` flag to snapshot freshness for specific sources. Eg: ```shell # Snapshot freshness for all Jaffle Shop tables: $ dbt source freshness --select source:jaffle_shop # Snapshot freshness for a particular source table: $ dbt source freshness --select source:jaffle_shop.orders # Snapshot freshness for multiple particular source tables: $ dbt source freshness --select source:jaffle_shop.orders source:jaffle_shop.customers ``` See the [`source freshness` command reference](https://docs.getdbt.com/reference/commands/source.md) for more information. --- ### How do I specify column types? Simply cast the column to the correct type in your model: ```sql select id, created::timestamp as created from some_other_table ``` You might have this question if you're used to running statements like this: ```sql create table dbt_alice.my_table id integer, created timestamp; insert into dbt_alice.my_table ( select id, created from some_other_table ) ``` In comparison, dbt would build this table using a `create table as` statement: ```sql create table dbt_alice.my_table as ( select id, created from some_other_table ) ``` So long as your model queries return the correct column type, the table you create will also have the correct column type. To define additional column options: * Rather than enforcing uniqueness and not-null constraints on your column, use dbt's [data testing](https://docs.getdbt.com/docs/build/data-tests.md) functionality to check that your assertions about your model hold true. * Rather than creating default values for a column, use SQL to express defaults (e.g. `coalesce(updated_at, current_timestamp()) as updated_at`) * In edge-cases where you *do* need to alter a column (e.g. column-level encoding on Redshift), consider implementing this via a [post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). --- ### How do I test and document seeds? To test and document seeds, use a [properties file](https://docs.getdbt.com/reference/configs-and-properties.md) and nest the configurations under a `seeds:` key #### Example seeds/properties.yml ```yml seeds: - name: country_codes description: A mapping of two letter country codes to country names columns: - name: country_code data_tests: - unique - not_null - name: country_name data_tests: - unique - not_null ``` --- ### How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. --- ### How do I transfer account ownership to another user? You can transfer your dbt [access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to another user by following the steps below, depending on your dbt account plan: | Account plan | Steps | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Developer** | You can transfer ownership by changing the email directly on your dbt profile page, which you can access using this URL when you replace `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/settings/profile`. Before doing this, please ensure that you unlink your GitHub profile. The email address of the new account owner cannot be associated with another dbt account. | | **Starter** | Existing account admins with account access can add users to, or remove users from the owner group. | | **Enterprise or Enterprise+** | Account admins can add users to, or remove users from a group with Account Admin permissions. | | **If all account owners left the company** | If the account owner has left your organization, you will need to work with *your* IT department to have incoming emails forwarded to the new account owner. Once your IT department has redirected the emails, you can request to reset the user password. Once you log in, you can change the email on the Profile page when you replace `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/settings/profile`. | When you make any account owner and email changes: * The new email address *must* be verified through our email verification process. * You can update any billing email address or [Notifications Settings](https://docs.getdbt.com/docs/deploy/job-notifications.md) to reflect the new account owner changes, if applicable. * When transferring account ownership, please ensure you [unlink](https://docs.getdbt.com/faqs/Accounts/git-account-in-use.md) your GitHub account in dbt. This is because you can only have your Git account linked to one dbt user account. --- ### How do I troubleshoot if cost data isn't appearing? If cost data isn't appearing in Cost Insights, check the following: * Verify that platform metadata credentials are configured in your account settings and that the credential test is passing. For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials). * Ensure you have one of the required permissions to view cost data. For more information, see [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * Confirm that at least one job is running in a production environment. Cost data only appears after jobs have executed. * Cost data refreshes daily and reflects the previous day's usage, which means there is a lag of up to one day between when a job runs and when its cost data appears. If you just ran a job, wait until the next day to see the data. * After enabling Cost Insights, dbt looks back 10 days to build baselines for cost reduction calculations. If you don't see cost reduction data, ensure you have sufficient job history within the last 10 days. --- ### How do I use the 'Custom Branch' settings in a dbt Environment? In dbt environments, you can change your git settings to use a different branch in your dbt project repositories besides the default branch. When you make this change, you run dbt on a custom branch. When specified, dbt executes models using the custom branch setting for that environment. Development and deployment environments have slightly different effects. To specify a custom branch: 1. Edit an existing environment or create a new one 2. Select **Only run on a custom branch** under General Settings 3. Specify the **branch name or tag** #### Development In a development environment, the primary branch (usually named `main`) is protected in your connected repositories. You can directly edit, format, or lint files and execute dbt commands in your protected default git branch. Since the Studio IDE prevents commits to the protected branch, you can commit those changes to a new branch when you're ready. Specifying a **Custom branch** overrides the default behavior. It makes the custom branch protected and enables you to create new development branches from it. You can directly edit, format, or lint files and execute dbt commands in your custom branch, but you cannot make commits to it. dbt prompts you to commit those changes to a new branch. Only one branch can be protected. If you specify a custom branch, the primary branch is no longer protected. If you want to protect the primary branch and prevent any commits on it, you need to set up branch protection rules in your git provider settings. This ensures your primary branch remains secure and no new commits can be made to it. For example, if you want to use the `develop` branch of a connected repository: 1. Go to an environment and click **Settings** > **Edit** to edit the environment. 2. Select **Only run on a custom branch** in **General settings**. 3. Enter **develop** as the name of your custom branch. 4. Click **Save**. [![Configuring a custom base repository branch](/img/docs/dbt-platform/platform-configuring-dbt-platform/dev-environment-custom-branch.png?v=2 "Configuring a custom base repository branch")](#)Configuring a custom base repository branch #### Deployment When running jobs in a deployment environment, dbt will clone your project from your connected repository before executing your models. By default, dbt uses the default branch of your repository (commonly the `main` branch). To specify a different version of your project for dbt to execute during job runs in a particular environment, you can edit the Custom Branch setting as shown in the previous steps. --- ### How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. --- ### How does dbt State calculate that a model has changed? dbt State only considers substantial changes to a model. Because dbt State understands the entire lineage of your models, it can see through things like whitespace and aliases to determine whether a model is the same or different across environments. By default, dbt State compares rendered SQL to detect code changes. Any change to the rendered SQL — including from non-deterministic macros or environment variables — triggers a rebuild. You can enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md) to also check the Jinja template (unrendered code). When enabled, a rebuild only occurs when both the template *and* the rendered SQL have changed. Non-deterministic values (for example, `{{ env_var('AIRFLOW_RUN_ID') }}` or a macro that calls `uuid()`) don't trigger rebuilds as long as the template itself is unchanged, which helps avoid unnecessary warehouse compute costs. --- ### How does increasing job frequency affect cost reduction estimates? Cost reduction metrics reflect how dbt optimizes compute costs by reusing existing results instead of running the same model again. When you increase your job run frequency (for example, because performance improvements make it easier to schedule jobs more often), dbt has more opportunities to reuse models. As reuse increases, dbt optimizes more compute, which means your reported cost reductions may also increase. This metric shows the efficiency impact of reuse within your current workload. It reflects the compute costs that dbt reduces by reusing models instead of rebuilding them, rather than showing your total warehouse spend reduction. --- ### How is data stored in dbt State? dbt State sends the following metadata to dbt Labs servers: * **Last-modified timestamps**: Used to determine whether upstream data has changed since the last run * **SQL statement hashes**: SQL statements are processed to detect and classify changes, then hashed. Only the hash is persisted for future comparisons. No actual data from your warehouse is transmitted. The dbt State service runs in a single US multi-tenant (MT) instance. The service never connects to your data warehouse. No actual data from your warehouse is transmitted. The only connection is to your running dbt process (CLI or platform) in order to exchange the metadata described above. For data retention details, refer to the [dbt Labs privacy policy](https://www.getdbt.com/cloud/privacy-policy). --- ### How is dbt State different from using state:modified? On its own, `state:modified` requires manual management of `manifest.json`, which is cumbersome and error-prone. dbt State is completely managed with almost zero setup and no workflow changes. In self-managed deployments, dbt State can also power `state:*` selectors directly — see the last item below. `state:modified` only checks if a file has changed. dbt State has semantic understanding of SQL, so meaningless changes like whitespace or table aliases are not counted as a change — making dbt State smarter about what actually needs to rebuild. `state:modified` does not consider upstream data changes. dbt State checks all sources to see if there is any new data or if the schema has been modified. This enables dbt State to skip running models if the result of the run would be the same as before. For example, if you run `dbt run` with `state:modified` twice, it runs all modified models both times. dbt State only reruns models the second time if upstream sources have changed. dbt State also has the ability to auto-defer refs and automatically clone tables when the result of the clone would have been the same as a full model run. `state:modified` has a limitation on seed files over 1MB, while dbt State does not. When dbt State is enabled in a self-managed deployment, all [`state:*` selectors](https://docs.getdbt.com/reference/node-selection/methods.md#state) can use dbt State as their comparison source. This provides per-node granularity — each node is compared against its own last execution in the [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) environment, rather than comparing all nodes against a single `manifest.json` from the most recent job. This applies automatically when no explicit [`--state`](https://docs.getdbt.com/reference/node-selection/configure-state.md) manifest has been provided. For more details, refer to [dbt State-powered `state:*` selectors](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md#dbt-state-powered-state-selectors). --- ### How often is cost data refreshed? Cost data refreshes daily and reflects the previous day's usage. This means there is a lag of up to one day between when a job runs and when its cost data appears in Cost Insights. --- ### How often should I run the snapshot command? Snapshots are a batch-based approach to [change data capture](https://en.wikipedia.org/wiki/Change_data_capture). The `dbt snapshot` command must be run on a schedule to ensure that changes to tables are actually recorded! While individual use-cases may vary, snapshots are intended to be run between hourly and daily. If you find yourself snapshotting more frequently than that, consider if there isn't a more appropriate way to capture changes in your source data tables. For best practices on when to use snapshots vs incremental models, refer to [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md). --- ### How should I structure my project? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). --- ### How to delete a job or environment in dbt? To delete an environment or job in dbt, you must have a `developer` [license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and have the necessary [access permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md). ##### Delete a job To delete a job or multiple jobs in dbt: 1. Click **Deploy** on the navigation header. 2. Click **Jobs** and select the job you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete job** to delete the job. <br /> [![Delete a job](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-job.png?v=2 "Delete a job")](#)Delete a job 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. 6. Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. ##### Delete an environment Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. Follow these steps to delete an environment in dbt: 1. Navigate to **Orchestration** > **Environments**. 2. Select the environment you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete** to delete the environment. [![Delete an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-environment.png?v=2 "Delete an environment")](#)Delete an environment 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. 6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. --- ### How to generate HAR files HTTP Archive (HAR) files are used to gather data from users’ browser, which dbt Support uses to troubleshoot network or resource issues. This information includes detailed timing information about the requests made between the browser and the server. The following sections describe how to generate HAR files using common browsers such as [Google Chrome](#google-chrome), [Mozilla Firefox](#mozilla-firefox), [Apple Safari](#apple-safari), and [Microsoft Edge](#microsoft-edge). info Remove or hide any confidential or personally identifying information before you send the HAR file to dbt Labs. You can edit the file using a text editor. ##### Google Chrome 1. Open Google Chrome. 2. Click on **View** --> **Developer Tools**. 3. Select the **Network** tab. 4. Ensure that Google Chrome is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Select **Preserve Log**. 6. Clear any existing logs by clicking **Clear network log** (🚫). 7. Go to the page where the issue occurred and reproduce the issue. 8. Click **Export HAR** (the down arrow icon) to export the file as HAR. The icon is located on the same row as the **Clear network log** button. 9. Save the HAR file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Mozilla Firefox 1. Open Firefox. 2. Click the application menu and then **More tools** --> **Web Developer Tools**. 3. In the developer tools docked tab, select **Network**. 4. Go to the page where the issue occurred and reproduce the issue. The page automatically starts recording as you navigate. 5. When you're finished, click **Pause/Resume recording network log**. 6. Right-click anywhere in the **File** column and select **Save All as HAR**. 7. Save the HAR file. 8. Upload the HAR file to the dbt Support ticket thread. ##### Apple Safari 1. Open Safari. 2. In case the **Develop** menu doesn't appear in the menu bar, go to **Safari** and then **Settings**. 3. Click **Advanced**. 4. Select the **Show features for web developers** checkbox. 5. From the **Develop** menu, select **Show Web Inspector**. 6. Click the **Network tab**. 7. Go to the page where the issue occurred and reproduce the issue. 8. When you're finished, click **Export**. 9. Save the file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Microsoft Edge 1. Open Microsoft Edge. 2. Click the **Settings and more** menu (...) to the right of the toolbar and then select **More tools** --> **Developer tools**. 3. Click **Network**. 4. Ensure that Microsoft Edge is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Go to the page where the issue occurred and reproduce the issue. 6. When you're finished, click **Stop recording network log**. 7. Click **Export HAR** (the down arrow icon) or press **Ctrl + S** to export the file as HAR. 8. Save the HAR file. 9. Upload the HAR file to the dbt Support ticket thread. ##### Additional resources Check out the [How to generate a HAR file in Chrome](https://www.loom.com/share/cabdb7be338243f188eb619b4d1d79ca) video for a visual guide on how to generate HAR files in Chrome. --- ### How to migrate git providers To migrate from one git provider to another, refer to the following steps to avoid minimal disruption: 1. Outside of dbt, you'll need to import your existing repository into your new provider. By default, connecting your repository in one account won't automatically disconnected it from another account. As an example, if you're migrating from GitHub to Azure DevOps, you'll need to import your existing repository (GitHub) into your new Git provider (Azure DevOps). For detailed steps on how to do this, refer to your Git provider's documentation (Such as [GitHub](https://docs.github.com/en/migrations/importing-source-code/using-github-importer/importing-a-repository-with-github-importer), [GitLab](https://docs.gitlab.com/ee/user/project/import/repo_by_url.html), [Azure DevOps](https://learn.microsoft.com/en-us/azure/devops/repos/git/import-git-repository?view=azure-devops)) 2. Go back to dbt and set up your [integration for the new Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md), if needed. 3. Disconnect the old repository in dbt by going to **Account Settings** and then **Projects**. 4. Click on the **Repository** link, then click **Edit** and **Disconnect**. [![Disconnect and reconnect your Git repository in your dbt Account settings page.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your Git repository in your dbt Account settings page.")](#)Disconnect and reconnect your Git repository in your dbt Account settings page. 5. Click **Confirm Disconnect**. 6. On the same page, connect to the new Git provider repository by clicking **Configure Repository** * If you're using the native integration, you may need to OAuth to it. 7. That's it, you should now be connected to the new Git provider! 🎉 Note — As a tip, we recommend you refresh your page and Studio IDE before performing any actions. --- ### How to upgrade a dbt account dbt offers [several plans](https://www.getdbt.com/pricing/) with different features that meet your needs. This document is for dbt admins and explains how to select a plan in order to continue using dbt. #### Prerequisites Before you begin: * You *must* be part of the [Owner](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md) user group to make billing changes. Users not included in this group will not see these options. * All amounts shown in dbt are in U.S. Dollars (USD) * When your trial expires, your account's default plan enrollment will be a Starter plan. #### Select a plan When your [14 day trial](https://www.getdbt.com/signup/) ends or if your subscription payment is past due , you'll need to select a plan in order to continue using your account: * Upon logging in, you should see an "Account locked" pop up message with instructions to unlock your account and update your payment details * Click **Go to Billing** to go to the billing page * Under **Billing**, you can review the available dbt [plans](https://www.getdbt.com/pricing/) and their features To unlock your account and select a plan, review the following guidance per plan type: ##### Developer plan 1. To select a Developer plan, click Select plan on the right. 2. Confirm your plan selection on the pop up message. 3. This automatically unlocks your dbt account, and you can now enjoy the benefits of the Developer plan. 🎉 [![](/img/docs/dbt-platform/downgrade-dev-flow.gif?v=2)](#) ###### Plan allocation If you select a plan but have too many seats or projects for that plan (for example, if you select the Developer plan but have more than one developer seat), you'll be directed to the users & projects pages to make edits. ##### Starter plan 1. When your trial expires, your account's default plan enrollment will be a Starter plan. 2. To unlock your account and continue using the Starter plan, click on **Select plan** under the Starter column. 3. Enter your payment information and seat purchases. Then click **Save**. 4. This automatically unlocks your dbt account, and you can now enjoy the benefits of the Starter plan. 🎉 [![](/img/docs/dbt-platform/trial-team-flow.png?v=2)](#) [![](/img/docs/dbt-platform/trial-team-payments-flow.png?v=2)](#) ##### Enterprise plan 1. If you're interested in one of our Enterprise-tier plans, select the Enterprise tab under **Billing**. 2. Click **Contact Sales** on the right. This opens a chat window for you to contact the dbt Support team, who will connect you to our Sales team. 3. Once you submit your request, our Sales team will contact you with more information. [![](/img/docs/dbt-platform/enterprise-upgrade.gif?v=2)](#) 4. Alternatively, you can [contact](https://www.getdbt.com/contact/) our Sales team directly to chat about how dbt can help you and your team. #### Related questions For commonly asked billings questions, refer to the dbt [pricing page](https://www.getdbt.com/pricing/). How does billing work? Starter plans are billed monthly on the credit card used to sign up, based on [developer seat count and usage](https://docs.getdbt.com/docs/platform/billing.md). You’ll also be sent a monthly receipt to the billing email of your choice. You can change any billing information in your **Account Settings** > **Billing page**. Enterprise-tier plan customers are billed annually based on the number of developer seats, as well as any additional services + features in your chosen plan. Can I upgrade or downgrade my plan? Yes, you can upgrade or downgrade at any time. Account Owners can access their dedicated billing section via the account settings page. If you’re not sure which plan is right for you, get in touch and we’ll be happy to help you find one that fits your needs. What happens when I upgrade from Developer to Starter? The free two-week trial is only available to new accounts at signup — but if you're already on the Developer plan, you can upgrade to Starter and immediately access all Starter features without needing to go through the trial. When you upgrade from Developer to Starter, you move directly to paid billing — you'll enter your payment information and select your developer license seats to complete the upgrade. Your existing projects and settings carry over. Can I pay by invoice? Currently, dbt Starter plan payments must be made with a credit card, and by default they will be billed monthly based on the number of [developer seats and usage](https://docs.getdbt.com/docs/platform/billing.md). We don’t have any plans to do invoicing for Starter plan accounts in the near future, but we do currently support invoices for companies on the dbt Enterprise-tier plan. Feel free to [contact](https://www.getdbt.com/contact/) us to build your Enterprise pricing plan. Why did I receive a **Failed payment** error email? This means we were unable to charge the credit card you have on file, or you have not provided an updated card for payment. If you're a current account owner with a card on file, contact your credit card issuer to inquire as to why your card was declined or update the credit card on your account. Your Account Owner can update payment details in the **Account Settings** -> **Billing** page. Click **Edit** next to your card details, double check your information is up-to-date, and we'll give it another go at the next billing run. --- ### I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. --- ### I need to use quotes to select from my source, what should I do? This is reasonably common on Snowflake in particular. By default, dbt will not quote the database, schema, or identifier for the source tables that you've specified. To force dbt to quote one of these values, use the [`quoting` property](https://docs.getdbt.com/reference/resource-properties/quoting.md): models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop quoting: database: true schema: true identifier: true tables: - name: order_items - name: orders # This overrides the `jaffle_shop` quoting config quoting: identifier: false ``` --- ### I'm getting a "Partial parsing enabled: 1 files deleted, 0 files added, 2 files changed" compilation error in dbt? If you're receiving this error, try deleting the `target/partial_parse.msgpack` file from your project and refresh your IDE. If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm getting a "Session occupied" error in dbt platform CLI? If you're receiving a `Session occupied` error in the dbt platform CLI or if you're experiencing a long-running session, you can use the `dbt invocation list` command in a separate terminal window to view the status of your active session. This helps debug the issue and identify the arguments that are causing the long-running session. To cancel an active session, use the `Ctrl + Z` shortcut. To learn more about the `dbt invocation` command, see the [dbt invocation command reference](https://docs.getdbt.com/reference/commands/invocation.md?version=2.0). Alternatively, you can reattach to your existing session with `dbt reattach` and then press `Control-C` and choose to cancel the invocation. --- ### I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job If you're receiving a `This run exceeded your account's run memory limits` error in your failed job, it means that the job exceeded the [memory limits](https://docs.getdbt.com/docs/deploy/job-scheduler.md#job-memory) set for your account. All dbt accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices. ##### Common reasons Some common reasons for higher memory usage are: * dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient. * dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries. ##### Resolution There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. Video example As an additional resource, check out [this example video](https://www.youtube.com/watch?v=sTqzNaFXiZ8), which demonstrates how to refactor the sample code by reducing the number of rows returned. If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! ##### Additional resources * [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) --- ### I'm receiving a "Permission denied while getting Drive credential" error when trying to query from Google Drive? If you're seeing the below error when you try to query a dataset from a Google Drive document in the Studio IDE, the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```text Access denied: BigQuery BigQuery: Permission denied while getting Drive credentials ``` Usually, this error indicates that you haven't granted the BigQuery service account access to the specific Google Drive document. If you're seeing this error, try giving the service account (Client email field seen [here](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md)) you are using for your BigQuery connection in dbt, permission to your Google Drive or Google Sheet. You'll want to do this directly in your Google Document, click the **Share** button, and enter the client email. If you are experiencing this error when using OAuth, and you have verified your access to the Google Sheet, you may need to grant permissions for gcloud to access Google Drive: ```text gcloud auth application-default login --disable-quota-project ``` For more info see the [gcloud auth application-default documentation](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) If you've tried the earlier steps and are still experiencing this behavior, try using the following command to log into Google Cloud and enable access to Google Drive. It also updates the Application Default Credentials (ADC) file, which many Google Cloud libraries use to authenticate API calls. ```text gcloud auth login --enable-gdrive-access --update-adc ``` For more info, refer to [gcloud auth login documentation](https://cloud.google.com/sdk/gcloud/reference/auth/login#--enable-gdrive-access). If you've tried the steps above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving a 403 error 'Forbidden: Access denied' when using service tokens All [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) traffic is subject to IP restrictions. When using a service token, the following 403 response error indicates the IP is not on the allowlist. To resolve this, you should add your third-party integration CIDRs (network addresses) to your allowlist. The following is an example of the 403 response error: ```json { "status": { "code": 403, "is_success": False, "user_message": ("Forbidden: Access denied"), "developer_message": None, }, "data": { "account_id": <account_id>, "user_id": <user_id>, "is_service_token": <boolean describing if it's a service token request>, "account_access_denied": True, }, } ``` --- ### I'm receiving a git rev-list master error in the IDE? If you're unable to access the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```shell git rev-list master..origin/main --count fatal: ambiguous argument 'master..origin/main': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' ``` Usually this error indicates that the "main" branch name has changed or it is possible that dbt was unable to determine what your primary branch was. No worries, we have a few workarounds for you to try: **Workaround 1** Take a look at your Environment Settings - If you **do not** have a custom branch filled in your Environment Settings: 1. Disconnect and reconnect your repository [connection](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) on your Project Settings page. This should then allow dbt to pick up that the "main" branch is now called `main`. 2. In the Environment Settings, set the custom branch to 'master' and refresh the Studio IDE. **Workaround 2** Take a look at your Environment Settings - If you **do** have a custom branch filled in your Environment Settings: 1. Disconnecting and reconnecting your repository [connection](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) on your Project Settings page. This should then allow dbt to pick up that the "main" branch is now called `main`. 2. In the Environment Settings, remove the custom branch and refresh the Studio IDE. If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving a NoneType object has no attribute error in the IDE? If you're unable to access the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```shell NoneType object has no attribute enumerate_fields' ``` Usually this error indicates that you tried connecting your database via [SSH tunnel](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md#connecting-using-an-ssh-tunnel). If you're seeing this error, double-check you have supplied the following items: * the hostname * username * port of bastion server If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving a Runtime Error Could not find profile named 'user' error? If you're unable to access the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```shell Running with dbt=1.9.0 Encountered an error while reading the project: ERROR: Runtime Error Could not find profile named 'user' Runtime Error Could not run dbt' ``` Usually this errors indicates that there is an issue with missing/stale credentials/authentication. No worries, we have a few workarounds for you to try: **In the Studio IDE:** Click your account name in the bottom left, select **Your profile**, then go to **Credentials**. Re-enter or re-authorize your credentials to resolve the error. **In a job:** This can happen if you changed the deployment environment and didn't re-enter your deployment credentials when saving. Go back to the deployment environment settings, re-enter your credentials (private key/passphrase or username and password), and kick off a new job run. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving an 'Your IDE session experienced an unknown error and was terminated. Please contact support'. If you're seeing the following error when you launch the Studio IDE, it could be due to a few scenarios but, commonly, it indicates a missing repository: ```shell Your Studio IDE session experienced an unknown error and was terminated. Please contact support. ``` You can try to resolve this by adding a repository like a [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or your preferred Git account. To add your Git account, navigate to **Project** > **Repository** and select your repository. If you're still running into this error, please contact the Support team at <support@getdbt.com> for help. --- ### I'm receiving an `Failed ALPN` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### I'm seeing a 'GitHub and dbt latest permissions' error If you see the error `This account needs to accept the latest permissions for the dbt GitHub App` in dbt — this usually occurs when the permissions for the dbt GitHub App are out-of-date. To solve this issue, you'll need to update the permissions for the dbt GitHub App in your GitHub account. This FAQ shares a couple of ways you can do it. #### Update permissions A GitHub organization admin will need to update the permissions in GitHub for the dbt GitHub App. If you're not the admin, reach out to your organization admin to request this. 1. Navigate to your GitHub account. Click on the top right profile icon and then **Settings** (or personal if using a non-organization account). 2. Then go to **Integrations** and then select **Applications** to identify any necessary permission changes. Note that a GitHub repository admin may not see the same permission request. [![Navigate to Application settings to identify permission changes.](/img/docs/dbt-platform/platform-configuring-dbt-platform/github-applications.png?v=2 "Navigate to Application settings to identify permission changes.")](#)Navigate to Application settings to identify permission changes. 3. Click on **Review request** and then click on the **Accept new permissions** button on the next page. [![Grant access to the dbt app by accepting the new permissions.](/img/docs/dbt-platform/platform-configuring-dbt-platform/github-review-request.png?v=2 "Grant access to the dbt app by accepting the new permissions.")](#)Grant access to the dbt app by accepting the new permissions. For more info on GitHub permissions, refer to [access permissions](https://docs.github.com/en/get-started/learning-about-github/access-permissions-on-github). Alternatively, try [disconnecting your GitHub account](#disconnect-github) in dbt, detailed in the following section. #### Disconnect GitHub Disconnect the GitHub and dbt integration in dbt. 1. In dbt, go to **Account Settings**. 2. In **Projects**, select the project experiencing the issue. 3. Click the repository link under **Repository**. 4. In the **Repository details** page, click **Edit**. 5. Click **Disconnect** to remove the GitHub integration. [![Disconnect and reconnect your git repository in your dbt Account settings pages.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your git repository in your dbt Account settings pages.")](#)Disconnect and reconnect your git repository in your dbt Account settings pages. 6. Click **Confirm Disconnect**. 7. Return to your **Project details** page and reconnect your repository by clicking the **Configure Repository** link. 8. Click **GitHub** and select your repository. #### Support If you've tried these workarounds and are still experiencing this behavior — reach out to the [dbt Support](mailto:support@getdbt.com) team and we'll be happy to help! --- ### I'm seeing a Gitlab authentication out of date error loop If you're seeing a 'GitLab Authentication is out of date' 500 server error page - this usually occurs when the deploy key in the repository settings in both dbt and GitLab do not match. No worries - this is a current issue the dbt Labs team is working on and we have a few workarounds for you to try: ###### First workaround 1. Disconnect repo from project in dbt. 2. Go to Gitlab and click on Settings > Repository. 3. Under Repository Settings, remove/revoke active dbt deploy tokens and deploy keys. 4. Attempt to reconnect your repository via dbt. 5. You would then need to check Gitlab to make sure that the new deploy key is added. 6. Once confirmed that it's added, refresh dbt and try developing once again. ###### Second workaround 1. Keep repo in project as is -- don't disconnect. 2. Copy the deploy key generated in dbt. 3. Go to Gitlab and click on Settings > Repository. 4. Under Repository Settings, manually add to your Gitlab project deploy key repo (with `Grant write permissions` box checked). 5. Go back to dbt, refresh your page and try developing again. If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm seeing a Gitlab authentication out of date error loop If you're seeing a 'GitLab Authentication is out of date' 500 server error page - this usually occurs when the deploy key in the repository settings in both dbt and GitLab do not match. No worries - this is a current issue the dbt Labs team is working on and we have a few workarounds for you to try: ##### 1st Workaround 1. Disconnect repo from project in dbt. 2. Go to Gitlab and click on Settings > Repository. 3. Under Repository Settings, remove/revoke active dbt deploy tokens and deploy keys. 4. Attempt to reconnect your repository via dbt. 5. You would then need to check Gitlab to make sure that the new deploy key is added. 6. Once confirmed that it's added, refresh dbt and try developing once again. ##### 2nd Workaround 1. Keep repo in project as is -- don't disconnect. 2. Copy the deploy key generated in dbt. 3. Go to Gitlab and click on Settings > Repository. 4. Under Repository Settings, manually add to your Gitlab project deploy key repo (with `Grant write permissions` box checked). 5. Go back to dbt, refresh your page and try developing again. If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### If I can name these files whatever I'd like, what should I name them? It's up to you! Here's a few options: * Default to the existing terminology: `schema.yml` (though this does make it hard to find the right file over time) * Use the same name as your directory (assuming you're using sensible names for your directories) * If you test and document one model (or seed, snapshot, macro etc.) per file, you can give it the same name as the model (or seed, snapshot, macro etc.) Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). --- ### If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. --- ### If models can only be `select` statements, how do I insert records? For those coming from an ETL (Extract Transform Load) paradigm, there's often a desire to write transformations as `insert` and `update` statements. In comparison, dbt will wrap your `select` query in a `create table as` statement, which can feel counter-productive. * If you wish to use `insert` statements for performance reasons (i.e. to reduce data that is processed), consider [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * If you wish to use `insert` statements since your source data is constantly changing (e.g. to create "Type 2 Slowly Changing Dimensions"), consider [snapshotting your source data](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness), and building models on top of your snapshots. --- ### Multi-cell or static subdomain account connection issues **Symptoms:** Connection errors when your account URL includes a prefix (for example, `abc123.us1.dbt.com`). **Solution (as of v1.14.0):** Set `DBT_HOST` to the full hostname including the prefix. If you're using PAT-based auth, also set `DBT_ACCOUNT_ID`. ```bash # ✅ Correct DBT_HOST=abc123.us1.dbt.com DBT_ACCOUNT_ID=12345 # required for PAT-based auth ``` You no longer need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. If you have these set from an older configuration, remove them. --- ### My compiled SQL has a lot of spaces and new lines, how can I get rid of it? This is known as "whitespace control". Use a minus sign (`-`, e.g. `{{- ... -}}`, `{%- ... %}`, `{#- ... -#}`) at the start or end of a block to strip whitespace before or after the block (more docs [here](https://jinja.palletsprojects.com/page/templates/#whitespace-control)). Check out the [tutorial on using Jinja](https://docs.getdbt.com/guides/using-jinja.md#use-whitespace-control-to-tidy-up-compiled-code) for an example. Take caution: it's easy to fall down a rabbit hole when it comes to whitespace control! --- ### OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). --- ### One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. --- ### Pasting full URLs instead of IDs **Symptoms:** Authentication errors, unexpected behavior, or the server failing to connect to the right environment. **Cause:** Environment variables like `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_ACCOUNT_ID` expect numeric integers, not full browser URLs. **Solution:** ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. --- ### Permission errors when creating the integration or MCP server `CREATE API INTEGRATION` and `CREATE EXTERNAL MCP SERVER` require `ACCOUNTADMIN` (or a role with account-level `CREATE INTEGRATION`). Run steps 1–2 as `ACCOUNTADMIN`, then grant `USAGE` on the MCP server and integration to the role that runs the agent (Step 3). --- ### Receiving a 'Could not parse dbt_project.yml' error in dbt job The error message `Could not parse dbt_project.yml: while scanning for...` in your dbt job run or development usually occurs for several reasons: * There's a parsing failure in a YAML file (such as a tab indentation or Unicode characters). * Your `dbt_project.yml` file has missing fields or incorrect formatting. * Your `dbt_project.yml` file doesn't exist in your dbt project repository. To resolve this issue, consider the following: * Use an online YAML parser or validator to check for any parsing errors in your YAML file. Some known parsing errors include missing fields, incorrect formatting, or tab indentation. * Or ensure your `dbt_project.yml` file exists. Once you've identified the issue, you can fix the error and rerun your dbt job. --- ### Receiving a `Failed to connect to DB` error when connecting to Snowflake 1. If you see the following error: ```text Failed to connect to DB: xxxxxxx.snowflakecomputing.com:443. The role requested in the connection, or the default role if none was requested in the connection ('xxxxx'), is not listed in the Access Token or was filtered. Please specify another role, or contact your OAuth Authorization server administrator. ``` 2. Edit your OAuth Security integration and explicitly specify this scope mapping attribute: ```sql ALTER INTEGRATION <my_int_name> SET EXTERNAL_OAUTH_SCOPE_MAPPING_ATTRIBUTE = 'scp'; ``` You can read more about this error in [Snowflake's documentation](https://community.snowflake.com/s/article/external-custom-oauth-error-the-role-requested-in-the-connection-is-not-listed-in-the-access-token). *** 1. If you see the following error: ```text Failed to connect to DB: xxxxxxx.snowflakecomputing.com:443. Incorrect username or password was specified. ``` * **Unique email addresses** — Each user in Snowflake must have a unique email address. You can't have multiple users (for example, a human user and a service account) using the same email, such as `alice@acme.com`, to authenticate to Snowflake. * **Match email addresses with identity provider** — The email address of your Snowflake user must exactly match the email address you use to authenticate with your Identity Provider (IdP). For example, if your Snowflake user's email is `alice@acme.com` but you log in to Entra or Okta with `alice_adm@acme.com`, this mismatch can cause an error. --- ### Reconnecting to Snowflake OAuth after authentication expires When you connect Snowflake to dbt platform using [OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md), dbt stores a refresh token. This allows your user credentials to be reused in tools like the Studio IDE and the dbt Semantic Layer without needing to re-authenticate each time. If you see an `authentication has expired` error when you try to run queries, you must renew your connection between Snowflake and the dbt platform. To resolve the issue, complete the following steps: 1. Go to **Your profile**, accessible from the navigation menu. 2. Navigate to **Credentials** and then choose the project where you're experiencing the issue. 3. Under **User credentials**, click the **Reconnect Snowflake Account** button. This will guide you through re-authenticating using your SSO workflow. Your Snowflake administrator can [configure the refresh token validity period](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md#create-a-security-integration), up to the maximum 90 days. If you've tried these step and are still getting this error, please contact the Support team at <support@getdbt.com> for further assistance. --- ### Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). --- ### Should I use separate files to declare resource properties, or one large file? It's up to you: * Some folks find it useful to have one file per model (or source / snapshot / seed etc) * Some find it useful to have one per directory, documenting and testing multiple models in one file Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). --- ### Studio IDE freezes when opening the Lineage tab If the Studio IDE freezes with two or more models open and the **Lineage** tab active, use the following workarounds. ##### Why this happens The Studio IDE can freeze when the **Lineage** tab is open or loads automatically, and when two or more models are open, especially if a model has a large or slow-loading DAG. #### What you might observe * The editor becomes unresponsive. * After a few minutes, your browser may display a **Wait** or **Exit** dialog. #### Workaround 1: Avoid the Lineage tab on startup 1. Refresh the page. 2. Keep the console focused on another tab (for example, **Results** or **Compiled code**) instead of **Lineage**. 3. Close extra model tabs before opening **Lineage**. 4. Open **Lineage** only when needed. #### Workaround 2: Clear Local Storage for your account URL This workaround is browser-dependent. In DevTools, select the Local Storage origin that matches the URL in your browser address bar. For example, your account access URL might look like `https://ab123.us1.dbt.com`. ##### Google Chrome and Microsoft Edge 1. Open the dbt platform page that’s freezing. 2. Open DevTools (right-click the page and select **Inspect**). 3. Click the **Application** tab. 4. In the left sidebar, expand **Storage** → **Local Storage**. 5. Right-click the origin that matches your current URL (for example, `https://ab123.us1.dbt.com`) and select **Clear**. 6. Refresh the page and sign in again if prompted. ##### Mozilla Firefox 1. Open the dbt platform page that’s freezing. 2. Open DevTools (right-click the page and select **Inspect**). 3. Open the **Storage** tab. * If you don’t see it, open DevTools **Settings** and enable **Storage**. 4. In the left sidebar, expand **Local Storage**. 5. Right-click the origin that matches your current URL (for example, `https://ab123.us1.dbt.com`) and select **Delete All**. 6. Refresh the page and sign in again if prompted. ##### Safari 1. Open Safari and go to the dbt platform page that’s freezing. 2. If needed, enable developer features: * Go to **Safari** > **Settings**. * Click **Advanced**. * Enable developer features. 3. Open Web Inspector from **Develop** > **Show Web Inspector**. 4. Open the **Storage** tab. 5. In the left sidebar, expand **Local Storage** and select the origin that matches your current URL (for example, `https://ab123.us1.dbt.com`). 6. Delete the Local Storage entries for that origin. 7. Refresh the page and sign in again if prompted. #### Need more help? If freezing continues after trying these workarounds, [contact dbt Support](mailto:support@getdbt.com) and include: * A HAR file (refer to [How to generate HAR files](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md)) * Your account URL * Browser and version * Approximate number of open model tabs * Whether the issue happens only when the **Lineage** tab is active --- ### The agent returns no metrics or empty results * Confirm the project you authorized has a configured [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) with metrics and dimensions. * Check that the user who connected has at least read-only access to that project — the agent only sees what the user can see. * If you scoped the OAuth connection to a single project, make sure it's the project that contains your metrics. --- ### The columns of my seed changed, and now I get an error when running the `seed` command, what should I do? If you changed the columns of your seed, you may get a `Database Error`: ##### Snowflake ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 1 seed file, 0 sources 12:12:27 | Concurrency: 8 threads (target='dev_snowflake') 12:12:27 | 12:12:27 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:12:30 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 2.78s] 12:12:31 | 12:12:31 | Finished running 1 seed in 10.05s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) 000904 (42000): SQL compilation error: error line 1 at position 62 invalid identifier 'COUNTRY_NAME' Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` ##### Redshift ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 149 macros, 0 operations, 1 seed file, 0 sources 12:14:46 | Concurrency: 1 threads (target='dev_redshift') 12:14:46 | 12:14:46 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:14:46 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 0.23s] 12:14:46 | 12:14:46 | Finished running 1 seed in 1.75s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) column "country_name" of relation "country_codes" does not exist Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` In this case, you should rerun the command with a `--full-refresh` flag, like so: ```text dbt seed --full-refresh ``` **Why is this the case?** When you typically run dbt seed, dbt truncates the existing table and reinserts the data. This pattern avoids a `drop cascade` command, which may cause downstream objects (that your BI users might be querying!) to get dropped. However, when column names are changed, or new columns are added, these statements will fail as the table structure has changed. The `--full-refresh` flag will force dbt to `drop cascade` the existing table before rebuilding it. --- ### The connector won't authorize or the OAuth flow fails * Confirm your account has a [static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). OAuth with MCP requires one. * Verify the host in `API_ALLOWED_PREFIXES`, `OAUTH_TOKEN_ENDPOINT`, `OAUTH_AUTHORIZATION_ENDPOINT`, and `OAUTH_RESOURCE_URL` all match your MCP URL host exactly, and that the integration `ENABLED = TRUE`. * AI features are enabled by default and admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * Make sure remote MCP OAuth is available for your account plan. --- ### Toolset unavailable or showing as disabled **Symptoms:** A toolset (Semantic Layer, Discovery, Admin API) is not available in your AI client even though you've configured credentials. **Cause:** Either the required variables are missing, or the toolset has been explicitly disabled. **Solution:** 1. Check that all required variables for the toolset are set — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). 2. Check whether you have any `DISABLE_*` variables set to `true` that might be turning off the toolset. 3. If you're using enable mode (`DBT_MCP_ENABLE_*`), make sure the toolset you need is listed. 4. Set `DBT_MCP_LOG_LEVEL=DEBUG` to see which toolsets are active at startup. --- ### Unable to trigger a CI job with GitLab When you connect dbt to a GitLab repository, GitLab automatically registers a webhook in the background, viewable under the repository settings. This webhook is also used to trigger [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) when you push to the repository. If you're unable to trigger a CI job, this usually indicates that the webhook registration is missing or incorrect. To resolve this issue, navigate to the repository settings in GitLab and view the webhook registrations by navigating to GitLab --> **Settings** --> **Webhooks**. Some things to check: * The webhook registration is enabled in GitLab. * The webhook registration is configured with the correct URL and secret. If you're still experiencing this issue, reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### What are the best practices for installing dbt v1 with pip? info dbt v2 is a next-generation, Rust-based engine that powers dbt development across the platform and local tooling. See [dbt v2](https://docs.getdbt.com/docs/introduction.md) for more information. #### Best practices Managing Python local environments can be challenging! You can use these best practices to improve the dbt v1 installation with `pip`. | Best practice | Recommendation | Why it matters | | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | [Install dbt v1 with an adapter](https://docs.getdbt.com/docs/local/install-dbt.md?version=1#installing-the-adapter) and keep versions in sync | Install with: `python -m pip install dbt-ADAPTER_NAME`<br /><br />(For example, `python -m pip install dbt-snowflake`)<br /><br />Match adapter versions to your dbt v1 version<br /><br /> | Provides a complete, compatible, and ready-to-run dbt setup<br /><br /><br /><br />Prevents runtime errors and adapter incompatibilities | | For tooling without a warehouse connection, install v1 without an adapter | `python -m pip install dbt-core` | Keeps your setup lean, predictable, and easier to maintain | | Use [virtual environments](https://docs.getdbt.com/faqs/Core/install-pip-best-practices.md#using-virtual-environments) | Install dbt in an isolated environment (for example, `venv`, `pipenv`, `poetry`) | Avoids dependency conflicts | | Reactivate your virtual environment for each session | Reactivate your virtual environment at the start of each new session before installing dependencies or running dbt commands | Keeps your dbt setup predictable, isolated, and reproducible | | [Create a project](https://docs.getdbt.com/reference/commands/init.md) | Use the `dbt init` command to create and initialize your first project | Creates a standard dbt project and verifies your installation | | Ensure you have the latest versions of `pip`, `wheel`, and `setuptools` | Before installing dbt, upgrade your Python packaging tools:<br /><br />`python -m pip install --upgrade pip wheel setuptools` | Helps ensure a smoother, more predictable dbt installation | <br /> Note, dbt adapters and dbt v1 are versioned independently to make it easier for us to maintain and independently evolve adapters going forward. ##### Using virtual environments We recommend using [virtual environments](https://docs.python-guide.org/dev/virtualenvs/) to namespace `pip` modules. Here's an example setup: ```shell python3 -m venv dbt-env # create the environment source dbt-env/bin/activate # activate the environment for Mac and Linux dbt-env\Scripts\activate # activate the environment for Windows ``` If you install `dbt` in a virtual environment, you need to reactivate that same virtual environment each time you create a shell window or session. *Tip:* You can create an alias for the `source` command in your `$HOME/.bashrc`, `$HOME/.zshrc`, or whichever rc file your shell draws from. For example, you can add a command like `alias env_dbt='source <PATH_TO_VIRTUAL_ENV_CONFIG>/bin/activate'`, replacing `<PATH_TO_VIRTUAL_ENV_CONFIG>` with the path to your virtual environment configuration. ##### Using the latest versions dbt installations are tested using the latest versions of `pip` and `setuptools`. Newer versions have improved behavior around dependency resolution, as well as much faster install times by using precompiled "wheels" when available for your operating system. Before installing dbt, make sure you have the latest versions: ```shell python -m pip install --upgrade pip wheel setuptools ``` --- ### What data tests are available for me to use in dbt? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. --- ### What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. --- ### What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)** as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). --- ### What happens if dbt State servers fail? If dbt State servers are unavailable, dbt gracefully falls back to normal dbt behavior. --- ### What happens if I add new columns to my snapshot query? When the columns of your source query changes, dbt will attempt to reconcile this change in the destination snapshot table. dbt does this by: 1. Creating new columns from the source query in the destination table 2. Expanding the size of string types where necessary (eg. `varchar`s on Redshift) dbt *will not* delete columns in the destination snapshot table if they are removed from the source query. It will also not change the type of a column beyond expanding the size of varchar columns. That is, if a `string` column is changed to a `date` column in the snapshot source query, dbt will not attempt to change the type of the column in the destination table. --- ### What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. --- ### What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. --- ### What if I work on multiple projects that each use their own dbt State? You can specify your org ID in `dbt_project.yml`: ```yaml dbt-cloud: state-org-id: <your-org-id> ``` --- ### What if my prod environment isn't named prod? You can specify the defer-to environment using the [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) config in `profiles.yml`: ```yaml my_project: outputs: prod: type: snowflake defer_to_target: production ``` `defer_to_target` only applies to self-managed deployments. If you're using the dbt platform, deferral is configured through your environment settings in the UI. For more details, refer to [Configuring deferral](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md). --- ### What if my source is in a different database to my target database? Use the [`database` property](https://docs.getdbt.com/reference/resource-properties/database.md) to define the database that the source is in. models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: jaffle_shop tables: - name: orders - name: customers ``` --- ### What if my source is in a poorly named schema or table? By default, dbt will use the `name:` parameters to construct the source reference. If these names are a little less-than-perfect, use the [schema](https://docs.getdbt.com/reference/resource-properties/schema.md) and [identifier](https://docs.getdbt.com/reference/resource-properties/identifier.md) properties to define the names as per the database, and use your `name:` property for the name that makes sense! models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: postgres_backend_public_schema tables: - name: orders identifier: api_orders ``` In a downstream model: ```sql select * from {{ source('jaffle_shop', 'orders') }} ``` Will get compiled to: ```sql select * from raw.postgres_backend_public_schema.api_orders ``` --- ### What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. --- ### What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. --- ### What parts of Jinja are dbt-specific? There are certain expressions that are specific to dbt — these are documented in the [Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) section of these docs. Further, docs blocks, snapshots, and materializations are custom Jinja *blocks* that exist only in dbt. --- ### What privileges does my database user need to use dbt? Your user will need to be able to: * `select` from raw data in your warehouse (i.e. data to be transformed) * `create` schemas, and therefore create tables/views within that schema¹ * read system views to generate documentation (i.e. views in `information_schema`) On Postgres, Redshift, Databricks, and Snowflake, use a series of `grants` to ensure that your user has the correct privileges. Check out [example permissions](https://docs.getdbt.com/reference/database-permissions/about-database-permissions.md) for these warehouses. On BigQuery, use the "BigQuery User" role to assign these privileges. *** ¹Alternatively, a separate user can create a schema for the dbt user, and then grant the user privileges to create within this schema. We generally recommend granting your dbt user the ability to create schemas, as it is less complicated to implement. --- ### What should I name my profile? We typically use a company name for a profile name, and then use targets to differentiate between `dev` and `prod`. Check out the docs on [environments in dbt v1](https://docs.getdbt.com/docs/local/dbt-environments.md) for more information. --- ### What should I name my target? We typically use targets to differentiate between development and production runs of dbt, naming the targets `dev` and `prod`, respectively. Check out the docs on [managing environments in dbt v1](https://docs.getdbt.com/docs/local/dbt-environments.md) for more information. --- ### What should my profiles.yml file look like for my warehouse? The structure of a profile looks different on each warehouse. Check out the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) page, and navigate to the `Profile Setup` section for your warehouse. --- ### What version of Python can I use? Use this table to match dbt v1 versions with their compatible Python versions. New [dbt minor versions](https://docs.getdbt.com/docs/dbt-versions.md#minor-versions) will add support for new Python3 minor versions when all dependencies can support it. In addition, dbt minor versions will withdraw support for old Python3 minor versions before their [end of life](https://endoflife.date/python). #### Python compatibility matrix | dbt-core version | v1.13 | v1.12 | v1.11 | v1.10 | v1.9 | v1.8 | v1.7 | v1.6 | v1.5 | v1.4 | v1.3 | v1.2 | v1.1 | v1.0 | | ---------------- | ----- | ----- | ----- | ----- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | Python 3.14 | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | Python 3.13 | ✅ | ✅ | ✅ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | Python 3.12 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | Python 3.11 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | Python 3.10 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ Python 3.13 is supported in dbt v1.10 for the Postgres adapter. Adapter plugins and their dependencies are not always compatible with the latest version of Python. Note that this shouldn't be confused with [dbt Python models](https://docs.getdbt.com/docs/build/python-models.md#specific-data-platforms). If you're using a data platform that supports Snowpark, use the `python_version` config to run a Snowpark model with [Python versions](https://docs.snowflake.com/en/developer-guide/snowpark/python/setup) 3.9, 3.10, or 3.11. --- ### When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). --- ### When should I use a UDF instead of a macro? Both user-defined functions (UDFs) and macros let you reuse logic across your dbt project, but they work in fundamentally different ways. Here's when to use each: ###### Use UDFs when:  You need logic accessible outside dbt UDFs are created in your warehouse and can be used by BI tools, data science notebooks, SQL clients, or any other tool that connects to your warehouse. Macros only work within dbt.  You want to standardize warehouse-native functions UDFs let you create reusable warehouse functions for data validation, custom formatting, or business-specific calculations that need to be consistent across all your data tools. Once created, they become part of your warehouse's function catalog.  You want dbt to manage the function lifecycle dbt manages UDFs as part of your DAG execution, ensuring they're created before models that reference them. You can version control UDF definitions alongside your models, test changes in development environments, and deploy them together through CI/CD pipelines.  Jinja compiles at creation time, not on each function call You can use Jinja (loops, conditionals, macros, `ref`, `source`, `var`) inside a UDF configuration. dbt resolves that Jinja **when the UDF is created**, and the resulting SQL body is what gets stored in your warehouse. Jinja influences the function when it’s created, whereas arguments influence it when it runs in the warehouse: * ✅ **Allowed:** Jinja that depends on project or build-time state — for example, `var(“can_do_things”)`, static `ref(‘orders’)`, or environment-specific logic. These are all evaluated once at creation time. * ❌ **Not allowed:** Jinja that depends on **function arguments** passed at runtime. The compiler can’t see those, so dynamic `ref(ref_name)` or conditional Jinja based on argument values won’t work.  You need Python logic that runs in your warehouse A Python UDF creates a Python function directly within your data warehouse, which you can invoke using SQL.<br />This makes it easier to apply complex transformations, calculations, or logic that would be difficult or verbose to express in SQL. Python UDFs support conditionals and looping within the function logic itself (using Python syntax), and execute at runtime, not at compile time like macros. Python UDFs are currently supported in Snowflake and BigQuery. ###### Use macros when:  You need to generate SQL at compile time Macros generate SQL dynamically **before** it's sent to the warehouse (at compile time). This is essential for: * Building different SQL for different warehouses * Generating repetitive SQL patterns (like creating dozens of similar columns) * Creating entire model definitions or DDL statements * Dynamically referencing models based on project structure UDFs execute **at query runtime** in the warehouse. While they can use Jinja templating in their definitions, they don't generate new SQL queries—they're pre-defined functions that get called by your SQL. Expanding UDFs Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You want to generate DDL or DML statements Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You need to adapt SQL across different warehouses Macros can use Jinja conditional logic to generate warehouse-specific SQL (see [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md)), making your dbt project portable across platforms. UDFs are warehouse-specific objects. Even though UDFs can include Jinja templating in their definitions, each warehouse has different syntax for creating functions, different supported data types, and different SQL dialects. You would need to define separate UDF files for each warehouse you support.  Your logic needs access to dbt context Both macros and UDFs can use Jinja, which means they can access dbt context variables like `{{ ref() }},` `{{ source() }}`, environment variables, and project configurations. You can even call a macro from within a UDF (and vice versa) to combine dynamic SQL generation with runtime execution. However, the difference between the two is *when* the logic runs: * Macros run at compile time, generating SQL before it’s sent to the warehouse. * UDFs run inside the warehouse at query time.  You want to avoid creating warehouse objects Macros don't create anything in your warehouse; they just generate SQL at compile time. UDFs create actual function objects in your warehouse that need to be managed. ###### Can I use both together? Yes! You can use a macro to call a UDF or call a macro from within a UDF, combining the benefits of both. So the following example shows how to use a macro to define default values for arguments alongside your logic, for your UDF ```sql {% macro cents_to_dollars(column_name, scale=2) %} {{ function('cents_to_dollars') }}({{ column_name }}, {{scale}}) {% endmacro %} ``` ###### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Jinja macros](https://docs.getdbt.com/docs/build/jinja-macros.md) --- ### Where can I find my user ID? Knowing your dbt user ID can help with interacting with support. To find your user ID in the dbt platform, read the following steps: 1. Click your account name at the bottom left-side menu and go to **Account settings** > **Users**. 2. Select your user. <br /> 3. Go to the address bar. The number after `/users` is your user ID. <br /> For example, if the URL is `https://YOUR_ACCESS_URL/settings/accounts/12345/users/67891` — the user ID is `67891`. <br /> 4. Copy that number and save it somewhere safe. <br /> --- ### Where can I find SCIM FAQs and troubleshooting? System for Cross-domain Identity Management (SCIM) lets you automatically provision, update, and deprovision dbt platform users and groups directly from your identity provider (IdP). Instead of managing users manually in dbt platform, your IdP becomes the source of truth. SCIM is available on [Enterprise plans](https://www.getdbt.com/pricing). For common questions and troubleshooting guidance, refer to the [SCIM FAQ and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md) page. --- ### Where can I find SSO FAQs and troubleshooting? Single sign-on (SSO) lets your team log into dbt platform using your existing identity provider (IdP) — such as Okta, Microsoft Entra ID, or Google Workspace \&mdsah; without needing a separate dbt platform password. SSO is available on [Enterprise tiered plans](https://www.getdbt.com/pricing). For common questions and troubleshooting guidance, refer to the [SSO FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/sso-faq.md) page. ddd --- ### Where does the metadata about last updated timestamp come from? Last updated timestamps come directly from the data warehouse, for example from `INFORMATION_SCHEMA` tables. --- ### Which docs should I use when writing Jinja or creating a macro? If you are stuck with a Jinja issue, it can get confusing where to check for more information. We recommend you check (in order): 1. [Jinja's Template Designer Docs](https://jinja.palletsprojects.com/page/templates/): This is the best reference for most of the Jinja you'll use 2. [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md): This maps which file types support Jinja expressions, built-in functions, and custom macros. 3. [Our Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md): This documents any additional functionality we've added to Jinja in dbt. 4. [Agate's table docs](https://agate.readthedocs.io/page/api/table.html): If you're operating on the result of a query, dbt will pass it back to you as an agate table. This means that the methods you call on the table belong to the Agate library rather than Jinja or dbt. --- ### Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. --- ### Which SQL dialect should I write my models in? Or which SQL dialect does dbt use? dbt can feel like magic, but it isn't actually magic. Under the hood, it's running SQL in your own warehouse — your data is not processed outside of your warehouse. As such, your models should just use the **SQL dialect of your own database**. Then, when dbt wraps your `select` statements in the appropriate DDL or DML, it will use the correct DML for your warehouse — all of this logic is written in to dbt. You can find more information about the databases, platforms, and query engines that dbt supports in the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) docs. Want to go a little deeper on how this works? Consider a snippet of SQL that works on each warehouse: models/test\_model.sql ```sql select 1 as my_column ``` To replace an existing table, here's an *illustrative* example of the SQL dbt will run on different warehouses (the actual SQL can get much more complicated than this!) ##### Redshift ```sql -- you can't create or replace on redshift, so use a transaction to do this in an atomic way begin; create table "dbt_alice"."test_model__dbt_tmp" as ( select 1 as my_column ); alter table "dbt_alice"."test_model" rename to "test_model__dbt_backup"; alter table "dbt_alice"."test_model__dbt_tmp" rename to "test_model" commit; begin; drop table if exists "dbt_alice"."test_model__dbt_backup" cascade; commit; ``` ##### BigQuery ```sql -- Make an API call to create a dataset (no DDL interface for this)!!; create or replace table `dbt-dev-87681`.`dbt_alice`.`test_model` as ( select 1 as my_column ); ``` ##### Snowflake ```sql create schema if not exists analytics.dbt_alice; create or replace table analytics.dbt_alice.test_model as ( select 1 as my_column ); ``` --- ### Why am I getting an "account in use" error? If you're receiving an 'Account in use' error when trying to integrate GitHub in your Profile page, this is because the Git integration is a 1-to-1 integration, so you can only have your Git account linked to one dbt user account. Here are some steps to take to get you unstuck: * Log in to the dbt account integrated with your Git account. Go to your user profile and click on Integrations to remove the link. If you don't remember which dbt account is integrated, please email dbt Support at <support@getdbt.com> and we'll do our best to disassociate the integration for you. --- ### Why am I receiving a Runtime Error in my packages? If you're receiving the runtime error below in your packages.yml folder, it may be due to an old version of your dbt\_utils package that isn't compatible with your current dbt version. ```shell Running with dbt=xxx Runtime Error Failed to read package: Runtime Error Invalid config version: 1, expected 2 Error encountered in dbt_utils/dbt_project.yml ``` Try updating the old version of the dbt\_utils package in your packages.yml to the latest version found in the [dbt hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/): ```shell packages: - package: dbt-labs/dbt_utils version: xxx ``` If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Why are profiles stored outside of my project? Profiles are stored separately to dbt projects to avoid checking credentials into version control. Database credentials are extremely sensitive information and should **never be checked into version control**. --- ### Why can't I just write DML in my transformations? ###### `select` statements make transformations accessible More people know how to write `select` statements, than DML, making the transformation layer accessible to more people! ###### Writing good DML is hard If you write the DDL / DML yourself you can end up getting yourself tangled in problems like: * What happens if the table already exists? Or this table already exists as a view, but now I want it to be a table? * What if the schema already exists? Or, should I check if the schema already exists? * How do I replace a model atomically (such that there's no down-time for someone querying the table) * What if I want to parameterize my schema so I can run these transformations in a development environment? * What order do I need to run these statements in? If I run a `cascade` does it break other things? Each of these problems *can* be solved, but they are unlikely to be the best use of your time. ###### dbt does more than generate SQL You can test your models, generate documentation, create snapshots, and more! ###### You reduce your vendor lock in SQL dialects tend to diverge the most in DML and DDL (rather than in `select` statements) — check out the example [here](https://docs.getdbt.com/faqs/Models/sql-dialect.md). By writing less SQL, it can make a migration to a new database technology easier. If you do need to write custom DML, there are ways to do this in dbt using [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). --- ### Why dbt compile needs a data platform connection `dbt compile` needs a data platform connection in order to gather the info it needs (including from introspective queries) to prepare the SQL for every model in your project. ##### dbt compile The [`dbt compile` command](https://docs.getdbt.com/reference/commands/compile.md) generates executable SQL from `source`, `model`, `test`, and `analysis` files. `dbt compile` is similar to `dbt run` except that it doesn't materialize the model's compiled SQL into an existing table. So, up until the point of materialization, `dbt compile` and `dbt run` are similar because they both require a data platform connection, run queries, and have an [`execute` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) set to `True`. However, here are some things to consider: * You don't need to execute `dbt compile` before `dbt run` * In dbt, `compile` doesn't mean `parse`. This is because `parse` validates your written `YAML`, configured tags, and so on. ##### Introspective queries To generate the compiled SQL for many models, dbt needs to run introspective queries, (which is when dbt needs to run SQL in order to pull data back and do something with it) against the data platform. These introspective queries include: * Populating the relation cache. For more information, refer to the [Create new materializations](https://docs.getdbt.com/guides/create-new-materializations.md) guide. Caching speeds up the metadata checks, including whether an [incremental model](https://docs.getdbt.com/docs/build/incremental-models.md) already exists in the data platform. * Resolving [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros), such as `run_query` or `dbt_utils.get_column_values` that you're using to template out your SQL. This is because dbt needs to run those queries during model SQL compilation. * [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) compiles your project by default (unless you pass [`--no-compile`](https://docs.getdbt.com/reference/commands/cmd-docs.md)), so introspective macros such as [`run_query`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) run against the warehouse during documentation builds the same way they do during other compile workflows. Refer to [`run_query`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) for how that works and for using `flags.WHICH` when you want to limitDML or other side-effecting SQL to specific dbt commands. Without a data platform connection, dbt can't perform these introspective queries and won't be able to generate the compiled SQL needed for the next steps in the dbt workflow. You can [`parse`](https://docs.getdbt.com/reference/commands/parse.md) a project and use the [`list`](https://docs.getdbt.com/reference/commands/list.md) resources in the project, without an internet or data platform connection. Parsing a project is enough to produce a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md), however, keep in mind that the written-out manifest won't include compiled SQL. To configure a project, you do need a [connection profile](https://docs.getdbt.com/docs/local/profiles.yml.md) (`profiles.yml` if using the CLI). You need this file because the project's configuration depends on its contents. For example, you may need to use [`{{target}}`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) for conditional configs or know what platform you're running against so that you can choose the right flavor of SQL. --- ### Why do I need to quote column names in Jinja? In the [macro example](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) we passed the column name `amount` quotes: ```sql {{ cents_to_dollars('amount') }} as amount_usd ``` We have to use quotes to pass the *string* `'amount'` to the macro. Without the quotes, the Jinja parser will look for a variable named `amount`. Since this doesn't exist, it will compile to nothing. Quoting in Jinja can take a while to get used to! The rule is that you're within a Jinja expression or statement (i.e. within `{% ... %}` or `{{ ... }}`), you'll need to use quotes for any arguments that are strings. Single and double quotes are equivalent in Jinja – just make sure you match them appropriately. And if you do need to pass a variable as an argument, make sure you [don't nest your curlies](https://docs.getdbt.com/best-practices/dont-nest-your-curlies.md). --- ### Why do model and source YAML files always start with `version: 2`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future --- ### Why does dbt reject my custom GitHub application details? dbt validates your GitHub application against GitHub before saving it, so an invalid configuration is rejected instead of quietly breaking your team's connection. If you see an error when you save, check these in order: * **App ID**: Use the **App ID** from your GitHub app's settings page, not the client ID or the installation ID. * **Private key**: Paste the entire contents of the `.pem` file, including the `-----BEGIN RSA PRIVATE KEY-----` and `-----END RSA PRIVATE KEY-----` lines. If you lost the file, generate a new private key in GitHub. * **Client ID**: Paste the **Client ID** from your GitHub app. * **GitHub base URL**: Use the hostname of your GitHub instance with no trailing slash and no `/api/v3` path, for example `https://github.yourgreatcompany.com`. * **Client secret**: GitHub only shows this once. If you didn't copy it, generate a new one. * **Network access**: Your GitHub instance must be reachable from dbt. If you use [IP restrictions](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md), confirm the relevant CIDRs are allowed. For setup steps, refer to [Custom GitHub application](https://docs.getdbt.com/docs/platform/git/connect-github.md#custom-github-application), available in private beta. --- ### Why does my dbt output have so many macros in it? The output of a dbt run counts over 100 macros in your project! ```shell $ dbt run Running with dbt=1.7.0 Found 1 model, 0 tests, 0 snapshots, 0 analyses, 138 macros, 0 operations, 0 seed files, 0 sources ``` This is because dbt ships with its own project, which also includes macros! You can learn more about this [here](https://discourse.getdbt.com/t/did-you-know-dbt-ships-with-its-own-project/764). --- ### Why does the BigQuery OAuth application require scopes to Google Drive? BigQuery supports external tables over both personal Google Drive files and shared files. For more information, refer to [Create Google Drive external tables](https://cloud.google.com/bigquery/docs/external-data-drive). --- ### Why doesn’t an indirectly referenced upstream public model appear in Explorer? For [project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) in Mesh, [Catalog](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) only displays directly referenced [public models](https://docs.getdbt.com/docs/mesh/govern/model-access.md) from upstream projects, even if an upstream model indirectly depends on another public model. So for example, if: * `project_b` adds `project_a` as a dependency * `project_b`'s model `downstream_c` references `project_a.upstream_b` * `project_a.upstream_b` references another public model, `project_a.upstream_a` Then: * In Explorer, only directly referenced public models (`upstream_b` in this case) appear. * In the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) lineage view, however, `upstream_a` (the indirect dependency) *will* appear because dbt dynamically resolves the full dependency graph. This behavior makes sure that Catalog only shows the immediate dependencies available to that specific project. --- ### Why is my model being rebuilt instead of reused? dbt State decides whether to reuse a model by parsing the rendered SQL into a syntax tree and comparing the hash. If the hash has changed (implying the model's logic has changed), dbt State rebuilds the model. dbt State prioritizes safety and precision; if it can't guarantee skipping a node is safe, then it rebuilds the node to be sure. A few patterns that commonly cause overeager rebuilds are listed on this page, along with recommendations to increase reuse rate. The following patterns commonly cause unexpected rebuilds: * [Views with `select *`](#views-with-select) * [Non-deterministic Jinja templating](#non-deterministic-jinja-templating) * [Models with external sources in BigQuery](#models-with-external-sources-in-bigquery) * [Models with custom materializations](#models-with-custom-materializations) #### Views with `select *` dbt State reuses a model when its compiled SQL matches the stored hash. When a view uses `select *` directly on a `ref()` or `source()`, dbt can't determine the column list at parse time — the upstream model or source table might have gained or lost columns since the last run. To be safe, dbt State forces a rebuild. For example, this view will be rebuilt even if `stg_orders` hasn't changed because dbt can't know at parse time whether `stg_orders` has the same columns as before: ```sql -- stg_orders_view.sql (materialized: view) select * from {{ ref('stg_orders') }} ``` However, if you use `select *` on a CTE, dbt can resolve the columns from the CTE definition and safely reuse the view: ```sql with renamed as ( select order_id, customer_id, order_total from {{ ref('stg_orders') }} ) select * from renamed ``` If a CTE explicitly names its columns, a `select *` that reads from that CTE won't force a rebuild even if an earlier CTE used `select *` on a `ref()` or `source()`. The typical staging pattern is reused: ```sql with source as ( select * from {{ source('jaffle_shop', 'orders') }} ), renamed as ( select id as order_id, user_id as customer_id, amount as order_total from source ) select * from renamed ``` tip To avoid forced rebuilds, use explicit column names when selecting directly from a `ref()` or `source()`. You can also exclude views from execution using `--exclude config.materialized:view`. #### Non-deterministic Jinja templating Some macros and environment variables can cause unexpected rebuilds. For example, `dbt_utils.get_relations_by_pattern` (an introspective macro) combined with `dbt_utils.union_relations` can return relations in a different order on each run, producing different rendered SQL even when your project logic hasn't changed. Similarly, environment variables that change between runs produce different rendered SQL on every run: ```sql select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` Because the query result order or the environment variable's value changes, the rendered SQL differs from the stored hash on every run. dbt State treats this as a code change and rebuilds the model, even though the underlying project logic hasn't changed. This pattern can affect any model type, not just views; if a base or staging model rebuilds on every run, all of its downstream models rebuild, too. To avoid these unnecessary rebuilds, enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md). When enabled, dbt State checks both the Jinja template and rendered SQL; non-deterministic values that don't change the template don't trigger a rebuild. For example: ```sql {{ config(state={"compare_unrendered_code": true}) }} select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` #### Models with external sources on BigQuery On BigQuery, models that use external sources (such as Google Sheets) always rebuild because BigQuery doesn't expose modification timestamps for external sources, so dbt State can't determine freshness. tip To prevent external sources from always being considered stale, configure [`loaded_at_field`](https://docs.getdbt.com/reference/resource-configs/freshness.md) or [`loaded_at_query`](https://docs.getdbt.com/reference/resource-configs/freshness.md) in your source definition to point to a timestamp field. This lets dbt State query a timestamp field directly to determine freshness, rather than relying on warehouse metadata. #### Models with custom materializations Models using custom materializations are always built and are never reused. Custom materializations may have side effects (for example, modifying table properties or writing to other schemas), and dbt State cannot safely determine whether skipping the run would produce the same result. #### How to diagnose After a run, use (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) to see why dbt State rebuilt, reused, or cloned a specific model. For a detailed breakdown, use the `--verbose` flag with `-s` to select your model: note The command name differs by version: dbt v2 uses `dbt state explain` (with a space), while dbt v1 uses `dbt-state explain` (with a hyphen). (Applies to dbt v2.0 and later) ```bash dbt state explain --verbose -s my_model_name ``` If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. --- ### Why is Run on Pull request grayed out? If you're unable to enable Run on Pull requests, you'll want to make sure your existing repo was not added via the Deploy Key auth method. If it was added via a deploy key method, you'll want to use the [GitHub auth method](https://docs.getdbt.com/docs/platform/git/connect-github.md) to enable CI in dbt. To go ahead and enable 'Run on Pull requests', you'll want to remove dbt from the Apps & Integration on GitHub and re-integrate it again via the GitHub app method. If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Why might my actual warehouse costs differ from displayed costs? Cost Insights shows estimates based on warehouse-reported usage and your configured pricing variables. These estimates are based on a retroactive analysis of historical runs and reflect actual usage, *not* forecasts of future costs. Adjustments and differences may occur if: * Your warehouse has custom pricing that differs from the default compute credit unit. * There are discounts or credits applied at the billing level that aren't reflected in usage tables. * Costs include other charges beyond compute. Costs Insights in the dbt platform is designed to be directionally accurate, showing you dbt-specific components rather than matching your billing exactly. --- ### Why would I want to impersonate a service account? You may want your models to be built using a dedicated service account that has elevated access to read or write data to the specified project or dataset. Typically, this requires you to create a service account key for running under development or on your CI server. By specifying the email address of the service account you want to build models as, you can use [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default) or the service's configured service account (when running in GCP) to assume the identity of the service account with elevated permissions. This allows you to reap the advantages of using federated identity for developers (via ADC) without needing to grant individual access to read and write data directly, and without needing to create separate service account and keys for each user. It also allows you to completely eliminate the need for service account keys in CI as long as your CI is running on GCP (Cloud Build, Jenkins, GitLab/Github Runners, etc). --- ## Fusion ### Arrow ADBC and dbt v2 This document provides technical guidance for dbt partners and vendors on how to design, build, and maintain ADBC (Apache Arrow Database Connectivity) drivers for dbt v2, the new dbt engine. dbt v2 leverages ADBC as a unified driver layer for seamless, high-performance integration with data platforms. Building an ADBC driver is the first step to connecting dbt v2 with a new platform. #### Why dbt v2 uses ADBC dbt v2 represents a major evolution in the dbt engine with minimal changes to the authoring layer. Built in Rust, dbt v2 delivers speed, language understanding, and seamless integration with numerous data warehouses. A key aspect of the new engine is its adoption of ADBC — a modern, open standard from the Apache Arrow project that simplifies columnar data interchange across platforms. Historically, dbt v1 adapters required bespoke connection logic for each data platform. dbt v2 improves on this model with a unified ADBC driver layer that offers several key advantages: * **Standardization**: ADBC standardizes common platform features across a single interface. * **Performance**: Drivers leverage Arrow's columnar memory format for efficient query execution with minimal transformations. * **Maintainability**: ADBC drivers follow a shared specification, reducing the complexity of implementing new adapters. #### Technical overview This technical specification covers the ADBC specification. The specification maintains backwards compatibility, so guidance here remains valid as the spec evolves. For the latest information and detailed documentation, refer to the [ADBC documentation](https://arrow.apache.org/adbc/current/), which is the source of truth. The ADBC API provides a powerful array of features, but you don't need to implement all of them. This section covers the API surface required for dbt v2 compatibility. ##### Programming language **tl;dr: Use Go.** One distinct advantage of Arrow ADBC is portability. You can write drivers in various languages and load them via driver managers. This portability allows dbt v2 (written in Rust) to leverage drivers written in other languages. For dbt v2 compatibility, drivers must: * Compile into shared libraries that can be loaded from any program * Produce a platform-specific, standalone binary We recommend **Go** as the language of choice, though Rust or C++ also work. Go has a runtime and garbage collector, but it's engineered to compile into well-behaved shared libraries—unlike languages like C# or Java. A standalone binary allows users to download and run the driver out of the box without setting up an interpreter. Compiled languages like Go also enable dbt v2 and its drivers to share memory directly over FFI without external dependencies. ##### ADBC specifications This section covers the minimum requirements for a dbt v2-compatible ADBC driver. For complete details on the ADBC specification and driver development, refer to the [ADBC driver authoring guide](https://arrow.apache.org/adbc/current/driver/authoring.html). Drivers consist of several key abstractions: 1. **Driver**: Load a driver to create databases. 2. **Database**: Create databases and set configuration options (including authentication). Use databases to open connections. 3. **Connection**: Establish connections to the warehouse. Connections create statements. 4. **Statement**: Set options and SQL queries on statements, then execute them against the warehouse. dbt v2 achieves high performance through aggressive parallelism, so expect many simultaneous connections during project execution. ##### Authentication Drivers handle authentication through key-value options set on the database. dbt v2 translates options from user-authored `profiles.yml` files before passing them to the driver. For example, what dbt calls `client_secret` in a Snowflake profile gets set on the driver as `adbc.snowflake.sql.client_option.client_secret`. For a complete example of how dbt v2 translates profile options, see the [Snowflake authentication source code](https://github.com/dbt-labs/dbt-fusion/blob/main/crates/dbt-auth/src/snowflake/mod.rs). For more information on profile configuration, refer to [dbt profiles](https://docs.getdbt.com/docs/local/profiles.yml.md). ###### Credential caching Simple authentication methods (like username/password stored in `profiles.yml`) support fully parallel connection creation with no special handling required. For authentication methods that require browser interaction (user-to-machine OAuth, SSO, or MFA), implement credential caching. Due to dbt v2's highly parallel execution, without caching, every new connection prompts the user for authentication repeatedly. Your credential cache for browser-based authentication must: * Block new connections until an initial connection establishes and stores a token in memory (avoiding the thundering herd problem). * Handle token refresh using the same blocking principle when invalidation occurs. * Use interprocess, file-system-based storage to support the LSP, which runs in a separate process. This caching is critical for any browser-based or MFA authentication option, but is not needed for simple credential-based authentication. #### Required APIs This section covers the minimum API set for dbt v2 compatibility. The requirements are: * Authentication via options. * SQL query execution. * Metadata queries to understand remote warehouse state (certain connection-level metadata functions can use cheaper or more performant APIs to pull table schemas). These requirements are not exhaustive. dbt Labs encourages implementing the full ADBC specification to benefit both dbt v2 and the broader ADBC community. ###### Driver | Method | Description | | ------------- | ------------------------------- | | `NewDatabase` | Create a new database instance. | ###### Database | Method | Description | | ------------ | ----------------------------------------------------- | | `SetOptions` | Set configuration options (including authentication). | | `Open` | Open a connection. | | `Close` | Close the database. | ###### Connection | Method | Description | | ---------------- | --------------------------------------- | | `GetObjects` | Pull metadata from the warehouse. | | `GetTableSchema` | Pull schema metadata for tables. | | `NewStatement` | Create a statement for query execution. | | `Close` | Close the connection. | ###### Statement | Method | Description | | --------------- | ----------------------------------- | | `SetOption` | Set options on queries. | | `SetSqlQuery` | Set the SQL query text. | | `ExecuteQuery` | Execute a query and return results. | | `ExecuteUpdate` | Execute DML queries. | | `Close` | Close the statement. | --- ### Compare changes during development Available in v2 | Local development | Login required info Compare changes needs a baseline to compare against — either [dbt platform deferral](#how-it-works), which uses your dbt platform account, or a local `manifest.json`, which doesn't. The dbt VS Code extension previews and compares how your local edits affect your data in your dbt platform account — including added/removed rows and join verification — without waiting on CI. Use compare changes to check impact early and validate changes before you open a PR or run a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md): * Validate outputs are correct when refactoring logic, adding or removing columns, or implementing join modifications. * It compares your current working copy against your `manifest.json` (for example, your last production state) and shows changes to primary keys, rows, and columns in the **Compare** tab. The dbt VS Code extension's compare changes feature is different from the [Advanced CI compare changes feature](#how-this-differs-from-advanced-ci), which runs at the PR stage in deployment rather than locally during development. info Compare changes in development is available for models only. Support for seeds, snapshots, ephemeral models, and Python models is coming soon. [![Make changes to a model and see the changes in the Compare tab](/img/docs/extension/vs-compare-changes.png?v=2 "Make changes to a model and see the changes in the Compare tab")](#)Make changes to a model and see the changes in the Compare tab #### Prerequisites To use the dbt VS Code extension compare changes feature, you need: * A dbt v2 [supported data platform](https://docs.getdbt.com/docs/dbt/supported-features.md?version=2.0#requirements) (BigQuery, Databricks, Redshift, or Snowflake) * The [dbt VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md) installed with a self-hosted installation of the [dbt v2](https://docs.getdbt.com/docs/dbt/get-started-dbt.md) * A baseline state to compare your changes against — refer to [How it works](#how-it-works) to choose between [automatic deferral](https://docs.getdbt.com/docs/platform/about-defer.md) or [`manifest.json`](https://docs.getdbt.com/reference/artifacts/manifest-json.md?version=2.0) manual setup. **Sign-in and authentication** Whether you need to sign in depends on how you set your baseline state: * **If you use dbt platform deferral (recommended):** [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) to a dbt platform account. You'll also need a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file in your local `.dbt` directory (`~/.dbt/dbt_cloud.yml` on macOS/Linux) so the extension can fetch the deferred manifest from your environment. * **If you set the baseline manually with a `manifest.json`:** No dbt platform account is required. Point the extension at a local `manifest.json` and compare changes runs against it. #### How it works Compare changes in development works by comparing two materialized models in your warehouse. Specifically, it compares the model built in your dev schema (determined by your active profile) against the model referenced in your `manifest.json` (for example, your last production state). Both sides of the comparison are always warehouse tables; it does not compare SQL file contents. The path you choose determines whether you need a dbt platform account: * **dbt platform deferral (recommended):** Requires a dbt platform account and at least one successful job run in the environment you're deferring to (usually staging or production). dbt v2 auto-downloads the deferred manifest and uses it as your baseline state. * **Manual `state` directory:** Point the extension at a local `manifest.json` (for example, copied from another environment). No dbt platform account is required; no job run is required. <br /> #### Use compare changes To use compare changes in development, follow these steps: 1. Open a SQL model file in your editor. 2. Make some changes to the model, like adding a new column or modifying an existing one (for example, `left_join_customers` to `right_join_customers`). 3. Run compare changes in the [VS Code interface](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette): * Command palette: Open the VS Code command palette (Shift + Command + P (Mac) / Ctrl + Shift + P (Windows/Linux)) and search for the [**dbt: Compare changes**](vscode://dbtLabsInc/dbt.compareModel) * Bottom panel: Click the **Compare** tab and then click the **Compare** button. [![Compare changes in development](/img/docs/extension/vs-compare-changes-options.png?v=2 "Compare changes in development")](#)Compare changes in development 4. Once you click the **Compare** button, the extension will execute a `dbt build` command to build the model you're working on and then runs the comparison. The [**Compare** tab](#compare-tab) displays the changes to the data's primary keys, rows, and columns. Clicking the tabs will display more details about the changes, like specific columns that were added or modified. 5. Once you've compared changes and see the changes in the **Compare** tab, you can then decide to commit your changes or continue editing. #### Compare tab results The **Compare** tab displays the changes to the data's primary keys, rows, and columns. Clicking the tabs will display more details about the changes, like specific columns that were added or modified. [![Example of the Compare tab](/img/docs/extension/vs-compare-changes-tab.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab * **Overview tab**: High-level summary about the changes to the models, such as the number of primary keys that were added or removed, rows modified, and so on. It will also include the relation between models that were added or modified. * **Primary keys tab**: Details about the changes to the records. * **Modified rows tab**: Details about the modified rows. * **Columns tab**: Details about the changes to the columns. #### FAQs  Are queries run on behalf of the developer? Yes. All comparison queries in development run using your development credentials or platform-based user credentials, directly from the dbt VS Code extension. If you authenticated with a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file, these are the credentials configured in your **Account settings**; otherwise it uses the credentials from your active dbt profile. dbt v2 uses your credentials to execute comparison queries in your warehouse. The results are stored in memory, so that we can keep them populated into the Compare tab for that file until you re-run.  Is this using my warehouse credits? Yes. Because the comparison runs in your development environment using your dev credentials, it will use your warehouse’s compute.  Do I need to run dbt build every time I make a change? No. When you click **Compare**, the extension builds the model into your development schema automatically, so you don’t need to run `dbt build` yourself. It then compares that development build against the version in your deferred environment (usually staging or production). If the model hasn’t been built yet in your deferred environment, the comparison can’t run. For more details, see [Defer to production](https://docs.getdbt.com/docs/platform/about-defer.md).  How is this different from Advanced CI compare changes? The dbt VS Code extension's compare changes feature applies only to your self-hosted dbt development environment. If you're looking to compare changes between your production environment and the pull request's latest commit, check out [Advanced CI compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes). | Aspect | In development (compare changes) | In deployment (Advanced CI) | | ------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Affects** | Development for one modified model at a time | Deployment for all modified models in a project | | **Trigger** | On-demand in editor | PR open/update and CI job | | **Scope** | Your working copy and local target | Branch head versus prod state in CI | | **Output** location | Compare panel in VS Code/Cursor. Does not create a PR comment in Git provider | Deployment job compare tab and PR summary comment in Git provider | | **Data caching** | Editor-side | dbt platform [caches](https://docs.getdbt.com/docs/deploy/advanced-ci.md#about-the-cached-data) limited samples | | **Governance** | Local development credentials | Production credentials | #### Related docs * [Advanced CI compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) --- ### dbt v2 readiness checklist dbt v2 is here and is now generally available for the dbt platform. Enable the dbt v2 readiness panel The dbt v2 readiness panel shows each project's eligibility status and blockers in the dbt platform. It's rolling out in phases — if it's not enabled for your account yet, an [account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) can turn it on in **Account settings** → **Account**. Refer to [Enable dbt v2 readiness features](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=2) for setup steps. If you have access to dbt Wizard, use the [dbt Wizard's dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) to help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed! #### Preparing for dbt v2 Use the following checklist to prepare your projects for dbt v2 For walkthroughs of both the preparation and upgrade processes, check out our detailed guides: * \[ ] [Upgrade to dbt v2 Pt. 1: Preparing to upgrade](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=1) * \[ ] [Upgrade to dbt v2 Pt. 2: Making the move](https://docs.getdbt.com/guides/upgrade-to-v2.md?step=1) ##### Upgrade to the latest dbt version (recommended) The **v1 Latest** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) has all of the most recent features to help you prepare for dbt v2. * \[ ] Make sure all your projects are on the **v1 Latest** release track across all deployment environments and jobs. This is not a strict requirement for upgrading, but it will ensure the simplest, most predictable experience by allowing you to pre-validate that your project doesn't rely on deprecated behaviors. ##### Resolve all deprecation warnings You must resolve deprecations while your projects are on a dbt v1 release track, as they result in warnings that will become errors once you upgrade to dbt v2. The autofix tool can automatically resolve many deprecations (such as moving arbitrary configs into the meta dictionary). For a full list of deprecations and how to resolve them, refer to [Deprecations](https://docs.getdbt.com/reference/deprecations.md). Start a new branch to begin resolving deprecation warnings using one of the following methods: * \[ ] **Run autofix in the dbt platform:** You can address deprecation warnings using the [autofix tool in the Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md). You can run the autofix tool on the **v1 Compatible** or **v1 Latest** release track. * \[ ] **Run autofix locally:** Use the [VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md). The extension has a built-in ["Getting Started" workflow](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started) that will debug your dbt project in the VS Code or Cursor IDE and execute the autofix tool. This has the added benefit of installing dbt v2 to your computer so you can begin testing locally before implementing in your dbt platform account. * \[ ] **Run autofix locally (without the extension):** Visit the autofix [GitHub repo](https://github.com/dbt-labs/dbt-autofix) to run the tool locally if you're not using VS Code or Cursor. This will only run the tool, it will not install dbt v2. * \[ ] **Review behavior change flag overrides:** dbt v2 removes the [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) marked `2.0` in the **Removed** column and always enables their new behavior. Delete any `flags:` overrides in your `dbt_project.yml` that opt out of those flags. Other flags remain configurable (including v2-specific flags on that page); validate that your project works with the mature / removed defaults before upgrading. ##### Upgrade YAML spec * \[ ] **Migrate Semantic Layer configs:** If your project uses the Semantic Layer, make sure your metric configurations use the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). ##### Validate and upgrade your dbt packages The most commonly used dbt Labs managed packages (such as `dbt_utils` and `dbt_project_evaluator`) are already compatible with dbt v2, as are a large number of external and community packages. Review [the dbt package hub](https://hub.getdbt.com) to see verified dbt v2-compatible packages by checking that the `require-dbt-version` configuration includes `2.0.0` or higher. Refer to [package support](https://docs.getdbt.com/docs/dbt/supported-features.md#package-support) for more information. * \[ ] Make sure that all of your packages are upgraded to the most recent version, many of which contain enhancements to support dbt v2. * \[ ] Check package repositories to make sure they're compatible with dbt v2. If a package you use is not yet compatible, we recommend opening an issue with the maintainer, making the contribution yourself, or removing the package temporarily before you upgrade. ##### Validate user-defined functions Check that dbt v2 supports all user-defined functions (UDFs) in your project. dbt v2 supports nearly all built-in data platform functions out of the box. However, data platforms continuously add new functions that dbt v2 may not yet support. If you see the error `dbt0209: No function <function name>`, you can resolve it depending on whether the function is a UDF or a built-in function: * \[ ] **For custom UDFs:** Recreate it as a [native dbt UDF](https://docs.getdbt.com/docs/build/udfs.md#defining-udfs-in-dbt) to get the full dbt v2 experience. With `static_analysis: baseline` (the default), most UDFs will work out of the box. * \[ ] **For Warehouse-native functions:** Submit a [GitHub issue](https://github.com/dbt-labs/dbt). dbt v2's `baseline` mode handles most cases, but will throw warnings and not errors. You can set `static_analysis: off` for specific models if needed. For more information about using `strict` in development and `baseline` in deployment, refer to [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md). ##### Check for known dbt v2 limitations Your project may implement features that dbt v2 currently [limits](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations) or doesn't support. * \[ ] Remove unnecessary features from your project to make it dbt v2 compatible. * \[ ] Monitor progress for critical features, knowing we are working to bring them to dbt v2. You can monitor their progress using the issues linked in the [limitations table](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations). ##### Review jobs configured in the dbt platform We determine dbt v2 eligibility using data from your job runs. * \[ ] Ensure you have at least one job running in each of your projects in the dbt platform. * \[ ] Ensure each job has run within the last 7 days. Jobs that haven't run recently are considered inactive and are ineligible for dbt v2. If you see a "no active jobs" ineligibility reason in the v2 readiness UI, run the job manually or adjust its schedule. * \[ ] Ensure all jobs are running on the [**v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md#which-release-tracks-are-available) for the smoothest upgrade experience. * \[ ] Resolve any job failures — all jobs must run successfully for eligibility checks to work. * \[ ] Delete any jobs that are no longer in use to ensure accurate eligibility reporting. * \[ ] Make sure you've promoted the changes for deprecation resolution and package upgrades to your git branches that map to your deployment environments. * \[ ] For eligible jobs, use **Debug on dbt v2** to debug in Studio IDE or run once on dbt v2. Refer to [Update your jobs](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=7). --- ### Feature availability Get started with many dbt features right away when you install dbt v2 and the dbt VS Code extension. Access dbt platform-based capabilities when you sign in with any dbt platform account, even the free developer tier! #### What you get with v2 For the best dbt experience, use dbt v2 with the dbt VS Code extension to access the following features: * **dbt v1 workflows** (except dbt docs v1) * [**Syntax error detection**](https://docs.getdbt.com/docs/dbt-extension-features.md?version=2#live-error-detection)\* (Jinja, YAML, SQL) * [**dbt lint**](https://docs.getdbt.com/reference/commands/lint.md?version=2) \* * [**dbt docs v2**](https://docs.getdbt.com/reference/commands/cmd-docs.md?version=2#dbt-docs-v2)\* (lite or full): Full includes column-level lineage. * [**LSP**](https://docs.getdbt.com/docs/about-dbt-lsp.md?version=2)\* (lite): Go-to ref, source, and macro * [**Full LSP**](https://docs.getdbt.com/docs/about-dbt-lsp.md?version=2): CTE, hover to see schema, and more * [**SQL comprehension, type checking, and impact analysis**](https://docs.getdbt.com/blog/the-levels-of-sql-comprehension) * [**Precise column-level lineage**](https://docs.getdbt.com/docs/explore/column-level-lineage.md?version=2) \* Free forever. Really! Connect your dbt platform account editor for additional enhanced capabilities like the **Catalog** tab. To learn more about VS Code-specific capabilities, refer to [dbt VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md). #### Related docs * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) locally * Install the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * Upgrade environments in the [dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) --- ### Get started with dbt ##### New to dbt? Start here Once you've caught up on everything [dbt has to offer](https://docs.getdbt.com/docs/introduction.md), start with the path that matches how you want to use dbt. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/local/install-dbt.md) ###### [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) [Install dbt from your command line.](https://docs.getdbt.com/docs/local/install-dbt.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/about-dbt-extension.md) ###### [Install the dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) [Get dbt's performance plus autocomplete, inline errors, and lineage in your editor.](https://docs.getdbt.com/docs/about-dbt-extension.md) [![](/img/icons/dbt-bit.svg)](https://www.getdbt.com/signup) ###### [Sign up for the dbt platform](https://www.getdbt.com/signup) [Create a dbt platform account to access dbt platform-based capabilities.](https://www.getdbt.com/signup) ##### Already using dbt? Start here Upgrade your existing projects to v2 and learn about the tools available to you once you're there. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) ###### [Preparing to upgrade from v1 to v2](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [Step-by-step guide to prepare your dbt platform projects for upgrading to v2.](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/upgrade-to-v2.md) ###### [Upgrade to v2](https://docs.getdbt.com/guides/upgrade-to-v2.md) [Learn how to upgrade your eligible projects on the dbt platform to v2.](https://docs.getdbt.com/guides/upgrade-to-v2.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) ###### [Hybrid dbt platform and local workflows](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) [Learn how to keep credentials, environment variables, and dbt versions in sync between dbt platform and local development.](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) --- ### Self-hosted dbt releases This page shows release information for local builds of dbt v2 only. v2 releases on the dbt platform adhere to the [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) categories, giving you control over release cadence and stability. Track current versions and full release history for v2. This data updates live from dbt release channels. Each of the versions on this page links to the matching section in the [v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) on GitHub. #### Release channels dbt v2 is distributed through three release channels: | Channel | Description | Stability | | -------- | -------------------------------------------- | ------------------------------------------------------------------- | | `latest` | The known `good` stable version | ✅ Recommended for production | | `canary` | The latest version to be officially released | ⚠️ Most recent stable version but still undergoing thorough testing | | `dev` | The latest development build | ❌ May be unstable; may not have passed all internal tests | #### Known-bad releases If a shipped v2 release is later found to contain a regression, dbt Labs flags it as a known-bad release. If you have a flagged version installed, the dbt VS Code extension shows a warning notification telling you which version to update to. To move off a flagged version, update it using your installation method (for example, pip or Homebrew). For details, including how air-gapped users receive these notifications, refer to [Known-bad releases](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases). #### dbt platform release tracks On dbt platform, each [environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) uses the account default or your chosen **release track**. Release tracks control how often that environment receives new v2 builds. They're separate from the local CLI release channels in the previous section. For cadence, plan availability, and API values (`nightly`, `stable`, and more), refer to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md#dbt-v2-release-tracks). To change the release track for an environment, follow [Upgrade dbt in dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). Live data below is for local CLI channels The **Current versions** cards and full release list below pull the public v2 manifest used for *local* installs (`dev`, `canary`, `latest`). You should use [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for dbt platform planning. Updating your self-hosted installation The following commands apply only to *local* installations of dbt. They don't affect which v2 build your dbt platform environments use. Instead, you can set a [release track](https://github.com/docs/dbt-versions/dbt-release-tracks#dbt-v2-release-tracks) per environment in dbt platform. Running the system update command without a version flag installs the `latest` stable release: ```shell dbt system update ``` To install a specific channel or version, pass the `--version` flag: ```shell dbt system update --version canary # Install the canary release dbt system update --version dev # Install the dev release dbt system update --version 2.0.0 # Install a specific version ``` ##### Current versions ###### Dev [`v2.1.0.dev20261030`](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261030 "View this version in the dbt v2 changelog") 2026-10-30 ###### Canary [`v2.0.10`](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#2010 "View this version in the dbt v2 changelog") 2026-10-28 ###### Latest [`v2.0.9`](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#209 "View this version in the dbt v2 changelog") 2026-10-19 ##### All releases [v2.1.0.dev20261017](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261017 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **github-merge-queue**Oct 17, 2026, 09:05 AM Automated promotion [v2.1.0.dev20261019](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261019 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **chayac**Oct 19, 2026, 07:08 PM Automated promotion [v2.1.0.dev20261020](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261020 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **github-merge-queue**Oct 20, 2026, 09:10 AM Automated promotion [v2.1.0.dev20261021](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261021 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **github-merge-queue**Oct 21, 2026, 09:11 AM Automated promotion [v2.1.0.dev20261022](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261022 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **github-merge-queue**Oct 22, 2026, 09:13 AM Automated promotion [v2.1.0.dev20261023](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261023 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **github-merge-queue**Oct 23, 2026, 09:11 AM Automated promotion [v2.1.0.dev20261024](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261024 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **kczimm**Oct 29, 2026, 04:02 PM Revert nightly track that the v2.0.10 patch rollout moved [v2.1.0a1](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210a1 "View this release in the dbt v2 changelog")Goodpreview Released by: **kczimm**Oct 27, 2026, 05:56 PM first promotion [v2.1.0a2](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210a2 "View this release in the dbt v2 changelog")Goodpreview Released by: **kczimm**Oct 27, 2026, 10:41 PM FS alpha release 2.1.0a2 [v2.1.0.dev20261030](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#210dev20261030 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **github-merge-queue**Oct 30, 2026, 09:08 AM Automated promotion [v2.0.10](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#2010 "View this release in the dbt v2 changelog")GoodDevnightlyCanary Released by: **chayac**Oct 28, 2026, 11:39 PM Planned orchestration promotion [v2.0.9](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#209 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **johnchappelledbt**Oct 29, 2026, 09:55 PM Automated ST snapshot [v2.0.8](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#208 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **johnchappelledbt**Oct 22, 2026, 09:05 PM Automated ST snapshot [v2.0.7](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#207 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **kczimm**Sep 25, 2026, 10:51 PM Automated promotion [v2.0.6](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#206 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLateststableST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **johnchappelledbt**Oct 7, 2026, 09:13 PM Automated ST snapshot [v2.0.5](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#205 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLateststable Released by: **xuliangs**Sep 17, 2026, 10:36 PM Bug fix for INC-8093 [v2.0.4](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#204 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLateststable Released by: **tauhid621**Sep 16, 2026, 07:40 AM planned [v2.0.3](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#203 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **tauhid621**Sep 16, 2026, 02:03 AM Automated promotion [v2.0.2](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#202 "View this release in the dbt v2 changelog")GoodDevnightlyLatest Released by: **mishamsk**Sep 15, 2026, 11:42 PM planned [v2.0.1](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#201 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystable Released by: **itsnamangoyal**Sep 14, 2026, 05:28 PM planned promotion of v2.0.1 to stable [v2.0.0-preview.221](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview221 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **jasonlin45**Sep 9, 2026, 03:12 AM Automated promotion [v2.0.0-preview.220](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview220 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **jasonlin45**Sep 3, 2026, 11:23 PM Automated promotion [v2.0.0-preview.219](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview219 "View this release in the dbt v2 changelog")GoodDevnightlyCanary Released by: **aiguofer**Sep 3, 2026, 07:56 PM planned promotion [v2.0.0-preview.218](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview218 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLatestST MondayST WednesdaystableST Thursdayst-monday-stablest-wednesday-stablest-thursday-stable Released by: **mishamsk**Sep 14, 2026, 04:22 PM 2.0.0 was accidentally promoted early [v2.0.0-preview.217](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview217 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **TIHan**Aug 27, 2026, 08:53 AM Automated promotion [v2.0.0-preview.216](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview216 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **github-merge-queue**Aug 26, 2026, 08:59 AM Automated promotion [v2.0.0-preview.215](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview215 "View this release in the dbt v2 changelog")Known BadDevnightlyCanary Released by: **mishamsk**Aug 25, 2026, 11:15 PM Planned canary promotion [v2.0.0-preview.214](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview214 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **mach-kernel**Aug 25, 2026, 03:50 PM Automated promotion [v2.0.0-preview.213](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview213 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLatest Released by: **mishamsk**Aug 26, 2026, 10:25 PM planned [v2.0.0-preview.212](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview212 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystablest-monday-stableLatestst-wednesday-stableextendedst-thursday-stablest-monday-extendedst-wednesday-extendedst-thursday-extendedfallbackst-monday-fallbackst-wednesday-fallbackst-thursday-fallback Released by: **johnchappelledbt**Oct 29, 2026, 09:55 PM Automated ST snapshot [v2.0.0-preview.210](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview210 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLateststableST MondayST Wednesday Released by: **jeremyhutt11**Aug 26, 2026, 11:46 PM Automated ST promotion [v2.0.0-preview.209](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview209 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Aug 20, 2026, 09:09 PM Automated ST snapshot [v2.0.0-preview.208](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview208 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **github-merge-queue**Aug 8, 2026, 08:59 AM Automated promotion [v2.0.0-preview.207](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview207 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **akbog**Aug 6, 2026, 12:33 AM Automated promotion [v2.0.0-preview.206](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview206 "View this release in the dbt v2 changelog")Known BadDevnightlyCanary Released by: **akbog**Aug 5, 2026, 01:11 AM Planned Promotion [v2.0.0-preview.205](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview205 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **akbog**Aug 14, 2026, 02:44 AM Incident Investigation [v2.0.0-preview.204](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview204 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatest Released by: **akbog**Aug 3, 2026, 08:18 PM Planned Promotion [v2.0.0-preview.203](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview203 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableextendedst-wednesday-stableST Thursdayst-thursday-stablest-monday-extendedst-wednesday-extendedst-thursday-extendedfallbackst-monday-fallbackst-wednesday-fallbackst-thursday-fallback Released by: **johnchappelledbt**Oct 7, 2026, 09:13 PM Automated ST snapshot [v2.0.0-preview.202](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview202 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 30, 2026, 09:26 PM Automated ST snapshot [v2.0.0-preview.201](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview201 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **mishamsk**Jul 20, 2026, 05:42 PM Automated promotion [v2.0.0-preview.200](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview200 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLateststableST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 23, 2026, 09:25 PM Automated ST snapshot [v2.0.0-preview.199](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview199 "View this release in the dbt v2 changelog")GoodDevCanarynightlystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **jeremyhutt11**Jul 16, 2026, 10:12 PM Automated ST snapshot [v2.0.0-preview.198](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview198 "View this release in the dbt v2 changelog")GoodDev Released by: **kczimm**Jul 9, 2026, 09:36 PM Automated promotion [v2.0.0-preview.197](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview197 "View this release in the dbt v2 changelog")GoodDevCanarynightly Released by: **akbog**Jul 2, 2026, 07:20 PM Planned promotion [v2.0.0-preview.196](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview196 "View this release in the dbt v2 changelog")GoodDevCanaryLateststableST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 9, 2026, 09:31 PM Automated ST snapshot [v2.0.0-preview.195](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview195 "View this release in the dbt v2 changelog")GoodDev Released by: **aiguofer**Jun 30, 2026, 12:08 AM Automated promotion [v2.0.0-preview.194](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview194 "View this release in the dbt v2 changelog")GoodDevCanarystableextendedst-monday-extendedst-wednesday-extendedst-thursday-extendedfallbackst-monday-fallbackst-wednesday-fallbackst-thursday-fallback Released by: **TimKlense**Sep 3, 2026, 09:03 PM Automated ST snapshot [v2.0.0-preview.193](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview193 "View this release in the dbt v2 changelog")GoodDevCanaryLatestnightlystableST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 2, 2026, 09:26 PM Automated ST snapshot [v2.0.0-preview.192](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview192 "View this release in the dbt v2 changelog")Known BadDevCanaryLatestnightlystable Released by: **akbog**Jun 18, 2026, 06:22 PM Planned promotion [v2.0.0-preview.191](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview191 "View this release in the dbt v2 changelog")Known BadDevCanarynightly Released by: **akbog**Jun 12, 2026, 09:14 PM Planned Promotion [v2.0.0-preview.190](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview190 "View this release in the dbt v2 changelog")GoodDevCanarynightlystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jun 25, 2026, 09:33 PM Automated ST snapshot [v2.0.0-preview.189](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview189 "View this release in the dbt v2 changelog")GoodDevCanaryLateststable Released by: **akbog**Jun 11, 2026, 03:15 AM Planned Promotion [v2.0.0-preview.188](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview188 "View this release in the dbt v2 changelog")Known BadDevCanarystablenightlyst-monday-stable Released by: **fa-assistant**Jun 8, 2026, 07:05 PM Automated ST snapshot [v2.0.0-preview.187](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview187 "View this release in the dbt v2 changelog")Known BadDev Released by: **akbog**Jun 6, 2026, 02:06 AM Automated promotion [v2.0.0-preview.186](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview186 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableextendedST Thursdayst-thursday-stablest-monday-extendedst-wednesday-extendedst-thursday-extendedfallbackst-monday-fallbackst-wednesday-fallbackst-thursday-fallback Released by: **laconc**Aug 6, 2026, 10:00 PM Automated ST snapshot [v2.0.0-preview.185](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview185 "View this release in the dbt v2 changelog")GoodDev Released by: **akbog**Jun 3, 2026, 07:49 AM Automated promotion [v2.0.0-preview.184](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview184 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestst-thursday-stable Released by: **fa-assistant**Jun 4, 2026, 09:34 PM Automated ST snapshot [v2.0.0-preview.183](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview183 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablenightlyst-wednesday-stable Released by: **fa-assistant**Jun 3, 2026, 10:03 PM Automated ST snapshot [v2.0.0-preview.182](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview182 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablenightlyST Mondayst-monday-stableST WednesdayST Thursday Released by: **fa-assistant**Jun 4, 2026, 09:32 PM Automated ST promotion [v2.0.0-preview.181](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview181 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestnightly Released by: **akbog**Jun 1, 2026, 06:55 AM Planned Promotion [v2.0.0-preview.180](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview180 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestnightly Released by: **akbog**May 30, 2026, 06:35 AM Planned promotion [v2.0.0-preview.179](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview179 "View this release in the dbt v2 changelog")GoodDev Released by: **kczimm**May 29, 2026, 05:01 PM Automated promotion [v2.0.0-preview.178](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview178 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablenightlyst-wednesday-stablest-thursday-stable Released by: **fa-assistant**May 29, 2026, 01:12 PM Automated ST snapshot [v2.0.0-preview.177](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview177 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestnightlyst-thursday-stableST Mondayst-monday-stableST WednesdayST Thursday Released by: **fa-assistant**May 29, 2026, 01:10 PM Automated ST promotion [v2.0.0-preview.176](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview176 "View this release in the dbt v2 changelog")GoodDevCanarynightlystableLatestst-wednesday-stablest-thursday-stableST MondayST WednesdayST Thursday Released by: **fa-assistant**May 21, 2026, 09:31 PM Automated ST promotion [v2.0.0-preview.175](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview175 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablest-thursday-stablest-monday-stablest-wednesday-stableST MondayextendedfallbacknightlyST WednesdayST Thursdayst-monday-extendedst-monday-fallbackst-wednesday-extendedst-wednesday-fallbackst-thursday-extendedst-thursday-fallback Released by: **TimKlense**Jul 9, 2026, 09:31 PM Automated ST snapshot [v2.0.0-preview.174](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview174 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Apr 29, 2026, 03:25 PM Planned orchestration promotion [v2.0.0-preview.173](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview173 "View this release in the dbt v2 changelog")release candidateGoodDevCanaryLatestST MondaynightlystableST WednesdayST Thursday Released by: **TimKlense**Apr 30, 2026, 11:02 PM manually updated for thursday [v2.0.0-preview.172](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview172 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Apr 15, 2026, 04:28 PM Planned orchestration promotion [v2.0.0-preview.171](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview171 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Apr 16, 2026, 09:21 PM Automated ST promotion [v2.0.0-preview.170](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview170 "View this release in the dbt v2 changelog")GoodDevCanaryLatestextendedfallbackst-thursday-extendedst-thursday-fallbackst-monday-extendedst-monday-fallbackst-wednesday-extendedst-wednesday-fallback Released by: **agelber-dbt**May 1, 2026, 02:22 PM Preparing for GA [v2.0.0-preview.169](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview169 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Apr 8, 2026, 12:06 PM Automated promotion [v2.0.0-preview.168](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview168 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Apr 8, 2026, 08:04 PM Planned orchestration promotion [v2.0.0-preview.167](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview167 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Apr 7, 2026, 12:17 PM Automated promotion [v2.0.0-preview.166](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview166 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Apr 7, 2026, 04:34 PM Planned orchestration promotion [v2.0.0-preview.165](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview165 "View this release in the dbt v2 changelog")GoodDev Released by: **akbog**Apr 4, 2026, 03:16 AM Automated promotion [v2.0.0-preview.164](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview164 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Apr 9, 2026, 09:21 PM Automated ST promotion [v2.0.0-preview.163](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview163 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Mar 26, 2026, 03:46 PM Planned orchestration promotion [v2.0.0-preview.162](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview162 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **dataders**Mar 23, 2026, 04:19 PM Automated promotion [v2.0.0-preview.161](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview161 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **james-durand-dbt**Mar 24, 2026, 04:22 PM Planned orchestration promotion [v2.0.0-preview.160](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview160 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **dataders**Mar 22, 2026, 07:14 PM Automated promotion [v2.0.0-preview.159](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview159 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Mar 20, 2026, 12:15 PM Automated promotion [v2.0.0-preview.158](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview158 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **kylepeirce**Mar 20, 2026, 03:50 PM Planned orchestration promotion [v2.0.0-preview.157](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview157 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Mar 18, 2026, 08:35 PM Automated promotion [v2.0.0-preview.156](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview156 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **kczimm**Mar 17, 2026, 08:55 PM Automated promotion [v2.0.0-preview.155](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview155 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Mar 16, 2026, 12:20 PM Automated promotion [v2.0.0-preview.154](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview154 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Mar 26, 2026, 09:14 PM Automated ST promotion [v2.0.0-preview.153](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview153 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **kylepeirce**Mar 12, 2026, 10:37 PM Bug fix for INC-5687 [v2.0.0-preview.152](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview152 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Mar 12, 2026, 12:13 PM Automated promotion [v2.0.0-preview.151](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview151 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **kylepeirce**Mar 12, 2026, 06:52 PM Planned orchestration promotion [v2.0.0-preview.150](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview150 "View this release in the dbt v2 changelog")GoodDev Released by: **kczimm**Mar 10, 2026, 10:53 PM Automated promotion [v2.0.0-preview.149](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview149 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Mar 10, 2026, 03:29 PM Planned orchestration promotion [v2.0.0-preview.148](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview148 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Mar 12, 2026, 10:10 PM Automated ST promotion [v2.0.0-preview.147](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview147 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **kczimm**Mar 7, 2026, 02:04 AM Automated promotion [v2.0.0-preview.146](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview146 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **kczimm**Mar 5, 2026, 11:58 PM Automated promotion [v2.0.0-preview.145](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview145 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **venkaa28**Mar 6, 2026, 04:22 AM INC-5500 [v2.0.0-preview.144](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview144 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Mar 4, 2026, 08:41 PM retrying scheduled promo for vscode ext [v2.0.0-preview.143](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview143 "View this release in the dbt v2 changelog")Known BadDevCanaryLatestST Monday Released by: **akbog**Mar 3, 2026, 03:43 PM inc-5454-minor-table-not-found-in-schema-errors [v2.0.0-preview.142](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview142 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Feb 27, 2026, 12:02 PM Automated promotion [v2.0.0-preview.141](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview141 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Feb 26, 2026, 10:31 PM Automated promotion [v2.0.0-preview.139](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview139 "View this release in the dbt v2 changelog")GoodDev Released by: **jasonlin45**Feb 26, 2026, 06:47 PM Automated promotion [v2.0.0-preview.137](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview137 "View this release in the dbt v2 changelog")GoodDev Released by: **jasonlin45**Feb 26, 2026, 01:55 AM Automated promotion [v2.0.0-preview.135](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview135 "View this release in the dbt v2 changelog")GoodDev Released by: **chayac**Feb 25, 2026, 12:45 AM Automated promotion [v2.0.0-preview.134](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview134 "View this release in the dbt v2 changelog")GoodDev Released by: **chayac**Feb 24, 2026, 10:42 PM Automated promotion [v2.0.0-preview.127](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview127 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Feb 20, 2026, 12:17 PM Automated promotion [v2.0.0-preview.126](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview126 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **mikaylacrawford**Feb 20, 2026, 07:38 PM Planned orchestration promotion [v2.0.0-preview.125](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview125 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue\[bot]**Feb 19, 2026, 06:05 PM Automated promotion [v2.0.0-preview.123](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview123 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Feb 18, 2026, 12:06 PM Automated promotion [v2.0.0-preview.121](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview121 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Feb 16, 2026, 12:09 PM Automated promotion [v2.0.0-preview.120](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview120 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Mar 5, 2026, 10:13 PM Automated ST promotion [v2.0.0-preview.119](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview119 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **dataders**Feb 13, 2026, 06:48 PM Automated promotion [v2.0.0-preview.118](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview118 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Feb 13, 2026, 12:15 PM Automated promotion [v2.0.0-preview.117](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview117 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jzhu13**Feb 13, 2026, 12:45 AM Automated promotion [v2.0.0-preview.116](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview116 "View this release in the dbt v2 changelog")GoodDev Released by: **jzhu13**Feb 12, 2026, 07:36 PM Automated promotion [v2.0.0-preview.114](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview114 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Feb 26, 2026, 10:15 PM Automated ST promotion [v2.0.0-preview.110](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview110 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Feb 6, 2026, 07:34 PM INC-5104 [v2.0.0-preview.108](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview108 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Feb 5, 2026, 04:42 PM Planned orchestration promotion [v2.0.0-preview.105](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview105 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Jan 30, 2026, 01:12 PM Automated promotion [v2.0.0-preview.104](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview104 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Feb 12, 2026, 10:13 PM Automated ST promotion [v2.0.0-preview.103](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview103 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Jan 27, 2026, 01:09 PM Automated promotion [v2.0.0-preview.102](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview102 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Jan 27, 2026, 08:06 PM Planned orchestration promotion [v2.0.0-preview.101](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview101 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Jan 29, 2026, 10:12 PM Automated ST promotion [v2.0.0-preview.100](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview100 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Jan 21, 2026, 01:42 PM Automated promotion [v2.0.0-preview.99](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview99 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **akbog**Jan 21, 2026, 07:10 AM Automated promotion [v2.0.0-preview.98](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview98 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Jan 19, 2026, 01:22 PM Automated promotion [v2.0.0-preview.97](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview97 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Jan 16, 2026, 01:14 PM Automated promotion [2.0.0-preview.97](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview97 "View this release in the dbt v2 changelog")Known Bad [v2.0.0-preview.96](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview96 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Jan 22, 2026, 10:10 PM Automated ST promotion [v2.0.0-preview.95](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview95 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Jan 15, 2026, 06:16 AM Automated promotion [v2.0.0-preview.94](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview94 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Jan 13, 2026, 07:26 PM Planned orchestration promotion [v2.0.0-preview.93](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview93 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Jan 9, 2026, 01:24 PM Automated promotion [v2.0.0-preview.92](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview92 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Jan 15, 2026, 10:10 PM Automated ST promotion [v2.0.0-preview.91](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview91 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Dec 22, 2025, 01:09 PM Automated promotion [v2.0.0-preview.90](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview90 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Dec 19, 2025, 01:04 PM Automated promotion [v2.0.0-preview.89](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview89 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **laconc**Dec 24, 2025, 12:06 AM st release [v2.0.0-preview.88](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview88 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Dec 18, 2025, 04:20 PM Planned orchestration promotion [v2.0.0-preview.87](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview87 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue\[bot]**Dec 16, 2025, 01:54 PM Automated promotion [v2.0.0-preview.86](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview86 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **james-durand-dbt**Dec 15, 2025, 08:35 PM INC-4737 [v2.0.0-preview.85](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview85 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Dec 18, 2025, 10:10 PM Automated ST promotion [v2.0.0-preview.84](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview84 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **james-durand-dbt**Dec 12, 2025, 06:04 PM Planned orchestration promotion [v2.0.0-preview.83](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview83 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Dec 10, 2025, 04:19 PM Planned orchestration promotion [v2.0.0-preview.82](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview82 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **xuliangs**Dec 9, 2025, 07:41 PM Automated promotion [v2.0.0-preview.81](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview81 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **ddk-dbt**Dec 9, 2025, 03:21 PM Automated promotion [v2.0.0-preview.80](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview80 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **venkaa28**Dec 8, 2025, 07:34 PM Automated promotion [v2.0.0-preview.79](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview79 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Dec 5, 2025, 03:18 PM Automated promotion [v2.0.0-preview.78](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview78 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **ddk-dbt**Dec 4, 2025, 03:24 PM Automated promotion [v2.0.0-preview.77](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview77 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **ddk-dbt**Dec 3, 2025, 10:26 PM Automated promotion [v2.0.0-preview.76](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview76 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **venkaa28**Dec 4, 2025, 01:36 AM preview.77 causing issues for IA. rolling back for incident. [v2.0.0-preview.75](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview75 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Nov 22, 2025, 04:09 AM Automated promotion [v2.0.0-preview.74](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview74 "View this release in the dbt v2 changelog")GoodDev Released by: **ddk-dbt**Nov 21, 2025, 03:05 PM Automated promotion [v2.0.0-preview.73](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview73 "View this release in the dbt v2 changelog")GoodDev Released by: **ddk-dbt**Nov 20, 2025, 03:19 PM Automated promotion [v2.0.0-preview.72](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview72 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Dec 11, 2025, 10:11 PM Automated ST promotion [v2.0.0-preview.71](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview71 "View this release in the dbt v2 changelog")GoodDev Released by: **venkaa28**Nov 19, 2025, 02:06 AM Automated promotion [v2.0.0-preview.70](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview70 "View this release in the dbt v2 changelog")GoodDev Released by: **ddk-dbt**Nov 17, 2025, 03:03 PM Automated promotion [v2.0.0-preview.69](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview69 "View this release in the dbt v2 changelog")GoodDev Released by: **ChenyuLInx**Nov 15, 2025, 06:57 AM Automated promotion [v2.0.0-preview.68](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview68 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **dbtlabs007**Nov 18, 2025, 12:44 AM Planned orchestration promotion [v2.0.0-preview.67](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview67 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **laconc**Nov 20, 2025, 10:47 PM Thursday ST release, manual run [v2.0.0-preview.66](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview66 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **mishamsk**Nov 12, 2025, 02:09 PM Automated promotion [v2.0.0-preview.65](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview65 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Nov 12, 2025, 10:00 PM Planned orchestration promotion [v2.0.0-preview.63](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview63 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **ddk-dbt**Nov 13, 2025, 11:05 PM scheduled promotion [v2.0.0-preview.62](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview62 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jzhu13**Nov 5, 2025, 11:18 PM Automated promotion [v2.0.0-preview.61](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview61 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Nov 5, 2025, 02:44 PM Automated promotion [v2.0.0-preview.60](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview60 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Nov 5, 2025, 08:15 PM Planned orchestration promotion [v2.0.0-preview.59](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview59 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Nov 5, 2025, 02:50 PM Planned orchestration promotion [v2.0.0-preview.58](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview58 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Nov 3, 2025, 02:39 PM Automated promotion [v2.0.0-preview.57](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview57 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Nov 6, 2025, 10:09 PM Automated ST promotion [v2.0.0-preview.56](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview56 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Oct 31, 2025, 03:24 PM Planned orchestration promotion [v2.0.0-preview.55](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview55 "View this release in the dbt v2 changelog")GoodDev Released by: **dataders**Oct 30, 2025, 07:08 PM Automated promotion [v2.0.0-preview.54](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview54 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Oct 30, 2025, 03:24 PM Automated promotion [v2.0.0-preview.53](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview53 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Oct 30, 2025, 04:47 PM Planned orchestration promotion [v2.0.0-preview.52](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview52 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Oct 29, 2025, 02:37 PM Automated promotion [v2.0.0-preview.51](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview51 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **dataders**Oct 29, 2025, 01:13 AM Automated promotion [v2.0.0-preview.50](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview50 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Oct 28, 2025, 06:46 PM Planned orchestration promotion [v2.0.0-preview.49](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview49 "View this release in the dbt v2 changelog")GoodDev Released by: **jzhu13**Oct 22, 2025, 06:12 PM Automated promotion [v2.0.0-preview.48](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview48 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST Monday Released by: **fa-assistant**Oct 27, 2025, 06:13 PM Automated promotion [v2.0.0-preview.47](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview47 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 22, 2025, 09:27 PM Planned orchestration promotion [v2.0.0-preview.45](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview45 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST Wednesday Released by: **johnchappelledbt**Oct 22, 2025, 10:16 PM 10-22-25 Wed ST release [v2.0.0-preview.44](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview44 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **dbtlabs007**Oct 9, 2025, 07:39 PM inc-3966-informational-releasing-dbt-lsp-hotfix-to-fix-hang-coalesce-thaw [v2.0.0-preview.43](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview43 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 9, 2025, 03:56 PM fs deployment freeze promotion [v2.0.0-preview.42](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview42 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Oct 8, 2025, 02:26 PM Automated promotion [v2.0.0-preview.41](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview41 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **ddk-dbt**Oct 8, 2025, 09:45 PM pre-coalesce sync [v2.0.0-preview.40](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview40 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **davidharting**Oct 7, 2025, 09:56 PM Automated promotion [v2.0.0-preview.39](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview39 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Oct 7, 2025, 02:33 PM Automated promotion [v2.0.0-preview.38](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview38 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 7, 2025, 04:19 PM orc release [v2.0.0-preview.37](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview37 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue\[bot]**Oct 6, 2025, 02:40 PM Automated promotion [v2.0.0-preview.36](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview36 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Oct 3, 2025, 02:21 PM Automated promotion [v2.0.0-preview.35](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview35 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST Monday Released by: **fa-assistant**Oct 6, 2025, 06:13 PM Automated promotion [v2.0.0-preview.34](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview34 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 3, 2025, 03:58 PM Planned orchestration promotion [v2.0.0-preview.33](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview33 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 2, 2025, 06:23 PM Planned orchestration promotion [v2.0.0-preview.32](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview32 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Oct 1, 2025, 03:21 PM Planned orchestration promotion [v2.0.0-preview.31](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview31 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue\[bot]**Sep 30, 2025, 02:44 PM Automated promotion [v2.0.0-preview.30](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview30 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Sep 30, 2025, 04:01 PM Planned orchestration promotion [v2.0.0-preview.29](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview29 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **ddk-dbt**Oct 3, 2025, 03:26 PM Automated promotion [v2.0.0-preview.28](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview28 "View this release in the dbt v2 changelog")GoodDev Sep 25, 2025, 02:08 PM [v2.0.0-preview.27](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview27 "View this release in the dbt v2 changelog")Known Bad [v2.0.0-preview.26](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview26 "View this release in the dbt v2 changelog")GoodCanary Sep 25, 2025, 03:16 PM [v2.0.0-preview.25](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview25 "View this release in the dbt v2 changelog")GoodST MondayST WednesdayLatestST Thursday Sep 25, 2025, 10:03 PM [v2.0.0-preview.23](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview23 "View this release in the dbt v2 changelog")Known Bad [v2.0.0-preview-nightly.176](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200-preview-nightly176 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **akbog**May 12, 2026, 08:21 PM Rollback incorrect release of 2.0.0 [v2.0.0](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md#200 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystable Released by: **itsnamangoyal**Sep 14, 2026, 12:48 PM planned promotion of v2.0.0 to stable --- ### Supported features Learn about the features supported by dbt v2, including requirements and limitations. When you install dbt, you get v2 by default. There's no separate feature set to choose between — v2 is just dbt, running faster, with more capability built in. #### Requirements To use v2 in your project you must: * Use a supported adapter and authentication method:  BigQuery * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks * Service Account / User Token * Native OAuth  Redshift * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  ClickHouse[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password  Apache Spark (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Thrift * Simple Authentication and Security Layer (SASL) PLAIN * No SASL (NOSASL) * Livy * Basic authentication (username and password) * When deployed on Amazon Web Services (AWS): AWS Signature Version 4 * Supports authentication using single sign-on, service accounts, or user tokens  DuckDB (CLI only) DuckDB does not require authentication — it runs locally on your machine. *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* * Be able to run your project on the latest version of v1 with no deprecation warnings or errors. * Migrate your Semantic Layer configurations to the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). #### Parity with dbt v1 dbt v2 supports nearly all of v1.x's capabilities today. Refer to [Limitations](#limitations) below for the small number of gaps that remain. v2 has also removed some deprecated features and introduced more rigorous validation of erroneous project code compared to v1.x. Refer to the [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) for details. #### Features and capabilities dbt v2 gives your team faster development workflows with semantic and syntax error detection, a faster linter, column-level lineage, language server and VS Code integration, docs v2 (full), and data diff. The dbt VS Code extension adds editor features like IntelliSense, hover info, and inline errors on top, powered by the LSP. Most v2 features work right away, with no login required. A few dbt platform-based capabilities become available once you sign in with a dbt platform account — free to create, no paid plan needed. For the full free-vs-login breakdown, refer to [v2 feature availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md). For LSP features specifically, refer to [About dbt LSP](https://docs.getdbt.com/docs/about-dbt-lsp.md). tip dbt platform [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) (like [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md), [dbt Mesh](https://docs.getdbt.com/docs/mesh/about-mesh.md), and more) are the enterprise layer on top of v2 — available no matter how you run dbt, depending on your [dbt plan](https://www.getdbt.com/pricing). #### Limitations If your project uses any of the following, you can still use v2, but full migration may not be possible yet: * Models that rely on materialization features v2 doesn't fully support, or that need configurations it's still missing * Tooling that depends on v1.x's exact log output * Workflows built around dbt platform features v2 doesn't yet support, like model-level notifications * Using the dbt VS Code extension in Cursor's Agent mode — lineage visualization only renders in Editor mode, so switch there if you need the full lineage tab * [Linting using SQLFluff](https://docs.getdbt.com/docs/deploy/continuous-integration.md#to-configure-sqlfluff-linting) — You use SQLFluff for linting in CI or local development. SQLFluff is not natively compatible with dbt v2 but we provide a [SQLFluff compatible high performance alternative](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) with dbt v2. Support varies by where you run it: * **dbt platform CI jobs**: CI jobs on a dbt v2 version invoke [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) instead of SQLFluff, so results can differ. Refer to [Rule parity with SQLFluff](https://docs.getdbt.com/reference/commands/lint.md?version=2.0#rule-parity-with-sqlfluff). * **Studio IDE**: SQLFluff linting works, but uses the dbt v1 engine templater rather than dbt v2. * **Local development**: You can run SQLFluff locally using the standalone dbt v1 engine templater as a workaround. A native dbt v2 linter is available with [`dbt lint` command](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) #### Package support To determine if a package is compatible with dbt v2, visit the [dbt package hub](https://hub.getdbt.com/) and look for the dbt v2-compatible badge, or review the package's [`require-dbt-version` configuration](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md#pin-to-a-range). * Packages with a `require-dbt-version` that equals or contains `2.0.0` are compatible with dbt v2. For example, `require-dbt-version: ">=1.10.0,<3.0.0"`. Even if a package doesn't reflect compatibility in the package hub, it may still work with v2. Work with package maintainers to track updates, and [thoroughly test packages](https://docs.getdbt.com/guides/dbt-package-compat?step=5) that aren't clearly compatible before deploying. * Package maintainers who would like to make their package compatible with v2 can refer to the [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) for instructions. Fivetran package considerations: * The Fivetran `source` and `transformation` packages have been combined into a single package. * If you manually installed source packages like `fivetran/github_source`, you need to ensure `fivetran/github` is installed and deactivate the transformation models. ###### Package compatibility messages Inconsistent v2 warnings and `dbt-autofix` logs dbt v2 warnings and `dbt-autofix` logs may show different messages about package compatibility. If you use [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) while upgrading to v2 in the Studio IDE or dbt VS Code extension, you may see different messages about package compatibility between `dbt-autofix` and v2 warnings. Here's why: * dbt v2 warnings are emitted based on a package's `require-dbt-version` and whether `require-dbt-version` contains `2.0.0`. * Some packages are already v2-compatible even though package maintainers haven't yet updated `require-dbt-version`. * `dbt-autofix` knows about these compatible packages and will not try to upgrade a package that it knows is already compatible. This means that even if you see a v2 warning for a package that `dbt-autofix` identifies as compatible, you don't need to change the package. The message discrepancy is temporary while we implement and roll out `dbt-autofix`'s enhanced compatibility detection to v2 warnings. Here's an example of a v2 warning in the Studio IDE that says a package isn't compatible with v2 but `dbt-autofix` indicates it is compatible: ```text dbt1065: Package 'dbt_utils' requires dbt version [>=1.30,<2.0.0], but current version is 2.0.0. This package may not be compatible with your dbt version. dbt(1065) [Ln 1, Col 1] ``` #### More information about dbt v2 * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ## Getting Started ### dbt platform configuration checklist dbt platform So, you've created a new cloud-hosted dbt platform account, and you're ready to explore its lightning-fast and intuitive features. Welcome! Before you begin, let’s ensure your account is properly configured so that you can easily onboard new users and take advantage of all the integrations dbt has to offer. For most organizations, this will require some collaboration with IT and/or security teams. Depending on the features you're using, you may need some of the following admin personas to help you get set up: * Data warehouse (Snowflake, BigQuery, Databricks, etc.) * Access control (Okta, Entra ID, Google, SAML 2.0) * Git (GitHub, GitLab, Azure DevOps, etc.) This checklist ensures you have everything in the right place, allowing you to deploy quickly and without any bottlenecks. #### Data warehouse The dbt platform supports [global connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management) for your data warehouses. This means that a single configured connection can be used across multiple projects and environments. The dbt platform supports multiple data warehouse connections, including (but not limited to) BigQuery, Databricks, Redshift, and Snowflake. One of the earliest account configuration steps you'll want to take is ensuring you have a working connection: * \[ ] Use the [connection set up documentation](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) to configure the data warehouse connection of your choice. * \[ ] Verify that dbt developers have proper roles and access in your data warehouse(s). * \[ ] Be sure the data warehouse has real data you can reference. This can be production or development data. We have a sandbox e-commerce project called [The Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) that you can use if you prefer. The Jaffle Shop includes mock data and ready-to-run models! * \[ ] Whether starting a brand new project or importing an existing dbt v1 project, you'll want to make sure you have the [proper structure configured](https://docs.getdbt.com/docs/build/projects.md). * \[ ] If you are migrating from dbt v1, there are some important things you'll need to know, so check out our [migration guide](https://docs.getdbt.com/guides/dbt-migration-2.md?step=1). * \[ ] Your users will need to [configure their credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) to connect to the development environment in the dbt Studio IDE. * \[ ] Ensure that all users who need access to work in the IDE have a [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) assigned in your account. * \[ ] dbt models are primarily written as [SELECT statements](https://docs.getdbt.com/docs/build/sql-models.md), so an early step for measuring success is having a developer run a simple select statement in the IDE and validating the results. * \[ ] You can also verify the connection by running basic SQL queries using [dbt Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md). * \[ ] Create a single model and ensure that you can [run it](https://docs.getdbt.com/reference/dbt-commands.md) successfully. * \[ ] For an easy to use drag-and-drop interface, try creating it with [dbt Canvas](https://docs.getdbt.com/docs/platform/canvas.md). * \[ ] Create a service account with proper access for your [production jobs](https://docs.getdbt.com/docs/deploy/jobs.md). #### Git configuration Git is, for many dbt environments, the backbone of your project. Git repositories are where your dbt files will live and where your developers will collaborate and manage version control of your project. * \[ ] Configure a [Git repository](https://docs.getdbt.com/docs/platform/git/configure-git.md) for your account. dbt supports integrations with: * [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md) * [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) * [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) * Other providers using [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) * If you aren't ready to integrate with an existing Git solution, dbt can provide you with a [managed Git repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md). * \[ ] Ensure developers can [checkout](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#git-overview) a new branch in your repo. * \[ ] Ensure developers in the IDE can [commit changes](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#basic-layout). #### Environments and jobs [Environments](https://docs.getdbt.com/docs/environments-in-dbt.md) separate your development data from your production data. dbt supports two environment types: Development and Deployment. There are three types of deployment environments: * Production - One per project * Staging - One per project * General - Multiple per project Additionally, you will have only one `Development` environment per project, but each developer will have their own unique access to the IDE, separate from the work of other developers. [Jobs](https://docs.getdbt.com/docs/deploy/jobs.md) dictate which commands are run in your environments and can be triggered manually, on a schedule, by other jobs, by APIs, or when pull requests are committed or merged. Once you connect your data warehouse and complete the Git integration, you can configure environments and jobs: * \[ ] Start by creating a new [Development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment) for your project. * \[ ] Create a [Production Deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md). * \[ ] (Optional) Create an additional Staging or General environment. * \[ ] [Create and schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) a deployment job. * \[ ] Validate the job by manually running it first. * \[ ] If needed, configure different databases for your environments. #### User access The dbt platform offers a variety of access control tools that you can leverage to grant or revoke user access, configure RBAC, and assign user licenses and permissions. * \[ ] Manually [invite users](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) to the dbt platform, and they can authenticate using [MFA (SMS or authenticator app)](https://docs.getdbt.com/docs/platform/manage-access/mfa.md). * \[ ] Configure [single sign-on or OAuth](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for advanced access control. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") accounts only. * \[ ] Create [SSO mappings](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#sso-mappings-) for groups * Configure [System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim.md) if available for your IdP. * \[ ] Ensure invited users are able to connect to the data warehouse from their personal profile. * \[ ] [Create groups](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#create-new-groups-) with granular permission sets assigned. * \[ ] Create [RBAC rules](https://docs.getdbt.com/docs/platform/manage-access/about-user-access#role-based-access-control-) to assign users to groups and permission sets upon sign in. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") accounts only. * \[ ] Enforce SSO for all non-admin users, and MFA is enforced for all password-based logins. #### Continue the journey Once you've completed this checklist, you're ready to start your dbt platform journey, but that journey has only just begun. Explore these additional resources to support you along the way: * \[ ] Review the [guides](https://docs.getdbt.com/guides.md) for quickstarts to help you get started with projects and features. * \[ ] Take a [dbt Learn](https://learn.getdbt.com/catalog) hands-on course. * \[ ] Review our [best practices](https://docs.getdbt.com/best-practices.md) for practical advice on structuring and deploying your dbt projects. * \[ ] Become familiar with the [references](https://docs.getdbt.com/reference/references-overview.md), as they are the product dictionary and offer detailed implementation examples. --- ### dbt Quickstarts Begin your dbt journey by choosing how you want to develop: * [**dbt platform** ](#the-dbt-platform)— Develop in your browser (Studio IDE or Canvas) or use local tools (VS Code extension, dbt platform CLI) that connect to your platform account. The platform provides hosted CI/CD, documentation, and more. Supports both the [dbt v2](https://docs.getdbt.com/docs/introduction.md) and [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) engines. * [**Self-hosted only**](#self-hosted-dbt-installations) — Use local tools like the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) or [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) locally, to develop and run dbt on your own infrastructure. You can use local tools with or without a dbt platform account. * **Local + dbt platform** — Use the VS Code extension or dbt platform CLI with a dbt platform account to develop locally while leveraging platform features like CI/CD, documentation hosting, Insights, Canvas, and more. * [**dbt Wizard**](#dbt-wizard) — The AI agent for analytics engineering, available in the dbt platform and from your terminal. Grounded in your project's lineage, model health, and semantic definitions. #### The dbt platform dbt provides a fully managed environment to develop, run, and deploy dbt projects—with CI/CD, documentation hosting, and more. Learn more about [dbt features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) and [start your free trial](https://www.getdbt.com/signup/) today. dbt v2 adds managed execution and a unified development experience so you can focus on building rather than infrastructure. Choose your warehouse to get started with a quickstart: [![](/img/icons/athena.svg)](https://docs.getdbt.com/guides/athena) ###### [Quickstart for dbt and Amazon Athena](https://docs.getdbt.com/guides/athena) [Integrate dbt with Amazon Athena for your data transformations.](https://docs.getdbt.com/guides/athena) [![](/img/icons/azure-synapse-analytics-2.svg)](https://docs.getdbt.com/guides/azure-synapse-analytics) ###### [Quickstart for dbt and Azure Synapse Analytics](https://docs.getdbt.com/guides/azure-synapse-analytics) [Discover how to integrate dbt with Azure Synapse Analytics for your data transformations.](https://docs.getdbt.com/guides/azure-synapse-analytics) [![](/img/icons/bigquery.svg)](https://docs.getdbt.com/guides/bigquery) ###### [Quickstart for dbt and BigQuery](https://docs.getdbt.com/guides/bigquery) [Discover how to leverage dbt with BigQuery to streamline your analytics workflows.](https://docs.getdbt.com/guides/bigquery) [![](/img/icons/databricks.svg)](https://docs.getdbt.com/guides/databricks) ###### [Quickstart for dbt and Databricks](https://docs.getdbt.com/guides/databricks) [Learn how to integrate dbt with Databricks for efficient data processing and analysis.](https://docs.getdbt.com/guides/databricks) [![](/img/icons/fabric.svg)](https://docs.getdbt.com/guides/microsoft-fabric) ###### [Quickstart for dbt and Microsoft Fabric](https://docs.getdbt.com/guides/microsoft-fabric) [Explore the synergy between dbt and Microsoft Fabric to optimize your data transformations.](https://docs.getdbt.com/guides/microsoft-fabric) [![](/img/icons/redshift.svg)](https://docs.getdbt.com/guides/redshift) ###### [Quickstart for dbt and Redshift](https://docs.getdbt.com/guides/redshift) [Learn how to connect dbt to Redshift for more agile data transformations.](https://docs.getdbt.com/guides/redshift) [![](/img/icons/snowflake.svg)](https://docs.getdbt.com/guides/snowflake) ###### [Quickstart for dbt and Snowflake](https://docs.getdbt.com/guides/snowflake) [Unlock the full potential of using dbt with Snowflake for your data transformations.](https://docs.getdbt.com/guides/snowflake) [![](/img/icons/starburst.svg)](https://docs.getdbt.com/guides/starburst-galaxy) ###### [Quickstart for dbt and Starburst Galaxy](https://docs.getdbt.com/guides/starburst-galaxy) [Leverage dbt with Starburst Galaxy to enhance your data transformation workflows.](https://docs.getdbt.com/guides/starburst-galaxy) [![](/img/icons/teradata.svg)](https://docs.getdbt.com/guides/teradata) ###### [Quickstart for dbt and Teradata](https://docs.getdbt.com/guides/teradata) [Discover and use dbt with Teradata to enhance your data transformation workflows.](https://docs.getdbt.com/guides/teradata) #### Self-hosted dbt installations When you install dbt into your Windows, macOS, or Linux environment, you get command-line tools and the VS Code extension that enable you to transform data using analytics engineering best practices. You can use self-hosted tools with or without a dbt platform account. With an account, the VS Code extension and dbt platform CLI sync with your platform project for CI/CD, documentation, and more. Without an account, you run dbt entirely on your own infrastructure. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/dbt.md?step=2) ###### [Self-hosted dbt v2 from a manual install](https://docs.getdbt.com/guides/dbt.md?step=2) [Learn how to install self-hosted v2 and set up a project.](https://docs.getdbt.com/guides/dbt.md?step=2) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/manual-install.md) ###### [Self-hosted dbt v1 from a manual install](https://docs.getdbt.com/guides/manual-install.md) [Learn how to install self-hosted v1 and set up a project.](https://docs.getdbt.com/guides/manual-install.md) [![](/img/icons/duckdb-seeklogo.svg)](https://docs.getdbt.com/guides/duckdb.md?step=1) ###### [Quickstart for dbt with DuckDB](https://docs.getdbt.com/guides/duckdb.md?step=1) [Learn how to connect dbt to DuckDB.](https://docs.getdbt.com/guides/duckdb.md?step=1) #### Connect your AI to your data You don't have to choose between learning dbt and using AI — you can absolutely pair them from day one: * Use the dbt Wizard and ask it to build, refactor, or document a model in natural language, grounded in your project's real lineage and tests. [Get started with dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). * Set up the dbt MCP server to connect to Claude, Cursor, or another AI tool to query your project, run dbt commands, and pull metrics directly. [Connect the dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md). * Connect to the Fivetran context layer to give any AI tool richer context about your data, so its answers are grounded in what your data actually means. [Explore the context layer](https://fivetran.com/docs/context-layer). #### Related docs Expand your dbt knowledge and expertise with these additional resources: * [Join the monthly demos](https://www.getdbt.com/resources/webinars/dbt-cloud-demos-with-experts) to see dbt in action and ask questions. * [dbt AWS marketplace](https://aws.amazon.com/marketplace/pp/prodview-tjpcf42nbnhko) contains information on how to deploy dbt on AWS, user reviews, and more. * [Best practices](https://docs.getdbt.com/best-practices.md) contains information on how dbt Labs approaches building projects through our current viewpoints on structure, style, and setup. * [dbt Learn](https://learn.getdbt.com) offers free online courses that cover dbt fundamentals, advanced topics, and more. * [Join the dbt Community](https://www.getdbt.com/community/join-the-community) to learn how other data practitioners globally are using dbt, share your own experiences, and get help with your dbt projects. --- ### What is dbt? dbt transforms raw warehouse data into trusted data products. You write simple SQL select statements, and dbt handles the heavy lifting by creating modular, maintainable data models that power analytics, operations, and AI -- replacing the need for complex and fragile transformation code. dbt is the industry standard for data transformation, helping teams work faster and produce higher-quality data. As you build in dbt, your project creates structured context — lineage, tests, contracts, metrics, and governance — that explains how your data connects, what it means, and what changes may affect. That context makes dbt especially powerful for AI and comes with features like [dbt Wizard](#dbt-wizard), which helps you investigate, build, validate, and ship with full project context and governance on by default. You can use dbt and its [framework](#dbt-framework) to: * Centralize and modularize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. * Collaborate on data models to safely deploy and monitor data transformations in production. * Apply software engineering best practices like version control, testing, modularity, CI/CD, and documentation to analytics workflows. * Build idempotent transformations that are safe to rerun and produce consistent results. Learn more about [Idempotence in dbt](https://docs.getdbt.com/best-practices/idempotence.md). Backed by a 100,000+ member [community](https://docs.getdbt.com/community/join.md), dbt helps teams build high-quality, trustworthy data pipelines faster. [![dbt works alongside your ingestion, visualization, and other data tools, so you can transform data directly in your cloud data platform.](/img/docs/platform-overview.jpg?v=2 "dbt works alongside your ingestion, visualization, and other data tools, so you can transform data directly in your cloud data platform.")](#)dbt works alongside your ingestion, visualization, and other data tools, so you can transform data directly in your cloud data platform. Read more about why we want to enable analysts to work more like software engineers in [The dbt Viewpoint](https://docs.getdbt.com/community/resources/viewpoint.md). Learn how other data practitioners around the world are using dbt by [joining the dbt Community](https://www.getdbt.com/community/join-the-community). #### dbt framework Use the dbt framework to quickly and collaboratively transform data and deploy analytics code following software engineering best practices like version control, modularity, portability, CI/CD, and documentation. This means anyone on the data team familiar with SQL can safely contribute to production-grade data pipelines. The dbt framework is composed of a *language* and an *engine*: * The *dbt language* is the code you write in your dbt project — SQL select statements, Jinja templating, YAML configs, tests, and more. It's the standard for the data industry and the foundation of the dbt framework. * The *dbt engine* compiles your project, executes your transformation graph, and produces metadata. Today, the current Rust-based version generation is v2. By default, installing dbt gives you SQL comprehension, editor features, and richer development workflows. ##### dbt versions dbt has two major versions: v1 and v2. **v2** is the current generation of dbt and the default when you [install it](https://docs.getdbt.com/docs/local/install-dbt.md). Installing dbt comes with richer developer tooling, linting, and more. Refer to the [Upgrade v2 guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) for more info. | Version | What it is | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **v2**<br />*Recommended* | The current Rust-based generation of dbt, built for the full modern development experience and powered by an open-source runtime.<br /><br />Many features work right away. Some dbt platform-based capabilities are available with a free sign-in — see [v2 feature availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md) for the full breakdown or [upgrade to v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). | | **v1** | The original Python-based generation of dbt, still maintained as dbt v1. | Refer to the [Licensing FAQs](https://www.getdbt.com/licenses-faq) for more info. #### The dbt engine The current generation of dbt is written in Rust with a native understanding of SQL across multiple engine dialects. That comprehension lets dbt catch errors before they reach your warehouse and powers editor features like autocomplete and inline errors as you type. v2 is the default experience when you [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). It builds on the Apache 2.0 open-source runtime CLI foundation for the dbt framework. dbt is free to use, with some dbt platform-based capabilities available when you sign in with any dbt platform account. ##### Enhance your development workflows As a developer, dbt can: * Immediately catch incorrect SQL in your dbt models, before they ever hit the warehouse * Give you autocomplete, hover info, and inline errors as you type * Preview inline CTEs for faster debugging * Trace model and column definitions across your entire project Get all of this, free, in the [dbt extension for VS Code](https://docs.getdbt.com/docs/about-dbt-extension.md), built to enhance the power of v2, but also available in v1. #### How to use dbt You can use dbt in different ways depending on your needs: * [With the dbt platform](#dbt-platform) (recommended for most users) * [Locally from your command line or code editor](#dbt-local-development) ##### dbt platform The dbt platform is the fastest way to run dbt: scheduling, CI/CD, documentation hosting, monitoring, and alerting, all in one place. It works with both v1 and v2, on every plan from Developer (free) through Enterprise+. Develop directly in the platform with the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or connect from your local machine with the dbt VS Code extension or dbt platform CLI. Learn more about [dbt platform features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md), explore [plans and pricing](https://www.getdbt.com/pricing/), or try a [quickstart](https://docs.getdbt.com/guides.md). ##### dbt local development [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) to run v2 locally from the command line, powered by an open-source runtime. For the best development experience, we recommend pairing v2 with the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) for autocomplete, inline errors, and lineage as you work. You can also run [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) to access dbt platform-based capabilities and create a free dbt platform account. To get started quickly, try the [dbt quickstart](https://docs.getdbt.com/guides/dbt.md). Other ways to run self-hosted dbt: * [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md?version=1.0): The original Python-based CLI. * [dbt OSS](https://docs.getdbt.com/docs/local/install-dbt-v2.md): The free, fully open-source (Apache 2.0) distribution of the new Rust-based dbt engine. Typically for organizations with a strict requirement to use this OSS runtime. To contribute to the open-source project, refer to the [GitHub repo](https://github.com/dbt-labs/dbt). #### Why use dbt As a dbt user, your main focus will be on writing models (select queries) that reflect core business logic – there's no need to write boilerplate code to create tables and views, or to define the order of execution of your models. Instead, dbt handles turning these models into objects in your warehouse for you. * **No boilerplate**: Write business logic with just a SQL `select` statement or a Python DataFrame. dbt handles materialization, transactions, DDL, and schema changes. * **Modular and reusable**: Build data models that can be referenced in subsequent work. Change a model once and the change propagates to all its dependencies, so you can publish canonical business logic without reimplementing it. * **Fast builds**: Use [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) and leverage metadata to optimize long-running models. * **Tested and documented** — Write [data quality tests](https://docs.getdbt.com/docs/build/data-tests.md) on your underlying data and auto-generate [documentation](https://docs.getdbt.com/docs/build/documentation.md) alongside your code. * **Software engineering workflows**: Version control, branching, pull requests, CI/CD, and [package management](https://docs.getdbt.com/docs/build/packages.md) for your data pipelines. Write DRYer code with [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) and [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md). * **AI-powered development**: Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to investigate, build, validate, and ship from natural language. dbt Wizard is grounded in your project's full context, validates its own work against lineage and tests, and includes governance and audit trails by default. #### Related docs * [Quickstarts for dbt](https://docs.getdbt.com/guides.md) * [Best practice guides](https://docs.getdbt.com/best-practices.md) * [What is a dbt project?](https://docs.getdbt.com/docs/build/projects.md) * [Supported features matrix](https://docs.getdbt.com/docs/dbt/supported-features.md) * [AI and agents](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md) * [Licensing](https://docs.getdbt.com/docs/dbt-licensing.md) * [v2 license agreement](https://www.getdbt.com/dbt-product-license-agreement) --- ## Guides ### Airflow and dbt [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Many organizations already use [Airflow](https://airflow.apache.org/) to orchestrate their data workflows. dbt works great with Airflow, letting you execute your dbt code in dbt while keeping orchestration duties with Airflow. This ensures your project's metadata (important for tools like Catalog) is available and up-to-date, while still enabling you to use Airflow for general tasks such as: * Scheduling other processes outside of dbt runs * Ensuring that a [dbt job](https://docs.getdbt.com/docs/deploy/job-scheduler.md) kicks off before or after another process outside of dbt * Triggering a dbt job only after another has completed In this guide, you'll learn how to: 1. Create a working local Airflow environment 2. Invoke a dbt job with Airflow 3. Reuse tested and trusted Airflow code for your specific use cases You’ll also gain a better understanding of how this will: * Reduce the cognitive load when building and maintaining pipelines * Avoid dependency hell (think: `pip install` conflicts) * Define clearer handoff of workflows between data engineers and analytics engineers #### Prerequisites * [dbt platform Enterprise or Enterprise+ account](https://www.getdbt.com/pricing/) (with [admin access](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md)) in order to create a service token. Permissions for service tokens can be found [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). * Service account token if you have [Developer license (with admins access)](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md)). Permissions for service tokens can be found [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens) and [here](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md?version=2.0). * [Personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens#create-a-personal-access-token) if you have developer, IT, or Read-only license. * A [free Docker account](https://hub.docker.com/signup) in order to sign in to Docker Desktop, which will be installed in the initial setup. * A local digital scratchpad for temporarily copy-pasting API keys and URLs 🙌 Let’s get started! 🙌 #### Install the Astro CLI Astro is a managed software service that includes key features for teams working with Airflow. In order to use Astro, we’ll install the Astro CLI, which will give us access to useful commands for working with Airflow locally. You can read more about Astro [here](https://docs.astronomer.io/astro/). In this example, we’re using Homebrew to install Astro CLI. Follow the instructions to install the Astro CLI for your own operating system [here](https://docs.astronomer.io/astro/install-cli). ```bash brew install astro ``` #### Install and start Docker Desktop Docker allows us to spin up an environment with all the apps and dependencies we need for this guide. Follow the instructions [here](https://docs.docker.com/desktop/) to install Docker desktop for your own operating system. Once Docker is installed, ensure you have it up and running for the next steps. #### Clone the airflow-dbt-cloud repository Open your terminal and clone the [airflow-dbt-cloud repository](https://github.com/dbt-labs/airflow-dbt-cloud). This contains example Airflow DAGs that you’ll use to orchestrate your dbt job. Once cloned, navigate into the `airflow-dbt-cloud` project. ```bash git clone https://github.com/dbt-labs/airflow-dbt-cloud.git cd airflow-dbt-cloud ``` For more information about cloning GitHub repositories, refer to "[Cloning a repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository)" in the GitHub documentation. #### Start the Docker container 1. From the `airflow-dbt-cloud` directory you cloned and opened in the prior step, run the following command to start your local Airflow deployment: ```bash astro dev start ``` When this finishes, you should see a message similar to the following: ```bash Airflow is starting up! This might take a few minutes… Project is running! All components are now available. Airflow Webserver: http://localhost:8080 Postgres Database: localhost:5432/postgres The default Airflow UI credentials are: admin:admin The default Postgres DB credentials are: postgres:postgres ``` 2. Open the Airflow interface. Launch your web browser and navigate to the address for the **Airflow Webserver** from your output above (for us, `http://localhost:8080`). This will take you to your local instance of Airflow. You’ll need to log in with the **default credentials**: * Username: admin * Password: admin [![Airflow login screen](/img/guides/orchestration/airflow-and-dbt-platform/airflow-login.png?v=2 "Airflow login screen")](#)Airflow login screen #### Create a dbt service token [Create a service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with `Job Admin` privileges from within dbt. Ensure that you save a copy of the token, as you won’t be able to access this later. As an alternative, you can [create a PAT](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md#create-a-personal-access-token) if you have a Developer, IT, or Read-only license. #### Create a dbt job [Create a job in your dbt account](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs), paying special attention to the information in the bullets below. * Configure the job with the full commands that you want to include when this job kicks off. This sample code has Airflow triggering the dbt job and all of its commands, instead of explicitly identifying individual models to run from inside of Airflow. * Ensure that the schedule is turned **off** since we’ll be using Airflow to kick things off. * Once you hit `save` on the job, make sure you copy the URL and save it for referencing later. The url will look similar to this: ```html https://YOUR_ACCESS_URL/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ ``` #### Connect dbt to Airflow Now you have all the working pieces to get up and running with Airflow + dbt. It's time to **set up a connection** and **run a DAG in Airflow** that kicks off a dbt job. 1. From the Airflow interface, navigate to Admin and click on **Connections** [![Airflow connections menu](/img/guides/orchestration/airflow-and-dbt-platform/airflow-connections-menu.png?v=2 "Airflow connections menu")](#)Airflow connections menu 2. Click on the `+` sign to add a new connection, then click on the drop down to search for the dbt Connection Type. [![Connection type](/img/guides/orchestration/airflow-and-dbt-platform/connection-type.png?v=2 "Connection type")](#)Connection type 3. Add in your connection details and your default dbt account id. This is found in your dbt URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/` [![Connection type](/img/guides/orchestration/airflow-and-dbt-platform/connection-type-configured.png?v=2 "Connection type")](#)Connection type #### Update the placeholders in the sample code Add your `account_id` and `job_id` to the python file [run\_dbt\_cloud\_job.py](https://github.com/dbt-labs/airflow-dbt-cloud/blob/main/dags/run_dbt_cloud_job.py). Both IDs are included inside of the dbt job URL as shown in the following snippets: ```python # For the dbt Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The account_id is 16173 and the job_id is 65767 # Update lines 34 and 35 ACCOUNT_ID = "16173" JOB_ID = "65767" ``` #### Run the Airflow DAG Turn on the DAG and trigger it to run. Verify the job succeeded after running. [![Airflow DAG](/img/guides/orchestration/airflow-and-dbt-platform/airflow-dag.png?v=2 "Airflow DAG")](#)Airflow DAG Click **Monitor Job Run** to open the run details in dbt. [![Task run instance](/img/guides/orchestration/airflow-and-dbt-platform/task-run-instance.png?v=2 "Task run instance")](#)Task run instance #### Cleaning up At the end of this guide, make sure you shut down your docker container. When you’re done using Airflow, use the following command to stop the container: ```bash $ astrocloud dev stop [+] Running 3/3 ⠿ Container airflow-dbt-cloud_e3fe3c-webserver-1 Stopped 7.5s ⠿ Container airflow-dbt-cloud_e3fe3c-scheduler-1 Stopped 3.3s ⠿ Container airflow-dbt-cloud_e3fe3c-postgres-1 Stopped 0.3s ``` To verify that the deployment has stopped, use the following command: ```bash astrocloud dev ps ``` This should give you an output like this: ```bash Name State Ports airflow-dbt-cloud_e3fe3c-webserver-1 exited airflow-dbt-cloud_e3fe3c-scheduler-1 exited airflow-dbt-cloud_e3fe3c-postgres-1 exited ``` #### Frequently asked questions ##### How can we run specific subsections of the dbt DAG in Airflow? Because the Airflow DAG references dbt jobs, your analytics engineers can take responsibility for configuring the jobs in dbt. For example, to run some models hourly and others daily, there will be jobs like `Hourly Run` or `Daily Run` using the commands `dbt run --select tag:hourly` and `dbt run --select tag:daily` respectively. Once configured in dbt, these can be added as steps in an Airflow DAG as shown in this guide. Refer to our full [node selection syntax docs here](https://docs.getdbt.com/reference/node-selection/syntax.md). ##### How can I re-run models from the point of failure? You can trigger re-run from point of failure with the `rerun` API endpoint. See the docs on [retrying jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) for more information. ##### Should Airflow run one big dbt job or many dbt jobs? dbt jobs are most effective when a build command contains as many models at once as is practical. This is because dbt manages the dependencies between models and coordinates running them in order, which ensures that your jobs can run in a highly parallelized fashion. It also streamlines the debugging process when a model fails and enables re-run from point of failure. As an explicit example, it's not recommended to have a dbt job for every single node in your DAG. Try combining your steps according to desired run frequency, or grouping by department (finance, marketing, customer success...) instead. ##### We want to kick off our dbt jobs after our ingestion tool (such as Fivetran) / data pipelines are done loading data. Any best practices around that? Astronomer's DAG registry has a sample workflow combining Fivetran, dbt and Census [here](https://registry.astronomer.io/dags/fivetran-dbt_cloud-census/versions/3.0.0). ##### How do you set up a CI/CD workflow with Airflow? Check out these two resources for accomplishing your own CI/CD pipeline: * [Continuous Integration with dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Astronomer's CI/CD Example](https://docs.astronomer.io/software/ci-cd/#example-cicd-workflow) ##### Can dbt dynamically create tasks in the DAG like Airflow can? As discussed above, we prefer to keep jobs bundled together and containing as many nodes as are necessary. If you must run nodes one at a time for some reason, then review [this article](https://www.astronomer.io/blog/airflow-dbt-1/) for some pointers. ##### Can you trigger notifications if a dbt job fails with Airflow? Yes, either through [Airflow's email/slack](https://www.astronomer.io/guides/error-notifications-in-airflow/) functionality, or [dbt's notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md), which support email and Slack notifications. You could also create a [webhook](https://docs.getdbt.com/docs/deploy/webhooks.md). ##### How should I plan my dbt + Airflow implementation? Check out [this recording](https://www.youtube.com/watch?v=n7IIThR8hGk) of a dbt meetup for some tips. --- ### Analyze your data in dbt Start with a stakeholder question and analyze the data to answer that question without writing any SQL [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction As a data analyst, you play a key role in transforming complex data into trusted, actionable insights for your team. With dbt, you can use built-in, AI-powered tools to build governed data models, explore how they’re built, and even run your own analysis. In this quickstart, you’ll learn how to: * Use Catalog to browse and understand data models across both dbt and Snowflake data assets * Use Insights to run queries for exploring and validating your data * Use Canvas to visually build your own data models * Build confidence using dbt as your workspace enhanced with AI Here's more about the tools you will use on your journey: * Catalog: View your project's resources (such as models, tests, and metrics), their lineage, and query patterns to gain a better understanding of its latest production state. * Insights: Explore, validate, and query data with an intuitive, context-rich interface that bridges technical and business users by combining metadata, documentation, AI-assisted tools, and powerful querying capabilities. * Canvas: Quickly access and transform data through a visual, drag-and-drop experience and with a built-in AI for custom code generation. #### Prerequisites Before you begin, make sure: * You have access to and credentials configured for a dbt project * Your team has already run a successful dbt job, so models are built and ready * You have a a git provider connected and authenticated #### Analyst workflows Kimiko, an analyst at the Jaffle Shop, notices they've been doing a lot of new sales and wants to investigate the most critical data they have in their warehouse. **Question: A stakeholder is curious how many customers you've acquired month by month, in the last 12 months.** Kimiko wonders, "How do I find data in our project that will help me answer their question?" ##### Explore a stakeholder question She navigates to the data catalog, Catalog, by signing into dbt and clicking Catalog in the left panel. Because the question was about customers, Kimiko begins by searching for "customers" in Catalog: [![Catalog search for customers](/img/guides/analyst-qs/catalog-search.png?v=2 "Catalog search for customers")](#)Catalog search for customers She finds a "customers" model, which might be what she needs. She clicks **customers** to open the model. The description reads, “Customer overview data Mart offering key details for each unique customer, one row per customer.” Next, Kimiko selects **Columns** to see which columns this model uses. [![Columns in customers table](/img/guides/analyst-qs/columns.png?v=2 "Columns in customers table")](#)Columns in customers table She notices these columns: `customer_ID`, `customer_names`, and `first_ordered_at`. The `first_ordered_at` column stands out to Kimiko, and she wonders if she might use it to see how many customers they've acquired based on when they placed their first order. But first, she decides to interact with the data to learn more. ##### Query data in Insights From the **Customer model page** in Catalog, Kimiko selects **Analyze data** from the **Open in...** dropdown. This enables her to query data for the Customer model. Once opened, Insights contains a query poised and ready to run. [![Select Analyze data from Catalog](/img/guides/analyst-qs/catalog-analyze-data.png?v=2 "Select Analyze data from Catalog")](#)Select Analyze data from Catalog [![Open query in Insights](/img/guides/analyst-qs/query.png?v=2 "Open query in Insights")](#)Open query in Insights When Kimiko runs the query, she can look at the data underlying it. The same context she saw in Catalog she now sees in her SQL editing experience. As she looks through the data, she sees information about each customer. She also notices the `first_ordered_at` column. Kimiko wants to code the query but her SQL is a little rusty so she uses natural language in dbt Wizard: *How many new customers did we get in each month last year? I'd like to use my customer model and the first ordered at field to do this analysis.* dbt Wizard writes SQL that Kimiko decides to use: ```sql select date_trunc('month', first_ordered_at) as month, count(customer_id) as new_customers from {{ ref('customers') }} where date_part('year', first_ordered_at) = date_part('year', current_date) - 1 and customer_type = 'new' group by 1 order by 1; ``` Kimiko clicks **Replace** to move all of the SQL into her editor and replaces the original query. She runs the new query and reviews the data but decides to limit the dates using dbt Wizard once again: *Can we limit the dates to 2024?* She verifies the new filter for 2024 and reruns this query: ```sql select date_trunc('month', first_ordered_at) as month, count(customer_id) as new_customers from {{ ref('customers') }} where date_part('year', first_ordered_at) = 2024 and customer_type = 'new' group by 1 order by 1; ``` She's happy with the results and clicks **Details** to see the AI-generated report, which includes a title and description, supplied SQL, and the compiled SQL. [![Details report tab](/img/guides/analyst-qs/details.png?v=2 "Details report tab")](#)Details report tab Once she's ready to get the insight to her stakeholder, she clicks **Chart** to view the chart prefilled with the data from the **Data** tab. She adds x- and y-axis labels, such as "Month of first order" and "Total new customers" to make it more comprehensible for the final report she'll share with her stakeholder. Next, she takes a screenshot to share with them. She often comes back to this data so Kimiko decides to bookmark the page by clicking **Bookmark** in the top right. She also exports it to a CSV file. ##### Visualize results Kimiko has a few conversations with teammates and she finds out they're running pretty similar one-off queries, so she decides to take her long running query that she previously bookmarked and turn it into a full-fledged dbt model using Canvas. She does this so she can share it with others, which de-duplicates work and helps her team become more efficient. To do this, she opens the query in Insights and clicks **Develop** then \***Develop in** Canvas. This opens the SQL query in a visual form, represented in a DAG. When she examines the model, she notes it's selecting from customers as expected, filtering for 2024, showing dates by month, and aggregating over that month. She runs it in her development environment and clicks **Commit** to submit a pull request. Now Kimiko's entire team, those who have the same access as her, can run this model and see the same results she does! What's more is they can help her improve the model as the stakeholder requests get more complicated, and she will benefit from their help. ##### The query becomes a model Going forward, Kimiko is able to return to her project in Catalog and run the model to get the most current results. From here, she can: * Manually run the model, which also runs tests and is versioned so Kimiko can track changes over time * Trigger a scheduled job to run the dbt model, like every Monday for her stakeholder report * Set up a Slack notification in case the job fails so she can rectify any problems --- ### Browse our guides [Skip to main content](#__docusaurus_skipToContent_fallback) [Relive the best of dbt Summit! Get access to the keynotes and breakout sessions from dbt Summit 2026. See the latest product innovations from dbt, and hear how data teams are leveling up for the agentic era.](https://www.getdbt.com/dbt-summit/registration/online) [![dbt Logo](/img/dbt-logo.svg?v=2)![dbt Logo](/img/dbt-logo-light.svg?v=2)](https://docs.getdbt.com/index.md) * [v2]() * [v1]() * [1.13 (beta)]() * [1.12]() * [1.11]() * On dbt platform? <br /> v2 = [dbt v2 release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks?version=2.0#dbt-v2-release-tracks) <br /> v1 = [dbt v1 release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks?version=2.0#dbt-v1-release-tracks) [Docs](#) * [Product docs](https://docs.getdbt.com/docs/introduction.md) * [References](https://docs.getdbt.com/reference/references-overview.md) * [Best practices](https://docs.getdbt.com/best-practices.md) * [Developer blog](https://docs.getdbt.com/blog) [Guides](https://docs.getdbt.com/guides.md)[APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) [Help](#) * [Release notes](https://docs.getdbt.com/docs/dbt-versions/release-notes.md) * [FAQs](https://docs.getdbt.com/docs/faqs.md) * [Support and billing](https://docs.getdbt.com/docs/dbt-support.md) * [Licensing](https://docs.getdbt.com/docs/dbt-licensing.md) * [Courses](https://learn.getdbt.com) [Community](#) * [Join the dbt Community](https://docs.getdbt.com/community/join.md) * [Become a contributor](https://docs.getdbt.com/community/contribute.md) * [Community forum](https://discourse.getdbt.com/) * [Webinars](https://www.getdbt.com/resources/webinars) * [Events](https://www.getdbt.com/events) [Account](#) * [Log in to dbt](https://login.dbt.com/) * [Create a free account](https://www.getdbt.com/signup) [Get started with dbt](https://docs.getdbt.com/docs/dbt/get-started-dbt.md?version=2) Search The dbt platform is the fastest and most reliable way to deploy dbt for scalable data transformation, while dbt v1 powers open-source transformation workflows. Together, they provide a seamless analytics engineering experience. Explore our step-by-step guides, quickstart tutorials, and troubleshooting resources to get started with dbt and your data platform. Search guides... ##### Filter by ▼ Choose a topic \[ ]Adapter creation\[ ]AI\[ ]Amazon \+ View more (37)Select all Choose a level \[ ]Advanced\[ ]Beginner\[ ]Intermediate Select all Clear all ##### Get started with dbt v2 [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/dbt.md) ###### [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md) [Start ](https://docs.getdbt.com/guides/dbt.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) ###### [Upgrade to dbt v2 part 1: Preparing to upgrade](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [Start ](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/upgrade-to-v2.md) ###### [Upgrade to dbt v2 part 2: Making the move](https://docs.getdbt.com/guides/upgrade-to-v2.md) [Start ](https://docs.getdbt.com/guides/upgrade-to-v2.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) ###### [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) [Start ](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/adapter-creation-v2.md) ###### [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md) [Start ](https://docs.getdbt.com/guides/adapter-creation-v2.md) ##### Foundations [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/terminal-guide.md) ###### [Getting started with the terminal](https://docs.getdbt.com/guides/terminal-guide.md) [Start ](https://docs.getdbt.com/guides/terminal-guide.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/clone-jaffle-shop.md) ###### [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) [Start ](https://docs.getdbt.com/guides/clone-jaffle-shop.md) ##### Popular [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/dbt-charts.md) ###### [Quickstart for dbt Charts](https://docs.getdbt.com/guides/dbt-charts.md) [Start ](https://docs.getdbt.com/guides/dbt-charts.md) [![](/img/icons/snowflake.svg)](https://docs.getdbt.com/guides/snowflake.md) ###### [Quickstart for dbt and Snowflake](https://docs.getdbt.com/guides/snowflake.md) [dbt v2 compatible](https://docs.getdbt.com/guides/snowflake.md) [Start ](https://docs.getdbt.com/guides/snowflake.md) [![](/img/icons/databricks.svg)](https://docs.getdbt.com/guides/databricks.md) ###### [Quickstart for dbt and Databricks](https://docs.getdbt.com/guides/databricks.md) [dbt v2 compatible](https://docs.getdbt.com/guides/databricks.md) [Start ](https://docs.getdbt.com/guides/databricks.md) [![](/img/icons/bigquery.svg)](https://docs.getdbt.com/guides/bigquery.md) ###### [Quickstart for dbt and BigQuery](https://docs.getdbt.com/guides/bigquery.md) [dbt v2 compatible](https://docs.getdbt.com/guides/bigquery.md) [Start ](https://docs.getdbt.com/guides/bigquery.md) [![](/img/icons/redshift.svg)](https://docs.getdbt.com/guides/redshift.md) ###### [Quickstart for dbt and Redshift](https://docs.getdbt.com/guides/redshift.md) [dbt v2 compatible](https://docs.getdbt.com/guides/redshift.md) [Start ](https://docs.getdbt.com/guides/redshift.md) ##### Troubleshooting [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/debug-schema-names.md) ###### [Debug schema names](https://docs.getdbt.com/guides/debug-schema-names.md) [Start ](https://docs.getdbt.com/guides/debug-schema-names.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/using-jinja.md) ###### [Use Jinja to improve your SQL code](https://docs.getdbt.com/guides/using-jinja.md) [Start ](https://docs.getdbt.com/guides/using-jinja.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/debug-errors.md) ###### [Debug errors](https://docs.getdbt.com/guides/debug-errors.md) [Start ](https://docs.getdbt.com/guides/debug-errors.md) ##### Advanced use cases [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/iceberg.md) ###### [Get hands-on with Snowflake Horizon and Apache Iceberg](https://docs.getdbt.com/guides/iceberg.md) [60 minutesStart ](https://docs.getdbt.com/guides/iceberg.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/dbt-package-compat.md) ###### [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) [Start ](https://docs.getdbt.com/guides/dbt-package-compat.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/qs-mcp-oauth.md) ###### [Connect dbt MCP to dbt platform](https://docs.getdbt.com/guides/qs-mcp-oauth.md) [Start ](https://docs.getdbt.com/guides/qs-mcp-oauth.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/qs-mcp-local.md) ###### [Run dbt MCP locally](https://docs.getdbt.com/guides/qs-mcp-local.md) [Start ](https://docs.getdbt.com/guides/qs-mcp-local.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) ###### [Airflow and dbt](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) [Start ](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/adapter-creation.md) ###### [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) [Start ](https://docs.getdbt.com/guides/adapter-creation.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/adapter-creation-v2.md) ###### [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md) [Start ](https://docs.getdbt.com/guides/adapter-creation-v2.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/dbt-migration-1.md) ###### [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md) [Total estimated time: 3-4 hoursStart ](https://docs.getdbt.com/guides/dbt-migration-1.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/create-new-materializations.md) ###### [Create new materializations](https://docs.getdbt.com/guides/create-new-materializations.md) [Start ](https://docs.getdbt.com/guides/create-new-materializations.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/customize-schema-alias.md) ###### [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md) [Start ](https://docs.getdbt.com/guides/customize-schema-alias.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/refactoring-legacy-sql.md) ###### [Refactoring legacy SQL to dbt](https://docs.getdbt.com/guides/refactoring-legacy-sql.md) [Start ](https://docs.getdbt.com/guides/refactoring-legacy-sql.md) Get started #### Start building with dbt. The free dbt VS Code extension is the best way to develop locally with dbt. [Install free extension](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt) [Request your demo](https://www.getdbt.com/contact) [![dbt Labs](/img/dbt-logo-light.svg?v=2)](https://docs.getdbt.com/index.md) ##### Resources [VS Code Extension](https://docs.getdbt.com/docs/about-dbt-extension.md) [Resource Hub](https://www.getdbt.com/resources) [dbt Learn](https://www.getdbt.com/dbt-learn) [Certification](https://www.getdbt.com/dbt-certification) [Developer Blog](https://docs.getdbt.com/blog) ##### Community [Join the Community](https://docs.getdbt.com/community/join.md) [Become a Contributor](https://docs.getdbt.com/community/contribute.md) [Open Source dbt Packages](https://hub.getdbt.com/) [Community Forum](https://discourse.getdbt.com/) ##### Support [Contact Support](https://docs.getdbt.com/docs/dbt-support.md) [Professional Services](https://www.getdbt.com/services) [Find a Partner](https://www.getdbt.com/partner-directory) [System Status](https://status.getdbt.com/) ##### Connect with Us [](https://github.com/dbt-labs/docs.getdbt.com "GitHub") [](https://www.linkedin.com/company/dbtlabs/mycompany/ "LinkedIn") [](https://www.youtube.com/channel/UCVpBwKK-ecMEV75y1dYLE5w "YouTube") [](https://www.instagram.com/dbt_labs/ "Instagram") [](https://x.com/dbt_labs "X") [](https://bsky.app/profile/getdbt.com "Bluesky") [](https://www.getdbt.com/community/join-the-community/ "Community Slack") © 2026 dbt Labs, LLC. All Rights Reserved. [Terms of Service](https://www.getdbt.com/terms-of-use/) [Privacy Policy](https://www.getdbt.com/cloud/privacy-policy/) [Security](https://www.getdbt.com/security/) Cookie Settings --- ### Browse our guides [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() --- ### Build a data lakehouse with dbt v1 and Dremio Cloud [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide will demonstrate how to build a data lakehouse with dbt 1.5 or newer and Dremio Cloud. You can simplify and optimize your data infrastructure with dbt's robust transformation framework and Dremio’s open and easy data lakehouse. The integrated solution empowers companies to establish a strong data and analytics foundation, fostering self-service analytics and enhancing business insights while simplifying operations by eliminating the necessity to write complex Extract, Transform, and Load (ETL) pipelines. ##### Prerequisites * You must have a [Dremio Cloud](https://docs.dremio.com/cloud/) account. * You must have Python 3 installed. * You must have dbt v1.5 or newer [installed](//docs/local/install-dbt). * You must have the Dremio adapter 1.5.0 or newer [installed and configured](https://docs.getdbt.com/docs/local/connect-data-platform/dremio-setup.md) for Dremio Cloud. * You must have basic working knowledge of Git and the command line interface (CLI). #### Validate your environment Validate your environment by running the following commands in your CLI and verifying the results: ```shell $ python3 --version Python 3.11.4 # Must be Python 3 ``` ```shell $ dbt --version Core: - installed: 1.5.0 # Must be 1.5 or newer - latest: 1.6.3 - Update available! Your version of dbt-core is out of date! You can find instructions for upgrading here: https://docs.getdbt.com/docs/installation Plugins: - dremio: 1.5.0 - Up to date! # Must be 1.5 or newer ``` #### Getting started 1. Clone the Dremio dbt v1 sample project from the [GitHub repo](https://github.com/dremio-brock/DremioDBTSample/tree/master/dremioSamples). 2. In your integrated development environment (Studio IDE), open the relation.py file in the Dremio adapter directory: `$HOME/Library/Python/3.9/lib/python/site-packages/dbt/adapters/dremio/relation.py` 3. Find and update lines 51 and 52 to match the following syntax: ```python PATTERN = re.compile(r"""((?:[^."']|"[^"]*"|'[^']*')+)""") return ".".join(PATTERN.split(identifier)[1::2]) ``` The complete selection should look like this: ```python def quoted_by_component(self, identifier, componentName): if componentName == ComponentName.Schema: PATTERN = re.compile(r"""((?:[^."']|"[^"]*"|'[^']*')+)""") return ".".join(PATTERN.split(identifier)[1::2]) else: return self.quoted(identifier) ``` You need to update this pattern because the plugin doesn’t support schema names in Dremio containing dots and spaces. #### Build your pipeline 1. Create a `profiles.yml` file in the `$HOME/.dbt/profiles.yml` path and add the following configs: ```yaml dremioSamples: outputs: cloud_dev: dremio_space: dev dremio_space_folder: no_schema object_storage_path: dev object_storage_source: $scratch pat: <this_is_the_personal_access_token> cloud_host: api.dremio.cloud cloud_project_id: <id_of_project_you_belong_to> threads: 1 type: dremio use_ssl: true user: <your_username> target: dev ``` 2. Execute the transformation pipeline: ```shell $ dbt run -t cloud_dev ``` If the above configurations have been implemented, the output will look something like this: ```shell 17:24:16 Running with dbt=1.5.0 17:24:17 Found 5 models, 0 tests, 0 snapshots, 0 analyses, 348 macros, 0 operations, 0 seed files, 2 sources, 0 exposures, 0 metrics, 0 groups 17:24:17 17:24:29 Concurrency: 1 threads (target='cloud_dev') 17:24:29 17:24:29 1 of 5 START sql view model Preparation.trips .................................. [RUN] 17:24:31 1 of 5 OK created sql view model Preparation. trips ............................. [OK in 2.61s] 17:24:31 2 of 5 START sql view model Preparation.weather ................................ [RUN] 17:24:34 2 of 5 OK created sql view model Preparation.weather ........................... [OK in 2.15s] 17:24:34 3 of 5 START sql view model Business.Transportation.nyc_trips .................. [RUN] 17:24:36 3 of 5 OK created sql view model Business.Transportation.nyc_trips ............. [OK in 2.18s] 17:24:36 4 of 5 START sql view model Business.Weather.nyc_weather ....................... [RUN] 17:24:38 4 of 5 OK created sql view model Business.Weather.nyc_weather .................. [OK in 2.09s] 17:24:38 5 of 5 START sql view model Application.nyc_trips_with_weather ................. [RUN] 17:24:41 5 of 5 OK created sql view model Application.nyc_trips_with_weather ............ [OK in 2.74s] 17:24:41 17:24:41 Finished running 5 view models in 0 hours 0 minutes and 24.03 seconds (24.03s). 17:24:41 17:24:41 Completed successfully 17:24:41 17:24:41 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 ``` Now that you have a running environment and a completed job, you can view the data in Dremio and expand your code. This is a snapshot of the project structure in an Studio IDE: [![Cloned repo in an IDE](/img/guides/dremio/dremio-cloned-repo.png?v=2 "Cloned repo in an IDE")](#)Cloned repo in an IDE #### About the schema.yml The `schema.yml` file defines Dremio sources and models to be used and what data models are in scope. In this guides sample project, there are two data sources: 1. The `NYC-weather.csv` stored in the **Samples** database and 2. The `sample_data` from the **Samples database**. The models correspond to both weather and trip data respectively and will be joined for analysis. The sources can be found by navigating to the **Object Storage** section of the Dremio Cloud UI. [![NYC-weather.csv location in Dremio Cloud](/img/guides/dremio/dremio-nyc-weather.png?v=2 "NYC-weather.csv location in Dremio Cloud")](#)NYC-weather.csv location in Dremio Cloud #### About the models **Preparation** — `preparation_trips.sql` and `preparation_weather.sql` are building views on top of the trips and weather data. **Business** — `business_transportation_nyc_trips.sql` applies some level of transformation on `preparation_trips.sql` view. `Business_weather_nyc.sql` has no transformation on the `preparation_weather.sql` view. **Application** — `application_nyc_trips_with_weather.sql` joins the output from the Business model. This is what your business users will consume. #### The Job output When you run the dbt job, it will create a **dev** space folder that has all the data assets created. This is what you will see in Dremio Cloud UI. Spaces in Dremio is a way to organize data assets which map to business units or data products. [![Dremio Cloud dev space](/img/guides/dremio/dremio-dev-space.png?v=2 "Dremio Cloud dev space")](#)Dremio Cloud dev space Open the **Application folder** and you will see the output of the simple transformation we did using dbt. [![Application folder transformation output](/img/guides/dremio/dremio-dev-application.png?v=2 "Application folder transformation output")](#)Application folder transformation output #### Query the data Now that you have run the job and completed the transformation, it's time to query your data. Click on the `nyc_trips_with_weather` view. That will take you to the SQL Runner page. Click **Show SQL Pane** on the upper right corner of the page. Run the following query: ```sql SELECT vendor_id, AVG(tip_amount) FROM dev.application."nyc_treips_with_weather" GROUP BY vendor_id ``` [![Sample output from SQL query](/img/guides/dremio/dremio-test-results.png?v=2 "Sample output from SQL query")](#)Sample output from SQL query This completes the integration setup and data is ready for business consumption. --- ### Build, test, document, and promote adapters [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() important dbt v1 This guide is for creating adapters for the Python-based dbt v1. For adapter creation on the Rust-based dbt v2, check out our [new guide](https://docs.getdbt.com/guides/adapter-creation-v2.md?step=1) #### Introduction Adapters are an essential component of dbt. At their most basic level, they are how dbt connects with the various supported data platforms. At a higher-level, dbt v1 adapters strive to give analytics engineers more transferable skills as well as standardize how analytics projects are structured. Gone are the days where you have to learn a new language or flavor of SQL when you move to a new job that has a different data platform. That is the power of adapters in dbt v1. Navigating and developing around the nuances of different databases can be daunting, but you are not alone. Visit [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM) Slack channel for additional help beyond the documentation. ##### All databases are not the same There's a tremendous amount of work that goes into creating a database. Here is a high-level list of typical database layers (from the outermost layer moving inwards): * SQL API * Client Library / Driver * Server Connection Manager * Query parser * Query optimizer * Runtime * Storage Access Layer * Storage There's a lot more there than just SQL as a language. Databases (and data warehouses) are so popular because you can abstract away a great deal of the complexity from your brain to the database itself. This enables you to focus more on the data. dbt allows for further abstraction and standardization of the outermost layers of a database (SQL API, client library, connection manager) into a framework that both: * Opens database technology to less technical users (a large swath of a DBA's role has been automated, similar to how the vast majority of folks with websites today no longer have to be "[webmasters](https://en.wikipedia.org/wiki/Webmaster)"). * Enables more meaningful conversations about how data warehousing should be done. This is where dbt adapters become critical. ##### What needs to be adapted? dbt adapters are responsible for *adapting* dbt's standard functionality to a particular database. Our prototypical database and adapter are PostgreSQL and dbt-postgres, and most of our adapters are somewhat based on the functionality described in dbt-postgres. Connecting dbt to a new database will require a new adapter to be built or an existing adapter to be extended. The outermost layers of a database map roughly to the areas in which the dbt adapter framework encapsulates inter-database differences. ##### SQL API Even amongst ANSI-compliant databases, there are differences in the SQL grammar. Here are some categories and examples of SQL statements that can be constructed differently: | Category | Area of differences | Examples | | -------------------------------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Statement syntax | The use of `IF EXISTS` | \* `IF <TABLE> EXISTS, DROP TABLE`\* `DROP <TABLE> IF EXISTS` | | Workflow definition & semantics | Incremental updates | \* `MERGE`\* `DELETE; INSERT` | | Relation and column attributes/configuration | Database-specific materialization configs | \* `DIST = ROUND_ROBIN` (Synapse)\* `DIST = EVEN` (Redshift) | | Permissioning | Grant statements that can only take one grantee at a time vs those that accept lists of grantees | \* `grant SELECT on table dinner.corn to corn_kid, everyone`\* `grant SELECT on table dinner.corn to corn_kid; grant SELECT on table dinner.corn to everyone` | ##### Python Client Library & Connection Manager The other big category of inter-database differences comes with how the client connects to the database and executes queries against the connection. To integrate with dbt, a data platform must have a pre-existing python client library or support ODBC, using a generic python library like pyodbc. | Category | Area of differences | Examples | | ---------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Credentials & authentication | Authentication | \* Username & password\* MFA with `boto3` or Okta token | | Connection opening/closing | Create a new connection to db | \* `psycopg2.connect(connection_string)`\* `google.cloud.bigquery.Client(...)` | | Inserting local data | Load seed .`csv` files into Python memory | \* `google.cloud.bigquery.Client.load_table_from_file(...)` (BigQuery)\* `INSERT ... INTO VALUES ...` prepared statement (most other databases) | ##### How dbt encapsulates and abstracts these differences Differences between databases are encoded into discrete areas: | Components | Code Path | Function | | ---------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | Python classes | `adapters/<adapter_name>` | Configuration (Refer to \[Python classes]\(#python classes) | | Macros | `include/<adapter_name>/macros/adapters/` | SQL API & statement syntax (for example, how to create schema or how to get table information) | | Materializations | `include/<adapter_name>/macros/materializations/` | Table/view/snapshot/ workflow definitions | ###### Python classes These classes implement all the methods responsible for: * Connecting to a database and issuing queries. * Providing dbt with database-specific configuration information. | Class | Description | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AdapterClass | High-level configuration type conversion and any database-specific python methods needed | | AdapterCredentials | Typed dictionary of possible profiles and associated methods | | AdapterConnectionManager | All the methods responsible for connecting to a database and issuing queries | | AdapterRelation | How relation names should be rendered, printed, and quoted. Do relation names use all three parts? `catalog.model_name` (two-part name) or `database.schema.model_name` (three-part name) | | AdapterColumn | How names should be rendered, and database-specific properties | ###### Macros A set of *macros* responsible for generating SQL that is compliant with the target database. ###### Materializations A set of *materializations* and their corresponding helper macros defined in dbt using Jinja and SQL. They codify for dbt how model files should be persisted into the database. ##### Adapter Architecture Below is a flow diagram illustrating how a `dbt run` command works with the `dbt-postgres` adapter. It shows the relationship between `dbt-core`, `dbt-adapters`, and individual adapters. [![Diagram of adapter architecture](/img/adapter-guide/adapter-architecture-diagram.png?v=2 "Diagram of adapter architecture")](#)Diagram of adapter architecture #### Prerequisites It is very important that you have the right skills, and understand the level of difficulty required to make an adapter for your data platform. The more you can answer Yes to the below questions, the easier your adapter development (and user-) experience will be. See the [New Adapter Information Sheet wiki](https://github.com/dbt-labs/dbt/wiki/New-Adapter-Information-Sheet) for even more specific questions. ##### Training * The developer (and any product managers) ideally will have substantial experience as an end-user of dbt. If not, it is highly advised that you at least take the [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) and [Advanced Materializations](https://learn.getdbt.com/courses/advanced-materializations) course. ##### Database * Does the database complete transactions fast enough for interactive development? * Can you execute SQL against the data platform? * Is there a concept of schemas? * Does the data platform support ANSI SQL, or at least a subset? ##### Driver / Connection Library * Is there a Python-based driver for interacting with the database that is db API 2.0 compliant (for example, Psycopg2 for Postgres and pyodbc for SQL Server) * Does it support: prepared statements, multiple statements, or single sign on token authorization to the data platform? ##### Open source software * Does your organization have an established process for publishing open source software? It is easiest to build an adapter for dbt when the following the data warehouse/platform in question has: * a conventional ANSI-SQL interface (or as close to it as possible), * a mature connection library/SDK that uses ODBC or Python DB 2 API, and * a way to enable developers to iterate rapidly with both quick reads and writes ##### Maintaining your new adapter When your adapter becomes more popular, and people start using it, you may quickly become the maintainer of an increasingly popular open source project. With this new role, comes some unexpected responsibilities that not only include code maintenance, but also working with a community of users and contributors. To help people understand what to expect of your project, you should communicate your intentions early and often in your adapter documentation or README. Answer questions like, Is this experimental work that people should use at their own risk? Or is this production-grade code that you're committed to maintaining into the future? ###### Keeping the code compatible with dbt v1 An adapter is compatible with dbt v1 if it has correctly implemented the interface defined in [dbt-adapters](https://github.com/dbt-labs/dbt-adapters/) and is tested by [dbt-tests-adapters](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter). Prior to dbt version 1.8, this interface was contained in `dbt-core`. New minor version releases of `dbt-adapters` may include changes to the Python interface for adapter plugins, as well as new or updated test cases. The maintainers of `dbt-adapters` will clearly communicate these changes in documentation and release notes, and they will aim for backwards compatibility whenever possible. Patch releases of `dbt-adapters` will *not* include breaking changes or new features to adapter-facing code. ###### Versioning and releasing your adapter dbt Labs strongly recommends you to adopt the following approach when versioning and releasing your plugin. * Declare major version compatibility with `dbt-adapters` and only set a boundary on the minor version if there is some known reason. * Do not import or rely on code from `dbt-core`. * Aim to release a new minor version of your plugin as you add substantial new features. Typically, this will be triggered by adding support for new features released in `dbt-adapters` or by changes to the data platform itself. * While your plugin is new and you're iterating on features, aim to offer backwards compatibility and deprecation notices for at least one minor version. As your plugin matures, aim to leave backwards compatibility and deprecation notices in place until the next major version (v2). * Release patch versions of your plugins whenever needed. These patch releases should only contain fixes. note Prior to dbt version 1.8, we recommended that the minor version of your plugin should match the minor version in `dbt-core` (for example, 1.1.x). #### Build a new adapter This step will walk you through the first creating the necessary adapter classes and macros, and provide some resources to help you validate that your new adapter is working correctly. Make sure you've familiarized yourself with the previous steps in this guide. Once the adapter is passing most of the functional tests in the previous "Testing a new adapter" step, please let the community know that is available to use by adding the adapter to the ["Supported Data Platforms"](https://docs.getdbt.com/docs/supported-data-platforms.md) page by following the steps given in "Documenting your adapter. For any questions you may have, don't hesitate to ask in the [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM) Slack channel. The community is very helpful and likely has experienced a similar issue as you. ##### Scaffolding a new adapter To create a new adapter plugin from scratch, you can use the [dbt-database-adapter-scaffold](https://github.com/dbt-labs/dbt-database-adapter-scaffold) to trigger an interactive session which will generate a scaffolding for you to build upon. Example usage: ```text $ cookiecutter gh:dbt-labs/dbt-database-adapter-scaffold ``` The generated boilerplate starting project will include a basic adapter plugin file structure, examples of macros, high level method descriptions, and more. One of the most important choices you will make during the cookiecutter generation will revolve around the field for `is_sql_adapter` which is a boolean used to correctly apply imports for either a `SQLAdapter` or `BaseAdapter`. Knowing which you will need requires a deeper knowledge of your selected database but a few good guides for the choice are. * Does your database have a complete SQL API? Can it perform tasks using SQL such as creating schemas, dropping schemas, querying an `information_schema` for metadata calls? If so, it is more likely to be a SQLAdapter where you set `is_sql_adapter` to `True`. * Most adapters do fall under SQL adapters which is why we chose it as the default `True` value. * It is very possible to build out a fully functional `BaseAdapter`. This will require a little more ground work as it doesn't come with some prebuilt methods the `SQLAdapter` class provides. See `dbt-bigquery` as a good guide. ##### Implementation details Regardless if you decide to use the cookiecutter template or manually create the plugin, this section will go over each method that is required to be implemented. The following table provides a high-level overview of the classes, methods, and macros you may have to define for your data platform. | File | Component | Purpose | | ------------------------------------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `./setup.py` | `setup()` function | adapter meta-data (package name, version, author, homepage, and more) | | `myadapter/dbt/adapters/myadapter/__init__.py` | `AdapterPlugin` | bundle all the information below into a dbt plugin | | `myadapter/dbt/adapters/myadapter/connections.py` | `MyAdapterCredentials` class | parameters to connect to and configure the database, using the chosen Python driver | | `myadapter/dbt/adapters/myadapter/connections.py` | `MyAdapterConnectionManager` class | telling dbt how to interact with the database, including opening and closing connections, executing queries, and fetching data. Effectively a wrapper around the db API or driver. | | `myadapter/dbt/include/bigquery/` | a dbt project of macro "overrides" in the format of "myadapter\_\_" | any differences in SQL syntax for regular db operations will be modified here from the global\_project (for example, "Create Table As Select" and "Get all relations in the current schema") | | `myadapter/dbt/adapters/myadapter/impl.py` | `MyAdapterConfig` | database- and relation-level configs and | | `myadapter/dbt/adapters/myadapter/impl.py` | `MyAdapterAdapter` | for changing *how* dbt performs operations like macros and other needed Python functionality | | `myadapter/dbt/adapters/myadapter/column.py` | `MyAdapterColumn` | for defining database-specific column such as datatype mappings | ##### Editing `setup.py` Edit the file at `myadapter/setup.py` and fill in the missing information. You can skip this step if you passed the arguments for `email`, `url`, `author`, and `dependencies` to the cookiecutter template script. If you plan on having nested macro folder structures, you may need to add entries to `package_data` so your macro source files get installed. ##### Editing the connection manager Edit the connection manager at `myadapter/dbt/adapters/myadapter/connections.py`. This file is defined in the sections below. ###### The Credentials class The credentials class defines all of the database-specific credentials (for example, `username` and `password`) that users will need in the [connection profile](https://docs.getdbt.com/docs/supported-data-platforms.md) for your new adapter. Each credentials contract should subclass dbt.adapters.base.Credentials, and be implemented as a python dataclass. Note that the base class includes required database and schema fields, as dbt uses those values internally. For example, if your adapter requires a host, integer port, username string, and password string, but host is the only required field, you'd add definitions for those new properties to the class as types, like this: connections.py ```python from dataclasses import dataclass from typing import Optional from dbt.adapters.base import Credentials @dataclass class MyAdapterCredentials(Credentials): host: str port: int = 1337 username: Optional[str] = None password: Optional[str] = None @property def type(self): return 'myadapter' @property def unique_field(self): """ Hashed and included in anonymous telemetry to track adapter adoption. Pick a field that can uniquely identify one team/organization building with this adapter """ return self.host def _connection_keys(self): """ List of keys to display in the `dbt debug` output. """ return ('host', 'port', 'database', 'username') ``` There are a few things you can do to make it easier for users when connecting to your database: * Be sure to implement the Credentials' `_connection_keys` method shown above. This method will return the keys that should be displayed in the output of the `dbt debug` command. As a general rule, it's good to return all the arguments used in connecting to the actual database except the password (even optional arguments). * Create a `profile_template.yml` to enable configuration prompts for a brand-new user setting up a connection profile using the [`dbt init` command](https://docs.getdbt.com/reference/commands/init.md). You will find more details in the following steps. * You may also want to define an `ALIASES` mapping on your Credentials class to include any config names you want users to be able to use in place of 'database' or 'schema'. For example if everyone using the MyAdapter database calls their databases "collections", you might do: connections.py ```python @dataclass class MyAdapterCredentials(Credentials): host: str port: int = 1337 username: Optional[str] = None password: Optional[str] = None ALIASES = { 'collection': 'database', } ``` Then users can use `collection` OR `database` in their `profiles.yml`, `dbt_project.yml`, or `config()` calls to set the database. ###### `ConnectionManager` class methods Once credentials are configured, you'll need to implement some connection-oriented methods. They are enumerated in the SQLConnectionManager docstring, but an overview will also be provided here. **Methods to implement:** * `open` * `get_response` * `cancel` * `exception_handler` * `standardize_grants_dict` ###### `open(cls, connection)` `open()` is a classmethod that gets a connection object (which could be in any state, but will have a `Credentials` object with the attributes you defined above) and moves it to the 'open' state. Generally this means doing the following: * if the connection is open already, log and return it. * If a database needed changes to the underlying connection before re-use, that would happen here * create a connection handle using the underlying database library using the credentials * on success: * set connection.state to `'open'` * set connection.handle to the handle object * this is what must have a `cursor()` method that returns a cursor! * on error: * set connection.state to `'fail'` * set connection.handle to `None` * raise a `dbt.exceptions.FailedToConnectException` with the error and any other relevant information For example: connections.py ```python @classmethod def open(cls, connection): if connection.state == 'open': logger.debug('Connection is already open, skipping open.') return connection credentials = connection.credentials try: handle = myadapter_library.connect( host=credentials.host, port=credentials.port, username=credentials.username, password=credentials.password, catalog=credentials.database ) connection.state = 'open' connection.handle = handle return connection ``` ###### `get_response(cls, cursor)` `get_response` is a classmethod that gets a cursor object and returns adapter-specific information about the last executed command. The return value should be an `AdapterResponse` object that includes items such as `code`, `rows_affected`, `bytes_processed`, and a summary `_message` for logging to stdout. connections.py ```python @classmethod def get_response(cls, cursor) -> AdapterResponse: code = cursor.sqlstate or "OK" rows = cursor.rowcount status_message = f"{code} {rows}" return AdapterResponse( _message=status_message, code=code, rows_affected=rows ) ``` ###### `cancel(self, connection)` `cancel` is an instance method that gets a connection object and attempts to cancel any ongoing queries, which is database dependent. Some databases don't support the concept of cancellation, they can simply implement it using `pass` and their adapter classes should implement an `is_cancelable` that returns False - On ctrl+c connections may remain running. This method must be implemented carefully, as the affected connection will likely be in use in a different thread. connections.py ```python def cancel(self, connection): tid = connection.handle.transaction_id() sql = 'select cancel_transaction({})'.format(tid) logger.debug("Cancelling query '{}' ({})".format(connection_name, pid)) _, cursor = self.add_query(sql, 'master') res = cursor.fetchone() logger.debug("Canceled query '{}': {}".format(connection_name, res)) ``` ###### `exception_handler(self, sql, connection_name='master')` `exception_handler` is an instance method that returns a context manager that will handle exceptions raised by running queries, catch them, log appropriately, and then raise exceptions dbt knows how to handle. If you use the (highly recommended) `@contextmanager` decorator, you only have to wrap a `yield` inside a `try` block, like so: connections.py ```python @contextmanager def exception_handler(self, sql: str): try: yield except myadapter_library.DatabaseError as exc: self.release(connection_name) logger.debug('myadapter error: {}'.format(str(e))) raise dbt.exceptions.DatabaseException(str(exc)) except Exception as exc: logger.debug("Error running SQL: {}".format(sql)) logger.debug("Rolling back transaction.") self.release(connection_name) raise dbt.exceptions.RuntimeException(str(exc)) ``` ###### `standardize_grants_dict(self, grants_table: agate.Table) -> dict` `standardize_grants_dict` is an method that returns the dbt-standardized grants dictionary that matches how users configure grants now in dbt. The input is the result of `SHOW GRANTS ON {{model}}` call loaded into an agate table. If there's any massaging of agate table containing the results, of `SHOW GRANTS ON {{model}}`, that can't easily be accomplished in SQL, it can be done here. For example, the SQL to show grants *should* filter OUT any grants TO the current user/role (for example, OWNERSHIP). If that's not possible in SQL, it can be done in this method instead. impl.py ```python @available def standardize_grants_dict(self, grants_table: agate.Table) -> dict: """ :param grants_table: An agate table containing the query result of the SQL returned by get_show_grant_sql :return: A standardized dictionary matching the `grants` config :rtype: dict """ grants_dict: Dict[str, List[str]] = {} for row in grants_table: grantee = row["grantee"] privilege = row["privilege_type"] if privilege in grants_dict.keys(): grants_dict[privilege].append(grantee) else: grants_dict.update({privilege: [grantee]}) return grants_dict ``` ##### Editing the adapter implementation Edit the connection manager at `myadapter/dbt/adapters/myadapter/impl.py` Very little is required to implement the adapter itself. On some adapters, you will not need to override anything. On others, you'll likely need to override some of the `convert_*` classmethods, or override the `is_cancelable` classmethod on others to return `False`. ###### `datenow()` This classmethod provides the adapter's canonical date function. This is not used but is required– anyway on all adapters. impl.py ```python @classmethod def date_function(cls): return 'datenow()' ``` ##### Editing SQL logic dbt implements specific SQL operations using Jinja macros. While reasonable defaults are provided for many such operations (like `create_schema`, `drop_schema`, `create_table`, and more), you may need to override one or more of macros when building a new adapter. ###### Required macros The following macros must be implemented, but you can override their behavior for your adapter using the "dispatch" pattern described below. Macros marked (required) do not have a valid default implementation, and are required for dbt to operate. * `alter_column_type` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/columns.sql#L37-L55)) * `check_schema_exists` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/metadata.sql#L43-L55)) * `create_schema` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/schema.sql#L1-L9)) * `drop_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/relation.sql#L34-L42)) * `drop_schema` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/schema.sql#L12-L20)) * `get_columns_in_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/columns.sql#L1-L8)) (required) * `list_relations_without_caching` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/metadata.sql#L58-L65)) (required) * `list_schemas` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/metadata.sql#L29-L40)) * `rename_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/relation.sql#L56-L65)) * `truncate_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/relation.sql#L45-L53)) * `current_timestamp` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/freshness.sql#L1-L8)) (required) * `copy_grants` ###### Adapter dispatch Most modern databases support a majority of the standard SQL spec. There are some databases that *do not* support critical aspects of the SQL spec however, or they provide their own nonstandard mechanisms for implementing the same functionality. To account for these variations in SQL support, dbt provides a mechanism called [multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch) for macros. With this feature, macros can be overridden for specific adapters. This makes it possible to implement high-level methods (like "create table") in a database-specific way. adapters.sql ```jinja2 {# dbt will call this macro by name, providing any arguments #} {% macro create_table_as(temporary, relation, sql) -%} {# dbt will dispatch the macro call to the relevant macro #} {{ return( adapter.dispatch('create_table_as')(temporary, relation, sql) ) }} {%- endmacro %} {# If no macro matches the specified adapter, "default" will be used #} {% macro default__create_table_as(temporary, relation, sql) -%} ... {%- endmacro %} {# Example which defines special logic for Redshift #} {% macro redshift__create_table_as(temporary, relation, sql) -%} ... {%- endmacro %} {# Example which defines special logic for BigQuery #} {% macro bigquery__create_table_as(temporary, relation, sql) -%} ... {%- endmacro %} ``` The `adapter.dispatch()` macro takes a second argument, `packages`, which represents a set of "search namespaces" in which to find potential implementations of a dispatched macro. This allows users of community-supported adapters to extend or "shim" dispatched macros from common packages, such as `dbt-utils`, with adapter-specific versions in their own project or other installed packages. See: * "Shim" package examples: [`spark-utils`](https://github.com/dbt-labs/spark-utils), [`tsql-utils`](https://github.com/dbt-msft/tsql-utils) * [`adapter.dispatch` docs](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) ###### Overriding adapter methods While much of dbt's adapter-specific functionality can be modified in adapter macros, it can also make sense to override adapter methods directly. In this example, assume that a database does not support a `cascade` parameter to `drop schema`. Instead, we can implement an approximation where we drop each relation and then drop the schema. impl.py ```python def drop_schema(self, relation: BaseRelation): relations = self.list_relations( database=relation.database, schema=relation.schema ) for relation in relations: self.drop_relation(relation) super().drop_schema(relation) ``` ###### Grants Macros See [this GitHub discussion](https://github.com/dbt-labs/dbt/discussions/5468) for information on the macros required for `GRANT` statements: ##### Behavior change flags Starting in `dbt-adapters==1.5.0` and `dbt-core==1.8.7`, adapter maintainers can implement their own behavior change flags. Refer to [Behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) for more information. Behavior Flags are not intended to be long-living feature flags. They should be implemented with the expectation that the behavior will be the default within an expected period of time. To implement a behavior change flag, you must provide a name for the flag, a default setting (`True` / `False`), an optional source, and a description and/or a link to the flag's documentation on docs.getdbt.com. We recommend having a description and documentation link whenever possible. The description and/or docs should provide end users context for why the flag exists, why they may see a warning, and why they may want to utilize the behavior flag. Behavior change flags can be implemented by overwriting `_behavior_flags()` on the adapter in `impl.py`: impl.py ```python class ABCAdapter(BaseAdapter): ... @property def _behavior_flags(self) -> List[BehaviorFlag]: return [ { "name": "enable_new_functionality_requiring_higher_permissions", "default": False, "source": "dbt-abc", "description": ( "The dbt-abc adapter is implementing a new method for sourcing metadata. " "This is a more performant way for dbt to source metadata but requires higher permissions on the platform. " "Enabling this without granting the requisite permissions will result in an error. " "This feature is expected to be required by Spring 2025." ), "docs_url": "https://docs.getdbt.com/reference/global-configs/behavior-changes#abc-enable_new_functionality_requiring_higher_permissions", } ] ``` Once a behavior change flag has been implemented, it can be referenced on the adapter both in `impl.py` and in Jinja macros: impl.py ```python class ABCAdapter(BaseAdapter): ... def some_method(self, *args, **kwargs): if self.behavior.enable_new_functionality_requiring_higher_permissions: # do the new thing else: # do the old thing ``` adapters.sql ```sql {% macro some_macro(**kwargs) %} {% if adapter.behavior.enable_new_functionality_requiring_higher_permissions %} {# do the new thing #} {% else %} {# do the old thing #} {% endif %} {% endmacro %} ``` Every time the behavior flag evaluates to `False,` it warns the user, informing them that a change will occur in the future. This warning doesn't display when the flag evaluates to `True` as the user is already in the new experience. Recognizing that the warnings can be disruptive and are not always necessary, you can evaluate the flag without triggering the warning. Simply append `.no_warn` to the end of the flag. impl.py ```python class ABCAdapter(BaseAdapter): ... def some_method(self, *args, **kwargs): if self.behavior.enable_new_functionality_requiring_higher_permissions.no_warn: # do the new thing else: # do the old thing ``` adapters.sql ```sql {% macro some_macro(**kwargs) %} {% if adapter.behavior.enable_new_functionality_requiring_higher_permissions.no_warn %} {# do the new thing #} {% else %} {# do the old thing #} {% endif %} {% endmacro %} ``` It's best practice to evaluate a behavior flag as few times as possible. This will make it easier to remove once the behavior change has matured. As a result, evaluating the flag earlier in the logic flow is easier. Then, take either the old or the new path. While this may create some duplication in code, using behavior flags in this way provides a safer way to implement a change, which we are already admitting is risky or even breaking in nature. ##### Other files ###### `profile_template.yml` In order to enable the [`dbt init` command](https://docs.getdbt.com/reference/commands/init.md) to prompt users when setting up a new project and connection profile, you should include a **profile template**. The filepath needs to be `dbt/include/<adapter_name>/profile_template.yml`. It's possible to provide hints, default values, and conditional prompts based on connection methods that require different supporting attributes. Users will also be able to include custom versions of this file in their own projects, with fixed values specific to their organization, to support their colleagues when using your dbt adapter for the first time. See examples: * [dbt-postgres](https://github.com/dbt-labs/dbt-postgres/blob/main/dbt/include/postgres/profile_template.yml) * [dbt-redshift](https://github.com/dbt-labs/dbt-redshift/blob/main/dbt/include/redshift/profile_template.yml) * [dbt-snowflake](https://github.com/dbt-labs/dbt-snowflake/blob/main/dbt/include/snowflake/profile_template.yml) * [dbt-bigquery](https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/include/bigquery/profile_template.yml) ###### `__version__.py` To assure that `dbt --version` provides the latest v1 version the adapter supports, be sure include a `__version__.py` file. The filepath will be `dbt/adapters/<adapter_name>/__version__.py`. We recommend using the latest v1 version and as the adapter is made compatible with later versions, this file will need to be updated. For a sample file, check out this [example](https://github.com/dbt-labs/dbt-snowflake/blob/main/dbt/adapters/snowflake/__version__.py). It should be noted that both of these files are included in the bootstrapped output of the `dbt-database-adapter-scaffold` so when using the scaffolding, these files will be included. #### Test your adapter This document has two sections: 1. Refer to "About the testing framework" for a description of the standard framework that we maintain for using pytest together with dbt. It includes an example that shows the anatomy of a simple test case. 2. Refer to "Testing your adapter" for a step-by-step guide for using our out-of-the-box suite of "basic" tests, which will validate that your adapter meets a baseline of dbt functionality. ##### Testing prerequisites * Your adapter must be compatible with dbt **v1.1** or newer * You should be familiar with **pytest**: <https://docs.pytest.org> ##### About the testing framework [dbt-adapters-tests](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter) offers a standard framework for running prebuilt functional tests, and for defining your own tests. The core testing framework is built using `pytest`, a mature and standard library for testing Python projects. It includes basic utilities for setting up pytest + dbt. These are used by all "prebuilt" functional tests, and make it possible to quickly write your own tests. Those utilities allow you to do three basic things: 1. **Quickly set up a dbt "project."** Define project resources using methods such as `models()` and `seeds()`. Use `project_config_update()` to pass configurations into `dbt_project.yml`. 2. **Define a sequence of dbt commands.** The most important utility is `run_dbt()`, which returns the [results](https://docs.getdbt.com/reference/dbt-classes.md#result-objects) of each dbt command. It takes a list of CLI specifiers (subcommand + flags), as well as an optional second argument, `expect_pass=False`, for cases where you expect the command to fail. 3. **Validate the results of those dbt commands.** For example, `check_relations_equal()` asserts that two database objects have the same structure and content. You can also write your own `assert` statements, by inspecting the results of a dbt command, or querying arbitrary database objects with `project.run_sql()`. You can see the full suite of utilities, with arguments and annotations, in [`util.py`](https://github.com/dbt-labs/dbt/blob/1.latest/core/dbt/tests/util.py). You'll also see them crop up across a number of test cases. While all utilities are intended to be reusable, you won't need all of them for every test. In the example below, we'll show a simple test case that uses only a few utilities. ###### Example: a simple test case This example will show you the anatomy of a test case using dbt + pytest. We will create reusable components, combine them to form a dbt "project", and define a sequence of dbt commands. Then, we'll use Python `assert` statements to ensure those commands succeed (or fail) as we expect. In ["Getting started running basic tests,"](#getting-started-running-basic-tests) we'll offer step-by-step instructions for installing and configuring `pytest`, so that you can run it on your own machine. For now, it's more important to see how the pieces of a test case fit together. This example includes a seed, a model, and two tests—one of which will fail. 1. Define Python strings that will represent the file contents in your dbt project. Defining these in a separate file enables you to reuse the same components across different test cases. The pytest name for this type of reusable component is "fixture." tests/functional/example/fixtures.py ```python # seeds/my_seed.csv my_seed_csv = """ id,name,some_date 1,Easton,1981-05-20T06:46:51 2,Lillian,1978-09-03T18:10:33 3,Jeremiah,1982-03-11T03:59:51 4,Nolan,1976-05-06T20:21:35 """.lstrip() # models/my_model.sql my_model_sql = """ select * from {{ ref('my_seed') }} union all select null as id, null as name, null as some_date """ # models/my_model.yml my_model_yml = """ version: 2 models: - name: my_model columns: - name: id data_tests: - unique - not_null # this test will fail """ ``` 2. Use the "fixtures" to define the project for your test case. These fixtures are always scoped to the **class**, where the class represents one test case—that is, one dbt project or scenario. (The same test case can be used for one or more actual tests, which we'll see in step 3.) Following the default pytest configurations, the file name must begin with `test_`, and the class name must begin with `Test`. tests/functional/example/test\_example\_failing\_test.py ```python import pytest from dbt.tests.util import run_dbt # our file contents from tests.functional.example.fixtures import ( my_seed_csv, my_model_sql, my_model_yml, ) # class must begin with 'Test' class TestExample: """ Methods in this class will be of two types: 1. Fixtures defining the dbt "project" for this test case. These are scoped to the class, and reused for all tests in the class. 2. Actual tests, whose names begin with 'test_'. These define sequences of dbt commands and 'assert' statements. """ # configuration in dbt_project.yml @pytest.fixture(scope="class") def project_config_update(self): return { "name": "example", "models": {"+materialized": "view"} } # everything that goes in the "seeds" directory @pytest.fixture(scope="class") def seeds(self): return { "my_seed.csv": my_seed_csv, } # everything that goes in the "models" directory @pytest.fixture(scope="class") def models(self): return { "my_model.sql": my_model_sql, "my_model.yml": my_model_yml, } # continues below ``` 3. Now that we've set up our project, it's time to define a sequence of dbt commands and assertions. We define one or more methods in the same file, on the same class (`TestExampleFailingTest`), whose names begin with `test_`. These methods share the same setup (project scenario) from above, but they can be run independently by pytest—so they shouldn't depend on each other in any way. tests/functional/example/test\_example\_failing\_test.py ```python # continued from above # The actual sequence of dbt commands and assertions # pytest will take care of all "setup" + "teardown" def test_run_seed_test(self, project): """ Seed, then run, then test. We expect one of the tests to fail An alternative pattern is to use pytest "xfail" (see below) """ # seed seeds results = run_dbt(["seed"]) assert len(results) == 1 # run models results = run_dbt(["run"]) assert len(results) == 1 # test tests results = run_dbt(["test"], expect_pass = False) # expect failing test assert len(results) == 2 # validate that the results include one pass and one failure result_statuses = sorted(r.status for r in results) assert result_statuses == ["fail", "pass"] @pytest.mark.xfail def test_build(self, project): """Expect a failing test""" # do it all results = run_dbt(["build"]) ``` 3. Our test is ready to run! The last step is to invoke `pytest` from your command line. We'll walk through the actual setup and configuration of `pytest` in the next section. terminal ```sh $ python3 -m pytest tests/functional/test_example.py =========================== test session starts ============================ platform ... -- Python ..., pytest-..., pluggy-... rootdir: ... plugins: ... tests/functional/test_example.py .X [100%] ======================= 1 passed, 1 xpassed in 1.38s ======================= ``` You can find more ways to run tests, along with a full command reference, in the [pytest usage docs](https://docs.pytest.org/how-to/usage.html). We've found the `-s` flag (or `--capture=no`) helpful to print logs from the underlying dbt invocations, and to step into an interactive debugger if you've added one. You can also use environment variables to set [global dbt configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md), such as (Applies to dbt v1.11 and later) `DBT_ENGINE_DEBUG` (to show debug-level logs). ##### Testing this adapter Anyone who installs `dbt-core`, and wishes to define their own test cases, can use the framework presented in the first section. The framework is especially useful for testing standard dbt behavior across different databases. To that end, we have built and made available a [package of reusable adapter test cases](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter), for creators and maintainers of adapter plugins. These test cases cover basic expected functionality, as well as functionality that frequently requires different implementations across databases. For the time being, this package is also located within the `dbt-labs/dbt` repository, but separate from the `dbt-core` Python package. ##### Categories of tests In the course of creating and maintaining your adapter, it's likely that you will end up implementing tests that fall into three broad categories: 1. **Basic tests** that every adapter plugin is expected to pass. These are defined in `tests.adapter.basic`. Given differences across data platforms, these may require slight modification or reimplementation. Significantly overriding or disabling these tests should be with good reason, since each represents basic functionality expected by dbt users. For example, if your adapter does not support incremental models, you should disable the test, [by marking it with `skip` or `xfail`](https://docs.pytest.org/en/latest/how-to/skipping.html), as well as noting that limitation in any documentation, READMEs, and usage guides that accompany your adapter. 2. **Optional tests**, for second-order functionality that is common across plugins, but not required for basic use. Your plugin can opt into these test cases by inheriting existing ones, or reimplementing them with adjustments. For now, this category includes all tests located outside the `basic` subdirectory. More tests will be added as we convert older tests defined on dbt-core and mature plugins to use the standard framework. 3. **Custom tests**, for behavior that is specific to your adapter / data platform. Each data warehouse has its own specialties and idiosyncrasies. We encourage you to use the same `pytest`-based framework, utilities, and fixtures to write your own custom tests for functionality that is unique to your adapter. If you run into an issue with the core framework, or the basic/optional test cases—or if you've written a custom test that you believe would be relevant and useful for other adapter plugin developers—please open an issue or PR in the `dbt-labs/dbt` repository on GitHub. ##### Getting started running basic tests In this section, we'll walk through the three steps to start running our basic test cases on your adapter plugin: 1. Install dependencies 2. Set up and configure pytest 3. Define test cases ##### Install dependencies You should already have a virtual environment with `dbt-core` and your adapter plugin installed. You'll also need to install: * [`pytest`](https://pypi.org/project/pytest/) * [`dbt-tests-adapter`](https://pypi.org/project/dbt-tests-adapter/), the set of common test cases * (optional) [`pytest` plugins](https://docs.pytest.org/en/7.0.x/reference/plugin_list.html)--we'll use `pytest-dotenv` below Or specify all dependencies in a requirements file like: dev\_requirements.txt ```txt pytest pytest-dotenv dbt-tests-adapter ``` ```sh python -m pip install -r dev_requirements.txt ``` ##### Set up and configure pytest First, set yourself up to run `pytest` by creating a file named `pytest.ini` at the root of your repository: pytest.ini ```python [pytest] filterwarnings = ignore:.*'soft_unicode' has been renamed to 'soft_str'*:DeprecationWarning ignore:unclosed file .*:ResourceWarning env_files = test.env # uses pytest-dotenv plugin # this allows you to store env vars for database connection in a file named test.env # rather than passing them in every CLI command, or setting in `PYTEST_ADDOPTS` # be sure to add "test.env" to .gitignore as well! testpaths = tests/functional # name per convention ``` Then, create a configuration file within your tests directory. In it, you'll want to define all necessary profile configuration for connecting to your data platform in local development and continuous integration. We recommend setting these values with environment variables, since this file will be checked into version control. tests/conftest.py ```python import pytest import os # Import the standard functional fixtures as a plugin # Note: fixtures with session scope need to be local pytest_plugins = ["dbt.tests.fixtures.project"] # The profile dictionary, used to write out profiles.yml # dbt will supply a unique schema per test, so we do not specify 'schema' here @pytest.fixture(scope="class") def dbt_profile_target(): return { 'type': '<myadapter>', 'threads': 1, 'host': os.getenv('HOST_ENV_VAR_NAME'), 'user': os.getenv('USER_ENV_VAR_NAME'), ... } ``` ##### Define test cases As in the example above, each test case is defined as a class, and has its own "project" setup. To get started, you can import all basic test cases and try running them without changes. tests/functional/adapter/test\_basic.py ```python import pytest from dbt.tests.adapter.basic.test_base import BaseSimpleMaterializations from dbt.tests.adapter.basic.test_singular_tests import BaseSingularTests from dbt.tests.adapter.basic.test_singular_tests_ephemeral import BaseSingularTestsEphemeral from dbt.tests.adapter.basic.test_empty import BaseEmpty from dbt.tests.adapter.basic.test_ephemeral import BaseEphemeral from dbt.tests.adapter.basic.test_incremental import BaseIncremental from dbt.tests.adapter.basic.test_generic_tests import BaseGenericTests from dbt.tests.adapter.basic.test_snapshot_check_cols import BaseSnapshotCheckCols from dbt.tests.adapter.basic.test_snapshot_timestamp import BaseSnapshotTimestamp from dbt.tests.adapter.basic.test_adapter_methods import BaseAdapterMethod class TestSimpleMaterializationsMyAdapter(BaseSimpleMaterializations): pass class TestSingularTestsMyAdapter(BaseSingularTests): pass class TestSingularTestsEphemeralMyAdapter(BaseSingularTestsEphemeral): pass class TestEmptyMyAdapter(BaseEmpty): pass class TestEphemeralMyAdapter(BaseEphemeral): pass class TestIncrementalMyAdapter(BaseIncremental): pass class TestGenericTestsMyAdapter(BaseGenericTests): pass class TestSnapshotCheckColsMyAdapter(BaseSnapshotCheckCols): pass class TestSnapshotTimestampMyAdapter(BaseSnapshotTimestamp): pass class TestBaseAdapterMethod(BaseAdapterMethod): pass ``` Finally, run pytest: ```sh python3 -m pytest tests/functional ``` ##### Modifying test cases You may need to make slight modifications in a specific test case to get it passing on your adapter. The mechanism to do this is simple: rather than simply inheriting the "base" test with `pass`, you can redefine any of its fixtures or test methods. For instance, on Redshift, we need to explicitly cast a column in the fixture input seed to use data type `varchar(64)`: tests/functional/adapter/test\_basic.py ```python import pytest from dbt.tests.adapter.basic.files import seeds_base_csv, seeds_added_csv, seeds_newcolumns_csv from dbt.tests.adapter.basic.test_snapshot_check_cols import BaseSnapshotCheckCols # set the datatype of the name column in the 'added' seed so it # can hold the '_update' that's added schema_seed_added_yml = """ version: 2 seeds: - name: added config: column_types: name: varchar(64) """ class TestSnapshotCheckColsRedshift(BaseSnapshotCheckCols): # Redshift defines the 'name' column such that it's not big enough # to hold the '_update' added in the test. @pytest.fixture(scope="class") def models(self): return { "base.csv": seeds_base_csv, "added.csv": seeds_added_csv, "seeds.yml": schema_seed_added_yml, } ``` As another example, the `dbt-bigquery` adapter asks users to "authorize" replacing a table with a view by supplying the `--full-refresh` flag. The reason: In the table materialization logic, a view by the same name must first be dropped; if the table query fails, the model will be missing. Knowing this possibility, the "base" test case offers a `require_full_refresh` switch on the `test_config` fixture class. For BigQuery, we'll switch it on: tests/functional/adapter/test\_basic.py ```python import pytest from dbt.tests.adapter.basic.test_base import BaseSimpleMaterializations class TestSimpleMaterializationsBigQuery(BaseSimpleMaterializations): @pytest.fixture(scope="class") def test_config(self): # effect: add '--full-refresh' flag in requisite 'dbt run' step return {"require_full_refresh": True} ``` It's always worth asking whether the required modifications represent gaps in perceived or expected dbt functionality. Are these simple implementation details, which any user of this database would understand? Are they limitations worth documenting? If, on the other hand, they represent poor assumptions in the "basic" test cases, which fail to account for a common pattern in other types of databases-—please open an issue or PR in the `dbt-labs/dbt` repository on GitHub. ##### Running with multiple profiles Some databases support multiple connection methods, which map to actually different functionality behind the scenes. For instance, the `dbt-spark` adapter supports connections to Apache Spark clusters *and* Databricks runtimes, which supports additional functionality out of the box, enabled by the Delta file format. tests/conftest.py ```python def pytest_addoption(parser): parser.addoption("--profile", action="store", default="apache_spark", type=str) # Using @pytest.mark.skip_profile('apache_spark') uses the 'skip_by_profile_type' # autouse fixture below def pytest_configure(config): config.addinivalue_line( "markers", "skip_profile(profile): skip test for the given profile", ) @pytest.fixture(scope="session") def dbt_profile_target(request): profile_type = request.config.getoption("--profile") elif profile_type == "databricks_sql_endpoint": target = databricks_sql_endpoint_target() elif profile_type == "apache_spark": target = apache_spark_target() else: raise ValueError(f"Invalid profile type '{profile_type}'") return target def apache_spark_target(): return { "type": "spark", "host": "localhost", ... } def databricks_sql_endpoint_target(): return { "type": "spark", "host": os.getenv("DBT_DATABRICKS_HOST_NAME"), ... } @pytest.fixture(autouse=True) def skip_by_profile_type(request): profile_type = request.config.getoption("--profile") if request.node.get_closest_marker("skip_profile"): for skip_profile_type in request.node.get_closest_marker("skip_profile").args: if skip_profile_type == profile_type: pytest.skip("skipped on '{profile_type}' profile") ``` If there are tests that *shouldn't* run for a given profile: tests/functional/adapter/basic.py ```python # Snapshots require access to the Delta file format, available on our Databricks connection, # so let's skip on Apache Spark @pytest.mark.skip_profile('apache_spark') class TestSnapshotCheckColsSpark(BaseSnapshotCheckCols): @pytest.fixture(scope="class") def project_config_update(self): return { "seeds": { "+file_format": "delta", }, "snapshots": { "+file_format": "delta", } } ``` Finally: ```sh python3 -m pytest tests/functional --profile apache_spark python3 -m pytest tests/functional --profile databricks_sql_endpoint ``` #### Document a new adapter If you've already built, and tested your adapter, it's time to document it so the dbt community will know that it exists and how to use it. ##### Making your adapter available Many community members maintain their adapter plugins under open source licenses. If you're interested in doing this, we recommend: * Hosting on a public git provider (for example, GitHub or Gitlab) * Publishing to [PyPI](https://pypi.org/) * Adding your adapter to the [community adapters](https://docs.getdbt.com/docs/community-adapters.md) list (refer to [Topics and pages to cover](#topics-and-pages-to-cover)) ##### General Guidelines To best inform the dbt community of the new adapter, you should contribute to the dbt's open-source documentation site, which uses the [Docusaurus project](https://docusaurus.io/). This is the site you're currently on! ##### Conventions Each `.md` file you create needs a header as shown below. The document id will also need to be added to the config file: `website/sidebars.js`. ```md --- title: "Documenting a new adapter" id: "documenting-a-new-adapter" --- ``` ##### Single Source of Truth We ask our adapter maintainers to use the [docs.getdbt.com repo](https://github.com/dbt-labs/docs.getdbt.com) (that is, this site) as the single-source-of-truth for documentation rather than having to maintain the same set of information in three different places. The adapter repo's `README.md` and the data platform's documentation pages should simply link to the corresponding page on this docs site. Keep reading for more information on what should and shouldn't be included on the dbt docs site. ##### Assumed Knowledge To simplify things, assume the reader of this documentation already knows how both dbt and your data platform works. There's already great material for how to learn dbt and the data platform out there. The documentation we're asking you to add should be what a user who is already proficient in both dbt and your data platform would need to know in order to use both. Effectively that boils down to two things: how to connect, and how to configure. ##### Topics and Pages to Cover The following subjects need to be addressed across three pages of this docs site to have your data platform be listed on our documentation. After the corresponding pull request is merged, we ask that you link to these pages from your adapter repo's `README` as well as from your product documentation. To contribute, all you will have to do make the changes listed in the table below. | How To... | File to change within `/website/docs/` | Action | Information to include | | -------------------- | --------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Connect | `/docs/local/connect-data-platform/{MY-DATA-PLATFORM}-setup.md` | Create | Give all information needed to define a target in `~/.dbt/profiles.yml` and get `dbt debug` to connect to the database successfully. All possible configurations should be mentioned. | | Configure | `reference/resource-configs/{MY-DATA-PLATFORM}-configs.md` | Create | What options and configuration specific to your data platform do users need to know? for example, table distribution and indexing options, column\_quoting policy, which incremental strategies are supported | | Discover and Install | `docs/community-adapters.md` | Modify | Add the adapter to the community adapters list. Include how to install the Python adapter package, ideally with pip and a PyPI package, or a `git+` link to the GitHub repo. | | Add link to sidebar | `website/sidebars.js` | Modify | Add the document id to the correct location in the sidebar menu | For example say I want to document my new adapter: `dbt-ders`. For the "Connect" page, I will make a new Markdown file, `ders-setup.md` and add it to the `/website/docs/local/connect-data-platform/` directory. ##### Example PRs to add new adapter documentation Below are some recent pull requests made by partners to document their data platform's adapter: * [TiDB](https://github.com/dbt-labs/docs.getdbt.com/pull/1309) * [SingleStore](https://github.com/dbt-labs/docs.getdbt.com/pull/1044) * [Firebolt](https://github.com/dbt-labs/docs.getdbt.com/pull/941) Note — Use the following re-usable component to auto-fill the frontmatter content on your new page: ```markdown import SetUpPages from '/snippets/_setup-pages-intro.md'; <SetUpPages meta={frontMatter.meta} /> ``` ##### Ongoing maintenance Keep the adapter usable after the first release. Cover the essential functionality of dbt v1, and support more of the feature set when you can. Essential functionality includes: * table, view, and seed materializations * dbt tests Document how to connect and configure the adapter on this site, and keep those pages up to date. Refer to [Topics and pages to cover](#topics-and-pages-to-cover) for which pages to add. For how to version releases and stay compatible with dbt v1, refer to [Versioning and releasing your adapter](https://docs.getdbt.com/guides/adapter-creation.md?step=2#versioning-and-releasing-your-adapter). ###### Stay current with dbt Read new releases of [dbt-adapters](https://github.com/dbt-labs/dbt-adapters) and support the features that apply to your platform. ###### Respond to the community When you can: * Answer questions and help people get started in the dbt Community Slack, in [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM) * Respond to issues in your public adapter repository * Review and merge community contributions when they are a good fit ###### Follow security practices Your adapter should not: * Write access credentials, or data from the underlying data platform, to logs or files * Make API calls other than those required for dbt features, or add extra logging * Hide code or behavior so it is hard to review To reduce supply-chain risk: * Use an automated service to keep Python dependencies up to date (for example, Dependabot) * Publish to PyPI from the adapter repository with a CI/CD process (for example, GitHub Actions) * Limit admin access to the code repository (GitHub) and the package repository (PyPI) * Scan for security vulnerabilities in CI/CD (for example, Snyk) Run the adapter test suite described in [Test your adapter](https://docs.getdbt.com/guides/adapter-creation.md?step=4). #### Promote a new adapter The most important thing here is recognizing that people are successful in the community when they join, first and foremost, to engage authentically. What does authentic engagement look like? It’s challenging to define explicit rules. One good rule of thumb is to treat people with dignity and respect. Contributors to the community should think of contribution *as the end itself,* not a means toward other business KPIs (leads, community members, and more). [We are a mission-driven company.](https://www.getdbt.com/dbt-labs/values/) Some ways to know if you’re authentically engaging: * Is an engagement’s *primary* purpose of sharing knowledge and resources or building brand engagement? * Imagine you didn’t work at the org you do — can you imagine yourself still writing this? * Is it written in formal / marketing language, or does it sound like you, the human? ##### Who should join the dbt community slack? * People who have insight into what it means to do hands-on [analytics engineering](https://www.getdbt.com/analytics-engineering/) work The dbt Community Slack workspace is fundamentally a place for analytics practitioners to interact with each other — the closer the users are in the community to actual data/analytics engineering work, the more natural their engagement will be (leading to better outcomes for partners and the community). * DevRel practitioners with strong focus DevRel practitioners often have a strong analytics background and a good understanding of the community. It’s essential to be sure they are focused on *contributing,* not on driving community metrics for partner org (such as signing people up for their slack or events). The metrics will rise naturally through authentic engagement. * Founder and executives who are interested in directly engaging with the community This is either incredibly successful or not at all depending on the profile of the founder. Typically, this works best when the founder has a practitioner-level of technical understanding and is interested in joining not to promote, but to learn and hear from users. * Software Engineers at partner products that are building and supporting integrations with either dbt v1 or the dbt platform This is successful when the engineers are familiar with dbt as a product or at least have taken our training course. The Slack is often a place where end-user questions and feedback is initially shared, so it is recommended that someone technical from the team be present. There are also a handful of channels aimed at those building integrations, which tend to be a font of knowledge. ##### Who might struggle in the dbt community * People in marketing roles dbt Slack is not a marketing channel. Attempts to use it as such invariably fall flat and can even lead to people having a negative view of a product. This doesn’t mean that dbt can’t serve marketing objectives, but a long-term commitment to engagement is the only proven method to do this sustainably. * People in product roles The dbt Community can be an invaluable source of feedback on a product. There are two primary ways this can happen — organically (community members proactively suggesting a new feature) and through direct calls for feedback and user research. Immediate calls for engagement must be done in your dedicated #tools channel. Direct calls should be used sparingly, as they can overwhelm more organic discussions and feedback. ##### Who is the audience for an adapter release? A new adapter is likely to drive huge community interest from several groups of people: * People who are currently using the database that the adapter is supporting * People who may be adopting the database in the near future. * People who are interested in dbt development in general. The database users will be your primary audience and the most helpful in achieving success. Engage them directly in the adapter’s dedicated Slack channel. If one does not exist already, reach out in #channel-requests, and we will get one made for you and include it in an announcement about new channels. The final group is where non-slack community engagement becomes important. Twitter and LinkedIn are both great places to interact with a broad audience. A well-orchestrated adapter release can generate impactful and authentic engagement. ##### How to message the initial rollout and follow-up content Tell a story that engages dbt users and the community. Highlight new use cases and functionality unlocked by the adapter in a way that will resonate with each segment. * Existing users of your technology who are new to dbt * Provide a general overview of the value dbt will deliver to your users. This can lean on dbt's messaging and talking points which are laid out in the [dbt viewpoint.](https://docs.getdbt.com/community/resources/viewpoint.md) * Give examples of a rollout that speaks to the overall value of dbt and your product. * Users who are already familiar with dbt and the community * Consider unique use cases or advantages your adapter provide over existing adapters. Who will be excited for this? * Contribute to the dbt Community and ensure that dbt users on your adapter are well supported (tutorial content, packages, documentation, and more). * Example of a rollout that is compelling for those familiar with dbt: [Firebolt](https://www.linkedin.com/feed/update/urn:li:activity:6879090752459182080/) ##### Tactically manage distribution of content about new or existing adapters There are tactical pieces on how and where to share that help ensure success. * On slack: * \#i-made-this channel — this channel has a policy against “marketing” and “content marketing” posts, but it should be successful if you write your content with the above guidelines in mind. Even with that, it’s important to post here sparingly. * Your own database / tool channel — this is where the people who have opted in to receive communications from you and always a great place to share things that are relevant to them. * On social media: * Twitter * LinkedIn * Social media posts *from the author* or an individual connected to the project tend to have better engagement than posts from a company or organization account. * Ask your partner representative about: * Retweets and shares from the official dbt Labs accounts. * Flagging posts internally at dbt Labs to get individual employees to share. ###### Measuring engagement You don’t need 1000 people in a channel to succeed, but you need at least a few active participants who can make it feel lived in. If you’re comfortable working in public, this could be members of your team, or it can be a few people who you know that are highly engaged and would be interested in participating. Having even 2 or 3 regulars hanging out in a channel is all that’s needed for a successful start and is, in fact, much more impactful than 250 people that never post. ##### How to announce a new adapter We’d recommend *against* boilerplate announcements and encourage finding a unique voice. That being said, there are a couple of things that we’d want to include: * A summary of the value prop of your database / technology for users who aren’t familiar. * The personas that might be interested in this news. * A description of what the adapter *is*. For example: > With the release of our new dbt adapter, you’ll be able to use dbt to model and transform your data in \[name-of-your-org] * Particular or unique use cases or functionality unlocked by the adapter. * Plans for future / ongoing support / development. * The link to the documentation for using the adapter on the dbt Labs docs site. * An announcement blog. ###### Announcing new release versions of existing adapters This can vary substantially depending on the nature of the release but a good baseline is the types of release messages that [we put out in the #dbt-releases](https://getdbt.slack.com/archives/C37J8BQEL/p1651242161526509) channel. [![Full release post](/img/adapter-guide/0-full-release-notes.png?v=2 "Full release post")](#)Full release post Breaking this down: * Visually distinctive announcement - make it clear this is a release [![title](/img/adapter-guide/1-announcement.png?v=2 "title")](#)title * Short written description of what is in the release [![description](/img/adapter-guide/2-short-description.png?v=2 "description")](#)description * Links to additional resources [![more resources](/img/adapter-guide/3-additional-resources.png?v=2 "more resources")](#)more resources * Implementation instructions: [![more installation](/img/adapter-guide/4-installation.png?v=2 "more installation")](#)more installation * Contributor recognition (if applicable) [![thank yous](/img/adapter-guide/6-thank-contribs.png?v=2 "thank yous")](#)thank yous --- ### Building dbt packages [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Creating packages is an **advanced use of dbt**. If you're new to the tool, we recommend that you first use the product for your own analytics before attempting to create a package for others. ##### Prerequisites A strong understanding of: * [packages](https://docs.getdbt.com/docs/build/packages.md) * administering a repository on GitHub * [semantic versioning](https://semver.org/) ##### Assess whether a package is the right solution Packages typically contain either: * macros that solve a particular analytics engineering problem — for example, [auditing the results of a query](https://hub.getdbt.com/dbt-labs/audit_helper/latest/), [generating code](https://hub.getdbt.com/dbt-labs/codegen/latest/), or [adding additional schema tests to a dbt project](https://hub.getdbt.com/calogica/dbt_expectations/latest/). * models for a common dataset — for example a dataset for software products like [MailChimp](https://hub.getdbt.com/fivetran/mailchimp/latest/) or [Snowplow](https://hub.getdbt.com/dbt-labs/snowplow/latest/), or even models for metadata about your data stack like [Snowflake query spend](https://hub.getdbt.com/gitlabhq/snowflake_spend/latest/) and [the artifacts produced by `dbt run`](https://hub.getdbt.com/tailsdotcom/dbt_artifacts/latest/). In general, there should be a shared set of industry-standard metrics that you can model (e.g. email open rate). We also recommend ensuring your package is compatible with [dbt v2](https://docs.getdbt.com/docs/introduction.md) and [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). To ensure dbt v2 compatibility, you can follow the steps in the [v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md). Packages are *not* a good fit for sharing models that contain business-specific logic, for example, writing code for marketing attribution, or monthly recurring revenue. Instead, consider sharing a blog post and a link to a sample repo, rather than bundling this code as a package (here's our blog post on [marketing attribution](https://blog.getdbt.com/modeling-marketing-attribution/) as an example). #### Create your new project Using the command line for package development We tend to use the command line interface for package development. The development workflow often involves installing a local copy of your package in another dbt project — at present dbt is not designed for this workflow. 1. Use the [dbt init](https://docs.getdbt.com/reference/commands/init.md) command to create a new dbt project, which will be your package: ```shell $ dbt init [package_name] ``` 2. Create a public GitHub¹ repo, named `dbt-<package-name>`, e.g. `dbt-mailchimp`. Follow the GitHub instructions to link this to the dbt project you just created. 3. Update the `name:` of the project in `dbt_project.yml` to your package name, e.g. `mailchimp`. 4. Define the allowed dbt versions by using the [`require-dbt-version` config](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md). ¹Currently, our package registry only supports packages that are hosted in GitHub. #### Develop your package We recommend that first-time package authors first develop macros and models for use in their own dbt project. Once your new package is created, you can get to work on moving them across, implementing some additional package-specific design patterns along the way. When working on your package, we often find it useful to install a local copy of the package in another dbt project — this workflow is described [here](https://discourse.getdbt.com/t/contributing-to-an-external-dbt-package/657). ##### Ensure dbt v2 compatibility If you're building a package, we recommend you ensure it's compatible with [dbt v2](https://docs.getdbt.com/docs/introduction.md) and [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). To ensure dbt v2 compatibility, you can follow the steps in the [v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md). Doing so will ensure your package is compatible with dbt v2 (and dbt v1), but will be displayed with a dbt v2-compatible badge in dbt package hub. ##### Follow best practices *Modeling packages only* Use our [dbt coding conventions](https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md), our article on [how we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md), and our [best practices](https://docs.getdbt.com/best-practices.md) for all of our advice on how to build your dbt project. This is where it comes in especially handy to have worked on your own dbt project previously. ##### Make the location of raw data configurable *Modeling packages only* Not every user of your package is going to store their Mailchimp data in a schema named `mailchimp`. As such, you'll need to make the location of raw data configurable. We recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and [variables](https://docs.getdbt.com/docs/build/project-variables.md) to achieve this. Check out [this package](https://github.com/fivetran/dbt_facebook_ads_source/blob/main/models/src_facebook_ads.yml#L5-L6) for an example — notably, the README [includes instructions](https://github.com/fivetran/dbt_facebook_ads_source#configuration) on how to override the default schema from a `dbt_project.yml` file. ##### Install upstream packages from hub.getdbt.com If your package relies on another package (for example, you use some of the cross-database macros from [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/)), we recommend you install the package from [hub.getdbt.com](https://hub.getdbt.com), specifying a version range like so: packages.yml ```yaml packages: - package: dbt-labs/dbt_utils version: [">0.6.5", "0.7.0"] ``` When packages are installed from hub.getdbt.com, dbt is able to handle duplicate dependencies. ##### Implement cross-database compatibility Many SQL functions are specific to a particular database. For example, the function name and order of arguments to calculate the difference between two dates varies between Redshift, Snowflake and BigQuery, and no similar function exists on Postgres! If you wish to support multiple warehouses, we have a number of tricks up our sleeve: * We've written a number of macros that compile to valid SQL snippets on each of the original four adapters. Where possible, leverage these macros. * If you need to implement cross-database compatibility for one of your macros, use the [`adapter.dispatch` macro](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) to achieve this. Check out the cross-database macros in dbt-utils for examples. * If you're working on a modeling package, you may notice that you need write different models for each warehouse (for example, if the EL tool you are working with stores data differently on each warehouse). In this case, you can write different versions of each model, and use the [`enabled` config](https://docs.getdbt.com/reference/resource-configs/enabled.md), in combination with [`target.type`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) to enable the correct models — check out [this package](https://github.com/fivetran/dbt_facebook_ads_creative_history/blob/main/dbt_project.yml#L11-L16) as an example. If your package has only been written to work for one data warehouse, make sure you document this in your package README. ##### Use specific model names *Modeling packages only* Many datasets have a concept of a "user" or "account" or "session". To make sure things are unambiguous in dbt, prefix all of your models with `[package_name]_`. For example, `mailchimp_campaigns.sql` is a good name for a model, whereas `campaigns.sql` is not. ##### Default to views *Modeling packages only* dbt makes it possible for users of your package to override your model materialization settings. In general, default to materializing models as `view`s instead of `table`s. The major exception to this is when working with data sources that benefit from incremental modeling (for example, web page views). Implementing incremental logic on behalf of your end users is likely to be helpful in this case. ##### Test and document your package It's critical that you [test](https://docs.getdbt.com/docs/build/data-tests.md) your models and sources. This will give your end users confidence that your package is actually working on top of their dataset as intended. Further, adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) via descriptions will help communicate your package to end users, and benefit their stakeholders that use the outputs of this package. ##### Include useful GitHub artifacts Over time, we've developed a set of useful GitHub artifacts that make administering our packages easier for us. In particular, we ensure that we include: * A useful README, that has: * installation instructions that refer to the latest version of the package on hub.getdbt.com, and includes any configurations requires ([example](https://github.com/dbt-labs/segment)) * Usage examples for any macros ([example](https://github.com/dbt-labs/dbt-audit-helper#macros)) * Descriptions of the main models included in the package ([example](https://github.com/dbt-labs/snowplow)) * GitHub templates, including PR templates and issue templates ([example](https://github.com/dbt-labs/dbt-audit-helper/tree/master/.github)) #### Add integration tests *Optional* We recommend that you implement integration tests to confirm that the package works as expected — this is an even *more* advanced step, so you may find that you build up to this. This pattern can be seen most packages, including the [`audit-helper`](https://github.com/dbt-labs/dbt-audit-helper/tree/master/integration_tests) and [`snowplow`](https://github.com/dbt-labs/snowplow/tree/master/integration_tests) packages. As a rough guide: 1. Create a subdirectory named `integration_tests` 2. In this subdirectory, create a new dbt project — you can use the `dbt init` command to do this. However, our preferred method is to copy the files from an existing `integration_tests` project, like the ones [here](https://github.com/dbt-labs/dbt-codegen/tree/HEAD/integration_tests) (removing the contents of the `macros`, `models` and `tests` folders since they are project-specific) 3. Install the package in the `integration_tests` subdirectory by using the `local` syntax, and then running `dbt deps` packages.yml ```yml packages: - local: ../ # this means "one directory above the current directory" ``` 4. Add resources to the package (seeds, models, tests) so that you can successfully run your project, and compare the output with what you expect. The exact approach here will vary depending on your packages. In general you will find that you need to: * Add mock data via a [seed](https://docs.getdbt.com/docs/build/seeds.md) with a few sample (anonymized) records. Configure the `integration_tests` project to point to the seeds instead of raw data tables. * Add more seeds that represent the expected output of your models, and use the [dbt\_utils.equality](https://github.com/dbt-labs/dbt-utils#equality-source) test to confirm the output of your package, and the expected output matches. 5. Confirm that you can run `dbt run` and `dbt test` from your command line successfully. 6. (Optional) Use a CI tool, like CircleCI or GitHub Actions, to automate running your dbt project when you open a new Pull Request. For inspiration, check out one of our [CircleCI configs](https://github.com/dbt-labs/snowplow/blob/main/.circleci/config.yml), which runs tests against our four main warehouses. Note: this is an advanced step — if you are going down this path, you may find it useful to say hi on [dbt Slack](https://community.getdbt.com/). #### Deploy the docs for your package *Optional* A dbt docs site can help a prospective user of your package understand the code you've written. As such, we recommend that you deploy the site generated by `dbt docs generate` and link to the deployed site from your package. The easiest way we've found to do this is to use [GitHub Pages](https://pages.github.com/). 1. On a new git branch, run `dbt docs generate`. If you have integration tests set up (above), use the integration-test project to do this. 2. Move the following files into a directory named `docs` ([example](https://github.com/fivetran/dbt_ad_reporting/tree/HEAD/docs)): `catalog.json`, `index.html`, `manifest.json`, `run_results.json`. 3. Merge these changes into the main branch 4. Enable GitHub pages on the repo in the settings tab, and point it to the “docs” subdirectory 5. GitHub should then deploy the docs at `<org-name>.github.io/<repo-name>`, like so: [fivetran.github.io/dbt\_ad\_reporting](https://fivetran.github.io/dbt_ad_reporting/) #### Release your package Create a new [release](https://docs.github.com/en/github/administering-a-repository/managing-releases-in-a-repository) once you are ready for others to use your work! Be sure to use [semantic versioning](https://semver.org/) when naming your release. In particular, if new changes will cause errors for users of earlier versions of the package, be sure to use *at least* a minor release (e.g. go from `0.1.1` to `0.2.0`). The release notes should contain an overview of the changes introduced in the new version. Be sure to call out any changes that break the existing interface! #### Add the package to hub.getdbt.com Our package registry, [hub.getdbt.com](https://hub.getdbt.com/), gets updated by the [hubcap script](https://github.com/dbt-labs/hubcap). To add your package to hub.getdbt.com, create a PR on the [hubcap repository](https://github.com/dbt-labs/hubcap) to include it in the `hub.json` file. --- ### Clone the Jaffle Shop sample project [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### What is Jaffle Shop? [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) is dbt Labs' canonical sample dbt project for a fictional cafe business. This guide shows you how to clone the project using Git from GitHub or GitLab. Cloning downloads the dbt project files to your machine: models, seeds, tests, and configuration. It doesn't install dbt or set up a database. If you only need the project files, follow the [Clone the repository](#clone-the-repository) and [Verify the clone](#verify-the-clone) sections. To run dbt commands like `dbt seed`, `dbt run`, and `dbt test`, you also need dbt installed and a database to connect to. Refer to [Next steps](#next-steps) for links to warehouse quickstarts. ##### Related content * [Install dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) * [About dbt projects](https://docs.getdbt.com/docs/build/projects.md) * [Example dbt projects](https://docs.getdbt.com/faqs/Project/example-projects.md) #### Prerequisites * [Git](https://git-scm.com/downloads) installed * A terminal * A [GitHub](https://github.com/join) or [GitLab](https://gitlab.com/users/sign_up) account (only if you plan to fork the repo, push changes, or clone from a private repository) Verify Git is installed: ```bash git --version ``` #### Clone the repository ##### GitHub 1. Open your terminal and navigate to where you keep projects: ```bash cd ~/Documents/Github ``` 2. Clone the repository: ```bash git clone https://github.com/dbt-labs/jaffle-shop.git ``` 3. Change into the project directory: ```bash cd jaffle-shop ``` ##### GitLab If your organization hosts Jaffle Shop on GitLab, or you've forked the repo there, clone from your GitLab URL instead: 1. Open your terminal and navigate to where you keep projects: ```bash cd ~/Documents/Github ``` 2. Clone the repository (replace `YOUR_USERNAME` with your GitLab username or group): ```bash git clone https://gitlab.com/YOUR_USERNAME/jaffle-shop.git ``` 3. Change into the project directory: ```bash cd jaffle-shop ``` ##### Other platforms If your organization has mirrored or forked Jaffle Shop on Bitbucket, Azure DevOps, or another Git host, use the clone URL from that platform. The `git clone` command works the same way: 1. Open your terminal and navigate to where you keep projects: ```bash cd ~/Documents/Github ``` 2. Clone the repository: ```bash git clone <your-repo-clone-url> ``` 3. Change into the project directory: ```bash cd jaffle-shop ``` #### Verify the clone Confirm the project files are present: ```bash ls ``` You should see files and folders including: * `dbt_project.yml` * `models/` * `seeds/` * `packages.yml` Cloning is complete! 🎉 You now have the Jaffle Shop project files on your machine. #### Explore the project structure Before you install dbt or connect a database, take a quick look at what you cloned. Jaffle Shop is a fictional cafe business. The project contains 13 models that transform cafe data about customers, locations (stores), products, supplies, and orders. It also includes: * **Seeds:** Six CSV files under `seeds/jaffle-data/` that provide the raw sample data. * **Data tests and unit tests:** YAML alongside the models that check uniqueness, not-null values, and a few unit-test cases. * **Macros:** Small helper macros in `macros/`. * **Packages:** Dependencies listed in `packages.yml` (install later with `dbt deps`). A simplified view of the project looks like this: ```text jaffle-shop/ ├── dbt_project.yml ├── packages.yml ├── models/ │ ├── staging/ # Clean and rename raw tables │ │ ├── stg_customers.sql │ │ ├── stg_orders.sql │ │ ├── stg_order_items.sql │ │ ├── stg_products.sql │ │ ├── stg_locations.sql │ │ ├── stg_supplies.sql │ │ └── __sources.yml │ └── marts/ # Business-ready tables for the cafe │ ├── customers.sql │ ├── orders.sql │ ├── order_items.sql │ ├── products.sql │ ├── locations.sql │ ├── supplies.sql │ └── metricflow_time_spine.sql ├── seeds/ │ └── jaffle-data/ # Sample CSV data (customers, orders, and more) ├── macros/ ├── analyses/ └── data-tests/ ``` Staging models sit closest to the raw seed data. Marts models join and shape that data into the tables you use for analysis. You don't need to read every file yet. This layout follows a standard [dbt project](https://docs.getdbt.com/docs/build/projects.md) pattern that appears in many real-world projects. #### Next steps To run or develop the project, you need dbt installed and a database connected. These links can help: * **[Install self-hosted dbt](https://docs.getdbt.com/docs/local/install-dbt.md):** Cloning doesn't install dbt. You need it to run commands. * **[Set up a virtual environment](https://docs.getdbt.com/docs/local/install-dbt.md):** Keeps dbt separate from other Python projects on your machine. * **[About dbt deps command](https://docs.getdbt.com/reference/commands/deps.md):** The repo lists packages in `packages.yml`. Run `dbt deps` after you install dbt. * **[About dbt projects](https://docs.getdbt.com/docs/build/projects.md):** Learn the project structure before editing models. * **[dbt Learn](https://learn.getdbt.com/):** Interactive courses for new users. ##### Warehouse quickstarts To run the project, you need a database and adapter configured in `profiles.yml`. Choose the quickstart for your warehouse or local setup: * [Quickstart for dbt and Snowflake](https://docs.getdbt.com/guides/snowflake.md) * [Quickstart for dbt and BigQuery](https://docs.getdbt.com/guides/bigquery.md) * [Quickstart for dbt and Databricks](https://docs.getdbt.com/guides/databricks.md) * [Quickstart for dbt and Redshift](https://docs.getdbt.com/guides/redshift.md) * [Quickstart for dbt v1 using DuckDB](https://docs.getdbt.com/guides/duckdb.md): Clone [`jaffle_shop_duckdb`](https://github.com/dbt-labs/jaffle_shop_duckdb) and follow the guide. You can also browse [all guides](https://docs.getdbt.com/guides.md) or other [example dbt projects](https://docs.getdbt.com/faqs/Project/example-projects.md). #### Optional cleanup If you cloned the repo only to test these steps and don't need the project anymore, you can remove the folder: ```bash rm -rf ~/Documents/Github/jaffle-shop ``` This deletes the cloned project from your machine. It doesn't affect the GitHub or GitLab repository. --- ### Connect dbt MCP to dbt platform [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() This quickstart uses the local MCP server: it runs on your machine using `uvx dbt-mcp`, connects to your dbt platform for Semantic Layer, Discovery, and SQL, and optionally runs local dbt. For self-hosted CLI only (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) or [Run self-hosted dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). To configure or disable specific tools, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) * A [dbt platform account](https://www.getdbt.com/signup) * For OAuth connections: * MCP OAuth is available for Starter, Enterprise, and Enterprise+ plans. #### Step 1: Choose your auth method and configure ##### OAuth *MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* OAuth is the fastest setup for dbt platform accounts, no tokens to copy or manage. A browser window opens to authenticate the first time you connect. For OAuth *without* a self-hosted installation, use the [remote MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md). Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. If your client does not support OAuth or you need token-based access, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication). Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. ###### Find your Access URL 1. Log in to your dbt platform account. 2. Go to **Account settings** and copy your **Access URL** (for example, `abc123.us1.dbt.com`). Multi-cell and DBT\_HOST format * The `DBT_HOST` field accepts both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com`. * If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ###### Add the config to your MCP client ##### Claude Desktop **Option A: Quick install (recommended)** 1. Go to the [latest dbt MCP release](https://github.com/dbt-labs/dbt-mcp/releases/latest) and download `dbt-mcp.mcpb`. 2. Double-click the file to open it in Claude Desktop. 3. Enter your **Access URL** as the dbt platform Host. 4. Enable the server. **Option B: Manual config** 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing `YOUR-ACCESS-URL` with your Access URL: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing `YOUR-ACCESS-URL` with your Access URL: ```shell claude mcp add dbt \ -e DBT_HOST=YOUR-ACCESS-URL \ -- uvx dbt-mcp ``` For example, if your Access URL is `abc123.us1.dbt.com`: ```shell claude mcp add dbt \ -e DBT_HOST=abc123.us1.dbt.com \ -- uvx dbt-mcp ``` ##### Cursor Click a link below with Cursor open to auto-configure, then replace the placeholder with your Access URL: * [dbt platform only (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJESVNBQkxFX0RCVF9DTEkiOiJ0cnVlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features only, no CLI * [dbt platform + CLI (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJEQlRfUFJPSkVDVF9ESVIiOiIvcGF0aC90by9wcm9qZWN0IiwiREJUX1BBVEgiOiJwYXRoL3RvL2RidC9leGVjdXRhYmxlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features + self-hosted dbt CLI commands After clicking, replace `YOUR-ACCESS-URL` with your actual Access URL (for example, `abc123.us1.dbt.com`) and save. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` Replace `YOUR-ACCESS-URL` with your Access URL (for example, `abc123.us1.dbt.com`) and save. ###### Optional: Add self-hosted dbt CLI commands To also run dbt platform CLI commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. ##### Tokens Token-based auth gives you more control and is better for shared or team setups. You'll need a service token or Personal Access Token (PAT). Which token should I use? * **PAT (Personal Access Token):** Required if you want to use `execute_sql`. Tied to your user account. * **Service token:** Works for all other platform toolsets. Better for shared or team setups. See [Choosing an auth method](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#choose-your-auth-method) for full guidance. ##### Find your paths and IDs You need the following values. See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. | Variable | Where to find it | | ----------------- | -------------------------------------------------------------------------- | | `DBT_HOST` | Your dbt platform hostname, found in **Account settings** → **Access URL** | | `DBT_TOKEN` | A service token or PAT from **Account settings** → **API tokens** | | `DBT_PROD_ENV_ID` | Your production environment ID, found in **Deploy** → **Environments** | | `DBT_DEV_ENV_ID` | Your development environment ID (required for `execute_sql`) | | `DBT_USER_ID` | Your numeric user ID (required for `execute_sql`) | | `DBT_ACCOUNT_ID` | Your account ID (required for Admin API tools) | Use values only, not full URLs These variables expect hostnames or numeric IDs — not full URLs: ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` Multi-cell accounts If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ##### Add the config to your MCP client Use the configuration below, replacing the placeholder values with your IDs from above. Include only the variables you need: ##### Claude Desktop 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing the placeholders with your actual values: ```bash claude mcp add dbt \ -e DBT_HOST=cloud.getdbt.com \ -e DBT_TOKEN=your-token-here \ -e DBT_PROD_ENV_ID=12345 \ -- uvx dbt-mcp ``` Add `-e DBT_DEV_ENV_ID=...` and `-e DBT_USER_ID=...` if you use `execute_sql`; add `-e DBT_ACCOUNT_ID=...` for Admin API. ##### Cursor 1. In Cursor, open **Settings** → **MCP** → **Edit config** (or your config file). 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save the configuration. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 4. Save `mcp.json` and restart VS Code.  Optional: add self-hosted dbt CLI commands To also run dbt commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. #### Step 2: Authenticate ##### OAuth The first time you connect, dbt MCP opens a browser window to complete OAuth. After signing in, your session is saved and future connections are automatic. If authentication doesn't start, close your client and run: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` Then restart your client. ##### Tokens No additional authentication step is needed — your token is already in the configuration. The server connects automatically when your MCP client starts. #### Step 3: Test your setup Ask your AI assistant a data-related question (for example, *"What models are in my dbt project?"* or *"What metrics are defined in my Semantic Layer?"*). If dbt MCP is working, the response will use your dbt metadata. #### What's available With the platform setup, your AI assistant can use: * Semantic Layer queries * Metadata Discovery (model lineage, test results, source freshness) * Admin API (trigger jobs, list runs, get artifacts) * SQL execution and text-to-SQL (requires a [PAT](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md)) * All dbt commands if you added `DBT_PROJECT_DIR` and `DBT_PATH` For the complete tool list, see [Available tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md). Looking for self-hosted dbt CLI only? If you only need to run dbt commands locally (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md). #### Troubleshooting Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). execute\_sql tool not working **Symptoms:** The `execute_sql` tool returns an authentication error or is unavailable. **Cause:** How you fix this depends on how you connected: * **Remote MCP with OAuth:** Sign in through your MCP client when prompted. You don't need a personal access token. * **Remote MCP with token-based auth:** You need a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. You also need `x-dbt-dev-environment-id` and `x-dbt-user-id` headers. The user ID must match the user who created the PAT. * **Local MCP:** Set `DBT_TOKEN` to a personal access token (PAT). Service tokens won't work. **Solution:** 1. If you use OAuth for remote MCP, confirm you completed sign-in and consent in your MCP client. 2. If you use token-based auth or local MCP, create a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) in **Account settings** → **API tokens** → **Personal tokens**. 3. For token-based remote MCP, add the PAT to the `Authorization` header as `Token YOUR_PAT`. For local MCP, set the PAT as your `DBT_TOKEN` value. 4. Also ensure `DBT_DEV_ENV_ID` and `DBT_USER_ID` are set (local MCP) or `x-dbt-dev-environment-id` and `x-dbt-user-id` headers are set (remote MCP). Refer to [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. Toolset unavailable or showing as disabled **Symptoms:** A toolset (Semantic Layer, Discovery, Admin API) is not available in your AI client even though you've configured credentials. **Cause:** Either the required variables are missing, or the toolset has been explicitly disabled. **Solution:** 1. Check that all required variables for the toolset are set — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). 2. Check whether you have any `DISABLE_*` variables set to `true` that might be turning off the toolset. 3. If you're using enable mode (`DBT_MCP_ENABLE_*`), make sure the toolset you need is listed. 4. Set `DBT_MCP_LOG_LEVEL=DEBUG` to see which toolsets are active at startup. Pasting full URLs instead of IDs **Symptoms:** Authentication errors, unexpected behavior, or the server failing to connect to the right environment. **Cause:** Environment variables like `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_ACCOUNT_ID` expect numeric integers, not full browser URLs. **Solution:** ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. Multi-cell or static subdomain account connection issues **Symptoms:** Connection errors when your account URL includes a prefix (for example, `abc123.us1.dbt.com`). **Solution (as of v1.14.0):** Set `DBT_HOST` to the full hostname including the prefix. If you're using PAT-based auth, also set `DBT_ACCOUNT_ID`. ```bash # ✅ Correct DBT_HOST=abc123.us1.dbt.com DBT_ACCOUNT_ID=12345 # required for PAT-based auth ``` You no longer need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. If you have these set from an older configuration, remove them. For all troubleshooting topics, see [dbt MCP FAQs and troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). #### Next steps * Run dbt commands locally: see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) * Configure specific toolsets: see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) * Understand toolset requirements: see [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance) --- ### Contribute a dbt v2 adapter [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() info This guide is for dbt v2. For dbt v1, see [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md). #### Step 1: Introduction dbt v2 adapters work very differently from v1, and it's worth understanding why before diving in. In dbt v1, every adapter was a **standalone Python package**, independently maintained by the community. That model didn't scale: maintenance cost grew with every new warehouse added, bugs had to be fixed separately in each package, and shared improvements rarely made it back across the ecosystem. dbt v2 flips this entirely. Adapters now live **inside a single Rust monorepo**, organized by feature area rather than by warehouse. A fix in the auth module benefits every adapter at once. When you contribute a new adapter, you're extending shared code, not building a whole new package from scratch. In Rust terms: adding arms to *existing* match expressions. The more adapters that exist, the easier each new adapter becomes. Your contribution makes the whole ecosystem better. Another advantage of dbt v2: connection management lives in its own lane outside of the adapter. In v1, each adapter had to own its connection logic, wrapping vendor SDKs or implementing the Python DB API spec. **In v2, ADBC drivers handle that responsibility**: pre-compiled binaries that you register but don't write. For a full explanation of ADBC and how dbt v2 uses it, see [ADBC in dbt v2](https://docs.getdbt.com/docs/dbt/adbc.md?version=2.0). What you need to contribute is the exact logic that varies by warehouse: credentials, relation naming, macros, and catalog queries. A complete community adapter touches ~13 files; that's the bar we're aiming for. This guide walks you through the architecture, crate structure, and how to contribute a dbt v2 adapter step by step. I’m Hope Watson, product manager for dbt v2 adapters at dbt Labs, and I put this together to help design, document, and shepherd this process for the community in the v2 world. The Exasol adapter, contributed by [Marco Nätlitz](https://www.linkedin.com/in/marco-naetlitz/), is used as the reference example throughout. The full file breakdown in the [reference section](#reference-file-by-file-implementation-guide) defines exactly what you're building. ##### How v1 vs. v2 adapters differ | Topic | dbt v1 | dbt v2 | | ------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Package structure** | Separate Python packages (for example, `dbt-snowflake` and `dbt-bigquery`), each inheriting from `dbt-adapters` | Part of the `dbt-labs/dbt` monorepo, contributed as PRs | | **Who maintains them** | dbt Labs maintains base adapter and 6 major adapters; others community-maintained independently | dbt Labs maintains the monorepo; community contributes through pull requests | | **Language** | Python | Rust (with Jinja SQL macros still used for SQL logic) | | **How they load** | Dynamically loaded at runtime through Python's plugin system | Compiled into the monorepo binary | | **Connection management** | Each adapter implements its own connection logic, built on vendor Python SDK connectors/drivers | ADBC drivers handle connection management: a unified driver interface that abstracts away wire protocol details | | **Code organization** | Organized by warehouse, each adapter is its own isolated codebase | Organized by feature area (vertical): a fix in one crate benefits all adapters | ##### What you're building As a community contributor, you're building the foundation: connect your warehouse, run dbt macros, and support basic materializations. Once your adapter is in place, `dbt build` and `dbt run` should work. The ~13 files in the [reference section](#reference-file-by-file-implementation-guide) define the complete scope. Think of your adapter contribution as the bridge between dbt v2 and your warehouse: everything dbt needs to talk to it, nothing more. The architecture diagram below shows where your work fits in the stack. [![dbt v2 adapter architecture diagram showing the community contribution zone within the monorepo, the ADBC C ABI boundary, the ADBC driver, and the warehouse](/img/adapter-guide/adapter-creation-v2-architecture.svg?v=2 "dbt v2 adapter architecture diagram showing the community contribution zone within the monorepo, the ADBC C ABI boundary, the ADBC driver, and the warehouse")](#)dbt v2 adapter architecture diagram showing the community contribution zone within the monorepo, the ADBC C ABI boundary, the ADBC driver, and the warehouse #### Step 2: Prerequisites **Rust familiarity**: You don't need to be a Rust expert. You need to read Rust, understand enum match expressions, and make sense of compiler errors. If Rust is new to you, skim [The Rust Book](https://doc.rust-lang.org/book/) Chapter 6 (enums and pattern matching). Those are the concepts you'll encounter most. **dbt fundamentals**: Understand how [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) works, what [materializations](https://docs.getdbt.com/docs/build/materializations.md) are, and what [adapter dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) macros do. **Deep knowledge of your warehouse**: As you work through this guide, you'll need to know how your warehouse behaves: * What character does it use for quoting identifiers? * Does it support two-part or three-part relation names? * What connection parameters does it require? * How does it handle transactions? * What datatypes map to standard SQL types like string, timestamp, boolean? * What's the most efficient way to list columns in a schema? **A note on AI-assisted development**: dbt v2's architecture is naturally legible to AI coding tools. Match arms are explicit, compiler errors are precise, and the scope of each file is narrow. You don't need to use an LLM, but if you do, the structure works in your favor. ##### Your ADBC driver *Do you have an ADBC driver for your warehouse? If no, stop here.* The ADBC driver is a firm requirement before creating a dbt v2 adapter. Check these sources to find out if one exists for your warehouse: * [apache/arrow-adbc](https://github.com/apache/arrow-adbc): upstream community repo * [dbt-labs/arrow-adbc](https://github.com/dbt-labs/arrow-adbc): the dbt Labs fork * [ADBC driver foundry](https://github.com/adbc-drivers): primary community home for ADBC drivers, with active drivers and Rust/Go frameworks for building new ones * Your warehouse vendor's own GitHub org If no driver exists yet, building one is a separate project that comes before the adapter contribution. This is outside the scope of dbt Labs. [Columnar](https://columnar.tech/) specializes in building ADBC drivers and may be a useful resource. Long-term: signed driver distribution dbt Labs is working toward a model where community drivers can be distributed and loaded safely through signed binaries, similar to how operating systems verify installers. The goal is to make third-party driver installation as seamless and trustworthy as first-party. This isn't in place yet, but dbt Labs is considering this direction. #### Step 3: Understanding the codebase shape before you build ##### The vertical model dbt v2 organizes adapter logic by *capability*, not by warehouse. There's no `SnowflakeAdapter` class, no `BigQueryAdapter` class. Instead, a single shared codebase handles all warehouses, with warehouse-specific behavior driven by `match adapter_type()` expressions. When you add a new warehouse, you're adding arms to those expressions across a set of feature areas, auth, relations, catalog introspection, Jinja macros, rather than creating a new top-level package. The structural diagram below shows how the layers fit together. Your work lives in the **ADAPTER** layer: the crates that handle warehouse identity, credential resolution, relation logic, and Jinja dispatch. XDBC (driver loading and connection pooling) sits just below and has a small registration step too. [![dbt v2 system layers diagram showing USER, CLI, PIPELINE, TASK RUNNER, ADAPTER (highlighted as the contribution zone), XDBC, and WAREHOUSE layers](/img/adapter-guide/adapter-creation-v2-structural.svg?v=2 "dbt v2 system layers diagram showing USER, CLI, PIPELINE, TASK RUNNER, ADAPTER (highlighted as the contribution zone), XDBC, and WAREHOUSE layers")](#)dbt v2 system layers diagram showing USER, CLI, PIPELINE, TASK RUNNER, ADAPTER (highlighted as the contribution zone), XDBC, and WAREHOUSE layers ##### Crate map In Rust, a **crate** is a package: the unit of compilation, roughly equivalent to a "library" or "module" in other languages. The `dbt-labs/dbt` repo is a monorepo of multiple crates, each responsible for one vertical slice of functionality across all warehouses. This is a quick-reference map of the six crates you'll touch to build your adapter, in the order you'll work through them in Step 5. | Crate | Location | What you touch | | ------------------ | -------------------------- | ------------------------------------------------------------------------------------------ | | `dbt-adapter-core` | `crates/dbt-adapter-core/` | `AdapterType` enum variant, `quote_char` arm | | `dbt-xdbc` | `crates/dbt-xdbc/` | ADBC driver registration (`Backend` enum, library name) | | `dbt-schemas` | `crates/dbt-schemas/` | `profiles.yml` config struct (`DbConfig` variant) | | `dbt-auth` | `crates/dbt-auth/` | Credential resolution, connection URI construction | | `dbt-adapter` | `crates/dbt-adapter/` | Relation quoting, metadata catalog queries, adapter match arms, column builder, sql\_types | | `dbt-loader` | `crates/dbt-loader/` | Jinja SQL macros (`dbt_macro_assets/dbt-<adapter>/`) | ##### Where your adapter fits in the execution model By the time your adapter is invoked, all the dbt work is already done. dbt has resolved `ref()` and `source()` calls, rendered every Jinja template, and produced dialect-specific SQL. Your adapter receives a finished string and executes it. The adapter never sees un-rendered SQL dbt resolves `ref()`, renders Jinja, and hands your adapter a finished string. You execute it; you don't template it. While you're coding, the compiler enforces completeness. When you add `AdapterType::MyWarehouse` to the enum, every `match adapter_type()` block that doesn't handle it fails at that exact line. Your to-do list is always visible. For example: ```rust // You add your variant to the enum in dbt-adapter-core: pub enum AdapterType { Snowflake, BigQuery, DuckDb, Trino, MyWarehouse, // ← you added this } // Now every match block in the codebase must handle it. // This one in adapter_impl.rs doesn't yet: match self.adapter_type() { AdapterType::Snowflake => { ... } AdapterType::BigQuery => { ... } AdapterType::DuckDb => { ... } AdapterType::Trino => { ... } // missing MyWarehouse → compile error } ``` ```text error[E0004]: non-exhaustive patterns: `AdapterType::MyWarehouse` not covered --> crates/dbt-adapter/src/adapter/adapter_impl.rs:142:18 | | match self.adapter_type() { | ^^^^^^^^^^^^^^^^^^^ pattern `AdapterType::MyWarehouse` not covered ``` Each failure is a concrete implementation task. ##### Are you porting an existing v1 adapter? No v1 adapter? Skip to [Step 4](#step-4-development-machine-setup). If you're porting, use this map to see exactly what transfers, what needs rethinking, and what dbt Labs now owns for you in v2. [![Diagram showing which v1 adapter components map directly to v2 (green), require rethinking (purple), or are absorbed by dbt Labs or the ADBC driver (red)](/img/adapter-guide/adapter-creation-v2-blocks.svg?v=2 "Diagram showing which v1 adapter components map directly to v2 (green), require rethinking (purple), or are absorbed by dbt Labs or the ADBC driver (red)")](#)Diagram showing which v1 adapter components map directly to v2 (green), require rethinking (purple), or are absorbed by dbt Labs or the ADBC driver (red) Most community contributors aren't building from scratch, they're porting an adapter they already maintain or use in Python for dbt v1. If that's you, **you have a significant head start**, and this is the most realistic path for the vast majority of contributors. Before writing any Rust, check a couple of things: 1. Does a v1 adapter already exist for your warehouse? Refer to the [community adapters](https://docs.getdbt.com/docs/community-adapters.md) list. If one exists, find its GitHub repo. The macro SQL and connection logic are almost directly reusable. 2. Did dbt Labs already add a placeholder for your warehouse? Some warehouses already appear in v2's `AdapterType` enum but aren't fully implemented yet: the enum variant exists, which means the boilerplate is partially in place. When you add the remaining code, the compiler shows you exactly what's still missing. The warehouses in this state are: | Warehouse | v1 adapter | dbt v2 status | | --------- | ------------------------------------------------------- | ----------------------------------------------------------------- | | Athena | [dbt-athena](https://github.com/dbt-athena/dbt-athena) | `AdapterType::Athena` exists: needs auth, macros, adapter arms | | Trino | [dbt-trino](https://github.com/starburstdata/dbt-trino) | `AdapterType::Trino` exists: needs auth, macros, adapter arms | | Starburst | [dbt-trino](https://github.com/starburstdata/dbt-trino) | `AdapterType::Starburst` exists: needs auth, macros, adapter arms | | Dremio | [dbt-dremio](https://github.com/dremio/dbt-dremio) | `AdapterType::Dremio` exists: needs auth, macros, adapter arms | | Oracle | [dbt-oracle](https://github.com/oracle/dbt-oracle) | `AdapterType::Oracle` exists: needs auth, macros, adapter arms | For warehouses not yet in `AdapterType` at all (MySQL, Hive, Vertica, SQL Server, Teradata, and more), you start from Step 5.1 by adding the `AdapterType` variant. ###### What transfers from v1 to v2 | v1 component | Where it goes in v2 | Notes | | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `macros/adapters.sql` | `dbt-loader/.../dbt-<wh>/macros/adapters.sql` | Mostly a direct port, same `<wh>__` dispatch prefix, same macro names, same Jinja patterns | | `macros/catalog.sql` | Same location | The catalog SQL (`list_relations_without_caching`, `get_catalog`) transfers almost verbatim | | Custom materializations and adapter-overrides | Same location | Look out for Jinja that might not yet be supported in dbt v2; that will need to be addressed separately | | Profile fields in `credentials.py` / `profile_template.yml` | `DbConfig` struct in `dbt-schemas` | Each profile field becomes a struct field: optional fields use `Option<T>` | | Connection URI / DSN construction in `connections.py` | `dbt-auth/src/<wh>/mod.rs` | The URI building logic maps cleanly to the auth module pattern | | `BaseRelation.quote_policy` / identifier casing behavior | `include_policy()` in `crates/dbt-adapter/src/relation/relation_impl.rs`, `Policy` is a type alias for `ResolvedQuoting` defined in `crates/dbt-schemas/src/schemas/relations/base.rs` | The 3-part vs. 2-part name structure and quote flags map 1:1 to the dbt v2 `Policy` struct | | Catalog introspection SQL in macros and `adapter.py` | `get_relation.rs` and Jinja macros | The system catalog table names and queries you already know transfer directly | ###### What doesn't transfer by design Four categories of v1 code simply don't exist in v2, which means less code to port: * **Connection management**: In v1 you owned the full connection lifecycle against the Python DB API. In v2 the ADBC driver handles this entirely; skip anything related to cursors, retries, or connection pooling. * **Query execution**: Shared Rust infrastructure handles raw query plumbing (`execute`, `add_query`). You don't implement it. * **Adapter class hierarchy**: The Python subclass structure is replaced by shared Rust match expressions. You register your adapter; you don't implement execution logic. * **Packaging and distribution**: No `setup.py`, no PyPI release. dbt Labs ships your adapter once the PR is merged. #### Step 4: Development machine setup ```shell # Rust rustup # install from https://rustup.rs rustup show # verify # Go (needed for driver builds) go version # verify # Clone the repo git clone https://github.com/dbt-labs/dbt cd dbt # Verify you can build cargo build --bin dbt ``` If you hit Z3 errors: ```shell brew install pkg-config z3 ``` If disk fills during build: ```shell cargo clean # frees old build artifacts; you'll do this often ``` ##### Development workflow The core development loop is the same whether you write code manually or with an AI assistant: 1. Add `AdapterType::MyWarehouse` to the enum (Step 5.1) 2. Run `cargo build -p <crate>`: the compiler lists every match arm missing for your new variant 3. Use the compiler output + the file you're editing + the equivalent reference file (from the breakdown at the bottom) to write each arm. If using an AI assistant, paste all three in as context. * *Example:* to fill in the `quote_char` arm, paste the error listing `AdapterType::MyWarehouse` as missing, the `quote_char` match block from `dbt-adapter-core/src/lib.rs`, and the Exasol line (`Exasol => '"'`) as the pattern to follow. 4. Fill in the arm; verify with `cargo build` again 5. Repeat for each crate until error-free Every missing case is a compile error, so an AI assistant always has a precise specification to work from. What context helps for each arm: * **The file you're editing**: paste the relevant `match` block or function. For example, the `match self.adapter_type()` block from `adapter_impl.rs` that needs a new arm. * **The equivalent reference file** from the breakdown below. For example, when writing your auth module, paste `crates/dbt-auth/src/exasol/mod.rs` as the pattern. * **The compiler error output.** For example: `error[E0004]: non-exhaustive patterns: AdapterType::MyWarehouse not covered` * **Your warehouse's specifics**: system catalog table names and the connection fields from your `profiles.yml`. Watch out for: * **Hallucinated file paths**: AI often invents dbt v2 paths. Use the file breakdown below as ground truth. * **Always verify with the type checker**: run `cargo build -p <crate>` after any AI-generated changes. * **SQL macro patterns from v1** may not apply cleanly in dbt v2. Compare against the reference `adapters.sql` at `crates/dbt-loader/src/dbt_macro_assets/dbt-exasol/macros/adapters.sql` in [dbt-labs/dbt](https://github.com/dbt-labs/dbt). #### Step 5: Build a new adapter This step walks you through each crate you need to touch. Work through them in order. Each builds on the last. After each sub-step, run the type checker to catch missed match arms and type errors: ```shell cargo build -p <crate-name> ``` Replace `<crate-name>` with the crate you just edited, for example `dbt-adapter-core`, `dbt-xdbc`, `dbt-schemas`, `dbt-auth`, `dbt-adapter`, or `dbt-loader`. ##### 5.1: Register the adapter type **Crate:** `crates/dbt-adapter-core/` `AdapterType` is the central enum that identifies your warehouse throughout the entire codebase. Add your variant here first. Rust's `match` expressions must handle every possible variant explicitly. So when you add `AdapterType::MyWarehouse`, every `match adapter_type()` block that doesn't handle your variant becomes a **compile error**. Run `cargo build -p <crate>` and the compiler hands you a complete list of exactly what still needs to be written. Nothing is hidden or implicit. ```rust // crates/dbt-adapter-core/src/lib.rs pub enum AdapterType { Postgres, Snowflake, Bigquery, Databricks, Redshift, Spark, DuckDB, Fabric, ClickHouse, Athena, Starburst, Trino, Datafusion, Dremio, Oracle, Exasol, // ← already added MyWarehouse, // ← add yours here } ``` Also add a `quote_char` arm in the same file. Use double quotes `'"'` for most warehouses; BigQuery and Databricks use a backtick instead: ```rust // crates/dbt-adapter-core/src/lib.rs fn quote_char(&self) -> char { match self { // ... existing adapters ... Exasol => '"', // double-quote (most warehouses) MyWarehouse => '"', // ← add yours here BigQuery | Databricks => '`', } } ``` ##### 5.2: Register the ADBC driver **Crate:** `crates/dbt-xdbc/` Your driver already exists (covered in Step 2). This step is where you register it so dbt v2 knows its library name and how to load it at runtime. | File | What to do | Required? | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `src/driver.rs` | Add a variant to the `Backend` enum; add the ADBC library name (for example, `"adbc_driver_exasol"`) and FFI protocol; also define a `LoadStrategy` | **Yes** | | `src/install.rs` | Add CDN download URL and platform strings | No, only for drivers distributed through the dbt Labs CDN, which requires separate coordination with dbt Labs | The `Backend` enum maps to the ADBC shared library name (`lib<name>.so` / `<name>.dll` / `lib<name>.dylib`). You're registering its identity so dbt v2 knows what to load. You're not writing the driver here. ###### CDN-distributed vs. manual install For these 11 adapters, dbt v2 automatically downloads the driver on first use. All other adapters require manual installation. Users take one extra setup step. Make this clear in your documentation (Step 7). | On the CDN: auto-downloaded | | ----------------------------------------------------------------------------------------------------------- | | Snowflake, BigQuery, ClickHouse, Postgres, Databricks, Redshift, DuckDB, DuckDB Extended, Spark, SQL Server | For **custom Arrow type mappings** (for example, DuckDB needed this for `HUGEINT`, `UTINYINT`): only add warehouse-specific type handling if your driver returns types that Arrow's standard schema doesn't cover. Most warehouses don't need this. ##### 5.3: Add your connection profile **Crate:** `crates/dbt-schemas/` | File | What to do | | ------------------------- | ----------------------------------------------------------------------- | | `src/schemas/profiles.rs` | Add a `DbConfig` variant and a config struct with all connection fields | The config struct should include everything a user would put in their `profiles.yml` for your warehouse. Example from Exasol: ```rust #[derive(Debug, Clone, Serialize, Deserialize, Default, DbtSchema, Merge)] pub struct ExasolDbConfig { pub user: Option<String>, #[serde(skip_serializing_if = "Option::is_none", alias = "pass")] pub password: Option<String>, pub host: Option<String>, pub port: Option<StringOrInteger>, pub database: Option<String>, pub schema: Option<String>, pub encryption: Option<bool>, pub certificate_validation: Option<bool>, pub certificate_fingerprint: Option<String>, pub connection_timeout: Option<StringOrInteger>, pub threads: Option<StringOrInteger>, } ``` After adding the config struct, also add `DbConfig::MyWarehouse(Box<MyWarehouseDbConfig>)` as a new variant to the `DbConfig` enum. The compiler will then point you at every place that reads from `DbConfig` and needs a new case for your warehouse. Follow those errors to wire it in. ##### 5.3.1: Add `dbt init` profile generation (optional but strongly encouraged) This is what makes your adapter easy to use. When someone runs `dbt init` and selects your warehouse, they get a guided prompt that builds their `profiles.yml`, without documentation hunting or manual YAML editing. Add support in `crates/dbt-init/`: 1. Register your adapter in `get_available_adapters()` in `crates/dbt-init/src/profile_setup.rs`: ```rust pub fn get_available_adapters() -> &'static [AdapterType] { &[ // ... existing adapters ... AdapterType::MyWarehouse, // ← add this ] } ``` 2. Create a config file `crates/dbt-init/src/adapter_config/<warehouse>_config.rs` implementing `InteractiveSetup` for your `DbConfig` struct: define the fields `dbt init` should prompt for and a `set_field()` handler. See `postgres_config.rs` as a minimal reference. 3. Export it from `crates/dbt-init/src/adapter_config/mod.rs`: ```rust pub mod mywarehouse_config; pub use mywarehouse_config::setup_mywarehouse_profile; ``` 4. Add a match arm in `create_profile_for_adapter()` in `profile_setup.rs`: ```rust AdapterType::MyWarehouse => { let config = match existing_config { Some(DbConfig::MyWarehouse(c)) => Some(c), _ => None, }; DbConfig::MyWarehouse(setup_mywarehouse_profile(config.map(Box::as_ref))?) } ``` ##### 5.4: Add authentication **Crate:** `crates/dbt-auth/` | File | What to do | Required? | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `src/<warehouse>/mod.rs` | Credential resolution, reads config fields, env vars, key files, tokens; builds the ADBC connection URI and credentials | **Yes** | | `src/<warehouse>/init.rs` | Init SQL generation, SQL that must run when the connection opens (for example, `USE SCHEMA`, `SET` statements, and extension loading) | Optional, only for warehouses that need SQL on connection open. Most don't. | | `src/lib.rs` | Register the new module with `mod <warehouse>;` and wire it into the auth dispatch match | **Yes** | The auth module turns a `DbConfig` into a live, authenticated ADBC connection. At minimum you need basic credential handling. More sophisticated auth (OAuth, SSO, key-pair) can be added incrementally. The pattern is: read config fields → construct URI → call `builder.with_parse_uri(uri)`, `builder.with_username(user)`, `builder.with_password(password)`. ##### 5.5: Build the adapter layer **Crate:** `crates/dbt-adapter/` This is the largest step. You're adding warehouse-specific behavior to the shared adapter layer using `match adapter_type()` arms. Simple vs. complex adapters Most adapters, including Exasol, Athena, Trino, Starburst, Dremio, Oracle, and ClickHouse, add match arms directly to the shared files in `src/relation/`. Only adapters with highly custom relation logic (Snowflake's multi-part names and case rules; BigQuery's project/dataset structure) have their own subdirectory under `src/relation/`. If your warehouse uses standard `schema.table` or `database.schema.table` naming with straightforward quoting, you're a simple adapter. Start simple and only add complexity if the compiler forces it. ###### Relation type and quoting **File:** `crates/dbt-adapter/src/relation/relation_impl.rs` Add a match arm that sets your quoting policy. The policy controls three things: whether the database is included in fully-qualified names, whether the schema is quoted, and whether the identifier is quoted. For example, Exasol uses a 2-part name (`schema.table`, no database prefix) and quotes both: ```rust // crates/dbt-adapter/src/relation/relation_impl.rs fn include_policy(adapter_type: AdapterType, path: &RelationPath) -> Policy { match adapter_type { // ... existing adapters ... AdapterType::MyWarehouse => Policy::new(false, true, true), // ↑ ↑ ↑ // database schema identifier // disabled quoted quoted _ => Policy::trues(), // default: all parts included and quoted } } ``` Decide up front whether your warehouse uses 2-part or 3-part names, and whether identifiers are case-sensitive. For example, Exasol uppercases unquoted identifiers by default, so all catalog lookup SQL uses `upper()` comparisons. **File:** `src/relation/factory.rs` Add your `AdapterType` to the `create_static_relation` match, wiring it to `RelationStatic` (the generic static relation used for Jinja's `api.Relation`): ```rust Databricks | Spark | Fabric | DuckDB | Exasol | Postgres | Redshift | Bigquery => { let relation_type = RelationStatic { adapter_type, quoting }; StaticBaseRelationObject::new(Arc::new(relation_type)) } ``` ###### Catalog introspection **File:** `src/metadata/get_relation.rs` Add a match arm and a function that queries your warehouse's system catalog to look up a single relation by name. Follow the reference implementations closely. This area of the codebase is expected to evolve: ```rust AdapterType::Exasol => exasol_get_relation( adapter, state, ctx, conn, database, schema, identifier, token, ), ``` ```rust fn exasol_get_relation( adapter: &AdapterImpl, state: &State, ctx: &QueryCtx, conn: &mut dyn Connection, database: &str, schema: &str, identifier: &str, token: CancellationToken, ) -> AdapterResult<Option<Box<dyn BaseRelation>>> { let q_schema = schema.to_uppercase(); let q_ident = identifier.to_uppercase(); let sql = format!( "select 'table' as \"type\" from sys.exa_all_tables \ where table_schema = '{q_schema}' and table_name = '{q_ident}' \ union all \ select 'view' from sys.exa_all_views \ where view_schema = '{q_schema}' and view_name = '{q_ident}'" ); // execute, read result, return relation } ``` Use `information_schema` if your warehouse supports standard SQL, or system catalog tables (`sys.*`, `information_schema.*`) as appropriate. ###### Adapter match arms **File:** `src/adapter/adapter_impl.rs` After adding `AdapterType::MyWarehouse`, the compiler will enumerate every `match self.adapter_type()` block that needs a new arm. Most arms simply delegate to an existing pattern. For example: ```rust // Column builder: delegate to postgres-like builder Exasol => Ok(Self::build_postgres_like(field, type_ops)), // Schema column name for listing relations Exasol => "name", // DATA_TYPE column name in information schema AdapterType::Exasol => "DATA_TYPE", // in src/sql_types.rs ``` For capabilities your adapter doesn't support yet (like `valid_incremental_strategies`), return `unimplemented!()`. That's fine for an initial community adapter contribution. ###### Column builder **File:** `src/column/column_builder.rs` Add a match arm for how your warehouse's Arrow record batches map to dbt column objects. Most adapters can delegate to `build_postgres_like`: ```rust Exasol => Ok(Self::build_postgres_like(field, type_ops)), ``` Only implement custom logic if your warehouse has unusual type handling. ##### 5.6: Write your SQL macros **Crate:** `crates/dbt-loader/` Create a new directory at `src/dbt_macro_assets/dbt-<adapter_type>/`. Registration is automatic The loader discovers adapter packages by scanning `src/dbt_macro_assets/`. You don't need to add any Rust code to register your directory. Creating it and the `dbt_project.yml` is sufficient. | File | What to do | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Adapter plugin project definition: `name: dbt_<warehouse>`, `macro-paths: ["macros"]` | | `macros/adapters.sql` | dbt v1 adapter macros: `create_schema`, `drop_schema`, `drop_relation`, `rename_relation`, `truncate_relation`, `create_table_as`, `create_view_as`, `list_schemas`, `check_schema_exists`, `information_schema_name`, `current_timestamp`, `get_columns_in_relation`, `list_relations_without_caching` | v2 uses the same `adapter.dispatch()` pattern as v1. Your macros use the `<warehouse>__` prefix to override defaults: ```sql {% macro exasol__create_table_as(temporary, relation, sql) -%} create or replace table {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__create_view_as(relation, sql) -%} create or replace view {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__drop_relation(relation) -%} {% call statement('drop_relation', auto_begin=False) -%} drop {{ relation.type }} if exists {{ relation }} cascade {%- endcall %} {% endmacro %} {% macro exasol__rename_relation(from_relation, to_relation) -%} {% call statement('rename_relation') -%} rename {{ from_relation.type }} {{ from_relation }} to {{ to_relation.identifier }} {%- endcall %} {% endmacro %} ``` Note: `rename_relation` uses only `to_relation.identifier`, not the full relation. Exasol's `RENAME` syntax doesn't take a fully qualified target. For catalog introspection, use your warehouse's system catalog. For example, Exasol queries `sys.*` instead of `information_schema`: ```sql {% macro exasol__list_relations_without_caching(schema_relation) -%} {% call statement('list_relations_without_caching', fetch_result=True) -%} select '{{ schema_relation.database }}' as "database", table_name as "name", table_schema as "schema", 'table' as "type" from sys.exa_all_tables where upper(table_schema) = upper('{{ schema_relation.schema }}') union all select '{{ schema_relation.database }}' as "database", view_name as "name", view_schema as "schema", 'view' as "type" from sys.exa_all_views where upper(view_schema) = upper('{{ schema_relation.schema }}') {%- endcall %} {{ return(load_result('list_relations_without_caching').table) }} {%- endmacro %} {% macro exasol__information_schema_name(database) -%} sys {%- endmacro %} ``` If your warehouse is similar to an existing one (for example, Postgres-compatible), start by delegating to that dialect's macros and only override where behavior differs: ```sql {% macro mywarehouse__create_table_as(temporary, relation, sql) -%} {{ return(postgres__create_table_as(temporary, relation, sql)) }} {%- endmacro %} ``` #### Step 6: Test your adapter ##### Type check after each crate Run the type checker after completing work in each crate to catch missed match arms and type errors immediately: ```shell cargo build -p dbt-adapter-core cargo build -p dbt-xdbc cargo build -p dbt-schemas cargo build -p dbt-auth cargo build -p dbt-adapter cargo build -p dbt-loader ``` ##### Smoke testing with a dbt project Run a real `dbt build` against your warehouse and ensure it builds error-free. At minimum, exercise table, view, incremental, and seed materializations. A clean `dbt build` on [jaffle-shop](https://github.com/dbt-labs/jaffle-shop) is one example of an acceptance bar for a community adapter. ```shell # Build the CLI cargo build --bin dbt # Create a test project (or clone jaffle-shop) ./target/debug/dbt init # Run against your warehouse ./target/debug/dbt build --project-dir <your-project> ``` ##### CI testing CI testing for community adapter PRs is coordinated with the dbt Labs adapters team: the test infrastructure is not publicly distributed. When your PR is ready, reach out in `#adapter-ecosystem` on the [dbt Community Slack](https://community.getdbt.com/), tag Hope Watson (`@Hope Watson (dbt Labs)`), and the adapters team will work with you on warehouse validation. Note that this may take some time and coordination. Known gap: CI requires coordination Community contributors cannot run CI independently. dbt Labs' CI pipeline requires certain checks. For certain adapters, that means warehouse credentials. dbt Labs is still defining this handoff process for dbt v2. Expect to coordinate closely with the adapters team, for the approval process to take some time, and for some back-and-forth. If you hit friction, flag it in `#adapter-ecosystem`. #### Step 7: Document your adapter Once your adapter is merged and available in a release, document it so users can find and configure it. ##### Write a setup guide Document the `profiles.yml` configuration for your warehouse: what fields are required, what's optional, and example values. Follow the format of existing [adapter setup guides](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md?version=2.0). Driver installation is critical to document Unlike adapters distributed through the CDN, your users won't get the driver automatically: dbt v2 won't download it for them. Your setup guide must explain where to get the driver binary and how to install it so dbt v2 can find it at runtime. Without this, users will configure a valid profile and still get a connection error. Include the exact library name dbt v2 looks for (for example, `libadbc_driver_<yourwarehouse>.dylib`) and where to put it. ##### General documentation guidelines * Assume the reader knows dbt fundamentals but is not an expert on your warehouse's inner workings. * Include a complete working `profiles.yml` example. * Document any warehouse-specific quirks (for example, 2-part versus 3-part naming and identifier case sensitivity). * Link to the warehouse vendor's ADBC driver documentation. #### Step 8: Promote your adapter Your PR must be merged first dbt Labs reviews and merges community adapter PRs into `dbt-labs/dbt`. Wait until the PR is merged and the adapter ships in a published release before directing users to it. ##### Community channels Join the [dbt Community Slack](https://community.getdbt.com/) and find: * **`#adapter-ecosystem`**: the main channel for adapter developers * **`#db-<yourwarehouse>`**: if a channel exists for your warehouse, let users know v2 support is available. Note: v1 adapter users will still be on the Python-based adapter and will need to migrate. ##### Before you announce Align with the adapters team on: which materializations you're targeting in the initial implementation, any known gaps in your ADBC driver, and timeline. This prevents surprises during review and sets accurate expectations for users. #### Reference: File-by-file implementation guide A community-contributed dbt v2 adapter touches roughly 13 files, all in the public [dbt-labs/dbt](https://github.com/dbt-labs/dbt) repo. The "Exasol example" column shows what it looks like in practice. Substitute your warehouse name and system catalog throughout. | Generic path | What it does | Exasol example | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `.changes/unreleased/Features-*.yaml` | Changelog entry | Same for all adapters | | `crates/dbt-adapter-core/src/lib.rs` | Add `AdapterType::<Warehouse>` variant and `quote_char` arm | `AdapterType::Exasol` | | `crates/dbt-xdbc/src/driver.rs` | Add `Backend::<Warehouse>` variant: the ADBC library name and FFI protocol | `Backend::Exasol`, library name `"adbc_driver_exasol"` | | `crates/dbt-schemas/src/schemas/profiles.rs` | `<Warehouse>DbConfig` struct + `DbConfig::<Warehouse>` variant, wired into all config match arms | `ExasolDbConfig` struct + `DbConfig::Exasol` variant | | `crates/dbt-auth/src/<warehouse>/mod.rs` | Auth module, reads config fields, resolves credentials, constructs connection URI | `src/exasol/mod.rs`, builds URI from host, port, user/pass, TLS options | | `crates/dbt-adapter/src/adapter/adapter_impl.rs` | All exhaustive `match adapter_type()` arms, most delegate to existing patterns; use `unimplemented!()` for features not yet supported | Exasol adds ~10 arms, most are 1–2 lines | | `crates/dbt-adapter/src/column/column_builder.rs` | Column builder match arm, maps Arrow record batches to dbt column objects | `Exasol => Ok(Self::build_postgres_like(...))` | | `crates/dbt-adapter/src/metadata/get_relation.rs` | Catalog lookup function, queries system tables to find a relation by name | `exasol_get_relation()`, queries `sys.exa_all_tables` / `sys.exa_all_views` | | `crates/dbt-adapter/src/relation/relation_impl.rs` | Add your adapter to the `include_policy()` match, controls whether the database prefix is included and whether parts are quoted | `Policy::new(false, true, true)`, database disabled, schema + identifier quoted. Default `Policy::trues()` includes and quotes all three parts. | | `crates/dbt-adapter/src/relation/factory.rs` | Add warehouse to `RelationStatic` arm in `create_static_relation` | Exasol added alongside Postgres, Redshift, DuckDB, and more. | | `crates/dbt-adapter/src/sql_types.rs` | Metadata column name arms, `DATA_TYPE` and schema listing column names | Exasol uses standard `DATA_TYPE` column name | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Macro plugin project definition | `dbt-exasol/dbt_project.yml` | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/macros/adapters.sql` | All required adapter macros | `dbt-exasol/macros/adapters.sql`, uses `sys.*` instead of `information_schema` | #### Reference: Useful commands ```shell # Type check a specific crate cargo build -p <crate-name> # Build the CLI binary cargo build --bin dbt # Run a full dbt build against your warehouse ./target/debug/dbt build --project-dir <your-project> # Free disk space cargo clean ``` --- ### Create Datadog events from dbt results [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide will teach you how to build and host a basic Python app which will add dbt job events to Datadog. To do this, when a dbt job completes it will create a log entry for each node that was run, containing all information about the node provided by the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md). In this example, we will use [fly.io](https://fly.io) for hosting/running the service. fly.io is a platform for running full stack apps without provisioning servers etc. This level of usage should comfortably fit inside of the Free tier. You can also use an alternative tool such as [AWS Lambda](https://ademoverflow.com/en/posts/tutorial-fastapi-aws-lambda-serverless/) or [Google Cloud Run](https://github.com/sekR4/FastAPI-on-Google-Cloud-Run). ##### Prerequisites This guide assumes some familiarity with: * [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * CLI apps * Deploying code to a serverless code runner like fly.io or AWS Lambda #### Clone the `dbt-cloud-webhooks-datadog` repo [This repository](https://github.com/dpguthrie/dbt-cloud-webhooks-datadog) contains the sample code for validating a webhook and creating logs in Datadog. #### Install `flyctl` and sign up for fly.io Follow the directions for your OS in the [fly.io docs](https://fly.io/docs/hands-on/install-flyctl/), then from your command line, run the following commands: Switch to the directory containing the repo you cloned in step 1: ```shell #example: replace with your actual path cd ~/Documents/GitHub/dbt-cloud-webhooks-datadog ``` Sign up for fly.io: ```shell flyctl auth signup ``` Your console should show `successfully logged in as YOUR_EMAIL` when you're done, but if it doesn't then sign in to fly.io from your command line: ```shell flyctl auth login ``` #### Launch your fly.io app Launching your app publishes it to the web and makes it ready to catch webhook events: ```shell flyctl launch ``` 1. You will see a message saying that an existing `fly.toml` file was found. Type `y` to copy its configuration to your new app. 2. Choose an app name of your choosing, such as `YOUR_COMPANY-dbt-cloud-webhook-datadog`, or leave blank and one will be generated for you. Note that your name can only contain numbers, lowercase letters and dashes. 3. Choose a deployment region, and take note of the hostname that is generated (normally `APP_NAME.fly.dev`). 4. When asked if you would like to set up Postgresql or Redis databases, type `n` for each. 5. Type `y` when asked if you would like to deploy now. Sample output from the setup wizard: \`\` `joel@Joel-Labes dbt-cloud-webhooks-datadog % flyctl launch An existing fly.toml file was found for app dbt-cloud-webhooks-datadog ? Would you like to copy its configuration to the new app? Yes Creating app in /Users/joel/Documents/GitHub/dbt-cloud-webhooks-datadog Scanning source code Detected a Dockerfile app ? Choose an app name (leave blank to generate one): demo-dbt-cloud-webhook-datadog automatically selected personal organization: Joel Labes Some regions require a paid plan (fra, maa). See https://fly.io/plans to set up a plan. ? Choose a region for deployment: [Use arrows to move, type to filter] ? Choose a region for deployment: Sydney, Australia (syd) Created app dbtlabs-dbt-cloud-webhook-datadog in organization personal Admin URL: https://fly.io/apps/demo-dbt-cloud-webhook-datadog Hostname: demo-dbt-cloud-webhook-datadog.fly.dev ? Would you like to set up a Postgresql database now? No ? Would you like to set up an Upstash Redis database now? No Wrote config file fly.toml ? Would you like to deploy now? Yes` ##### 4. Create a Datadog API Key [Create an API Key for your Datadog account](https://docs.datadoghq.com/account_management/api-app-keys/) and make note of it and your Datadog site (e.g. `datadoghq.com`) for later. #### Configure a new webhook in dbt 1. See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Your event should be **Run completed**. 2. Set the webhook URL to the host name you created earlier (`APP_NAME.fly.dev`). 3. Make note of the Webhook Secret Key for later. *Do not test the endpoint*; it won't work until you have stored the auth keys (next step) #### Store secrets The application requires four secrets to be set, using these names: * `DBT_CLOUD_SERVICE_TOKEN`: a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with at least the `Metadata Only` permission. * `DBT_CLOUD_AUTH_TOKEN`: the Secret Key for the dbt webhook you created earlier. * `DD_API_KEY`: the API key you created earlier. * `DD_SITE`: The Datadog site for your organisation, e.g. `datadoghq.com`. Set these secrets as follows, replacing `abc123` etc with actual values: ```shell flyctl secrets set DBT_CLOUD_SERVICE_TOKEN=abc123 DBT_CLOUD_AUTH_TOKEN=def456 DD_API_KEY=ghi789 DD_SITE=datadoghq.com ``` #### Deploy your app After you set your secrets, fly.io will redeploy your application. When it has completed successfully, go back to the dbt webhook settings and click **Test Endpoint**. --- ### Create new materializations [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction The model materializations you're familiar with, `table`, `view`, and `incremental` are implemented as macros in a package that's distributed along with dbt. You can check out the [source code for these materializations](https://github.com/dbt-labs/dbt-adapters/tree/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations). If you need to create your own materializations, reading these files is a good place to start. Continue reading below for a deep-dive into dbt materializations. caution This is an advanced feature of dbt. Let us know if you need a hand! We're always happy to [chat](http://community.getdbt.com/). (Applies to dbt v2.0 and later) Custom materializations and static analysis Models that use a custom materialization follow the normal [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md#custom-materializations) rules. If you use `strict` and your materialization changes the model's schema (for example, by adding, renaming, or retyping columns), set `static_analysis: off` on those models. Because `off` cascades, their downstream models lose static analysis too. Refer to [Custom materializations](https://docs.getdbt.com/docs/build/about-static-analysis.md#custom-materializations) for details. #### Creating a materialization Learn by video! For video tutorials on Materializations, go to dbt Learn and check out the [Materializations fundamentals course](https://learn.getdbt.com/courses/materializations-fundamentals). Materialization blocks make it possible for dbt to load custom materializations from packages. The materialization blocks work very much like `macro` blocks, with a couple of key exceptions. Materializations are defined as follows: ```sql {% materialization [materialization name], ["specified adapter" | default] %} ... {% endmaterialization %} ``` Materializations can be given a name, and they can be tied to a specific adapter. dbt will pick the materialization tied to the currently-in-use adapter if one exists, or it will fall back to the `default` adapter. In practice, this looks like: macros/my\_materialization.sql ```sql {% materialization my_materialization_name, default %} -- cross-adapter materialization... assume Redshift is not supported {% endmaterialization %} {% materialization my_materialization_name, adapter='redshift' %} -- override the materialization for Redshift {% endmaterialization %} ``` info dbt's ability to dynamically pick the correct materialization based on the active database target is called [multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch). This feature unlocks a whole world of cross-database compatibility features -- if you're interested in this, please let us know on Slack! ##### Anatomy of a materialization Materializations are responsible for taking a dbt model SQL statement and turning it into a transformed dataset in a database. As such, materializations generally take the following shape: 1. Prepare the database for the new model 2. Run pre-hooks 3. Execute any SQL required to implement the desired materialization 4. Run post-model hooks 5. Clean up the database as required 6. Update the Relation cache Each of these tasks are explained in sections below. ##### Prepare the database Materializations are responsible for creating new tables or views in the database, or inserting/updating/deleting data from existing tables. As such, materializations need to know about the state of the database to determine exactly what SQL they should run. Here is some pseudocode for the "setup" phase of the **table** materialization: ```sql -- Refer to the table materialization (linked above) for an example of real syntax -- This code will not work and is only intended for demonstration purposes {% set existing = adapter.get_relation(this) %} {% if existing and existing.is_view %} {% do adapter.drop_relation(existing) %} {% endif %} ``` In this example, the `get_relation` method is used to fetch the state of the currently-executing model from the database. If the model exists as a view, then the view is dropped to make room for the table that will be built later in the materialization. This is a simplified example, and the setup phase for a materialization can become quite complicated indeed! When building a materialization, be sure to consider the state of the database and any supplied [flags](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md) (ie. `--full-refresh`) to ensure that the materialization code behaves correctly in different scenarios. ##### Run pre-hooks Pre- and post-hooks can be specified for any model -- be sure that your materialization plays nicely with these settings. Two variables, `pre_hooks` and `post_hooks` are automatically injected into the materialization context. Invoke these hooks at the appropriate time with: ```sql ... {{ run_hooks(pre_hooks) }} .... ``` ##### Executing SQL Construct your materialization DML to account for the different permutations of table existence, materialization flags, etc. There are a number of [adapter functions](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md) and context variables that can help you here. Be sure to consult the Reference section of this site for a full list of variables and functions at your disposal. ##### Run post-hooks See the section above on pre-hooks for more information on running post-hooks. ##### Clean up The "cleanup" phase of the materialization typically renames or drops relations and commits the transaction opened in "preparation" step above. The `table` materialization, for instance, executes the following cleanup code: ```text {{ drop_relation_if_exists(backup_relation) }} ``` Be sure to `commit` the transaction in the `cleanup` phase of the materialization with `{{ adapter.commit() }}`. If you do not commit this transaction, it will be rolled back by dbt and the transformations applied in your materialization will be discarded. ##### Update the Relation cache Materializations should [return](https://docs.getdbt.com/reference/dbt-jinja-functions/return.md) the list of Relations that they have created at the end of execution. dbt will use this list of Relations to update the relation cache in order to reduce the number of queries executed against the database's `information_schema`. If a list of Relations is not returned, then dbt will raise a Deprecation Warning and infer the created relation from the model's configured database, schema, and alias. macros/my\_view\_materialization.sql ```sql {%- materialization my_view, default -%} {%- set target_relation = api.Relation.create( identifier=this.identifier, schema=this.schema, database=this.database, type='view') -%} -- ... setup database ... -- ... run pre-hooks... -- build model {% call statement('main') -%} {{ create_view_as(target_relation, sql) }} {%- endcall %} -- ... run post-hooks ... -- ... clean up the database... -- Return the relations created in this materialization {{ return({'relations': [target_relation]}) }} {%- endmaterialization -%} ``` If a materialization solely creates a single relation, then returning that relation at the end of the materialization is sufficient to synchronize the dbt Relation cache. If the materialization *renames* or *drops* Relations other than the relation returned by the materialization, then additional work is required to keep the cache in sync with the database. To explicitly remove a relation from the cache, use [adapter.drop\_relation](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md). To explicitly rename a relation in the cache, use [adapter.rename\_relation](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md). Calling these methods is preferable to executing the corresponding SQL directly, as they will mutate the cache as required. If you do need to execute the SQL to drop or rename relations directly, use the `adapter.cache_dropped` and `adapter.cache_renamed` methods to synchronize the cache. #### Materialization Configuration Materializations support custom configuration. You might be familiar with some of these configs from materializations like `unique_key` in [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) or `strategy` in [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) . ##### Specifying configuration options Materialization configurations can either be "optional" or "required". If a user fails to provide required configurations, then dbt will raise a compilation error. You can define these configuration options with the `config.get` and `config.require` functions. ```text # optional config.get('optional_config_name', default="the default") # required config.require('required_config_name') ``` For more information on the `config` dbt Jinja function, see the [config](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md) reference. #### Materialization precedence dbt will pick the materialization macro in the following order (lower takes priority): 1. global project - default 2. global project - plugin specific 3. imported package - default 4. imported package - plugin specific 5. local project - default 6. local project - plugin specific In each of the stated search spaces, a materialization can only be defined once. Two different imported packages may not supply the same materialization - an error will be raised. Specific materializations can be selected by using the dot-notation when selecting a materialization from the context. We recommend *not* overriding materialization names directly, and instead using a prefix or suffix to denote that the materialization changes the behavior of the default implementation (eg. my\_project\_incremental). --- ### Customize dbt models database, schema, and alias [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide explains how to customize the [schema](https://docs.getdbt.com/docs/build/custom-schemas.md), [database](https://docs.getdbt.com/docs/build/custom-databases.md), and [alias](https://docs.getdbt.com/docs/build/custom-aliases.md) naming conventions in dbt to fit your data warehouse governance and design needs. When we develop dbt models and execute certain [commands](https://docs.getdbt.com/reference/dbt-commands.md) (such as `dbt run` or `dbt build`), objects (like tables and views) get created in the data warehouse based on these naming conventions. A word on naming Different warehouses have different names for *logical databases*. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. The following is dbt's out-of-the-box default behavior: * The database where the object is created is defined by the database configured at the [environment level in dbt](https://docs.getdbt.com/docs/dbt-platform-environments.md) or in the [`profiles.yml` file](https://docs.getdbt.com/docs/local/profiles.yml.md) in dbt v1. * The schema depends on whether you have defined a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) for the model: * If you haven't defined a custom schema, dbt creates the object in the default schema. In dbt, this is typically `dbt_username` for development and the default schema for deployment environments. In dbt v1, it uses the schema specified in the `profiles.yml` file. * If you define a custom schema, dbt concatenates the schema mentioned earlier with the custom one. * For example, if the configured schema is `dbt_myschema` and the custom one is `marketing`, the objects will be created under `dbt_myschema_marketing`. * Note that for automated CI jobs, the schema name derives from the job number and PR number: `dbt_cloud_pr_<job_id>_<pr_id>`. * The object name depends on whether an [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) has been defined on the model: * If no alias is defined, the object will be created with the same name as the model, without the `.sql` or `.py` at the end. * For example, suppose that we have a model where the SQL file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and no custom alias is configured. The resulting model will be created in `dbt_myschema_marketing.fct_orders_complete` in the dev environment. * If an alias is defined, the object will be created with the configured alias. * For example, suppose that we have a model where the SQL file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and the alias is configured to be `fct_orders`. The resulting model will be created in `dbt_myschema_marketing.fct_orders` These default rules are a great starting point, and many organizations choose to stick with those without any customization required. The defaults allow developers to work in their isolated schemas (sandboxes) without overwriting each other's work — even if they're working on the same tables. #### How to customize this behavior While the default behavior will fit the needs of most organizations, there are occasions where this approach won't work. For example, dbt expects that it has permission to create schemas as needed (and we recommend that the users running dbt have this ability), but it might not be allowed at your company. Or, based on how you've designed your warehouse, you may wish to minimize the number of schemas in your dev environment (and avoid schema sprawl by not creating the combination of all developer schemas and custom schemas). Alternatively, you may even want your dev schemas to be named after feature branches instead of the developer name. For this reason, dbt offers three macros to customize what objects are created in the data warehouse: * [`generate_database_name()`](https://docs.getdbt.com/docs/build/custom-databases.md#generate_database_name) * [`generate_schema_name()`](https://docs.getdbt.com/docs/build/custom-schemas.md#how-does-dbt-generate-a-models-schema-name) * [`generate_alias_name()`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_alias_name) By overwriting one or multiple of those macros, we can tailor where dbt objects are created in the data warehouse and align with any existing requirement. Key concept Models run from two different contexts must result in unique objects in the data warehouse. For example, a developer named Suzie is working on enhancements to `fct_player_stats`, but Darren is developing against the exact same object. In order to prevent overwriting each other's work, both Suzie and Darren should each have their unique versions of `fct_player_stats` in the development environment. Further, the staging version of `fct_player_stats` should exist in a unique location apart from the development versions, and the production version. We often leverage the following when customizing these macros: * In dbt, we recommend utilizing [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to define where the dbt invocation is occurring (dev/stg/prod). * They can be set at the environment level and all jobs will automatically inherit the default values. We'll add Jinja logic (`if/else/endif`) to identify whether the run happens in dev, prod, Ci, and more. * Or as an alternative to environment variables, you can use `target.name`. For more information, you can refer to [About target variables](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md). [![Custom schema environmental variables target name.](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/custom-schema-env-var.png?v=2> "Custom schema environmental variables target name.")](#)Custom schema environmental variables target name. To allow the database/schema/object name to depend on the current branch, you can use the out-of-the-box `DBT_CLOUD_GIT_BRANCH` environment variable in dbt [special environment variables](https://docs.getdbt.com/docs/build/environment-variables.md#special-environment-variables). #### Example use cases Here are some typical examples we've encountered with dbt users leveraging those 3 macros and different logic. note Note that the following examples are not comprehensive and do not cover all the available options. These examples are meant to be templates for you to develop your own behaviors. * [Use custom schema without concatenating target schema in production](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#1-custom-schemas-without-target-schema-concatenation-in-production) * [Add developer identities to tables](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#2-static-schemas-add-developer-identities-to-tables) * [Use branch name as schema prefix](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#3-use-branch-name-as-schema-prefix) * [Use a static schema for CI](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#4-use-a-static-schema-for-ci) ##### 1. Custom schemas without target schema concatenation in production The most common use case is using the custom schema without concatenating it with the default schema name when in production. To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- elif env_var('DBT_ENV_TYPE','DEV') == 'PROD' -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------------------ | | Developer 1 | dev | dbt\_dev1 | dev.dbt\_dev1\_marketing.my\_model | | Developer 2 | dev | dbt\_dev2 | dev.dbt\_dev2\_marketing.my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.dbt\_pr\_123\_marketing.my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.dbt\_pr\_234\_marketing.my\_model | | Production | prod | analytics | prod.marketing.my\_model | note We added logic to check if the current dbt run is happening in production or not. This is important, and we explain why in the [What not to do](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#what-not-to-do) section. ##### 2. Static schemas: Add developer identities to tables Occasionally, we run into instances where the security posture of the organization prevents developers from creating schemas and all developers have to develop in a single schema. In this case, we can: * Create a new file called generate\_schema\_name.sql under your macros folder with the following code: * Change `generate_schema_name()` to use a single schema for all developers, even if a custom schema is set. * Update `generate_alias_name()` to append the developer alias and the custom schema to the front of the table name in the dev environment. * This method is not ideal, as it can cause long table names, but it will let developers see in which schema the model will be created in production. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- elif env_var('DBT_ENV_TYPE','DEV') != 'CI' -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` macros/generate\_alias\_name.sql ```jinja {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} {%- if custom_alias_name -%} {{ target.schema }}__{{ custom_alias_name | trim }} {%- elif node.version -%} {{ target.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} {%- else -%} {{ target.schema }}__{{ node.name }} {%- endif -%} {%- else -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- elif node.version -%} {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} {%- else -%} {{ node.name }} {%- endif -%} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------------------ | | Developer 1 | dev | dbt\_dev1 | dev.marketing.dbt\_dev1\_my\_model | | Developer 2 | dev | dbt\_dev2 | dev.marketing.dbt\_dev2\_my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.dbt\_pr\_123\_marketing.my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.dbt\_pr\_234\_marketing.my\_model | | Production | prod | analytics | prod.marketing.my\_model | ##### 3. Use branch name as schema prefix For teams who prefer to isolate work based on the feature branch, you may want to take advantage of the `DBT_CLOUD_GIT_BRANCH` special environment variable. Please note that developers will write to the exact same schema when they are on the same feature branch. note The `DBT_CLOUD_GIT_BRANCH` variable is only available within the Studio IDE and not the dbt platform CLI. We’ve also seen some organizations prefer to organize their dev databases by branch name. This requires implementing similar logic in `generate_database_name()` instead of the `generate_schema_name()` macro. By default, dbt will not automatically create the databases. Refer to the [Tips and tricks](https://docs.getdbt.com/guides/customize-schema-alias.md?step=5) section to learn more. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} {#- we replace characters not allowed in the schema names by "_" -#} {%- set re = modules.re -%} {%- set cleaned_branch = re.sub("\W", "_", env_var('DBT_CLOUD_GIT_BRANCH')) -%} {%- if custom_schema_name is none -%} {{ cleaned_branch }} {%- else -%} {{ cleaned_branch }}_{{ custom_schema_name | trim }} {%- endif -%} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Branch | Target database | Target schema | Resulting object | | ----------- | ------------ | --------------- | ------------- | ------------------------------------ | | Developer 1 | `featureABC` | dev | dbt\_dev1 | dev.featureABC\_marketing.my\_model | | Developer 2 | `featureABC` | dev | dbt\_dev2 | dev.featureABC\_marketing.my\_model | | Developer 1 | `feature123` | dev | dbt\_dev1 | dev.feature123\_marketing.my\_model | | CI PR 123 | | ci | dbt\_pr\_123 | ci.dbt\_pr\_123\_marketing.my\_model | | CI PR 234 | | ci | dbt\_pr\_234 | ci.dbt\_pr\_234\_marketing.my\_model | | Production | | prod | analytics | prod.marketing.my\_model | When developer 1 and developer 2 are checked out on the same branch, they will generate the same object in the data warehouse. This shouldn't be a problem as being on the same branch means the model's code will be the same for both developers. ##### 4. Use a static schema for CI Some organizations prefer to write their CI jobs to a single schema with the PR identifier prefixed to the front of the table name. It's important to note that this will result in long table names. To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name=none, node=none) -%} {%- set default_schema = target.schema -%} {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} {# {%- if target.name == 'CI' -%} #} {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} ci_schema {%- elif custom_schema_name is none -%} {{ default_schema }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` macros/generate\_alias\_name.sql ```jinja {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} {# {%- if target.name == 'CI' -%} #} {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} {%- if custom_alias_name -%} {{ target.schema }}__{{ node.config.schema }}__{{ custom_alias_name | trim }} {%- elif node.version -%} {{ target.schema }}__{{ node.config.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} {%- else -%} {{ target.schema }}__{{ node.config.schema }}__{{ node.name }} {%- endif -%} {%- else -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- elif node.version -%} {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} {%- else -%} {{ node.name }} {%- endif -%} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------------------------------ | | Developer 1 | dev | dbt\_dev1 | dev.dbt\_dev1\_marketing.my\_model | | Developer 2 | dev | dbt\_dev2 | dev.dbt\_dev2\_marketing.my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.ci\_schema.dbt\_pr\_123\_marketing\_my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.ci\_schema.dbt\_pr\_234\_marketing\_my\_model | | Production | prod | analytics | prod.marketing.my\_model | #### What not to do This section will provide an outline of what users should avoid doing when customizing their schema and alias due to the issues that may arise. ##### Update generate\_schema\_name() to always use the custom schema Some people prefer to only use the custom schema when it is set instead of concatenating the default schema with the custom one, as it happens in the out of the box behavior. ##### Problem When modifying the default macro for `generate_schema_name()`, this might result in creating this new version. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} # The following is incorrect as it omits {{ default_schema }} before {{ custom_schema_name | trim }}. {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` While it may provide the expected output for production, where a dedicated database is used, it will generate conflicts anywhere people share a database. Let’s look at the example of a model called `my_model` with a custom schema of `marketing`. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------ | | Production | prod | analytics | prod.marketing.my\_model | | Developer 1 | dev | dbt\_dev1 | dev.marketing.my\_model | | Developer 2 | dev | dbt\_dev2 | dev.marketing.my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.marketing.my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.marketing.my\_model | We can see that both developer 1 and developer 2 get the same object for `my_model`. This means that if they both work on this model at the same time, it will be impossible to know if the version currently in the data warehouse is the one from developer 1 and developer 2. Similarly, different PRs will result in the exact same object in the data warehouse. If different PRs are open at the same time and modifying the same models, it is very likely that we will get issues, slowing down the whole development and code promotion. ##### Solution As described in the previous example, update the macro to check if dbt is running in production. Only in production should we remove the concatenation and use the custom schema alone. #### Tips and tricks This section will provide some useful tips on how to properly adjust your `generate_database_name()` and `generate_alias_name()` macros. ##### Creating non existing databases from dbt dbt will automatically try to create a schema if it doesn’t exist and if an object needs to be created in it, but it won’t automatically try to create a database that doesn’t exist. So, if your `generate_database_name()` configuration points to different databases, which might not exist, dbt will fail if you do a simple `dbt build`. It is still possible to get it working in dbt by creating some macros that will check if a database exists and if not, dbt will create it. You can then call those macros either in [a `dbt run-operation ...` step](https://docs.getdbt.com/reference/commands/run-operation.md) in your jobs or as a [`on-run-start` hook](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md). ##### Assuming context using environment variables rather than `target.name` We prefer to use [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) over `target.name` For a further read, have a look at ([About target variables](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md)) to decipher the context of the dbt invocation. * `target.name` cannot be set at the environment-level. Therefore, every job within the environment must explicitly specify the `target.name` override. If the job does not have the appropriate `target.name` value set, the database/schema/alias may not resolve properly. Alternatively, environment variable values are inherited by the jobs within their corresponding environment. The environment variable values can also be overwritten within the jobs if needed. [![Customize schema alias env var.](/img/docs/dbt-platform/using-dbt-platform/custom-schema-env-var-targetname.png?v=2 "Customize schema alias env var.")](#)Customize schema alias env var. * `target.name` requires every developer to input the same value (often ‘dev’) into the target name section of their project user credentials. If a developer doesn’t have the appropriate target name value set, their database/schema/alias may not resolve properly. [![User credentials.](/img/docs/dbt-platform/using-dbt-platform/development-credentials.png?v=2 "User credentials.")](#)User credentials. ##### Always enforce custom schemas Some users prefer to enforce custom schemas on all objects within their projects. This avoids writing to unintended “default” locations. You can add this logic to your `generate_schema_name()` macro to [raise a compilation error](https://docs.getdbt.com/reference/dbt-jinja-functions/exceptions.md) if a custom schema is not defined for an object. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- set node_custom_schema = node.config.get('schema') -%} {%- if custom_schema_name is none and node_custom_schema is none and node.resource_type == 'model' -%} {{ exceptions.raise_compiler_error("Error: No Custom Schema Defined for the model " ~ node.name ) }} {%- elif custom_schema_name is none -%} {{ default_schema }} {%- elif env_var('DBT_ENV_TYPE','DEV') == 'PROD' -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` --- ### Customizing CI/CD with custom pipelines [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction One of the core tenets of dbt is that analytic code should be version controlled. This provides a ton of benefit to your organization in terms of collaboration, code consistency, stability, and the ability to roll back to a prior version. There’s an additional benefit that is provided with your code hosting platform that is often overlooked or underutilized. Some of you may have experience using dbt’s [webhook functionality](https://docs.getdbt.com/docs/deploy/continuous-integration.md) to run a job when a PR is created. This is a fantastic capability, and meets most use cases for testing your code before merging to production. However, there are circumstances when an organization needs additional functionality, like running workflows on every commit (linting), or running workflows after a merge is complete. In this article, we will show you how to setup custom pipelines to lint your project and trigger a dbt job via the API. A note on parlance in this article since each code hosting platform uses different terms for similar concepts. The terms `pull request` (PR) and `merge request` (MR) are used interchangeably to mean the process of merging one branch into another branch. ##### What are pipelines? Pipelines (which are known by many names, such as workflows, actions, or build steps) are a series of pre-defined jobs that are triggered by specific events in your repository (PR created, commit pushed, branch merged, etc). Those jobs can do pretty much anything your heart desires assuming you have the proper security access and coding chops. Jobs are executed on [runners](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#runners), which are virtual servers. The runners come pre-configured with Ubuntu Linux, macOS, or Windows. That means the commands you execute are determined by the operating system of your runner. You’ll see how this comes into play later in the setup, but for now just remember that your code is executed on virtual servers that are, typically, hosted by the code hosting platform. [![Diagram of how pipelines work](/img/guides/orchestration/custom-cicd-pipelines/pipeline-diagram.png?v=2 "Diagram of how pipelines work")](#)Diagram of how pipelines work Please note, runners hosted by your code hosting platform provide a certain amount of free time. After that, billing charges may apply depending on how your account is setup. You also have the ability to host your own runners. That is beyond the scope of this article, but checkout the links below for more information if you’re interested in setting that up: * Repo-hosted runner billing information: * [GitHub](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions) * [GitLab](https://docs.gitlab.com/ee/ci/pipelines/cicd_minutes.html) * [Bitbucket](https://bitbucket.org/product/features/pipelines#) * Self-hosted runner information: * [GitHub](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) * [GitLab](https://docs.gitlab.com/runner/) * [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/runners/) Additionally, if you’re using the free tier of GitLab you can still follow this guide, but it may ask you to provide a credit card to verify your account. You’ll see something like this the first time you try to run a pipeline: [![Warning from GitLab showing payment information is required](/img/guides/orchestration/custom-cicd-pipelines/gitlab-cicd-payment-warning.png?v=2 "Warning from GitLab showing payment information is required")](#)Warning from GitLab showing payment information is required ##### How to setup pipelines This guide provides details for multiple code hosting platforms. Where steps are unique, they are presented without a selection option. If code is specific to a platform (i.e. GitHub, GitLab, Bitbucket) you will see a selection option for each. Pipelines can be triggered by various events. The [dbt webhook](https://docs.getdbt.com/docs/deploy/continuous-integration.md) process already triggers a run if you want to run your jobs on a merge request, so this guide focuses on running pipelines for every push and when PRs are merged. Since pushes happen frequently in a project, we’ll keep this job super simple and fast by linting with SQLFluff. The pipeline that runs on merge requests will run less frequently, and can be used to call the dbt API to trigger a specific job. This can be helpful if you have specific requirements that need to happen when code is updated in production, like running a `--full-refresh` on all impacted incremental models. Here’s a quick look at what this pipeline will accomplish: [![Diagram showing the pipelines to be created and the programs involved](/img/guides/orchestration/custom-cicd-pipelines/pipeline-programs-diagram.png?v=2 "Diagram showing the pipelines to be created and the programs involved")](#)Diagram showing the pipelines to be created and the programs involved #### Run a dbt job on merge This job will take a bit more to setup, but is a good example of how to call the dbt API from a CI/CD pipeline. The concepts presented here can be generalized and used in whatever way best suits your use case. Run on merge If your Git provider has a native integration with dbt, you can take advantage of setting up [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) in the UI. The setup below shows how to call the dbt API to run a job every time there's a push to your [default branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch). ##### 1. Get your dbt API key When running a CI/CD pipeline you’ll want to use a service token instead of any individual’s API key. There are [detailed docs](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) available on this, but below is a quick rundown (this must be performed by an Account Admin): 1. Log in to your dbt account. 2. Click your account name at the bottom left-hand menu and go to **Account settings**. 3. Click [**Service tokens**](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) on the left. 4. Click **+ Create service token** to create a new token specifically for CI/CD API calls. 5. Name your token something like “CICD Token”. 6. Click the **+Add permission** button under **Access**, and grant this token the **Job Admin** permission. 7. Click **Save** and you’ll see a grey box appear with your token. Copy that and save it somewhere safe (this is a password, and should be treated as such). [![View of the dbt page where service tokens are created](/img/guides/orchestration/custom-cicd-pipelines/dbt-service-token-page.png?v=2 "View of the dbt page where service tokens are created")](#)View of the dbt page where service tokens are created [![Creating a new service token](/img/guides/orchestration/custom-cicd-pipelines/dbt-new-service-token-page.png?v=2 "Creating a new service token")](#)Creating a new service token ##### 2. Put your dbt API key into your repo Adding you dbt API key to your repo happens in your code hosting platform. You need to save your API key from the previous section as a repository secret so the job we create can access it. Never save passwords or API keys in your code. The following steps ensure that your key stays secure, but is still usable for your pipelines. ##### GitHub * Open up your repository where you want to run the pipeline (the same one that houses your dbt project). * Click *Settings* to open up the repository options. * On the left click the *Secrets and variables* dropdown in the *Security* section. * From that list, click on *Actions*. * Towards the middle of the screen, click the *New repository secret* button. * It will ask you for a name, so let’s call ours `DBT_API_KEY`. * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the *Secret* section, paste in the key you copied from dbt. * Click *Add secret* and you’re all set! \*\* A quick note on security: while using a repository secret is the most straightforward way to setup this secret, there are other options available to you in GitHub. They’re beyond the scope of this guide, but could be helpful if you need to create a more secure environment for running actions. Checkout GitHub’s documentation on secrets [here](https://docs.github.com/en/actions/security-guides/encrypted-secrets).\* Here’s a video showing these steps: ##### GitLab * Open up your repository where you want to run the pipeline (the same one that houses your dbt project). * Click *Settings* > *CI/CD*. * Under the *Variables* section, click *Expand,* then click *Add variable*. * It will ask you for a name, so let’s call ours `DBT_API_KEY`. * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the *Value* section, paste in the key you copied from dbt. * Make sure the check box next to *Protect variable* is unchecked, and the box next to *Mask variable* is selected (see below). * “Protected” means that the variable is only available in pipelines that run on protected branches or protected tags - that won’t work for us because we want to run this pipeline on multiple branches. “Masked” means that it will be available to your pipeline runner, but will be masked in the logs. [![\[View of the GitLab window for entering DBT\_API\_KEY](/img/guides/orchestration/custom-cicd-pipelines/dbt-api-key-gitlab.png?v=2 "\[View of the GitLab window for entering DBT_API_KEY")](#)\[View of the GitLab window for entering DBT\_API\_KEY Here’s a video showing these steps: ##### Azure DevOps In Azure: * Open up your Azure DevOps project where you want to run the pipeline (the same one that houses your dbt project). * Click on *Pipelines* and then *Create Pipeline*. * Select where your git code is located. It should be *Azure Repos Git*. * Select your git repository from the list. * Select *Starter pipeline* (this will be updated later in Step 4). * Click on *Variables* and then *New variable*. * In the *Name* field, enter the `DBT_API_KEY`. * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the *Value* section, paste in the key you copied from dbt. * Make sure the check box next to *Keep this value secret* is checked. This will mask the value in logs, and you won't be able to see the value for the variable in the UI. * Click *OK* and then *Save* to save the variable. * Save your new Azure pipeline. [![View of the Azure pipelines window for entering DBT\_API\_KEY](/img/guides/orchestration/custom-cicd-pipelines/dbt-api-key-azure.png?v=2 "View of the Azure pipelines window for entering DBT_API_KEY")](#)View of the Azure pipelines window for entering DBT\_API\_KEY ##### Bitbucket In Bitbucket: * Open up your repository where you want to run the pipeline (the same one that houses your dbt project). * In the left menu, click **Repository Settings**. * Scroll to the bottom of the left menu, and select **Repository variables**. * In the **Name** field, input `DBT_API_KEY` * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the Value section, paste in the key you copied from dbt * Make sure the check box next to Secured is checked. This will mask the value in logs, and you won't be able to see the value for the variable in the UI. * Click **Add** to save the variable [![View of the Bitbucket window for entering DBT\_API\_KEY](/img/guides/orchestration/custom-cicd-pipelines/dbt-api-key-bitbucket.png?v=2 "View of the Bitbucket window for entering DBT_API_KEY")](#)View of the Bitbucket window for entering DBT\_API\_KEY Here’s a video showing these steps: ##### CodeCommit In AWS, store your API key in Systems Manager (SSM) Parameter Store instead of a repository secret: 1. Open the AWS console and go to **Systems Manager** → **Parameter Store**. 2. Click **Create parameter**. 3. Enter a name for the parameter (for example, `DBT_API_KEY`). 4. Set the type to **SecureString** and paste in your dbt service token as the value. 5. Click **Create parameter**. You’ll reference this parameter in the CodeBuild project environment in step 4. ##### 3. Create script to trigger dbt job via an API call In your project, create a new folder at the root level named `python`. In that folder, create a file named `run_and_monitor_dbt_job.py`. You'll copy/paste the contents from this [gist](https://gist.github.com/b-per/f4942acb8584638e3be363cb87769b48) into that file. ```yaml my_awesome_project ├── python │ └── run_and_monitor_dbt_job.py ``` The following Python file has everything you need to call the dbt API, but requires a few inputs as outlined below. These inputs will be fed to this script through environment variables that will be defined in the next step. ```python #------------------------------------------------------------------------------ # get environment variables #------------------------------------------------------------------------------ api_base = os.getenv('DBT_URL', 'https://cloud.getdbt.com/') # default to multitenant url job_cause = os.getenv('DBT_JOB_CAUSE', 'API-triggered job') # default to generic message git_branch = os.getenv('DBT_JOB_BRANCH', None) # default to None schema_override = os.getenv('DBT_JOB_SCHEMA_OVERRIDE', None) # default to None api_key = os.environ['DBT_API_KEY'] # no default here, just throw an error here if key not provided account_id = os.environ['DBT_ACCOUNT_ID'] # no default here, just throw an error here if id not provided project_id = os.environ['DBT_PROJECT_ID'] # no default here, just throw an error here if id not provided job_id = os.environ['DBT_PR_JOB_ID'] # no default here, just throw an error here if id not provided ``` **Required input:** In order to call the dbt API, there are a few pieces of info the script needs. The easiest way to get these values is to open up the job you want to run in dbt. The URL when you’re inside the job has all the values you need: * `DBT_ACCOUNT_ID` - this is the number just after `accounts/` in the URL * `DBT_PROJECT_ID` - this is the number just after `projects/` in the URL * `DBT_PR_JOB_ID` - this is the number just after `jobs/` in the URL [![Image of a dbt job URL with the pieces for account, project, and job highlighted](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-url.png?v=2 "Image of a dbt job URL with the pieces for account, project, and job highlighted")](#)Image of a dbt job URL with the pieces for account, project, and job highlighted ##### 4. Update your project to include the new API call ##### GitHub For this new job, we'll add a file for the dbt API call named `dbt_run_on_merge.yml`. ```yaml my_awesome_project ├── python │ └── run_and_monitor_dbt_job.py ├── .github │ ├── workflows │ │ └── dbt_run_on_merge.yml │ │ └── lint_on_push.yml ``` The YAML file will look pretty similar to our earlier job, but there is a new section called `env` that we’ll use to pass in the required variables. Update the variables below to match your setup based on the comments in the file. It’s worth noting that we changed the `on:` section to now run **only** when there are pushes to a branch named `main` (for example, a pull request is merged). Have a look through [GitHub documentation](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows) on these filters for additional use cases. For information about `github` context property names and their use cases, refer to the [GitHub documentation](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs). ```yaml name: run dbt job on push # This filter says only run this job when there is a push to the default branch # This works off the assumption that you've restricted this branch to only all PRs to push to the default branch # Update the name to match the name of your default branch on: push: branches: - 'main' jobs: # the job calls the dbt API to run a job run_dbt_cloud_job: name: Run dbt Job runs-on: ubuntu-latest # Set the environment variables needed for the run env: DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id DBT_API_KEY: ${{ secrets.DBT_API_KEY }} DBT_URL: https://cloud.getdbt.com # enter a URL that matches your job DBT_JOB_CAUSE: 'GitHub Pipeline CI Job' DBT_JOB_BRANCH: ${{ github.head_ref }} # Resolves to the head_ref or source branch of the pull request in a workflow run. steps: - uses: "actions/checkout@v4" - uses: "actions/setup-python@v5" with: python-version: "3.9" - name: Run dbt job run: "python python/run_and_monitor_dbt_job.py" ``` ##### GitLab For this job, we'll set it up using the `gitlab-ci.yml` file as in the prior step (see Step 1 of the linting setup for more info). The YAML file will look pretty similar to our earlier job, but there is a new section called `variables` that we’ll use to pass in the required variables to the Python script. Update this section to match your setup based on the comments in the file. Please note that the `rules:` section now says to run **only** when there are pushes to a branch named `main`, such as a PR being merged. Have a look through [GitLab’s docs](https://docs.gitlab.com/ee/ci/yaml/#rules) on these filters for additional use cases. ##### Only dbt job ```yaml image: python:3.9 variables: DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id DBT_API_KEY: $DBT_API_KEY # secret variable in gitlab account DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'GitLab Pipeline CI Job' DBT_JOB_BRANCH: $CI_COMMIT_BRANCH stages: - build # this job calls the dbt API to run a job run-dbt-cloud-job: stage: build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == 'main' script: - python python/run_and_monitor_dbt_job.py ``` ##### Lint and dbt job ```yaml image: python:3.9 variables: DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id DBT_API_KEY: $DBT_API_KEY # secret variable in gitlab account DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'GitLab Pipeline CI Job' DBT_JOB_BRANCH: $CI_COMMIT_BRANCH stages: - pre-build - build # this job runs SQLFluff with a specific set of rules # note the dialect is set to Snowflake, so make that specific to your setup # details on linter rules: https://docs.sqlfluff.com/en/stable/rules.html lint-project: stage: pre-build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != 'main' script: - python -m pip install sqlfluff==0.13.1 - sqlfluff lint models --dialect snowflake --rules L019,L020,L021,L022 # this job calls the dbt API to run a job run-dbt-cloud-job: stage: build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == 'main' script: - python python/run_and_monitor_dbt_job.py ``` ##### Azure DevOps For this new job, open the existing Azure pipeline you created above and select the *Edit* button. We'll want to edit the corresponding Azure pipeline YAML file with the appropriate configuration, instead of the starter code, along with including a `variables` section to pass in the required variables. Copy the below YAML file into your Azure pipeline and update the variables below to match your setup based on the comments in the file. It's worth noting that we changed the `trigger` section so that it will run *only* when there are pushes (for example, when a PR is merged) to `main`, but you should use the name of your [default branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch). Read through [Azure's docs](https://learn.microsoft.com/en-us/azure/devops/pipelines/build/triggers?view=azure-devops) on these filters for additional use cases. ```yaml name: Run dbt Job trigger: [ main ] # runs on pushes to main variables: DBT_URL: https://cloud.getdbt.com # no trailing backslash, adjust this accordingly for single-tenant deployments DBT_JOB_CAUSE: 'Azure Pipeline CI Job' # provide a descriptive job cause here for easier debugging down the road DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id steps: - task: UsePythonVersion@0 inputs: versionSpec: '3.7' displayName: 'Use Python 3.7' - script: | python -m pip install requests displayName: 'Install python dependencies' - script: | python -u ./python/run_and_monitor_dbt_job.py displayName: 'Run dbt job ' env: DBT_API_KEY: $(DBT_API_KEY) # Set these values as secrets in the Azure pipelines Web UI ``` ##### Bitbucket For this job, we'll set it up using the `bitbucket-pipelines.yml` file as in the prior step (see Step 1 of the linting setup for more info). The YAML file will look pretty similar to our earlier job, but we’ll pass in the required variables to the Python script using `export` statements. Update this section to match your setup based on the comments in the file. ##### Only job ```yaml image: python:3.11.1 pipelines: branches: 'main': # override if your default branch doesn't run on a branch named "main" - step: name: 'Run dbt Job' script: - export DBT_URL="https://cloud.getdbt.com" # if you have a single-tenant deployment, adjust this accordingly - export DBT_JOB_CAUSE="Bitbucket Pipeline CI Job" - export DBT_ACCOUNT_ID=00000 # enter your account id here - export DBT_PROJECT_ID=00000 # enter your project id here - export DBT_PR_JOB_ID=00000 # enter your job id here - python python/run_and_monitor_dbt_job.py ``` ##### Lint and dbt job ```yaml image: python:3.11.1 pipelines: branches: '**': # this sets a wildcard to run on every branch unless specified by name below - step: name: Lint dbt project script: - python -m pip install sqlfluff==0.13.1 - sqlfluff lint models --dialect snowflake --rules L019,L020,L021,L022 'main': # override if your default branch doesn't run on a branch named "main" - step: name: 'Run dbt Job' script: - export DBT_URL="https://cloud.getdbt.com" # if you have a single-tenant deployment, adjust this accordingly - export DBT_JOB_CAUSE="Bitbucket Pipeline CI Job" - export DBT_ACCOUNT_ID=00000 # enter your account id here - export DBT_PROJECT_ID=00000 # enter your project id here - export DBT_PR_JOB_ID=00000 # enter your job id here - python python/run_and_monitor_dbt_job.py ``` ##### CodeCommit For CodeCommit, you'll configure a CodeBuild project and an EventBridge rule to trigger a dbt job every time code is pushed to your default branch (for example, when a pull request is merged to `main`). Add the following files to your dbt project: ```text my_awesome_project ├── ci-configuration │ └── buildspec-merge.yml ├── python │ └── run_and_monitor_dbt_cloud_job.py ``` **Create `python/run_and_monitor_dbt_cloud_job.py` with the following content:** ```python import requests import os import time api_base = os.getenv('DBT_URL', 'https://cloud.getdbt.com') job_cause = os.getenv('DBT_JOB_CAUSE', 'API-triggered job') git_branch = os.getenv('DBT_JOB_BRANCH', None) schema_override = os.getenv('DBT_JOB_SCHEMA_OVERRIDE', None) api_key = os.environ['DBT_API_KEY'] account_id = os.environ['DBT_ACCOUNT_ID'] project_id = os.environ['DBT_PROJECT_ID'] job_id = os.environ['DBT_PR_JOB_ID'] print(f""" Configuration: api_base: {api_base} job_cause: {job_cause} git_branch: {git_branch} schema_override: {schema_override} account_id: {account_id} project_id: {project_id} job_id: {job_id} """) req_auth_header = {'Authorization': f'Token {api_key}'} req_job_url = f'{api_base}/api/v2/accounts/{account_id}/jobs/{job_id}/run/' run_status_map = { 1: 'Queued', 2: 'Starting', 3: 'Running', 10: 'Success', 20: 'Error', 30: 'Cancelled', } def run_job(url, headers, cause, branch=None, schema_override=None) -> int: req_payload = {'cause': cause} if branch: req_payload['git_branch'] = branch.replace('refs/heads/', '') if schema_override: req_payload['schema_override'] = schema_override.replace('-', '_') print(f'Triggering job:\n\turl: {url}\n\tpayload: {req_payload}') run_job_resp = requests.post(url, headers=headers, data=req_payload).json() return run_job_resp['data']['id'] def get_run_status(url, headers) -> str: req_status_resp = requests.get(url, headers=headers).json() run_status_code = req_status_resp['data']['status'] return run_status_map[run_status_code] def main(): print('Beginning request for job run...') try: run_id = run_job(req_job_url, req_auth_header, job_cause, git_branch, schema_override) except Exception as e: print(f'ERROR! - Could not trigger job:\n {e}') raise req_status_url = f'{api_base}/api/v2/accounts/{account_id}/runs/{run_id}/' run_status_link = f'{api_base}/#/accounts/{account_id}/projects/{project_id}/runs/{run_id}/' print(f'Job running! See job status at {run_status_link}') time.sleep(30) while True: status = get_run_status(req_status_url, req_auth_header) print(f'Run status -> {status}') if status in ['Error', 'Cancelled']: raise Exception(f'Run failed or canceled. See why at {run_status_link}') if status == 'Success': print(f'Job completed successfully! See details at {run_status_link}') return time.sleep(10) if __name__ == "__main__": main() ``` **Create `ci-configuration/buildspec-merge.yml` with the following content:** Replace the placeholder values with your actual dbt account details: * `YOUR_DBT_ACCOUNT_ID`: The number after `accounts/` in your dbt job URL * `YOUR_DBT_PROJECT_ID`: The number after `projects/` in your dbt job URL * `YOUR_DBT_MERGE_JOB_ID`: The number after `jobs/` in the URL of the dbt job you want to run on merge * `YOUR_SSM_PARAMETER_NAME`: The name of the SSM parameter you created in step 2 (for example, `DBT_API_KEY`) ```yaml version: 0.2 env: variables: DBT_ACCOUNT_ID: YOUR_DBT_ACCOUNT_ID DBT_PROJECT_ID: YOUR_DBT_PROJECT_ID DBT_PR_JOB_ID: YOUR_DBT_MERGE_JOB_ID DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'CodeCommit Merge CI Job' parameter-store: DBT_API_KEY: "YOUR_SSM_PARAMETER_NAME" phases: install: commands: - apt-get update -y - python -m pip install --upgrade pip - pip install requests build: commands: - python -u ./python/run_and_monitor_dbt_cloud_job.py ``` **Create a CodeBuild project** Follow the [AWS CodeBuild documentation](https://docs.aws.amazon.com/codebuild/latest/userguide/create-project-console.html) to create a project. Configure the following settings: * **Source**: Select **AWS CodeCommit**, set the reference type to **Branch**, and select your main branch * **Environment**: Select **Ubuntu** as the operating system * **Buildspec**: Select **Use a buildspec file** and enter `ci-configuration/buildspec-merge.yml` as the path Note the service role name shown in the **Environment** section. In **IAM**, find that role and add an inline policy granting permission to read and decrypt the SSM parameter: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["ssm:GetParameter", "ssm:GetParameters"], "Resource": "arn:aws:ssm:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:parameter/YOUR_SSM_PARAMETER_NAME" }, { "Effect": "Allow", "Action": ["kms:Decrypt"], "Resource": "arn:aws:kms:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:key/alias/aws/ssm" } ] } ``` **Create an EventBridge rule** 1. In the AWS console, go to **Amazon EventBridge** → **Rules** → **Create rule** 2. Give the rule a name (for example, `dbt-run-on-merge`) 3. Under **Event pattern**, select **Custom pattern** and paste the following JSON. Substitute `YOUR_AWS_ACCOUNT_ID`, `YOUR_AWS_REGION`, and `YOUR_CODECOMMIT_REPO_NAME` with your values: ```json { "source": ["aws.codecommit"], "account": ["YOUR_AWS_ACCOUNT_ID"], "region": ["YOUR_AWS_REGION"], "detail-type": ["CodeCommit Repository State Change"], "resources": ["arn:aws:codecommit:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:YOUR_CODECOMMIT_REPO_NAME"], "detail": { "event": ["referenceUpdated"], "referenceType": ["branch"], "referenceName": ["main"] } } ``` 4. Under **Target**, select **AWS service** → **CodeBuild** and choose the project you created above 5. When prompted, allow EventBridge to create a new IAM role for the target automatically 6. Save the rule ##### 5. Test your new action Now that you have a shiny new action, it’s time to test it out! Since this change is setup to only run on merges to your default branch, you’ll need to create and merge this change into your main branch. Once you do that, you’ll see a new pipeline job has been triggered to run the dbt job you assigned in the variables section. Additionally, you’ll see the job in the run history of dbt. It should be fairly easy to spot because it will say it was triggered by the API, and the *INFO* section will have the branch you used for this guide. ##### GitHub [![dbt run on merge job in GitHub](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-github.png?v=2 "dbt run on merge job in GitHub")](#)dbt run on merge job in GitHub [![dbt job showing it was triggered by GitHub](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-github-triggered.png?v=2 "dbt job showing it was triggered by GitHub")](#)dbt job showing it was triggered by GitHub ##### GitLab [![dbt run on merge job in GitLab](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-gitlab.png?v=2 "dbt run on merge job in GitLab")](#)dbt run on merge job in GitLab [![dbt job showing it was triggered by GitLab](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-gitlab-triggered.png?v=2 "dbt job showing it was triggered by GitLab")](#)dbt job showing it was triggered by GitLab ##### Azure DevOps [![dbt run on merge job in ADO](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-azure.png?v=2 "dbt run on merge job in ADO")](#)dbt run on merge job in ADO [![ADO-triggered job in dbt](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-azure-triggered.png?v=2 "ADO-triggered job in dbt")](#)ADO-triggered job in dbt ##### Bitbucket [![dbt run on merge job in Bitbucket](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-bitbucket.png?v=2 "dbt run on merge job in Bitbucket")](#)dbt run on merge job in Bitbucket [![dbt job showing it was triggered by Bitbucket](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-bitbucket-triggered.png?v=2 "dbt job showing it was triggered by Bitbucket")](#)dbt job showing it was triggered by Bitbucket ##### CodeCommit To test the pipeline, merge a pull request (or push a commit directly) to your main branch. You should see a new build start in your CodeBuild project within a few seconds. You can monitor the run in the CodeBuild console under **Build history**. [![Find new build under build history of CodeBuild console](/img/guides/orchestration/custom-cicd-pipelines/codecommit-run-on-pr.png?v=2 "Find new build under build history of CodeBuild console")](#)Find new build under build history of CodeBuild console In dbt, the job will appear in your run history with a cause of "CodeCommit Merge CI Job". [![Check dbt run history for CodeCommit build triggered](/img/guides/orchestration/custom-cicd-pipelines/codecommit-dbt-cloud-pr.png?v=2 "Check dbt run history for CodeCommit build triggered")](#)Check dbt run history for CodeCommit build triggered #### Run a dbt job on pull request If your git provider is not one with a native integration with dbt, but you still want to take advantage of CI builds, you've come to the right spot! With just a bit of work it's possible to setup a job that will run a dbt job when a pull request (PR) is created. Run on PR If your git provider has a native integration with dbt, you can take advantage of the [setup instructions](https://docs.getdbt.com/docs/deploy/ci-jobs.md). This section is only for those projects that connect to their git repository using an SSH key. The setup for this pipeline will use the same steps as the prior page. Before moving on, make sure you follow [steps 1-5](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=2) from the previous page. ##### 1. Create a pipeline job that runs when PRs are created ##### Bitbucket For this job, we’ll set it up using the `bitbucket-pipelines.yml` file as in the prior step. The YAML file will look pretty similar to our earlier job, but we’ll pass in the required variables to the Python script using `export` statements. Update this section to match your setup based on the comments in the file. **What is this pipeline going to do?**<br />The setup below will trigger a dbt job to run every time a PR is opened in this repository. It will also run a fresh version of the pipeline for every commit that is made on the PR until it is merged. For example, if you open a PR, it will run the pipeline. If you then decide additional changes are needed, and commit/push to the PR branch, a new pipeline will run with the updated code. The following variables control this job: * `DBT_JOB_BRANCH`: Tells the dbt job to run the code in the branch that created this PR * `DBT_JOB_SCHEMA_OVERRIDE`: Tells the dbt job to run this into a custom target schema * The format of this will look like: `DBT_CLOUD_PR_{REPO_KEY}_{PR_NUMBER}` ```yaml image: python:3.11.1 pipelines: # This job will run when pull requests are created in the repository pull-requests: '**': - step: name: 'Run dbt PR Job' script: # Check to only build if PR destination is master (or other branch). # Comment or remove line below if you want to run on all PR’s regardless of destination branch. - if [ "${BITBUCKET_PR_DESTINATION_BRANCH}" != "main" ]; then printf ‘PR Destination is not master, exiting.’; exit; fi - export DBT_URL="https://cloud.getdbt.com" - export DBT_JOB_CAUSE="Bitbucket Pipeline CI Job" - export DBT_JOB_BRANCH=$BITBUCKET_BRANCH - export DBT_JOB_SCHEMA_OVERRIDE="DBT_CLOUD_PR_"$BITBUCKET_PROJECT_KEY"_"$BITBUCKET_PR_ID - export DBT_ACCOUNT_ID=00000 # enter your account id here - export DBT_PROJECT_ID=00000 # enter your project id here - export DBT_PR_JOB_ID=00000 # enter your job id here - python python/run_and_monitor_dbt_job.py ``` ##### CodeCommit The setup for CodeCommit follows the same steps as the prior page. Before moving on, follow steps 1–5 from the [prior page](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=2) to store your API key, add the Python script to your repository, and configure the merge pipeline. Prerequisites Before proceeding, make sure you have also: * Connected your dbt project to CodeCommit by following the [CodeCommit integration guide](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#aws-codecommit) * Configured the [pull request URL template for CodeCommit](https://docs.getdbt.com/docs/platform/git/pr-template.md#aws-codecommit) in your dbt project settings * Created a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) in dbt — do not configure it to trigger on pull requests, since this pipeline will trigger it through the API The pull request pipeline uses a separate CodeBuild project from the merge pipeline, because it runs a different dbt job (your CI job) and requires branch and schema values that are dynamically passed at build time from the pull request event. **1. Add `ci-configuration/buildspec.yml` to your project** This is separate from the `buildspec-merge.yml` you created in the prior step. It references your CI job and leaves `DBT_JOB_BRANCH` and `DBT_JOB_SCHEMA_OVERRIDE` empty — a Lambda function will pass those values at build time. Replace the placeholder values with your actual dbt account details: * `YOUR_DBT_ACCOUNT_ID`: The number after `accounts/` in your dbt job URL * `YOUR_DBT_PROJECT_ID`: The number after `projects/` in your dbt job URL * `YOUR_DBT_PR_JOB_ID`: The number after `jobs/` in the URL of your CI job * `YOUR_SSM_PARAMETER_NAME`: The name of the SSM parameter you created in step 2 of the prior page (for example, `DBT_API_KEY`) ```yaml version: 0.2 env: variables: DBT_ACCOUNT_ID: YOUR_DBT_ACCOUNT_ID DBT_PROJECT_ID: YOUR_DBT_PROJECT_ID DBT_PR_JOB_ID: YOUR_DBT_PR_JOB_ID DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'CodeCommit Pipeline CI Job' DBT_JOB_BRANCH: '' DBT_JOB_SCHEMA_OVERRIDE: '' parameter-store: DBT_API_KEY: "YOUR_SSM_PARAMETER_NAME" phases: install: commands: - apt-get update -y - python -m pip install --upgrade pip - pip install requests build: commands: - python -u ./python/run_and_monitor_dbt_cloud_job.py ``` **2. Create a CodeBuild project for pull requests** Follow the [AWS CodeBuild documentation](https://docs.aws.amazon.com/codebuild/latest/userguide/create-project-console.html) to create a second project (separate from the one you created for merge). Configure the following settings: * **Source**: Select **AWS CodeCommit**, set the reference type to **Branch**, and select your main branch * **Environment**: Select **Ubuntu** as the operating system * **Buildspec**: Select **Use a buildspec file** and enter `ci-configuration/buildspec.yml` as the path Note the service role name shown in the **Environment** section. In **IAM**, find that role and add an inline policy granting permission to read and decrypt the SSM parameter: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["ssm:GetParameter", "ssm:GetParameters"], "Resource": "arn:aws:ssm:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:parameter/YOUR_SSM_PARAMETER_NAME" }, { "Effect": "Allow", "Action": ["kms:Decrypt"], "Resource": "arn:aws:kms:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:key/alias/aws/ssm" } ] } ``` **3. Create a Lambda trigger function** The Lambda function receives the pull request event from EventBridge, extracts the PR branch and PR ID, then starts CodeBuild with those values as environment variable overrides. This is what ensures the dbt CI job runs against your PR branch with a PR-specific schema. 1. In the AWS console, go to **Lambda** → **Create function** 2. Select **Author from scratch**, name the function (for example, `dbt-ci-trigger`), and choose **Python 3.12** as the runtime 3. Click **Create function**, then replace the default code with the following: ```python import boto3 def lambda_handler(event, context): cb = boto3.client('codebuild') source_ref = event['detail']['sourceReference'] pr_id = event['detail']['pullRequestId'] repo_name = event['detail']['repositoryNames'][0] branch = source_ref.replace('refs/heads/', '') schema_override = f'DBT_CLOUD_PR_{repo_name}_{pr_id}'.replace('-', '_').upper() print(f'Triggering CodeBuild for branch: {branch}, schema_override: {schema_override}') cb.start_build( projectName='YOUR_CODEBUILD_PROJECT_NAME', environmentVariablesOverride=[ {'name': 'DBT_JOB_BRANCH', 'value': branch, 'type': 'PLAINTEXT'}, {'name': 'DBT_JOB_SCHEMA_OVERRIDE', 'value': schema_override, 'type': 'PLAINTEXT'} ] ) ``` Replace `YOUR_CODEBUILD_PROJECT_NAME` with the name of the CodeBuild project you created above, then click **Deploy**. 4. Go to **Configuration** → **Permissions** → click the execution role name to open IAM 5. Add an inline policy granting permission to start the CodeBuild project: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "codebuild:StartBuild", "Resource": "arn:aws:codebuild:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:project/YOUR_CODEBUILD_PROJECT_NAME" } ] } ``` **4. Create an EventBridge rule** 1. In the AWS console, go to **Amazon EventBridge** → **Rules** → **Create rule** 2. Give the rule a name (for example, `dbt-ci-on-pr-created`) 3. Under **Event pattern**, select **Custom pattern** and paste the following JSON. Substitute `YOUR_AWS_ACCOUNT_ID`, `YOUR_AWS_REGION`, and `YOUR_CODECOMMIT_REPO_NAME` with your values: ```json { "source": ["aws.codecommit"], "account": ["YOUR_AWS_ACCOUNT_ID"], "region": ["YOUR_AWS_REGION"], "detail-type": ["CodeCommit Pull Request State Change"], "resources": ["arn:aws:codecommit:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:YOUR_CODECOMMIT_REPO_NAME"], "detail": { "event": ["pullRequestCreated"], "repositoryNames": ["YOUR_CODECOMMIT_REPO_NAME"] } } ``` 4. Under **Target**, select **AWS service** → **Lambda function**, and choose the Lambda function you created above 5. Save the rule ##### 2. Confirm the pipeline runs Now that you have a new pipeline, it's time to run it and make sure it works. Since this only triggers when a PR is created, you'll need to create a new PR on a branch that contains the code above. Once you do that, you should see a pipeline that looks like this: ##### Bitbucket Bitbucket pipeline: [![dbt run on PR job in Bitbucket](/img/guides/orchestration/custom-cicd-pipelines/bitbucket-run-on-pr.png?v=2 "dbt run on PR job in Bitbucket")](#)dbt run on PR job in Bitbucket dbt job: [![dbt job showing it was triggered by Bitbucket](/img/guides/orchestration/custom-cicd-pipelines/bitbucket-dbt-cloud-pr.png?v=2 "dbt job showing it was triggered by Bitbucket")](#)dbt job showing it was triggered by Bitbucket ##### CodeCommit To test the pipeline, create a new pull request in your CodeCommit repository on a branch that contains the files you added above. Once you open the pull request, the EventBridge rule fires, Lambda extracts the branch and PR ID, and CodeBuild starts automatically. CodeBuild run: [![CodeBuild run triggered by a CodeCommit pull request](/img/guides/orchestration/custom-cicd-pipelines/codecommit-run-on-pr.png?v=2 "CodeBuild run triggered by a CodeCommit pull request")](#)CodeBuild run triggered by a CodeCommit pull request In dbt, the job run appears in your run history. It will show the PR branch and a schema override in the format `DBT_CLOUD_PR_{REPO_NAME}_{PR_NUMBER}`: [![dbt job showing it was triggered by a CodeCommit pull request](/img/guides/orchestration/custom-cicd-pipelines/codecommit-dbt-cloud-pr.png?v=2 "dbt job showing it was triggered by a CodeCommit pull request")](#)dbt job showing it was triggered by a CodeCommit pull request ##### 3. Handle those extra schemas in your database As noted above, when the PR job runs it will create a new schema based on the PR. To avoid having your database overwhelmed with PR schemas, consider adding a "cleanup" job to your dbt account. This job can run on a scheduled basis to cleanup any PR schemas that haven't been updated/used recently. Add this as a macro to your project. It takes 2 arguments that lets you control which schema get dropped: * `age_in_days`: The number of days since the schema was last altered before it should be dropped (default 10 days) * `database_to_clean`: The name of the database to remove schemas from ```sql {# This macro finds PR schemas older than a set date and drops them The macro defaults to 10 days old, but can be configured with the input argument age_in_days Sample usage with different date: dbt run-operation pr_schema_cleanup --args "{'database_to_clean': 'analytics','age_in_days':'15'}" #} {% macro pr_schema_cleanup(database_to_clean, age_in_days=10) %} {% set find_old_schemas %} select 'drop schema {{ database_to_clean }}.'||schema_name||';' from {{ database_to_clean }}.information_schema.schemata where catalog_name = '{{ database_to_clean | upper }}' and schema_name ilike 'DBT_CLOUD_PR%' and last_altered <= (current_date() - interval '{{ age_in_days }} days') {% endset %} {% if execute %} {{ log('Schema drop statements:' ,True) }} {% set schema_drop_list = run_query(find_old_schemas).columns[0].values() %} {% for schema_to_drop in schema_drop_list %} {% do run_query(schema_to_drop) %} {{ log(schema_to_drop ,True) }} {% endfor %} {% endif %} {% endmacro %} ``` This macro goes into a dbt job that is run on a schedule. The command will look like this (text below for copy/paste): [![dbt job showing the run operation command for the cleanup macro](/img/guides/orchestration/custom-cicd-pipelines/dbt-macro-cleanup-pr.png?v=2 "dbt job showing the run operation command for the cleanup macro")](#)dbt job showing the run operation command for the cleanup macro `dbt run-operation pr_schema_cleanup --args "{ 'database_to_clean': 'development','age_in_days':15}"` #### Consider risk of conflicts when using multiple orchestration tools Running dbt jobs through a CI/CD pipeline is a form of job orchestration. If you also run jobs using dbt’s built in scheduler, you now have 2 orchestration tools running jobs. The risk with this is that you could run into conflicts - you can imagine a case where you are triggering a pipeline on certain actions and running scheduled jobs in dbt, you would probably run into job clashes. The more tools you have, the more you have to make sure everything talks to each other. That being said, if **the only reason you want to use pipelines is for adding a lint check or run on merge**, you might decide the pros outweigh the cons, and as such you want to go with a hybrid approach. Just keep in mind that if two processes try and run the same job at the same time, dbt will queue the jobs and run one after the other. It’s a balancing act but can be accomplished with diligence to ensure you’re orchestrating jobs in a manner that does not conflict. --- ### dbt v2 package upgrade guide Learn how to upgrade your packages to be compatible with dbt v2. [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Thank you for being part of the [dbt's package hub community](https://hub.getdbt.com/) and maintaining [packages](https://docs.getdbt.com/docs/build/packages.md)! Your work makes dbt’s ecosystem possible and helps thousands of teams reuse trusted models and macros to build faster, more reliable analytics. This guide helps you upgrade your dbt packages to be [dbt v2](https://docs.getdbt.com/docs/introduction.md)-compatible. A dbt v2-compatible package: * Supports [dbt v2](https://docs.getdbt.com/docs/introduction.md) version `2.0.0` * Uses the [`require-dbt-version` config](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) to signal compatibility in the dbt package hub * Aligns with the latest JSON schema introduced in dbt v1.10.0 In this guide, we'll go over: * Updating your package to be compatible with dbt v2 * Testing your package with dbt v2 * Updating the `require-dbt-version` config to include `2.0.0` * Updating your README to note that the package is compatible with dbt v2 ##### Who is this for? This guide is for any dbt package maintainer, like [`dbt-utils`](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/), that's looking to upgrade their package to be compatible with dbt v2. Updating your package ensures users have the latest version of your package, your package stays trusted on dbt package hub, and users benefit from the latest features and bug fixes. A user stores their package in a `packages.yml` or `dependencies.yml` file. If a package excludes `2.0.0`, dbt v2 warns today and errors in a future release, matching dbt v1 behavior. This guide assumes you're using the command line and Git to make changes in your package repository. If you're interested in creating a new package from scratch, we recommend using the [dbt package guide](https://docs.getdbt.com/guides/building-packages.md) to get started. #### Prerequisites Before you begin, make sure you meet the following: * dbt package maintainer — You maintain a package on [dbt's package hub](https://hub.getdbt.com/) or are interested in [creating one](https://docs.getdbt.com/guides/building-packages.md?step=1). * `dbt-autofix` installed — [Install `dbt-autofix`](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation) to automatically update the package's YAML files to align with the latest dbt updates and best practices. We recommend [using/installing uv/uvx](https://docs.astral.sh/uv/getting-started/installation/) to run the tool. * Run the command `uvx dbt-autofix` for the latest version of the tool. For more installation options, see the [official `dbt-autofix` doc](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation). * Repository access — You’ll need permission to create a branch and release updates/a new version of your package. You’ll need to tag a new version of your package once it’s dbt v2-compatible. * A dbt v2 installation or test environment — You can use dbt v2 locally (using the `dbtf` binary) or in your CI pipeline to validate compatibility. * CLI and Git usage — You’re comfortable using the command line and Git to update the repository. #### Upgrade the package This section covers how to upgrade your package to be compatible with dbt v2 by: * [Using `dbt-autofix` to automatically update your YAML files](https://docs.getdbt.com/guides/dbt-package-compat.md?step=) * [Testing your package with dbt v2](https://docs.getdbt.com/guides/dbt-package-compat.md?step=5) * [Updating your `require-dbt-version` config](https://docs.getdbt.com/guides/dbt-package-compat.md?step=6) * [Publishing a new release of your package](https://docs.getdbt.com/guides/dbt-package-compat.md?step=7) If you're ready to get started, let's begin! #### Run dbt-autofix 1. Before you begin, make sure you have `dbt-autofix` installed. If you don't have it installed, run the command `uvx dbt-autofix`. For more installation options, see the [official `dbt-autofix` doc](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation). 2. In your dbt package repository, create a branch to work in. For example: ```bash git checkout -b fusion-compat ``` 3. Run `dbt-autofix deprecations` in your package directory so it automatically updates your package code and rewrites YAML to conform to the latest JSON schema: ```bash dbt-autofix deprecations ``` #### Test package with dbt v2 Now that you've run `dbt-autofix`, let's test your package with dbt v2 to ensure it's compatible before [updating](https://docs.getdbt.com/guides/dbt-package-compat?step=6) your `require-dbt-version` config. Refer to the [dbt v2 limitations documentation](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations) for more information on what to look out for. You can test your package two ways: * [Running your integration tests with dbt v2](#running-your-integration-tests-with-v2) — Use if your package has [integration tests](https://docs.getdbt.com/guides/building-packages?step=4) using an `integration_tests/` folder. * [Manually validating your package](#manually-validating-your-package) — Use if your package doesn't have [integration tests](https://docs.getdbt.com/guides/building-packages?step=4). Consider creating one to help validate your package. ###### Running your integration tests with v2 If your package includes an `integration_tests/` folder ([like `dbt-utils`](https://github.com/dbt-labs/dbt-utils/tree/main/integration_tests)), follow these steps: 1. Navigate to the folder (`cd integration_tests`) to run your tests. If you don't have an `integration_tests/` folder, you can either [create one](https://docs.getdbt.com/guides/building-packages?step=4) or navigate to the folder that contains your tests. 2. Then, run your tests with dbt v2 by running the following `dbtf build` command (or whatever dbt v2 executable is available in your environment). 3. If there are no errors, your package likely supports dbt v2 and you're ready to [update your `require-dbt-version`](https://docs.getdbt.com//guides/dbt-package-compat?step=5#update-your-require-dbt-version). If there are errors, you'll need to fix them first before updating your `require-dbt-version`. ###### Manually validating your package If your package doesn't have integration tests, follow these steps: 1. Create a small, dbt v2-compatible dbt project that installs your package and has a `packages.yml` or `dependencies.yml` file. 2. Run it with dbt v2 using the `dbtf run` command. 3. Confirm that models build successfully and that there are no warnings. If there are errors/warnings, you'll need to fix them first. If you still have issues, reach out to the [#package-ecosystem channel](https://getdbt.slack.com/archives/CU4MRJ7QB) on Slack for help. #### Update `require-dbt-version` Only update the [`require-dbt-version` config](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) after testing and confirming that your package works with dbt v2. 1. Update the `require-dbt-version` in your `dbt_project.yml` to include `2.0.0`. We recommend using a range to ensure stability across releases: ```yaml require-dbt-version: [">=1.10.0,<3.0.0"] ``` This signals that your package supports both dbt v1 and dbt v2. dbt Labs uses this release metadata to mark your package with a dbt v2-compatible badge in the [dbt package hub](https://hub.getdbt.com/). Packages without this metadata don't display the dbt v2-compatible badge. 2. Commit and push your changes to your repository. #### Publish a new release 1. After committing and pushing your changes, publish a new release of your package by merging your branch into main (or whatever branch you're using for your package). 2. Update your `README` to note that the package is dbt v2-compatible. 3. (Optional) Announce it in [#package-ecosystem on dbt Slack](https://getdbt.slack.com/archives/CU4MRJ7QB) if you’d like. CI dbt v2 testing When possible, add a step to your CI pipeline that runs `dbtf build` or equivalent to ensure ongoing dbt v2 compatibility. Your package is now dbt v2-compatible and the dbt package hub reflects these changes. To summarize, you've now: * Created a v2 compatible branch * Run `dbt-autofix` deprecations * Reviewed, committed, and tested changes * Updated `require-dbt-version: [">=1.10.0,<3.0.0"]` to include `2.0.0` * Published a new release * Announced the update (optional) * Celebrate your new dbt v2-compatible badge 🎉 #### Final thoughts Now that you've upgraded your package to be dbt v2-compatible, users can use your package with dbt v2! 🎉 By upgrading now, you’re ensuring a smoother experience for users, paving the way for the next generation of dbt projects, and helping dbt v2 reach full stability. If you have questions or run into issues: * Join the conversation in the [#package-ecosystem channel](https://getdbt.slack.com/archives/CU4MRJ7QB) on Slack. * Open an issue in the [dbt-autofix repository](https://github.com/dbt-labs/dbt-autofix/issues) on GitHub. Lastly, thank you for your help in making the dbt ecosystem stronger — one package at a time 💜. #### Frequently asked questions The following are some frequently asked questions about upgrading your package to be dbt v2-compatible.  Why do we need to update our package? dbt v2 and dbt v1.10+ use the same new authoring layer. Ensuring your package supports `2.0.0` in your `require-dbt-version` config ensures your package is compatible with both. Updating your package ensures users have the latest version of your package, your package stays trusted on dbt package hub, and users benefit from the latest features and bug fixes. dbt v2-compatible packages display a badge in the dbt package hub. If a package excludes `2.0.0`, dbt v2 will warn today and error in a future release, matching dbt v1 behavior.  How do I test v2 in CI? Add a separate job that installs dbt v2 (`dbtf`) and runs `dbtf build`. See this [PR](https://github.com/godatadriven/dbt-date/pull/31) for a working example. You want to do this to ensure any changes to your package remain compatible with dbt v2.  How will users know my package is v2-compatible? Users can identify your package as dbt v2-compatible by checking for 2.0.0 or higher in the `require-dbt-version` range config. dbt v2-compatible packages also display a badge in the dbt package hub. This is automatically determined based on your package’s metadata and version requirements. --- ### Debug errors [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### General process of debugging Learning how to debug is a skill, and one that will make you great at your role! 1. Read the error message — when writing the code behind dbt, we try our best to make error messages as useful as we can. The error message dbt produces will normally contain the type of error (more on these error types below), and the file where the error occurred. 2. Inspect the file that was known to cause the issue, and see if there's an immediate fix. 3. Isolate the problem — for example, by running one model a time, or by undoing the code that broke things. 4. Get comfortable with compiled files and the logs. * The `target/compiled` directory contains `select` statements that you can run in any query editor. * The `target/run` directory contains the SQL dbt executes to build your models. * The `logs/dbt.log` file contains all the queries that dbt runs, and additional logging. Recent errors will be at the bottom of the file. * **dbt users**: Use the above, or the `Details` tab in the command output. * **dbt v1 users**: Note that your code editor *may* be hiding these files from the tree view [VSCode help](https://stackoverflow.com/questions/42891463/how-can-i-show-ignored-files-in-visual-studio-code)). 5. If you are really stuck, try [asking for help](https://docs.getdbt.com/community/resources/getting-help.md). Before doing so, take the time to write your question well so that others can diagnose the problem quickly. #### Types of errors Below, we've listed some of common errors. It's useful to understand what dbt is doing behind the scenes when you execute a command like `dbt run`. | Step | Description | Error type | | ---------------- | --------------------------------------------------------------------------------- | ------------------- | | Initialize | Check that this a dbt project, and that dbt can connect to the warehouse | `Runtime Error` | | Parsing | Check that the Jinja snippets in `.sql` files valid, and that `.yml` files valid. | `Compilation Error` | | Graph validation | Compile the dependencies into a graph. Check that it's acyclic. | `Dependency Error` | | SQL execution | Run the models | `Database Error` | Let's dive into some of these errors and how to debug 👇. Note: not all errors are covered here! #### Runtime Errors *Note: If you're using the Studio IDE to work on your project, you're unlikely to encounter these errors.* ##### Not a dbt project ```text Running with dbt=1.7.1 Encountered an error: Runtime Error fatal: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file ``` Debugging * Use `pwd` to check that you're in the right directory. If not, `cd` your way there! * Check that you have a file named `dbt_project.yml` in the root directory of your project. You can use `ls` to list files in the directory, or also open the directory in a code editor and see files in the "tree view". ##### Could not find profile ```text Running with dbt=1.7.1 Encountered an error: Runtime Error Could not run dbt Could not find profile named 'jaffle_shops' ``` Debugging * Check the `profile:` key in your `dbt_project.yml`. For example, this project uses the `jaffle_shops` (note plural) profile: dbt\_project.yml ```yml profile: jaffle_shops # note the plural ``` * Check the profiles you have in your `profiles.yml` file. For example, this profile is named `jaffle_shop` (note singular). profiles.yml ```yaml jaffle_shop: # this does not match the profile: key target: dev outputs: dev: type: postgres schema: dbt_alice ... # other connection details ``` * Update these so that they match. * If you can't find your `profiles.yml` file, run `dbt debug --config-dir` for help: ```text $ dbt debug --config-dir Running with dbt=1.7.1 To view your profiles.yml file, run: open /Users/alice/.dbt ``` * Then execute `open /Users/alice/.dbt` (adjusting accordingly), and check that you have a `profiles.yml` file. If you do not have one, set one up using [these docs](https://docs.getdbt.com/docs/local/profiles.yml.md) ##### Failed to connect ```text Encountered an error: Runtime Error Database error while listing schemas in database "analytics" Database Error 250001 (08001): Failed to connect to DB: your_db.snowflakecomputing.com:443. Incorrect username or password was specified. ``` Debugging * Open your `profiles.yml` file (if you're unsure where this is, run `dbt debug --config-dir`) * Confirm that your credentials are correct — you may need to work with a DBA to confirm this. * After updating the credentials, run `dbt debug` to check you can connect ```text $ dbt debug Running with dbt=1.7.1 Using profiles.yml file at /Users/alice/.dbt/profiles.yml Using dbt_project.yml file at /Users/alice/jaffle-shop-dbt/dbt_project.yml Configuration: profiles.yml file [OK found and valid] dbt_project.yml file [OK found and valid] Required dependencies: - git [OK found] Connection: ... Connection test: OK connection ok ``` ##### Invalid `dbt_project.yml` file ```text Encountered an error while reading the project: ERROR: Runtime Error at path []: Additional properties are not allowed ('hello' was unexpected) Error encountered in /Users/alice/jaffle-shop-dbt/dbt_project.yml Encountered an error: Runtime Error Could not run dbt ``` Debugging * Open your `dbt_project.yml` file. * Find the offending key (e.g. `hello`, as per "'hello' was unexpected") dbt\_project.yml ```yml name: jaffle_shop hello: world # this is not allowed ``` * Use the reference section for [`dbt_project.yml` files](https://docs.getdbt.com/reference/dbt_project.yml.md) to correct this issue. * If you're using a key that is valid according to the documentation, check that you're using the latest version of dbt with `dbt --version`. #### Compilation Errors *Note: if you're using the Studio IDE to work on your dbt project, this error often shows as a red bar in your command prompt as you work on your dbt project. For dbt v1 users, these won't get picked up until you run `dbt run` or `dbt compile`.* ##### Invalid `ref` function ```text $ dbt run -s customers Running with dbt=1.1.0 Encountered an error: Compilation Error in model customers (models/customers.sql) Model 'model.jaffle_shop.customers' (models/customers.sql) depends on a node named 'stg_customer' which was not found ``` Debugging * Open the `models/customers.sql` file. * `cmd + f` (or equivalent) for `stg_customer`. There must be a file named `stg_customer.sql` for this to work. * Replace this reference with a reference to another model (i.e. the filename for another model), in this case `stg_customers`. OR rename your model to `stg_customer` ##### Invalid Jinja ```text $ dbt run Running with dbt=1.7.1 Compilation Error in macro (macros/cents_to_dollars.sql) Reached EOF without finding a close tag for macro (searched from line 1) ``` Debugging Here, we rely on the Jinja library to pass back an error, and then just pass it on to you. This particular example is for a forgotten `{% endmacro %}` tag, but you can also get errors like this for: * Forgetting a closing `}` * Closing a `for` loop before closing an `if` statement To fix this: * Navigate to the offending file (e.g. `macros/cents_to_dollars.sql`) as listed in the error message * Use the error message to find your mistake To prevent this: * *(dbt v1 only)* Use snippets to auto-complete pieces of Jinja ([atom-dbt package](https://github.com/dbt-labs/atom-dbt)) ##### Invalid YAML dbt wasn't able to turn your YAML into a valid dictionary. ```text $ dbt run Running with dbt=1.7.1 Encountered an error: Compilation Error Error reading jaffle_shop: schema.yml - Runtime Error Syntax error near line 5 ------------------------------ 2 | 3 | models: 4 | - name: customers 5 | columns: 6 | - name: customer_id 7 | data_tests: 8 | - unique Raw Error: ------------------------------ mapping values are not allowed in this context in "<unicode string>", line 5, column 12 ``` Debugging Usually, it's to do with indentation — here's the offending YAML that caused this error: ```yaml models: - name: customers columns: # this is indented too far! - name: customer_id data_tests: - unique - not_null ``` To fix this: * Open the offending file (e.g. `schema.yml`) * Check the line in the error message (e.g. `line 5`) * Find the mistake and fix it To prevent this: * (dbt v1 users) Turn on indentation guides in your code editor to help you inspect your files * Use a YAML validator ([example](http://www.yamllint.com/)) to debug any issues ##### Incorrect YAML spec Slightly different error — the YAML structure is right (i.e. the YAML parser can turn this into a python dictionary), *but* there's a key that dbt doesn't recognize. ```text $ dbt run Running with dbt=1.7.1 Encountered an error: Compilation Error Invalid models config given in models/schema.yml @ models: {'name': 'customers', 'hello': 'world', 'columns': [{'name': 'customer_id', 'tests': ['unique', 'not_null']}], 'original_file_path': 'models/schema.yml', 'yaml_key': 'models', 'package_name': 'jaffle_shop'} - at path []: Additional properties are not allowed ('hello' was unexpected) ``` Debugging * Open the file (e.g. `models/schema.yml`) as per the error message * Search for the offending key (e.g. `hello`, as per "**'hello'** was unexpected") * Fix it. Use the [model properties](https://docs.getdbt.com/reference/model-properties.md) docs to find valid keys * If you are using a valid key, check that you're using the latest version of dbt with `dbt --version` #### Dependency Errors ```text $ dbt run Running with dbt=1.7.1-rc Encountered an error: Found a cycle: model.jaffle_shop.customers --> model.jaffle_shop.stg_customers --> model.jaffle_shop.customers ``` Your dbt DAG is not acyclic, and needs to be fixed! * Update the `ref` functions to break the cycle. * If you need to reference the current model, use the [`{{ this }}` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md) instead. #### Database Errors The thorniest errors of all! These errors come from your data warehouse, and dbt passes the message on. You may need to use your warehouse docs (i.e. the Snowflake docs, or BigQuery docs) to debug these. ```text $ dbt run ... Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) 001003 (42000): SQL compilation error: syntax error line 14 at position 4 unexpected 'from'. compiled SQL at target/run/jaffle_shop/models/customers.sql ``` 90% of the time, there's a mistake in the SQL of your model. To fix this: 1. Open the offending file: * **dbt:** Open the model (in this case `models/customers.sql` as per the error message) * **dbt v1:** Open the model as above. Also open the compiled SQL (in this case `target/run/jaffle_shop/models/customers.sql` as per the error message) — it can be useful to show these side-by-side in your code editor. 2. Try to re-execute the SQL to isolate the error: * **dbt:** Use the `Preview` button from the model file * **dbt v1:** Copy and paste the compiled query into a query runner (e.g. the Snowflake UI, or a desktop app like DataGrip / TablePlus) and execute it 3. Fix the mistake. 4. Rerun the failed model. In some cases, these errors might occur as a result of queries that dbt runs "behind-the-scenes". These include: * Introspective queries to list objects in your database * Queries to `create` schemas * `pre-hooks`s, `post-hooks`, `on-run-end` hooks and `on-run-start` hooks * For incremental models, and snapshots: merge, update and insert statements In these cases, you should check out the logs — this contains *all* the queries dbt has run. * **dbt**: Use the `Details` in the command output to see logs, or check the `logs/dbt.log` file * **dbt v1**: Open the `logs/dbt.log` file. Isolating errors in the logs If you're hitting a strange `Database Error`, it can be a good idea to clean out your logs by opening the file, and deleting the contents. Then, re-execute `dbt run` for *just* the problematic model. The logs will *just* have the output you're looking for. #### Common pitfalls ##### `Preview` vs. `dbt run` *(Studio IDE users only)* There's two interfaces that look similar: * The `Preview` button executes whatever SQL statement is in the active tab. It is the equivalent of grabbing the compiled `select` statement from the `target/compiled` directory and running it in a query editor to see the results. * The `dbt run` command builds relations in your database Using the `Preview` button is useful when developing models and you want to visually inspect the results of a query. However, you'll need to make sure you have executed `dbt run` for any upstream models — otherwise dbt will try to select `from` tables and views that haven't been built. ##### Forgetting to save files before running We've all been there. dbt uses the last-saved version of a file when you execute a command. In most code editors, and in the Studio IDE, a dot next to a filename indicates that a file has unsaved changes. Make sure you hit `cmd + s` (or equivalent) before running any dbt commands — over time it becomes muscle memory. ##### Editing compiled files *(More likely for dbt v1 users)* If you just opened a SQL file in the `target/` directory to help debug an issue, it's not uncommon to accidentally edit that file! To avoid this, try changing your code editor settings to grey out any files in the `target/` directory — the visual cue will help avoid the issue. #### FAQs Here are some useful FAQs to help you debug your dbt project: * How to generate HAR files HTTP Archive (HAR) files are used to gather data from users’ browser, which dbt Support uses to troubleshoot network or resource issues. This information includes detailed timing information about the requests made between the browser and the server. The following sections describe how to generate HAR files using common browsers such as [Google Chrome](#google-chrome), [Mozilla Firefox](#mozilla-firefox), [Apple Safari](#apple-safari), and [Microsoft Edge](#microsoft-edge). info Remove or hide any confidential or personally identifying information before you send the HAR file to dbt Labs. You can edit the file using a text editor. ##### Google Chrome 1. Open Google Chrome. 2. Click on **View** --> **Developer Tools**. 3. Select the **Network** tab. 4. Ensure that Google Chrome is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Select **Preserve Log**. 6. Clear any existing logs by clicking **Clear network log** (🚫). 7. Go to the page where the issue occurred and reproduce the issue. 8. Click **Export HAR** (the down arrow icon) to export the file as HAR. The icon is located on the same row as the **Clear network log** button. 9. Save the HAR file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Mozilla Firefox 1. Open Firefox. 2. Click the application menu and then **More tools** --> **Web Developer Tools**. 3. In the developer tools docked tab, select **Network**. 4. Go to the page where the issue occurred and reproduce the issue. The page automatically starts recording as you navigate. 5. When you're finished, click **Pause/Resume recording network log**. 6. Right-click anywhere in the **File** column and select **Save All as HAR**. 7. Save the HAR file. 8. Upload the HAR file to the dbt Support ticket thread. ##### Apple Safari 1. Open Safari. 2. In case the **Develop** menu doesn't appear in the menu bar, go to **Safari** and then **Settings**. 3. Click **Advanced**. 4. Select the **Show features for web developers** checkbox. 5. From the **Develop** menu, select **Show Web Inspector**. 6. Click the **Network tab**. 7. Go to the page where the issue occurred and reproduce the issue. 8. When you're finished, click **Export**. 9. Save the file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Microsoft Edge 1. Open Microsoft Edge. 2. Click the **Settings and more** menu (...) to the right of the toolbar and then select **More tools** --> **Developer tools**. 3. Click **Network**. 4. Ensure that Microsoft Edge is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Go to the page where the issue occurred and reproduce the issue. 6. When you're finished, click **Stop recording network log**. 7. Click **Export HAR** (the down arrow icon) or press **Ctrl + S** to export the file as HAR. 8. Save the HAR file. 9. Upload the HAR file to the dbt Support ticket thread. ##### Additional resources Check out the [How to generate a HAR file in Chrome](https://www.loom.com/share/cabdb7be338243f188eb619b4d1d79ca) video for a visual guide on how to generate HAR files in Chrome. * Reconnecting to Snowflake OAuth after authentication expires When you connect Snowflake to dbt platform using [OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md), dbt stores a refresh token. This allows your user credentials to be reused in tools like the Studio IDE and the dbt Semantic Layer without needing to re-authenticate each time. If you see an `authentication has expired` error when you try to run queries, you must renew your connection between Snowflake and the dbt platform. To resolve the issue, complete the following steps: 1. Go to **Your profile**, accessible from the navigation menu. 2. Navigate to **Credentials** and then choose the project where you're experiencing the issue. 3. Under **User credentials**, click the **Reconnect Snowflake Account** button. This will guide you through re-authenticating using your SSO workflow. Your Snowflake administrator can [configure the refresh token validity period](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md#create-a-security-integration), up to the maximum 90 days. If you've tried these step and are still getting this error, please contact the Support team at <support@getdbt.com> for further assistance. * Receiving a 'Could not parse dbt\_project.yml' error in dbt job The error message `Could not parse dbt_project.yml: while scanning for...` in your dbt job run or development usually occurs for several reasons: * There's a parsing failure in a YAML file (such as a tab indentation or Unicode characters). * Your `dbt_project.yml` file has missing fields or incorrect formatting. * Your `dbt_project.yml` file doesn't exist in your dbt project repository. To resolve this issue, consider the following: * Use an online YAML parser or validator to check for any parsing errors in your YAML file. Some known parsing errors include missing fields, incorrect formatting, or tab indentation. * Or ensure your `dbt_project.yml` file exists. Once you've identified the issue, you can fix the error and rerun your dbt job. * How can I fix my .gitignore file? A gitignore file specifies which files Git should intentionally ignore. You can identify these files in your project by their italics formatting. If you can't revert changes, check out a branch, or click commit — this is usually do to your project missing a [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file OR your gitignore file doesn't contain the necessary content inside the folder. To fix this, complete the following steps: 1. In the Studio IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 2. Save your changes but *don't commit* 3. Restart the Studio IDE by clicking on the three dots next to the **Studio IDE Status button** on the lower right of the Studio IDE. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 4. Select **Restart Studio IDE**. 5. Go back to the **File explorer** in the IDE and delete the following files or folders if you have them: * `target`, `dbt_modules`, `dbt_packages`, `logs` 6. **Save** and then **Commit and sync** your changes. 7. Restart the Studio IDE again. 8. Create a pull request (PR) under the **Version Control** menu to integrate your new changes. 9. Merge the PR on your git provider page. 10. Switch to your main branch and click on **Pull from remote** to pull in all the changes you made to your main branch. You can verify the changes by making sure the files/folders in the .gitignore file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics). For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. * I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job If you're receiving a `This run exceeded your account's run memory limits` error in your failed job, it means that the job exceeded the [memory limits](https://docs.getdbt.com/docs/deploy/job-scheduler.md#job-memory) set for your account. All dbt accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices. ##### Common reasons Some common reasons for higher memory usage are: * dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient. * dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries. ##### Resolution There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. Video example As an additional resource, check out [this example video](https://www.youtube.com/watch?v=sTqzNaFXiZ8), which demonstrates how to refactor the sample code by reducing the number of rows returned. If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! ##### Additional resources * [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) * Why am I receiving a Runtime Error in my packages? If you're receiving the runtime error below in your packages.yml folder, it may be due to an old version of your dbt\_utils package that isn't compatible with your current dbt version. ```shell Running with dbt=xxx Runtime Error Failed to read package: Runtime Error Invalid config version: 1, expected 2 Error encountered in dbt_utils/dbt_project.yml ``` Try updating the old version of the dbt\_utils package in your packages.yml to the latest version found in the [dbt hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/): ```shell packages: - package: dbt-labs/dbt_utils version: xxx ``` If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! * \[Error] Could not find my\_project package If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain *any* macros, dbt will raise an error like: ```shell Compilation Error In dispatch: Could not find package 'my_project' ``` This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! * What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. --- ### Debug schema names [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction If a model uses the [`schema` config](https://docs.getdbt.com/reference/resource-properties/schema.md) but builds under an unexpected schema, here are some steps for debugging the issue. The full explanation of custom schemas can be found [here](https://docs.getdbt.com/docs/build/custom-schemas.md). You can also follow along via this video: #### Search for a macro named `generate_schema_name` Do a file search to check if you have a macro named `generate_schema_name` in the `macros` directory of your project. ##### You do not have a macro named `generate_schema_name` in your project This means that you are using dbt's default implementation of the macro, as defined [here](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/get_custom_name/get_custom_schema.sql) ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` Note that this logic is designed so that two dbt users won't accidentally overwrite each other's work by writing to the same schema. ##### You have a `generate_schema_name` macro in a project that calls another macro If your `generate_schema_name` macro looks like so: ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {{ generate_schema_name_for_env(custom_schema_name, node) }} {%- endmacro %} ``` Your project is switching out the `generate_schema_name` macro for another macro, `generate_schema_name_for_env`. Similar to the above example, this is a macro which is defined in dbt's global project, [here](https://github.com/dbt-labs/dbt/blob/main/crates/dbt-loader/src/dbt_macro_assets/dbt-adapters/macros/get_custom_name/get_custom_schema.sql). ```sql {% macro generate_schema_name_for_env(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if target.name == 'prod' and custom_schema_name is not none -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }} {%- endif -%} {%- endmacro %} ``` ##### You have a `generate_schema_name` macro with custom logic If this is the case — it might be a great idea to reach out to the person who added this macro to your project, as they will have context here — you can use [GitHub's blame feature](https://docs.github.com/en/free-pro-team@latest/github/managing-files-in-a-repository/tracking-changes-in-a-file) to do this. In all cases take a moment to read through the Jinja to see if you can follow the logic. #### Confirm your `schema` config Check if you are using the [`schema` config](https://docs.getdbt.com/reference/resource-properties/schema.md) in your model, either via a `{{ config() }}` block, or from `dbt_project.yml`. In both cases, dbt passes this value as the `custom_schema_name` parameter of the `generate_schema_name` macro. #### Confirm your target values Most `generate_schema_name` macros incorporate logic from the [`target` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md), in particular `target.schema` and `target.name`. Use the docs [here](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) to help you find the values of each key in this dictionary. #### Put the two together Now, re-read through the logic of your `generate_schema_name` macro, and mentally plug in your `customer_schema_name` and `target` values. You should find that the schema dbt is constructing for your model matches the output of your `generate_schema_name` macro. Be careful. Snapshots do not follow this behavior if target\_schema is set. To have environment-aware snapshots in v1.9+ or dbt, remove the [target\_schema config](https://docs.getdbt.com/reference/resource-configs/target_schema.md) from your snapshots. If you still want a custom schema for your snapshots, use the [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) config instead. #### Prefixed schema names By default, dbt combines `target.schema` and `custom_schema_name` using the following pattern: `{target.schema}_{custom_schema_name}`. For example, when `target.schema` is `public` and a model sets `+schema: silver`, dbt builds the model in `public_silver`, not `silver`. This behavior is intentional. Including `target.schema` helps prevent developers and continuous integration (CI) jobs from building into the same schema and overwriting one another’s relations. If you want to use dedicated schema names such as `silver` and `gold` in production, use the environment-aware [`generate_schema_name_for_env` pattern](#you-have-a-generate_schema_name-macro-in-a-project-that-calls-another-macro) shown earlier. This pattern uses the custom schema name when `target.name` is `prod`, while retaining the target schema in development and CI environments. The macro looks like this: ```sql {% macro generate_schema_name_for_env(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if target.name == 'prod' and custom_schema_name is not none -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }} {%- endif -%} {%- endmacro %} ``` Verify actual relation locations with `dbt ls --output json` or by querying your warehouse catalog (`pg_views`, `information_schema.tables`, or equivalent). #### Adjust as necessary Now that you understand how a model's schema is being generated, you can adjust as necessary: * You can adjust the logic in your `generate_schema_name` macro (or add this macro to your project if you don't yet have one and adjust from there) * You can also adjust your `target` details (for example, changing the name of a target) If you change the logic in `generate_schema_name`, it's important that you consider whether two users will end up writing to the same schema when developing dbt models. This consideration is the reason why the default implementation of the macro concatenates your target schema and custom schema together — we promise we were trying to be helpful by implementing this behavior, but acknowledge that the resulting schema name is unintuitive. --- ### Get hands-on with Snowflake Horizon and Apache Iceberg [Back to guides](https://docs.getdbt.com/guides.md) 60 minutes [Menu ]() This guide shows you how to run dbt transformations across two engines (Snowflake and DuckDB) against a single, shared set of Apache Iceberg tables. You'll build your raw and staging layers as Snowflake-managed Iceberg tables, then use Snowflake Horizon's open Iceberg REST catalog to let DuckDB read and write those exact same tables directly. The result is a multi-engine workflow where each transformation runs on the engine that fits it best, your data is never copied or synced between systems, and your governance and lineage stay intact throughout. #### Access the same Iceberg tables from Snowflake and DuckDB By the end of this guide, your `fusion-jaffle-shop` project will transform data in two different engines against one shared set of tables: * **Snowflake:** Builds your raw and staging layers as Snowflake-managed Apache Iceberg tables, registered in Snowflake's built-in Horizon catalog. * **DuckDB:** Reads those exact same Iceberg tables through Horizon's open Iceberg REST catalog, builds a downstream model (`orders`), and writes the result *back* as an Iceberg table in the same catalog. The DuckDB step is the interesting part. Because Iceberg is an open table format and Horizon speaks the open Iceberg REST protocol, an external engine like DuckDB can operate directly on your governed Snowflake tables. Both solutions are reading and writing the same files, through the same catalog, without a separate copy of the data. Concretely, the transformation runs in a local DuckDB process (embedded in dbt v2); Horizon serves catalog metadata and vends short-lived storage credentials so DuckDB can access the underlying files, which live in your own S3 bucket the whole time. Snowflake warehouses are highly performant. Even a single-node XSMALL handles most production transformations well, and for the majority of your workloads, running them in Snowflake is still the right call. But there are times when it's useful to reach the same governed tables from somewhere else: a quick exploratory query from a laptop, a step that fits naturally into a pipeline already running on another engine, or a workload where you'd rather not spin up a warehouse at all. Horizon's Iceberg REST catalog is what makes that possible without duplicating data or losing governance. Why that's a big deal: * **One governed copy, many engines**: There's a single physical copy of the data, in open Parquet + Iceberg metadata, registered in Horizon. Snowflake, DuckDB, Spark, Trino, and others can all read and write it directly. No syncing, no ETL between systems. * **Governance stays put**: The tables are still first-class Snowflake objects. They show up in the catalog, honor grants, and can carry masking/row policies and lineage — even the ones DuckDB wrote. * **Flexibility to match engine to job**: Because Horizon exposes the same tables everywhere, you can route each transformation to whichever engine suits it best — including running a step locally in DuckDB — without duplicating data or re-architecting your pipeline. This is the foundation of cross-platform [dbt Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh): one project, one catalog, many engines. #### Notes about this guide Before you begin, there are a few considerations: * This is an advanced guide and assumes that you have fundamental knowledge of dbt, Snowflake, DuckDB, related tools, and how to install them. * While dbt handles SQL transformations with grace, some of the tools used in this guide are very specific about the syntax they accept. We highly recommend you remove code comments from examples in this guide before using them in a live environment. * The demo project used is a feature rich example of the existing Jaffle Shop project. This is to demonstrate some of the considerations you'll need to make in your own projects as you implement these workflows. To ensure a smooth outcome, we recommend you use the project link in this guide and not any other existing Jaffle Shop projects. * If you are using a Snowflake account that has not been configured for Python (for example, a brand new trial account), you may run into errors with the Python models in the dbt project. Deleting them for the duration of this guide will remove those errors (though there will be some non-blocking warnings). #### Prerequisites There are a number of prerequisites and personas you'll need to complete this guide. ##### Accounts and access To complete the required setup steps, you'll need: * A **Snowflake account** where you can act as `ACCOUNTADMIN`. This guide builds everything from scratch, so a brand-new account (a trial works) is exactly the assumed starting point. * An **AWS account** where you can create an S3 bucket and an IAM role + policy. * A local clone of the [**`fusion-jaffle-shop`** project](https://github.com/matthewshaver/jaffle-shop-multi-adapter), with the seed CSVs present in `seeds/`. * Comfort running SQL in Snowsight and basic commands in a terminal. ##### Tools to install and verify Install and verify these before you start: | Requirement | Why | Check | | ------------------------------------ | --------------------------------------------------------- | ----------------------------- | | **dbt v2** (v2, `preview.194`+) | `catalogs.yml` v2 + DuckDB attach support | `dbt --version` | | **DuckDB 1.5.4+** | Iceberg write-compat (`ATTACH`, credential vending) | `duckdb --version` | | **An AWS account** | Hosts the S3 bucket that stores the Iceberg data/metadata | — | | **A new Snowflake account** | Where Horizon + the managed Iceberg tables live | — | | **The `fusion-jaffle-shop` project** | Cloned locally, with the seed CSVs present in `seeds/` | `ls seeds/` shows `raw_*.csv` | ##### Placeholders used in this guide Throughout this guide, replace these placeholders with your own values: * `<ACCOUNT_IDENTIFIER>`: Your Snowflake account identifier in `ORG-ACCOUNT` form (for example, `ABCDEFG-HI12345`). Find it in Snowsight under your account menu → **Account** → **View account details**, or in the account URL `https://<ORG>-<ACCOUNT>.snowflakecomputing.com`. * `<AWS_ACCOUNT_ID>`: Your 12-digit AWS account number. * `<YOUR_USER>`: Your Snowflake login username. * `<REGION>`: The AWS region for your bucket. This guide uses `us-east-2`; use whatever is closest to your Snowflake account's region. This guide is for v2 only. The `catalogs.yml` mechanism it relies on is not available in the legacy Python `dbt-duckdb` adapter. #### The project This guide will use a heavily augmented copy of the traditional Jaffle Shop project. The project itself contains more information than required to simply setup an Iceberg workflow, so you can use it to run more trials and observe results. Clone the [`fusion-jaffle-shop` project](https://github.com/matthewshaver/jaffle-shop-multi-adapter) from GitHub: ```bash git clone https://github.com/matthewshaver/jaffle-shop-multi-adapter.git ``` Then navigate into the project directory: ```bash cd fusion-jaffle-shop ``` #### Part 1: Set up the Snowflake account Log into Snowsight (the Snowflake web UI) with your admin user. Open a new SQL worksheet and run each block below. ##### 1.1 Create the warehouse, database, and schema ```sql USE ROLE ACCOUNTADMIN; CREATE WAREHOUSE IF NOT EXISTS COMPUTE_WH WAREHOUSE_SIZE = 'XSMALL' AUTO_SUSPEND = 60 AUTO_RESUME = TRUE; CREATE DATABASE IF NOT EXISTS DBT_ICEBERG; CREATE SCHEMA IF NOT EXISTS DBT_ICEBERG.RAW; ``` ##### 1.2 Create the `TRANSFORMER` role and grant it everything it needs You'll use one role, `TRANSFORMER`, for everything. Both the Snowflake-side build and the DuckDB offload. Because `TRANSFORMER` will *own* every table it creates, it automatically has read/write on them and you won't need extra `SELECT` grants later. ```sql USE ROLE ACCOUNTADMIN; CREATE ROLE IF NOT EXISTS TRANSFORMER; -- Let TRANSFORMER use compute and the database objects GRANT USAGE ON WAREHOUSE COMPUTE_WH TO ROLE TRANSFORMER; GRANT USAGE ON DATABASE DBT_ICEBERG TO ROLE TRANSFORMER; GRANT USAGE ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; -- Let TRANSFORMER create the tables dbt will build (seeds + Iceberg models) GRANT CREATE TABLE ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; GRANT CREATE ICEBERG TABLE ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; GRANT CREATE VIEW ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; -- Give the role to yourself so you can use it, and make it your default GRANT ROLE TRANSFORMER TO USER <YOUR_USER>; ALTER USER <YOUR_USER> SET DEFAULT_ROLE = TRANSFORMER; ``` We'll grant `TRANSFORMER` access to the external volume in Part 3, after it exists. #### Part 2: Create the S3 bucket and IAM role in AWS Snowflake-managed Iceberg tables store their data and metadata files in object storage that you own. Snowflake reaches that bucket by assuming an IAM role you create. This is a two-way handshake, so the order matters. ##### 2.1 Create the S3 bucket In your AWS console 1. Navigate to **S3 → Create bucket**. 2. **Bucket name**: pick a globally unique name. A good convention is `iceberg-<AWS_ACCOUNT_ID>-<REGION>`, like `iceberg-1234567890-us-east-2`. 3. **Region**: choose `<REGION>` (for example, `us-east-2`). 4. Leave the rest as defaults and click **Create bucket**. 5. (Optional but tidy) open the bucket and create a folder/prefix named `jaffle-iceberg/` for all Iceberg files to live under. ##### 2.2 Create the IAM permission policy In your AWS console: 1. Navigate to **IAM → Policies → Create policy** and switch to the **JSON** tab. 2. Paste the following, replacing the bucket name in both `Resource` blocks: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject", "s3:GetObjectVersion", "s3:DeleteObject", "s3:DeleteObjectVersion"], "Resource": "arn:aws:s3:::iceberg-<AWS_ACCOUNT_ID>-<REGION>/jaffle-iceberg/*" }, { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": "arn:aws:s3:::iceberg-<AWS_ACCOUNT_ID>-<REGION>", "Condition": { "StringLike": { "s3:prefix": ["jaffle-iceberg/*"] } } } ] } ``` 3. Click **Next**, name it `snowflake-iceberg-policy`, and **Create policy**. ##### 2.3 Create the IAM role (with a placeholder trust) You can't fill in the real trust policy yet as it needs values Snowflake generates in Part 3. So create the role with a temporary trust, then you'll fix it later. In the AWS console: 1. Navigate to **IAM → Roles → Create role**. 2. **Trusted entity type**: choose **AWS account**. 3. Select **This account (`<AWS_ACCOUNT_ID>`)**. Leave both **"Require external ID"** and **"Require MFA"** **unchecked**. (This is a throwaway trust; you'll replace it in Part 3.) 4. Click **Next**. On the permissions page, search for and check **`snowflake-iceberg-policy`**. 5. Click **Next**, name the role **`snowflake-iceberg-role`**, and **Create role**. 6. Open the new role and copy its **ARN** (`arn:aws:iam::<AWS_ACCOUNT_ID>:role/snowflake-iceberg-role`) — you'll need it next. #### Part 3: Connect Snowflake to S3 with an external volume An external volume is Snowflake's secure link to the S3 bucket where your Iceberg data and metadata live. In this part, you'll create the volume, complete the trust handshake between Snowflake and your IAM role, and set the volume as your database default so every Iceberg table knows where to store its files. ##### 3.1 Create the external volume Back in Snowsight (as `ACCOUNTADMIN`), copy and paste the following (replace `<PLACEHOLDERS>` with your info): ```sql USE ROLE ACCOUNTADMIN; CREATE OR REPLACE EXTERNAL VOLUME ICEBERG_EXT_VOL STORAGE_LOCATIONS = (( NAME = 'iceberg-s3-<REGION>' STORAGE_PROVIDER = 'S3' STORAGE_BASE_URL = 's3://iceberg-<AWS_ACCOUNT_ID>-<REGION>/jaffle-iceberg/' STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<AWS_ACCOUNT_ID>:role/snowflake-iceberg-role' )) ALLOW_WRITES = TRUE; ``` ##### 3.2 Get Snowflake's identity and finish the IAM trust ```sql DESCRIBE EXTERNAL VOLUME ICEBERG_EXT_VOL; ``` In the output, find the `STORAGE_LOCATION_1` row and read two values out of its JSON: * `STORAGE_AWS_IAM_USER_ARN` (looks like `arn:aws:iam::123456789012:user/abc1-s`) * `STORAGE_AWS_EXTERNAL_ID` (looks like `ABC12345_SFCRole=...=`) Now, go back to the AWS console **IAM → Roles → `snowflake-iceberg-role` → Trust relationships → Edit trust policy**, delete what's there, and paste this (with your two values): ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "AWS": "<STORAGE_AWS_IAM_USER_ARN>" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "<STORAGE_AWS_EXTERNAL_ID>" } } }] } ``` Click **Update policy**. Copy the external ID *exactly*, including any trailing `=` and internal `/`. It's case-sensitive. ##### 3.3 Grant the volume and set it as the database default ```sql USE ROLE ACCOUNTADMIN; -- TRANSFORMER needs USAGE on the volume (required for credential vending) GRANT USAGE ON EXTERNAL VOLUME ICEBERG_EXT_VOL TO ROLE TRANSFORMER; -- Make this volume the default for Iceberg tables in the database. -- REQUIRED: when DuckDB creates a table through the REST catalog it can't -- name a volume, so Snowflake falls back to this default. ALTER DATABASE DBT_ICEBERG SET EXTERNAL_VOLUME = ICEBERG_EXT_VOL; ALTER DATABASE DBT_ICEBERG SET CATALOG = 'SNOWFLAKE'; ``` #### Part 4: Create a Programmatic Access Token for DuckDB DuckDB authenticates to Horizon's REST catalog over OAuth2, using a Programmatic Access Token (PAT) as its credential. To add the PAT in Snowsight: ```sql USE ROLE ACCOUNTADMIN; ALTER USER <YOUR_USER> ADD PROGRAMMATIC ACCESS TOKEN duckdb_pat ROLE_RESTRICTION = 'TRANSFORMER' DAYS_TO_EXPIRY = 90; ``` The result grid has two columns. Copy the long `token_secret` value (not `token_name`). You only see it once; save it somewhere safe for Part 6. **Verify the token works:** Before wiring it into dbt, in a terminal, run the OAuth2 exchange (this is exactly what DuckDB does under the hood). Note the **empty username** before the colon in `-u ":..."` — this is essential (see FAQ): ```bash curl -s "https://<ACCOUNT_IDENTIFIER>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \ -u ":<TOKEN_SECRET>" \ --data-urlencode 'grant_type=client_credentials' \ --data-urlencode 'scope=session:role:TRANSFORMER' ``` A JSON response containing `"access_token"` means you're good. If you see `Programmatic access token is invalid`, re-copy the token. **If you see an error mentioning a network policy, see the [FAQ](https://docs.getdbt.com/guides/iceberg.md?step=12#faq)**. Some Snowflake accounts require a network policy before a PAT can be created or used. #### Part 5: Load the seed data In this part, you'll load the dbt project's raw CSV seed data into Snowflake as regular tables. You'll run the seed against the prod target so you get the full historical dataset, rather than dev, which filters orders to the last year. ##### 5.1 Seed commands From the project root in your dbt CLI, install packages and seed the raw tables into Snowflake. (Make sure the seed CSVs are present in `seeds/` first.) ```bash dbt deps ``` ```bash dbt seed --target prod ``` ##### 5.2 First seed The project's `on-run-start` hook (`insert_freshness_heartbeat()`) queries a raw source table that doesn't exist until the seeds load, so if the *first* `dbt seed` fails with `Object 'DBT_ICEBERG.RAW.RAW_STORES' does not exist`. Before your first seed, comment out the `on-run-start` block in `dbt_project.yml`: ```yaml # on-run-start: # - "{{ insert_freshness_heartbeat() }}" ``` Uncomment it once the seeds exist. This loads the following into `DBT_ICEBERG.RAW` as regular Snowflake tables: * `raw_customers` * `raw_orders` * `raw_items` * `raw_products` * `raw_stores` * `raw_supplies` * `raw_tweets` We use the `prod` target — you'll define it in the next section. Why `prod` and not `dev`? This project's `stg_orders` model applies a `limit_in_dev` macro that filters orders to the last year **whenever the target is named `dev`**. The seed data is historical, so building on `dev` yields an **empty** `stg_orders` (and therefore empty `orders`). Building on any non-`dev` target skips that filter and gives you the full dataset. See the [FAQ](https://docs.getdbt.com/guides/iceberg.md?step=12#faq) for more information. #### Part 6: Configure the project ##### 6.1 `profiles.yml` Create/edit `~/.dbt/profiles.yml`. Define three targets: `dev` and `prod` (both Snowflake) and `duckdb` (the offload). Paste your PAT into `client_secret`. *You can, optionally, run `dbt init` to generate the `profiles.yml` in the proper location and configure the Snowflake profile.* If you already have a `~/.dbt/profiles.yml`, add the `jaffle_shop` profile below **before** running any commands. On an existing config, `dbt init` may not surface a `prod` target and later steps will fail to find it. ```yaml jaffle_shop: target: prod outputs: # Everyday Snowflake target. NOTE: stg_orders is limited to the last year here. dev: type: snowflake account: "<ACCOUNT_IDENTIFIER>" user: "<YOUR_USER>" authenticator: externalbrowser # or your preferred auth role: TRANSFORMER database: DBT_ICEBERG schema: RAW warehouse: COMPUTE_WH threads: 8 # Full-data Snowflake target used to build the Iceberg parents. prod: type: snowflake account: "<ACCOUNT_IDENTIFIER>" user: "<YOUR_USER>" authenticator: externalbrowser role: TRANSFORMER database: DBT_ICEBERG schema: RAW warehouse: COMPUTE_WH threads: 8 # DuckDB target reads/writes Iceberg via Horizon with ZERO Snowflake compute. duckdb: type: duckdb path: ':memory:' schema: RAW secrets: - type: iceberg name: horizon_secret client_id: "" # MUST be empty (see FAQ) client_secret: "<TOKEN_SECRET>" # your PAT from Part 4 oauth2_server_uri: "https://<ACCOUNT_IDENTIFIER>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" oauth2_scope: "session:role:TRANSFORMER" ``` The v2 project's `dbt_project.yml` sets `profile: default`. Either rename the profile key above to `default`, or set `profile: jaffle_shop`. Keep it consistent. If `externalbrowser` fails For example `390190 (08004) ... SAML Identity Provider account parameter`, swap it for password auth on the `dev` and `prod` targets and keep the secret out of the file with an environment variable: ```yaml # authenticator: externalbrowser password: "{{ env_var('SNOWFLAKE_PASSWORD') }}" ``` ##### 6.2 `catalogs.yml` Create `catalogs.yml` in the project root. This one entry describes the *same* Horizon catalog for both engines. ```yaml catalogs: - name: horizon_catalog type: horizon table_format: iceberg config: snowflake: external_volume: ICEBERG_EXT_VOL duckdb: warehouse: DBT_ICEBERG # Horizon REST "warehouse" = the Snowflake database name endpoint: "https://<ACCOUNT_IDENTIFIER>.snowflakecomputing.com/polaris/api/catalog" secret: horizon_secret # references the profiles.yml secret above default_schema: RAW access_delegation_mode: VENDED_CREDENTIALS # Horizon vends temp S3 creds; no AWS keys needed ``` ##### 6.3 `dbt_project.yml` Add the behavior flag that enables `catalogs.yml`, and remove the project-level `marts` grants (they don't work on DuckDB — see 6.4 and the FAQ). Add near the top: ```yaml flags: use_catalogs_v2: true ``` Then, under `models: jaffle_shop: marts:`, **delete** the `+grants` block: ```yaml # DELETE these two lines from the marts config: +grants: select: ["ACCOUNTADMIN"] ``` ##### 6.4 Model changes These edits make the three models materialize as Iceberg and behave on both engines. * **`models/staging/stg_orders.sql`**: Add the config header and cast the timestamp to microsecond precision (Iceberg rejects Snowflake's default nanosecond scale): ```sql {{ config(materialized='table', catalog_name='horizon_catalog', alias='STG_ORDERS') }} ``` ...and change the timestamp line to: ```sql cast({{ dbt.date_trunc('day','ordered_at') }} as timestamp_ntz(6)) as ordered_at ``` * **`models/marts/order_items.sql`**: add at the very top: ```sql {{ config(materialized='table', catalog_name='horizon_catalog', alias='ORDER_ITEMS') }} ``` * **`models/marts/orders.sql`**: replace its config header with: ```sql {{ config( materialized='table', catalog_name='horizon_catalog', alias='ORDERS', grants={} if target.name == 'duckdb' else {'select': ['ACCOUNTADMIN']}, persist_docs={'relation': false, 'columns': false} if target.name == 'duckdb' else {'relation': true, 'columns': true} ) }} ``` * **`models/marts/orders.yml`**: Turn off the enforced contract (a single contract can't hold both Snowflake and DuckDB type names): ```yaml contract: enforced: false ``` * **`macros/insert_freshness_heartbeat.sql`**: Exclude the `duckdb` target (the heartbeat writes to the raw Snowflake source, which isn't attached in a DuckDB session): ```jinja {% if target.name not in ('ci', 'dev', 'duckdb') %} ``` * **Why uppercase `alias`?** Snowflake stores unquoted identifiers in UPPERCASE, and Iceberg catalogs are case-sensitive. DuckDB quotes model names in lowercase, so without the uppercase alias it would look for `"stg_orders"` and miss `STG_ORDERS`. #### Part 7: Build it This is where it all comes together! Snowflake builds the raw and staging Iceberg parents, then DuckDB reads those same tables through Horizon to build and write back the final orders model. All of this against the one shared catalog. ##### 7.1 Snowflake pass: Build the Iceberg parents ```bash dbt run --target prod -s +orders --exclude orders ``` This builds `stg_orders` and `order_items` as Snowflake-managed Iceberg tables in `DBT_ICEBERG.RAW`, registered in Horizon. Because you run as `TRANSFORMER`, it owns those tables. ##### 7.2 DuckDB pass: Read and write through Horizon ```bash dbt run --target duckdb -s orders ``` DuckDB attaches Horizon, reads `STG_ORDERS` + `ORDER_ITEMS` (fetching temp S3 credentials from Horizon), builds `orders` locally, and commits it back as an Iceberg table through Horizon's REST catalog without involving a Snowflake warehouse. ##### 7.3 Verify ```bash dbt show --target duckdb --inline "select count(*) from {{ ref('orders') }}" ``` You should get a real row count (tens of thousands). You'll also see `ORDERS` listed as an **Iceberg table** in Snowsight under `DBT_ICEBERG.RAW` and written entirely by DuckDB. **Confirm the cross-engine write:** in Snowsight, open **Admin → Cost Management** (or query `snowflake.account_usage.warehouse_metering_history`). You'll see credits for the `prod` build in 7.1, and none for the DuckDB run in 7.2 — confirming that DuckDB read and wrote the Iceberg tables directly through Horizon's REST catalog, rather than through a Snowflake warehouse. #### Congratulations! You've just built a single set of Apache Iceberg tables and transformed them with two different engines! Snowflake for the raw and staging layers, DuckDB for the final orders model, and all against one shared Horizon catalog. Because Horizon speaks the open Iceberg REST protocol, DuckDB was able to read and write those governed Snowflake tables directly. No data copies, no lock-in, and governance and lineage that stay intact no matter which engine does the work. This is the foundation of cross-platform dbt Mesh: one project, one catalog, and the freedom to route every job to whichever engine fits it best. #### FAQ  OAuth token exchange returns \`invalid\_scope\` for every role, even \`PUBLIC\`. The PAT must be sent with an **empty `client_id`** — as the HTTP Basic *password* with no username (`curl -u ":<PAT>"`), which is what `client_id: ""` produces in the DuckDB secret. If you pass a real username as `client_id`, Snowflake tries to resolve it as a registered OAuth client and rejects the scope. Empty client\_id is the fix.  The exchange returns \`Programmatic access token is invalid\`. The token value is wrong — you likely copied `token_name` instead of the long `token_secret`, or truncated it. Re-create the PAT and copy the `token_secret` carefully.  PAT creation or use fails complaining about a network policy. Some Snowflake accounts require a network policy to be in effect before a PAT can be created or used. If you hit this, create a network policy scoped to your egress IP range and attach it to your user, then retry. (Intentionally out of scope for this guide's main flow.)  DuckDB gets \`401 Unauthorized\` from the catalog. Auth problem. Check: the PAT is valid (test with the curl in Part 4); `client_id` is `""`; `oauth2_scope` is `session:role:TRANSFORMER` and your user actually holds `TRANSFORMER`; and the account in `oauth2_server_uri`/`endpoint` matches the account the PAT was minted in.  DuckDB gets \`404 Not Found\` on \`/v1/config\`. The `warehouse` value in `catalogs.yml` is wrong. For Horizon it's the **Snowflake database name** (`DBT_ICEBERG`), not the `catalogs.yml` entry name.  The catalog returns the namespace but lists no tables, or DuckDB can't find a table it should see. Two common causes: (1) a role lacks visibility — but if you built as `TRANSFORMER` (this guide) it owns the tables and sees them. If you built as a *different* role, note that `GRANT SELECT ON ALL TABLES` does **not** cover Iceberg tables; you must `GRANT SELECT ON ALL ICEBERG TABLES IN SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER` (and `FUTURE ICEBERG TABLES`). (2) Case mismatch — see the uppercase-alias note below.  \`Catalog Error: Table with name raw.stg\_orders does not exist\`. Case sensitivity. Snowflake stores `STG_ORDERS`; DuckDB quotes `"stg_orders"`. Give the Iceberg models an uppercase `alias` (`alias='STG_ORDERS'`). For a whole project you'd standardize identifier casing rather than aliasing each model.  \`SQL compilation error: Invalid time type scale ... TIMESTAMP\_NTZ(9)\`. Iceberg only supports microsecond timestamps. Cast to `timestamp_ntz(6)`.  \`403 Forbidden\` reading a \`.metadata.json\` from S3, with \`region ''\` / \`No credentials provided\`. DuckDB needs storage credentials to read the data files. Set `access_delegation_mode: VENDED_CREDENTIALS` in the `duckdb` catalog block so Horizon vends temporary, region-scoped S3 credentials — no AWS keys required. If this occurs intermittently right after rebuilding a table, just re-run; a fresh process re-fetches current metadata and credentials.  \`403 Authorization failed\` on \`POST .../namespaces/RAW/tables\` (create). The database has no default external volume, so a REST `createTable` can't resolve where to write. Run `ALTER DATABASE DBT_ICEBERG SET EXTERNAL_VOLUME = ICEBERG_EXT_VOL;` (and `SET CATALOG = 'SNOWFLAKE';`). Also confirm `TRANSFORMER` has `CREATE ICEBERG TABLE` on the schema and `USAGE` on the volume.  \`500 ... Object 'ORDERS' already exists as TABLE\`. A regular (non-Iceberg) table of the same name already occupies that namespace (e.g. from an earlier full `dbt build`). Drop it: `DROP TABLE IF EXISTS DBT_ICEBERG.RAW.ORDERS;` and re-run so DuckDB can create the Iceberg version.  \`unknown method: map has no method named warn\_once\` during grants. The dbt-duckdb `apply_grants` macro hits a v2 incompatibility. Grants are meaningless on DuckDB anyway — make `grant_config` empty on the DuckDB target (`grants={} if target.name == 'duckdb' else {...}`) and remove the project-level `marts` `+grants` so nothing gets merged back in.  \`Not implemented Error: Only ALTER TABLE is supported for Iceberg\`. `persist_docs` emits `COMMENT` statements DuckDB's Iceberg writer doesn't support. Disable it on the DuckDB target (`persist_docs={'relation': false, 'columns': false} if target.name == 'duckdb' else {...}`).  \`Type with name number does not exist\` (DuckDB suggests \`numeric\`). An **enforced model contract** is generating a typed `CREATE TABLE` with Snowflake type names (`number`, `timestamp_ntz`) that DuckDB doesn't recognize. A single contract can't satisfy two engines, so set `contract: enforced: false` on models you build on DuckDB.  \`orders\` builds successfully but has 0 rows. `stg_orders` came out empty because you built it on the `dev` target, where `limit_in_dev` restricts orders to the last year and the seed data is older. Build the parents on `prod` (or any non-`dev` target).  \`TRANSFORMER\` can't run \`count(\*)\` / \`must specify a warehouse\`. Metadata commands (`show`, `describe`) don't need a warehouse, but queries do. Ensure `GRANT USAGE ON WAREHOUSE COMPUTE_WH TO ROLE TRANSFORMER`. To read row counts without any warehouse, read `summary.total-records` from the Iceberg table metadata via the REST `loadTable` endpoint.  How do I extend this beyond \`orders\`? Move the DuckDB-specific overrides (empty `grants`, disabled `persist_docs`) to a **target-aware `marts` config** in `dbt_project.yml` instead of per-model, standardize identifier casing project-wide, and add `catalog_name`/`alias` to each model you want materialized as Iceberg. Everything Snowflake writes, DuckDB can read, and vice-versa. --- ### Get started with Continuous Integration tests [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction By validating your code *before* it goes into production, you don't need to spend your afternoon fielding messages from people whose reports are suddenly broken. A solid CI setup is critical to preventing avoidable downtime and broken trust. dbt uses **sensible defaults** to get you up and running in a performant and cost-effective way in minimal time. After that, there's time to get fancy, but let's walk before we run. In this guide, we're going to add a **CI environment**, where proposed changes can be validated in the context of the entire project without impacting production systems. We will use a single set of deployment credentials (like the Prod environment), but models are built in a separate location to avoid impacting others (like the Dev environment). Your git flow will look like this: [![git flow diagram](/img/best-practices/environment-setup/one-branch-git.png?v=2 "git flow diagram")](#)git flow diagram ##### Prerequisites As part of your initial dbt setup, you should already have Development and Production environments configured. Let's recap what each does: * Your **Development environment** powers the Studio IDE. Each user has individual credentials, and builds into an individual dev schema. Nothing you do here impacts any of your colleagues. * Your **Production environment** brings the canonical version of your project to life for downstream consumers. There is a single set of deployment credentials, and everything is built into your production schema(s). #### Create a new CI environment See [Create a new environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-deployment-environment). The environment should be called **CI**. Just like your existing Production environment, it will be a Deployment-type environment. When setting a Schema in the **Deployment Credentials** area, remember that dbt will automatically generate a custom schema name for each PR to ensure that they don't interfere with your deployed models. This means you can safely set the same Schema name as your Production job. ##### 1. Double-check your Production environment is identified Go into your existing Production environment, and ensure that the **Set as Production environment** checkbox is set. It'll make things easier later. ##### 2. Create a new job in the CI environment Use the **Continuous Integration Job** template, and call the job **CI Check**. In the Execution Settings, your command will be preset to `dbt build --select state:modified+`. Let's break this down: * [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) runs all nodes (seeds, models, snapshots, tests) at once in DAG order. If something fails, nodes that depend on it will be skipped. * The [`state:modified+` selector](https://docs.getdbt.com/reference/node-selection/methods.md#state) means that only modified nodes and their children will be run ("Slim CI"). In addition to [not wasting time](https://discourse.getdbt.com/t/how-we-sped-up-our-ci-runs-by-10x-using-slim-ci/2603) building and testing nodes that weren't changed in the first place, this significantly reduces compute costs. To be able to find modified nodes, dbt needs to have something to compare against. dbt uses the last successful run of any job in your Production environment as its [comparison state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). As long as you identified your Production environment in Step 2, you won't need to touch this. If you didn't, pick the right environment from the dropdown. If you point CI at a non-production environment (staging, QA, UAT, or similar) that runs many jobs, comparison manifests can change unpredictably, or lag behind merges to your integration branch (for example, `develop`). Use CI to test your metrics If you've [built semantic nodes](https://docs.getdbt.com/docs/build/build-metrics-intro.md) in your dbt project, you can [validate them in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. ##### 3. Test your process That's it! There are other steps you can take to be even more confident in your work, such as validating your structure follows best practices and linting your code. For more information, refer to [Get started with Continuous Integration tests](https://docs.getdbt.com/guides/set-up-ci.md). To test your new flow, create a new branch in the Studio IDE then add a new file or modify an existing one. Commit it, then create a new Pull Request (not a draft). Within a few seconds, you’ll see a new check appear in your git provider. ##### Things to keep in mind * If you make a new commit while a CI run based on older code is in progress, it will be automatically canceled and replaced with the fresh code. * An unlimited number of CI jobs can run at once. If 10 developers all commit code to different PRs at the same time, each person will get their own schema containing their changes. Once each PR is merged, dbt will drop that schema. * CI jobs will never block a production run. #### Keep comparison manifests stable Deferring to Production is straightforward when one primary deploy job owns artifacts and merges are infrequent. With high merge volume, Production’s manifest can still move while other pull requests stay open, so CI may select extra `state:modified` nodes until those branches incorporate the latest merges (or you add a fast [merge triggered](https://docs.getdbt.com/docs/deploy/merge-jobs.md) `dbt parse` in the deferred environment). Busy staging, QA, or UAT environments add another failure mode as many jobs with different settings overwrite `manifest.json`. Any deferred environment can also be stale for a short window right after a merge until the next successful run refreshes artifacts. #### Enforce best practices with dbt project evaluator dbt Project Evaluator is a package designed to identify deviations from best practices common to many dbt projects, including modeling, testing, documentation, structure and performance problems. For an introduction to the package, read its [launch blog post](https://docs.getdbt.com/blog/align-with-dbt-project-evaluator). ##### 1. Install the package As with all packages, add a reference to `dbt-labs/dbt_project_evaluator` to your `packages.yml` file. See the [dbt Package Hub](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) for full installation instructions. ##### 2. Define test severity with an environment variable As noted in the [documentation](https://dbt-labs.github.io/dbt-project-evaluator/latest/ci-check/), tests in the package are set to `warn` severity by default. To have these tests fail in CI, create a new environment called `DBT_PROJECT_EVALUATOR_SEVERITY`. Set the project-wide default to `warn`, and set it to `error` in the CI environment. In your `dbt_project.yml` file, override the severity configuration: ```yaml data_tests: dbt_project_evaluator: +severity: "{{ env_var('DBT_PROJECT_EVALUATOR_SEVERITY', 'warn') }}" ``` ##### 3. Update your CI commands Because these tests should only run after the rest of your project has been built, your existing CI command will need to be updated to exclude the dbt\_project\_evaluator package. You will then add a second step which builds *only* the package's models and tests. Update your steps to: ```bash dbt build --select state:modified+ --exclude package:dbt_project_evaluator dbt build --select package:dbt_project_evaluator ``` ##### 4. Apply any customizations Depending on the state of your project when you roll out the evaluator, you may need to skip some tests or allow exceptions for some areas. To do this, refer to the documentation on: * [disabling tests](https://dbt-labs.github.io/dbt-project-evaluator/latest/customization/customization/) * [excluding groups of models from a specific test](https://dbt-labs.github.io/dbt-project-evaluator/latest/customization/exceptions/) * [excluding packages or sources/models based on path](https://dbt-labs.github.io/dbt-project-evaluator/latest/customization/excluding-packages-and-paths/) If you create a seed to exclude groups of models from a specific test, remember to disable the default seed and include `dbt_project_evaluator_exceptions` in your second `dbt build` command above. #### Run linting checks with SQLFluff By [linting](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#lint) your project during CI, you can ensure that code styling standards are consistently enforced, without spending human time nitpicking comma placement. Seamlessly enable [SQL linting for your CI job](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) in dbt to invoke [SQLFluff](https://docs.sqlfluff.com/en/stable/), a modular and configurable SQL linter that warns you of complex functions, syntax, formatting, and compilation errors. SQL linting in CI lints all the changed SQL files in your project (compared to the last deferred production state). Available on dbt [Starter, Enterprise, or Enterprise+ accounts](https://www.getdbt.com/pricing) using [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). ##### Manually set up SQL linting in CI You can run SQLFluff as part of your pipeline even if you don't have access to [SQL linting in CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting). The following steps walk you through setting up a CI job using SQLFluff to scan your code for linting errors. If you're new to SQLFluff rules in dbt, check out [our recommended config file](https://docs.getdbt.com/best-practices/how-we-style/2-how-we-style-our-sql.md). ##### 1. Create a YAML file to define your pipeline The YAML files defined below are what tell your code hosting platform the steps to run. In this setup, you’re telling the platform to run a SQLFluff lint job every time a commit is pushed. ##### GitHub GitHub Actions are defined in the `.github/workflows` directory. To define the job for your action, add a new file named `lint_on_push.yml` under the `workflows` folder. Your final folder structure will look like this: ```sql my_awesome_project ├── .github │ ├── workflows │ │ └── lint_on_push.yml ``` **Key pieces:** * `on:` defines when the pipeline is run. This workflow will run whenever code is pushed to any branch except `main`. For other trigger options, check out [GitHub’s docs](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows). * `runs-on: ubuntu-latest` - this defines the operating system we’re using to run the job * `uses:` - When the Ubuntu server is created, it is completely empty. [`checkout`](https://github.com/actions/checkout#checkout-v3) and [`setup-python`](https://github.com/actions/setup-python#setup-python-v3) are public GitHub Actions which enable the server to access the code in your repo, and set up Python correctly. * `run:` - these steps are run at the command line, as though you typed them at a prompt yourself. This will install sqlfluff and lint the project. Be sure to set the correct `--dialect` for your project. For a full breakdown of the properties in a workflow file, see [Understanding the workflow file](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#understanding-the-workflow-file) on GitHub's website. ```yaml name: lint dbt project on push on: push: branches-ignore: - 'main' jobs: # this job runs SQLFluff with a specific set of rules # note the dialect is set to Snowflake, so make that specific to your setup # details on linter rules: https://docs.sqlfluff.com/en/stable/rules.html lint_project: name: Run SQLFluff linter runs-on: ubuntu-latest steps: - uses: "actions/checkout@v3" - uses: "actions/setup-python@v4" with: python-version: "3.9" - name: Install SQLFluff run: "python -m pip install sqlfluff" - name: Lint project run: "sqlfluff lint models --dialect snowflake" ``` ##### GitLab Create a `.gitlab-ci.yml` file in your **root directory** to define the triggers for when to execute the script below. You’ll put the code below into this file. ```sql my_awesome_project ├── dbt_project.yml ├── .gitlab-ci.yml ``` **Key pieces:** * `image: python:3.9` - this defines the virtual image we’re using to run the job * `rules:` - defines when the pipeline is run. This workflow will run whenever code is pushed to any branch except `main`. For other rules, refer to [GitLab’s documentation](https://docs.gitlab.com/ee/ci/yaml/#rules). * `script:` - this is how we’re telling the GitLab runner to execute the Python script we defined above. ```yaml image: python:3.9 stages: - pre-build # this job runs SQLFluff with a specific set of rules # note the dialect is set to Snowflake, so make that specific to your setup # details on linter rules: https://docs.sqlfluff.com/en/stable/rules.html lint-project: stage: pre-build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != 'main' script: - python -m pip install sqlfluff - sqlfluff lint models --dialect snowflake ``` ##### Bitbucket Create a `bitbucket-pipelines.yml` file in your **root directory** to define the triggers for when to execute the script below. You’ll put the code below into this file. ```sql my_awesome_project ├── bitbucket-pipelines.yml ├── dbt_project.yml ``` **Key pieces:** * `image: python:3.11.1` - this defines the virtual image we’re using to run the job * `'**':` - this is used to filter when the pipeline runs. In this case we’re telling it to run on every push event, and you can see at line 12 we're creating a dummy pipeline for `main`. More information on filtering when a pipeline is run can be found in [Bitbucket's documentation](https://support.atlassian.com/bitbucket-cloud/docs/pipeline-triggers/) * `script:` - this is how we’re telling the Bitbucket runner to execute the Python script we defined above. ```yaml image: python:3.11.1 pipelines: branches: '**': # this sets a wildcard to run on every branch - step: name: Lint dbt project script: - python -m pip install sqlfluff==0.13.1 - sqlfluff lint models --dialect snowflake --rules L019,L020,L021,L022 'main': # override if your default branch doesn't run on a branch named "main" - step: script: - python --version ``` ##### 2. Commit and push your changes to make sure everything works After you finish creating the YAML files, commit and push your code to trigger your pipeline for the first time. If everything goes well, you should see the pipeline in your code platform. When you click into the job you’ll get a log showing that SQLFluff was run. If your code failed linting you’ll get an error in the job with a description of what needs to be fixed. If everything passed the lint check, you’ll see a successful job run. ##### GitHub In your repository, click the *Actions* tab [![Image showing the GitHub action for lint on push](/img/guides/orchestration/custom-cicd-pipelines/lint-on-push-github.png?v=2 "Image showing the GitHub action for lint on push")](#)Image showing the GitHub action for lint on push Sample output from SQLFluff in the `Run SQLFluff linter` job: [![Image showing the logs in GitHub for the SQLFluff run](/img/guides/orchestration/custom-cicd-pipelines/lint-on-push-logs-github.png?v=2 "Image showing the logs in GitHub for the SQLFluff run")](#)Image showing the logs in GitHub for the SQLFluff run ##### GitLab In the menu option go to *CI/CD > Pipelines* [![Image showing the GitLab action for lint on push](/img/guides/orchestration/custom-cicd-pipelines/lint-on-push-gitlab.png?v=2 "Image showing the GitLab action for lint on push")](#)Image showing the GitLab action for lint on push Sample output from SQLFluff in the `Run SQLFluff linter` job: [![Image showing the logs in GitLab for the SQLFluff run](/img/guides/orchestration/custom-cicd-pipelines/lint-on-push-logs-gitlab.png?v=2 "Image showing the logs in GitLab for the SQLFluff run")](#)Image showing the logs in GitLab for the SQLFluff run ##### Bitbucket In the left menu pane, click on *Pipelines* [![Image showing the Bitbucket action for lint on push](/img/guides/orchestration/custom-cicd-pipelines/lint-on-push-bitbucket.png?v=2 "Image showing the Bitbucket action for lint on push")](#)Image showing the Bitbucket action for lint on push Sample output from SQLFluff in the `Run SQLFluff linter` job: [![Image showing the logs in Bitbucket for the SQLFluff run](/img/guides/orchestration/custom-cicd-pipelines/lint-on-push-logs-bitbucket.png?v=2 "Image showing the logs in Bitbucket for the SQLFluff run")](#)Image showing the logs in Bitbucket for the SQLFluff run #### Advanced: Create a release train with additional environments Large and complex enterprises sometimes require additional layers of validation before deployment. Learn how to add these checks with dbt. Are you sure you need this? This approach can increase release safety, but creates additional manual steps in the deployment process as well as a greater maintenance burden. As such, it may slow down the time it takes to get new features into production. The team at Sunrun maintained a SOX-compliant deployment in dbt while reducing the number of environments. Check out [their Coalesce presentation](https://www.youtube.com/watch?v=vmBAO2XN-fM) to learn more. In this section, we will add a new **QA** environment. New features will branch off from and be merged back into the associated `qa` branch, and a member of your team (the "Release Manager") will create a PR against `main` to be validated in the CI environment before going live. The git flow will look like this: [![git flow diagram with an intermediary branch](/img/best-practices/environment-setup/many-branch-git.png?v=2 "git flow diagram with an intermediary branch")](#)git flow diagram with an intermediary branch ##### Advanced prerequisites * You have the **Development**, **CI**, and **Production** environments, as described in [the Baseline setup](https://docs.getdbt.com/guides/set-up-ci.md). ##### 1. Create a `release` branch in your git repo As noted above, this branch will outlive any individual feature, and will be the base of all feature development for a period of time. Your team might choose to create a new branch for each sprint (`qa/sprint-01`, `qa/sprint-02`, etc), tie it to a version of your data product (`qa/1.0`, `qa/1.1`), or just have a single `qa` branch which remains active indefinitely. ##### 2. Update your Development environment to use the `qa` branch See [Custom branch behavior](https://docs.getdbt.com/docs/dbt-platform-environments.md#custom-branch-behavior). Setting `qa` as your custom branch ensures that the IDE creates new branches and PRs with the correct target, instead of using `main`. [![A demonstration of configuring a custom branch for an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/dev-environment-custom-branch.png?v=2 "A demonstration of configuring a custom branch for an environment")](#)A demonstration of configuring a custom branch for an environment ##### 3. Create a new QA environment See [Create a new environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-deployment-environment). The environment should be called **QA**. Just like your existing Production and CI environments, it will be a Deployment-type environment. Set its branch to `qa` as well. ##### 4. Create a new job Use the **Continuous Integration Job** template, and call the job **QA Check**. In the Execution Settings, your command will be preset to `dbt build --select state:modified+`. Let's break this down: * [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) runs all nodes (seeds, models, snapshots, tests) at once in DAG order. If something fails, nodes that depend on it will be skipped. * The [`state:modified+` selector](https://docs.getdbt.com/reference/node-selection/methods.md#state) means that only modified nodes and their children will be run ("Slim CI"). In addition to [not wasting time](https://discourse.getdbt.com/t/how-we-sped-up-our-ci-runs-by-10x-using-slim-ci/2603) building and testing nodes that weren't changed in the first place, this significantly reduces compute costs. To be able to find modified nodes, dbt needs to have something to compare against. Normally, we use the Production environment as the source of truth, but in this case there will be new code merged into `qa` long before it hits the `main` branch and Production environment. Because of this, we'll want to defer the Release environment to itself. ##### Optional: also add a compile-only job dbt uses the last successful run of any job in that environment as its [comparison state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). If you have a lot of PRs in flight, the comparison state could switch around regularly. Adding a regularly-scheduled job inside of the QA environment whose only command is `dbt compile` can regenerate a more stable manifest for comparison purposes. ##### 5. Test your process When the Release Manager is ready to cut a new release, they will manually open a PR from `qa` into `main` from their git provider (e.g. GitHub, GitLab, Azure DevOps). dbt will detect the new PR, at which point the existing check in the CI environment will trigger and run. When using the [baseline configuration](https://docs.getdbt.com/guides/set-up-ci.md), it's possible to kick off the PR creation from inside of the Studio IDE. Under this paradigm, that button will create PRs targeting your QA branch instead. To test your new flow, create a new branch in the Studio IDE then add a new file or modify an existing one. Commit it, then create a new Pull Request (not a draft) against your `qa` branch. You'll see the integration tests begin to run. Once they complete, manually create a PR against `main`, and within a few seconds you’ll see the tests run again but this time incorporating all changes from all code that hasn't been merged to main yet. --- ### Getting started with the terminal [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### What is the terminal? The terminal (also called the command line, shell, or CLI) is a text-based interface for running commands on your computer. Many dbt tools — including dbt v2, dbt v1, and the dbt Wizard CLI — run from the terminal. You don't need to be a terminal expert to use dbt. This guide covers the basics. #### Open the terminal ##### macOS **Option 1: Spotlight** Press `⌘ Space`, type `Terminal`, and press Enter. **Option 2: Applications folder** Go to **Applications** → **Utilities** → **Terminal**. **Option 3: VS Code integrated terminal** In VS Code, press `` ⌃` `` (Control + backtick) to open a terminal panel directly in your editor. ##### Windows **Option 1: Command Prompt or PowerShell** Press `Win + R`, type `cmd` or `powershell`, and press Enter. **Option 2: Windows Terminal** Search for "Terminal" in the Start menu. Windows Terminal supports Command Prompt, PowerShell, and WSL side by side. **Option 3: VS Code integrated terminal** In VS Code, press `` Ctrl+` `` to open a terminal panel directly in your editor. Using dbt on Windows? dbt runs natively on Windows via PowerShell. For the best experience, consider using [WSL (Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl/install). ##### Linux Open your distribution's terminal emulator — usually found in the applications menu, or press `Ctrl + Alt + T` on most desktop environments. #### Navigate your file system When the terminal opens, you're in a directory (folder). These commands help you move around: | Command | What it does | Example | | ------------- | ------------------------------------ | ---------------------- | | `pwd` | Print the current directory | `pwd` → `/Users/alice` | | `ls` | List files and folders (macOS/Linux) | `ls` | | `dir` | List files and folders (Windows) | `dir` | | `cd <folder>` | Change into a folder | `cd my-dbt-project` | | `cd ..` | Go up one folder | `cd ..` | | `cd ~` | Go to your home directory | `cd ~` | **Tips:** * Press **Tab** to autocomplete folder and file names — saves a lot of typing. * Press **↑ / ↓** to scroll through previous commands. * Press **Ctrl+C** to cancel a running command. #### Navigate to your dbt project Your dbt project is a folder on your computer containing a `dbt_project.yml` file. Before running any dbt commands, navigate into that folder: ```bash cd ~/path/to/your-dbt-project ``` Verify you're in the right place: ```bash ls # macOS/Linux — you should see dbt_project.yml dir # Windows ``` #### Run your first dbt command After installing dbt, you can run your first dbt command to verify it's installed and working. Make sure you're in your project folder before running any dbt commands: ```bash cd ~/path/to/your-dbt-project ``` Then run your first dbt command: ```bash dbt --version ``` You should see output similar to the following: ```bash dbt-fusion 2.0.1 ``` .... and that's it! Congrats, you're ready to start using dbt in the terminal! 🎉 #### Common issues **`command not found`** The tool isn't installed or isn't on your PATH. Double-check the install instructions for [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) or [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). **`Permission denied`** You may need to run the command with elevated permissions, or check that the file is executable. **Nothing happens after I type** Make sure you pressed **Enter** after the command. #### Next steps * [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) — use the dbt Wizard CLI in your terminal * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) — set up dbt v2 or dbt v1 locally * [dbt commands reference](https://docs.getdbt.com/reference/dbt-commands.md) — all available dbt CLI commands --- ### How to use prompts for dbt Wizard [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Overview Learn how to write effective prompts for dbt dbt Wizard to generate accurate SQL, models, metrics, and macros. Each recipe is self-contained with its own realistic example. dbt dbt Wizard is an AI assistant that generates SQL, YAML, documentation, tests, semantic models, and macros based on your project's context. The quality of output depends on the clarity of your prompts. This cookbook provides independent recipes for common prompting tasks. Jump to any section that matches your current need. This cookbook covers the following topics: * [Prompt best practices](https://docs.getdbt.com/guides/prompt-cookbook?step=2) * [Generate SQL queries](https://docs.getdbt.com/guides/prompt-cookbook?step=3) * [Use what you already have](https://docs.getdbt.com/guides/prompt-cookbook?step=4) * [Create semantic models and metrics](https://docs.getdbt.com/guides/prompt-cookbook?step=5) * [Create reusable macros](https://docs.getdbt.com/guides/prompt-cookbook?step=6) * [Troubleshoot errors and issues](https://docs.getdbt.com/guides/prompt-cookbook?step=7) * [Conclusion](https://docs.getdbt.com/guides/prompt-cookbook?step=8) #### Prompt best practices Writing effective prompts is about giving dbt Wizard the right context and clear direction. Follow these principles: * [Provide rich context](#provide-rich-context) * [Break complex logic into smaller steps](#break-complex-logic-into-smaller-steps) * [State the business question, not just the output](#state-the-business-question-not-just-the-output) * [Be clear and explicit about the result](#be-clear-and-explicit-about-the-result) ##### Provide rich context In your prompt, include table names, column types, and example values to describe how they relate to each other. Include the following: * Table relationships (such as `orders` connects to `customers` on `customer_id`) * Data types (such as `signup_date` is a timestamp) * Sample values (such as `plan_type` can be "monthly" or "annual") tip The following example uses SQL terminology (like data types and joins) because it's generating a SQL query. However, the principle of providing rich context applies to all dbt Wizard tasks—whether you're generating macros, documentation, or YAML configurations. **Example: Santi's neighborhood café** Let's say you run a neighborhood café and folks get a free drink after 10 visits: **Without rich context** (vague): ```text I need a query using customers, subscriptions, and activity tables to see weekly regulars. ``` **With rich context** (specific): ```text Context: I run a café loyalty program where customers earn a free drink after 10 visits. Tables and relationships: - customers (customer_id INT or integer, name STRING, email STRING, signup_date TIMESTAMP) - subscriptions (subscription_id INT, customer_id INT, plan_type STRING, start_date DATE, end_date DATE) - Joins to customers on customer_id - plan_type values: "monthly", "annual", null for non-subscribers - activity (activity_id INT, customer_id INT, visit_date DATE, visit_count INT) - Joins to customers on customer_id - visit_count tracks cumulative visits (resets after redemption) Business question: Show me which customers visit weekly (3+ times per week for 4+ weeks) and compare conversion rates: do high-frequency punch-card users convert to our 'beans of the month' subscription at a higher rate than casual visitors? ``` **Why it works:** The AI now knows exact data types, how tables relate, what values to expect, and the specific business logic (3+ visits/week defines "regulars"). ##### Break complex logic into smaller steps Common misconception Many users try to ask for everything at once in a single prompt. Breaking your request into smaller, sequential steps consistently produces better results. For multi-part tasks, write them as a sequence of clear instructions. dbt Wizard handles step-by-step logic better than complex, all-in-one requests. **Example:** ```text 1. Filter the dataset to active users in the last 90 days. 2. Calculate their average session duration. 3. Join to subscription data and group by plan tier. ``` **Why this works:** Each step is clear and actionable. You can always iterate on your prompt to refine results — start simple, then build complexity. ##### State the business question, not just the output Describe the decision or insight the query supports, and avoid only technical-like prompts. This means, instead of "count users", you can say "count active users per week to analyze engagement trends." **Example: The sneaker drop** Let's say you run an online sneaker shop and just launched a new feature: customers can view 3D previews of sneakers before buying. ```text We launched a 3D preview feature with our latest limited-edition sneaker drop. Did customers who used the 3D preview convert to buyers at a higher rate than those who only saw photos? Show me weekly conversion rates: browsers who became buyers, segmented by whether they used the 3D preview. If preview users convert 20%+ higher, we'll add 3D to all products. If not, we'll improve the feature before expanding. ``` **Why it works:** You've described the feature, the behavior you're measuring, specific success criteria (20%+ lift), and the decision you'll make based on results. ##### Be clear and explicit about the result Define the expected output clearly. Mention the expected columns in the final result and state whether results should be sorted, limited, or filtered. **What to specify:** * Expected column names and formats * Sort order and any limits (for example, "top 10 products by revenue") * Output format examples (for example, "`conversion_rate` as a percentage") **Example: The fitness challenge** In this example, you run a fitness app with a 2-week challenge, Kimiko's kettlebell challenge. ```text Give me a weekly trend with the date, active folks, and a simple 'engagement per person.' Then a summary by launch week with 'trial starts,' 'upgrades in 30 days,' and an 'upgrade rate' as a percentage. Each week, show active challengers and total workouts. By challenge start week, show how many upgraded to paid within 30 days and what their average workouts looked like. ``` **Why it works:** Specific metrics that are ready to present. #### Generate SQL queries Let's say you want to build a query to find top-spending customers. ```text Context: I have two tables: - customers (customer_id, name, email) - orders (order_id, customer_id, order_total, order_date) Relationship: orders.customer_id connects to customers.customer_id Business question: Show me the top 10 customers by total spending in 2024. Output: - customer_id - customer_name - total_spent - order_count Sort by total_spent descending, limit to 10 rows. ``` **What dbt Wizard generates:** ```sql select c.customer_id, c.name as customer_name, sum(o.order_total) as total_spent, count(o.order_id) as order_count from {{ ref('customers') }} c inner join {{ ref('orders') }} o on c.customer_id = o.customer_id where year(o.order_date) = 2024 group by c.customer_id, c.name order by total_spent desc limit 10 ``` **Why it works:** * Clear context about tables and their relationship * Specific business question with a defined time period * Explicit output requirements and sorting logic **Pro tip:** Start simple, then iterate. If dbt Wizard's first attempt isn't perfect, no worries! Refine your prompt with more specific details and let dbt Wizard do its magic, it usually gets there in the end ✨ #### Use what you already have You don't need to write everything from scratch. Pull in documentation, definitions, and sample data you already have—it helps dbt Wizard understand your specific business context. dbt Insights integration When using dbt Wizard in [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md), you can easily cross-reference between dbt Wizard's generated SQL and metadata from [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). This embedded integration makes it seamless to access documentation, definitions, and sample data while building queries. ##### Define your business rules Instead of just saying "active customer," explain the rule: ```text Active customer = at least one paid purchase in the last 90 days, excluding refunds Net revenue = gross sales minus discounts and returns ``` **Pull from:** Metrics glossaries, KPI catalogs, product requirement docs, data dictionaries ##### Show sample values Give dbt Wizard examples of what the data actually looks like, especially edge cases: ```text Order statuses: - `customer_id: C-12, created_at: 2025-05-03T09:07:00Z, status: 'completed'` - `customer_id: C-14, created_at: 2025-05-03T09:02:00Z, status: 'cancelled'` - `customer_id: C-13, created_at: 2020-01-02T06:40:00Z, status: 'pending'` ``` **Pull from:** Data profiling reports, QA test datasets, BI dashboard filters ##### Start with a draft, refine later Frame your model first, then iterate. Start with a clean outline that gets the basic structure right: ```text From stg_orders and dim_customers, draft a minimal model with order_id, customer_id, order_date, net_revenue = gross - coalesce(discount, 0), and join to dim_customers on customer_id. Filter to the last 30 days for preview only. ``` **Pull from:** Source-to-target mapping sheets (join keys and transformations), data dictionaries (primary and foreign keys) #### Create semantic models and metrics Fast-track your semantic layer strategy with AI-generated YAML using dbt Wizard. dbt platform provides built-in generation buttons that automatically [generate code](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md), [documentation](https://docs.getdbt.com/docs/build/documentation.md), [data tests](https://docs.getdbt.com/docs/build/data-tests.md), [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md), and [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) for you with the click of a button in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md), [Canvas](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md), and [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md). These features understand your model's structure and generate YAML in the correct location. **How to generate semantic models:** 1. Navigate to the Studio IDE and select a SQL model file in the **File explorer** 2. In the **Console** section (under the **Editor**), click the **dbt Wizard** icon to view AI options 3. Select **Semantic model** to create a semantic model based on your SQL model 4. Review and refine the generated YAML as needed You can also use dbt Wizard to generate documentation, tests, and metrics. These built-in features automatically understand your model's columns, data types, and relationships, which means you don't need to manually describe your schema or copy-paste between file types. **Typical workflow:** 1. Build your SQL model using dbt Wizard conversational prompts 2. Use built-in buttons to add documentation, tests, and semantic models 3. Refine the generated YAML as needed For more details, check out the [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) docs. #### Create reusable macros In this section, we'll look at how to create reusable macros using dbt Wizard. * [Turn repetitive code into reusable logic](#turn-repetitive-code-into-reusable-logic) * [Lower the barrier to entry](#lower-the-barrier-to-entry) * [Accelerate complex logic design](#accelerate-complex-logic-design) ##### Turn repetitive code into reusable logic A junior analyst keeps copy-pasting CASE statements across models. **What to give dbt Wizard:** ```text Turn this CASE pattern into a reusable macro: CASE WHEN amount >= 1000 THEN 'high' WHEN amount >= 500 THEN 'medium' ELSE 'low' END Macro requirements: - Name: categorize_amount - Parameters: column name, high threshold (default 1000), medium threshold (default 500) - Include docstring with usage example - Handle null values by returning 'unknown' ``` **Why it works:** Clear input (the CASE statement), clear requirements, clear output expectations. ##### Lower the barrier to entry You need a macro but don't know Jinja syntax well. **What to ask dbt Wizard:** ```text I need a macro that calculates the number of days between two date columns, excluding weekends. Parameters: - start_date_column (required) - end_date_column (required) Include a docstring explaining how to use it. ``` **Outcome:** dbt Wizard generates proper Jinja syntax, handles parameters, and includes documentation. You learn Jinja patterns while getting working code. ##### Accelerate complex logic design This is best for advanced users who are comfortable with Jinja. **What to ask dbt Wizard:** ```text I need a macro that builds a grouped aggregation with optional filters. Parameters: - relation (the model/table to query) - group_by (list of columns to group by) - metrics (list of columns to aggregate) - where (optional filter condition) Include defaults and guardrails for empty lists. Add a docstring with parameter descriptions and usage example. ``` **Why this works:** You've outlined the interface (parameters) and edge cases (empty lists), letting dbt Wizard handle the Jinja boilerplate while you focus on design. This approach accelerates iteration so you can refine the structure without getting stuck in syntax details. #### Troubleshoot errors and issues dbt Wizard acts as a fast, context-aware reviewer for failing SQL and macros. It reads errors, inspects your query structure, and suggests minimal fixes. Troubleshooting with dbt Wizard gives you: * Faster diagnosis by using plain-language translation of errors with likely root causes * Safer fixes by biasing toward small, targeted changes * Better learning by generating explanations you can paste into docs or PR descriptions ##### Troubleshoot errors When something breaks, give dbt Wizard the error message, your code, and what you expected to happen. Here are a couple of examples to show you how to use dbt Wizard to troubleshoot errors. **Example: SQL error** ```text Error: "SQL compilation error: Column 'product_name' must appear in GROUP BY" Query: SELECT product_id, product_name, SUM(quantity) as total_quantity FROM inventory GROUP BY product_id Warehouse: Snowflake Expected: Group by product and show product name. What's wrong and how do I fix it? ``` **Example: Macro not working** ```text This macro should calculate discount but returns wrong values: {% macro calculate_discount(amount, rate) %} {{ amount }} * {{ rate }} {% endmacro %} When I call {{ calculate_discount(100, 0.1) }} I expect 10 but get an error. Show me the rendered SQL from target/compiled and explain what's wrong. ``` **Tip:** Include your warehouse type (Snowflake, BigQuery, Databricks and so on.) — this is because the syntax can vary across data platforms. #### Conclusion Congrats, you've now learned some tips on how to create and use prompts for dbt dbt Wizard 🎉! You can: * Boost your prompting skills by providing rich context and stating clear business questions. Applicable for SQL, macros, documentation, tests, metrics, and semantic models. * Amplify your workflow by using existing documentation and project context * Generate Jinja macros to build more scalable and maintainable systems * Troubleshoot your code to diagnose issues fast and apply safe, explainable fixes ##### Quick reference checklist When writing prompts for dbt dbt Wizard: * ✅ Provide rich context: Table names, columns, data types, relationships, sample values * ✅ Break down complex logic: Write multi-part queries as a sequence of steps * ✅ State the business question: What decision or insight you're supporting, not just "write a query" * ✅ Be clear and explicit: Expected columns, sort order, filters, and output format For troubleshooting: * ✅ Include complete error messages: Full warehouse error with line numbers * ✅ Show the failing code: Both the dbt model and compiled SQL (from `target/compiled/`) * ✅ Provide sample data: Representative rows that trigger the issue * ✅ State your warehouse: Snowflake, BigQuery, Databricks, etc. ##### Next steps Start with one task—automating documentation, generating a test, or refactoring a model—and build the habit from there. The more you use dbt Wizard, the more you'll discover ways to accelerate your analytics engineering workflow. Check out the following docs to learn more about how to use dbt Wizard: * [About dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Generate resources](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#generate-resources) * [Build visual models with dbt Wizard](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md) * [Build queries in dbt Insights](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md) --- ### Hybrid development with dbt platform and dbt v2 [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Hybrid dbt deployments are becoming increasingly common. dbt v2 adopters are frequently working in several places at once: in the dbt platform for production runs and IDE-based development, and on their local machine using the dbt platform CLI or the dbt VS Code extension. These paths are fully supported for dbt platform users. Keeping the environments in sync across credentials, environment variables, and engine versions is one of the first operational challenges teams encounter. This guide walks through command routing, credentials, environment variables, dbt v2 versions, and Mesh or deferral, with concrete, copy-paste-ready steps to keep everything aligned. If you run both the dbt platform CLI and a local dbt v2 build from the same project, start with [Choosing which dbt runs](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md?step=3#choosing-which-dbt-runs). Both tools are invoked as `dbt`, and the rest of this guide assumes you can tell them apart. #### Prerequisites * You have a dbt platform account with at least one project using dbt v2. * You have either the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or the [dbt VS Code extension + local dbt](https://docs.getdbt.com/docs/local/install-dbt.md) installed. #### Choosing which dbt runs If you install both the dbt platform CLI and a local dbt v2 build, you have two separate programs on your machine that are both invoked by typing `dbt`. Before you configure credentials, environment variables, or versions, make it unambiguous which one you're calling. Skip this section if you only ever install one of the two. ##### The two execution paths Both the dbt platform CLI and the local dbt v2 execution paths read the same project files — one clone of your repository, one `dbt_project.yml`, one set of models, macros, and tests. What differs is where the work happens and where the connection details come from. | Area | dbt platform CLI | Local dbt v2 | | ------------------ | ------------------------------------------------ | ----------------------------------------------- | | **What it is** | A client that sends your command to dbt platform | A dbt executable that runs on your machine | | **Where dbt runs** | On dbt platform infrastructure | Locally, or inside your agent's virtual machine | ##### Give each tool its own command Because both programs are installed as `dbt`, whichever one appears first in your `$PATH` opens when you use `dbt`. To avoid relying on `$PATH` order, you can assign at least one of them an alias that you can use on the command line to call each one explicitly. The dbt v2 [installation script](https://docs.getdbt.com/docs/local/install-dbt.md) already provides `dbtf` as an alias and points to the local dbt v2 binary, or executable program. You can also add a `dbt-cli` alias for the dbt platform CLI so each command calls exactly what it says, for example: ```shell dbtf build --select my_model # Runs locally, on your installed v2 binary dbt-cli build --select my_model # Runs on dbt platform, on your environment's release track ``` Follow these steps to set up an alias: 1. Find out what `dbt` resolves to today, and whether more than one is installed: ```shell which -a dbt ``` 2. Install the tools you need: * Install dbt v2 using the [installation script](https://docs.getdbt.com/docs/local/install-dbt.md), which puts it in `$HOME/.local/bin/dbt` on macOS and Linux, or `C:\Users\USERNAME\.local\bin\dbt.exe` on Windows, and adds the `dbtf` alias. * Install the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) separately. 3. Using the path you identified in Step 1 for your dbt platform CLI install, add an alias to your shell profile that points to its installed program. For example, on macOS with Homebrew you can create an alias for the executable at `/opt/homebrew/bin/dbt`: ```shell # ~/.zshrc or ~/.bashrc alias dbt-cli="/opt/homebrew/bin/dbt" ``` Reload your shell profile: ```shell source ~/.zshrc # or source ~/.bashrc ``` 4. Confirm each command resolves to the tool you expect and the two version strings differ: ```shell dbtf --version dbt-cli --version ``` 5. Decide which program opens when you run bare `dbt` on your machine and create a best practice for your team. Regardless, `dbtf` and `dbt-cli` clearly run the named program. Shell aliases don't apply everywhere `dbtf` and `dbt-cli` are shell aliases. They're available in your interactive terminal, but not in `Makefile` recipes, shell scripts, CI jobs, or commands an AI agent runs in a non-interactive shell. In those contexts, use the absolute path to the binary, or control `$PATH` ordering so that bare `dbt` resolves to the tool you want. ##### Commands that mean different things in each tool Most commands (`build`, `run`, `test`, `compile`) behave equivalently, but the run happens in a different place. A few are specific to one tool, and running them against the other either fails or does something you didn't intend: * `dbt system update` and `dbt system uninstall` manage a local dbt v2 install. They have no meaning for the dbt platform CLI and no effect on dbt platform. * `dbt init` hydrates a local `profiles.yml`. You need it for the local dbt v2 path, not for the dbt platform CLI, which doesn't use `profiles.yml`. * `dbt debug` inspects a local profile, target, and connection. Use [`dbt environment`](https://docs.getdbt.com/reference/commands/dbt-environment.md) for dbt platform CLI environment and connection details. When both tools are installed, write these as `dbtf system update`, `dbtf init`, and `dbtf debug` so they can't be misread. ##### How each editor and agent picks a dbt Each tool in your workflow resolves `dbt` on its own terms. Configuring one does not configure the others. | Where you run dbt | How it picks a dbt | What to configure | | ----------------------------------------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **dbt VS Code extension** | The `dbt.fusionPath` setting, or a v2 build the extension downloads and manages itself | Leave `dbt.fusionPath` unset to let the extension manage v2. If you install v2 yourself, set it to the absolute path of the v2 binary | | **VS Code or Cursor integrated terminal** | Your shell `$PATH` and shell profile aliases | The `dbtf` and `dbt-cli` aliases described earlier | | **Coding agents, such as Claude Code or Cursor agent mode** | A non-interactive shell — `$PATH` applies, shell aliases usually don't | Absolute binary paths, plus a written command-routing rule in the agent's instructions file | | **Remote agent virtual machines** | The virtual machine's own `$PATH`, not your workstation's | Install both tools in the virtual machine, persist them in its setup configuration so they survive fresh sessions, and store credentials in that platform's secrets manager | `dbt.fusionPath` is not a terminal setting `dbt.fusionPath` tells the dbt VS Code extension which binary to start the LSP and the extension's own menu actions from. It has no effect on commands you type in an integrated terminal, and no effect on commands an agent runs. It must point to a valid dbt v2 binary — an absolute filesystem path, not an alias name and not the dbt platform CLI. Find the path to pass it with: ```shell command -v dbtf ``` Keep machine-specific absolute paths in your user settings rather than committing them to workspace settings, so the setting doesn't break for teammates whose paths differ. ##### Tell your coding agent which command to use Agents run shell commands the same way a script does, so they inherit `$PATH` but not your interactive aliases, and they have no way to guess which execution path you intended. State the convention in the instructions file the agent reads — `CLAUDE.md`, `AGENTS.md`, `.cursor/rules/`, or the equivalent for your tool: ```markdown ## dbt command routing Replace the paths below with the output of `which -a dbt` on this machine. - Use `$HOME/.local/bin/dbt` for local dbt v2 commands. It runs on this machine against `profiles.yml`. - Use `/opt/homebrew/bin/dbt` for dbt platform CLI commands. It runs on dbt platform against the environment's release track. - Never substitute one for the other. - Run dbt commands from the repository root. - Ask before running any command that creates or modifies warehouse objects. ``` Use absolute paths in agent instructions rather than the `dbtf` and `dbt-cli` aliases, because the agent's shell may not load your shell profile. Discover the real paths on the machine the agent runs on with `which -a dbt`, and update the instructions file when they change. On a remote agent virtual machine, confirm the paths inside a fresh session, because tools installed ad hoc in an earlier session may not persist. #### Managing credentials How you authenticate to your data warehouse locally depends on which self-hosted tool you use: * [dbt platform CLI](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md?step=4#dbt-platform-cli): For a CLI-only development experience (without the dbt VS Code extension), use the dbt platform CLI with dbt v2 set as your platform release track. Warehouse credentials are managed centrally in dbt platform and passed through automatically — no `profiles.yml` required. * [dbt VS Code extension](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md?step=4#dbt-vs-code-extension-profilesyml-required): For IDE-based local development, the dbt VS Code extension runs dbt v2 and its LSP features in a local process. This path requires a `profiles.yml` to connect directly to your warehouse. ##### dbt platform CLI The [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) is the lowest-friction path for dbt platform users who want a self-hosted CLI-only workflow without VS Code. It authenticates using your dbt platform session, and your warehouse credentials are managed centrally in dbt platform and passed through automatically. For detailed installation instructions, refer to [Install the dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md?version=1.10). The dbt platform CLI is installed from your local command prompt. The configuration file downloaded from your dbt platform **Account settings** will facilitate the connection and authentication with your existing credentials. This is the lowest-friction path for teams that don't need full IDE integration locally. ##### dbt VS Code extension (profiles.yml required) The dbt VS Code extension runs dbt v2 and its language server in a local process and connects directly to your warehouse. For this reason, you need a `profiles.yml` for local extension development sessions. Download your [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) from your dbt platform **Account settings** and dbt v2 attempts to hydrate non-sensitive credential metadata from dbt platform automatically. If you get access to a new project, re-download the `dbt_cloud.yml` file before working on it locally. To switch between projects already listed in your file, update [`context.active-project`](https://docs.getdbt.com/reference/dbt_cloud.yml.md#update-or-switch-projects). To avoid manually recreating your warehouse configuration, use `dbt init`. ```shell dbt init ``` dbt v2 pulls down fields such as your **username**, **role**, **warehouse**, **database**, and **schema**, but never sensitive values like passwords or tokens. If your authentication mechanism is passwordless (such as `externalbrowser` or SSO-based OAuth), dbt v2 configures that too, so you can work without storing secrets locally. note This hydration happens once during initial setup and does not stay in sync automatically. When your warehouse configuration changes in dbt platform, run `dbt init` again to refresh your local `profiles.yml`. The dbt VS Code extension first-time setup flow prompts you through this process, so you usually don't need to run `dbt init` manually. Coming soon We're working on a solution that lets you develop locally in the dbt VS Code extension while you manage credentials entirely in dbt platform, without a local `profiles.yml`. We'll update this page when that ships. #### Managing environment variables Environment variables you set in dbt platform apply to production runs and the Studio IDE sessions. For local development, you manage environment variables separately. ##### dbt platform CLI When you use the dbt platform CLI, dbt platform injects the same environment variables you use in production into your dbt platform CLI session. You don't need extra setup. ##### VS Code extension (.env file) The dbt VS Code extension runs dbt v2 as a local process, so environment variables from dbt platform are not automatically available. Instead, use a [`.env` file](https://dotenvx.com/docs) at the root of your dbt project: ```shell # .env DBT_MY_DATABASE=my_database DBT_MY_SCHEMA=my_dev_schema DBT_TARGET_SCHEMA=analytics_dev ``` dbt v2 and the dbt VS Code extension automatically load values from this file. You can also view and override individual environment variables from the extension's settings UI. Reference these variables in your `profiles.yml` or elsewhere in your dbt project using the [`env_var` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md): ```yaml # profiles.yml my_profile: target: dev outputs: dev: type: snowflake account: my_account database: "{{ env_var('DBT_MY_DATABASE') }}" schema: "{{ env_var('DBT_MY_SCHEMA') }}" ``` For a full walkthrough of `.env` file usage and variable precedence, see [Environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) for more information on where to find your configured variables and [Set environment variables locally](https://docs.getdbt.com/docs/configure-dbt-extension.md?version=2.0#set-environment-variables-locally) for local configuration instructions. ##### Keeping platform and local variables in sync Environment variables in dbt platform and a local `.env` file don't sync automatically. Commit a `.env.example` file with the list of required variables to your repo. Each developer copies it locally and fills in their own values and their copy is never committed. ```shell # .env.example (committed to version control) DBT_MY_DATABASE= # Your development database name DBT_MY_SCHEMA= # Your personal dev schema, for example dbt_yourname DBT_TARGET_SCHEMA= # Target schema for dbt output ``` ```shell # Developer setup: copy the example and fill in your values cp .env.example .env ``` Do not commit .env dbt v2 and the dbt VS Code extension only load from a file named exactly `.env`, so each developer needs their own copy. Make sure `.env` is in your `.gitignore` so credentials are never committed. Running `dbt init` adds this automatically. ```shell echo ".env" >> .gitignore ``` When environment variables change in dbt platform (you add variables or rename values), update `.env.example` in the same pull request so local developers know to update their own `.env`. For teams with strict security requirements Consider a script that fetches variables from your secrets manager (for example, AWS Secrets Manager or 1Password) and writes them to `.env` at the start of a session, instead of storing values in a file long term. #### Managing dbt v2 versions The **v2 Stable** release track on dbt platform updates continuously as dbt v2 ships new releases. If your local version falls behind, you might see inconsistent behavior. The same query could compile differently locally than in production, or a feature might exist in dbt platform but not in your local binary. Stay current to avoid these mismatches. A v2 release track does not change your local dbt Moving a dbt platform environment to a v2 release track changes the engine that dbt platform uses for that environment. It does not install, update, replace, or select the dbt v2 executable on your machine, and it does not turn the dbt platform CLI into dbt v2. The reverse is also true: running `dbt system update` locally updates your local install only. It has no effect on which build your dbt platform environments run. Treat the two version settings as independent, and keep them aligned yourself using the steps in this section. ##### Versions on the dbt platform On dbt platform, dbt v2 follows a versionless release track model. The default release track is **v2 Stable**, which always runs the most recent stable release. For details on release tracks and their stability levels, see [dbt v2 releases](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md#dbt-v2-release-tracks). ##### Versions installed locally By default, the dbt v2 [installation instructions](https://docs.getdbt.com/docs/local/install-dbt.md) install the latest stable release, the same version that ships with the **v2 Stable** release track on dbt platform: ```shell # macOS / Linux pip install dbt ``` To update your self-hosted installation to the latest stable release at any time: ```shell dbt system update ``` To check your current version: ```shell dbt --version ``` ##### Keeping versions in sync: dev containers (recommended) Use a [VS Code dev container](https://code.visualstudio.com/docs/devcontainers/containers) for the most reliable match between local dbt v2 versions and dbt platform. A dev container runs your environment inside a Docker image that rebuilds at the start of each session and performs a fresh dbt v2 install each time, so everyone on your team uses the same version as dbt platform without manual updates. Our friends at Brooklyn Data have published a ready-to-use dbt v2 dev container: * **Dev container template:** [brooklyn-data/dbt-fusion-devcontainer](https://github.com/brooklyn-data/dbt-fusion-devcontainer) * **Blog post:** [Why you should use dev containers with dbt v2](https://www.brooklyndata.co/ideas/2025/06/11/why-you-should-use-dev-containers-with-dbt-fusion) To get started with their template: ```shell # Clone the devcontainer template into your project curl -fsSL https://raw.githubusercontent.com/brooklyn-data/dbt-fusion-devcontainer/main/setup.sh | sh ``` Then open your project in VS Code and select **Reopen in Container** when prompted. VS Code builds the image and installs the latest stable dbt v2 release automatically. Coming soon We're introducing additional dbt v2 release tracks on dbt platform beyond **v2 Stable**. When they're available, we'll update this guide with steps to pin your dev container to a specific track. ##### Without dev containers: update at the start of each session If dev containers aren't an option for your team, run `dbt system update` at the start of each development session instead. That installs the latest stable release, the same version as the **v2 Stable** track on dbt platform, so your local binary stays current: ```shell dbt system update && dbt debug ``` Pinning to a specific version number does not work long term here: the **v2 Stable** track on dbt platform keeps advancing, and a pinned self-hosted installation falls behind. Aim to stay on **v2 Stable** instead of locking to one release. To make this easy to remember, add a `dev` target to your project's `Makefile`: ```makefile # Makefile .PHONY: dev dev: dbt system update dbt debug ``` Then developers start their session with: ```shell make dev ``` You can also document this convention in your project's `CONTRIBUTING.md` so it's part of your onboarding checklist. *** #### dbt Mesh and deferral If your project uses [dbt Mesh](https://docs.getdbt.com/docs/mesh/about-mesh.md), referencing models from other dbt projects via cross-project refs, dbt v2 handles this automatically during development when a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) is present. ##### How it works When dbt v2 detects upstream projects defined in your `dependencies.yml`, it downloads the publication artifact for each upstream project from dbt platform before resolving cross-project refs. Then `ref('upstream_project', 'model_name')` works locally without manual setup. Your logs include lines such as the following while dbt v2 resolves cross-project refs: ```text Downloading publication artifact for <upstream_project> (resolving cross-project refs) Downloaded publication artifact for <upstream_project> to <path> (resolving cross-project refs) ``` dbt v2 caches downloaded publication artifacts for up to one hour, so subsequent runs in the same session skip the download and resolve refs from the local cache. Auto-deferral is also on by default. When a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) is present, dbt v2 defers to your project's configured deferral environment, so you build only modified models and their downstream dependencies while the rest resolve against the production state. ##### Disabling deferral * **In the VS Code extension:** Disable auto-deferral in the extension settings. Search for `Dbt > Flag: Defer` and uncheck the option: [![dbt VS Code extension deferral settings](/img/v2/vsce-defer-settings.png?v=2 "dbt VS Code extension deferral settings")](#)dbt VS Code extension deferral settings * **On the CLI:** Pass `--no-defer` to any command to skip both deferral and the publication artifact download: ```shell dbt run --no-defer dbt compile --no-defer ``` #### Reference table The following table summarizes the key differences between the two development paths covered in this guide: | Area | dbt platform CLI | dbt VS Code extension | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | **Command when both are installed** | `dbt-cli` (alias you add) | `dbtf` (alias the installer adds) | | **Credentials** | Managed through your dbt platform session, no `profiles.yml` needed | `profiles.yml` required; use `dbt init` to hydrate from dbt platform | | **Environment variables** | Same env vars as in dbt platform automatically | Use a `.env` file at the project root | | **Version management** | `dbt system update` to stay current | Dev container recommended for automatic sync | | **dbt Mesh / deferral** | Auto-enabled when [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) present; `--no-defer` to disable | Auto-enabled when [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) present; toggle off in extension settings | #### Related docs * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform CLI installation](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) * [dbt extension settings, including `dbt.fusionPath`](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings) * [`dbt environment` command](https://docs.getdbt.com/reference/commands/dbt-environment.md) * [dbt v2 releases and release channels](https://docs.getdbt.com/docs/dbt/dbt-releases.md) * [About profiles.yml](https://docs.getdbt.com/docs/local/profiles.yml.md) * [Environment variables (local)](https://docs.getdbt.com/docs/local/configure-environment-variables.md) * [VS Code dev containers](https://code.visualstudio.com/docs/devcontainers/containers) * [dbt Mesh overview](https://docs.getdbt.com/docs/mesh/about-mesh.md) * [Deferral in dbt](https://docs.getdbt.com/docs/platform/about-defer.md) --- ### Integrate with dbt Semantic Layer using best practices [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction To fit your tool within the world of the Semantic Layer, dbt Labs offers some best practice recommendations for how to expose metrics and allow users to interact with them seamlessly. This is an evolving guide that is meant to provide recommendations based on our experience. If you have any feedback, we'd love to hear it! 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). ##### Prerequisites To build a Semantic Layer integration: * We offer a [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) API and [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md). Refer to the dedicated [Semantic Layer API](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) for more technical integration details. * Familiarize yourself with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md)'s key concepts. There are two main objects: * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) — Nodes in your semantic graph, connected via entities as edges. MetricFlow takes semantic models defined in YAML configuration files as inputs and creates a semantic graph that you can use to query metrics. * [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) — Can be defined in the same YAML files as your semantic models, or split into separate YAML files into any other subdirectories (provided that these subdirectories are also within the same dbt project repo). ##### Connection parameters The dbt Semantic Layer APIs authenticate with `environmentId`, `SERVICE_TOKEN`, and `host`. We recommend you provide users with separate input fields with these components for authentication (dbt will surface these parameters for the user). ##### Exposing metadata to dbt Labs When building an integration, we recommend you expose certain metadata in the request for analytics and troubleshooting purpose. Please send us the following header with every query: `'X-dbt-partner-source': 'Your-Application-Name'` Additionally, it would be helpful if you also included the email and username of the person generating the query from your application. #### Use best practices when exposing metrics Best practices for exposing metrics are summarized into five themes: * [Governance](#governance-and-traceability) — Recommendations on how to establish guardrails for governed data work. * [Discoverability](#discoverability) — Recommendations on how to make user-friendly data interactions. * [Organization](#organization) — Organize metrics and dimensions for all audiences, use [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md). * [Query flexibility](#query-flexibility) — Allow users to query either one metric alone without dimensions or multiple metrics with dimensions. * [Context and interpretation](#context-and-interpretation) — Contextualize metrics for better analysis; expose definitions, metadata, lineage, and freshness. ##### Governance and traceability When working with more governed data, it's essential to establish clear guardrails. Here are some recommendations: * **Aggregations control** — Users shouldn't generally be allowed to modify aggregations unless they perform post-processing calculations on Semantic Layer data (such as year-over-year analysis). * **Time series alignment and using metric\_time** — Make sure users view metrics across the correct time series. When displaying metric graphs, using a non-default time aggregation dimension might lead to misleading interpretations. While users can still group by other time dimensions, they should be careful not to create trend lines with incorrect time axes.<br /><br />When looking at one or multiple metrics, users should use `metric_time` as the main time dimension to guarantee they are looking at the right time series for the metric(s).<br /><br />As such, when building an application, we recommend exposing `metric_time` as a separate, "special" time dimension on its own. This dimension is always going to align with all metrics and be common across them. Other time dimensions can still be looked at and grouped by, but having a clear delineation between the `metric_time` dimension and the other time dimensions is clarifying so that people do not confuse how metrics should be plotted.<br /><br />Also, when a user requests a time granularity change for the main time series, the query that your application runs should use `metric_time` as this will always give you the correct slice. Related to this, we also strongly recommend that you have a way to expose what dimension `metric_time` actually maps to for users who may not be familiar. Our APIs allow you to fetch the actual underlying time dimensions that makeup metric\_time (such as `transaction_date`) so you can expose them to your users. * **Units consistency** — If units are supported, it's vital to avoid plotting data incorrectly with different units. Ensuring consistency in unit representation will prevent confusion and misinterpretation of the data. * **Traceability of metric and dimension changes** — When users change names of metrics and dimensions for reports, it's crucial to have a traceability mechanism in place to link back to the original source metric name. ##### Discoverability * Consider treating [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) as first-class objects rather than measures. Metrics offer a higher-level and more contextual way to interact with data, reducing the burden on end-users to manually aggregate data. * **Easy metric interactions** — Provide users with an intuitive approach to: * Search for Metrics — Users should be able to easily search and find relevant metrics. Metrics can serve as the starting point to lead users into exploring dimensions. * Search for Dimensions — Users should be able to query metrics with associated dimensions, allowing them to gain deeper insights into the data. * Filter by Dimension Values — Expose and enable users to filter metrics based on dimension values, encouraging data analysis and exploration. * Filter additional metadata — Allow users to filter metrics based on other available metadata, such as metric type and default time granularity. * **Suggested metrics** — Ideally, the system should intelligently suggest relevant metrics to users based on their team's activities. This approach encourages user exposure, facilitates learning, and supports collaboration among team members. By implementing these recommendations, the data interaction process becomes more user-friendly, empowering users to gain valuable insights without the need for extensive data manipulation. ##### Organization We recommend organizing metrics and dimensions in ways that a non-technical user can understand the data model, without needing much context: * **Organizing dimensions** — To help non-technical users understand the data model better, we recommend organizing dimensions based on the entity they originated from. For example, consider dimensions like `user__country` and `product__category`.<br /><br />You can create groups by extracting `user` and `product` and then nest the respective dimensions under each group. This way, dimensions align with the entity or semantic model they belong to and make them more user-friendly and accessible. Additionally, we recommending adding a `label` parameter to dimensions in order to define the value displayed in downstream tools. * **Organizing metrics** — The goal is to organize metrics into a hierarchy in our configurations, instead of presenting them in a long list.<br /><br />This hierarchy helps you organize metrics based on specific criteria, such as business unit or team. By providing this structured organization, users can find and navigate metrics more efficiently, enhancing their overall data analysis experience. * **Using saved queries** — The Semantic Layer has a concept of [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) which allows users to pre-build slices of metrics, dimensions, filters to be easily accessed. You should surface these as first class objects in your integration. Refer to the [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) and [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) APIs for syntax. ##### Query flexibility Allow users to query either one metric alone without dimensions or multiple metrics with dimensions. * Allow toggling between metrics/dimensions seamlessly. * Be clear on exposing what dimensions are queryable with what metrics and hide things that don’t apply. (Our APIs provide calls for you to get relevant dimensions for metrics, and vice versa). * Only expose time granularities (monthly, daily, yearly) that match the available metrics. * For example, if a dbt model and its resulting semantic model have a monthly granularity, make sure querying data with a 'daily' granularity isn't available to the user. Our APIs have functionality that will help you surface the correct granularities * We recommend that time granularity is treated as a general time dimension-specific concept and that it can be applied to more than just the primary aggregation (or `metric_time`). Consider a situation where a user wants to look at `sales` over time by `customer signup month`; in this situation, having the ability to apply granularities to both time dimensions is crucial. Our APIs include information to fetch the granularities for the primary (metric\_time) dimensions, as well as all time dimensions. You can treat each time dimension and granularity selection independently in your application. Note: Initially, as a starting point, it makes sense to only support `metric_time` or the primary time dimension, but we recommend expanding that as your solution evolves. * You should allow users to filter on date ranges and expose a calendar and nice presets for filtering these. * For example, last 30 days, last week, and so on. ##### Context and interpretation For better analysis, it's best to have the context of the metrics close to where the analysis is happening. We recommend the following: * Expose business definitions of the metrics as well as logical definitions. * Expose additional metadata from the Semantic layer (measures, type parameters). * Use the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) to enhance the metric and build confidence in its accuracy: * Check if the metric is fresh and when it was last updated. * Include lineage information to understand the metric's origin. * Allow for creating other metadata that’s useful for the metric. We can provide some of this information in our configuration (Display name, Default Granularity for View, Default Time range), but there may be other metadata that your tool wants to provide to make the metric richer. ##### Transparency and using compile For transparency and additional context, we recommend you have an easy way for the user to obtain the SQL that MetricFlow generates. Depending on what API you are using, you can do this by using our `compile` parameter. This is incredibly powerful and emphasizes transparency and openness, particularly for technically inclined users. ##### Where filters and optimization In the cases where our APIs support either a string or a filter list for the `where` clause, we always recommend that your application utilizes the filter list in order to gain maximum pushdown benefits. The `where` string may be more intuitive for users writing queries during testing, but it will not have the performance benefits of the filter list in a production environment. #### Understand stages of an integration These are recommendations on how to evolve a Semantic Layer integration and not a strict runbook. **Stage 1 - The basic** * Supporting and using [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) or [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) is the first step. Refer to the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) for more technical details. **Stage 2 - More discoverability and basic querying** * Support listing metrics defined in the project * Listing available dimensions based on one or many metrics * Querying defined metric values on their own or grouping by available dimensions * Display metadata from [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) and other context * Expose [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), which are pre-built metrics, dimensions, and filters that Semantic Layer developers create for easier analysis. You can expose them in your application. Refer to the [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) and [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) APIs for syntax. **Stage 3 - More querying flexibility and better user experience (UX)** * More advanced filtering * Time filters with good presets/calendar UX * Filtering metrics on a pre-populated set of dimension values * Make dimension values more user-friendly by organizing them effectively * Intelligent filtering of metrics based on available dimensions and vice versa **Stage 4 - More custom user interface (UI) / Collaboration** * A place where users can see all the relevant information about a given metric * Organize metrics by hierarchy and more advanced search features (such as filter on the type of metric or other metadata) * Use and expose more metadata * Querying dimensions without metrics and other more advanced querying functionality * Suggest metrics to users based on teams/identity, and so on. ##### Related docs * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) * [Use the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) to learn about the product. * [Build your metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) for more info about MetricFlow and its components. * [Semantic Layer integrations page](https://www.getdbt.com/product/semantic-layer-integrations) for information about the available partner integrations. --- ### Leverage dbt to generate analytics and ML-ready pipelines with SQL and Python with Snowflake [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction The focus of this workshop will be to demonstrate how we can use both *SQL and python together* in the same workflow to run *both analytics and machine learning models* on dbt. All code in today’s workshop can be found on [GitHub](https://github.com/dbt-labs/python-snowpark-formula1/tree/python-formula1). ##### What you'll use during the lab * A [Snowflake account](https://trial.snowflake.com/) with ACCOUNTADMIN access * A [dbt account](https://www.getdbt.com/signup/) ##### What you'll learn * How to build scalable data transformation pipelines using dbt, and Snowflake using SQL and Python * How to leverage copying data into Snowflake from a public S3 bucket ##### What you need to know * Basic to intermediate SQL and python. * Basic understanding of dbt fundamentals. We recommend the [dbt Fundamentals course](https://learn.getdbt.com) if you're interested. * High level machine learning process (encoding, training, testing) * Simple ML algorithms — we will use logistic regression to keep the focus on the *workflow*, not algorithms! ##### What you'll build * A set of data analytics and prediction pipelines using Formula 1 data leveraging dbt and Snowflake, making use of best practices like data quality tests and code promotion between environments * We will create insights for: 1. Finding the lap time average and rolling average through the years (is it generally trending up or down)? 2. Which constructor has the fastest pit stops in 2021? 3. Predicting the position of each driver given using a decade of data (2010 - 2020) As inputs, we are going to leverage Formula 1 datasets hosted on a dbt Labs public S3 bucket. We will create a Snowflake Stage for our CSV files then use Snowflake’s `COPY INTO` function to copy the data in from our CSV files into tables. The Formula 1 is available on [Kaggle](https://www.kaggle.com/datasets/rohanrao/formula-1-world-championship-1950-2020). The data is originally compiled from the [Ergast Developer API](http://ergast.com/mrd/). Overall we are going to set up the environments, build scalable pipelines in dbt, establish data tests, and promote code to production. #### Configure Snowflake 1. Log in to your trial Snowflake account. You can [sign up for a Snowflake Trial Account using this form](https://signup.snowflake.com/) if you don’t have one. 2. Ensure that your account is set up using **AWS** in the **US East (N. Virginia)**. We will be copying the data from a public AWS S3 bucket hosted by dbt Labs in the us-east-1 region. By ensuring our Snowflake environment setup matches our bucket region, we avoid any multi-region data copy and retrieval latency issues. [![Snowflake trial](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/1-snowflake-trial-AWS-setup.png?v=2 "Snowflake trial")](#)Snowflake trial 3. After creating your account and verifying it from your sign-up email, Snowflake will direct you back to the UI called Snowsight. 4. When Snowsight first opens, your window should look like the following, with you logged in as the ACCOUNTADMIN with demo worksheets open: [![Snowflake trial demo worksheets](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/2-new-snowflake-account.png?v=2 "Snowflake trial demo worksheets")](#)Snowflake trial demo worksheets 5. Navigate to **Admin > Billing & Terms**. Click **Enable > Acknowledge & Continue** to enable Anaconda Python Packages to run in Snowflake. [![Anaconda terms](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/3-accept-anaconda-terms.jpeg?v=2 "Anaconda terms")](#)Anaconda terms [![Enable Anaconda](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/4-enable-anaconda.png?v=2 "Enable Anaconda")](#)Enable Anaconda 1. Log in to your [trial Snowflake account](https://app.snowflake.com). 2. In the Snowflake UI, click the Create icon **+** in the upper left (under the Snowflake logo) to open a dropdown. 3. Select the first option, **SQL File**. #### Connect to data source We need to obtain our data source by copying our Formula 1 data into Snowflake tables from a public S3 bucket that dbt Labs hosts. 1. When a new Snowflake account is created, there should be a preconfigured warehouse in your account named `COMPUTE_WH`. 2. If for any reason your account doesn’t have this warehouse, we can create a warehouse using the following script: ```sql create or replace warehouse COMPUTE_WH with warehouse_size=XSMALL ``` 3. Rename the worksheet to `data setup script` since we will be placing code in this worksheet to ingest the Formula 1 data. Make sure you are still logged in as the **ACCOUNTADMIN** and select the **COMPUTE\_WH** warehouse. [![Rename worksheet and select warehouse](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/1-rename-worksheet-and-select-warehouse.png?v=2 "Rename worksheet and select warehouse")](#)Rename worksheet and select warehouse 4. Copy the following code into the main body of the Snowflake worksheet. You can also find this setup script under the `setup` folder in the [Git repository](https://github.com/dbt-labs/python-snowpark-formula1/blob/main/setup/setup_script_s3_to_snowflake.sql). The script is long since it's bring in all of the data we'll need today! ```sql -- create and define our formula1 database create or replace database formula1; use database formula1; create or replace schema raw; use schema raw; -- define our file format for reading in the CSVs create or replace file format CSVformat type = CSV field_delimiter =',' field_optionally_enclosed_by = '"', skip_header=1; -- create or replace stage formula1_stage file_format = CSVformat url = 's3://formula1-dbt-cloud-python-demo/formula1-kaggle-data/'; -- load in the 8 tables we need for our demo -- we are first creating the table then copying our data in from s3 -- think of this as an empty container or shell that we are then filling create or replace table formula1.raw.circuits ( CIRCUITID NUMBER(38,0), CIRCUITREF VARCHAR(16777216), NAME VARCHAR(16777216), LOCATION VARCHAR(16777216), COUNTRY VARCHAR(16777216), LAT FLOAT, LNG FLOAT, ALT NUMBER(38,0), URL VARCHAR(16777216) ); -- copy our data from public s3 bucket into our tables copy into circuits from @formula1_stage/circuits.csv on_error='continue'; create or replace table formula1.raw.constructors ( CONSTRUCTORID NUMBER(38,0), CONSTRUCTORREF VARCHAR(16777216), NAME VARCHAR(16777216), NATIONALITY VARCHAR(16777216), URL VARCHAR(16777216) ); copy into constructors from @formula1_stage/constructors.csv on_error='continue'; create or replace table formula1.raw.drivers ( DRIVERID NUMBER(38,0), DRIVERREF VARCHAR(16777216), NUMBER VARCHAR(16777216), CODE VARCHAR(16777216), FORENAME VARCHAR(16777216), SURNAME VARCHAR(16777216), DOB DATE, NATIONALITY VARCHAR(16777216), URL VARCHAR(16777216) ); copy into drivers from @formula1_stage/drivers.csv on_error='continue'; create or replace table formula1.raw.lap_times ( RACEID NUMBER(38,0), DRIVERID NUMBER(38,0), LAP NUMBER(38,0), POSITION FLOAT, TIME VARCHAR(16777216), MILLISECONDS NUMBER(38,0) ); copy into lap_times from @formula1_stage/lap_times.csv on_error='continue'; create or replace table formula1.raw.pit_stops ( RACEID NUMBER(38,0), DRIVERID NUMBER(38,0), STOP NUMBER(38,0), LAP NUMBER(38,0), TIME VARCHAR(16777216), DURATION VARCHAR(16777216), MILLISECONDS NUMBER(38,0) ); copy into pit_stops from @formula1_stage/pit_stops.csv on_error='continue'; create or replace table formula1.raw.races ( RACEID NUMBER(38,0), YEAR NUMBER(38,0), ROUND NUMBER(38,0), CIRCUITID NUMBER(38,0), NAME VARCHAR(16777216), DATE DATE, TIME VARCHAR(16777216), URL VARCHAR(16777216), FP1_DATE VARCHAR(16777216), FP1_TIME VARCHAR(16777216), FP2_DATE VARCHAR(16777216), FP2_TIME VARCHAR(16777216), FP3_DATE VARCHAR(16777216), FP3_TIME VARCHAR(16777216), QUALI_DATE VARCHAR(16777216), QUALI_TIME VARCHAR(16777216), SPRINT_DATE VARCHAR(16777216), SPRINT_TIME VARCHAR(16777216) ); copy into races from @formula1_stage/races.csv on_error='continue'; create or replace table formula1.raw.results ( RESULTID NUMBER(38,0), RACEID NUMBER(38,0), DRIVERID NUMBER(38,0), CONSTRUCTORID NUMBER(38,0), NUMBER NUMBER(38,0), GRID NUMBER(38,0), POSITION FLOAT, POSITIONTEXT VARCHAR(16777216), POSITIONORDER NUMBER(38,0), POINTS NUMBER(38,0), LAPS NUMBER(38,0), TIME VARCHAR(16777216), MILLISECONDS NUMBER(38,0), FASTESTLAP NUMBER(38,0), RANK NUMBER(38,0), FASTESTLAPTIME VARCHAR(16777216), FASTESTLAPSPEED FLOAT, STATUSID NUMBER(38,0) ); copy into results from @formula1_stage/results.csv on_error='continue'; create or replace table formula1.raw.status ( STATUSID NUMBER(38,0), STATUS VARCHAR(16777216) ); copy into status from @formula1_stage/status.csv on_error='continue'; ``` 5. Ensure all the commands are selected before running the query — an easy way to do this is to use Ctrl-a to highlight all of the code in the worksheet. Select **run** (blue triangle icon). Notice how the dot next to your **COMPUTE\_WH** turns from gray to green as you run the query. The **status** table is the final table of all 8 tables loaded in. [![Load data from S3 bucket](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/2-load-data-from-s3.png?v=2 "Load data from S3 bucket")](#)Load data from S3 bucket 6. Let’s unpack that pretty long query we ran into component parts. We ran this query to load in our 8 Formula 1 tables from a public S3 bucket. To do this, we: * Created a new database called `formula1` and a schema called `raw` to place our raw (untransformed) data into. * Defined our file format for our CSV files. Importantly, here we use a parameter called `field_optionally_enclosed_by =` since the string columns in our Formula 1 CSV files use quotes. Quotes are used around string values to avoid parsing issues where commas `,` and new lines `/n` in data values could cause data loading errors. * Created a stage to locate our data we are going to load in. Snowflake Stages are locations where data files are stored. Stages are used to both load and unload data to and from Snowflake locations. Here we are using an external stage, by referencing an S3 bucket. * Created our tables for our data to be copied into. These are empty tables with the column name and data type. Think of this as creating an empty container that the data will then fill into. * Used the `copy into` statement for each of our tables. We reference our staged location we created and upon loading errors continue to load in the rest of the data. You should not have data loading errors but if you do, those rows will be skipped and Snowflake will tell you which rows caused errors 7. Now let's take a look at some of our cool Formula 1 data we just loaded up! 1. Create a new worksheet by selecting the **+** then **New Worksheet**. [![Create new worksheet to query data](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/3-create-new-worksheet-to-query-data.png?v=2 "Create new worksheet to query data")](#)Create new worksheet to query data 2. Navigate to **Database > Formula1 > RAW > Tables**. 3. Query the data using the following code. There are only 76 rows in the circuits table, so we don’t need to worry about limiting the amount of data we query. ```sql select * from formula1.raw.circuits ``` 4. Run the query. From here on out, we’ll use the keyboard shortcuts Command-Enter or Control-Enter to run queries and won’t explicitly call out this step. 5. Review the query results, you should see information about Formula 1 circuits, starting with Albert Park in Australia! 6. Finally, ensure you have all 8 tables starting with `CIRCUITS` and ending with `STATUS`. Now we are ready to connect into dbt! [![Query circuits data](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/4-query-circuits-data.png?v=2 "Query circuits data")](#)Query circuits data #### Configure dbt 1. We are going to be using [Snowflake Partner Connect](https://docs.snowflake.com/en/user-guide/ecosystem-partner-connect.html) to set up a dbt account. Using this method will allow you to spin up a fully fledged dbt account with your [Snowflake connection](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md), [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md), environments, and credentials already established. 2. Navigate out of your worksheet back by selecting **home**. 3. In Snowsight, confirm that you are using the **ACCOUNTADMIN** role. 4. Navigate to the **Data Products** **> Partner Connect**. Find **dbt** either by using the search bar or navigating the **Data Integration**. Select the **dbt** tile. [![Open Partner Connect](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/1-open-partner-connect.png?v=2 "Open Partner Connect")](#)Open Partner Connect 5. You should now see a new window that says **Connect to dbt**. Select **Optional Grant** and add the `FORMULA1` database. This will grant access for your new dbt user role to the FORMULA1 database. [![Partner Connect Optional Grant](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/2-partner-connect-optional-grant.png?v=2 "Partner Connect Optional Grant")](#)Partner Connect Optional Grant 6. Ensure the `FORMULA1` is present in your optional grant before clicking **Connect**.  This will create a dedicated dbt user, database, warehouse, and role for your dbt trial. [![Connect to dbt](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/3-connect-to-dbt.png?v=2 "Connect to dbt")](#)Connect to dbt 7. When you see the **Your partner account has been created** window, click **Activate**. 8. You should be redirected to a dbt registration page. Fill out the form. Make sure to save the password somewhere for login in the future. [![dbt sign up](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/4-dbt-cloud-sign-up.png?v=2 "dbt sign up")](#)dbt sign up 9. Select **Complete Registration**. You should now be redirected to your dbt account, complete with a connection to your Snowflake account, a deployment and a development environment, and a sample job. 10. To help you version control your dbt project, we have connected it to a [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md), which means that dbt Labs will be hosting your repository for you. This will give you access to a Git workflow without you having to create and host the repository yourself. You will not need to know Git for this workshop; dbt will help guide you through the workflow. In the future, when you’re developing your own project, [feel free to use your own repository](https://docs.getdbt.com/docs/platform/git/connect-github.md). This will allow you to learn more about features like [Slim CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md) builds after this workshop. #### Change development schema name and navigate the IDE 1. First we are going to change the name of our default schema to where our dbt models will build. By default, the name is `dbt_`. We will change this to `dbt_<YOUR_NAME>` to create your own personal development schema. To do this, click on your account name in the left side menu and select **Account settings**. [![Settings menu](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/1-settings-gear-icon.png?v=2 "Settings menu")](#)Settings menu 2. Navigate to the **Credentials** menu and select **Partner Connect Trial**, which will expand the credentials menu. [![Credentials edit schema name](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/2-credentials-edit-schema-name.png?v=2 "Credentials edit schema name")](#)Credentials edit schema name 3. Click **Edit** and change the name of your schema from `dbt_` to `dbt_YOUR_NAME` replacing `YOUR_NAME` with your initials and name. Be sure to click **Save** for your changes! [![Save new schema name](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/3-save-new-schema-name.png?v=2 "Save new schema name")](#)Save new schema name 4. We now have our own personal development schema, amazing! When we run our first dbt models they will build into this schema. 5. Let’s open up dbt’s Integrated Development Environment (Studio IDE) and familiarize ourselves. Choose **Develop** at the top of the UI. 6. When the Studio IDE is done loading, click **Initialize dbt project**. The initialization process creates a collection of files and folders necessary to run your dbt project. [![Initialize dbt project](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/4-initialize-dbt-project.png?v=2 "Initialize dbt project")](#)Initialize dbt project 7. After the initialization is finished, you can view the files and folders in the file tree menu. As we move through the workshop we'll be sure to touch on a few key files and folders that we'll work with to build out our project. 8. Next click **Commit and sync** to commit the new files and folders from the initialize step. We always want our commit messages to be relevant to the work we're committing, so be sure to provide a message like `initialize project` and select **Commit Changes**. [![First commit and push](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/5-first-commit-and-push.png?v=2 "First commit and push")](#)First commit and push [![Commit Changes button](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/6-initalize-project.png?v=2 "Commit Changes button")](#)Commit Changes button 9. [Committing](https://www.atlassian.com/git/tutorials/saving-changes/git-commit) your work here will save it to the managed git repository that was created during the Partner Connect signup. This initial commit is the only commit that will be made directly to our `main` branch and from *here on out we'll be doing all of our work on a development branch*. This allows us to keep our development work separate from our production code. 10. There are a couple of key features to point out about the Studio IDE before we get to work. It is a text editor, an SQL and Python runner, and a CLI with Git version control all baked into one package! This allows you to focus on editing your SQL and Python files, previewing the results with the SQL runner (it even runs Jinja!), and building models at the command line without having to move between different applications. The Git workflow in dbt allows both Git beginners and experts alike to be able to easily version control all of their work with a couple clicks. [![IDE overview](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/7-IDE-overview.png?v=2 "IDE overview")](#)IDE overview 11. Let's run our first dbt models! Two example models are included in your dbt project in the `models/examples` folder that we can use to illustrate how to run dbt at the command line. Type `dbt run` into the command line and click **Enter** on your keyboard. When the run bar expands you'll be able to see the results of the run, where you should see the run complete successfully. [![dbt run example models](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/8-dbt-run-example-models.png?v=2 "dbt run example models")](#)dbt run example models 12. The run results allow you to see the code that dbt compiles and sends to Snowflake for execution. To view the logs for this run, select one of the model tabs using the  **>** icon and then **Details**. If you scroll down a bit you'll be able to see the compiled code and how dbt interacts with Snowflake. Given that this run took place in our development environment, the models were created in your development schema. [![Details about the second model](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/9-second-model-details.png?v=2 "Details about the second model")](#)Details about the second model 13. Now let's switch over to Snowflake to confirm that the objects were actually created. Click on the three dots **…** above your database objects and then **Refresh**. Expand the **PC\_DBT\_DB** database and you should see your development schema. Select the schema, then **Tables**  and **Views**. Now you should be able to see `MY_FIRST_DBT_MODEL` as a table and `MY_SECOND_DBT_MODEL` as a view. [![Confirm example models are built in Snowflake](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/10-confirm-example-models-built-in-snowflake.png?v=2 "Confirm example models are built in Snowflake")](#)Confirm example models are built in Snowflake #### Create branch and set up project configs In this step, we’ll need to create a development branch and set up project level configurations. 1. To get started with development for our project, we'll need to create a new Git branch for our work. Select **create branch** and name your development branch. We'll call our branch `snowpark_python_workshop` then click **Submit**. 2. The first piece of development we'll do on the project is to update the `dbt_project.yml` file. Every dbt project requires a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. The [dbt\_project.yml](https://docs.getdbt.com/reference/dbt_project.yml.md) file also contains important information that tells dbt how to operate on your project. 3. Select the `dbt_project.yml` file from the file tree to open it and replace all of the existing contents with the following code below. When you're done, save the file by clicking **save**. You can also use the Command-S or Control-S shortcut from here on out. ```yaml # Name your project! Project names should contain only lowercase characters # and underscores. A good package name should reflect your organization's # name or the intended use of these models name: 'snowflake_dbt_python_formula1' version: '1.3.0' require-dbt-version: '>=1.3.0' config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: 'default' # These configurations specify where dbt should look for different types of files. # The `model-paths` config, for example, states that models in this project can be # found in the "models/" directory. You probably won't need to change these! model-paths: ["models"] analysis-paths: ["analyses"] test-paths: ["tests"] seed-paths: ["seeds"] macro-paths: ["macros"] snapshot-paths: ["snapshots"] target-path: "target" # directory which will store compiled SQL files clean-targets: # directories to be removed by `dbt clean` - "target" - "dbt_packages" models: snowflake_dbt_python_formula1: staging: +docs: node_color: "CadetBlue" marts: +materialized: table aggregates: +docs: node_color: "Maroon" +tags: "bi" core: +docs: node_color: "#800080" intermediate: +docs: node_color: "MediumSlateBlue" ml: prep: +docs: node_color: "Indigo" train_predict: +docs: node_color: "#36454f" ``` 4. The key configurations to point out in the file with relation to the work that we're going to do are in the `models` section. * `require-dbt-version` — Tells dbt which version of dbt to use for your project. We are requiring 1.3.0 and any newer version to run python models and node colors. * `materialized` — Tells dbt how to materialize models when compiling the code before it pushes it down to Snowflake. All models in the `marts` folder will be built as tables. * `tags` — Applies tags at a directory level to all models. All models in the `aggregates` folder will be tagged as `bi` (abbreviation for business intelligence). * `docs` — Specifies the `node_color` either by the plain color name or a hex value. 5. [Materializations](https://docs.getdbt.com/docs/build/materializations.md) are strategies for persisting dbt models in a warehouse, with `tables` and `views` being the most commonly utilized types. By default, all dbt models are materialized as views and other materialization types can be configured in the `dbt_project.yml` file or in a model itself. It’s very important to note *Python models can only be materialized as tables or incremental models.* Since all our Python models exist under `marts`, the following portion of our `dbt_project.yml` ensures no errors will occur when we run our Python models. Starting with [dbt version 1.4](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.4.md#updates-to-python-models>), Python files will automatically get materialized as tables even if not explicitly specified. ```yaml marts:     +materialized: table ``` #### Create folders and organize files dbt Labs has developed a [project structure guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) that contains a number of recommendations for how to build the folder structure for your project. Do check out that guide if you want to learn more. Right now we are going to create some folders to organize our files: * Sources — This is our Formula 1 dataset and it will be defined in a source properties YAML file. * Staging models — These models have a 1:1 with their source table. * Intermediate — This is where we will be joining some Formula staging models. * Marts models — Here is where we perform our major transformations. It contains these subfolders: * aggregates * core * ml 1. In your file tree, use your cursor and hover over the `models` subdirectory, click the three dots **…** that appear to the right of the folder name, then select **Create Folder**. We're going to add two new folders to the file path, `staging` and `formula1` (in that order) by typing `staging/formula1` into the file path. [![Create folder](/img/guides/dbt-ecosystem/dbt-python-snowpark/7-folder-structure/1-create-folder.png?v=2 "Create folder")](#)Create folder [![Set file path](/img/guides/dbt-ecosystem/dbt-python-snowpark/7-folder-structure/2-file-path.png?v=2 "Set file path")](#)Set file path * If you click into your `models` directory now, you should see the new `staging` folder nested within `models` and the `formula1` folder nested within `staging`. 2. Create two additional folders the same as the last step. Within the `models` subdirectory, create new directories `marts/core`. 3. We will need to create a few more folders and subfolders using the UI. After you create all the necessary folders, your folder tree should look like this when it's all done: [![File tree of new folders](/img/guides/dbt-ecosystem/dbt-python-snowpark/7-folder-structure/3-tree-of-new-folders.png?v=2 "File tree of new folders")](#)File tree of new folders Remember you can always reference the entire project in [GitHub](https://github.com/dbt-labs/python-snowpark-formula1/tree/python-formula1) to view the complete folder and file structure. #### Create source and staging models In this section, we are going to create our source and staging models. Sources allow us to create a dependency between our source database object and our staging models which will help us when we look at data lineage later. Also, if your source changes database or schema, you only have to update it in your `f1_sources.yml` file rather than updating all of the models it might be used in. Staging models are the base of our project, where we bring all the individual components we're going to use to build our more complex and useful models into the project. Since we want to focus on dbt and Python in this workshop, check out our [sources](https://docs.getdbt.com/docs/build/sources.md) and [staging](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) docs if you want to learn more (or take our [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) course which covers all of our core functionality). ##### 1. Create sources We're going to be using each of our 8 Formula 1 tables from our `formula1` database under the `raw`  schema for our transformations and we want to create those tables as sources in our project. 1. Create a new file called `f1_sources.yml` with the following file path: `models/staging/formula1/f1_sources.yml`. 2. Then, paste the following code into the file before saving it: ```yaml version: 2 sources: - name: formula1 description: formula 1 datasets with normalized tables database: formula1 schema: raw tables: - name: circuits description: One record per circuit, which is the specific race course. columns: - name: circuitid data_tests: - unique - not_null - name: constructors description: One record per constructor. Constructors are the teams that build their formula 1 cars. columns: - name: constructorid data_tests: - unique - not_null - name: drivers description: One record per driver. This table gives details about the driver. columns: - name: driverid data_tests: - unique - not_null - name: lap_times description: One row per lap in each race. Lap times started being recorded in this dataset in 1984 and joined through driver_id. - name: pit_stops description: One row per pit stop. Pit stops do not have their own id column, the combination of the race_id and driver_id identify the pit stop. columns: - name: stop data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: [1,2,3,4,5,6,7,8] quote: false - name: races description: One race per row. Importantly this table contains the race year to understand trends. columns: - name: raceid data_tests: - unique - not_null - name: results columns: - name: resultid data_tests: - unique - not_null description: One row per result. The main table that we join out for grid and position variables. - name: status description: One status per row. The status contextualizes whether the race was finished or what issues arose e.g. collisions, engine, etc. columns: - name: statusid data_tests: - unique - not_null ``` ##### 2. Create staging models The next step is to set up the staging models for each of the 8 source tables. Given the one-to-one relationship between staging models and their corresponding source tables, we'll build 8 staging models here. We know it’s a lot and in the future, we will seek to update the workshop to make this step less repetitive and more efficient. This step is also a good representation of the real world of data, where you have multiple hierarchical tables that you will need to join together! 1. Let's go in alphabetical order to easily keep track of all our staging models! Create a new file called `stg_f1_circuits.sql` with this file path `models/staging/formula1/stg_f1_circuits.sql`. Then, paste the following code into the file before saving it: ```sql with source as ( select * from {{ source('formula1','circuits') }} ), renamed as ( select circuitid as circuit_id, circuitref as circuit_ref, name as circuit_name, location, country, lat as latitude, lng as longitude, alt as altitude -- omit the url from source ) select * from renamed ``` All we're doing here is pulling the source data into the model using the `source` function, renaming some columns, and omitting the column `url` with a commented note since we don’t need it for our analysis. 2. Create `stg_f1_constructors.sql` with this file path `models/staging/formula1/stg_f1_constructors.sql`. Paste the following code into it before saving the file: ```sql with source as ( select * from {{ source('formula1','constructors') }} ), renamed as ( select constructorid as constructor_id, constructorref as constructor_ref, name as constructor_name, nationality as constructor_nationality -- omit the url from source ) select * from renamed ``` We have 6 other stages models to create. We can do this by creating new files, then copy and paste the code into our `staging` folder. 3. Create `stg_f1_drivers.sql` with this file path `models/staging/formula1/stg_f1_drivers.sql`: ```sql with source as ( select * from {{ source('formula1','drivers') }} ), renamed as ( select driverid as driver_id, driverref as driver_ref, number as driver_number, code as driver_code, forename, surname, dob as date_of_birth, nationality as driver_nationality -- omit the url from source ) select * from renamed ``` 4. Create `stg_f1_lap_times.sql` with this file path `models/staging/formula1/stg_f1_lap_times.sql`: ```sql with source as ( select * from {{ source('formula1','lap_times') }} ), renamed as ( select raceid as race_id, driverid as driver_id, lap, position, time as lap_time_formatted, milliseconds as lap_time_milliseconds from source ) select * from renamed ``` 5. Create `stg_f1_pit_stops.sql` with this file path `models/staging/formula1/stg_f1_pit_stops.sql`: ```sql with source as ( select * from {{ source('formula1','pit_stops') }} ), renamed as ( select raceid as race_id, driverid as driver_id, stop as stop_number, lap, time as lap_time_formatted, duration as pit_stop_duration_seconds, milliseconds as pit_stop_milliseconds from source ) select * from renamed order by pit_stop_duration_seconds desc ``` 6. Create `stg_f1_races.sql` with this file path `models/staging/formula1/stg_f1_races.sql`: ```sql with source as ( select * from {{ source('formula1','races') }} ), renamed as ( select raceid as race_id, year as race_year, round as race_round, circuitid as circuit_id, name as circuit_name, date as race_date, to_time(time) as race_time, -- omit the url fp1_date as free_practice_1_date, fp1_time as free_practice_1_time, fp2_date as free_practice_2_date, fp2_time as free_practice_2_time, fp3_date as free_practice_3_date, fp3_time as free_practice_3_time, quali_date as qualifying_date, quali_time as qualifying_time, sprint_date, sprint_time from source ) select * from renamed ``` 7. Create `stg_f1_results.sql` with this file path `models/staging/formula1/stg_f1_results.sql`: ```sql with source as ( select * from {{ source('formula1','results') }} ), renamed as ( select resultid as result_id, raceid as race_id, driverid as driver_id, constructorid as constructor_id, number as driver_number, grid, position::int as position, positiontext as position_text, positionorder as position_order, points, laps, time as results_time_formatted, milliseconds as results_milliseconds, fastestlap as fastest_lap, rank as results_rank, fastestlaptime as fastest_lap_time_formatted, fastestlapspeed::decimal(6,3) as fastest_lap_speed, statusid as status_id from source ) select * from renamed ``` 8. Last one! Create `stg_f1_status.sql` with this file path: `models/staging/formula1/stg_f1_status.sql`: ```sql with source as ( select * from {{ source('formula1','status') }} ), renamed as ( select statusid as status_id, status from source ) select * from renamed ``` After the source and all the staging models are complete for each of the 8 tables, your staging folder should look like this: [![Staging folder](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/1-staging-folder.png?v=2 "Staging folder")](#)Staging folder 9. It’s a good time to delete our example folder since these two models are extraneous to our formula1 pipeline and `my_first_model` fails a `not_null` test that we won’t spend time investigating. dbt will warn us that this folder will be permanently deleted, and we are okay with that so select **Delete**. [![Delete example folder](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/2-delete-example.png?v=2 "Delete example folder")](#)Delete example folder 10. Now that the staging models are built and saved, it's time to create the models in our development schema in Snowflake. To do this we're going to enter into the command line `dbt build` to run all of the models in our project, which includes the 8 new staging models and the existing example models. Your run should complete successfully and you should see green checkmarks next to all of your models in the run results. We built our 8 staging models as views and ran 13 source tests that we configured in the `f1_sources.yml` file with not that much code, pretty cool! [![Successful dbt build in Snowflake](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/3-successful-run-in-snowflake.png?v=2 "Successful dbt build in Snowflake")](#)Successful dbt build in Snowflake Let's take a quick look in Snowflake, refresh database objects, open our development schema, and confirm that the new models are there. If you can see them, then we're good to go! [![Confirm models](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/4-confirm-models.png?v=2 "Confirm models")](#)Confirm models Before we move onto the next section, be sure to commit your new models to your Git branch. Click **Commit and push** and give your commit a message like `profile, sources, and staging setup` before moving on. #### Transform SQL Now that we have all our sources and staging models done, it's time to move into where dbt shines — transformation! We need to: * Create some intermediate tables to join tables that aren’t hierarchical * Create core tables for business intelligence (BI) tool ingestion * Answer the two questions about: * fastest pit stops * lap time trends about our Formula 1 data by creating aggregate models using python! ##### Intermediate models We need to join lots of reference tables to our results table to create a human readable dataframe. What does this mean? For example, we don’t only want to have the numeric `status_id` in our table, we want to be able to read in a row of data that a driver could not finish a race due to engine failure (`status_id=5`). By now, we are pretty good at creating new files in the correct directories so we won’t cover this in detail. All intermediate models should be created in the path `models/intermediate`. 1. Create a new file called `int_lap_times_years.sql`. In this model, we are joining our lap time and race information so we can look at lap times over years. In earlier Formula 1 eras, lap times were not recorded (only final results), so we filter out records where lap times are null. ```sql with lap_times as ( select * from {{ ref('stg_f1_lap_times') }} ), races as ( select * from {{ ref('stg_f1_races') }} ), expanded_lap_times_by_year as ( select lap_times.race_id, driver_id, race_year, lap, lap_time_milliseconds from lap_times left join races on lap_times.race_id = races.race_id where lap_time_milliseconds is not null ) select * from expanded_lap_times_by_year ``` 2. Create a file called `in_pit_stops.sql`. Pit stops are a many-to-one (M:1) relationship with our races. We are creating a feature called `total_pit_stops_per_race` by partitioning over our `race_id` and `driver_id`, while preserving individual level pit stops for rolling average in our next section. ```sql with stg_f1__pit_stops as ( select * from {{ ref('stg_f1_pit_stops') }} ), pit_stops_per_race as ( select race_id, driver_id, stop_number, lap, lap_time_formatted, pit_stop_duration_seconds, pit_stop_milliseconds, max(stop_number) over (partition by race_id,driver_id) as total_pit_stops_per_race from stg_f1__pit_stops ) select * from pit_stops_per_race ``` 3. Create a file called `int_results.sql`. Here we are using 4 of our tables — `races`, `drivers`, `constructors`, and `status` — to give context to our `results` table. We are now able to calculate a new feature `drivers_age_years` by bringing the `date_of_birth` and `race_year` into the same table. We are also creating a column to indicate if the driver did not finish (dnf) the race, based upon if their `position` was null called, `dnf_flag`. ```sql with results as ( select * from {{ ref('stg_f1_results') }} ), races as ( select * from {{ ref('stg_f1_races') }} ), drivers as ( select * from {{ ref('stg_f1_drivers') }} ), constructors as ( select * from {{ ref('stg_f1_constructors') }} ), status as ( select * from {{ ref('stg_f1_status') }} ), int_results as ( select result_id, results.race_id, race_year, race_round, circuit_id, circuit_name, race_date, race_time, results.driver_id, results.driver_number, forename ||' '|| surname as driver, cast(datediff('year', date_of_birth, race_date) as int) as drivers_age_years, driver_nationality, results.constructor_id, constructor_name, constructor_nationality, grid, position, position_text, position_order, points, laps, results_time_formatted, results_milliseconds, fastest_lap, results_rank, fastest_lap_time_formatted, fastest_lap_speed, results.status_id, status, case when position is null then 1 else 0 end as dnf_flag from results left join races on results.race_id=races.race_id left join drivers on results.driver_id = drivers.driver_id left join constructors on results.constructor_id = constructors.constructor_id left join status on results.status_id = status.status_id ) select * from int_results ``` 4. Create a *Markdown* file `intermediate.md` that we will go over in depth in the Test and Documentation sections of the [Leverage dbt to generate analytics and ML-ready pipelines with SQL and Python with Snowflake](https://docs.getdbt.com/guides/dbt-python-snowpark.md) guide. ```markdown # the intent of this .md is to allow for multi-line long form explanations for our intermediate transformations # below are descriptions {% docs int_results %} In this query we want to join out other important information about the race results to have a human readable table about results, races, drivers, constructors, and status. We will have 4 left joins onto our results table. {% enddocs %} {% docs int_pit_stops %} There are many pit stops within one race, aka a M:1 relationship. We want to aggregate this so we can properly join pit stop information without creating a fanout. {% enddocs %} {% docs int_lap_times_years %} Lap times are done per lap. We need to join them out to the race year to understand yearly lap time trends. {% enddocs %} ``` 5. Create a *YAML* file `intermediate.yml` that we will go over in depth during the Test and Document sections of the [Leverage dbt to generate analytics and ML-ready pipelines with SQL and Python with Snowflake](https://docs.getdbt.com/guides/dbt-python-snowpark.md) guide. ```yaml version: 2 models: - name: int_results description: '{{ doc("int_results") }}' - name: int_pit_stops description: '{{ doc("int_pit_stops") }}' - name: int_lap_times_years description: '{{ doc("int_lap_times_years") }}' ``` That wraps up the intermediate models we need to create our core models! ##### dbt models 1. Create a file `fct_results.sql`. This is what I like to refer to as the “mega table” — a really large denormalized table with all our context added in at row level for human readability. Importantly, we have a table `circuits` that is linked through the table `races`. When we joined `races` to `results` in `int_results.sql` we allowed our tables to make the connection from `circuits` to `results` in `fct_results.sql`. We are only taking information about pit stops at the result level so our join would not cause a [fanout](https://community.looker.com/technical-tips-tricks-1021/what-is-a-fanout-23327). ```sql with int_results as ( select * from {{ ref('int_results') }} ), int_pit_stops as ( select race_id, driver_id, max(total_pit_stops_per_race) as total_pit_stops_per_race from {{ ref('int_pit_stops') }} group by 1,2 ), circuits as ( select * from {{ ref('stg_f1_circuits') }} ), base_results as ( select result_id, int_results.race_id, race_year, race_round, int_results.circuit_id, int_results.circuit_name, circuit_ref, location, country, latitude, longitude, altitude, total_pit_stops_per_race, race_date, race_time, int_results.driver_id, driver, driver_number, drivers_age_years, driver_nationality, constructor_id, constructor_name, constructor_nationality, grid, position, position_text, position_order, points, laps, results_time_formatted, results_milliseconds, fastest_lap, results_rank, fastest_lap_time_formatted, fastest_lap_speed, status_id, status, dnf_flag from int_results left join circuits on int_results.circuit_id=circuits.circuit_id left join int_pit_stops on int_results.driver_id=int_pit_stops.driver_id and int_results.race_id=int_pit_stops.race_id ) select * from base_results ``` 2. Create the file `pit_stops_joined.sql`. Our results and pit stops are at different levels of dimensionality (also called grain). Simply put, we have multiple pit stops per a result. Since we are interested in understanding information at the pit stop level with information about race year and constructor, we will create a new table `pit_stops_joined.sql` where each row is per pit stop. Our new table tees up our aggregation in Python. ```sql with base_results as ( select * from {{ ref('fct_results') }} ), pit_stops as ( select * from {{ ref('int_pit_stops') }} ), pit_stops_joined as ( select base_results.race_id, race_year, base_results.driver_id, constructor_id, constructor_name, stop_number, lap, lap_time_formatted, pit_stop_duration_seconds, pit_stop_milliseconds from base_results left join pit_stops on base_results.race_id=pit_stops.race_id and base_results.driver_id=pit_stops.driver_id ) select * from pit_stops_joined ``` 3. Enter in the command line and execute `dbt build` to build out our entire pipeline to up to this point. Don’t worry about “overriding” your previous models – dbt workflows are designed to be idempotent so we can run them again and expect the same results. 4. Let’s talk about our lineage so far. It’s looking good 😎. We’ve shown how SQL can be used to make data type, column name changes, and handle hierarchical joins really well; all while building out our automated lineage! [![The DAG](/img/guides/dbt-ecosystem/dbt-python-snowpark/9-sql-transformations/1-dag.png?v=2 "The DAG")](#)The DAG 5. Time to **Commit and push** our changes and give your commit a message like `intermediate and fact models` before moving on. #### Running dbt Python models Up until now, SQL has been driving the project (car pun intended) for data cleaning and hierarchical joining. Now it’s time for Python to take the wheel (car pun still intended) for the rest of our lab! For more information about running Python models on dbt, check out our [docs](https://docs.getdbt.com/docs/build/python-models.md). To learn more about dbt python works under the hood, check out [Snowpark for Python](https://docs.snowflake.com/en/developer-guide/snowpark/python/index.html), which makes running dbt Python models possible. There are quite a few differences between SQL and Python in terms of the dbt syntax and DDL, so we’ll be breaking our code and model runs down further for our python models. ##### Pit stop analysis First, we want to find out: which constructor had the fastest pit stops in 2021? (constructor is a Formula 1 team that builds or “constructs” the car). 1. Create a new file called `fastest_pit_stops_by_constructor.py` in our `aggregates` (this is the first time we are using the `.py` extension!). 2. Copy the following code into the file: ```python import numpy as np import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas","numpy"]) # get upstream data pit_stops_joined = dbt.ref("pit_stops_joined").to_pandas() # provide year so we do not hardcode dates year=2021 # describe the data pit_stops_joined["PIT_STOP_SECONDS"] = pit_stops_joined["PIT_STOP_MILLISECONDS"]/1000 fastest_pit_stops = pit_stops_joined[(pit_stops_joined["RACE_YEAR"]==year)].groupby(by="CONSTRUCTOR_NAME")["PIT_STOP_SECONDS"].describe().sort_values(by='mean') fastest_pit_stops.reset_index(inplace=True) fastest_pit_stops.columns = fastest_pit_stops.columns.str.upper() return fastest_pit_stops.round(2) ``` 3. Let’s break down what this code is doing step by step: * First, we are importing the Python libraries that we are using. A *library* is a reusable chunk of code that someone else wrote that you may want to include in your programs/projects. We are using `numpy` and `pandas`in this Python model. This is similar to a dbt *package*, but our Python libraries do *not* persist across the entire project. * Defining a function called `model` with the parameter `dbt` and `session`. The parameter `dbt` is a class compiled by dbt, which enables you to run your Python code in the context of your dbt project and DAG. The parameter `session` is a class representing your Snowflake’s connection to the Python backend. The `model` function *must return a single DataFrame*. You can see that all the data transformation happening is within the body of the `model` function that the `return` statement is tied to. * Then, within the context of our dbt model library, we are passing in a configuration of which packages we need using `dbt.config(packages=["pandas","numpy"])`. * Use the `.ref()` function to retrieve the data frame `pit_stops_joined` that we created in our last step using SQL. We cast this to a pandas dataframe (by default it's a Snowpark Dataframe). * Create a variable named `year` so we aren’t passing a hardcoded value. * Generate a new column called `PIT_STOP_SECONDS` by dividing the value of `PIT_STOP_MILLISECONDS` by 1000. * Create our final data frame `fastest_pit_stops` that holds the records where year is equal to our year variable (2021 in this case), then group the data frame by `CONSTRUCTOR_NAME` and use the `describe()` and `sort_values()` and in descending order. This will make our first row in the new aggregated data frame the team with the fastest pit stops over an entire competition year. * Finally, it resets the index of the `fastest_pit_stops` data frame. The `reset_index()` method allows you to reset the index back to the default 0, 1, 2, etc indexes. By default, this method will keep the "old" indexes in a column named "index"; to avoid this, use the drop parameter. Think of this as keeping your data “flat and square” as opposed to “tiered”. If you are new to Python, now might be a good time to [learn about indexes for 5 minutes](https://towardsdatascience.com/the-basics-of-indexing-and-slicing-python-lists-2d12c90a94cf) since it's the foundation of how Python retrieves, slices, and dices data. The `inplace` argument means we override the existing data frame permanently. Not to fear! This is what we want to do to avoid dealing with multi-indexed dataframes! * Convert our Python column names to all uppercase using `.upper()`, so Snowflake recognizes them. * Finally we are returning our dataframe with 2 decimal places for all the columns using the `round()` method. 4. Zooming out a bit, what are we doing differently here in Python from our typical SQL code: * Method chaining is a technique in which multiple methods are called on an object in a single statement, with each method call modifying the result of the previous one. The methods are called in a chain, with the output of one method being used as the input for the next one. The technique is used to simplify the code and make it more readable by eliminating the need for intermediate variables to store the intermediate results. * The way you see method chaining in Python is the syntax `.().()`. For example, `.describe().sort_values(by='mean')` where the `.describe()` method is chained to `.sort_values()`. * The `.describe()` method is used to generate various summary statistics of the dataset. It's used on pandas dataframe. It gives a quick and easy way to get the summary statistics of your dataset without writing multiple lines of code. * The `.sort_values()` method is used to sort a pandas dataframe or a series by one or multiple columns. The method sorts the data by the specified column(s) in ascending or descending order. It is the pandas equivalent to `order by` in SQL. We won’t go as in depth for our subsequent scripts, but will continue to explain at a high level what new libraries, functions, and methods are doing. 5. Build the model using the UI which will **execute**: ```bash dbt run --select fastest_pit_stops_by_constructor ``` in the command bar. Let’s look at some details of our first Python model to see what our model executed. There two major differences we can see while running a Python model compared to an SQL model: * Our Python model was executed as a stored procedure. Snowflake needs a way to know that it's meant to execute this code in a Python runtime, instead of interpreting in a SQL runtime. We do this by creating a Python stored proc, called by a SQL command. * The `snowflake-snowpark-python` library has been picked up to execute our Python code. Even though this wasn’t explicitly stated this is picked up by the dbt class object because we need our Snowpark package to run Python! Python models take a bit longer to run than SQL models, however we could always speed this up by using [Snowpark-optimized Warehouses](https://docs.snowflake.com/en/user-guide/warehouses-snowpark-optimized.html) if we wanted to. Our data is sufficiently small, so we won’t worry about creating a separate warehouse for Python versus SQL files today. [![We can see our python model is run a stored procedure in our personal development schema](/img/guides/dbt-ecosystem/dbt-python-snowpark/10-python-transformations/1-python-model-details-output.png?v=2 "We can see our python model is run a stored procedure in our personal development schema")](#)We can see our python model is run a stored procedure in our personal development schema The rest of our **Details** output gives us information about how dbt and Snowpark for Python are working together to define class objects and apply a specific set of methods to run our models. So which constructor had the fastest pit stops in 2021? Let’s look at our data to find out! 6. We can't preview Python models directly, so let’s create a new file using the **+** button or the Control-n shortcut to create a new scratchpad. 7. Reference our Python model: ```sql select * from {{ ref('fastest_pit_stops_by_constructor') }} ``` and preview the output: [![Looking at our new python data model we can see that Red Bull had the fastest pit stops!](/img/guides/dbt-ecosystem/dbt-python-snowpark/10-python-transformations/2-fastest-pit-stops-preview.png?v=2 "Looking at our new python data model we can see that Red Bull had the fastest pit stops!")](#)Looking at our new python data model we can see that Red Bull had the fastest pit stops! Not only did Red Bull have the fastest average pit stops by nearly 40 seconds, they also had the smallest standard deviation, meaning they are both fastest and most consistent teams in pit stops. By using the `.describe()` method we were able to avoid verbose SQL requiring us to create a line of code per column and repetitively use the `PERCENTILE_COUNT()` function. Now we want to find the lap time average and rolling average through the years (is it generally trending up or down)? 8. Create a new file called `lap_times_moving_avg.py` in our `aggregates` folder. 9. Copy the following code into the file: ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"]) # get upstream data lap_times = dbt.ref("int_lap_times_years").to_pandas() # describe the data lap_times["LAP_TIME_SECONDS"] = lap_times["LAP_TIME_MILLISECONDS"]/1000 lap_time_trends = lap_times.groupby(by="RACE_YEAR")["LAP_TIME_SECONDS"].mean().to_frame() lap_time_trends.reset_index(inplace=True) lap_time_trends["LAP_MOVING_AVG_5_YEARS"] = lap_time_trends["LAP_TIME_SECONDS"].rolling(5).mean() lap_time_trends.columns = lap_time_trends.columns.str.upper() return lap_time_trends.round(1) ``` 10. Breaking down our code a bit: * We’re only using the `pandas` library for this model and casting it to a pandas data frame `.to_pandas()`. * Generate a new column called `LAP_TIMES_SECONDS` by dividing the value of `LAP_TIME_MILLISECONDS` by 1000. * Create the final dataframe. Get the lap time per year. Calculate the mean series and convert to a data frame. * Reset the index. * Calculate the rolling 5 year mean. * Round our numeric columns to one decimal place. 11. Now, run this model by using the UI **Run model** or ```bash dbt run --select lap_times_moving_avg ``` in the command bar. 12. Once again previewing the output of our data using the same steps for our `fastest_pit_stops_by_constructor` model. [![Viewing our lap trends and 5 year rolling trends](/img/guides/dbt-ecosystem/dbt-python-snowpark/10-python-transformations/3-lap-times-trends-preview.png?v=2 "Viewing our lap trends and 5 year rolling trends")](#)Viewing our lap trends and 5 year rolling trends We can see that it looks like lap times are getting consistently faster over time. Then in 2010 we see an increase occur! Using outside subject matter context, we know that significant rule changes were introduced to Formula 1 in 2010 and 2011 causing slower lap times. 13. Now is a good time to checkpoint and commit our work to Git. Click **Commit and push** and give your commit a message like `aggregate python models` before moving on. ##### The dbt model, .source(), .ref() and .config() functions Let’s take a step back before starting machine learning to both review and go more in-depth at the methods that make running dbt python models possible. If you want to know more outside of this lab’s explanation read the documentation [here](https://docs.getdbt.com/docs/build/python-models.md?version=1). * dbt model(dbt, session). For starters, each Python model lives in a .py file in your models/ folder. It defines a function named `model()`, which takes two parameters: * dbt — A class compiled by dbt, unique to each model, enables you to run your Python code in the context of your dbt project and DAG. * session — A class representing your data platform’s connection to the Python backend. The session is needed to read in tables as DataFrames and to write DataFrames back to tables. In PySpark, by convention, the SparkSession is named spark, and available globally. For consistency across platforms, we always pass it into the model function as an explicit argument called session. * The `model()` function must return a single DataFrame. On Snowpark (Snowflake), this can be a Snowpark or pandas DataFrame. * `.source()` and `.ref()` functions. Python models participate fully in dbt's directed acyclic graph (DAG) of transformations. If you want to read directly from a raw source table, use `dbt.source()`. We saw this in our earlier section using SQL with the source function. These functions have the same execution, but with different syntax. Use the `dbt.ref()` method within a Python model to read data from other models (SQL or Python). These methods return DataFrames pointing to the upstream source, model, seed, or snapshot. * `.config()`. Just like SQL models, there are three ways to configure Python models: * In a dedicated `.yml` file, within the `models/` directory * Within the model's `.py` file, using the `dbt.config()` method * Calling the `dbt.config()` method will set configurations for your model within your `.py` file, similar to the `{{ config() }} macro` in `.sql` model files: ```python def model(dbt, session): # setting configuration dbt.config(materialized="table") ``` * There's a limit to how complex you can get with the `dbt.config()` method. It accepts only literal values (strings, booleans, and numeric types). Passing another function or a more complex data structure is not possible. The reason is that dbt statically analyzes the arguments to `.config()` while parsing your model without executing your Python code. If you need to set a more complex configuration, we recommend you define it using the config property in a [properties YAML file](https://docs.getdbt.com/reference/resource-properties/config.md). Learn more about configurations [here](https://docs.getdbt.com/reference/model-configs.md). #### Prepare for machine learning: cleaning, encoding, and splits Now that we’ve gained insights and business intelligence about Formula 1 at a descriptive level, we want to extend our capabilities into prediction. We’re going to take the scenario where we censor the data. This means that we will pretend that we will train a model using earlier data and apply it to future data. In practice, this means we’ll take data from 2010-2019 to train our model and then predict 2020 data. In this section, we’ll be preparing our data to predict the final race position of a driver. At a high level we’ll be: * Creating new prediction features and filtering our dataset to active drivers * Encoding our data (algorithms like numbers) and simplifying our target variable called `position` * Splitting our dataset into training, testing, and validation ##### ML data prep 1. To keep our project organized, we’ll need to create two new subfolders in our `ml` directory. Under the `ml` folder, make the subfolders `prep` and `train_predict`. 2. Create a new file under `ml/prep` called `ml_data_prep.py`. Copy the following code into the file and **Save**. ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"]) # get upstream data fct_results = dbt.ref("fct_results").to_pandas() # provide years so we do not hardcode dates in filter command start_year=2010 end_year=2020 # describe the data for a full decade data = fct_results.loc[fct_results['RACE_YEAR'].between(start_year, end_year)] # convert string to an integer data['POSITION'] = data['POSITION'].astype(float) # we cannot have nulls if we want to use total pit stops data['TOTAL_PIT_STOPS_PER_RACE'] = data['TOTAL_PIT_STOPS_PER_RACE'].fillna(0) # some of the constructors changed their name over the year so replacing old names with current name mapping = {'Force India': 'Racing Point', 'Sauber': 'Alfa Romeo', 'Lotus F1': 'Renault', 'Toro Rosso': 'AlphaTauri'} data['CONSTRUCTOR_NAME'].replace(mapping, inplace=True) # create confidence metrics for drivers and constructors dnf_by_driver = data.groupby('DRIVER').sum(numeric_only=True)['DNF_FLAG'] driver_race_entered = data.groupby('DRIVER').count()['DNF_FLAG'] driver_dnf_ratio = (dnf_by_driver/driver_race_entered) driver_confidence = 1-driver_dnf_ratio driver_confidence_dict = dict(zip(driver_confidence.index,driver_confidence)) dnf_by_constructor = data.groupby('CONSTRUCTOR_NAME').sum(numeric_only=True)['DNF_FLAG'] constructor_race_entered = data.groupby('CONSTRUCTOR_NAME').count()['DNF_FLAG'] constructor_dnf_ratio = (dnf_by_constructor/constructor_race_entered) constructor_relaiblity = 1-constructor_dnf_ratio constructor_relaiblity_dict = dict(zip(constructor_relaiblity.index,constructor_relaiblity)) data['DRIVER_CONFIDENCE'] = data['DRIVER'].apply(lambda x:driver_confidence_dict[x]) data['CONSTRUCTOR_RELAIBLITY'] = data['CONSTRUCTOR_NAME'].apply(lambda x:constructor_relaiblity_dict[x]) #removing retired drivers and constructors active_constructors = ['Renault', 'Williams', 'McLaren', 'Ferrari', 'Mercedes', 'AlphaTauri', 'Racing Point', 'Alfa Romeo', 'Red Bull', 'Haas F1 Team'] active_drivers = ['Daniel Ricciardo', 'Kevin Magnussen', 'Carlos Sainz', 'Valtteri Bottas', 'Lance Stroll', 'George Russell', 'Lando Norris', 'Sebastian Vettel', 'Kimi Räikkönen', 'Charles Leclerc', 'Lewis Hamilton', 'Daniil Kvyat', 'Max Verstappen', 'Pierre Gasly', 'Alexander Albon', 'Sergio Pérez', 'Esteban Ocon', 'Antonio Giovinazzi', 'Romain Grosjean','Nicholas Latifi'] # create flags for active drivers and constructors so we can filter downstream data['ACTIVE_DRIVER'] = data['DRIVER'].apply(lambda x: int(x in active_drivers)) data['ACTIVE_CONSTRUCTOR'] = data['CONSTRUCTOR_NAME'].apply(lambda x: int(x in active_constructors)) return data ``` 3. As usual, let’s break down what we are doing in this Python model: * We’re first referencing our upstream `fct_results` table and casting it to a pandas dataframe. * Filtering on years 2010-2020 since we’ll need to clean all our data we are using for prediction (both training and testing). * Filling in empty data for `total_pit_stops` and making a mapping active constructors and drivers to avoid erroneous predictions * ⚠️ You might be wondering why we didn’t do this upstream in our `fct_results` table! The reason for this is that we want our machine learning cleanup to reflect the year 2020 for our predictions and give us an up-to-date team name. However, for business intelligence purposes we can keep the historical data at that point in time. Instead of thinking of one table as “one source of truth” we are creating different datasets fit for purpose: one for historical descriptions and reporting and another for relevant predictions. * Create new confidence features for drivers and constructors * Generate flags for the constructors and drivers that were active in 2020 4. Execute the following in the command bar: ```bash dbt run --select ml_data_prep ``` 5. There are more aspects we could consider for this project, such as normalizing the driver confidence by the number of races entered. Including this would help account for a driver’s history and consider whether they are a new or long-time driver. We’re going to keep it simple for now, but these are some of the ways we can expand and improve our machine learning dbt projects. Breaking down our machine learning prep model: * Lambda functions — We use some lambda functions to transform our data without having to create a fully-fledged function using the `def` notation. So what exactly are lambda functions? * In Python, a lambda function is a small, anonymous function defined using the keyword "lambda". Lambda functions are used to perform a quick operation, such as a mathematical calculation or a transformation on a list of elements. They are often used in conjunction with higher-order functions, such as `apply`, `map`, `filter`, and `reduce`. * `.apply()` method — We used `.apply()` to pass our functions into our lambda expressions to the columns and perform this multiple times in our code. Let’s explain apply a little more: * The `.apply()` function in the pandas library is used to apply a function to a specified axis of a DataFrame or a Series. In our case the function we used was our lambda function! * The `.apply()` function takes two arguments: the first is the function to be applied, and the second is the axis along which the function should be applied. The axis can be specified as 0 for rows or 1 for columns. We are using the default value of 0 so we aren’t explicitly writing it in the code. This means that the function will be applied to each *row* of the DataFrame or Series. 6. Let’s look at the preview of our clean dataframe after running our `ml_data_prep` model: [![What our clean dataframe fit for machine learning looks like](/img/guides/dbt-ecosystem/dbt-python-snowpark/11-machine-learning-prep/1-completed-ml-data-prep.png?v=2 "What our clean dataframe fit for machine learning looks like")](#)What our clean dataframe fit for machine learning looks like ##### Covariate encoding In this next part, we’ll be performing covariate encoding. Breaking down this phrase a bit, a *covariate* is a variable that is relevant to the outcome of a study or experiment, and *encoding* refers to the process of converting data (such as text or categorical variables) into a numerical format that can be used as input for a model. This is necessary because most machine learning algorithms can only work with numerical data. Algorithms don’t speak languages, have eyes to see images, etc. so we encode our data into numbers so algorithms can perform tasks by using calculations they otherwise couldn’t. 🧠 We’ll think about this as : “algorithms like numbers”. 1. Create a new file under `ml/prep` called `covariate_encoding` copy the code below and save. ```python import pandas as pd import numpy as np from sklearn.preprocessing import StandardScaler,LabelEncoder,OneHotEncoder from sklearn.linear_model import LogisticRegression def model(dbt, session): # dbt configuration dbt.config(packages=["pandas","numpy","scikit-learn"]) # get upstream data data = dbt.ref("ml_data_prep").to_pandas() # list out covariates we want to use in addition to outcome variable we are modeling - position covariates = data[['RACE_YEAR','CIRCUIT_NAME','GRID','CONSTRUCTOR_NAME','DRIVER','DRIVERS_AGE_YEARS','DRIVER_CONFIDENCE','CONSTRUCTOR_RELAIBLITY','TOTAL_PIT_STOPS_PER_RACE','ACTIVE_DRIVER','ACTIVE_CONSTRUCTOR', 'POSITION']] # filter covariates on active drivers and constructors # use fil_cov as short for "filtered_covariates" fil_cov = covariates[(covariates['ACTIVE_DRIVER']==1)&(covariates['ACTIVE_CONSTRUCTOR']==1)] # Encode categorical variables using LabelEncoder # TODO: we'll update this to both ohe in the future for non-ordinal variables! le = LabelEncoder() fil_cov['CIRCUIT_NAME'] = le.fit_transform(fil_cov['CIRCUIT_NAME']) fil_cov['CONSTRUCTOR_NAME'] = le.fit_transform(fil_cov['CONSTRUCTOR_NAME']) fil_cov['DRIVER'] = le.fit_transform(fil_cov['DRIVER']) fil_cov['TOTAL_PIT_STOPS_PER_RACE'] = le.fit_transform(fil_cov['TOTAL_PIT_STOPS_PER_RACE']) # Simply target variable "position" to represent 3 meaningful categories in Formula1 # 1. Podium position 2. Points for team 3. Nothing - no podium or points! def position_index(x): if x<4: return 1 if x>10: return 3 else : return 2 # we are dropping the columns that we filtered on in addition to our training variable encoded_data = fil_cov.drop(['ACTIVE_DRIVER','ACTIVE_CONSTRUCTOR'],axis=1)) encoded_data['POSITION_LABEL']= encoded_data['POSITION'].apply(lambda x: position_index(x)) encoded_data_grouped_target = encoded_data.drop(['POSITION'],axis=1)) return encoded_data_grouped_target ``` 2. Execute the following in the command bar: ```bash dbt run --select covariate_encoding ``` 3. In this code, we are using a ton of functions from libraries! This is really cool, because we can utilize code other people have developed and bring it into our project simply by using the `import` function. [Scikit-learn](https://scikit-learn.org/stable/), “sklearn” for short, is an extremely popular data science library. Sklearn contains a wide range of machine learning techniques, including supervised and unsupervised learning algorithms, feature scaling and imputation, as well as tools model evaluation and selection. We’ll be using Sklearn for both preparing our covariates and creating models (our next section). 4. Our dataset is pretty small data so we are good to use pandas and `sklearn`. If you have larger data for your own project in mind, consider `dask` or `category_encoders`. 5. Breaking it down a bit more: * We’re selecting a subset of variables that will be used as predictors for a driver’s position. * Filter the dataset to only include rows using the active driver and constructor flags we created in the last step. * The next step is to use the `LabelEncoder` from scikit-learn to convert the categorical variables `CIRCUIT_NAME`, `CONSTRUCTOR_NAME`, `DRIVER`, and `TOTAL_PIT_STOPS_PER_RACE` into numerical values. * Create a new variable called `POSITION_LABEL`, which is a derived from our position variable. * 💭 Why are we changing our position variable? There are 20 total positions in Formula 1 and we are grouping them together to simplify the classification and improve performance. We also want to demonstrate you can create a new function within your dbt model! * Our new `position_label` variable has meaning: * In Formula1 if you are in: * Top 3 you get a “podium” position * Top 10 you gain points that add to your overall season total * Below top 10 you get no points! * We are mapping our original variable position to `position_label` to the corresponding places above to 1,2, and 3 respectively. * Drop the active driver and constructor flags since they were filter criteria and additionally drop our original position variable. ##### Splitting into training and testing datasets Now that we’ve cleaned and encoded our data, we are going to further split in by time. In this step, we will create dataframes to use for training and prediction. We’ll be creating two dataframes 1) using data from 2010-2019 for training, and 2) data from 2020 for new prediction inferences. We’ll create variables called `start_year` and `end_year` so we aren’t filtering on hardcasted values (and can more easily swap them out in the future if we want to retrain our model on different timeframes). 1. Create a file called `train_test_dataset.py` copy and save the following code: ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"], tags="train") # get upstream data encoding = dbt.ref("covariate_encoding").to_pandas() # provide years so we do not hardcode dates in filter command start_year=2010 end_year=2019 # describe the data for a full decade train_test_dataset = encoding.loc[encoding['RACE_YEAR'].between(start_year, end_year)] return train_test_dataset ``` 2. Create a file called `hold_out_dataset_for_prediction.py` copy and save the following code below. Now we’ll have a dataset with only the year 2020 that we’ll keep as a hold out set that we are going to use similar to a deployment use case. ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"], tags="predict") # get upstream data encoding = dbt.ref("covariate_encoding").to_pandas() # variable for year instead of hardcoding it year=2020 # filter the data based on the specified year hold_out_dataset = encoding.loc[encoding['RACE_YEAR'] == year] return hold_out_dataset ``` 3. Execute the following in the command bar: ```bash dbt run --select train_test_dataset hold_out_dataset_for_prediction ``` To run our temporal data split models, we can use this syntax in the command line to run them both at once. Make sure you use a *space* [syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) between the model names to indicate you want to run both! 4. **Commit and push** our changes to keep saving our work as we go using `ml data prep and splits` before moving on. 👏 Now that we’ve finished our machine learning prep work we can move onto the fun part — training and prediction! #### Training a model to predict in machine learning We’re ready to start training a model to predict the driver’s position. Now is a good time to pause and take a step back and say, usually in ML projects you’ll try multiple algorithms during development and use an evaluation method such as cross validation to determine which algorithm to use. You can definitely do this in your dbt project, but for the content of this lab we’ll have decided on using a logistic regression to predict position (we actually tried some other algorithms using cross validation outside of this lab such as k-nearest neighbors and a support vector classifier but that didn’t perform as well as the logistic regression and a decision tree that overfit). There are 3 areas to break down as we go since we are working at the intersection all within one model file: 1. Machine Learning 2. Snowflake and Snowpark 3. dbt Python models If you haven’t seen code like this before or use joblib files to save machine learning models, we’ll be going over them at a high level and you can explore the links for more technical in-depth along the way! Because Snowflake and dbt have abstracted away a lot of the nitty gritty about serialization and storing our model object to be called again, we won’t go into too much detail here. There’s *a lot* going on here so take it at your pace! ##### Training and saving a machine learning model 1. Project organization remains key, so let’s make a new subfolder called `train_predict` under the `ml` folder. 2. Now create a new file called `train_test_position.py` and copy and save the following code: ```python import snowflake.snowpark.functions as F from sklearn.model_selection import train_test_split import pandas as pd from sklearn.metrics import confusion_matrix, balanced_accuracy_score import io from sklearn.linear_model import LogisticRegression from joblib import dump, load import joblib import logging import sys from joblib import dump, load logger = logging.getLogger("mylog") def save_file(session, model, path, dest_filename): input_stream = io.BytesIO() joblib.dump(model, input_stream) session._conn.upload_stream(input_stream, path, dest_filename) return "successfully created file: " + path def model(dbt, session): dbt.config( packages = ['numpy','scikit-learn','pandas','numpy','joblib','cachetools'], materialized = "table", tags = "train" ) # Create a stage in Snowflake to save our model file session.sql('create or replace stage MODELSTAGE').collect() #session._use_scoped_temp_objects = False version = "1.0" logger.info('Model training version: ' + version) # read in our training and testing upstream dataset test_train_df = dbt.ref("train_test_dataset") # cast snowpark df to pandas df test_train_pd_df = test_train_df.to_pandas() target_col = "POSITION_LABEL" # split out covariate predictors, x, from our target column position_label, y. split_X = test_train_pd_df.drop([target_col], axis=1) split_y = test_train_pd_df[target_col] # Split out our training and test data into proportions X_train, X_test, y_train, y_test = train_test_split(split_X, split_y, train_size=0.7, random_state=42) train = [X_train, y_train] test = [X_test, y_test] # now we are only training our one model to deploy # we are keeping the focus on the workflows and not algorithms for this lab! model = LogisticRegression() # fit the preprocessing pipeline and the model together model.fit(X_train, y_train) y_pred = model.predict_proba(X_test)[:,1] predictions = [round(value) for value in y_pred] balanced_accuracy = balanced_accuracy_score(y_test, predictions) # Save the model to a stage save_file(session, model, "@MODELSTAGE/driver_position_"+version, "driver_position_"+version+".joblib" ) logger.info('Model artifact:' + "@MODELSTAGE/driver_position_"+version+".joblib") # Take our pandas training and testing dataframes and put them back into snowpark dataframes snowpark_train_df = session.write_pandas(pd.concat(train, axis=1, join='inner'), "train_table", auto_create_table=True, create_temp_table=True) snowpark_test_df = session.write_pandas(pd.concat(test, axis=1, join='inner'), "test_table", auto_create_table=True, create_temp_table=True) # Union our training and testing data together and add a column indicating train vs test rows return snowpark_train_df.with_column("DATASET_TYPE", F.lit("train")).union(snowpark_test_df.with_column("DATASET_TYPE", F.lit("test"))) ``` 3. Execute the following in the command bar: ```bash dbt run --select train_test_position ``` 4. Breaking down our Python script here: * We’re importing some helpful libraries. * Defining a function called `save_file()` that takes four parameters: `session`, `model`, `path` and `dest_filename` that will save our logistic regression model file. * `session` — an object representing a connection to Snowflake. * `model` — an object that needs to be saved. In this case, it's a Python object that is a scikit-learn that can be serialized with joblib. * `path` — a string representing the directory or bucket location where the file should be saved. * `dest_filename` — a string representing the desired name of the file. * Creating our dbt model * Within this model we are creating a stage called `MODELSTAGE` to place our logistic regression `joblib` model file. This is really important since we need a place to keep our model to reuse and want to ensure it's there. When using Snowpark commands, it's common to see the `.collect()` method to ensure the action is performed. Think of the session as our “start” and collect as our “end” when [working with Snowpark](https://docs.snowflake.com/en/developer-guide/snowpark/python/working-with-dataframes.html) (you can use other ending methods other than collect). * Using `.ref()` to connect into our `train_test_dataset` model. * Now we see the machine learning part of our analysis: * Create new dataframes for our prediction features from our target variable `position_label`. * Split our dataset into 70% training (and 30% testing), train\_size=0.7 with a `random_state` specified to have repeatable results. * Specify our model is a logistic regression. * Fit our model. In a logistic regression this means finding the coefficients that will give the least classification error. * Round our predictions to the nearest integer since logistic regression creates a probability between for each class and calculate a balanced accuracy to account for imbalances in the target variable. * Right now our model is only in memory, so we need to use our nifty function `save_file` to save our model file to our Snowflake stage. We save our model as a joblib file so Snowpark can easily call this model object back to create predictions. We really don’t need to know much else as a data practitioner unless we want to. It’s worth noting that joblib files aren’t able to be queried directly by SQL. To do this, we would need to transform the joblib file to an SQL queryable format such as JSON or CSV (out of scope for this workshop). * Finally we want to return our dataframe, but create a new column indicating what rows were used for training and those for training. 5. Viewing our output of this model: [![Preview which rows of our model were used for training and testing](/img/guides/dbt-ecosystem/dbt-python-snowpark/12-machine-learning-training-prediction/1-preview-train-test-position.png?v=2 "Preview which rows of our model were used for training and testing")](#)Preview which rows of our model were used for training and testing 6. Let’s pop back over to Snowflake and check that our logistic regression model has been stored in our `MODELSTAGE` using the command: ```sql list @modelstage ``` [![List the objects in our Snowflake stage to check for our logistic regression to predict driver position](/img/guides/dbt-ecosystem/dbt-python-snowpark/12-machine-learning-training-prediction/2-list-snowflake-stage.png?v=2 "List the objects in our Snowflake stage to check for our logistic regression to predict driver position")](#)List the objects in our Snowflake stage to check for our logistic regression to predict driver position 7. To investigate the commands run as part of `train_test_position` script, navigate to Snowflake query history to view it **Activity > Query History**. We can view the portions of query that we wrote such as `create or replace stage MODELSTAGE`, but we also see additional queries that Snowflake uses to interpret python code. [![View Snowflake query history to see how python models are run under the hood](/img/guides/dbt-ecosystem/dbt-python-snowpark/12-machine-learning-training-prediction/3-view-snowflake-query-history.png?v=2 "View Snowflake query history to see how python models are run under the hood")](#)View Snowflake query history to see how python models are run under the hood ##### Predicting on new data 1. Create a new file called `predict_position.py` and copy and save the following code: ```python import logging import joblib import pandas as pd import os from snowflake.snowpark import types as T DB_STAGE = 'MODELSTAGE' version = '1.0' # The name of the model file model_file_path = 'driver_position_'+version model_file_packaged = 'driver_position_'+version+'.joblib' # This is a local directory, used for storing the various artifacts locally LOCAL_TEMP_DIR = f'/tmp/driver_position' DOWNLOAD_DIR = os.path.join(LOCAL_TEMP_DIR, 'download') TARGET_MODEL_DIR_PATH = os.path.join(LOCAL_TEMP_DIR, 'ml_model') TARGET_LIB_PATH = os.path.join(LOCAL_TEMP_DIR, 'lib') # The feature columns that were used during model training # and that will be used during prediction FEATURE_COLS = [ "RACE_YEAR" ,"CIRCUIT_NAME" ,"GRID" ,"CONSTRUCTOR_NAME" ,"DRIVER" ,"DRIVERS_AGE_YEARS" ,"DRIVER_CONFIDENCE" ,"CONSTRUCTOR_RELAIBLITY" ,"TOTAL_PIT_STOPS_PER_RACE"] def register_udf_for_prediction(p_predictor ,p_session ,p_dbt): # The prediction udf def predict_position(p_df: T.PandasDataFrame[int, int, int, int, int, int, int, int, int]) -> T.PandasSeries[int]: # Snowpark currently does not set the column name in the input dataframe # The default col names are like 0,1,2,... Hence we need to reset the column # names to the features that we initially used for training. p_df.columns = [*FEATURE_COLS] # Perform prediction. this returns an array object pred_array = p_predictor.predict(p_df) # Convert to series df_predicted = pd.Series(pred_array) return df_predicted # The list of packages that will be used by UDF udf_packages = p_dbt.config.get('packages') predict_position_udf = p_session.udf.register( predict_position ,name=f'predict_position' ,packages = udf_packages ) return predict_position_udf def download_models_and_libs_from_stage(p_session): p_session.file.get(f'@{DB_STAGE}/{model_file_path}/{model_file_packaged}', DOWNLOAD_DIR) def load_model(p_session): # Load the model and initialize the predictor model_fl_path = os.path.join(DOWNLOAD_DIR, model_file_packaged) predictor = joblib.load(model_fl_path) return predictor # ------------------------------- def model(dbt, session): dbt.config( packages = ['snowflake-snowpark-python' ,'scipy','scikit-learn' ,'pandas' ,'numpy'], materialized = "table", tags = "predict" ) session._use_scoped_temp_objects = False download_models_and_libs_from_stage(session) predictor = load_model(session) predict_position_udf = register_udf_for_prediction(predictor, session ,dbt) # Retrieve the data, and perform the prediction hold_out_df = (dbt.ref("hold_out_dataset_for_prediction") .select(*FEATURE_COLS) ) # Perform prediction. new_predictions_df = hold_out_df.withColumn("position_predicted" ,predict_position_udf(*FEATURE_COLS) ) return new_predictions_df ``` 2. Execute the following in the command bar: ```bash dbt run --select predict_position ``` 3. **Commit and push** our changes to keep saving our work as we go using the commit message `logistic regression model training and application` before moving on. 4. At a high level in this script, we are: * Retrieving our staged logistic regression model * Loading the model in * Placing the model within a user defined function (UDF) to call in line predictions on our driver’s position 5. At a more detailed level: * Import our libraries. * Create variables to reference back to the `MODELSTAGE` we just created and stored our model to. * The temporary file paths we created might look intimidating, but all we’re doing here is programmatically using an initial file path and adding to it to create the following directories: * LOCAL\_TEMP\_DIR ➡️ /tmp/driver\_position * DOWNLOAD\_DIR ➡️ /tmp/driver\_position/download * TARGET\_MODEL\_DIR\_PATH ➡️ /tmp/driver\_position/ml\_model * TARGET\_LIB\_PATH ➡️ /tmp/driver\_position/lib * Provide a list of our feature columns that we used for model training and will now be used on new data for prediction. * Next, we are creating our main function `register_udf_for_prediction(p_predictor ,p_session ,p_dbt):`. This function is used to register a user-defined function (UDF) that performs the machine learning prediction. It takes three parameters: `p_predictor` is an instance of the machine learning model, `p_session` is an instance of the Snowflake session, and `p_dbt` is an instance of the dbt library. The function creates a UDF named `predict_churn` which takes a pandas dataframe with the input features and returns a pandas series with the predictions. * ⚠️ Pay close attention to the whitespace here. We are using a function within a function for this script. * We have 2 simple functions that are programmatically retrieving our file paths to first get our stored model out of our `MODELSTAGE` and downloaded into the session `download_models_and_libs_from_stage` and then to load the contents of our model in (parameters) in `load_model` to use for prediction. * Take the model we loaded in and call it `predictor` and wrap it in a UDF. * Return our dataframe with both the features used to predict and the new label. 🧠 Another way to read this script is from the bottom up. This can help us progressively see what is going into our final dbt model and work backwards to see how the other functions are being referenced. 6. Let’s take a look at our predicted position alongside our feature variables. Open a new scratchpad and use the following query. I chose to order by the prediction of who would obtain a podium position: ```sql select * from {{ ref('predict_position') }} order by position_predicted ``` 7. We can see that we created predictions in our final dataset, we are ready to move on to testing! #### Test your data models We have now completed building all the models for today’s lab, but how do we know if they meet our assertions? Put another way, how do we know the quality of our data models are any good? This brings us to testing! We test data models for mainly two reasons: * Ensure that our source data is clean on ingestion before we start data modeling/transformation (aka avoid garbage in, garbage out problem). * Make sure we don’t introduce bugs in the transformation code we wrote (stop ourselves from creating bad joins/fanouts). Testing in dbt comes in two flavors: [generic](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests) and [singular](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests). You define them in a test block (similar to a macro) and once defined, you can reference them by name in your `.yml` files (applying them to models, columns, sources, snapshots, and seeds). You might be wondering: *what about testing Python models?* Since the output of our Python models are tables, we can test SQL and Python models the same way! We don’t have to worry about any syntax differences when testing SQL versus Python data models. This means we use `.yml` and `.sql` files to test our entities (tables, views, etc.). Under the hood, dbt is running an SQL query on our tables to see if they meet assertions. If no rows are returned, dbt will surface a passed test. Conversely, if a test results in returned rows, it will fail or warn depending on the configuration (more on that later). ##### Generic tests 1. To implement generic out-of-the-box tests dbt comes with, we can use YAML files to specify information about our models. To add generic tests to our aggregates model, create a file called `aggregates.yml`, copy the code block below into the file, and save. [![The aggregates.yml file in our file tree](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/1-generic-testing-file-tree.png?v=2 "The aggregates.yml file in our file tree")](#)The aggregates.yml file in our file tree ```yaml models: - name: fastest_pit_stops_by_constructor description: Use the python .describe() method to retrieve summary statistics table about pit stops by constructor. Sort by average stop time ascending so the first row returns the fastest constructor. columns: - name: constructor_name description: team that makes the car data_tests: - unique - name: lap_times_moving_avg description: Use the python .rolling() method to calculate the 5 year rolling average of pit stop times alongside the average for each year. columns: - name: race_year description: year of the race data_tests: - relationships: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. to: ref('int_lap_times_years') field: race_year ``` 2. Let’s unpack the code we have here. We have both our aggregates models with the model name to know the object we are referencing and the description of the model that we’ll populate in our documentation. At the column level (a level below our model), we are providing the column name followed by our tests. We want to ensure our `constructor_name` is unique since we used a pandas `groupby` on `constructor_name` in the model `fastest_pit_stops_by_constructor`. Next, we want to ensure our `race_year` has referential integrity from the model we selected from `int_lap_times_years` into our subsequent `lap_times_moving_avg` model. 3. Finally, if we want to see how tests were deployed on sources and SQL models, we can look at other files in our project such as the `f1_sources.yml` we created in our Sources and staging section. ##### Using macros for testing 1. Under your `macros` folder, create a new file and name it `test_all_values_gte_zero.sql`. Copy the code block below and save the file. For clarity, “gte” is an abbreviation for greater than or equal to. [![macro file for reusable testing code](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/2-macro-testing.png?v=2 "macro file for reusable testing code")](#)macro file for reusable testing code ```sql {% macro test_all_values_gte_zero(table, column) %} select * from {{ ref(table) }} where {{ column }} < 0 {% endmacro %} ``` 2. Macros in Jinja are pieces of code that can be reused multiple times in our SQL models — they are analogous to "functions" in other programming languages, and are extremely useful if you find yourself repeating code across multiple models. 3. We use the `{% macro %}` to indicate the start of the macro and `{% endmacro %}` for the end. The text after the beginning of the macro block is the name we are giving the macro to later call it. In this case, our macro is called `test_all_values_gte_zero`. Macros take in *arguments* to pass through, in this case the `table` and the `column`. In the body of the macro, we see an SQL statement that is using the `ref` function to dynamically select the table and then the column. You can always view macros without having to run them by using `dbt run-operation`. You can learn more [here](https://docs.getdbt.com/reference/commands/run-operation.md). 4. Great, now we want to reference this macro as a test! Let’s create a new test file called `macro_pit_stops_mean_is_positive.sql` in our `tests` folder. [![creating a test on our pit stops model referencing the macro](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/3-gte-macro-applied-to-pit-stops.png?v=2 "creating a test on our pit stops model referencing the macro")](#)creating a test on our pit stops model referencing the macro 5. Copy the following code into the file and save: ```sql {{ config( enabled=true, severity='warn', tags = ['bi'] ) }} {{ test_all_values_gte_zero('fastest_pit_stops_by_constructor', 'mean') }} ``` 6. In our testing file, we are applying some configurations to the test including `enabled`, which is an optional configuration for disabling models, seeds, snapshots, and tests. Our severity is set to `warn` instead of `error`, which means our pipeline will still continue to run. We have tagged our test with `bi` since we are applying this test to one of our bi models. Then, in our final line, we are calling the `test_all_values_gte_zero` macro that takes in our table and column arguments and inputting our table `'fastest_pit_stops_by_constructor'` and the column `'mean'`. ##### Custom singular tests to validate Python models The simplest way to define a test is by writing the exact SQL that will return failing records. We call these "singular" tests, because they're one-off assertions usable for a single purpose. These tests are defined in `.sql` files, typically in your `tests` directory (as defined by your test-paths config). You can use Jinja in SQL models (including ref and source) in the test definition, just like you can when creating models. Each `.sql` file contains one select statement, and it defines one test. Let’s add a custom test that asserts that the moving average of the lap time over the last 5 years is greater than zero (it’s impossible to have time less than 0!). It is easy to assume if this is not the case the data has been corrupted. 1. Create a file `lap_times_moving_avg_assert_positive_or_null.sql` under the `tests` folder. [![custom singular test for testing lap times are positive values](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/4-custom-singular-test.png?v=2 "custom singular test for testing lap times are positive values")](#)custom singular test for testing lap times are positive values 2. Copy the following code and save the file: ```sql {{ config( enabled=true, severity='error', tags = ['bi'] ) }} with lap_times_moving_avg as ( select * from {{ ref('lap_times_moving_avg') }} ) select * from lap_times_moving_avg where lap_moving_avg_5_years < 0 and lap_moving_avg_5_years is not null ``` ##### Putting all our tests together 1. Time to run our tests! Altogether, we have created 4 tests for our 2 Python models: * `fastest_pit_stops_by_constructor` * Unique `constructor_name` * Lap times are greater than 0 or null (to allow for the first leading values in a rolling calculation) * `lap_times_moving_avg` * Referential test on `race_year` * Mean pit stop times are greater than or equal to 0 (no negative time values) 2. To run the tests on both our models, we can use this syntax in the command line to run them both at once, similar to how we did our data splits earlier. Execute the following in the command bar: ```bash dbt test --select fastest_pit_stops_by_constructor lap_times_moving_avg ``` [![running tests on our python models](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/5-running-tests-on-python-models.png?v=2 "running tests on our python models")](#)running tests on our python models 3. All 4 of our tests passed (yay for clean data)! To understand the SQL being run against each of our tables, we can click into the details of the test. 4. Navigating into the **Details** of the `unique_fastest_pit_stops_by_constructor_name`, we can see that each line `constructor_name` should only have one row. [![view details of testing our python model that used SQL to test data assertions](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/6-testing-output-details.png?v=2 "view details of testing our python model that used SQL to test data assertions")](#)view details of testing our python model that used SQL to test data assertions #### Document your dbt project When it comes to documentation, dbt brings together both column and model level descriptions that you can provide as well as details from your Snowflake information schema in a static site for consumption by other data team members and stakeholders. We are going to revisit 2 areas of our project to understand our documentation: * `intermediate.md` file * `dbt_project.yml` file To start, let’s look back at our `intermediate.md` file. We can see that we provided multi-line descriptions for the models in our intermediate models using [docs blocks](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks). Then we reference these docs blocks in our `.yml` file. Building descriptions with doc blocks in Markdown files gives you the ability to format your descriptions with Markdown and are particularly helpful when building long descriptions, either at the column or model level. In our `dbt_project.yml`, we added `node_colors` at folder levels. 1. To see all these pieces come together, execute this in the command bar: ```bash dbt docs generate ``` This will generate the documentation for your project. Click the book button, as shown in the screenshot below to access the docs. [![dbt docs book icon](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/1-docs-icon.png?v=2 "dbt docs book icon")](#)dbt docs book icon 2. Go to our project area and view `int_results`. View the description that we created in our doc block. [![Docblock description within docs site](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/2-view-docblock-description.png?v=2 "Docblock description within docs site")](#)Docblock description within docs site 3. View the mini-lineage that looks at the model we are currently selected on (`int_results` in this case). [![Mini lineage view on docs site](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/3-mini-lineage-docs.png?v=2 "Mini lineage view on docs site")](#)Mini lineage view on docs site 4. In our `dbt_project.yml`, we configured `node_colors` depending on the file directory. By color coding your project, it can help you cluster together similar models or steps and more easily troubleshoot when viewing lineage in your docs. [![Full project DAG on docs site](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/4-full-dag-docs.png?v=2 "Full project DAG on docs site")](#)Full project DAG on docs site #### Deploy your code Before we jump into deploying our code, let's have a quick primer on environments. Up to this point, all of the work we've done in the Studio IDE has been in our development environment, with code committed to a feature branch and the models we've built created in our development schema in Snowflake as defined in our Development environment connection. Doing this work on a feature branch, allows us to separate our code from what other coworkers are building and code that is already deemed production ready. Building models in a development schema in Snowflake allows us to separate the database objects we might still be modifying and testing from the database objects running production dashboards or other downstream dependencies. Together, the combination of a Git branch and Snowflake database objects form our environment. Now that we've completed testing and documenting our work, we're ready to deploy our code from our development environment to our production environment and this involves two steps: * Promoting code from our feature branch to the production branch in our repository. * Generally, the production branch is going to be named your main branch and there's a review process to go through before merging code to the main branch of a repository. Here we are going to merge without review for ease of this workshop. * Deploying code to our production environment. * Once our code is merged to the main branch, we'll need to run dbt in our production environment to build all of our models and run all of our tests. This will allow us to build production-ready objects into our production environment in Snowflake. Luckily for us, the Partner Connect flow has already created our deployment environment and job to facilitate this step. 1. Before getting started, let's make sure that we've committed all of our work to our feature branch. If you still have work to commit, you'll be able to select the **Commit and push**, provide a message, and then select **Commit** again. 2. Once all of your work is committed, the git workflow button will now appear as **Merge this branch to main**. Click **Merge this branch to main** and the merge process will automatically run in the background. [![Merge this branch to main](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/1-merge-to-main-branch.png?v=2 "Merge this branch to main")](#)Merge this branch to main 3. When it's completed, you should see the git button read **Create branch** and the branch you're currently looking at will become **main**. 4. Now that all of our development work has been merged to the main branch, we can build our deployment job. Given that our production environment and production job were created automatically for us through Partner Connect, all we need to do here is update some default configurations to meet our needs. 5. In the left-hand menu, go to **Orchestration** > **Environments**. 6. You should see two environments listed and you'll want to select the **Deployment** environment then **Settings** to modify it. 7. Before making any changes, let's touch on what is defined within this environment. The Snowflake connection shows the credentials that dbt is using for this environment and in our case they are the same as what was created for us through Partner Connect. Our deployment job will build in our `PC_DBT_DB` database and use the default Partner Connect role and warehouse to do so. The deployment credentials section also uses the info that was created in our Partner Connect job to create the credential connection. However, it is using the same default schema that we've been using as the schema for our development environment. 8. Let's update the schema to create a new schema specifically for our production environment. Click **Edit** to allow you to modify the existing field values. Navigate to **Deployment Credentials >** **schema.** 9. Update the schema name to **production**. Remember to select **Save** after you've made the change. [![Update the deployment credentials schema to production](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/3-update-deployment-credentials-production.png?v=2 "Update the deployment credentials schema to production")](#)Update the deployment credentials schema to production 10. By updating the schema for our production environment to **production**, it ensures that our deployment job for this environment will build our dbt models in the **production** schema within the `PC_DBT_DB` database as defined in the Snowflake Connection section. 11. Now let's switch over to our production job. Click on the deploy tab again and then select **Jobs**. You should see an existing and preconfigured **Partner Connect Trial Job**. Similar to the environment, click on the job, then select **Settings** to modify it. Let's take a look at the job to understand it before making changes. * The Environment section is what connects this job with the environment we want it to run in. This job is already defaulted to use the Deployment environment that we just updated and the rest of the settings we can keep as is. * The Execution settings section gives us the option to generate docs, run source freshness, and defer to a previous run state. For the purposes of our lab, we're going to keep these settings as is as well and stick with just generating docs. * The Commands section is where we specify exactly which commands we want to run during this job, and we also want to keep this as is. We want our seed to be uploaded first, then run our models, and finally test them. The order of this is important as well, considering that we need our seed to be created before we can run our incremental model, and we need our models to be created before we can test them. * Finally, we have the Triggers section, where we have a number of different options for scheduling our job. Given that our data isn't updating regularly here and we're running this job manually for now, we're also going to leave this section alone. So, what are we changing then? Just the name! Click **Edit** to allow you to make changes. Then update the name of the job to **Production Job** to denote this as our production deployment job. After that's done, click **Save**. 12. Now let's go to run our job. Clicking on the job name in the path at the top of the screen will take you back to the job run history page where you'll be able to click **Run run** to kick off the job. If you encounter any job failures, try running the job again before further troubleshooting. [![Run production job](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/4-run-production-job.png?v=2 "Run production job")](#)Run production job [![View production job details](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/5-job-details.png?v=2 "View production job details")](#)View production job details 13. Let's go over to Snowflake to confirm that everything built as expected in our production schema. Refresh the database objects in your Snowflake account and you should see the production schema now within our default Partner Connect database. If you click into the schema and everything ran successfully, you should be able to see all of the models we developed. [![Check all our models in our pipeline are in Snowflake](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/6-all-models-generated.png?v=2 "Check all our models in our pipeline are in Snowflake")](#)Check all our models in our pipeline are in Snowflake ##### Conclusion Fantastic! You’ve finished the workshop! We hope you feel empowered in using both SQL and Python in your dbt workflows with Snowflake. Having a reliable pipeline to surface both analytics and machine learning is crucial to creating tangible business value from your data. For more help and information join our [dbt community Slack](https://www.getdbt.com/community/) which contains more than 50,000 data practitioners today. We have a dedicated slack channel #db-snowflake to Snowflake related content. Happy dbt'ing! --- ### Migrate between dbt platform tenancy types [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() Contract change required Migrating between tenancy types requires a contract change. Make sure you contact your account manager before starting this process to confirm eligibility, timeline, and initiate the necessary contract updates. *Do not* begin migration steps until a contract change is confirmed. #### Overview This guide walks you through a self-serve migration between [dbt platform tenancy types](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) — for example, moving from multi-tenant to single-tenant. It's meant for accounts that have been assessed by the dbt Labs team as low complexity. Your dbt Labs contact will provide a migration window before you begin — do not start until you've received it. Use the following table to confirm this guide is the right path for your account. If your account crosses the high-complexity threshold for even one signal, we recommend a Professional Services-led migration instead. | Signal | Low complexity (self-serve) | High complexity (PS-led) | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Active jobs (run in last 30 days) | Fewer than 10 | 10 or more | | Projects | Fewer than 3 | 3 or more | | Environments | Fewer than 5 | 5 or more | | **How to proceed** | If all signals are low complexity, follow this guide or contact [Professional Services migration](https://www.getdbt.com/services) for help | If one or more signals are high complexity, contact your account manager to scope a [Professional Services migration](https://www.getdbt.com/services) | The following factors add coordination overhead regardless of account size — flag these to your account manager if they apply: * CI or webhook-triggered jobs * SSO or SCIM provisioning * Privatelink endpoints * Multiple warehouse connection types ##### Related docs * [Tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) * [Access, regions, and IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) #### What doesn't migrate automatically The migration tool automates most configuration, but some things require manual setup after you apply the Terraform configuration. **Credentials and secrets** — these must be recreated manually: * SSH keys and repository credentials * Service account tokens * Environment variables containing secrets * Warehouse connections and environment-level credentials * dbt platform profiles **Project data** — not transferred: * Job run history, artifacts, and logs **User management** — requires manual verification: * Users must already exist in the target account * Permissions must be manually verified after migration * SSO, keypairs, and warehouse authentication credentials must be reconfigured **Unsupported resources:** * PrivateLink endpoints require assistance from the dbt Labs infrastructure team — [open a support ticket](https://www.getdbt.com/support) #### Prerequisites Before starting your migration, confirm the following: * A contract change has been initiated and confirmed with your account manager * You have received a migration window from the dbt Labs Support team * You have admin access to both the source and target dbt platform accounts * The target account is provisioned in the new environment * You have a service token with Account Admin permissions on the source account #### Migration steps This migration uses two open-source tools: * [dbt platform migration tool](https://github.com/dpguthrie/dbt-cloud-migration-tool) — exports your source account configuration and applies it to the target account via Terraform * [dbtcloud-terraforming](https://github.com/dbt-labs/dbtcloud-terraforming) — generates Terraform HCL from an existing dbt platform account info Credentials, secrets, and some other items don't migrate automatically — plan to recreate them manually. Refer to [What doesn't migrate automatically](#what-doesnt-migrate-automatically) for the full list. ##### Mac/Linux ##### Step 1: Install prerequisites You need Python, Terraform, and `dbtcloud-terraforming`. Install them using any method you prefer — we use Homebrew as an example here. Create and activate a Python virtual environment: ```shell python3 -m venv tf_acct_migrator_env source tf_acct_migrator_env/bin/activate ``` Install Terraform: ```shell brew tap hashicorp/tap && brew install hashicorp/tap/terraform ``` Install `dbtcloud-terraforming`: ```shell brew install dbt-labs/dbt-cli/dbtcloud-terraforming ``` ##### Step 2: Set up the migration tool Clone the migration tool repo and navigate into it: ```shell git clone https://github.com/dpguthrie/dbt-cloud-migration-tool cd dbt-cloud-migration-tool ``` Set environment variables for your source dbt platform account: ```shell export DBT_CLOUD_HOST_URL="https://YOUR_SOURCE_REGION.getdbt.com/api" export DBT_CLOUD_TOKEN="your_source_service_token" export DBT_CLOUD_ACCOUNT_ID="your_source_account_id" ``` ##### Step 3: Generate and apply Terraform configuration Run the migration tool to generate a `resources.tf` file with your source account's resources: ```shell sh migrate.sh ``` Copy the example vars file and add your target account's details: ```shell cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with your target account details ``` Apply the configuration to your target account: ```shell terraform init terraform plan terraform apply ``` Terraform will output the newly created resources and any warnings or errors. You may need to re-run `migrate.sh` multiple times to catch all required resource types. ##### Windows ##### Step 1: Install prerequisites You need Python, Terraform, and `dbtcloud-terraforming`. Install them using any method you prefer — we use Chocolatey as an example here. Install Python and Terraform: ```shell choco install python terraform -y ``` After installation, restart your terminal and verify: ```shell python --version terraform --version ``` Install `dbtcloud-terraforming` by adding it to your Terraform installation directory. If you used Chocolatey, the path is typically `C:\ProgramData\chocolatey\bin`. ##### Step 2: Set up the migration tool The following steps should be run in a bash environment (for example, Git Bash). Clone the migration tool repo and navigate into it: ```shell git clone https://github.com/dpguthrie/dbt-cloud-migration-tool.git cd dbt-cloud-migration-tool ``` ##### Step 3: Configure and run the migration Set environment variables for your source dbt platform account (these reset when you close the terminal): ```shell export DBT_CLOUD_HOST_URL="https://YOUR_SOURCE_REGION.getdbt.com/api" export DBT_CLOUD_TOKEN="your_source_service_token" export DBT_CLOUD_ACCOUNT_ID="your_source_account_id" ``` Run the migration script: ```shell sh migrate.sh ``` Copy the example vars file and add your target account's details: ```shell cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with your target account details ``` Apply the configuration to your target account: ```shell terraform init terraform plan terraform apply ``` Terraform will output the newly created resources and any warnings or errors. You may need to re-run `migrate.sh` multiple times to catch all required resource types. #### Post-migration checklist * \[ ] Verify all connections, permissions, and secrets in the target account * \[ ] Run test jobs to confirm end-to-end functionality * \[ ] Confirm IP allowlist and network settings are updated for the new tenant * \[ ] Update internal bookmarks, SSO configurations, and API integrations to the new account URL --- ### Migrate from dbt-spark to dbt-databricks [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction You can migrate your projects from using the `dbt-spark` adapter to using the [dbt-databricks adapter](https://github.com/databricks/dbt-databricks). In collaboration with dbt Labs, Databricks built this adapter using dbt-spark as the foundation and added some critical improvements. With it, you get an easier set up — requiring only three inputs for authentication — and more features such as support for [Unity Catalog](https://www.databricks.com/product/unity-catalog). ##### Prerequisite * For dbt, you need administrative (admin) privileges to migrate dbt projects. ##### Simpler authentication Previously, you had to provide a `cluster` or `endpoint` ID which was hard to parse from the `http_path` that you were given. Now, it doesn't matter if you're using a cluster or an SQL endpoint because the [dbt-databricks setup](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) requires the *same* inputs for both. All you need to provide is: * hostname of the Databricks workspace * HTTP path of the Databricks SQL warehouse or cluster * appropriate credentials ##### Better defaults The `dbt-databricks` adapter provides better defaults than `dbt-spark` does. The defaults help optimize your workflow so you can get the fast performance and cost-effectiveness of Databricks. They are: * The dbt models use the [Delta](https://docs.databricks.com/delta/index.html) table format. You can remove any declared configurations of `file_format = 'delta'` since they're now redundant. * Accelerate your expensive queries with the [Photon engine](https://docs.databricks.com/runtime/photon.html). * The `incremental_strategy` config is set to `merge`. With dbt-spark, however, the default for `incremental_strategy` is `append`. If you want to continue using `incremental_strategy=append`, you must set this config specifically on your incremental models. If you already specified `incremental_strategy=merge` on your incremental models, you don't need to change anything when moving to dbt-databricks; but, you can keep your models clean (tidy) by removing the config since it's redundant. Read [About incremental\_strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) to learn more. For more information on defaults, see [Caveats](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md#caveats). ##### Pure Python If you use dbt v1, you no longer have to download an independent driver to interact with Databricks. The connection information is all embedded in a pure-Python library called `databricks-sql-connector`. #### Migrate your dbt projects in dbt You can migrate your projects to the Databricks-specific adapter from the generic Apache Spark adapter. If you're using dbt v1, then skip to Step 4. The migration to the `dbt-databricks` adapter from `dbt-spark` shouldn't cause any downtime for production jobs. dbt Labs recommends that you schedule the connection change when usage of the IDE is light to avoid disrupting your team. To update your Databricks connection in dbt: 1. Select **Account Settings** in the main navigation bar. 2. On the **Projects** tab, find the project you want to migrate to the dbt-databricks adapter. 3. Click the hyperlinked Connection for the project. 4. Click **Edit** in the top right corner. 5. Select **Databricks** for the warehouse 6. Enter the: 1. `hostname` 2. `http_path` 3. (optional) catalog name 7. Click **Save**. Everyone in your organization who uses dbt must refresh the Studio IDE before starting work again. It should refresh in less than a minute. #### Configure your credentials When you update the Databricks connection in dbt, your team will not lose their credentials. This makes migrating easier since it only requires you to delete the Databricks connection and re-add the cluster or endpoint information. These credentials will not get lost when there's a successful connection to Databricks using the `dbt-spark` ODBC method: * The credentials you supplied to dbt to connect to your Databricks workspace. * The personal access tokens your team added in their dbt profile so they can develop in the Studio IDE for a given project. * The access token you added for each deployment environment so dbt can connect to Databricks during production jobs. #### Migrate dbt projects in dbt To migrate your dbt v1 projects to the `dbt-databricks` adapter from `dbt-spark`, you: 1. Install the [dbt-databricks adapter](https://github.com/databricks/dbt-databricks) in your environment 2. Update your Databricks connection by modifying your `target` in your `~/.dbt/profiles.yml` file Anyone who's using your project must also make these changes in their environment. #### Try these examples You can use the following examples of the `profiles.yml` file to see the authentication setup with `dbt-spark` compared to the simpler setup with `dbt-databricks` when connecting to an SQL endpoint. A cluster example would look similar. An example of what authentication looks like with `dbt-spark`: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: spark method: odbc driver: '/opt/simba/spark/lib/64/libsparkodbc_sb64.so' schema: my_schema host: dbc-l33t-nwb.cloud.databricks.com endpoint: 8657cad335ae63e3 token: [my_secret_token] ``` An example of how much simpler authentication is with `dbt-databricks`: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: databricks schema: my_schema host: dbc-l33t-nwb.cloud.databricks.com http_path: /sql/1.0/endpoints/8657cad335ae63e3 token: [my_secret_token] ``` --- ### Migrate from DDL, DML, and stored procedures [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction One of the more common situations that new dbt adopters encounter is a historical codebase of transformations written as a hodgepodge of DDL and DML statements, or stored procedures. Going from DML statements to dbt models is often a challenging hump for new users to get over, because the process involves a significant paradigm shift between a procedural flow of building a dataset (e.g. a series of DDL and DML statements) to a declarative approach to defining a dataset (e.g. how dbt uses SELECT statements to express data models). This guide aims to provide tips, tricks, and common patterns for converting DML statements to dbt models. ##### Preparing to migrate Before getting into the meat of conversion, it’s worth noting that DML statements will not always illustrate a comprehensive set of columns and column types that an original table might contain. Without knowing the DDL to create the table, it’s impossible to know precisely if your conversion effort is apples-to-apples, but you can generally get close. If your data warehouse supports `SHOW CREATE TABLE`, that can be a quick way to get a comprehensive set of columns you’ll want to recreate. If you don’t have the DDL, but are working on a substantial stored procedure, one approach that can work is to pull column lists out of any DML statements that modify the table, and build up a full set of the columns that appear. As for ensuring that you have the right column types, since models materialized by dbt generally use `CREATE TABLE AS SELECT` or `CREATE VIEW AS SELECT` as the driver for object creation, tables can end up with unintended column types if the queries aren’t explicit. For example, if you care about `INT` versus `DECIMAL` versus `NUMERIC`, it’s generally going to be best to be explicit. The good news is that this is easy with dbt: you just cast the column to the type you intend. We also generally recommend that column renaming and type casting happen as close to the source tables as possible, typically in a layer of staging transformations, which helps ensure that future dbt modelers will know where to look for those transformations! See [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) for more guidance on overall project structure. ##### Operations we need to map There are four primary DML statements that you are likely to have to convert to dbt operations while migrating a procedure: * `INSERT` * `UPDATE` * `DELETE` * `MERGE` Each of these can be addressed using various techniques in dbt. Handling `MERGE`s is a bit more involved than the rest, but can be handled effectively via dbt. The first three, however, are fairly simple to convert. #### Map INSERTs An `INSERT` statement is functionally the same as using dbt to `SELECT` from an existing source or other dbt model. If you are faced with an `INSERT`-`SELECT` statement, the easiest way to convert the statement is to just create a new dbt model, and pull the `SELECT` portion of the `INSERT` statement out of the procedure and into the model. That’s basically it! To really break it down, let’s consider a simple example: ```sql INSERT INTO returned_orders (order_id, order_date, total_return) SELECT order_id, order_date, total FROM orders WHERE type = 'return' ``` Converting this with a first pass to a [dbt model](https://docs.getdbt.com/guides/bigquery.md?step=8) (in a file called returned\_orders.sql) might look something like: ```sql SELECT order_id as order_id, order_date as order_date, total as total_return FROM {{ ref('orders') }} WHERE type = 'return' ``` Functionally, this would create a model (which could be materialized as a table or view depending on needs) called `returned_orders` that contains three columns: `order_id`, `order_date`, `total_return`) predicated on the type column. It achieves the same end as the `INSERT`, just in a declarative fashion, using dbt. ##### **A note on `FROM` clauses** In dbt, using a hard-coded table or view name in a `FROM` clause is one of the most serious mistakes new users make. dbt uses the ref and source macros to discover the ordering that transformations need to execute in, and if you don’t use them, you’ll be unable to benefit from dbt’s built-in lineage generation and pipeline execution. In the sample code throughout the remainder of this article, we’ll use ref statements in the dbt-converted versions of SQL statements, but it is an exercise for the reader to ensure that those models exist in their dbt projects. ##### **Sequential `INSERT`s to an existing table can be `UNION ALL`’ed together** Since dbt models effectively perform a single `CREATE TABLE AS SELECT` (or if you break it down into steps, `CREATE`, then an `INSERT`), you may run into complexities if there are multiple `INSERT` statements in your transformation that all insert data into the same table. Fortunately, this is a simple thing to handle in dbt. Effectively, the logic is performing a `UNION ALL` between the `INSERT` queries. If I have a transformation flow that looks something like (ignore the contrived nature of the scenario): ```sql CREATE TABLE all_customers INSERT INTO all_customers SELECT * FROM us_customers INSERT INTO all_customers SELECT * FROM eu_customers ``` The dbt-ified version of this would end up looking something like: ```sql SELECT * FROM {{ ref('us_customers') }} UNION ALL SELECT * FROM {{ ref('eu_customers') }} ``` The logic is functionally equivalent. So if there’s another statement that `INSERT`s into a model that I’ve already created, I can just add that logic into a second `SELECT` statement that is just `UNION ALL`'ed with the first. Easy! #### Map UPDATEs `UPDATE`s start to increase the complexity of your transformations, but fortunately, they’re pretty darn simple to migrate, as well. The thought process that you go through when translating an `UPDATE` is quite similar to how an `INSERT` works, but the logic for the `SELECT` list in the dbt model is primarily sourced from the content in the `SET` section of the `UPDATE` statement. Let’s look at a simple example: ```sql UPDATE orders SET type = 'return' WHERE total < 0 ``` The way to look at this is similar to an `INSERT`-`SELECT` statement. The table being updated is the model you want to modify, and since this is an `UPDATE`, that model has likely already been created, and you can either: * add to it with subsequent transformations * create an intermediate model that builds off of the original model – perhaps naming it something like `int_[entity]_[verb].sql`. The `SELECT` list should contain all of the columns for the table, but for the specific columns being updated by the DML, you’ll use the computation on the right side of the equals sign as the `SELECT`ed value. Then, you can use the target column name on the left of the equals sign as the column alias. If I were building an intermediate transformation from the above query would translate to something along the lines of: ```sql SELECT CASE WHEN total < 0 THEN 'return' ELSE type END AS type, order_id, order_date FROM {{ ref('stg_orders') }} ``` Since the `UPDATE` statement doesn’t modify every value of the type column, we use a `CASE` statement to apply the contents’ `WHERE` clause. We still want to select all of the columns that should end up in the target table. If we left one of the columns out, it wouldn’t be passed through to the target table at all due to dbt’s declarative approach. Sometimes, you may not be sure what all the columns are in a table, or in the situation as above, you’re only modifying a small number of columns relative to the total number of columns in the table. It can be cumbersome to list out every column in the table, but fortunately dbt contains some useful utility macros that can help list out the full column list of a table. Another way I could have written the model a bit more dynamically might be: ```sql SELECT {{ dbt_utils.star(from=ref('stg_orders'), except=['type']) }}, CASE WHEN total < 0 THEN 'return' ELSE type END AS type, FROM {{ ref('stg_orders') }} ``` The `dbt_utils.star()` macro will print out the full list of columns in the table, but skip the ones I’ve listed in the except list, which allows me to perform the same logic while writing fewer lines of code. This is a simple example of using dbt macros to simplify and shorten your code, and dbt can get a lot more sophisticated as you learn more techniques. Read more about the [dbt\_utils package](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and the [star macro](https://github.com/dbt-labs/dbt-utils/tree/0.8.6/#star-source). #### Map DELETEs One of the biggest differences between a procedural transformation and how dbt models data is that dbt, in general, will never destroy data. While there are ways to execute hard `DELETE`s in dbt that are outside of the scope of this article, the general best practice for handling deleted data is to just use soft deletes, and filter out soft-deleted data in a final transformation. Let’s consider a simple example query: ```sql DELETE FROM stg_orders WHERE order_status IS NULL ``` In a dbt model, you’ll need to first identify the records that should be deleted and then filter them out. There are really two primary ways you might translate this query: ```sql SELECT * FROM {{ ref('stg_orders') }} WHERE order_status IS NOT NULL ``` This first approach just inverts the logic of the DELETE to describe the set of records that should remain, instead of the set of records that should be removed. This ties back to the way dbt declaratively describes datasets. You reference the data that should be in a dataset, and the table or view gets created with that set of data. Another way you could achieve this is by marking the deleted records, and then filtering them out. For example: ```sql WITH soft_deletes AS ( SELECT *, CASE WHEN order_status IS NULL THEN true ELSE false END AS to_delete FROM {{ ref('stg_orders') }} ) SELECT * FROM soft_deletes WHERE to_delete = false ``` This approach flags all of the deleted records, and the final `SELECT` filters out any deleted data, so the resulting table contains only the remaining records. It’s a lot more verbose than just inverting the `DELETE` logic, but for complex `DELETE` logic, this ends up being a very effective way of performing the `DELETE` that retains historical context. It’s worth calling out that while this doesn’t enable a hard delete, hard deletes can be executed a number of ways, the most common being to execute a dbt [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) via as a [run-operation](https://docs.getdbt.com/reference/commands/run-operation.md), or by using a [post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) to perform a `DELETE` statement after the records to-be-deleted have been marked. These are advanced approaches outside the scope of this guide. #### Map MERGEs dbt has a concept called [materialization](https://docs.getdbt.com/docs/build/materializations.md), which determines how a model is physically or logically represented in the warehouse. `INSERT`s, `UPDATE`s, and `DELETE`s will typically be accomplished using table or view materializations. For incremental workloads accomplished via commands like `MERGE` or `UPSERT`, dbt has a particular materialization called [incremental](https://docs.getdbt.com/docs/build/incremental-models.md). The incremental materialization is specifically used to handle incremental loads and updates to a table without recreating the entire table from scratch on every run. ##### Step 1: Map the MERGE like an INSERT/UPDATE to start Before we get into the exact details of how to implement an incremental materialization, let’s talk about logic conversion. Extracting the logic of the `MERGE` and handling it as you would an `INSERT` or an `UPDATE` is the easiest way to get started migrating a `MERGE` command. . To see how the logic conversion works, we’ll start with an example `MERGE`. In this scenario, imagine a ride sharing app where rides are loaded into a details table daily, and tips may be updated at some later date, and need to be kept up-to-date: ```sql MERGE INTO ride_details USING ( SELECT ride_id, subtotal, tip FROM rides_to_load AS rtl ON ride_details.ride_id = rtl.ride_id WHEN MATCHED THEN UPDATE SET ride_details.tip = rtl.tip WHEN NOT MATCHED THEN INSERT (ride_id, subtotal, tip) VALUES (rtl.ride_id, rtl.subtotal, NVL(rtl.tip, 0, rtl.tip) ); ``` The content of the `USING` clause is a useful piece of code because that can easily be placed in a CTE as a starting point for handling the match statement. I find that the easiest way to break this apart is to treat each match statement as a separate CTE that builds on the previous match statements. We can ignore the `ON` clause for now, as that will only come into play once we get to a point where we’re ready to turn this into an incremental. As with `UPDATE`s and `INSERT`s, you can use the `SELECT` list and aliases to name columns appropriately for the target table, and `UNION` together `INSERT` statements (taking care to use `UNION`, rather than `UNION ALL` to avoid duplicates). The `MERGE` would end up translating to something like this: ```sql WITH using_clause AS ( SELECT ride_id, subtotal, tip FROM {{ ref('rides_to_load') }} ), updates AS ( SELECT ride_id, subtotal, tip FROM using_clause ), inserts AS ( SELECT ride_id, subtotal, NVL(tip, 0, tip) FROM using_clause ) SELECT * FROM updates UNION inserts ``` To be clear, this transformation isn’t complete. The logic here is similar to the `MERGE`, but will not actually do the same thing, since the updates and inserts CTEs are both selecting from the same source query. We’ll need to ensure we grab the separate sets of data as we transition to the incremental materialization. One important caveat is that dbt does not natively support `DELETE` as a `MATCH` action. If you have a line in your `MERGE` statement that uses `WHEN MATCHED THEN DELETE`, you’ll want to treat it like an update and add a soft-delete flag, which is then filtered out in a follow-on transformation. ##### Step 2: Convert to incremental materialization As mentioned above, incremental materializations are a little special in that when the target table does not exist, the materialization functions in nearly the same way as a standard table materialization, and executes a `CREATE TABLE AS SELECT` statement. If the target table does exist, however, the materialization instead executes a `MERGE` statement. Since a `MERGE` requires a `JOIN` condition between the `USING` clause and the target table, we need a way to specify how dbt determines whether or not a record triggers a match or not. That particular piece of information is specified in the dbt model configuration. We can add the following `config()` block to the top of our model to specify how it should build incrementally: ```sql {{ config( materialized='incremental', unique_key='ride_id', incremental_strategy='merge' ) }} ``` The three configuration fields in this example are the most important ones. * Setting `materialized='incremental'` tells dbt to apply UPSERT logic to the target table. * The `unique_key` should be a primary key of the target table. This is used to match records with the existing table. * `incremental_strategy` here is set to MERGE any existing rows in the target table with a value for the `unique_key` which matches the incoming batch of data. There are [various incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) for different situations and warehouses. The bulk of the work in converting a model to an incremental materialization comes in determining how the logic should change for incremental loads versus full backfills or initial loads. dbt offers a special macro, `is_incremental()`, which evaluates false for initial loads or for backfills (called full refreshes in dbt parlance), but true for incremental loads. This macro can be used to augment the model code to adjust how data is loaded for subsequent loads. How that logic should be added will depend a little bit on how data is received. Some common ways might be: 1. The source table is truncated ahead of incremental loads, and only contains the data to be loaded in that increment. 2. The source table contains all historical data, and there is a load timestamp column that identifies new data to be loaded. In the first case, the work is essentially done already. Since the source table always contains only the new data to be loaded, the query doesn’t have to change for incremental loads. The second case, however, requires the use of the `is_incremental()` macro to correctly handle the logic. Taking the converted `MERGE` statement that we’d put together previously, we’d augment it to add this additional logic: ```sql WITH using_clause AS ( SELECT ride_id, subtotal, tip, max(load_timestamp) as load_timestamp FROM {{ ref('rides_to_load') }} {% if is_incremental() %} WHERE load_timestamp > (SELECT max(load_timestamp) FROM {{ this }}) {% endif %} ), updates AS ( SELECT ride_id, subtotal, tip, load_timestamp FROM using_clause {% if is_incremental() %} WHERE ride_id IN (SELECT ride_id FROM {{ this }}) {% endif %} ), inserts AS ( SELECT ride_id, subtotal, NVL(tip, 0, tip), load_timestamp FROM using_clause WHERE ride_id NOT IN (SELECT ride_id FROM updates) ) SELECT * FROM updates UNION inserts ``` There are a couple important concepts to understand here: 1. The code in the `is_incremental()` conditional block only executes for incremental executions of this model code. If the target table doesn’t exist, or if the `--full-refresh` option is used, that code will not execute. 2. `{{ this }}` is a special keyword in dbt that when used in a Jinja block, self-refers to the model for which the code is executing. So if you have a model in a file called `my_incremental_model.sql`, `{{ this }}` will refer to `my_incremental_model` (fully qualified with database and schema name if necessary). By using that keyword, we can leverage the current state of the target table to inform the source query. #### Migrate Stores procedures The techniques shared above are useful ways to get started converting the individual DML statements that are often found in stored procedures. Using these types of patterns, legacy procedural code can be rapidly transitioned to dbt models that are much more readable, maintainable, and benefit from software engineering best practices like DRY principles. Additionally, once transformations are rewritten as dbt models, it becomes much easier to test the transformations to ensure that the data being used downstream is high-quality and trustworthy. --- ### Migrate off legacy dbt versions [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() Legacy dbt versions v1.3–v1.7 are being deprecated on January 31, 2027. To keep your work running and supported, move your environments and jobs to a [dbt release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) (v1 or v2) now. A release track keeps you on a supported version automatically. Which release track to choose When migrating off a legacy version: * **v2 (recommended)**: [dbt v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md?version=2) is generally available. Since you're already upgrading, move directly to v2 to get all the powerful features and capabilities it has to offer. * **v1 (minimum requirement)**: To keep your environments and jobs running after January 31, 2027, move to a v1 release track to stay supported. Your project code, connections, and history stay accessible throughout. #### Identify projects using legacy versions What you migrate is driven by a job's effective version: the version pinned on the job if one is set, otherwise the version its environment inherits. 1. Navigate to **Orchestration** > **Environments** and note any environment whose **dbt version** is v1.3–v1.7. The current version is displayed below the environment name. 2. Open the jobs in your supported environments and check for any job with a **version override** pinned to v1.3–v1.7. Find the row that matches each environment, then follow the linked steps. | Environment version | Job version | What you do | | ------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Legacy (v1.3–v1.7) | Inherits from environment, or pinned to a legacy version | [Migrate the environment to a release track](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=3\&version=2) | | Legacy (v1.3–v1.7) | At least one job pinned to a supported version | [Migrate the environment to a release track](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=3\&version=2); the supported jobs are retained | | Supported | One or more jobs pinned to a legacy version | [Update the job's version](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=5\&version=2); the environment is untouched | | Legacy and **dormant** (unused 12+ months) | — | [Delete the environment](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=2\&version=2), or migrate it if you still need it | If you don't migrate a legacy environment or clear a legacy job, it's cleaned up when legacy versions are retired: a legacy environment (and its jobs) is deleted, a legacy environment that already has a job on a supported version is **updated** with only the legacy jobs removed, and a supported environment with legacy-pinned jobs keeps the environment and loses **only those jobs**. #### Delete a dormant environment Before you migrate a project off a deprecated version, consider whether it's needed at all. If a legacy environment has been dormant for a year or more, you probably no longer need it, so delete it. Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. Follow these steps to delete an environment in dbt: 1. Navigate to **Orchestration** > **Environments**. 2. Select the environment you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete** to delete the environment. [![Delete an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-environment.png?v=2 "Delete an environment")](#)Delete an environment 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. 6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Migrate an environment to a release track Start with a development or test environment. Moving it to a release track first lets you find and fix any compatibility issues in your project before you touch your other environments. 1. Navigate to the Settings page of the environment, then click **Edit**. 2. Click the **dbt version** dropdown and select the recommend **V2 Stable** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). If you need to stay on v1, select the **v1 Latest** release track. 3. Save your changes. You can also set the version through the [Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) or Terraform. Refer to [Upgrade versions in dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) for the full walkthrough. Once the environment is on the new version, fix any compatibility issues it surfaces before you migrate the rest of your environments and jobs. #### Fix compatibility issues in your project Moving to a release track can surface deprecation warnings that weren't visible on your legacy version. Resolve these in your project code before you migrate your remaining environments, either by hand or with autofix. ##### Autofix (recommended) Autofix runs the [dbt-autofix](https://github.com/dbt-labs/dbt-autofix) script to resolve many deprecation warnings for you. It's available from both the Studio IDE and the VS Code extension. ###### Studio IDE 1. Open the environment in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). 2. Select **[Check & fix deprecations](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md)** from the IDE's three-dot menu to find and resolve deprecation warnings. 3. Click **Commit and sync** to commit the changes to your project repository. ###### dbt VS Code extension 1. In the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md), run `dbt parse` against the updated environment to surface any deprecation warnings. The **Problems** tab will also surface useful information. 2. Resolve them with the [dbt-autofix](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation) tool, either from the command line or by following the prompts in the extension's upgrade assistant. 3. If you're moving this project to dbt v2 (recommended), use the extension's [agentic migration](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md#agentic-migration) flow instead, which runs autofix as part of the full v1-to-v2 upgrade. ##### Manually fix 1. Run `dbt parse --no-partial-parse --show-all-deprecations` to list the deprecations in your project, or view them from the **Dashboard** in your dbt platform account. 2. Look up each warning in [Deprecations](https://docs.getdbt.com/reference/deprecations.md) and update your project code accordingly. 3. Rerun `dbt parse` to confirm the warnings are resolved, then commit your changes. #### Update a job's version To clear or change an override for a job on a supported environment that's pinned to a legacy version: 1. Open the job's settings and find the **dbt version** setting. 2. Either clear the override so the job inherits the environment's version, or set it to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). 3. Save your changes. #### Validate your migration Moving from a pinned legacy version to a release track can surface behavior changes, dependency issues, or adapter differences that weren't visible before. To reduce risk: * Test in a development environment before upgrading your production and default development environments. * Review your compile, build, and job run results after the change. * Establish a fallback path in case you need to roll back. #### Get help If you hit a blocker you can't resolve, [contact Support](mailto:support@getdbt.com) with your project ID, environment ID, affected job run IDs and logs, your current version, and your target release track. --- ### Move to the dbt platform: Get started [Back to guides](https://docs.getdbt.com/guides.md) Total estimated time: 3-4 hours [Menu ]() #### Introduction Moving from self-hosted dbt to the dbt platform streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. The data layer is the foundation for trusted analytics and AI; the dbt platform gives you the governance, shared definitions, and reliability to scale both — without the hidden cost of self-hosting in engineer hours and wasted compute. Explore our 3-part-guide series on moving from self-hosted dbt to the dbt platform. This series is ideal for users aiming for streamlined workflows and enhanced analytics: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.  What is dbt? * The dbt platform is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. Infrastructure is managed for you — no custom scripts or fragile orchestration. State-aware orchestration only builds what's changed, so you waste less compute and time. Browser-based development and dbt Wizard open up development to analysts, so you're no longer the bottleneck for every change. With end-to-end lineage, shared metric definitions, and CI that catches regressions before production, you spend less time debugging and more time building. The dbt platform also supports: * Development experiences tailored to multiple personas ([Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md)) * Out-of-the-box [CI/CD workflows](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) for consistent metrics * Domain ownership of data with multi-project [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setups * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for easier data discovery and understanding Learn more about [dbt platform features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md). * dbt is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for 'single players' and small technical teams, all development happens on a command-line interface (CLI), and production deployments must be self-hosted and maintained. You absorb the cost of every upgrade, every broken CI run, and every request that pulls you away from real work: maintaining infrastructure, debugging the CI pipeline, and fielding every change that requires CLI access. Compute runs unchecked, upgrades are risky, and there's no easy way to trace what broke or why. This requires significant, costly work that adds up over time to maintain and scale — and without governance, shared definitions, or reliable testing. #### What you'll learn This guide outlines the steps you need to take to move from self-hosted dbt to the dbt platform and highlights the necessary technical changes: * [Account setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=4): Learn how to create a dbt platform account, invite team members, and configure it for your team. * [Data platform setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=5): Find out about connecting your data platform to the dbt platform. * [Git setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=6): Learn to link your dbt project's Git repository with the dbt platform. * [Developer setup:](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7) Understand the setup needed for developing in the dbt platform. * [Environment variables](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8): Discover how to manage environment variables in the dbt platform, including their priority. * [Orchestration setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=9): Learn how to prepare your dbt platform environment and jobs for orchestration. * [Models configuration](https://docs.getdbt.com/guides/dbt-migration-1.md?step=10): Get insights on validating and running your models in the dbt platform, using either the Studio IDE or dbt platform CLI. * [What's next?](https://docs.getdbt.com/guides/dbt-migration-1.md?step=11): Summarizes key takeaways and introduces what to expect in the following guides. ##### Related docs * [Learn dbt](https://learn.getdbt.com) on-demand video learning. * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. #### Prerequisites * You have an existing dbt project connected to a Git repository and data platform supported in the [dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * You have a dbt platform account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**! #### Account setup This section outlines the steps to set up your dbt platform account and configure it for your team. 1. [Create your dbt platform account](https://www.getdbt.com/signup). 2. Provide user [access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) and [invite users](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to your dbt platform account and project. 3. Configure [Single Sign-On (SSO)](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) or [Role-based access control (RBAC)](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control) for easy and secure access. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * This removes the need to save passwords and secret environment variables locally. ##### Additional configuration Explore these additional configurations for performance and reliability improvements: 1. In **Account settings**, enable [partial parsing](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing) to only reparse changed files, saving time. 2. In **Account settings**, enable [Git repo caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) for job reliability & third-party outage protection. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### Data platform setup This section outlines the considerations and methods to connect your data platform to the dbt platform. 1. In the dbt platform, set up your [data platform connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) and [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md). The dbt platform can connect with a variety of data platform providers including: * [AlloyDB](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md) * [Amazon Athena](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-amazon-athena.md) * [Amazon Redshift](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md) * [Apache Spark](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-apache-spark.md) * [Azure Synapse Analytics](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-azure-synapse-analytics.md) * [Databricks](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-databricks.md) * [Google BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md) * [Microsoft Fabric](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-microsoft-fabric.md) * [PostgreSQL](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md) * [Snowflake](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md) * [Starburst or Trino](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-starburst-trino.md) * [Teradata](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-teradata.md) 2. You can verify your data platform connections by clicking the **Test connection** button in your deployment and user credentials settings. ##### Additional configuration Explore these additional configurations to optimize your data platform setup further: 1. Use [OAuth connections](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md), which enables secure authentication using your data platform's SSO. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### Git setup Your existing dbt project source code should live in a Git repository. In this section, you will connect your existing dbt project source code from Git to the dbt platform. 1. Ensure your dbt project is in a Git repository. 2. In **Account settings**, select **Integrations** to [connect your Git repository](https://docs.getdbt.com/docs/platform/git/configure-git.md) to the dbt platform: * (**Recommended**) Connect with one of the [native integrations](https://docs.getdbt.com/docs/platform/git/configure-git.md) in the dbt platform (such as GitHub, GitLab, and Azure DevOps). This method is preferred for its simplicity, security features (including secure OAuth logins and automated workflows like CI builds on pull requests), and overall ease of use. * [Import a Git repository](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) from any valid Git URL that points to a dbt project. #### Developer setup This section highlights the development configurations you'll need for your dbt project. The following categories are covered in this section: * [dbt platform environments](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#dbt-cloud-environments) * [Initial setup steps](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#initial-setup-steps) * [Additional configuration](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#additional-configuration-2) * [dbt platform commands](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#dbt-cloud-commands) ##### dbt environments The most common data environments are production, staging, and development. The way self-hosted dbt manages [environments](https://docs.getdbt.com/docs/environments-in-dbt.md) is through `target`, which are different sets of connection details. [dbt platform environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) go further by: * Integrating with features such as job scheduling or version control, making it easier to manage the full lifecycle of your dbt projects within a single platform. * Streamlining the process of switching between development, staging, and production contexts. * Making it easy to configure environments through the dbt platform UI instead of manually editing the `profiles.yml` file. You can also [set up](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) or [customize](https://docs.getdbt.com/docs/build/custom-target-names.md) target names in the dbt platform. * Adding `profiles.yml` attributes with [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes), which explains where to configure them for deployment and development environments. * Using [Git repo caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) to protect you from third-party outages, Git auth failures, and more. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### Initial setup steps 1. **Set up development environment** — Set up your [development](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment) environment and [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide). You'll need this to access your dbt project and start developing. 2. **dbt version** — In your dbt platform environment, select a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for ongoing dbt version upgrades. If your team plans to use both self-hosted dbt and the dbt platform for developing or deploying your dbt project, you can run `dbt --version` in the command line to find out which version of dbt you're using. * When using self-hosted dbt, you need to think about which version you're using and manage your own upgrades. When using the dbt platform, leverage [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) so you don't have to. 3. **Connect to your data platform** — When using the dbt platform, you can [connect to your data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) directly in the UI. * Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project. 4. **Development tools** — Set up your development workspace with the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) (command line interface or code editor) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (browser-based) to build, test, run, and version control your dbt code in your tool of choice. * If you've previously installed dbt, the [dbt platform CLI installation doc](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md?install=pip#install-dbt-cloud-cli) has more information on how to install the dbt platform CLI, create aliases, or uninstall dbt for a smooth transition. ##### Additional configuration Explore these additional configurations to optimize your developer setup further: 1. **Custom target names** — Using [`custom target.names`](https://docs.getdbt.com/docs/build/custom-target-names.md) in your dbt projects helps identify different environments (like development, staging, and production). While you can specify the `custom target.name` values in your development credentials or orchestration setup, we recommend using [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) as the preferred method. They offer a clearer way to handle different environments and are better supported by dbt's partial parsing feature, unlike using [`{{ target }}` logic](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) which is meant for defining the data warehouse connection. ##### dbt commands 1. Review the [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) supported for dbt platform development. For example, `dbt init` isn't needed in the dbt platform as you can create a new project directly in the dbt platform. #### Environment variables This section will help you understand how to set up and manage dbt platform environment variables for your project. The following categories are covered: * [Environment variables in the dbt platform](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#environment-variables-in-dbt-cloud) * [dbt platform environment variables order of precedence](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#dbt-cloud-environment-variables-order-of-precedence) * [Set environment variables in the dbt platform](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#set-environment-variables-in-dbt-cloud) In the dbt platform, you can set [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) in the dbt platform user interface (UI). Read [Set up environment variables](#set-environment-variables-in-dbt-cloud) for more info. In self-hosted dbt, environment variables, or the [`env_var` function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md), are defined manually by the developer or within the external application running dbt. ##### Environment variables in dbt * dbt platform environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`). * If your self-hosted dbt environment variables don't follow this naming convention, perform a ["find and replace"](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features) in your project to make sure all references to these environment variables contain the proper naming conventions. * The dbt platform secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET`to obscure them in logs and the UI. [![Setting project level and environment level values](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/project-environment-view.png?v=2> "Setting project level and environment level values")](#)Setting project level and environment level values ##### dbt environment variables order of precedence Environment variables in the dbt platform are managed with a clear [order of precedence](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables), allowing users to define values at four levels (highest to lowest order of precedence): * The job level (job override) or in the Studio IDE for an individual developer (personal override). *Highest precedence* * The environment level, which can be overridden by the job level or personal override. * A project-wide default value, which can be overridden by the environment level, job level, or personal override. * The optional default argument supplied to the `env_var` Jinja function in the code. *Lowest precedence* [![Environment variables order of precedence](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/env-var-precdence.png?v=2> "Environment variables order of precedence")](#)Environment variables order of precedence ##### Set environment variables in dbt * To set these variables for an entire project or specific environments, navigate to **Deploy** > **Environments** > **Environment variables** tab. * To set these variables at the job level, navigate to **Deploy** > **Jobs** > **Select your job** > **Settings** > **Advanced settings**. * To set these variables at the personal override level, navigate to **Your profile** > **Credentials** > **Select your project** > **Environment variables**. #### Orchestration setup This section outlines the considerations and methods to set up your dbt platform environments and jobs for orchestration. The following categories are covered in this section: * [dbt platform environments](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#dbt-cloud-environments-1) * [Initial setup steps](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#initial-setup-steps-1) * [Additional configuration](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#additional-configuration-3) * [CI/CD setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#cicd-setup) ##### dbt environments To use the [dbt platform's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md), set up one environment as the production environment. This is the [deployment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) environment. You can set up multiple environments for different stages of your deployment pipeline, such as development, staging/QA, and production. ##### Initial setup steps 1. **dbt version** — In your environment settings, configure the dbt platform with the same dbt version. * Once your full migration is complete, we recommend upgrading your environments to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to always get the latest features and more. You only need to do this once. 2. **Configure your jobs** — [Create jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job. * Note that alongside [jobs in the dbt platform](https://docs.getdbt.com/docs/deploy/jobs.md), discover other ways to schedule and run your dbt jobs with the help of other tools. Refer to [Integrate with other tools](https://docs.getdbt.com/docs/deploy/deployment-tools.md) for more information. ##### Additional configuration Explore these additional configurations to optimize your dbt platform orchestration setup further: 1. **Custom target names** — Use environment variables to set a `custom target.name` for every [corresponding dbt platform job](https://docs.getdbt.com/docs/build/custom-target-names.md) at the environment level. 2. **dbt commands** — Add any relevant [dbt commands](https://docs.getdbt.com/docs/deploy/job-commands.md) to execute your dbt platform jobs runs. 3. **Notifications** — Set up [notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) by configuring email and Slack alerts to monitor your jobs. 4. **Monitoring tools** — Use [monitoring tools](https://docs.getdbt.com/docs/deploy/monitor-jobs.md) like run history, job retries, job chaining, dashboard status tiles, and more for a seamless orchestration experience. 5. **API access** — Create [API auth tokens](https://docs.getdbt.com/docs/dbt-apis/authentication.md) and access to [dbt platform APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) as needed. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") 6. **Catalog** — If you use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and run production jobs with an external orchestrator, ensure your production jobs run `dbt run` or `dbt build` to update and view models and their [metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) in Catalog. Running `dbt compile` alone will not update model metadata. In addition, features like column-level lineage also requires catalog metadata produced through running `dbt docs generate`. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### CI/CD setup Building a custom solution to efficiently check code upon pull requests is complicated. With the dbt platform, you can enable [continuous integration / continuous deployment (CI/CD)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) and configure the dbt platform to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. [![Workflow of continuous integration in dbt](/img/docs/dbt-platform/using-dbt-platform/ci-workflow.png?v=2 "Workflow of continuous integration in dbt")](#)Workflow of continuous integration in dbt This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners. 1. Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in the dbt platform. 2. Create [a CI/CD job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) to automate quality checks before code is deployed to production. 3. Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against. #### Model development and discovery In this section, you'll be able to validate whether your models run or compile correctly in your development tool of choice: The [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). You'll want to make sure you set up your [development environment and credentials](https://docs.getdbt.com/docs/dbt-platform-environments.md#set-developer-credentials). 1. In your [development tool](https://docs.getdbt.com/docs/platform/about-develop-dbt.md) of choice, you can review your dbt project, ensure it's set up correctly, and run some [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md): * Run `dbt compile` to make sure your project compiles correctly. * Run a few models in the Studio IDE or dbt platform CLI to ensure you're experiencing accurate results in development. 2. Once your first job has successfully run in your production environment, use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their data lineage to gain a better understanding of its latest production state. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### What's next? Congratulations on completing the first part of your move to the dbt platform 🎉! You have learned: * How to set up your dbt platform account * How to connect your data platform and Git repository * How to configure your development, orchestration, and CI/CD environments * How to set up environment variables and validate your models For the next steps, you can continue exploring our 3-part-guide series on moving from self-hosted dbt to the dbt platform: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. ##### Related docs * [Learn dbt](https://learn.getdbt.com) video courses for on-demand learning. * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights. * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. * [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of self-hosted dbt and the dbt platform. * Subscribe to the [dbt platform RSS alerts](https://status.getdbt.com/) --- ### Move to the dbt platform: Optimization tips [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Moving from self-hosted dbt to the dbt platform streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. It's not just better tooling — it's about lowering total cost of ownership, powering AI with trusted data, and scaling with governed self-service. Explore our 3-part-guide series on moving from self-hosted dbt to the dbt platform. The series is ideal for users aiming for streamlined workflows and enhanced analytics: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. #### What you'll learn You may have already started your move to the dbt platform and are looking for tips to help you optimize your dbt platform experience. This guide includes tips and caveats for the following areas: * [Adapters and connections](https://docs.getdbt.com/guides/dbt-migration-3.md?step=3) * [Development tools](https://docs.getdbt.com/guides/dbt-migration-3.md?step=4) * [Orchestration](https://docs.getdbt.com/guides/dbt-migration-3.md?step=5) * [Mesh](https://docs.getdbt.com/guides/dbt-migration-3.md?step=6) * [Semantic Layer](https://docs.getdbt.com/guides/dbt-migration-3.md?step=7) * [Catalog](https://docs.getdbt.com/guides/dbt-migration-3.md?step=8) #### Adapters and connections In the dbt platform, you can natively connect to your data platform and test its [connection](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) with a click of a button. This is especially useful for users who are new to the dbt platform or are looking to streamline their connection setup. Here are some tips and caveats to consider: ##### Tips * Manage [dbt versions](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) and ensure team collaboration with dbt platform's one-click feature, eliminating the need for manual updates and version discrepancies. Select a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for ongoing updates, to always stay up to date with fixes and (optionally) get early access to new functionality for your dbt project. * The dbt platform supports a whole host of [cloud providers](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md), including Snowflake, Databricks, BigQuery, Fabric, and Redshift (to name a few). * Use [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) to set a [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) snippet that changes how the dbt platform connects to the data platform. That page explains where to configure them for deployment and development environments. * For example, if you have a field in your `profiles.yml` that you’d like to add to the dbt platform adapter user interface, you can use Extended attributes to set it. ##### Caveats * Not all parameters are available for adapters. * A project can only use one warehouse type. #### Development tools The dbt platform empowers data practitioners to develop in the tool of their choice. It ships with a [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) (local) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (browser-based) to build, test, run, and version control your dbt projects. Both development tools are tailored to suit different audiences and preferences within your team. To streamline your team's workflow, it's important to know who will prefer the Studio IDE and who might lean towards the dbt platform CLI. This section aims to clarify these preferences. ##### Studio IDE A web-based interface for building, testing, running, and version-controlling dbt projects. It compiles dbt code into SQL and executes it directly on your database. The Studio IDE makes developing fast and easy for new and seasoned data practitioners to build and test changes. **Who might prefer the Studio IDE?** * New dbt users or those transitioning from other tools who appreciate a more guided experience through a browser-based interface. * Team members focused on speed and convenience for getting started with a new or existing project. * Individuals who prioritize direct feedback from the Studio IDE, such as seeing unsaved changes. **Key features** * The Studio IDE has simplified Git functionality: * Create feature branches from the branch configured in the development environment. * View saved but not-committed code changes directly in the Studio IDE. * [Format or lint](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) your code with `sqlfluff` or `sqlfmt`. This includes support for adding your custom linting rules. * Allows users to natively [defer to production](https://docs.getdbt.com/docs/platform/about-defer.md#defer-in-dbt-cli) metadata directly in their development workflows, reducing the number of objects. * Support running multiple dbt commands at the same time through [safe parallel execution](https://docs.getdbt.com/reference/dbt-commands.md#parallel-execution), a [feature](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) available in dbt platform's infrastructure. In contrast, local dbt *doesn't support* safe parallel execution for multiple invocations in the same process. The Studio IDE provides a simplified interface that's accessible to all users, regardless of their technical background. However, there are some capabilities that are intentionally not available in the Studio IDE due to its focus on simplicity and ease of use: * Pre-commit for automated checks before *committing* code is not available (yet). * Mass-generating files / interacting with the file system are not available. * Combining/piping commands, such as `dbt run -s (bash command)`, is not available. ##### dbt platform CLI The dbt platform CLI allows you to run dbt [commands](https://docs.getdbt.com/reference/dbt-commands.md#available-commands) against your dbt platform development environment from your local command line. For users who seek full control over their development environment and ideal for those comfortable with the command line. When moving from self-hosted dbt to the dbt platform, make sure you check the `.gitignore` file contains the [necessary folders](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-gitignore-file). dbt doesn't interact with git so the dbt platform doesn't automatically add or verify entries in the `.gitignore` file. Additionally, if the repository already contains dbt code and doesn't require initialization, the dbt platform won't add any missing entries to the `.gitignore file`. **Who might prefer the dbt platform CLI?** * Data practitioners accustomed to working with a specific set of development tooling. * Users looking for granular control over their Git workflows (such as pre-commits for automated checks before committing code). * Data practitioners who need to perform complex operations, like mass file generation or specific command combinations. **Key features** * Allows users to run dbt commands against their dbt platform development environment from their local command line with minimal configuration. * Allows users to natively [defer to production](https://docs.getdbt.com/docs/platform/about-defer.md#defer-in-dbt-cli) metadata directly in their development workflows, reducing the number of objects. * Support running multiple dbt commands at the same time through [safe parallel execution](https://docs.getdbt.com/reference/dbt-commands.md#parallel-execution), a [feature](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) available in dbt platform's infrastructure. In contrast, local dbt *doesn't support* safe parallel execution for multiple invocations in the same process. * Able to use Visual Studio (VS) Code extensions #### Orchestration The dbt platform provides robust orchestration that enables you to schedule, run, and monitor dbt jobs with ease. Here are some tips and caveats to consider when using dbt platform's orchestration features: ##### Tips * Enable [partial parsing](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing) between jobs in the dbt platform to significantly speed up project parsing by only processing changed files, optimizing performance for large projects. * [Run multiple CI/CD](https://docs.getdbt.com/docs/deploy/continuous-integration.md) jobs at the same time which will not block production runs. The Job scheduler automatically cancels stale runs when a newer commit is pushed. This is because each PR will run in its own schema. * The dbt platform automatically [cancels](https://docs.getdbt.com/docs/deploy/job-scheduler.md#run-cancellation-for-over-scheduled-jobs) a scheduled run if the existing run is still executing. This prevents unnecessary, duplicative executions. * Protect you and your data freshness from third-party outages by enabling dbt platform's [Git repository caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching), which keeps a cache of the project's Git repository. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * [Link deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion) across dbt platform projects by configuring your job or using the [Create Job API](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) to do this. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * [Rerun your jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) from the start or the point of failure if your dbt job run completed with a status of **`Error.`** ##### Caveats * To automate the setup and configuration of your dbt platform, you can store your job configurations as code within a repository: * Check out our [Terraform provider.](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) * Alternatively, check out our [jobs-as-code](https://github.com/dbt-labs/dbt-jobs-as-code) repository, which is a tool built to handle dbt platform jobs as a well-defined YAML file. * dbt platform users and external emails can receive notifications if a job fails, succeeds, or is cancelled. To get notifications for warnings, you can create a [webhook subscription](https://docs.getdbt.com/guides/zapier-slack.md) and post to Slack. #### dbt Mesh [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. It allows you to make use of multiple interconnected dbt projects instead of a single large, monolithic project. It enables you to interface and navigate between different projects and models with [cross-project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref), enhancing collaboration and data governance. Here are some tips and caveats to consider when using Mesh: ##### Tips * To dynamically resolve [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref), all developers need to develop with the dbt platform (either with the dbt platform CLI or Studio IDE). Cross-project references aren't natively supported in self-hosted dbt, except by installing the source code from upstream projects [as packages](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) * Link models across projects for a modular and scalable approach for your project and teams. * Manage access to your dbt models both within and across projects using: * **[Groups](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups)** — Organize nodes in your dbt DAG that share a logical connection and assign an owner to the entire group. * **[Model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers)** — Control which other models or projects can reference this model. * **[Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md)** — Enable adoption and deprecation of models as they evolve. * **[Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md)** — Set clear expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. ##### Caveats * To use cross-project references in dbt, each dbt project must correspond to just one dbt platform project. We strongly discourage defining multiple projects for the same codebase, even if you're trying to manage access permissions, connect to different data warehouses, or separate production and non-production data. While this was required historically, features like [Staging environments](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments), Environment-level RBAC (*coming soon*), and [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) will make it unnecessary. * Project dependencies are uni-directional, meaning they go in one direction. This means dbt checks for cycles across projects (circular dependencies) and raise errors if any are detected. However, we are considering support to allow projects to depend on each other in both directions in the future, with dbt still checking for node-level cycles while allowing cycles at the project level. * Everyone in the account can view public model metadata, which helps users find data products more easily. This is separate from who can access the actual data, which is controlled by permissions in the data warehouse. For use cases where even metadata about a reusable data asset is sensitive, we are [considering](https://github.com/dbt-labs/dbt/issues/9340) an optional extension of protected models. Refer to the [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md) for more questions. #### dbt Semantic Layer Leverage the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), powered by MetricFlow, to create a unified view of your business metrics, ensuring consistency across all analytics tools. The data layer is the foundation for AI as well as analytics — shared definitions and lineage give AI and BI the same trusted context so answers are consistent and actionable. Here are some tips and caveats to consider when using Semantic Layer: ##### Tips * Define semantic models and metrics once in the dbt platform with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) (powered by MetricFlow). Reuse them across various analytics platforms, reducing redundancy and errors. * Use the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) to query metrics in downstream tools for consistent, reliable data metrics. * Connect to several data applications, from business intelligence tools to notebooks, spreadsheets, data catalogs, and more, to query your metrics. [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) include Tableau, Google Sheets, Hex, and more. * Use [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to write commonly used queries directly within your data platform, on a schedule. ##### Caveats * Semantic Layer currently supports the Deployment environment for querying. Development querying experience coming soon. * Run queries/semantic layer commands in the dbt platform CLI, however running queries/semantic layer commands in the Studio IDE isn't supported *yet.* * Semantic Layer doesn't support using [Single sign-on (SSO)](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for Semantic Layer [production credentials](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens), however, SSO is supported for development user accounts. Refer to the [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) for more information. #### dbt Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) enhances your ability to discover, understand, and troubleshoot your data assets through rich metadata and lineage visualization. Lineage and discovery are essential for governance and for feeding reliable context to AI workflows. Here are some tips and caveats to consider when using Catalog: ##### Tips * Use the search and filter capabilities in Catalog to quickly locate models, sources, and tests, streamlining your workflow. * View all the [different projects](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) and public models in the account, where the public models are defined, and how they are used to gain a better understanding of your cross-project resources. * Use the [Lenses](https://docs.getdbt.com/docs/explore/explore-projects.md#lenses) feature, which are map-like layers for your DAG, available from your project's lineage graph. Lenses help you further understand your project's contextual metadata at scale, especially to distinguish a particular model or a subset of models. * Access column-level lineage (CLL) for the resources in your dbt project. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### Caveats * There must be at least one successful job run in the production deployment environment for Catalog to populate information. Familiarize yourself with Catalog's features to fully leverage its capabilities to avoid missed opportunities for efficiency gains. Refer to the [Catalog FAQs](https://docs.getdbt.com/docs/explore/dbt-explorer-faqs.md) for more information. #### What's next? Congratulations on making it through the guide 🎉! We hope you're equipped with useful insights and tips to help you with your move. Something to note is that moving from self-hosted dbt to the dbt platform isn't just about evolving your data projects, it's about exploring new levels of collaboration, governance, efficiency, and innovation within your team, and building a data layer that's ready for AI. For the next steps, continue exploring our 3-part-guide series on moving from self-hosted dbt to the dbt platform: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. ##### Resources If you need any additional help or have some questions, use the following resources: * [dbt Learn courses](https://learn.getdbt.com) for on-demand video learning. * Our [Support team](https://docs.getdbt.com/docs/dbt-support.md) is always available to help you troubleshoot your dbt platform issues. * Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices. * Subscribe to the [dbt platform RSS alerts](https://status.getdbt.com/) * Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more. * [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of self-hosted dbt and the dbt platform. For tailored assistance, you can use the following resources: * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and move. --- ### Move to the dbt platform: What you need to know [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Moving from self-hosted dbt to the dbt platform streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. It's not just better tooling — it's about lowering total cost of ownership, powering AI with trusted data, and scaling with governed self-service. Explore our 3-part-guide series on moving from self-hosted dbt to the dbt platform. The series is ideal for users aiming for streamlined workflows and enhanced analytics: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.  What are dbt and dbt v1? * The dbt platform is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports: * Development experiences tailored to multiple personas ([Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md)) * Out-of-the-box [CI/CD workflows](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) for consistent metrics * Domain ownership of data with multi-project [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setups * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for easier data discovery and understanding Learn more about [dbt platform features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md). * dbt is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for 'single players' and small technical teams, all development happens on a command-line interface, and production deployments must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale. #### What you'll learn Today thousands of companies, with data teams ranging in size from 2 to 2,000, rely on the dbt platform to accelerate data work, increase collaboration, and win the trust of the business. Understanding what you'll need to do in order to move between the dbt platform and your current self-hosted dbt deployment will help you strategize and plan for your move. The guide outlines the following steps: * [Considerations](https://docs.getdbt.com/guides/dbt-migration-2.md?step=3): Learn about the most important things you need to think about when moving from self-hosted dbt to the dbt platform. * [Plan your move](https://docs.getdbt.com/guides/dbt-migration-2.md?step=4): Considerations you need to make, such as user roles and permissions, onboarding order, current workflows, and more. * [Move to the dbt platform](https://docs.getdbt.com/guides/dbt-migration-2.md?step=5): Review the steps to move your dbt project to the dbt platform, including setting up your account, data platform, and Git repository. * [Test and validate](https://docs.getdbt.com/guides/dbt-migration-2.md?step=6): Discover how to ensure model accuracy and performance post-move. * [Transition and training](https://docs.getdbt.com/guides/dbt-migration-2.md?step=7): Learn how to fully transition to the dbt platform and what training and support you may need. * [Summary](https://docs.getdbt.com/guides/dbt-migration-2.md?step=8): Summarizes key takeaways and what you've learned in this guide. * [What's next?](https://docs.getdbt.com/guides/dbt-migration-2.md?step=9): Introduces what to expect in the following guides. #### Considerations If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. This guide shares the technical adjustments and team collaboration strategies you'll need to know to move your project from self-hosted dbt to the dbt platform. Each "build your own" deployment of dbt will look a little different, but after seeing hundreds of teams make the migration, there are many things in common. The most important things you need to think about when moving from self-hosted dbt to the dbt platform: * How is your team structured? Are there natural divisions of domain? * Should you have one project or multiple? Which dbt resources do you want to standardize & keep central? * Who should have permission to view, develop, and administer? * How are you scheduling your dbt models to run in production? * How are you currently managing Continuous integration/Continuous deployment (CI/CD) of logical changes (if at all)? * How do your data developers prefer to work? * How do you manage different data environments and the different behaviors in those environments? The dbt platform provides standard mechanisms for tackling these considerations, all of which deliver long-term benefits to your organization: * Cross-team collaboration * Access control * Orchestration * Isolated data environments If you have rolled out your own self-hosted dbt deployment, you have probably come up with different answers. #### Plan your move As you plan your move, consider your workflow and team layout to ensure a smooth transition. Here are some key considerations to keep in mind:  Start small to minimize risk and maximize learning You don't need to move every team and every developer's workflow all at once. Many customers with large dbt deployments start by moving one team and one project. Once the benefits of a consolidated platform are clear, move the rest of your teams and workflows. While long-term 'hybrid' deployments can be challenging, it may make sense as a temporary on-ramp.  User roles and responsibilities Assess the users or personas involved in the pre-move, during the move, and post-move. * **Administrators**: Plan for new [access controls](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) in the dbt platform, such as deciding what teams can manage themselves and what should be standardized. Determine who will be responsible for setting up and maintaining projects, data platform connections, and environments. * **Data developers** (data analysts, data engineers, analytics engineers, business analysts): Determine onboarding order, workflow adaptation in the dbt platform, training on [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) usage, and role changes. * **Data consumers:** Discover data insights by using [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's resources (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")  Onboarding order If you have multiple teams of dbt developers, think about how to start your onboarding sequence for the dbt platform: * Start with downstream (like business-embedded teams) who may benefit from the Studio IDE as dev experience (less technical users) and sharing features (like auto-deferral and Catalog) to share with their stakeholders, moving to more technical teams later. * Consider setting up a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) in the dbt platform (even before development or production jobs) to streamline development workflows. This is especially beneficial if there's no existing CI process.  Analyze current workflows, review processes, and team structures Discover how the dbt platform can help simplify development, orchestration, and testing: * **Development**: Develop dbt models, allowing you to build, test, run, and version control your dbt projects using the dbt platform CLI (command line interface or code editor) or Studio IDE (browser-based). * **Orchestration**: Create custom schedules to run your production jobs. Schedule jobs by day of the week, time of day, or a recurring interval. * Set up [a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) to ensure developer effectiveness, and CD jobs to deploy changes as soon as they're merged. * Link deploy jobs together by [triggering a job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion) when another one is completed. * For the most flexibility, use the [dbt platform API](https://docs.getdbt.com/dbt-cloud/api-v2#/) to trigger jobs. This makes sense when you want to integrate dbt execution with other data workflows. * **Continuous integration (CI)**: Use [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. This build-on-PR functionality is a great way to catch bugs before deploying to production. * For many teams, dbt platform CI represents a major improvement compared to their previous development workflows. * **How are you defining tests today?**: While testing production data is important, it's not the most efficient way to catch logical errors introduced by developers You can use [unit testing](https://docs.getdbt.com/docs/build/unit-tests.md) to allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production.  Understand access control Transition to the dbt platform's [access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) mechanisms to ensure security and proper access management. dbt platform administrators can use the dbt platform's permission model to control user-level access in a dbt platform account: * **License-based access controls:** Users are configured with account-wide license types. These licenses control the things a user can do within the application: view project metadata, develop changes within those projects, or administer access to those projects. * **Role-based Access Control (RBAC):** Users are assigned to *groups* with specific permissions on specific projects or all projects in the account. A user may be a member of multiple groups, and those groups may have permissions on multiple projects. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")  Manage environments If you require isolation between production and pre-production data environments due to sensitive data, the dbt platform can support Development, Staging, and Production data [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md). This provides developers with the benefits of an enhanced workflow while ensuring isolation between Staging and Production data, and locking down permissions on Prod. #### Move to dbt This guide is your roadmap to help you think about migration strategies and what moving from self-hosted dbt to the dbt platform could look like. After reviewing the considerations and planning your move, you may want to start moving your dbt project to the dbt platform: * Check out the detailed [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) guide for useful tasks and insights for a smooth transition from self-hosted dbt to the dbt platform. For a more detailed comparison of self-hosted dbt and the dbt platform, check out [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud). #### Test and validate After [setting the foundations of the dbt platform](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1), it's important to validate your migration to ensure seamless functionality and data integrity: * **Review your dbt project:** Ensure your project compiles correctly and that you can run commands. Make sure your models are accurate and monitor performance post-move. * **Start cutover:** You can start the cutover to the dbt platform by creating a dbt platform job with commands that only run a small subset of the DAG. Validate the tables are being populated in the proper database/schemas as expected. Then continue to expand the scope of the job to include more sections of the DAG as you gain confidence in the results. * **Precision testing:** Use [unit testing](https://docs.getdbt.com/docs/build/unit-tests.md) to allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production. * **Access and permissions**: Review and adjust [access controls and permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) within the dbt platform to maintain security protocols and safeguard your data. #### Transition and training Once you've confirmed that dbt platform orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you're still using an external orchestrator (such as Airflow), and you've swapped out local dbt execution for dbt platform execution (through the [API](https://docs.getdbt.com/docs/dbt-apis/overview.md)). Familiarize your team with the dbt platform's [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) and optimize development and deployment processes. Some key features to consider include: * **Release tracks:** Choose a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for automatic dbt version upgrades, at the cadence appropriate for your team — removing the hassle of manual updates and the risk of version discrepancies. You can also get early access to new functionality, ahead of self-hosted dbt. * **Development tools**: Use the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) to build, test, run, and version control your dbt projects. * **Documentation and Source freshness:** Automate storage of [documentation](https://docs.getdbt.com/docs/build/documentation.md) and track [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) in the dbt platform, which streamlines project maintenance. * **Notifications and logs:** Receive immediate [notifications](https://docs.getdbt.com/docs/deploy/monitor-jobs.md) for job failures, with direct links to the job details. Access comprehensive logs for all job runs to help with troubleshooting. * **CI/CD:** Use the dbt platform's [CI/CD](https://docs.getdbt.com/docs/deploy/ci-jobs.md) feature to run your dbt projects in a temporary schema whenever new commits are pushed to open pull requests. This helps with catching bugs before deploying to production. ##### Beyond your move Now that you've chosen the dbt platform as your platform, you've unlocked the power of streamlining collaboration, enhancing workflow efficiency, and leveraging powerful [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) for analytics engineering teams. Here are some additional features you can use to unlock the full potential of the dbt platform: * **Audit logs:** Use [audit logs](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md) to review actions performed by people in your organization. Audit logs contain audited user and system events in real time. You can even [export](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md#exporting-logs) *all* the activity (beyond the 90 days you can view in the dbt platform). [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **dbt platform APIs:** Use the dbt platform's robust [APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) to create, read, update, and delete (CRUD) projects/jobs/environments project. The [dbt platform Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) facilitate programmatic access and configuration storage. While the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) offers extensive metadata querying capabilities, such as job data, model configurations, usage, and overall project health. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **Catalog**: Use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their [lineage](https://docs.getdbt.com/terms/data-lineage) to gain a better understanding of its latest production state. (Once you have a successful job in a Production environment). [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **dbt Semantic Layer:** The [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) allows you to define universal metrics on top of your models that can then be queried in your [business intelligence (BI) tool](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). This means no more inconsistent metrics — there's now a centralized way to define these metrics and create visibility in every component of the data flow. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **dbt Mesh:** Use [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) to share data models across organizations, enabling data teams to collaborate on shared data models and leverage the work of other teams. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### Additional help * **dbt Learn courses**: Access our free [Learn dbt](https://learn.getdbt.com) video courses for on-demand training. * **dbt Community:** Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices. * **dbt Support team:** Our [dbt Support team](https://docs.getdbt.com/docs/dbt-support.md) is always available to help you troubleshoot your dbt issues. Create a support ticket in the dbt platform and we'll be happy to help! * **Account management** Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### Summary This guide should now have given you some insight and equipped you with a framework for moving from self-hosted dbt to the dbt platform. This guide has covered the following key areas: * **Considerations:** Understanding the foundational steps required for a successful migration, including evaluating your current setup and identifying key considerations unique to your team's structure and workflow needs. * **Plan you move**: Highlighting the importance of workflow redesign, role-specific responsibilities, and the adoption of new processes to harness the dbt platform's collaborative and efficient environment. * **Move to the dbt platform**: Linking to [the guide](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) that outlines technical steps required to transition your dbt project to the dbt platform, including setting up your account, data platform, and Git repository. * **Test and validate**: Emphasizing technical transitions, including testing and validating your dbt projects within the dbt platform ecosystem to ensure data integrity and performance. * **Transition and training**: Share useful transition, training, and onboarding information for your team. Fully leverage the dbt platform's capabilities, from development tools (dbt platform CLI and Studio IDE) to advanced features such as Catalog, the Semantic Layer, and Mesh. #### What's next? Congratulations on finishing this guide, we hope it's given you insight into the considerations you need to take to best plan your move to the dbt platform. For the next steps, you can continue exploring our 3-part-guide series on moving from self-hosted dbt to the dbt platform: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. ##### Related content * [Learn dbt](https://learn.getdbt.com) courses * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. * [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of self-hosted dbt and the dbt platform. * Subscribe to the [dbt platform RSS alerts](https://status.getdbt.com/) --- ### Optimize and troubleshoot dbt models on Databricks [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Building on the [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md) guide, we'd like to discuss performance optimization. In this follow-up post, we outline simple strategies to optimize for cost, performance, and simplicity when you architect data pipelines. We’ve encapsulated these strategies in this acronym-framework: * Platform Components * Patterns & Best Practices * Performance Troubleshooting #### Platform Components As you start to develop your dbt projects, one of the first decisions you will make is what kind of backend infrastructure to run your models against. Databricks offers SQL warehouses, All-Purpose Compute, and Jobs Compute, each optimized to workloads they are catered to. Our recommendation is to use Databricks SQL warehouses for all your SQL workloads. SQL warehouses are optimized for SQL workloads when compared to other compute options, additionally, they can scale both vertically to support larger workloads and horizontally to support concurrency. Also, SQL warehouses are easier to manage and provide out-of-the-box features such as query history to help audit and optimize your SQL workloads. Between Serverless, Pro, and Classic SQL Warehouse types that Databricks offers, our standard recommendation for you is to leverage Databricks serverless warehouses. You can explore features of these warehouse types in the [Compare features section](https://www.databricks.com/product/pricing/databricks-sql?_gl=1*2rsmlo*_ga*ZmExYzgzZDAtMWU0Ny00N2YyLWFhYzEtM2RhZTQzNTAyZjZi*_ga_PQSEQ3RZQC*MTY3OTYwMDg0Ni4zNTAuMS4xNjc5NjAyMDMzLjUzLjAuMA..&_ga=2.104593536.1471430337.1679342371-fa1c83d0-1e47-47f2-aac1-3dae43502f6b) on the Databricks pricing page. With serverless warehouses, you greatly decrease spin-up time waiting for the cluster to warm up and scale time when your cluster needs to horizontally scale. This mitigates the need to keep clusters idle as serverless warehouses will spin up quickly when the workload begins and then spin down when the workload is complete. Plus, serverless warehouses leverage our Photon engine out of the box for optimal performance in both ELT and serving workloads. The next step would be to decide how big to make your serverless SQL warehouse. This is not an exact science but these subsections provide you with some quick tips that will drive huge improvements in performance. ##### Sizing your SQL warehouses To select the appropriate size of your SQL warehouse, consider the use case and workload you are running and its corresponding latency requirements. You can select a T-shirt size based on the amount of data and auto-scaling based on concurrency needs. A good rule of thumb to follow is to start with a Medium warehouse and work from there. For large and complex workloads, bigger warehouses are the way to go and that won’t necessarily mean higher costs. This is because larger warehouses take a shorter time to complete a unit of work. For example, if a Small warehouse takes an hour to complete a pipeline, it will only take half an hour with a Medium. This linear trend continues as long as there’s enough work for the warehouse to perform. ##### Provision warehouses by workload Another technique worth implementing is to provision separate SQL warehouses for building dbt pipelines instead of ad hoc, interactive SQL analysis. This is because the query design patterns and compute usage are different for these two types of workloads. Choose T-shirt sizes based on data volumes and SLAs (scale-up principle), and choose auto-scaling based on concurrency requirements (scale-out principle). For larger deployments, this approach could be expanded to map different workload sizes to multiple “pipeline” warehouses, if needed. On the dbt side, take into account the [number of threads you have](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads), meaning how many dbt models you can run in parallel. The higher the thread count, the more compute you will require. ##### Configure auto-stop Because of the ability of serverless warehouses to spin up in a matter of seconds, setting your auto-stop configuration to a lower threshold will not impact SLAs and end-user experience. From the SQL Workspace UI, the default value is 10 minutes and  you can set it to 5 minutes for a lower threshold with the UI. If you would like more custom settings, you can set the threshold to as low as 1 minute with the [API](https://docs.databricks.com/sql/api/sql-endpoints.html#). #### Patterns & Best Practices Now that we have a solid sense of the infrastructure components, we can shift our focus to best practices and design patterns on pipeline development.  We recommend the staging/intermediate/mart approach which is analogous to the medallion architecture bronze/silver/gold approach that’s recommended by Databricks. Let’s dissect each stage further. dbt has guidelines on how you can [structure your dbt project](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) which you can learn more about. ##### Bronze / Staging Layer: There are a few different options for materializing bronze delta tables on Databricks. In the recommended dbt workflow, you should load your flat files into a table first before using dbt to transform on it. To do so, you can use an EL tool to handle this ingestion. However, we know this isn't always possible so for data sets in cloud storage, we recommend that you either leverage our `COPY INTO` functionality or stage the external table. In terms of the `COPY INTO` approach, you would have a few different options. The first option would be to run the `COPY INTO` logic as a pre-hook before building your silver/intermediate models. The second option would be to invoke the databricks `COPY INTO` macro with `dbt run-operation` and then subsequently execute your model runs. You can see an example implementation of the [COPY INTO macro](https://github.com/databricks/dbt-databricks/blob/main/docs/databricks-copy-into-macro-aws.md) in the dbt-databricks docs. The main benefit of leveraging `COPY INTO` is that it's an incremental operation and it ensures that data is written in Delta format (when we refer to Delta, we are simply referring to the open Parquet tables with a transaction log). If you instead opt to stage an external table, the bronze table retains its raw structure (whether it is CSV, Parquet, JSON, etc.). This would prevent the ability to leverage the performance, reliability, and governance advantages inherent in Delta. Further, external Parquet tables require additional manual work such as running repair operations to ensure new partition metadata is accounted for. Nevertheless, staging external tables could be a feasible option if you are migrating to Databricks from another cloud warehouse system, where you heavily leveraged this functionality. ##### Silver / Intermediate Layer Now that we have our bronze table taken care of, we can proceed with the silver layer. For cost and performance reasons, many customers opt to implement an incremental pipeline approach. The main benefit with this approach is that you process a lot less data when you insert new records into the silver layer, rather than re-create the table each time with all the data from the bronze layer. However it should be noted that by default, [dbt recommends using views and tables](https://docs.getdbt.com/best-practices/materializations/1-guide-overview.md) to start out with and then moving to incremental as you require more performance optimization. dbt has an [incremental model materialization](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#the-merge-strategy) to facilitate this framework. How this works at a high level is that Databricks will create a temp view with a snapshot of data and then merge that snapshot into the silver table. You can customize the time range of the snapshot to suit your specific use case by configuring the `where` conditional in your `is_incremental` logic. The most straightforward implementation is to merge data using a timestamp that’s later than the current max timestamp in the silver table, but there are certainly valid use cases for increasing the temporal range of the source snapshot. While merge should be fairly performant out of the box but if you have particularly tight SLAs, there are some more advanced tuning techniques that you can incorporate into your logic. Let us discuss several examples in further detail. ##### File Compaction Most compute engines work best when file sizes are between 32 MB and 256 MB. In Databricks, we take care of optimal file sizing under the hood with our [auto optimize](https://docs.databricks.com/optimizations/auto-optimize.html) features. Auto optimize consists of two distinct features: auto compaction and optimized writes. In Databricks SQL warehouses, optimized writes are enabled by default. We recommend that you [opt in to auto compaction](https://docs.databricks.com/optimizations/auto-optimize.html#when-to-opt-in-to-auto-compaction). ##### Data skipping Under the hood, Databricks will naturally [cluster data based on when it was ingested](https://www.databricks.com/blog/2022/11/18/introducing-ingestion-time-clustering-dbr-112.html). Since many queries include timestamps in `where` conditionals, this will naturally lead to a large amount of file skipping for enhanced performance. Nevertheless, if you have other high cardinality columns (basically columns with a large amount of distinct values such as id columns) that are frequently used in `join` keys or `where` conditionals, performance can typically be augmented further by leveraging Z-order. The SQL syntax for the Z-Order command is `OPTIMIZE table_name ZORDER BY (col1,col2,col3,etc)`. One caveat to be aware of is that you will rarely want to Z-Order by more than three columns. You will likely want to either run Z-order on run end after your model builds or run Z-Order as a separate scheduled job on a consistent cadence, whether it is daily, weekly, or monthly. ```sql config( materialized='incremental', zorder="column_A" | ["column_A", "column_B"] ) ``` ##### Analyze Table The `ANALYZE TABLE` command ensures that our system has the most up-to-date statistics to select the optimal join plan. You will likely want to either run analyze table posthook after your model builds or run analyze table as a separate scheduled dbt job on a consistent cadence, whether it is daily, weekly, or monthly.  The SQL syntax for this is: ```sql ANALYZE TABLE mytable COMPUTE STATISTICS FOR COLUMNS col1, col2, col3 ``` An important item to clarify is that you will want to prioritize statistics for columns that are frequently used in joins. ##### Vacuum When you delete a record from a Delta table, it is a soft delete. What this means is that the record is deleted from the transaction log and is not included in subsequent queries, but the underlying file still remains in cloud storage. If you want to delete the underlying files as well (whether for reducing storage cost or augmenting performance on merges), you can run a vacuum command. The factor you will want to be very cognizant of is restoring older versions of the table. Let’s say  you vacuum a table to delete all unused files that’s older than 7 days. You won’t be  able to restore versions of the table from over 7 days ago that rely on those deleted  files, so use with caution. If/when you choose to leverage vacuum, you will likely want to run vacuum using the dbt functionality [on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) after your model builds or run vacuum as a separate scheduled dbt job on a consistent cadence (whether it is daily, weekly, or monthly) using the dbt [run-operation](https://docs.getdbt.com/reference/commands/run-operation.md) command (with the vacuum statement in a macro). ##### Gold / Marts Layer Now onto the most final layer — the gold marts that business stakeholders typically interact with from their preferred BI tool. The considerations here will be fairly similar to the silver layer except that these marts are more likely to handling aggregations. Further, you will likely want to be even more intentional about Z-Ordering these tables as SLAs tend to be lower with these direct stakeholder facing tables. In addition, these tables are well suited for defining [metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) on to ensure simplicity and consistency across your key business KPIs! Using the [MetricFlow](https://github.com/dbt-labs/metricflow), you can query the metrics inside of your own dbt project even. With the upcoming Semantic Layer Integration, you can also then query the metrics in any of the partner integrated tools. ##### Filter rows in target and/or source It can be done using `incremental_predicates` like in this example: ```sql {{ config( materialized='incremental', incremental_strategy = 'merge', unique_key = 'id', incremental_predicates = [ "dbt_internal_target.create_at >= '2023-01-01'", "dbt_internal_source.create_at >= '2023-01-01'"], ) }} ``` #### Performance Troubleshooting Performance troubleshooting refers to the process of identifying and resolving issues that impact the performance of your dbt models and overall data pipelines. By improving the speed and performance of your Lakehouse platform, you will be able to process data faster, process large and complex queries more effectively, and provide faster time to market.  Let’s go into detail the three effective strategies that you can implement. ##### SQL warehouse query profile The SQL warehouse query profile is an effective tool found inside the Databricks SQL workspace. It’s used to troubleshoot slow-running queries, optimize query execution plans, and analyze granular metrics to see where compute resources are being spent. The query profile includes these high level capability areas: * Detailed information about the three main components of query execution, which are time spent in tasks, number of rows processed, and memory consumption. * Two types of graphical representations. A tree view to easily spot slow operations at a glance, and a graph view that breaks down how data is transformed across tasks. * Ability to understand mistakes and performance bottlenecks in queries. The three common examples of performance bottlenecks that can be surfaced by the query profile are: ##### Inefficient file pruning By default, Databricks Delta tables collect statistics on the *first 32 columns* defined in your table schema. When transforming data from the Bronze/staging layer to the Silver/intermediate layer, it is advised to reorder your columns to account for these file-level stats and improve overall performance. Move numerical keys and high cardinality query predicates to the left of the 32nd ordinal position, and move strings and complex data types after the 32nd ordinal position of the table. It is worth mentioning that while you can change the default table property to collect statistics on more columns, it will add more overhead as you write files. You may change this default value by using the [table property](https://docs.databricks.com/delta/table-properties.html), `delta.dataSkippingNumIndexedCols`. ##### Full Table Scans The Query Profile provides metrics that allow you to identify the presence of full table scans. Full table scans is a query operation that involves scanning the entire table to retrieve records. It can be a performance issue especially for large tables with billions or trillions of rows. This is because scanning an entire table can be time-consuming and resource-intensive, leading to high memory and CPU usage and slower response times. Table layout techniques such as file compaction and Z-Ordering described in the earlier section of this article will help alleviate this problem. ##### Exploding Joins The concept of *exploding joins* refers to a `join` operation that produces a much larger table result set than either of the input tables used, resulting in a Cartesian product. This performance issue can be determined by enabling the verbose mode setting in the Query Profile, by looking at the number of records produced by a join operator. There are several steps you can take to prevent exploding joins. As a first step, make the join conditions more specific to reduce the number of rows that are being matched. Another step is to utilize data preprocessing techniques such as aggregating, filtering, and performing data sampling before the join operation. These techniques can reduce the size of the input tables and help prevent exploding joins. ##### Materialization Best Practices   Remember that data is stored as files, so the unit of I/O work is a file, not a row. That’s a lot of work if we’re dealing with TBs of data. Therefore we recommend relying on merge strategy as the recommended strategy for the majority of incremental models. Databricks is committed to continuously improving its performance. For example, in Delta and DBSQL, we’ve greatly improved performance of MERGE operations recently with [low-shuffle merge and Photon](https://www.databricks.com/blog/2022/10/17/faster-merge-performance-low-shuffle-merge-and-photon.html). With many future implementations in the pipeline such as deletion vectors for efficient deletes & upserts. Here’s the basic strategies to speed it up: 1. Only read partitions that are important by pushing down filters to scan source and target using filters in *model* and *incremental\_predicates* 2. Only update important rows 3. Improve key lookup by defining only *one* materialized key 4. Only update important columns ##### dbt Discovery API Now you might be wondering, how do you identify opportunities for performance improvement inside of dbt? Well, with each job run, dbt generates metadata on the timing, configuration, and freshness of models in your dbt project. The [dbt Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) is a GraphQL service that supports queries on this metadata, using  the [graphical explorer](https://metadata.cloud.getdbt.com/graphiql) or the endpoint itself. Teams can pipe this data into their data warehouse and analyze it like any other data source in a business intelligence platform. dbt users can also use the data from the [Model Timing tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing) to visually identify models that take the most time and may require refactoring. ##### dbt Admin API With the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md), you can  pull the dbt artifacts from your dbt run,  put the generated `manifest.json` into an S3 bucket, stage it, and model the data using the [dbt artifacts package](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/). That package can help you identify inefficiencies in your dbt models and pinpoint where opportunities for improvement are. ##### Conclusion This builds on the content in [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). We welcome you to try these strategies on our example open source TPC-H implementation and to provide us with thoughts/feedback as you start to incorporate these features into production. Looking forward to your feedback on [#db-databricks-and-spark](https://getdbt.slack.com/archives/CNGCW8HKL) Slack channel! --- ### Post to Microsoft Teams when a job finishes [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide will show you how to set up an integration between dbt jobs and Microsoft Teams using [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) and Zapier, similar to the [native Slack integration](https://docs.getdbt.com/docs/deploy/job-notifications.md#slack-notifications). Want Microsoft Teams notifications without Zapier? If you only need job status alerts in Microsoft Teams (for example, “job succeeded/failed”) and *don’t* need to process webhook payloads, you can use **Job notifications** instead by sending notifications to a Teams channel email address (External Email). See **Job notifications**: [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications) When a dbt job finishes running, the integration will: * Receive a webhook notification in Zapier, * Extract the results from the dbt admin API, and * Post a summary to a Microsoft Teams channel. [![Screenshot of a message in MS Teams showing a summary of a dbt run which failed](/img/guides/orchestration/webhooks/zapier-ms-teams/ms-teams-ui.png?v=2 "Screenshot of a message in MS Teams showing a summary of a dbt run which failed")](#)Screenshot of a message in MS Teams showing a summary of a dbt run which failed ##### Prerequisites In order to set up the integration, you should have familiarity with: * [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * Zapier #### Set up the connection between Zapier and Microsoft Teams * Install the [Zapier app in Microsoft Teams](https://appsource.microsoft.com/en-us/product/office/WA200002044) and [grant Zapier access to your account](https://zapier.com/blog/how-to-automate-microsoft-teams/). **Note**: To receive the message, add the Zapier app to the team's channel during installation. #### Create a new Zap in Zapier Use **Webhooks by Zapier** as the Trigger, and **Catch Raw Hook** as the Event. If you don't intend to [validate the authenticity of your webhook](https://docs.getdbt.com/docs/deploy/webhooks.md#validate-a-webhook) (not recommended!) then you can choose **Catch Hook** instead. Press **Continue**, then copy the webhook URL. [![Screenshot of the Zapier UI, showing the webhook URL ready to be copied](/img/guides/orchestration/webhooks/zapier-common/catch-raw-hook.png?v=2 "Screenshot of the Zapier UI, showing the webhook URL ready to be copied")](#)Screenshot of the Zapier UI, showing the webhook URL ready to be copied ##### 3. Configure a new webhook in dbt See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Choose either **Run completed** or **Run errored**, but not both, or you'll get double messages when a run fails. Make note of the Webhook Secret Key for later. Once you've tested the endpoint in dbt, go back to Zapier and click **Test Trigger**, which will create a sample webhook body based on the test event dbt sent. The sample body's values are hard-coded and not reflective of your project, but they give Zapier a correctly-shaped object during development. #### Store secrets In the next step, you will need the Webhook Secret Key from the prior step, and a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps), which prevents your keys from being displayed in plaintext in the Zap code. You will be able to access them via the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the names for the secret keys are: `DBT_CLOUD_SERVICE_TOKEN` and `DBT_WEBHOOK_KEY`. If you're using different names, make sure you update all references to it in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ###### a. Create a Storage by Zapier connection If you haven't already got one, go to <https://zapier.com/app/connections/storage> and create a new connection. Remember the UUID secret you generate for later. ###### b. Add a temporary code step Choose **Run Python** as the Event. Run the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_WEBHOOK_KEY', 'abc123') #replace with webhook secret store.set('DBT_CLOUD_SERVICE_TOKEN', 'abc123') #replace with your dbt API token ``` Test the step. You can delete this Action when the test succeeds. The key will remain stored as long as it is accessed at least once every three months. #### Add a code action Select **Code by Zapier** as the App, and **Run Python** as the Event. In the **Set up action** area, add two items to **Input Data**: `raw_body` and `auth_header`. Map those to the `1. Raw Body` and `1. Headers Http Authorization` fields from the **Catch Raw Hook** step above. [![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/img/guides/orchestration/webhooks/zapier-common/run-python.png?v=2 "Screenshot of the Zapier UI, showing the mappings of raw_body and auth_header")](#)Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header In the **Code** field, paste the following code, replacing `YOUR_SECRET_HERE` with the secret you created when setting up the Storage by Zapier integration. Remember that this is not your dbt secret. The code below will validate the authenticity of the request, extract the run logs for the completed job from the Admin API, and then build a summary message that pulls out any error messages from the end-of-invocation logs created by dbt. ````python import hashlib import hmac import json import re auth_header = input_data['auth_header'] raw_body = input_data['raw_body'] # Access secret credentials secret_store = StoreClient('YOUR_SECRET_HERE') hook_secret = secret_store.get('DBT_WEBHOOK_KEY') api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') # Validate the webhook came from dbt signature = hmac.new(hook_secret.encode('utf-8'), raw_body.encode('utf-8'), hashlib.sha256).hexdigest() if signature != auth_header: raise Exception("Calculated signature doesn't match contents of the Authorization header. This webhook may not have been sent from dbt.") full_body = json.loads(raw_body) hook_data = full_body['data'] # Steps derived from these commands won't have their error details shown inline, as they're messy commands_to_skip_logs = ['dbt source', 'dbt docs'] # When testing, you will want to hardcode run_id and account_id to IDs that exist; the sample webhook won't work. run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Admin API url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() run_data_results = run_data_response.json()['data'] # Overall run summary outcome_message = f""" **[{hook_data['runStatus']} for Run #{run_id} on Job \"{hook_data['jobName']}\"]({run_data_results['href']})** **Environment:** {hook_data['environmentName']} | **Trigger:** {hook_data['runReason']} | **Duration:** {run_data_results['duration_humanized']} """ # Step-specific summaries for step in run_data_results['run_steps']: if step['status_humanized'] == 'Success': outcome_message += f""" ✅ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ else: outcome_message += f""" ❌ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ show_logs = not any(cmd in step['name'] for cmd in commands_to_skip_logs) if show_logs: full_log = step['logs'] # Remove timestamp and any colour tags full_log = re.sub('\x1b?\[[0-9]+m[0-9:]*', '', full_log) summary_start = re.search('(?:Completed with \d+ error.* and \d+ warnings?:|Database Error|Compilation Error|Runtime Error)', full_log) line_items = re.findall('(^.*(?:Failure|Error) in .*\n.*\n.*)', full_log, re.MULTILINE) if len(line_items) == 0: relevant_log = f'```{full_log[summary_start.start() if summary_start else 0:]}```' else: relevant_log = summary_start[0] for item in line_items: relevant_log += f'\n```\n{item.strip()}\n```\n' outcome_message += f""" {relevant_log} """ # Zapier looks for the `output` dictionary for use in subsequent steps output = {'outcome_message': outcome_message} ```` #### Add the Microsoft Teams action Select **Microsoft Teams** as the App, and **Send Channel Message** as the Action. In the **Set up action** area, choose the team and channel. Set the **Message Text Format** to **markdown**, then put **2. Outcome Message** from the Run Python in Code by Zapier output into the **Message Text** field. [![Screenshot of the Zapier UI, showing the mappings of prior steps to an MS Teams message](/img/guides/orchestration/webhooks/zapier-ms-teams/ms-teams-zap-config.png?v=2 "Screenshot of the Zapier UI, showing the mappings of prior steps to an MS Teams message")](#)Screenshot of the Zapier UI, showing the mappings of prior steps to an MS Teams message #### Test and deploy As you have gone through each step, you should have tested the outputs, so you can now try posting a message into your Teams channel. When you're happy with it, remember to ensure that your `run_id` and `account_id` are no longer hardcoded, then publish your Zap. ##### Other notes * If you post to a chat instead of a team channel, you don't need to add the Zapier app to Microsoft Teams. * If you post to a chat instead of a team channel, note that markdown is not supported and you will need to remove the markdown formatting. * If you chose the **Catch Hook** trigger instead of **Catch Raw Hook**, you will need to pass each required property from the webhook as an input instead of running `json.loads()` against the raw body. You will also need to remove the validation code. --- ### Post to Slack with error context when a job fails [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide will show you how to set up an integration between dbt jobs and Slack using [dbt webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) and Zapier. It builds on the native [native Slack integration](https://docs.getdbt.com/docs/deploy/job-notifications.md#slack-notifications) by attaching error message details of models and tests in a thread. Note: Because there is not a webhook for Run Cancelled, you may want to keep the standard Slack integration installed to receive those notifications. You could also use the [alternative integration](#alternate-approach) that augments the native integration without replacing it. When a dbt job finishes running, the integration will: * Receive a webhook notification in Zapier * Extract the results from the dbt admin API * Post a brief summary of the run to a Slack channel * Create a threaded message attached to that post which contains any reasons that the job failed [![Screenshot of a message in Slack showing a summary of a dbt run which failed](/img/guides/orchestration/webhooks/zapier-slack/slack-thread-example.png?v=2 "Screenshot of a message in Slack showing a summary of a dbt run which failed")](#)Screenshot of a message in Slack showing a summary of a dbt run which failed ##### Prerequisites In order to set up the integration, you should have familiarity with: * [dbt webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * Zapier #### Create a new Zap in Zapier 1. Use **Webhooks by Zapier** as the Trigger, and **Catch Raw Hook** as the Event. If you don't intend to [validate the authenticity of your webhook](https://docs.getdbt.com/docs/deploy/webhooks.md#validate-a-webhook) (not recommended!) then you can choose **Catch Hook** instead. 2. Click **Continue**, then copy the webhook URL. [![Screenshot of the Zapier UI, showing the webhook URL ready to be copied](/img/guides/orchestration/webhooks/zapier-common/catch-raw-hook.png?v=2 "Screenshot of the Zapier UI, showing the webhook URL ready to be copied")](#)Screenshot of the Zapier UI, showing the webhook URL ready to be copied #### Configure a new webhook in dbt See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Choose **Run completed** as the Event. You can alternatively choose **Run errored**, but you will need to account for the fact that the necessary metadata [might not be available immediately](https://docs.getdbt.com/docs/deploy/webhooks.md#completed-errored-event-difference). Remember the Webhook Secret Key for later. Once you've tested the endpoint in dbt, go back to Zapier and click **Test Trigger**. This creates a sample webhook body based on the test event dbt sent. The sample body's values are hardcoded and not reflective of your project, but they give Zapier a correctly-shaped object during development. #### Store secrets In the next step, you will need the Webhook Secret Key from the prior step, and a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps). This prevents your keys from being displayed as plaintext in the Zap code. You can access them with the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the names for the secret keys are: `DBT_CLOUD_SERVICE_TOKEN` and `DBT_WEBHOOK_KEY`. If you're using different names, make sure you update all references to it in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ###### a. Create a Storage by Zapier connection If you haven't already got one, go to <https://zapier.com/app/connections/storage> and create a new connection. Remember the UUID secret you generate for later. ###### b. Add a temporary code step Choose **Run Python** as the Event. Run the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_WEBHOOK_KEY', 'abc123') #replace with webhook secret store.set('DBT_CLOUD_SERVICE_TOKEN', 'abc123') #replace with your dbt API token ``` Test the step. You can delete this Action when the test succeeds. The key will remain stored as long as it is accessed at least once every three months. #### Add a code action Select **Code by Zapier** as the App, and **Run Python** as the Event. In the **Set up action** section, add two items to **Input Data**: `raw_body` and `auth_header`. Map those to the `1. Raw Body` and `1. Headers Http Authorization` fields from the previous **Catch Raw Hook** step. [![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/img/guides/orchestration/webhooks/zapier-common/run-python.png?v=2 "Screenshot of the Zapier UI, showing the mappings of raw_body and auth_header")](#)Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header In the **Code** field, paste the following code, replacing `YOUR_SECRET_HERE` with the secret you created when setting up the Storage by Zapier integration. Remember that this is not your dbt secret. This example code validates the authenticity of the request, extracts the run logs for the completed job from the Admin API, and then builds two messages: a summary message containing the outcome of each step and its duration, and a message for inclusion in a thread displaying any error messages extracted from the end-of-invocation logs created by dbt. ````python import hashlib import hmac import json import re auth_header = input_data['auth_header'] raw_body = input_data['raw_body'] # Access secret credentials secret_store = StoreClient('YOUR_SECRET_HERE') hook_secret = secret_store.get('DBT_WEBHOOK_KEY') api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') # Validate the webhook came from dbt signature = hmac.new(hook_secret.encode('utf-8'), raw_body.encode('utf-8'), hashlib.sha256).hexdigest() if signature != auth_header: raise Exception("Calculated signature doesn't match contents of the Authorization header. This webhook may not have been sent from dbt.") full_body = json.loads(raw_body) hook_data = full_body['data'] # Steps derived from these commands won't have their error details shown inline, as they're messy commands_to_skip_logs = ['dbt source', 'dbt docs'] # When testing, you will want to hardcode run_id and account_id to IDs that exist; the sample webhook won't work. run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Admin API url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() run_data_results = run_data_response.json()['data'] # Overall run summary step_summary_post = f""" *<{run_data_results['href']}|{hook_data['runStatus']} for Run #{run_id} on Job \"{hook_data['jobName']}\">* *Environment:* {hook_data['environmentName']} | *Trigger:* {hook_data['runReason']} | *Duration:* {run_data_results['duration_humanized']} """ threaded_errors_post = "" # Step-specific summaries for step in run_data_results['run_steps']: if step['status_humanized'] == 'Success': step_summary_post += f""" ✅ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ else: step_summary_post += f""" ❌ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ # Don't try to extract info from steps that don't have well-formed logs show_logs = not any(cmd in step['name'] for cmd in commands_to_skip_logs) if show_logs: full_log = step['logs'] # Remove timestamp and any colour tags full_log = re.sub('\x1b?\[[0-9]+m[0-9:]*', '', full_log) summary_start = re.search('(?:Completed with \d+ error.* and \d+ warnings?:|Database Error|Compilation Error|Runtime Error)', full_log) line_items = re.findall('(^.*(?:Failure|Error) in .*\n.*\n.*)', full_log, re.MULTILINE) if not summary_start: continue threaded_errors_post += f""" *{step['name']}* """ # If there are no line items, the failure wasn't related to dbt nodes, and we want the whole rest of the message. # If there are, then we just want the summary line and then to log out each individual node's error. if len(line_items) == 0: relevant_log = f'```{full_log[summary_start.start():]}```' else: relevant_log = summary_start[0] for item in line_items: relevant_log += f'\n```\n{item.strip()}\n```\n' threaded_errors_post += f""" {relevant_log} """ send_error_thread = len(threaded_errors_post) > 0 # Zapier looks for the `output` dictionary for use in subsequent steps output = {'step_summary_post': step_summary_post, 'send_error_thread': send_error_thread, 'threaded_errors_post': threaded_errors_post} ```` #### Add Slack actions in Zapier Select **Slack** as the App, and **Send Channel Message** as the Action. In the **Action** section, choose which **Channel** to post to. Set the **Message Text** field to **2. Step Summary Post** from the Run Python in Code by Zapier output. Configure the other options as you prefer (for example, **Bot Name** and **Bot Icon**). [![Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message](/img/guides/orchestration/webhooks/zapier-slack/parent-slack-config.png?v=2 "Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message")](#)Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message Add another step, **Filter**. In the **Filter setup and testing** section, set the **Field** to **2. Send Error Thread** and the **condition** to **(Boolean) Is true**. This prevents the Zap from failing if the job succeeded and you try to send an empty Slack message in the next step. [![Screenshot of the Zapier UI, showing the correctly configured Filter step](/img/guides/orchestration/webhooks/zapier-slack/filter-config.png?v=2 "Screenshot of the Zapier UI, showing the correctly configured Filter step")](#)Screenshot of the Zapier UI, showing the correctly configured Filter step Add another **Send Channel Message in Slack** action. In the **Action** section, choose the same channel as last time, but set the **Message Text** to **2. Threaded Errors Post** from the same Run Python step. Set the **Thread** value to **3. Message Ts**, which is the timestamp of the post created by the first Slack action. This tells Zapier to add this post as a threaded reply to the main message, which prevents the full (potentially long) output from cluttering your channel. [![Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message](/img/guides/orchestration/webhooks/zapier-slack/thread-slack-config.png?v=2 "Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message")](#)Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message #### Test and deploy When you're done testing your Zap, make sure that your `run_id` and `account_id` are no longer hardcoded in the Code step, then publish your Zap. #### Alternately, use a dbt app Slack message to trigger Zapier Instead of using a webhook as your trigger, you can keep the existing dbt app installed in your Slack workspace and use its messages being posted to your channel as the trigger. In this case, you can skip validating the webhook and only need to load the context from the thread. ##### 1. Create a new Zap in Zapier Use **Slack** as the initiating app, and **New Message Posted to Channel** as the Trigger. In the **Trigger** section, select the channel where your Slack alerts are being posted, and set **Trigger for Bot Messages?** to **Yes**. [![Screenshot of the Zapier UI, showing the correctly configured Message trigger step](/img/guides/orchestration/webhooks/zapier-slack/message-trigger-config.png?v=2 "Screenshot of the Zapier UI, showing the correctly configured Message trigger step")](#)Screenshot of the Zapier UI, showing the correctly configured Message trigger step Test your Zap to find an example record. You might need to load additional samples until you get one that relates to a failed job, depending on whether you post all job events to Slack or not. ##### 2. Add a Filter step Add a **Filter** step with the following conditions: * **1. Text contains failed on Job** * **1. User Is Bot Is true** * **1. User Name Exactly matches dbt** [![Screenshot of the Zapier UI, showing the correctly configured Filter step](/img/guides/orchestration/webhooks/zapier-slack/message-trigger-filter.png?v=2 "Screenshot of the Zapier UI, showing the correctly configured Filter step")](#)Screenshot of the Zapier UI, showing the correctly configured Filter step ##### 3. Extract the run ID Add a **Format** step with the **Event** of **Text**, and the Action **Extract Number**. For the **Input**, select **1. Text**. [![Screenshot of the Zapier UI, showing the Transform step configured to extract a number from the Slack message's Text property](/img/guides/orchestration/webhooks/zapier-slack/extract-number.png?v=2 "Screenshot of the Zapier UI, showing the Transform step configured to extract a number from the Slack message's Text property")](#)Screenshot of the Zapier UI, showing the Transform step configured to extract a number from the Slack message's Text property Test your step and validate that the run ID has been correctly extracted. ##### 4. Add a Delay Sometimes dbt posts the message about the run failing before the run's artifacts are available through the API. For this reason, it's recommended to add a brief delay to increase the likelihood that the data is available. On certain plans, Zapier will automatically retry a job that fails from to a 404 error, but its standdown period is longer than is normally necessary so the context will be missing from your thread for longer. A one-minute delay is generally sufficient. ##### 5. Store secrets In the next step, you will need either a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps). This prevents your keys from being displayed as plaintext in the Zap code. You can access them with the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the name for the secret key is `DBT_CLOUD_SERVICE_TOKEN`. If you're using a different name, make sure you update all references to it in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ###### a. Create a Storage by Zapier connection If you haven't already got one, go to <https://zapier.com/app/connections/storage> and create a new connection. Remember the UUID secret you generate for later. ###### b. Add a temporary code step Choose **Run Python** as the Event. Run the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_CLOUD_SERVICE_TOKEN', 'abc123') #replace with your dbt API token ``` Test the step. You can delete this Action when the test succeeds. The key will remain stored as long as it is accessed at least once every three months. ##### 6. Add a Code action Select **Code by Zapier** as the App, and **Run Python** as the Event. This step is very similar to the one described in the main example, but you can skip a lot of the initial validation work. In the **Action** section, add two items to **Input Data**: `run_id` and `account_id`. Map those to the `3. Output` property and your hardcoded dbt Account ID, respectively. [![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/img/guides/orchestration/webhooks/zapier-slack/code-example-alternate.png?v=2 "Screenshot of the Zapier UI, showing the mappings of raw_body and auth_header")](#)Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header In the **Code** field, paste the following code, replacing `YOUR_SECRET_HERE` with the secret you created when setting up the Storage by Zapier integration. Remember that this is not your dbt secret. This example code extracts the run logs for the completed job from the Admin API, and then builds a message displaying any error messages extracted from the end-of-invocation logs created by dbt (which will be posted in a thread). ````python import re # Access secret credentials secret_store = StoreClient('YOUR_SECRET_HERE') api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') # Steps derived from these commands won't have their error details shown inline, as they're messy commands_to_skip_logs = ['dbt source', 'dbt docs'] run_id = input_data['run_id'] account_id = input_data['account_id'] url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} response = requests.get(url, headers=headers) response.raise_for_status() results = response.json()['data'] threaded_errors_post = "" for step in results['run_steps']: show_logs = not any(cmd in step['name'] for cmd in commands_to_skip_logs) if not show_logs: continue if step['status_humanized'] != 'Success': full_log = step['logs'] # Remove timestamp and any colour tags full_log = re.sub('\x1b?\[[0-9]+m[0-9:]*', '', full_log) summary_start = re.search('(?:Completed with \d+ error.* and \d+ warnings?:|Database Error|Compilation Error|Runtime Error)', full_log) line_items = re.findall('(^.*(?:Failure|Error) in .*\n.*\n.*)', full_log, re.MULTILINE) if not summary_start: continue threaded_errors_post += f""" *{step['name']}* """ # If there are no line items, the failure wasn't related to dbt nodes, and we want the whole rest of the message. # If there are, then we just want the summary line and then to log out each individual node's error. if len(line_items) == 0: relevant_log = f'```{full_log[summary_start.start():]}```' else: relevant_log = summary_start[0] for item in line_items: relevant_log += f'\n```\n{item.strip()}\n```\n' threaded_errors_post += f""" {relevant_log} """ output = {'threaded_errors_post': threaded_errors_post} ```` ##### 7. Add Slack action in Zapier Add a **Send Channel Message in Slack** action. In the **Action** section, set the channel to **1. Channel Id**, which is the channel that the triggering message was posted in. Set the **Message Text** to **5. Threaded Errors Post** from the Run Python step. Set the **Thread** value to **1. Ts**, which is the timestamp of the triggering Slack post. This tells Zapier to add this post as a threaded reply to the main message, which prevents the full (potentially long) output from cluttering your channel. [![Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message](/img/guides/orchestration/webhooks/zapier-slack/thread-slack-config-alternate.png?v=2 "Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message")](#)Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message ##### 8. Test and deploy When you're done testing your Zap, publish it. --- ### Productionize your dbt Databricks project [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Welcome to the third installment of our comprehensive series on optimizing and deploying your data pipelines using Databricks and dbt. In this guide, we'll dive into delivering these models to end users while incorporating best practices to ensure that your production data remains reliable and timely. ##### Prerequisites If you don't have any of the following requirements, refer to the instructions in the [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md) for help meeting these requirements: * You have [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). * You have [optimized your dbt models for peak performance](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md). * You have created two catalogs in Databricks: *dev* and *prod*. * You have created Databricks Service Principal to run your production jobs. * You have at least one [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) in dbt. To get started, let's revisit the deployment environment created for your production data. ##### Deployment environments In software engineering, environments play a crucial role in allowing engineers to develop and test code without affecting the end users of their software. Similarly, you can design [data lakehouses](https://www.databricks.com/product/data-lakehouse) with separate environments. The *production* environment includes the relations (schemas, tables, and views) that end users query or use, typically in a BI tool or ML model. In dbt, [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) come in two flavors: * Deployment — Defines the settings used for executing jobs created within that environment. * Development — Determine the settings used in the Studio IDE for a particular dbt project. Each dbt project can have multiple deployment environments, but only one development environment per user. #### Create and schedule a production job With your deployment environment set up, it's time to create a production job to run in your *prod* environment. To deploy our data transformation workflows, we will utilize [dbt’s built-in job scheduler](https://docs.getdbt.com/docs/deploy/deploy-jobs.md). The job scheduler is designed specifically to streamline your dbt project deployments and runs, ensuring that your data pipelines are easy to create, monitor, and modify efficiently. Leveraging dbt's job scheduler allows data teams to own the entire transformation workflow. You don't need to learn and maintain additional tools for orchestration or rely on another team to schedule code written by your team. This end-to-end ownership simplifies the deployment process and accelerates the delivery of new data products. Let’s [create a job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) in dbt that will transform data in our Databricks *prod* catalog. 1. Create a new job by clicking **Deploy** in the header, click **Jobs** and then **Create job**. 2. **Name** the job “Daily refresh”. 3. Set the **Environment** to your *production* environment. * This will allow the job to inherit the catalog, schema, credentials, and environment variables defined in [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). 4. Under **Execution Settings** * Check the **Generate docs on run** checkbox to configure the job to automatically generate project docs each time this job runs. This will ensure your documentation stays evergreen as models are added and modified. * Select the **Run on source freshness** checkbox to configure dbt [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) as the first step of this job. Your sources will need to be configured to [snapshot freshness information](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) for this to drive meaningful insights. Add the following three **Commands:** * `dbt source freshness` * This will check if any sources are stale. We don’t want to recompute models with data that hasn’t changed since our last run. * `dbt test --models source:*` * This will test the data quality our source data, such as checking making sure ID fields are unique and not null. We don’t want bad data getting into production models. * `dbt build --exclude source:* --fail-fast` * dbt build is more efficient than issuing separate commands for dbt run and dbt test separately because it will run then test each model before continuing. * We are excluding source data because we already tested it in step 2. * The fail-fast flag will make dbt exit immediately if a single resource fails to build. If other models are in-progress when the first model fails, then dbt will terminate the connections for these still-running models. 5. Under **Triggers**, use the toggle to configure your job to [run on a schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#schedule-days). You can enter specific days and timing or create a custom cron schedule. * If you want your dbt job scheduled by another orchestrator, like Databricks Workflows, see the [Advanced Considerations](#advanced-considerations) section below. This is just one example of an all-or-nothing command list designed to minimize wasted computing. The [job command list](https://docs.getdbt.com/docs/deploy/job-commands.md) and [selectors](https://docs.getdbt.com/reference/node-selection/syntax.md) provide a lot of flexibility on how your DAG will execute. You may want to design yours to continue running certain models if others fail. You may want to set up multiple jobs to refresh models at different frequencies. See our [Job Creation Best Practices discourse](https://discourse.getdbt.com/t/job-creation-best-practices-in-dbt-cloud-feat-my-moms-lasagna/2980) for more job design suggestions. After your job is set up and runs successfully, configure your **[project artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md)** to make this job inform your production docs site and data sources dashboard that can be reached from the UI. This will be our main production job to refresh data that will be used by end users. Another job everyone should include in their dbt project is a continuous integration job. #### Add a CI job CI/CD, or Continuous Integration and Continuous Deployment/Delivery, has become a standard practice in software development for rapidly delivering new features and bug fixes while maintaining high quality and stability. dbt enables you to apply these practices to your data transformations. The steps below show how to create a CI test for your dbt project. CD in dbt requires no additional steps, as your jobs will automatically pick up the latest changes from the branch assigned to the environment your job is running in. You may choose to add steps depending on your deployment strategy. If you want to dive deeper into CD options, check out [this blog on adopting CI/CD with dbt](https://www.getdbt.com/blog/adopting-ci-cd-with-dbt-cloud/). dbt allows you to write [data tests](https://docs.getdbt.com/docs/build/data-tests.md) for your data pipeline, which can be run at every step of the process to ensure the stability and correctness of your data transformations. The main places you’ll use your dbt tests are: 1. **Daily runs:** Regularly running tests on your data pipeline helps catch issues caused by bad source data, ensuring the quality of data that reaches your users. 2. **Development**: Running tests during development ensures that your code changes do not break existing assumptions, enabling developers to iterate faster by catching problems immediately after writing code. 3. **CI checks**: Automated CI jobs run and test your pipeline end-to end when a pull request is created, providing confidence to developers, code reviewers, and end users that the proposed changes are reliable and will not cause disruptions or data quality issues Your CI job will ensure that the models build properly and pass any tests applied to them. We recommend creating a separate *test* environment and having a dedicated service principal. This will ensure the temporary schemas created during CI tests are in their own catalog and cannot unintentionally expose data to other users. Repeat the steps in [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md) to create your *prod* environment to create a *test* environment. After setup, you should have: * A catalog called *test* * A service principal called *dbt\_test\_sp* * A new dbt environment called *test* that defaults to the *test* catalog and uses the *dbt\_test\_sp* token in the deployment credentials We recommend setting up a dbt CI job. This will decrease the job’s runtime by running and testing only modified models, which also reduces compute spend on the lakehouse. To create a CI job, refer to [Set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) for details. With dbt tests and SlimCI, you can feel confident that your production data will be timely and accurate even while delivering at high velocity. #### Monitor your jobs Keeping a close eye on your dbt jobs is crucial for maintaining a robust and efficient data pipeline. By monitoring job performance and quickly identifying potential issues, you can ensure that your data transformations run smoothly. dbt provides three entry points to monitor the health of your project: run history, deployment monitor, and status tiles. The [run history](https://docs.getdbt.com/docs/deploy/run-visibility.md#run-history) dashboard in dbt provides a detailed view of all your project's job runs, offering various filters to help you focus on specific aspects. This is an excellent tool for developers who want to check recent runs, verify overnight results, or track the progress of running jobs. To access it, select **Run History** from the **Deploy** menu. The deployment monitor in dbt offers a higher-level view of your run history, enabling you to gauge the health of your data pipeline over an extended period of time. This feature includes information on run durations and success rates, allowing you to identify trends in job performance, such as increasing run times or more frequent failures. The deployment monitor also highlights jobs in progress, queued, and recent failures. To access the deployment monitor click on the dbt logo in the top left corner of the dbt UI. [![The Deployment Monitor Shows Job Status Over Time Across Environments](/img/guides/databricks-guides/deployment_monitor_dbx.png?v=2 "The Deployment Monitor Shows Job Status Over Time Across Environments")](#)The Deployment Monitor Shows Job Status Over Time Across Environments By adding [data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) to your BI dashboards, you can give stakeholders visibility into the health of your data pipeline without leaving their preferred interface. Data tiles instill confidence in your data and help prevent unnecessary inquiries or context switching. To implement dashboard status tiles, you'll need to have dbt docs with [exposures](https://docs.getdbt.com/docs/build/exposures.md) defined. #### Set up notifications Setting up [notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) in dbt allows you to receive alerts via email or a Slack channel whenever a run ends. This ensures that the appropriate teams are notified and can take action promptly when jobs fail or are canceled. To set up notifications: 1. Navigate to your dbt project settings. 2. Select the **Notifications** tab. 3. Choose the desired notification type (Email or Slack) and configure the relevant settings. If you require notifications through other means than email or Slack, you can use dbt's outbound [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) feature to relay job events to other tools. Webhooks enable you to integrate dbt with a wide range of SaaS applications, extending your pipeline’s automation into other systems. #### Troubleshooting When a disruption occurs in your production pipeline, it's essential to know how to troubleshoot issues effectively to minimize downtime and maintain a high degree of trust with your stakeholders. The five key steps for troubleshooting dbt issues are: 1. Read the error message: dbt error messages usually indicate the error type and the file where the issue occurred. 2. Inspect the problematic file and look for an immediate fix. 3. Isolate the problem by running one model at a time in the Studio IDE or undoing the code that caused the issue. 4. Check for problems in compiled files and logs. Consult the [Debugging errors documentation](https://docs.getdbt.com/guides/debug-errors.md) for a comprehensive list of error types and diagnostic methods. To troubleshoot issues with a dbt job, navigate to the "Deploy > Run History" tab in your dbt project and select the failed run. Then, expand the run steps to view [console and debug logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs) to review the detailed log messages. To obtain additional information, open the Artifacts tab and download the compiled files associated with the run. If your jobs are taking longer than expected, use the [model timing](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing) dashboard to identify bottlenecks in your pipeline. Analyzing the time taken for each model execution helps you pinpoint the slowest components and optimize them for better performance. The Databricks [Query History](https://docs.databricks.com/sql/admin/query-history.html) lets you inspect granular details such as time spent in each task, rows returned, I/O performance, and execution plan. For more on performance tuning, see our guide on [How to Optimize and Troubleshoot dbt Models on Databricks](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md). #### Advanced considerations As you become more experienced with dbt and Databricks, you might want to explore advanced techniques to further enhance your data pipeline and improve the way you manage your data transformations. The topics in this section are not requirements but will help you harden your production environment for greater security, efficiency, and accessibility. ##### Refreshing your data with Databricks Workflows The dbt job scheduler offers several ways to trigger your jobs. If your dbt transformations are just one step of a larger orchestration workflow, use the dbt API to trigger your job from Databricks Workflows. This is a common pattern for analytics use cases that want to minimize latency between ingesting bronze data into the lakehouse with a notebook, transforming that data into gold tables with dbt, and refreshing a dashboard. It is also useful for data science teams who use dbt for feature extraction before using the updated feature store to train and register machine learning models with MLflow. The API enables integration between your dbt jobs and the Databricks workflow, ensuring that your data transformations are effectively managed within the broader context of your data processing pipeline. Inserting dbt jobs into a Databricks Workflows allows you to chain together external tasks while still leveraging these benefits of dbt: * UI Context: The dbt UI enables you to define the job within the context of your dbt environments, making it easier to create and manage relevant configs. * Logs and Run History: Accessing logs and run history becomes more convenient when using dbt. * Monitoring and Notification Features: dbt comes equipped with monitoring and notification features like the ones described above that can help you stay informed about the status and performance of your jobs. To trigger your dbt job from Databricks, follow the instructions in our [Databricks Workflows to run dbt jobs guide](https://docs.getdbt.com/guides/databricks-workflows.md). #### Data masking Our [Best Practices for dbt and Unity Catalog](https://docs.getdbt.com/best-practices/dbt-unity-catalog-best-practices.md) guide recommends using separate catalogs *dev* and *prod* for development and deployment environments, with Unity Catalog and dbt handling configurations and permissions for environment isolation. Ensuring security while maintaining efficiency in your development and deployment environments is crucial. Additional security measures may be necessary to protect sensitive data, such as personally identifiable information (PII). Databricks leverages [Dynamic Views](https://docs.databricks.com/data-governance/unity-catalog/create-views.html#create-a-dynamic-view) to enable data masking based on group membership. Because views in Unity Catalog use Spark SQL, you can implement advanced data masking by using more complex SQL expressions and regular expressions. You can now also apply fine grained access controls like row filters in preview and column masks in preview on tables in Databricks Unity Catalog, which will be the recommended approach to protect sensitive data once this goes GA. Additionally, in the near term, Databricks Unity Catalog will also enable Attribute Based Access Control natively, which will make protecting sensitive data at scale simpler. To implement data masking in a dbt model, ensure the model materialization configuration is set to view. Next, add a case statement using the is\_account\_group\_member function to identify groups permitted to view plain text values. Then, use regex to mask data for all other users. For example: ```sql CASE WHEN is_account_group_member('auditors') THEN email ELSE regexp_extract(email, '^.*@(.*)$', 1) END ``` It is recommended not to grant users the ability to read tables and views referenced in the dynamic view. Instead, assign your dbt sources to dynamic views rather than raw data, allowing developers to run end-to-end builds and source freshness commands securely. Using the same sources for development and deployment environments enables testing with the same volumes and frequency you will see in production. However, this may cause development runs to take longer than necessary. To address this issue, consider using the Jinja variable target.name to [limit data when working in the development environment](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md#use-targetname-to-limit-data-in-dev). #### Pairing dbt Docs and Unity Catalog Though there are similarities between dbt docs and Databricks Unity Catalog, they are ultimately used for different purposes and complement each other well. By combining their strengths, you can provide your organization with a robust and user-friendly data management ecosystem. dbt docs is a documentation site generated from your dbt project that provides an interface for developers and non-technical stakeholders to understand the data lineage and business logic applied to transformations without requiring full access to dbt or Databricks. It gives you additional options on how you can organize and search for your data. You can automatically [build and view your dbt docs using dbt](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) to keep the documentation evergreen. Unity Catalog is a unified governance solution for your lakehouse. It provides a data explorer that can be used for discovery of datasets that have not been defined in dbt. The data explorer also captures [column-level lineage](https://docs.databricks.com/data-governance/unity-catalog/data-lineage.html#capture-and-explore-lineage),  when you need to trace the lineage of a specific column. To get the most out of both tools, you can use the [persist docs config](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) to push table and column descriptions written in dbt into Unity Catalog, making the information easily accessible to both tools' users. Keeping the descriptions in dbt ensures they are version controlled and can be reproduced after a table is dropped. ##### Related docs * [Advanced Deployment course](https://learn.getdbt.com/courses/advanced-deployment) if you want a deeper dive into these topics * [Autoscaling CI: The intelligent Slim CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Trigger a dbt Job in your automated workflow with Python](https://discourse.getdbt.com/t/triggering-a-dbt-cloud-job-in-your-automated-workflow-with-python/2573) * [Databricks + dbt Quickstart Guide](https://docs.getdbt.com/guides/databricks.md) * Reach out to your Databricks account team to get access to preview features on Databricks. --- ### Quickstart for dbt and Amazon Athena [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with Amazon Athena. It will show you how to: * Create an S3 bucket for Athena query results. * Create an Athena database. * Access sample data in a public dataset. * Connect dbt to Amazon Athena. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have an [AWS account](https://aws.amazon.com/). * You have set up [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html). ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) #### Getting started For the following guide you can use an existing S3 bucket or [create a new one](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). Download the following CSV files (the Jaffle Shop sample data) and upload them to your S3 bucket: * [jaffle\_shop\_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) * [jaffle\_shop\_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) * [stripe\_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) #### Configure Amazon Athena 1. Log into your AWS account and navigate to the **Athena console**. * If this is your first time in the Athena console (in your current AWS Region), click **Explore the query editor** to open the query editor. Otherwise, Athena opens automatically in the query editor. 2. Open **Settings** and find the **Location of query result box** field. 1. Enter the path of the S3 bucket (prefix it with `s3://`). 2. Navigate to **Browse S3**, select the S3 bucket you created, and click **Choose**. 3. **Save** these settings. 4. In the **query editor**, create a database by running `create database YOUR_DATABASE_NAME`. 5. To make the database you created the one you `write` into, select it from the **Database** list on the left side menu. 6. Access the Jaffle Shop data in the S3 bucket using one of these options: 1. Manually create the tables. 2. Create a glue crawler to recreate the data as external tables (recommended). 7. Once the tables have been created, you will able to `SELECT` from them. #### Set up security access to Athena To setup the security access for Athena, determine which access method you want to use: * Obtain `aws_access_key_id` and `aws_secret_access_key` (recommended) * Obtain an **AWS credentials** file. ##### AWS access key (recommended) To obtain your `aws_access_key_id` and `aws_secret_access_key`: 1. Open the **AWS Console**. 2. Click on your **username** near the top right and click **Security Credentials**. 3. Click on **Users** in the sidebar. 4. Click on your **username** (or the name of the user for whom to create the key). 5. Click on the **Security Credentials** tab. 6. Click **Create Access Key**. 7. Click **Show User Security Credentials** and Save the `aws_access_key_id` and `aws_secret_access_key` for a future step. ##### AWS credentials file To obtain your AWS credentials file: 1. Follow the instructions for [configuring the credentials file](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) using the AWS CLI 2. Locate the `~/.aws/credentials` file on your computer 1. Windows: `%USERPROFILE%\.aws\credentials` 2. Mac/Linux: `~/.aws/credentials` Retrieve the `aws_access_key_id` and `aws_secret_access_key` from the `~/.aws/credentials` file for a future step. #### Configure the connection in dbt To configure the Athena connection in dbt: 1. Click your **account name** on the left-side menu and click **Account settings**. 2. Click **Connections** and click **New connection**. 3. Click **Athena** and fill out the required fields (and any optional fields). 1. **AWS region name** — The AWS region of your environment. 2. **Database (catalog)** — Enter the database name created in earlier steps (lowercase only). 3. **AWS S3 staging directory** — Enter the S3 bucket created in earlier steps. 4. Click **Save** ##### Configure your environment To configure the Athena credentials in your environment: 1. Click **Deploy** on the left-side menu and click **Environments**. 2. Click **Create environment** and fill out the **General settings**. * Your **dbt version** must be on a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to use the Athena connection. 3. Select the Athena connection from the **Connection** dropdown. 4. Fill out the `aws_access_key` and `aws_access_id` recorded in previous steps, as well as the `Schema` to write to. 5. Click **Test connection** and once it succeeds, **Save** the environment. Repeat the process to create a [development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments). #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from jaffle_shop.customers ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) section below. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned up data rather than raw data. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and Azure Synapse Analytics [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with [Azure Synapse Analytics](https://azure.microsoft.com/en-us/products/synapse-analytics/). It will show you how to: * Load the Jaffle Shop sample data (provided by dbt Labs) into your Azure Synapse Analytics warehouse. * Connect dbt to Azure Synapse Analytics. * Turn a sample query into a model in your dbt project. A model in dbt is a SELECT statement. * Add tests to your models. * Document your models. * Schedule a job to run. ##### Prerequisites * You have a [dbt](https://www.getdbt.com/signup/) account. * You have an Azure Synapse Analytics account. For a free trial, refer to [Synapse Analytics](https://azure.microsoft.com/en-us/free/synapse-analytics/) in the Microsoft docs. * As a Microsoft admin, you’ve enabled service principal authentication. You must add the service principal to the Synapse workspace with either a Member (recommended) or Admin permission set. For details, refer to [Create a service principal using the Azure portal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal) in the Microsoft docs. dbt needs these authentication credentials to connect to Azure Synapse Analytics. ##### Related content * [dbt Learn courses](https://learn.getdbt.com) * [About continuous integration jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) #### Load data into your Azure Synapse Analytics 1. Log in to your [Azure portal account](https://portal.azure.com/#home). 2. On the home page, select the **SQL databases** tile. 3. From the **SQL databases** page, navigate to your organization’s workspace or create a new workspace; refer to [Create a Synapse workspace](https://learn.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-workspace) in the Microsoft docs for more details. 4. From the workspace's sidebar, select **Data**. Click the three dot menu on your database and select **New SQL script** to open the SQL editor. 5. Copy these statements into the SQL editor to load the Jaffle Shop example data: ```sql CREATE TABLE dbo.customers ( [ID] [bigint], [FIRST_NAME] [varchar](8000), [LAST_NAME] [varchar](8000) ); COPY INTO [dbo].[customers] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/jaffle_shop_customers.parquet' WITH ( FILE_TYPE = 'PARQUET' ); CREATE TABLE dbo.orders ( [ID] [bigint], [USER_ID] [bigint], [ORDER_DATE] [date], [STATUS] [varchar](8000) ); COPY INTO [dbo].[orders] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/jaffle_shop_orders.parquet' WITH ( FILE_TYPE = 'PARQUET' ); CREATE TABLE dbo.payments ( [ID] [bigint], [ORDERID] [bigint], [PAYMENTMETHOD] [varchar](8000), [STATUS] [varchar](8000), [AMOUNT] [bigint], [CREATED] [date] ); COPY INTO [dbo].[payments] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/stripe_payments.parquet' WITH ( FILE_TYPE = 'PARQUET' ); ``` [![Example of loading data](/img/quickstarts/dbt-platform/example-load-data-azure-syn-analytics.png?v=2 "Example of loading data")](#)Example of loading data #### Connect dbt to Azure Synapse Analytics 1. Create a new project in dbt. Click on your account name in the left side menu, select **Account settings**, and click **+ New Project**. 2. Enter a project name and click **Continue**. 3. Choose **Synapse** as your connection and click **Next**. 4. In the **Configure your environment** section, enter the **Settings** for your new project: * **Server** — Use the service principal's **Synapse host name** value (without the trailing `, 1433` string) for the Synapse test endpoint. * **Port** — 1433 (which is the default). * **Database** — Use the service principal's **database** value for the Synapse test endpoint. 5. Enter the **User credentials** for your new project: * **Authentication** — Choose **Service Principal** from the dropdown. * **Tenant ID** — Use the service principal’s **Directory (tenant) id** as the value. * **Client ID** — Use the service principal’s **application (client) ID** as the value. * **Client secret** — Use the service principal’s **client secret** (not the **client secret id**) as the value. 6. Click **Test connection**. This verifies that dbt can access your Azure Synapse Analytics account. 7. Click **Next** when the test succeeds. If it failed, you might need to check your Microsoft service principal. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit Changes**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. 1. Under **Version Control** on the left, click **Create branch**. You can name it `add-customers-model`. You need to create a new branch since the main branch is set to read-only mode. 2. Click the three dot menu (**...**) next to the `models` directory, then select **Create file**. 3. Name the file `customers.sql`, then click **Create**. 4. Copy the following query into the file and click **Save**. customers.sql ```sql with customers as ( select ID as customer_id, FIRST_NAME as first_name, LAST_NAME as last_name from dbo.customers ), orders as ( select ID as order_id, USER_ID as customer_id, ORDER_DATE as order_date, STATUS as status from dbo.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by customer_id ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 5. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned up data rather than raw data. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select ID as customer_id, FIRST_NAME as first_name, LAST_NAME as last_name from dbo.customers ``` models/stg\_orders.sql ```sql select ID as order_id, USER_ID as customer_id, ORDER_DATE as order_date, STATUS as status from dbo.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by customer_id ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and BigQuery > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * This guide uses dbt platform. Don’t switch the reader to a local setup. If the reader wants to set up dbt locally, use the <https://docs.getdbt.com/guides/dbt> guide instead > * Never ask for passwords or keys in chat. > * When the reader needs a dbt account, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=bigquery-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with BigQuery. It will show you how to: * Create a Google Cloud Platform (GCP) project. * Access sample data in a public dataset. * Connect dbt to BigQuery. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have a [Google account](https://support.google.com/accounts/answer/27441?hl=en). * You can use a personal or work account to set up BigQuery through [Google Cloud Platform (GCP)](https://cloud.google.com/free). ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. #### Create a new GCP project​ 1. Go to the [BigQuery Console](https://console.cloud.google.com/bigquery) after you log in to your Google account. If you have multiple Google accounts, make sure you’re using the correct one. 2. Create a new project from the [Manage resources page](https://console.cloud.google.com/projectcreate?previousPage=%2Fcloud-resource-manager%3Fwalkthrough_id%3Dresource-manager--create-project%26project%3D%26folder%3D%26organizationId%3D%23step_index%3D1\&walkthrough_id=resource-manager--create-project). For more information, refer to [Creating a project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project) in the Google Cloud docs. GCP automatically populates the Project name field for you. You can change it to be more descriptive for your use. For example, `dbt Learn - BigQuery Setup`. #### Create BigQuery datasets 1. From the [BigQuery Console](https://console.cloud.google.com/bigquery), click **Editor**. Make sure to select your newly created project, which is available at the top of the page. 2. Verify that you can run SQL queries. Copy and paste these queries into the Query Editor: ```sql select * from `dbt-tutorial.jaffle_shop.customers`; select * from `dbt-tutorial.jaffle_shop.orders`; select * from `dbt-tutorial.stripe.payment`; ``` Click **Run**, then check for results from the queries. For example: [![Bigquery Query Results](/img/bigquery/query-results.png?v=2 "Bigquery Query Results")](#)Bigquery Query Results 3. Create new datasets from the [BigQuery Console](https://console.cloud.google.com/bigquery). For more information, refer to [Create datasets](https://cloud.google.com/bigquery/docs/datasets#create-dataset) in the Google Cloud docs. Datasets in BigQuery are equivalent to schemas in a traditional database. On the **Create dataset** page: * **Dataset ID** — Enter a name that fits the purpose. This name is used like schema in fully qualified references to your database objects such as `database.schema.table`. As an example for this guide, create one for `jaffle_shop` and another one for `stripe` afterward. * **Data location** — Leave it blank (the default). It determines the GCP location of where your data is stored. The current default location is the US multi-region. All tables within this dataset will share this location. * **Enable table expiration** — Leave it unselected (the default). The default for the billing table expiration is 60 days. Because billing isn’t enabled for this project, GCP defaults to deprecating tables. * **Google-managed encryption key** — This option is available under **Advanced options**. Allow Google to manage encryption (the default). [![Bigquery Create Dataset ID](/img/bigquery/create-dataset-id.png?v=2 "Bigquery Create Dataset ID")](#)Bigquery Create Dataset ID 4. After you create the `jaffle_shop` dataset, create one for `stripe` with all the same values except for **Dataset ID**. #### Generate BigQuery credentials In order to let dbt connect to your warehouse, you'll need to generate a keyfile. This is analogous to using a database username and password with most other data warehouses. 1. Start the [GCP credentials wizard](https://console.cloud.google.com/apis/credentials/wizard). Make sure your new project is selected in the header. If you do not see your account or project, click your profile picture to the right and verify you are using the correct email account. For **Credential Type**: * From the **Select an API** dropdown, choose **BigQuery API** * Select **Application data** for the type of data you will be accessing * Click **Next** to create a new service account. 2. Create a service account for your new project from the [Service accounts page](https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts?supportedpurview=project). For more information, refer to [Create a service account](https://developers.google.com/workspace/guides/create-credentials#create_a_service_account) in the Google Cloud docs. As an example for this guide, you can: * Type `dbt-user` as the **Service account name** * From the **Select a role** dropdown, choose **BigQuery Job User**, **BigQuery Read Session User**, and **BigQuery Data Editor** roles and click **Continue** * Leave the **Grant users access to this service account** fields blank * Click **Done** 3. Create a service account key for your new project from the [Service accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts?walkthrough_id=iam--create-service-account-keys\&start_index=1#step_index=1). For more information, refer to [Create a service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating) in the Google Cloud docs. When downloading the JSON file, make sure to use a filename you can easily remember. For example, `dbt-user-creds.json`. For security reasons, dbt Labs recommends that you protect this JSON file like you would your identity credentials; for example, don't check the JSON file into your version control software. #### Connect dbt to BigQuery​ 1. In [dbt](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md), click your account name in the left side menu. 2. Navigate to **Account settings** and click **+ New project**. 3. Enter a project name and click **Continue**. 4. For the warehouse, click **BigQuery** then **Next** to set up your connection. 5. Click **Upload a Service Account JSON File** in settings. 6. Select the JSON file you downloaded in Generate BigQuery credentials and dbt will fill in all the necessary fields. 7. (Optional) dbt Enterprise plans can configure developer OAuth with BigQuery, providing an additional layer of security. For more information, refer to [Set up BigQuery OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md). 8. Set up your personal user credentials by navigating to **Your profile** > **Credentials**. 9. Select your project that uses the BigQuery connection. 10. Click **Edit**. 11. Enter your **User credentials** for BigQuery with: * **Authentication Method** — Select **Service Account JSON**. This uses the service account you uploaded when you set up the project connection. * **Dataset** — You may notice that the dataset name has been auto-created for you. By convention, this is `dbt_<first-initial><last-name>`. This is the dataset connected directly to your development environment, and it's where your models will be built when running dbt within the Studio IDE. * **Target name** — Leave as the default. * **Threads** — Leave as the default (6). This is the number of simultaneous connections that dbt will make to build models concurrently. [![BigQuery User credentials](/img/bigquery/dbt_platform_bigquery_development_credentials.png?v=2 "BigQuery User credentials")](#)BigQuery User credentials 12. Click **Test Connection**. This verifies that dbt can access your BigQuery account. 13. Click **Next** if the test succeeded. If it failed, you might need to go back and regenerate your BigQuery credentials. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from `dbt-tutorial.jaffle_shop.customers` ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from `dbt-tutorial`.jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from `dbt-tutorial`.jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned up data rather than raw data. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from `dbt-tutorial`.jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from `dbt-tutorial`.jaffle_shop.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. #### Build models on top of sources Sources make it possible to name and describe the data loaded into your warehouse by your extract and load tools. By declaring these tables as sources in dbt, you can: * select from source tables in your models using the `{{ source() }}` function, helping define the lineage of your data * test your assumptions about your source data * calculate the freshness of your source data 1. Create a new YML file `models/sources.yml`. 2. Declare the sources by copying the following into the file and clicking **Save**. models/sources.yml ```yml sources: - name: jaffle_shop description: This is a replica of the Postgres database used by our app database: dbt-tutorial schema: jaffle_shop tables: - name: customers description: One record per customer. - name: orders description: One record per order. Includes cancelled and deleted orders. ``` 3. Edit the `models/stg_customers.sql` file to select from the `customers` table in the `jaffle_shop` source. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from {{ source('jaffle_shop', 'customers') }} ``` 4. Edit the `models/stg_orders.sql` file to select from the `orders` table in the `jaffle_shop` source. models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from {{ source('jaffle_shop', 'orders') }} ``` 5. Execute `dbt run`. The results of your `dbt run` will be exactly the same as the previous step. Your `stg_customers` and `stg_orders` models will still query from the same raw data source in BigQuery. By using `source`, you can test and document your raw data and also understand the lineage of your sources. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and Databricks > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * This guide uses dbt platform. Don’t switch the reader to a local setup. If the reader wants to set up dbt locally, use the <https://docs.getdbt.com/guides/dbt> guide instead > * Never ask for passwords or keys in chat. > * When the reader needs a dbt account, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=databricks-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with Databricks. It will show you how to: * Create a Databricks workspace. * Load sample data into your Databricks account. * Connect dbt to Databricks. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have an account with a cloud service provider (such as AWS, GCP, and Azure) and have permissions to create an S3 bucket with this account. For demonstrative purposes, this guide uses AWS as the cloud service provider. ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. #### Create a Databricks workspace 1. Use your existing account or [sign up for a Databricks account](https://databricks.com/). Complete the form with your user information and click **Continue**. [![Sign up for Databricks](/img/databricks_tutorial/images/signup_form.png?v=2 "Sign up for Databricks")](#)Sign up for Databricks 2. On the next screen, select your cloud provider. This tutorial uses AWS as the cloud provider, but if you use Azure or GCP internally, please select your platform. The setup process will be similar. Do not select the **Get started with Community Edition** option, as this will not provide the required compute for this guide. [![Choose cloud provider](/img/databricks_tutorial/images/choose_provider.png?v=2 "Choose cloud provider")](#)Choose cloud provider 3. Check your email and complete the verification process. 4. After completing the verification processes, you will be brought to the first setup screen. Databricks defaults to the `Premium` plan and you can change the trial to `Enterprise` on this page. [![Choose Databricks Plan](/img/databricks_tutorial/images/choose_plan.png?v=2 "Choose Databricks Plan")](#)Choose Databricks Plan 5. Now, it's time to create your first workspace. A Databricks workspace is an environment for accessing all of your Databricks assets. The workspace organizes objects like notebooks, SQL warehouses, clusters, and more so into one place. Provide the name of your workspace, choose the appropriate AWS region, and click **Start Quickstart**. You might get the checkbox of **I have data in S3 that I want to query with Databricks**. You do not need to check this off for this tutorial. [![Create AWS resources](/img/databricks_tutorial/images/start_quickstart.png?v=2 "Create AWS resources")](#)Create AWS resources 6. By clicking on `Start Quickstart`, you will be redirected to AWS and asked to log in if you haven’t already. After logging in, you should see a page similar to this. [![Create AWS resources](/img/databricks_tutorial/images/quick_create_stack.png?v=2 "Create AWS resources")](#)Create AWS resources tip If you get a session error and don’t get redirected to this page, you can go back to the Databricks UI and create a workspace from the interface. All you have to do is click **create workspaces**, choose the quickstart, fill out the form and click **Start Quickstart**. 7. There is no need to change any of the pre-filled out fields in the Parameters. Just add in your Databricks password under **Databricks Account Credentials**. Check off the Acknowledgement and click **Create stack**. [![Parameters](/img/databricks_tutorial/images/parameters.png?v=2 "Parameters")](#)Parameters [![Capabilities](/img/databricks_tutorial/images/create_stack.png?v=2 "Capabilities")](#)Capabilities 8. Go back to the Databricks tab. You should see that your workspace is ready to use. [![A Databricks Workspace](/img/databricks_tutorial/images/workspaces.png?v=2 "A Databricks Workspace")](#)A Databricks Workspace 9. Now let’s jump into the workspace. Click **Open** and log into the workspace using the same login as you used to log into the account. #### Load data 1. Download these CSV files (the Jaffle Shop sample data) that you will need for this guide: * [jaffle\_shop\_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) * [jaffle\_shop\_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) * [stripe\_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) 2. First we need a SQL warehouse. Find the drop down menu and toggle into the SQL space. [![SQL space](/img/databricks_tutorial/images/go_to_sql.png?v=2 "SQL space")](#)SQL space 3. We will be setting up a SQL warehouse now. Select **SQL Warehouses** from the left hand side console. You will see that a default SQL Warehouse exists. 4. Click **Start** on the Starter Warehouse. This will take a few minutes to get the necessary resources spun up. 5. Once the SQL Warehouse is up, click **New** and then **File upload** on the dropdown menu. [![New File Upload Using Databricks SQL](/img/databricks_tutorial/images/new_file_upload_using_databricks_SQL.png?v=2 "New File Upload Using Databricks SQL")](#)New File Upload Using Databricks SQL 6. Let's load the Jaffle Shop Customers data first. Drop in the `jaffle_shop_customers.csv` file into the UI. [![Databricks Table Loader](/img/databricks_tutorial/images/databricks_table_loader.png?v=2 "Databricks Table Loader")](#)Databricks Table Loader 7. Update the Table Attributes at the top: * **data\_catalog** = hive\_metastore * **database** = default * **table** = jaffle\_shop\_customers * Make sure that the column data types are correct. The way you can do this is by hovering over the datatype icon next to the column name. * **ID** = bigint * **FIRST\_NAME** = string * **LAST\_NAME** = string [![Load jaffle shop customers](/img/databricks_tutorial/images/jaffle_shop_customers_upload.png?v=2 "Load jaffle shop customers")](#)Load jaffle shop customers 8. Click **Create** on the bottom once you’re done. 9. Now let’s do the same for `Jaffle Shop Orders` and `Stripe Payments`. [![Load jaffle shop orders](/img/databricks_tutorial/images/jaffle_shop_orders_upload.png?v=2 "Load jaffle shop orders")](#)Load jaffle shop orders [![Load stripe payments](/img/databricks_tutorial/images/stripe_payments_upload.png?v=2 "Load stripe payments")](#)Load stripe payments 10. Once that's done, make sure you can query the training data. Navigate to the `SQL Editor` through the left hand menu. This will bring you to a query editor. 11. Ensure that you can run a `select *` from each of the tables with the following code snippets. ```sql select * from default.jaffle_shop_customers select * from default.jaffle_shop_orders select * from default.stripe_payments ``` [![Query Check](/img/databricks_tutorial/images/query_check.png?v=2 "Query Check")](#)Query Check 12. To ensure any users who might be working on your dbt project has access to your object, run this command. ```sql grant all privileges on schema default to users; ``` #### Connect dbt to Databricks There are two ways to connect dbt to Databricks. The first option is Partner Connect, which provides a streamlined setup to create your dbt account from within your new Databricks trial account. The second option is to create your dbt account separately and build the Databricks connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt setup flow, dbt Labs recommends connecting manually. #### Set up the integration from Partner Connect note Partner Connect is intended for trial partner accounts. If your organization already has a dbt account, connect manually. Refer to [Connect to dbt manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. To connect dbt to Databricks using Partner Connect, do the following: 1. In the sidebar of your Databricks account, click **Partner Connect**. 2. Click the **dbt tile**. 3. Select a catalog from the drop-down list, and then click **Next**. The drop-down list displays catalogs you have read and write access to. If your workspace isn't `<UC>-enabled`, the legacy Hive metastore (`hive_metastore`) is used. 4. If there are SQL warehouses in your workspace, select a SQL warehouse from the drop-down list. If your SQL warehouse is stopped, click **Start**. 5. If there are no SQL warehouses in your workspace: 1. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. 2. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. 3. Return to the Partner Connect tab in your browser, and then close the **dbt tile**. 4. Re-open the **dbt tile**. 5. Select the SQL warehouse you just created from the drop-down list. 6. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you have read and write access to. You can repeat this step to add multiple schemas. Partner Connect creates the following resources in your workspace: * A Databricks service principal named **DBT\_CLOUD\_USER**. * A Databricks personal access token that is associated with the **DBT\_CLOUD\_USER** service principal. Partner Connect also grants the following privileges to the **DBT\_CLOUD\_USER** service principal: * (Unity Catalog) **USE CATALOG**: Required to interact with objects within the selected catalog. * (Unity Catalog) **USE SCHEMA**: Required to interact with objects within the selected schema. * (Unity Catalog) **CREATE SCHEMA**: Grants the ability to create schemas in the selected catalog. * (Hive metastore) **USAGE**: Required to grant the **SELECT** and **READ\_METADATA** privileges for the schemas you selected. * **SELECT**: Grants the ability to read the schemas you selected. * (Hive metastore) **READ\_METADATA**: Grants the ability to read metadata for the schemas you selected. * **CAN\_USE**: Grants permissions to use the SQL warehouse you selected. 7. Click **Next**. The **Email** box displays the email address for your Databricks account. dbt Labs uses this email address to prompt you to create a trial dbt account. 8. Click **Connect to dbt**. A new tab opens in your web browser, which displays the getdbt.com website. 9. Complete the on-screen instructions on the getdbt.com website to create your trial dbt account. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from default.jaffle_shop_customers ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop_customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop_orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned up data rather than raw data. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop_customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop_orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and Microsoft Fabric > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * This guide uses dbt platform. Don’t switch the reader to a local setup. If the reader wants to set up dbt locally, use the <https://docs.getdbt.com/guides/dbt> guide instead > * Never ask for passwords or keys in chat. > * When the reader needs a dbt account, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=microsoft-fabric-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with [Microsoft Fabric](https://www.microsoft.com/en-us/microsoft-fabric). It will show you how to: * Load the Jaffle Shop sample data (provided by dbt Labs) into your Microsoft Fabric warehouse. * Connect dbt to Microsoft Fabric. * Turn a sample query into a model in your dbt project. A model in dbt is a SELECT statement. * Add tests to your models. * Document your models. * Schedule a job to run. ##### Prerequisites * You have a [dbt](https://www.getdbt.com/signup/) account. * You have started the Microsoft Fabric (Preview) trial. For details, refer to [Microsoft Fabric (Preview) trial](https://learn.microsoft.com/en-us/fabric/get-started/fabric-trial) in the Microsoft docs. * As a Microsoft admin, you’ve enabled service principal authentication. You must add the service principal to the Microsoft Fabric workspace with either a Member (recommended) or Admin permission set. The service principal must also have `CONNECT` privileges to the database in the warehouse. For details, refer to [Enable service principal authentication](https://learn.microsoft.com/en-us/fabric/admin/metadata-scanning-enable-read-only-apis) in the Microsoft docs. dbt needs these authentication credentials to connect to Microsoft Fabric. ##### Related content * [dbt Learn courses](https://learn.getdbt.com) * [About continuous integration jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) #### Load data into your Microsoft Fabric warehouse 1. Log in to your [Microsoft Fabric](http://app.fabric.microsoft.com) account. 2. From **Workspaces** on the left sidebar, navigate to your organization’s workspace. Or, you can create a new workspace; refer to [Create a workspace](https://learn.microsoft.com/en-us/fabric/get-started/create-workspaces) in the Microsoft docs for more details. 3. Choose your warehouse from the table. Or, create a new one by selecting **+ New item** > **Warehouse** in your workspace; refer to [Create a warehouse](https://learn.microsoft.com/en-us/fabric/data-warehouse/create-warehouse) in the Microsoft docs for more details. 4. Open the SQL editor by selecting **New SQL query** from the top bar. 5. Copy these statements into the SQL editor to load the Jaffle Shop example data: ```sql DROP TABLE IF EXISTS dbo.customers; CREATE TABLE dbo.customers ( [ID] [int], [FIRST_NAME] varchar(8000), [LAST_NAME] varchar(8000) ); COPY INTO [dbo].[customers] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/jaffle_shop_customers.parquet' WITH ( FILE_TYPE = 'PARQUET' ); DROP TABLE IF EXISTS dbo.orders; CREATE TABLE dbo.orders ( [ID] [int], [USER_ID] [int], -- [ORDER_DATE] [int], [ORDER_DATE] [date], [STATUS] varchar(8000) ); COPY INTO [dbo].[orders] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/jaffle_shop_orders.parquet' WITH ( FILE_TYPE = 'PARQUET' ); DROP TABLE IF EXISTS dbo.payments; CREATE TABLE dbo.payments ( [ID] [int], [ORDERID] [int], [PAYMENTMETHOD] varchar(8000), [STATUS] varchar(8000), [AMOUNT] [int], [CREATED] [date] ); COPY INTO [dbo].[payments] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/stripe_payments.parquet' WITH ( FILE_TYPE = 'PARQUET' ); ``` [![Example of loading data](/img/quickstarts/dbt-platform/example-load-data-ms-fabric.png?v=2 "Example of loading data")](#)Example of loading data #### Connect dbt to Microsoft Fabric 1. Create a new project in dbt. Navigate to **Account settings** (by clicking on your account name in the left side menu), and click **+ New Project**. 2. Enter a project name and click **Continue**. 3. Choose **Fabric** as your connection and click **Next**. 4. In the **Configure your environment** section, enter the **Settings** for your new project: * **Server** — Use the service principal's **host** value for the Fabric test endpoint. * **Port** — 1433 (which is the default). * **Database** — Use the service principal's **database** value for the Fabric test endpoint. 5. Enter the **User credentials** for your new project: * **Authentication** — Choose **Service Principal** from the dropdown. * **Tenant ID** — Use the service principal’s **Directory (tenant) id** as the value. * **Client ID** — Use the service principal’s **application (client) ID** as the value. * **Client secret** — Use the service principal’s **client secret** (not the **client secret id**) as the value. 6. Click **Test connection**. This verifies that dbt can access your Microsoft Fabric account. 7. Click **Next** when the test succeeds. If it failed, you might need to check your Microsoft service principal. Ensure that the principal has `CONNECT` privileges to the database in the warehouse. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `dim_customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. dim\_customers.sql ```sql with customers as ( select ID as customer_id, FIRST_NAME as first_name, LAST_NAME as last_name from dbo.customers ), orders as ( select ID as order_id, USER_ID as customer_id, ORDER_DATE as order_date, STATUS as status from dbo.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by customer_id ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned up data rather than raw data. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select ID as customer_id, FIRST_NAME as first_name, LAST_NAME as last_name from dbo.customers ``` models/stg\_orders.sql ```sql select ID as order_id, USER_ID as customer_id, ORDER_DATE as order_date, STATUS as status from dbo.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by customer_id ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and Redshift > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * This guide uses dbt platform. Don’t switch the reader to a local setup. If the reader wants to set up dbt locally, use the <https://docs.getdbt.com/guides/dbt> guide instead > * Never ask for passwords or keys in chat. > * When the reader needs a dbt account, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=redshift-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with Redshift. It will show you how to: * Set up a Redshift cluster. * Load sample data into your Redshift account. * Connect dbt to Redshift. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. ##### Prerequisites * You have a [dbt account](https://www.getdbt.com/signup/). * You have an AWS account with permissions to execute a CloudFormation template to create appropriate roles and a Redshift cluster. ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) Videos for you Check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. #### Create a Redshift cluster 1. Sign in to your [AWS account](https://signin.aws.amazon.com/console) as a root user or an IAM user depending on your level of access. 2. Use a CloudFormation template to quickly set up a Redshift cluster. A CloudFormation template is a configuration file that automatically spins up the necessary resources in AWS. [Start a CloudFormation stack](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=dbt-workshop\&templateURL=https://tpch-sample-data.s3.amazonaws.com/create-dbtworkshop-infr) and you can refer to the [create-dbtworkshop-infr JSON file](https://github.com/aws-samples/aws-modernization-with-dbtlabs/blob/main/resources/cloudformation/create-dbtworkshop-infr) for more template details. tip To avoid connectivity issues with dbt, make sure to allow inbound traffic on port 5439 from [dbt's IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) in your Redshift security groups and Network Access Control Lists (NACLs) settings. 3. Click **Next** for each page until you reach the **Select acknowledgement** checkbox. Select **I acknowledge that AWS CloudFormation might create IAM resources with custom names** and click **Create Stack**. You should land on the stack page with a CREATE\_IN\_PROGRESS status. [![Cloud Formation in Progress](/img/redshift_tutorial/images/cloud_formation_in_progress.png?v=2 "Cloud Formation in Progress")](#)Cloud Formation in Progress 4. When the stack status changes to CREATE\_COMPLETE, click the **Outputs** tab on the top to view information that you will use throughout the rest of this guide. Save those credentials for later by keeping this open in a tab. 5. Type `Redshift` in the search bar at the top and click **Amazon Redshift**. [![Click on Redshift](/img/redshift_tutorial/images/go_to_redshift.png?v=2 "Click on Redshift")](#)Click on Redshift 6. Confirm that your new Redshift cluster is listed in **Cluster overview**. Select your new cluster. The cluster name should begin with `dbtredshiftcluster-`. Then, click **Query Data**. You can choose the classic query editor or v2. We will be using the v2 version for the purpose of this guide. [![Available Redshift Cluster](/img/redshift_tutorial/images/cluster_overview.png?v=2 "Available Redshift Cluster")](#)Available Redshift Cluster 7. You might be asked to Configure account. For this sandbox environment, we recommend selecting “Configure account”. 8. Select your cluster from the list. In the **Connect to** popup, fill out the credentials from the output of the stack: * **Authentication** — Use the default which is **Database user name and password**. * **Database** — `dbtworkshop` * **User name** — `dbtadmin` * **Password** — Use the autogenerated `RSadminpassword` from the output of the stack and save it for later. [![Redshift Query Editor v2](/img/redshift_tutorial/images/redshift_query_editor.png?v=2 "Redshift Query Editor v2")](#)Redshift Query Editor v2 [![Connect to Redshift Cluster](/img/redshift_tutorial/images/connect_to_redshift_cluster.png?v=2 "Connect to Redshift Cluster")](#)Connect to Redshift Cluster 9. Click **Create connection**. #### Load data Now we are going to load our sample data into the S3 bucket that our Cloudformation template created. S3 buckets are simple and inexpensive way to store data outside of Redshift. 1. The data used in this course is stored as CSVs in a public S3 bucket. You can use the following URLs to download these files. Download these to your computer to use in the following steps. * [jaffle\_shop\_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) * [jaffle\_shop\_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) * [stripe\_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) 2. Now we are going to use the S3 bucket that you created with CloudFormation and upload the files. Go to the search bar at the top and type in `S3` and click on S3. There will be sample data in the bucket already, feel free to ignore it or use it for other modeling exploration. The bucket will be prefixed with `dbt-data-lake`. [![Go to S3](/img/redshift_tutorial/images/go_to_S3.png?v=2 "Go to S3")](#)Go to S3 3. Click on the `name of the bucket` S3 bucket. If you have multiple S3 buckets, this will be the bucket that was listed under “Workshopbucket” on the Outputs page. [![Go to your S3 Bucket](/img/redshift_tutorial/images/s3_bucket.png?v=2 "Go to your S3 Bucket")](#)Go to your S3 Bucket 4. Click **Upload**. Drag the three files into the UI and click the **Upload** button. [![Upload your CSVs](/img/redshift_tutorial/images/upload_csv.png?v=2 "Upload your CSVs")](#)Upload your CSVs 5. Remember the name of the S3 bucket for later. It should look like this: `s3://dbt-data-lake-xxxx`. You will need it for the next section. 6. Now let’s go back to the Redshift query editor. Search for Redshift in the search bar, choose your cluster, and select Query data. 7. In your query editor, execute this query below to create the schemas that we will be placing your raw data into. You can highlight the statement and then click on Run to run them individually. If you are on the Classic Query Editor, you might need to input them separately into the UI. You should see these schemas listed under `dbtworkshop`. ```sql create schema if not exists jaffle_shop; create schema if not exists stripe; ``` 8. Now create the tables in your schema with these queries using the statements below. These will be populated as tables in the respective schemas. ```sql create table jaffle_shop.customers( id integer, first_name varchar(50), last_name varchar(50) ); create table jaffle_shop.orders( id integer, user_id integer, order_date date, status varchar(50) ); create table stripe.payment( id integer, orderid integer, paymentmethod varchar(50), status varchar(50), amount integer, created date ); ``` 9. Now we need to copy the data from S3. This enables you to run queries in this guide for demonstrative purposes; it's not an example of how you would do this for a real project. Make sure to update the S3 location, iam role, and region. You can find the S3 and iam role in your outputs from the CloudFormation stack. Find the stack by searching for `CloudFormation` in the search bar, then clicking **Stacks** in the CloudFormation tile. ```sql copy jaffle_shop.customers( id, first_name, last_name) from 's3://dbt-data-lake-xxxx/jaffle_shop_customers.csv' iam_role 'arn:aws:iam::XXXXXXXXXX:role/RoleName' region 'us-east-1' delimiter ',' ignoreheader 1 acceptinvchars; copy jaffle_shop.orders(id, user_id, order_date, status) from 's3://dbt-data-lake-xxxx/jaffle_shop_orders.csv' iam_role 'arn:aws:iam::XXXXXXXXXX:role/RoleName' region 'us-east-1' delimiter ',' ignoreheader 1 acceptinvchars; copy stripe.payment(id, orderid, paymentmethod, status, amount, created) from 's3://dbt-data-lake-xxxx/stripe_payments.csv' iam_role 'arn:aws:iam::XXXXXXXXXX:role/RoleName' region 'us-east-1' delimiter ',' ignoreheader 1 Acceptinvchars; ``` Ensure that you can run a `select *` from each of the tables with the following code snippets. ```sql select * from jaffle_shop.customers; select * from jaffle_shop.orders; select * from stripe.payment; ``` #### Connect dbt to Redshift 1. Create a new project in [dbt](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Navigate to **Account settings** (by clicking on your account name in the left side menu), and click **+ New Project**. 2. Enter a project name and click **Continue**. 3. In the **Configure your development environment** section, click the **Connection** dropdown menu and select **Add new connection**. This directs you to the connection configuration settings. 4. In the **Type** section, select **Redshift**. 5. Enter your Redshift settings. Reference your credentials you saved from the CloudFormation template. * **Hostname** — Your entire hostname. * **Port** — `5439` * **Database** (under **Optional settings**) — `dbtworkshop` [![dbt - Redshift Cluster Settings](/img/redshift_tutorial/images/dbt_cloud_redshift_account_settings.png?v=2 "dbt - Redshift Cluster Settings")](#)dbt - Redshift Cluster Settings Avoid connection issues To avoid connection issues with dbt, ensure you follow these minimal but essential AWS network setup steps because Redshift network access isn't configured automatically: * Allow inbound traffic on port `5439` from [dbt's IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) in your Redshift security groups and Network Access Control Lists settings. * Configure your Virtual Private Cloud with the necessary route tables, IP gateways (like an internet or NAT gateway), and inbound rules. For more information, see [AWS documentation on configuring Redshift security group communication](https://docs.aws.amazon.com/redshift/latest/mgmt/rs-security-group-public-private.html). 6. Click **Save**. 7. Set up your personal user credentials by going to **Your profile** > **Credentials**. 8. Select your project that uses the Redshift connection. 9. Click the **configure your development environment and add a connection** link. This directs you to a page where you can enter your personal user credentials. 10. Set your user credentials. These credentials will be used by dbt to connect to Redshift. Those credentials (as provided in your CloudFormation output) will be: * **Username** — `dbtadmin` * **Password** — This is the autogenerated password that you used earlier in the guide * **Schema** — dbt automatically generates a schema name for you. By convention, this is `dbt_<first-initial><last-name>`. This is the schema connected directly to your development environment, and it's where your models will be built when running dbt within the Studio IDE. [![dbt - Redshift User credentials](/img/redshift_tutorial/images/dbt_cloud_redshift_development_credentials.png?v=2 "dbt - Redshift User credentials")](#)dbt - Redshift User credentials 11. Click **Test connection**. This verifies that dbt can access your Redshift cluster. 12. If the test succeeded, click **Save** to complete the configuration. If it failed, you might need to check your Redshift settings and credentials. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from jaffle_shop.customers ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned up data rather than raw data. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and Snowflake > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader's prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * This guide uses dbt platform. Don't switch the reader to a local setup. If the reader wants to set up dbt locally, use the <https://docs.getdbt.com/guides/dbt> guide instead > * Never ask for passwords or keys in chat. > * When the reader needs a dbt account, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=snowflake-qs> > * Ask rather than guess when something is unclear. [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with Snowflake. It will show you how to: * Create a new Snowflake SQL file. * Load sample data into your Snowflake account. * Connect dbt to Snowflake. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add sources to your dbt project. Sources allow you to name and describe the raw data already loaded into Snowflake. * Add tests to your models. * Document your models. * Schedule a job to run. Snowflake also provides a quickstart for you to learn how to use dbt. It makes use of a different public dataset (Knoema Economy Data Atlas) than what's shown in this guide. For more information, refer to [Accelerating Data Teams with dbt & Snowflake](https://quickstarts.snowflake.com/guide/accelerating_data_teams_with_snowflake_and_dbt_cloud_hands_on_lab/) in the Snowflake docs. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have a [trial Snowflake account](https://signup.snowflake.com/). During trial account creation, make sure to choose the **Enterprise** Snowflake edition so you have `ACCOUNTADMIN` access. For a full implementation, you should consider organizational questions when choosing a cloud provider. For more information, see [Introduction to Cloud Platforms](https://docs.snowflake.com/en/user-guide/intro-cloud-platforms.html) in the Snowflake docs. For the purposes of this setup, all cloud providers and regions will work so choose whichever you’d like. ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [How we configure Snowflake](https://blog.getdbt.com/how-we-configure-snowflake/) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. You can also watch the [YouTube video on dbt and Snowflake](https://www.youtube.com/watch?v=kbCkwhySV_I\&list=PL0QYlrC86xQm7CoOH6RS7hcgLnd3OQioG). #### Create a new Snowflake SQL file 1. Log in to your [trial Snowflake account](https://app.snowflake.com). 2. In the Snowflake UI, click the Create icon **+** in the upper left (under the Snowflake logo) to open a dropdown. 3. Select the first option, **SQL File**. #### Load data The data used here is stored as CSV files in a public S3 bucket and the following steps will guide you through how to prepare your Snowflake account for that data and upload it. 1. Create a new virtual warehouse, two new databases (one for raw data, the other for future dbt development), and two new schemas (one for `jaffle_shop` data, the other for `stripe` data). To do this, run these SQL commands by typing them into the Editor of your new Snowflake SQL file and clicking **Run**: ```sql create warehouse transforming; create database raw; create database analytics; create schema raw.jaffle_shop; create schema raw.stripe; ``` 2. In the `raw` database and `jaffle_shop` and `stripe` schemas, create three tables and load relevant data into them: * First, delete all contents (empty) in the Editor of the Snowflake SQL file. Then, run this SQL command to create the `customer` table: ```sql create table raw.jaffle_shop.customers ( id integer, first_name varchar, last_name varchar ); ``` * Delete all contents in the Editor, then run this command to load data into the `customer` table: ```sql copy into raw.jaffle_shop.customers (id, first_name, last_name) from 's3://dbt-tutorial-public/jaffle_shop_customers.csv' file_format = ( type = 'CSV' field_delimiter = ',' skip_header = 1 ); ``` * Delete all contents in the Editor (empty), then run this command to create the `orders` table: ```sql create table raw.jaffle_shop.orders ( id integer, user_id integer, order_date date, status varchar, _etl_loaded_at timestamp default current_timestamp ); ``` * Delete all contents in the Editor, then run this command to load data into the `orders` table: ```sql copy into raw.jaffle_shop.orders (id, user_id, order_date, status) from 's3://dbt-tutorial-public/jaffle_shop_orders.csv' file_format = ( type = 'CSV' field_delimiter = ',' skip_header = 1 ); ``` * Delete all contents in the Editor (empty), then run this command to create the `payment` table: ```sql create table raw.stripe.payment ( id integer, orderid integer, paymentmethod varchar, status varchar, amount integer, created date, _batched_at timestamp default current_timestamp ); ``` * Delete all contents in the Editor, then run this command to load data into the `payment` table: ```sql copy into raw.stripe.payment (id, orderid, paymentmethod, status, amount, created) from 's3://dbt-tutorial-public/stripe_payments.csv' file_format = ( type = 'CSV' field_delimiter = ',' skip_header = 1 ); ``` 3. Verify that the data is loaded by running these SQL queries. Confirm that you can see output for each one. ```sql select * from raw.jaffle_shop.customers; select * from raw.jaffle_shop.orders; select * from raw.stripe.payment; ``` You should see rows returned for all three queries. If a `create` or `copy into` statement fails with an insufficient privileges error, check that your Snowflake trial uses the **Enterprise** edition and that you're using the `ACCOUNTADMIN` role. Continue when all three tables return data. #### Connect dbt to Snowflake There are two ways to connect dbt to Snowflake. The first option is Partner Connect, which provides a streamlined setup to create your dbt account from within your new Snowflake trial account. The second option is to create your dbt account separately and build the Snowflake connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt setup flow, dbt Labs recommends connecting manually. ##### Use Partner Connect Using Partner Connect allows you to create a complete dbt account with your [Snowflake connection](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md), [a managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md), [environments](https://docs.getdbt.com/docs/build/custom-schemas.md#managing-environments), and credentials. 1. On the left sidebar of the Snowflake UI, go to **Admin > Partner Connect**. Find the dbt tile under the **Data Integration** section or search for dbt in the search bar. Click the tile to connect to dbt. [![Snowflake Partner Connect Box](/img/snowflake_tutorial/snowflake_partner_connect_box.png?v=2 "Snowflake Partner Connect Box")](#)Snowflake Partner Connect Box If you’re using the classic version of the Snowflake UI, you can click the **Partner Connect** button in the top bar of your account. From there, click on the dbt tile to open up the connect box. [![Snowflake Classic UI - Partner Connect](/img/snowflake_tutorial/snowflake_classic_ui_partner_connect.png?v=2 "Snowflake Classic UI - Partner Connect")](#)Snowflake Classic UI - Partner Connect 2. In the **Connect to dbt** popup, find the **Optional Grant** option and select the **RAW** and **ANALYTICS** databases. This will grant access for your new dbt user role to each database. Then, click **Connect**. [![Snowflake Classic UI - Connection Box](/img/snowflake_tutorial/snowflake_classic_ui_connection_box.png?v=2 "Snowflake Classic UI - Connection Box")](#)Snowflake Classic UI - Connection Box [![Snowflake New UI - Connection Box](/img/snowflake_tutorial/snowflake_new_ui_connection_box.png?v=2 "Snowflake New UI - Connection Box")](#)Snowflake New UI - Connection Box 3. Click **Activate** when a popup appears: [![Snowflake Classic UI - Actviation Window](/img/snowflake_tutorial/snowflake_classic_ui_activation_window.png?v=2 "Snowflake Classic UI - Actviation Window")](#)Snowflake Classic UI - Actviation Window [![Snowflake New UI - Activation Window](/img/snowflake_tutorial/snowflake_new_ui_activation_window.png?v=2 "Snowflake New UI - Activation Window")](#)Snowflake New UI - Activation Window 4. After the new tab loads, you will see a form. If you already created a dbt account, you will be asked to provide an account name. If you haven't created account, you will be asked to provide an account name and password. 5. After you have filled out the form and clicked **Complete Registration**, you will be logged into dbt automatically. 6. Go to the left side menu and click your account name, then select **Account settings**, choose the "Partner Connect Trial" project, and select **snowflake** in the overview table. Select edit and update the fields **Database** and **Warehouse** to be `analytics` and `transforming`, respectively. [![dbt - Snowflake Project Overview](/img/snowflake_tutorial/dbt_cloud_snowflake_project_overview.png?v=2 "dbt - Snowflake Project Overview")](#)dbt - Snowflake Project Overview [![dbt - Update Database and Warehouse](/img/snowflake_tutorial/dbt_cloud_update_database_and_warehouse.png?v=2 "dbt - Update Database and Warehouse")](#)dbt - Update Database and Warehouse You should now be logged into dbt with a project that has a Snowflake connection using the `analytics` database and `transforming` warehouse. Partner Connect also created a managed repository for you, so skip to [Initialize your dbt project and start developing](#initialize-your-dbt-project-and-start-developing). ##### Connect manually 1. Create a new project in dbt. Navigate to **Account settings** (by clicking on your account name in the left side menu), and click **+ New Project**. 2. Enter a project name and click **Continue**. 3. In the **Configure your development environment** section, click the **Connection** dropdown menu and select **Add new connection**. This directs you to the connection configuration settings. 4. In the **Type** section, select **Snowflake**. [![dbt - Choose Snowflake Connection](/img/snowflake_tutorial/dbt_cloud_setup_snowflake_connection_start.png?v=2 "dbt - Choose Snowflake Connection")](#)dbt - Choose Snowflake Connection 5. Enter your **Settings** for Snowflake with: * **Account** — Find your account by using the Snowflake trial account URL and removing `snowflakecomputing.com`. The order of your account information will vary by Snowflake version. For example, Snowflake's Classic console URL might look like: `oq65696.west-us-2.azure.snowflakecomputing.com`. The AppUI or Snowsight URL might look more like: `snowflakecomputing.com/west-us-2.azure/oq65696`. In both examples, your account will be: `oq65696.west-us-2.azure`. For more information, see [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) in the Snowflake docs.   ✅ `db5261993` or `db5261993.east-us-2.azure`<br />  ❌ `db5261993.eu-central-1.snowflakecomputing.com` * **Role** — Leave blank for now. You can update this to a default Snowflake role later. * **Database** — `analytics`. This tells dbt to create new models in the analytics database. * **Warehouse** — `transforming`. This tells dbt to use the transforming warehouse that was created earlier. [![dbt - Snowflake Account Settings](/img/snowflake_tutorial/dbt_cloud_snowflake_account_settings.png?v=2 "dbt - Snowflake Account Settings")](#)dbt - Snowflake Account Settings 6. Click **Save**. 7. Set up your personal user credentials by going to **Your profile** > **Credentials**. 8. Select your project that uses the Snowflake connection. 9. Click the **configure your development environment and add a connection** link. This directs you to a page where you can enter your personal user credentials. 10. Enter your **User credentials** for Snowflake with: * **Username** — The username you created for Snowflake. The username is not your email address and is usually your first and last name together in one word. * **Password** — The password you set when creating your Snowflake account. * **Schema** — You’ll notice that the schema name has been auto created for you. By convention, this is `dbt_<first-initial><last-name>`. This is the schema connected directly to your development environment, and it's where your models will be built when running dbt within the Studio IDE. * **Target name** — Leave as the default. * **Threads** — Leave as 4. This is the number of simultaneous connects that dbt will make to build models concurrently. [![dbt - Snowflake User credentials](/img/snowflake_tutorial/dbt_cloud_snowflake_development_credentials.png?v=2 "dbt - Snowflake User credentials")](#)dbt - Snowflake User credentials 11. Click **Test connection**. This verifies that dbt can access your Snowflake account. 12. If the test succeeded, click **Save** to complete the configuration. If it failed, check your Snowflake settings and credentials. The most common cause is an **Account** value that still includes `snowflakecomputing.com`. #### Set up a dbt managed repository If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as the Partner Connect provides you with a managed repository. Otherwise, you will need to create your repository connection. When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize your project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit`. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from raw.jaffle_shop.customers ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. If you get an insufficient privileges error, follow the steps in the next note before continuing. info If you receive an insufficient privileges error on Snowflake at this point, it may be because your Snowflake role doesn't have permission to access the raw source data, to build target tables and views, or both. To troubleshoot, use a role with sufficient privileges (like `ACCOUNTADMIN`) and run the following commands in Snowflake. **Note**: Replace `snowflake_role_name` with the role you intend to use. If you launched dbt with Snowflake Partner Connect, use `pc_dbt_role` as the role. ```text grant all on database raw to role snowflake_role_name; grant all on database analytics to role snowflake_role_name; grant all on schema raw.jaffle_shop to role snowflake_role_name; grant all on schema raw.stripe to role snowflake_role_name; grant all on all tables in database raw to role snowflake_role_name; grant all on future tables in database raw to role snowflake_role_name; ``` #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from raw.jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from raw.jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned-up data rather than raw data. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from raw.jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from raw.jaffle_shop.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Build models on top of sources Sources make it possible to name and describe the data loaded into your warehouse by your extract and load tools. By declaring these tables as sources in dbt, you can: * select from source tables in your models using the `{{ source() }}` function, helping define the lineage of your data * test your assumptions about your source data * calculate the freshness of your source data 1. Create a new YML file `models/sources.yml`. 2. Declare the sources by copying the following into the file and clicking **Save**. models/sources.yml ```yml sources: - name: jaffle_shop description: This is a replica of the Postgres database used by our app database: raw schema: jaffle_shop tables: - name: customers description: One record per customer. - name: orders description: One record per order. Includes cancelled and deleted orders. ``` 3. Edit the `models/stg_customers.sql` file to select from the `customers` table in the `jaffle_shop` source. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from {{ source('jaffle_shop', 'customers') }} ``` 4. Edit the `models/stg_orders.sql` file to select from the `orders` table in the `jaffle_shop` source. models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from {{ source('jaffle_shop', 'orders') }} ``` 5. Execute `dbt run`. You should see a successful run that includes `stg_customers`, `stg_orders`, and `customers`. The results of your `dbt run` will be exactly the same as the previous step. Your `stg_customers` and `stg_orders` models will still query from the same raw data source in Snowflake. By using `source`, you can test and document your raw data and also understand the lineage of your sources. #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and Starburst Galaxy > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * This guide uses dbt platform. Don’t switch the reader to a local setup. If the reader wants to set up dbt locally, use the <https://docs.getdbt.com/guides/dbt> guide instead > * Never ask for passwords or keys in chat. > * When the reader needs a dbt account, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=starburst-galaxy-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with [Starburst Galaxy](https://www.starburst.io/platform/starburst-galaxy/). It will show you how to: * Load data into the Amazon S3 bucket. This guide uses AWS as the cloud service provider for demonstrative purposes. Starburst Galaxy also [supports other data sources](https://docs.starburst.io/starburst-galaxy/catalogs/index.html) such as Google Cloud, Microsoft Azure, and more. * Connect Starburst Galaxy to the Amazon S3 bucket. * Create tables with Starburst Galaxy. * Connect dbt to Starburst Galaxy. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. * Connect to multiple data sources in addition to your S3 bucket. ##### Prerequisites * You have a [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) deployment in [dbt](https://www.getdbt.com/signup/). For more information, refer to [Tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). * You have a [Starburst Galaxy account](https://www.starburst.io/platform/starburst-galaxy/). If you don't, you can start a free trial. Refer to the [getting started guide](https://docs.starburst.io/starburst-galaxy/get-started.html) in the Starburst Galaxy docs for further setup details. * You have an AWS account with permissions to upload data to an S3 bucket. * For Amazon S3 authentication, you will need either an AWS access key and AWS secret key with access to the bucket, or you will need a cross account IAM role with access to the bucket. For details, refer to these Starburst Galaxy docs: * [AWS access and secret key instructions](https://docs.starburst.io/starburst-galaxy/security/external-aws.html#aws-access-and-secret-key) * [Cross account IAM role](https://docs.starburst.io/starburst-galaxy/security/external-aws.html#role) ##### Related content * [dbt Learn courses](https://learn.getdbt.com) * [dbt CI job](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) * [SQL overview for Starburst Galaxy](https://docs.starburst.io/starburst-galaxy/sql/index.html) Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. You can also watch the [Build Better Data Pipelines with dbt and Starburst](https://www.youtube.com/watch?v=tfWm4dWgwRg) YouTube video produced by Starburst Data, Inc. #### Load data to an Amazon S3 bucket Using Starburst Galaxy, you can create tables and also transform them with dbt. Start by loading the Jaffle Shop data (provided by dbt Labs) to your Amazon S3 bucket. Jaffle Shop is a fictional cafe selling food and beverages in several US cities. 1. Download these CSV files to your local machine: * [jaffle\_shop\_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) * [jaffle\_shop\_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) * [stripe\_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) 2. Upload these files to S3. For details, refer to [Upload objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html) in the Amazon S3 docs. When uploading these files, you must create the following folder structure and upload the appropriate file to each folder: ```text <bucket/blob> dbt-quickstart (folder) jaffle-shop-customers (folder) jaffle_shop_customers.csv (file) jaffle-shop-orders (folder) jaffle_shop_orders.csv (file) stripe-payments (folder) stripe-payments.csv (file) ``` #### Connect Starburst Galaxy to the Amazon S3 bucket If your Starburst Galaxy instance is not already connected to your S3 bucket, you need to create a cluster, configure a catalog that allows Starburst Galaxy to connect to the S3 bucket, add the catalog to your new cluster, and configure privilege settings. In addition to Amazon S3, Starburst Galaxy supports many other data sources. To learn more about them, you can refer to the [Catalogs overview](https://docs.starburst.io/starburst-galaxy/catalogs/index.html) in the Starburst Galaxy docs. 1. Create a cluster. Click **Clusters** on the left sidebar of the Starburst Galaxy UI, then click **Create cluster** in the main body of the page. 2. In the **Create a new cluster** modal, you only need to set the following options. You can use the defaults for the other options. * **Cluster name** — Type a name for your cluster. * **Cloud provider region** — Select the AWS region. When done, click **Create cluster**. 3. Create a catalog. Click **Catalogs** on the left sidebar of the Starburst Galaxy UI, then click **Create catalog** in the main body of the page. 4. On the **Create a data source** page, select the Amazon S3 tile. 5. In the **Name and description** section of the **Amazon S3** page, fill out the fields. 6. In the **Authentication to S3** section of the **Amazon S3** page, select the [AWS (S3) authentication mechanism](#prerequisites) you chose to connect with. 7. In the **Metastore configuration** section, set these options: * **Default S3 bucket name** — Enter the name of your S3 bucket you want to access. * **Default directory name** — Enter the folder name of where the Jaffle Shop data lives in the S3 bucket. This is the same folder name you used in [Load data to an Amazon S3 bucket](#load-data-to-s3). * **Allow creating external tables** — Enable this option. * **Allow writing to external tables** — Enable this option. The **Amazon S3** page should look similar to this, except for the **Authentication to S3** section which is dependant on your setup: [![Amazon S3 connection settings in Starburst Galaxy](/img/quickstarts/dbt-platform/starburst-galaxy-config-s3.png?v=2 "Amazon S3 connection settings in Starburst Galaxy")](#)Amazon S3 connection settings in Starburst Galaxy 8. Click **Test connection**. This verifies that Starburst Galaxy can access your S3 bucket. 9. Click **Connect catalog** if the connection test passes. [![Successful connection test](/img/quickstarts/dbt-platform/test-connection-success.png?v=2 "Successful connection test")](#)Successful connection test 10. On the **Set permissions** page, click **Skip**. You can add permissions later if you want. 11. On the **Add to cluster** page, choose the cluster you want to add the catalog to from the dropdown and click **Add to cluster**. 12. Add the location privilege for your S3 bucket to your role in Starburst Galaxy. Click **Access control > Roles and privileges** on the left sidebar of the Starburst Galaxy UI. Then, in the **Roles** table, click the role name **accountadmin**. If you're using an existing Starburst Galaxy cluster and don't have access to the accountadmin role, then select a role that you do have access to. To learn more about access control, refer to [Access control](https://docs.starburst.io/starburst-galaxy/security/access-control.html) in the Starburst Galaxy docs. 13. On the **Roles** page, click the **Privileges** tab and click **Add privilege**. 14. On the **Add privilege** page, set these options: * **What would you like to modify privileges for?** — Choose **Location**. * **Enter a storage location provide** — Enter the storage location of *your S3 bucket* and the folder of where the Jaffle Shop data lives. Make sure to include the `/*` at the end of the location. * **Create SQL** — Enable the option. When done, click **Add privileges**. [![Add privilege to accountadmin role](/img/quickstarts/dbt-platform/add-privilege.png?v=2 "Add privilege to accountadmin role")](#)Add privilege to accountadmin role #### Create tables with Starburst Galaxy To query the Jaffle Shop data with Starburst Galaxy, you need to create tables using the Jaffle Shop data that you [loaded to your S3 bucket](#load-data-to-s3). You can do this (and run any SQL statement) from the [query editor](https://docs.starburst.io/starburst-galaxy/query/query-editor.html). 1. Click **Query > Query editor** on the left sidebar of the Starburst Galaxy UI. The main body of the page is now the query editor. 2. Configure the query editor so it queries your S3 bucket. In the upper right corner of the query editor, select your cluster in the first gray box and select your catalog in the second gray box: [![Set the cluster and catalog in query editor](/img/quickstarts/dbt-platform/starburst-galaxy-editor.png?v=2 "Set the cluster and catalog in query editor")](#)Set the cluster and catalog in query editor 3. Copy and paste these queries into the query editor. Then **Run** each query individually. Replace `YOUR_S3_BUCKET_NAME` with the name of your S3 bucket. These queries create a schema named `jaffle_shop` and also create the `jaffle_shop_customers`, `jaffle_shop_orders`, and `stripe_payments` tables: ```sql CREATE SCHEMA jaffle_shop WITH (location='s3://YOUR_S3_BUCKET_NAME/dbt-quickstart/'); CREATE TABLE jaffle_shop.jaffle_shop_customers ( id VARCHAR, first_name VARCHAR, last_name VARCHAR ) WITH ( external_location = 's3://YOUR_S3_BUCKET_NAME/dbt-quickstart/jaffle-shop-customers/', format = 'csv', type = 'hive', skip_header_line_count=1 ); CREATE TABLE jaffle_shop.jaffle_shop_orders ( id VARCHAR, user_id VARCHAR, order_date VARCHAR, status VARCHAR ) WITH ( external_location = 's3://YOUR_S3_BUCKET_NAME/dbt-quickstart/jaffle-shop-orders/', format = 'csv', type = 'hive', skip_header_line_count=1 ); CREATE TABLE jaffle_shop.stripe_payments ( id VARCHAR, order_id VARCHAR, paymentmethod VARCHAR, status VARCHAR, amount VARCHAR, created VARCHAR ) WITH ( external_location = 's3://YOUR_S3_BUCKET_NAME/dbt-quickstart/stripe-payments/', format = 'csv', type = 'hive', skip_header_line_count=1 ); ``` 4. When the queries are done, you can see the following hierarchy on the query editor's left sidebar: [![Hierarchy of data in query editor](/img/quickstarts/dbt-platform/starburst-data-hierarchy.png?v=2 "Hierarchy of data in query editor")](#)Hierarchy of data in query editor 5. Verify that the tables were created successfully. In the query editor, run the following queries: ```sql select * from jaffle_shop.jaffle_shop_customers; select * from jaffle_shop.jaffle_shop_orders; select * from jaffle_shop.stripe_payments; ``` #### Connect dbt to Starburst Galaxy 1. Make sure you are still logged in to [Starburst Galaxy](https://galaxy.starburst.io/login). 2. If you haven’t already, set your account’s role to accountadmin. Click your email address in the upper right corner, choose **Switch role** and select **accountadmin**. If this role is not listed for you, choose the role you selected in [Connect Starburst Galaxy to the Amazon S3 bucket](#connect-to-s3-bucket) when you added location privilege for your S3 bucket. 3. Click **Clusters** on the left sidebar. 4. Find your cluster in the **View clusters** table and click **Connection info**. Choose **dbt** from the **Select client** dropdown. Keep the **Connection information** modal open. You will use details from that modal in dbt. 5. In another browser tab, log in to [dbt](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). 6. Create a new project in dbt. Click on your account name in the left side menu, select **Account settings**, and click **+ New Project**. 7. Enter a project name and click **Continue**. 8. Choose **Starburst** as your connection and click **Next**. 9. Enter the **Settings** for your new project: * **Host** – The **Host** value from the **Connection information** modal in your Starburst Galaxy tab. * **Port** – 443 (which is the default) 10. Enter the **User credentials** for your new project: * **User** – The **User** value from the **Connection information** modal in your Starburst Galaxy tab. Make sure to use the entire string, including the account's role which is the `/` and all the characters that follow. If you don’t include it, your default role is used and that might not have the correct permissions for project development. * **Password** – The password you use to log in to your Starburst Galaxy account. * **Database** – The Starburst catalog you want to save your data to (for example, when writing new tables). For future reference, database is synonymous to catalog between dbt and Starburst Galaxy. * Leave the remaining options as is. You can use their default values. 11. Click **Test Connection**. This verifies that dbt can access your Starburst Galaxy cluster. 12. Click **Next** if the test succeeded. If it failed, you might need to check your Starburst Galaxy settings and credentials. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from dbt_quickstart.jaffle_shop.jaffle_shop_customers ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from dbt_quickstart.jaffle_shop.jaffle_shop_customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from dbt_quickstart.jaffle_shop.jaffle_shop_orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools (like [dbt Charts](https://dbtcharts.com/)) to these views and tables so they only read cleaned up data rather than raw data. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from dbt_quickstart.jaffle_shop.jaffle_shop_customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from dbt_quickstart.jaffle_shop.jaffle_shop_orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. (Applies to dbt v2.0 and later) You don't need to add a docs step. Every job run refreshes your project's metadata in [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) automatically. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, (Applies to dbt v2.0 and later) click **Catalog** in the navigation to explore your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. #### Connect to multiple data sources This quickstart focuses on using dbt to run models against a data lake (S3) by using Starburst Galaxy as the query engine. In most real world scenarios, the data that is needed for running models is actually spread across multiple data sources and is stored in a variety of formats. With Starburst Galaxy, Starburst Enterprise, and Trino, you can run your models on any of the data you need, no matter where it is stored. If you want to try this out, you can refer to the [Starburst Galaxy docs](https://docs.starburst.io/starburst-galaxy/catalogs/) to add more data sources and load the Jaffle Shop data into the source you select. Then, extend your models to query the new data source and the data source you created in this quickstart. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt and Teradata > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * This guide uses dbt platform. Don’t switch the reader to a local setup. If the reader wants to set up dbt locally, use the <https://docs.getdbt.com/guides/dbt> guide instead > * Never ask for passwords or keys in chat. > * When the reader needs a dbt account, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=teradata-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with Teradata Vantage. It will show you how to: * Create a new Teradata Clearscape instance * Load sample data into your Teradata Database * Connect dbt to Teradata. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have access to a Teradata Vantage instance. You can provision one for free at <https://clearscape.teradata.com>. See [the ClearScape Analytics Experience guide](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/) for details. ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [How we provision Teradata Clearscape Vantage instance](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. #### Load data The following steps will guide you through how to get the data stored as CSV files in a public S3 bucket and insert it into the tables. SQL Studio IDE If you created your Teradata Vantage database instance at <https://clearscape.teradata.com> and you don't have an SQL Studio IDE handy, use the JupyterLab bundled with your database to execute SQL: 1. Navigate to [ClearScape Analytics Experience dashboard](https://clearscape.teradata.com/dashboard) and click the **Run Demos** button. The demo will launch JupyterLab. 2. In JupyterLab, go to **Launcher** by clicking the blue **+** icon in the top left corner. Find the **Notebooks** section and click **Teradata SQL**. 3. In the notebook's first cell, connect to the database using `connect` magic. You will be prompted to enter your database password when you execute it: ```ipynb %connect local ``` 4. Use additional cells to type and run SQL statements. 1) Use your preferred SQL IDE editor to create the database, `jaffle_shop`: ```sql CREATE DATABASE jaffle_shop AS PERM = 1e9; ``` 2) In `jaffle_shop` database, create three foreign tables and reference the respective csv files located in object storage: ```sql CREATE FOREIGN TABLE jaffle_shop.customers ( id integer, first_name varchar (100), last_name varchar (100), email varchar (100) ) USING ( LOCATION ('/gs/storage.googleapis.com/clearscape_analytics_demo_data/dbt/raw_customers.csv') ) NO PRIMARY INDEX; CREATE FOREIGN TABLE jaffle_shop.orders ( id integer, user_id integer, order_date date, status varchar(100) ) USING ( LOCATION ('/gs/storage.googleapis.com/clearscape_analytics_demo_data/dbt/raw_orders.csv') ) NO PRIMARY INDEX; CREATE FOREIGN TABLE jaffle_shop.payments ( id integer, orderid integer, paymentmethod varchar (100), amount integer ) USING ( LOCATION ('/gs/storage.googleapis.com/clearscape_analytics_demo_data/dbt/raw_payments.csv') ) NO PRIMARY INDEX; ``` #### Connect dbt to Teradata 1. Create a new project in dbt. Click on your account name in the left side menu, select **Account settings**, and click **+ New Project**. 2. Enter a project name and click **Continue**. 3. In the **Configure your development environment** section, click the **Connection** dropdown menu and select **Add new connection**. 4. In the **Type** section, select **Teradata**. 5. Enter your Teradata settings and click **Save**. [![dbt - Choose Teradata Connection](/img/teradata/dbt_cloud_teradata_setup_connection_start.png?v=2 "dbt - Choose Teradata Connection")](#)dbt - Choose Teradata Connection [![dbt - Teradata Account Settings](/img/teradata/dbt_cloud_teradata_account_settings.png?v=2 "dbt - Teradata Account Settings")](#)dbt - Teradata Account Settings 6. Set up your personal user credentials by navigating to **Your profile** > **Credentials**. 7. Select your project that uses the Teradata connection. 8. Click the **configure your development environment and add a connection** link. This directs you to a page where you can enter your personal user credentials. 9. Enter your **User credentials** for Teradata with: * **Username** — The username of Teradata database. * **Password** — The password of Teradata database. * **Schema** — The default database to use. [![dbt - Teradata User credentials](/img/teradata/dbt_cloud_teradata_development_credentials.png?v=2 "dbt - Teradata User credentials")](#)dbt - Teradata User credentials 10. Click **Test Connection** to verify that dbt can access your Teradata Vantage instance. 11. If the test succeeded, click **Save** to complete the configuration. If it failed, you may need to check your Teradata settings and credentials. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize your project** to build out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` to create the first commit to your managed repo. Once you’ve created the commit, you can open a branch to add new dbt code. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: my_new_project: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: my_new_project: +materialized: table ``` 3. Save your changes. 4. Commit your changes and merge to the main branch. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, lint files, or execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will receive a prompt to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `bi_customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. You can connect your business intelligence (BI) tools to these views and tables so they only read cleaned-up data rather than raw data in your BI tool. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? dbt Wizard can build the same model for you, right in the Studio IDE or [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md?version=2\§ion=qs-build-with-ai), grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Open dbt Wizard from the Studio IDE. [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. 2. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 3. Review the SQL dbt Wizard generates, then accept it to save the model. 4. Type `dbt run` to build it. Either way, you end up with the same working model. Learn more about [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?section=qs-build-with-ai). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table ``` * Click **Save**. 2. Enter the `dbt run` command. Your `bi_customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/bi_customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/bi\_customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `bi_customers`, should now build as a view. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in your original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop.customers ``` 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in your original query. models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ``` 3. Edit the SQL in your `models/bi_customers.sql` file as follows: models/bi\_customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, it created separate views/tables for `stg_customers`, `stg_orders`, and `customers`. dbt inferred the order in which these models should run. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You don’t need to define these dependencies explicitly. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Build models on top of sources Sources make it possible to name and describe the data loaded into your warehouse by your extract and load tools. By declaring these tables as sources in dbt, you can: * Select from source tables in your models using the `{{ source() }}` function, helping define the lineage of your data * Test your assumptions about your source data * Calculate the freshness of your source data 1. Create a new YML file, `models/sources.yml`. 2. Declare the sources by copying the following into the file and clicking **Save**. models/sources.yml ```yml version: 2 sources: - name: jaffle_shop description: This is a replica of the Postgres database used by the app database: raw schema: jaffle_shop tables: - name: customers description: One record per customer. - name: orders description: One record per order. Includes canceled and deleted orders. ``` 3. Edit the `models/stg_customers.sql` file to select from the `customers` table in the `jaffle_shop` source. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from {{ source('jaffle_shop', 'customers') }} ``` 4. Edit the `models/stg_orders.sql` file to select from the `orders` table in the `jaffle_shop` source. models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from {{ source('jaffle_shop', 'orders') }} ``` 5. Execute `dbt run`. Your `dbt run` results will be the same as those in the previous step. Your `stg_customers` and `stg_orders` models will still query from the same raw data source in Teradata. By using `source`, you can test and document your raw data and also understand the lineage of your sources. #### Add data tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new properties YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: bi_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each data test. Each query will return the number of records that fail the test. If this number is 0, then the data test is successful. ###### FAQs What data tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. 1. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml models: - name: bi_customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 2. Run `dbt docs generate` to generate the documentation for your project. dbt introspects your project and your warehouse to generate a JSON file with rich documentation about your project. 3. Click the book icon in the Develop interface to launch documentation in a new tab. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. In the upper left, select **Deploy**, then click **Environments**. 2. Click **Create Environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. In the **dbt Version** field, select the latest version from the dropdown. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as `jaffle_shop_prod`. This will allow dbt to build and work with that dataset. 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `bi_customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Deploy** in the upper left, then click **Jobs**. 2. Click **+ Create job** and then select **Deploy job**. Provide a name, for example, "Production run", and link it to the Environment you just created. 3. Scroll down to the **Execution Settings** section. 4. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 5. Select the **Generate docs on run** checkbox to automatically [generate updated project docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) each time your job runs. 6. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 7. Select **Save**, then click **Run now** to run your job. 8. Click the run and watch its progress under "Run history." 9. Once the run is complete, click **View Documentation** to see the docs for your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt Canvas [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Canvas offers a quick and straightforward way for anyone to build analytics models, no background in analytics engineering is required! In this guide, you will learn about: * Accessing Canvas and creating a new model * Navigating the interface * Building a model using operators * Committing your changes to Git * Locating your Canvas model and data #### Canvas prerequisites Before using Canvas, you should: * Have a [dbt Enterprise or Enterprise+](https://www.getdbt.com/pricing) account. * Have a [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) with user credentials set up. * Be using one of the following adapters: * Bigquery * Databricks * Redshift * Snowflake * Trino * You can access the Canvas with adapters not listed, but some features may be missing at this time. * Use [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md), [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md), or [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) as your Git provider, connected to dbt via HTTPS. * SSH connections aren't supported at this time. * Self-hosted or on-premises deployments of any Git provider aren't supported for Canvas at this time. * Have an existing dbt project already created with a Staging or Production run completed. * Verify your Development environment is on a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing updates. * Have read-only access to the [Staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) with the data to be able to execute `run` in the Canvas. To customize the required access for the Canvas user group, refer to [Set up environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md) for more information. Prerequisite for using the Jaffle Shop The examples in this guide use the [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) GitHub repo sample project. You can use your own data, but the Jaffle Shop offers a full-featured project useful for testing dbt features. Ask your dbt administrator about importing it to a project in your environment. They must also execute `dbt run` on the Jaffle Shop project in your `Production` environment before you begin, or you will be unable to reference the source models. #### Access Canvas To access Canvas: 1. Click **Canvas** on the left-side menu. 2. From the right side, click **Create new workspace**. This will open a new workspace with a blank untitled model. You don't need to take any additional action to continue with this guide, but in scenarios where you want to create a new model, click **+Add** on the top navigation bar and click **Create new model**. [![Create a new model from the Canvas landing page.](/img/docs/dbt-platform/canvas/canvas-create-new-model.png?v=2 "Create a new model from the Canvas landing page.")](#)Create a new model from the Canvas landing page. #### Navigating the interface Canvas comprises a series of menus activated by clicking icons surrounding the border of the canvas workspace area. With none of the menu items activated, the workspace looks like this: [![The Canvas workspace screen. The number of items is defined in this section.](/img/docs/dbt-platform/canvas/canvas-screen.png?v=2 "The Canvas workspace screen. The number of items is defined in this section.")](#)The Canvas workspace screen. The number of items is defined in this section. Click on an icon to expand its section or execute an action depending on its purpose. The options are as follows: 1. The main menu (click on the **dbt logo**) and the workspace's title. The title default is random but can be edited anytime by clicking on it. 2. The **current model tab** and name. The name for the model is set with the **Output** operator. 3. The **model icon** button. Manage your models in the workspace. 4. The **Runs** pane displays run data, including warnings and errors. 5. The **Previews** pane that displays previews data for individual operators. 6. The **Add** option for creating new models, editing existing, or adding seed files. 7. The **Operators** toolbar (`Input`, `Transform`, and `Output`) contains the building blocks for creating a model with the editor. 8. The [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) icon (where available). Use natural language to build your Canvas models. 9. The **SQL code** area displays the SQL that compiles your model. 10. The **Run** command executes `dbt run` for the model. 11. This button is initially a **Commit** command for your integrated Git provider. It will change to "Open pull request" once your first commit is made. The button will not initially appear until you begin working in the canvas area. 12. The navigation tab that has icons for (from top to bottom): * Zoom in * Zoom out * Center the model to fit the screen * Zoom to selection (the operator selected on or off screen will be zoomed and centered) * Auto-layout option for the individual operator tiles #### Create a model This section will walk you through creating a model with operators using sample data from the [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) project. With this guide, you will create a basic model that transforms two datasets to build a view of repeat customer purchases while you consider a loyalty program for your shop. The operators are the heart of your model. They determine what data will be transformed and how. Click the **+** icon to open the operator menu. Operators are divided into three types: * **Input:** Input operators configure the source data. * **Transform:** Transform operators change and shape your data. * **Output:** Output operators define your model name and location. [![The operator’s menu on the side of the Canvas workspace.](/img/docs/dbt-platform/canvas/operators.png?v=2 "The operator’s menu on the side of the Canvas workspace.")](#)The operator’s menu on the side of the Canvas workspace. Read more about the [individual operators](https://docs.getdbt.com/docs/platform/canvas-interface.md#operators) to understand the basic purpose of each. The dbt model created by the Canvas builds off of existing models. In this guide, there will be input (source) models and an output model (what you are building) which will be *your model*. More about operator tiles The operators are drag-and-drop from their menu to the canvas, and when they are dropped they will create a tile. The tiles have the same basic setup with different fields depending on their function. All operators except for **Model** must be connected to another tile before configuring. Once configured, they’ll have the same basic layout. [![An operator tile with configurations filled out.](/img/docs/dbt-platform/canvas/operator-tile.png?v=2 "An operator tile with configurations filled out.")](#)An operator tile with configurations filled out. 1. **The connectors:** Click-and-drag to the connector on another operator to link them. Some connectors have L and R markers. When implementing joins, they designate the left and right joins, respectively. 2. **The title:** Click to change. The examples and images in this guide will use the default names. 3. **Play icon and menu:** Preview the data at any point in its transformation by clicking the tile's **play icon**. The dropdown menu contains the option to **Delete** a tile. 4. **Column icon:** The number next to it represents the number of columns in the data at that point in its transformation. tip Make operator tile titles unique compared to your column names to avoid confusion, and the same applies to any aliases you create. ##### Create your model from pre-existing models To get started: 1. Click the **Input** menu and drag the **Input Model** operator over to the canvas. 2. Click **Choose a model** and then select the source `stg_models` from the dropdown. 3. Click the **Select model** option in the window that lists the columns. [![A single model operator.](/img/docs/dbt-platform/canvas/one-model-operator.png?v=2 "A single model operator.")](#)A single model operator. You now have your first input model in Canvas! 4. Drag a new **Input Model** operator to the canvas below the first and repeat the previous steps, but this time set the source model to `stg_order_items`. [![Two model operators in the canvas.](/img/docs/dbt-platform/canvas/two-model-operators.png?v=2 "Two model operators in the canvas.")](#)Two model operators in the canvas. Now, you have two input models and are ready to transform the data! tip Don't see a pre-existing model you're looking for? Ask your dbt admins to ensure it's been run in your Production environment recently and hasn't gone stale. ##### Create a join 1. From the **Operators** menu, click **Transform** and drag the **Join** operator onto the canvas to the right of the source models. [![A join that has not been connected to the models](/img/docs/dbt-platform/canvas/join-not-connected.png?v=2 "A join that has not been connected to the models")](#)A join that has not been connected to the models 2. Click and drag a line from the **+** connector below the `L` on the join border to the **+** on the `stg_orders` model. Do the same for the `R` connector to the `stg_order_items` model. [![The join is connected to two model operators.](/img/docs/dbt-platform/canvas/join-connected.png?v=2 "The join is connected to two model operators.")](#)The join is connected to two model operators. 3. In the **Join** tile, click **Configure inputs.** 4. Set the **Join type** to `Inner`. 5. In the pair of dropdowns, set both `stg_orders` and `stg_order_items` to `ORDER_ID`. 6. Click **Select and rename columns** and click **Configure columns** select the following columns: * From `stg_orders` click `ORDER_ID` and `CUSTOMER_ID`. * From `stg_order_items` click `PRODUCT_ID`. * Note: These will appear in the order they are clicked. 7. You've now built your join! Test it by clicking the **Play icon** in the top right corner of the join tile. Your data will populate in the **Runs and previews** pane. [![A completed join with the sample data.](/img/docs/dbt-platform/canvas/preview-join.png?v=2 "A completed join with the sample data.")](#)A completed join with the sample data. tip Your work in the Canvas is automatically saved as you progress, so if you need a break, you can always come back to a session later. Just be sure to give it a unique title! #### Enhance your model You've got the basics going with your Canvas model! It has successfully joined two pre-existing input models, but you want to transform the data further to get what you need: a list of customers who buy repeat items as you consider a loyalty club rewards program. ##### Aggregate data Multiple options for transforming your data include custom formulas, filters, and unions. Keep it simple and add an aggregation operator to tell you which customers buy the most repeat products. 1. From **Transform**, drag the **Aggregation** operator over to the right of the join. 2. Connect the aggregation operator to the join operator. 3. Click **Configure aggregation** in the **Aggregation tile**. 4. Click in the **Group by** field and first select `CUSTOMER_ID` then `PRODUCT_ID`. 5. Configure the next three fields with the following: * **Function:** Count * **Column:** PRODUCT\_ID * **Alias:** count\_PRODUCT\_ID [![The configured aggregation operator tile.](/img/docs/dbt-platform/canvas/aggregation.png?v=2 "The configured aggregation operator tile.")](#)The configured aggregation operator tile. 6. Click the **Play icon** to preview the data. You're starting to see the results you're looking for, but the data is scattered. Let's clean it up a bit more. tip As your model grows, you can zoom in and out to view your needs. Click and hold in empty canvas space to drag your setup across the screen. Click the **Fit view** icon to see your entire model on the screen. Click the **Auto layout** icon to auto-arrange the tiles efficiently. ##### Add some order There's a lot of data there. Dozens of customers are buying hundreds of products. You will sort it so that the customers are listed ascending by their CUSTOMER\_ID number, with the most purchased products listed in descending order. 1. From **Transform**, drag the **Order** operator over to the right of the **Aggregation** tile and connect them. 2. Click the **pencil edit icon**. 3. In the **Sort order** field click **Select column** and click `Aggregation1.CUSTOMER_ID` from the dropdown. Set it to `Asc`. 4. Click **Add sorting** and in the new **Select column** field select `Aggregation1.count_PRODUCT_ID`. Set it to `Desc`. 5. Press the **Play icon** to preview the new data. [![The ordered data operator tile config.](/img/docs/dbt-platform/canvas/order.png?v=2 "The ordered data operator tile config.")](#)The ordered data operator tile config. tip Want to practice on your own? Try adding a **Filter** operator that removes items with less than 10 sales for any customer ID. Be sure to run the preview and verify the data is correct. #### Configure your output model Now that you've built your model, you need to customize the output name and location: 1. From **Output**, drag the **Output Model** operator to the right of your **Order** operator. 2. Connect the **Order** and **Output Model** operators. 3. The **Output Model** configuration will default to the name of your Canvas project and the default models directory. Click the **pencil edit icon** to configure the optional fields: * Edit the **Model name** field if you want the name to be different than that of your project. * Edit the **File path** if you have a custom directory for your Canvas models. * Hover over a column name and click the **-** icon to remove it from the output model. 4. Click the **play icon** to preview your final model. [![The output model configures your final model's name and location.](/img/docs/dbt-platform/canvas/output-model.png?v=2 "The output model configures your final model's name and location.")](#)The output model configures your final model's name and location. Model locations You can customize the location for Canvas models to keep them separate from other dbt models. Check with your dbt admins for best practices and ideas for Canvas model locations and naming conventions. #### Run and share your model Now that you've built a model that results in the data you want, it's time to run it and push it to your Git repo. Before you run your model, keep a few items in mind: * When you run previews (at any stage in the process), it does not affect the state of your warehouse. So, you can test and develop in the Canvas without impacting anything outside of the dbt Development environment. * When you're ready to use this model in a downstream tool, you can run it to materialize it in your data warehouse development schema. * Once your model is ready for production and ready to be used by others or orchestrated, commit it and open a pull request. ##### Run To run your model, you only need to click the big **Run** button. With the Canvas, there is no command line and no need to memorize a list of commands; there is only **Run**. Click it to see the results populate in the **Runs and previews** pane. [![The results of a successful run in the 'Runs and previews' pane.](/img/docs/dbt-platform/canvas/run-results.png?v=2 "The results of a successful run in the 'Runs and previews' pane.")](#)The results of a successful run in the 'Runs and previews' pane. This will [materialize](https://docs.getdbt.com/docs/build/materializations.md) the data as a `view` in your developer schema in the database. Once the model has been merged with your project and `dbt run` is executed in your Staging or Production environments, it will be materialized as a view in related schemas. [![Preview of the transformed data in Snowflake.](/img/docs/dbt-platform/canvas/preview-data.png?v=2 "Preview of the transformed data in Snowflake.")](#)Preview of the transformed data in Snowflake. tip Have dbt [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) enabled for your dbt Enterprise account? Clear the board and try using natural language to build the model in this guide without manually configuring any operators. ##### Git commit The models built in the Canvas are a part of your larger dbt project. They are stored in the `visual_editor` folder of your `/models` directory. This is all done automatically; you don't have to configure any paths or directories. [![Example of the Canvas model path in GitHub.](/img/docs/dbt-platform/canvas/ve-model-folder.png?v=2 "Example of the Canvas model path in GitHub.")](#)Example of the Canvas model path in GitHub. However, it won't be created in your Git repo until you commit your first model. So, back in the model's view: 1. Click **Commit** in the top right. * If you've already created a commit and wish to make more, click the arrow next to **Create a pull request** to see the **Commit** option. 2. Fill out the **Description** field with information about your model. If it's long, part of it will be included in the pull request title, and the rest will be in the body. That's okay! You can correct it during the PR creation process. 3. Click **Commit**. 4. The **Commit** button will change to **Create a pull request**. You can add more commits, but click the **Create a pull request** button for now. You will then be redirected to your Git provider in a new tab. The following example uses GitHub as the provider: [![Example of the screen you're taken to in GitHub when you create a pull request from Canvas.](/img/docs/dbt-platform/canvas/demo-model-github.png?v=2 "Example of the screen you're taken to in GitHub when you create a pull request from Canvas.")](#)Example of the screen you're taken to in GitHub when you create a pull request from Canvas. 5. Click **Create pull request** in the GitHub window. 6. Complete the **Add a title** and **Add a description** fields. If your description is split between both, copy all the contents to the description field and give it a shorter title. 7. Click **Create pull request**. You've just submitted your first model from the Canvas for review. Once approved and merged, the model will be included in your organization’s project and run whenever `dbt run` is executed in any environment your model is in. You're now on your way to becoming an expert in data transformation! tip Want to take your skills to the next level? Try taking the SQL output from your Canvas model and using it to create a model in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). Want to learn more? Be sure to check out our [Canvas fundamentals course](https://learn.getdbt.com/learn/course/canvas-fundamentals) on [dbt Learn](https://learn.getdbt.com/catalog). --- ### Quickstart for dbt Charts [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction dbt Charts is a visualization layer that turns YAML files into interactive dashboards. You define queries, charts, and layouts as text and store them in Git alongside your dbt project. Because dbt Charts queries your dbt models directly, your dashboards stay in sync with the same transformations your team already trusts. You develop dbt Charts locally with the `dct` command-line tool: it reads a board file, runs its queries against the connection in your dbt `profiles.yml`, and serves the result in your browser with hot reload as you edit. In this quickstart, you'll learn how to: * Create the `charts/` directory and a board file by hand. * Write your first five charts — a line, bar, area, pie, and table — from your dbt models. * Install dbt Charts and preview the board locally. This guide uses the [`fusion-jaffle-shop`](https://github.com/matthewshaver/jaffle-shop-multi-adapter) sample project, which is built for v1.12+ and includes marts such as `order_items` and `location_performance`. When you finish, you'll have a board that trends revenue over time, compares performance across store locations, and lists location-level key performance indicators (KPIs). ##### Prerequisites * You have [dbt v1.12+ installed](https://docs.getdbt.com/docs/local/install-dbt.md?version=2). This guide assumes dbt v2. * If you're using v2, you must install the v1 adapter for your dbt Charts project to load in your browser. You don't have to do any additional configuration. For example, if you're using Snowflake, run `pip install dbt-snowflake` before installing dbt Charts. As development on dbt Charts continues, this prerequisite will eventually be removed. * You have [Git](https://git-scm.com/downloads) and basic familiarity with the command line and YAML. * You have [Python](https://www.python.org/downloads/) 3.9 or later and a package manager such as [uv](https://docs.astral.sh/uv/) or `pip`, to install the `dct` tool. * You have a working connection to your data platform (Snowflake in this guide) that dbt can build into. ##### Related content * [Build your dbt models](https://docs.getdbt.com/docs/build/models.md) * [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [dbt Charts query types](https://docs.dataface.com/queries/) * [dbt Charts documentation](https://docs.dataface.com/) #### Clone the sample project and build it dbt Charts queries tables that already exist in your data platform, so build the project's models before you chart them. 1. Clone the sample project and change into it: ```bash git clone https://github.com/matthewshaver/jaffle-shop-multi-adapter.git cd fusion-jaffle-shop ``` 2. Add a `profiles.yml` for your data platform. The project's `dbt_project.yml` sets `profile: default`, so name your profile `default` and give it a `dev` target. dbt reads a `profiles.yml` in the project root, or falls back to `~/.dbt/profiles.yml`. For platform-specific fields, refer to [Connection profiles](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md). 3. Install the project's packages: ```bash dbt deps ``` 4. Seed the raw CSVs and build the models and tests in one command: ```bash dbt build ``` The `dbt build` command runs seeds, models, snapshots, and tests in dependency order, so you don't need a separate `dbt seed` step. For the full list of commands, refer to [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md). 5. Confirm that a mart the charts read returns rows: ```bash dbt show --inline "select count(*) from {{ ref('order_items') }}" ``` note Record the database and schema that dbt built into — your chart SQL references them. To find them, run `dbt list --output json` and read each model's `relation_name`. #### Create the charts directory and board file The sample project doesn't ship any dbt Charts files, so you create them yourself. dbt Charts needs two things: a project config at the root that declares a data source, and at least one board file under `charts/`. 1. From the project root, create the directory that holds your boards: ```bash mkdir charts ``` Every `.yaml`, `.yml`, and `.md` file under `charts/` becomes a served page, and nested directories become nested paths. 2. Create `dbt_charts.yml` in the project root. Locally, dbt Charts reuses your dbt connection — there's no separate credential setup. Declare a source that points at a dbt profile and target, and boards reference it by name: dbt\_charts.yml ```yaml sources: analytics: type: dbt_profile profile: default target: dev ``` The `profile` and `target` match your `profiles.yml` (`default` and `dev` in this project). Credentials stay in `profiles.yml`, including Snowflake key-pair auth through `private_key_path` if your account requires it. 3. Create an empty board file at `charts/analytics.yml` and give it a title and a source: charts/analytics.yml ```yaml title: "Jaffle Shop overview" source: analytics ``` Over the next two steps, you add the queries and charts to this file, then the layout that positions them. #### Add your first five charts Now add five charts to `charts/analytics.yml`, one for each of the most common chart types. Each chart references a SQL query — defined in a `queries:` block — and maps that query's output columns to the chart's channels (`x`, `y`, `theta`, `color`). Each snippet below shows the query and the chart to add. Charts don't render until you add the layout in the next step, and the complete file appears in [Lay out your board](https://docs.getdbt.com/guides/dbt-charts.md?step=5). Each query references its model by fully qualified location. Replace `MY_DATABASE.MY_SCHEMA` with the database and schema that dbt built your marts into. The SQL uses Snowflake syntax, so adjust functions like `date_trunc` if you build on another platform. ##### 1. Line chart: revenue over time Line charts show trends over time. This query aggregates revenue by month from the `order_items` model. ```yaml queries: revenue_by_month: sql: | SELECT date_trunc('month', ordered_at) AS month, sum(product_price) AS revenue FROM MY_DATABASE.MY_SCHEMA.order_items GROUP BY 1 ORDER BY 1 charts: revenue_trend: type: line query: revenue_by_month x: month y: revenue title: "Revenue by month" ``` ##### 2. Bar chart: revenue by location Bar charts compare a value across categories. This query reads pre-aggregated revenue per store from the `location_performance` mart. ```yaml queries: revenue_by_location: sql: | SELECT location_name, total_revenue AS revenue FROM MY_DATABASE.MY_SCHEMA.location_performance ORDER BY total_revenue DESC charts: revenue_by_location: type: bar query: revenue_by_location x: location_name y: revenue title: "Revenue by location" ``` ##### 3. Area chart: cumulative revenue Area charts emphasize accumulation over time. This query uses a window function to running-total monthly revenue. ```yaml queries: cumulative_revenue: sql: | SELECT month, sum(revenue) OVER (ORDER BY month) AS cumulative_revenue FROM ( SELECT date_trunc('month', ordered_at) AS month, sum(product_price) AS revenue FROM MY_DATABASE.MY_SCHEMA.order_items GROUP BY 1 ) ORDER BY month charts: cumulative_revenue: type: area query: cumulative_revenue x: month y: cumulative_revenue title: "Cumulative revenue" ``` ##### 4. Pie chart: revenue by product type Pie (sector) charts show part-to-whole comparisons. This query splits revenue into food and drink. A sector chart uses `theta` for the wedge size and `color` for the category. ```yaml queries: revenue_by_type: sql: | SELECT CASE WHEN is_food_item THEN 'Food' ELSE 'Drink' END AS product_type, sum(product_price) AS revenue FROM MY_DATABASE.MY_SCHEMA.order_items GROUP BY 1 charts: revenue_by_type: type: pie query: revenue_by_type theta: revenue color: product_type title: "Revenue by product type" ``` ##### 5. Table: location KPIs Tables show detailed values. This query lists several columns per location for a scannable KPI table. ```yaml queries: location_kpis: sql: | SELECT location_name, total_revenue, total_orders, avg_order_value FROM MY_DATABASE.MY_SCHEMA.location_performance ORDER BY total_revenue DESC charts: location_kpis_table: type: table query: location_kpis title: "Location KPIs" ``` #### Lay out your board Charts don't appear until you reference them in a layout. Boards use `rows:` to stack content vertically and `cols:` to place content side by side. Combine the queries and charts from the previous step under single `queries:` and `charts:` blocks, then add a layout that references each chart by its ID. Your complete `charts/analytics.yml` should now look like this, with `MY_DATABASE.MY_SCHEMA` replaced by the database and schema that dbt built your marts into: charts/analytics.yml ```yaml title: "Jaffle Shop overview" source: analytics queries: revenue_by_month: sql: | SELECT date_trunc('month', ordered_at) AS month, sum(product_price) AS revenue FROM MY_DATABASE.MY_SCHEMA.order_items GROUP BY 1 ORDER BY 1 revenue_by_location: sql: | SELECT location_name, total_revenue AS revenue FROM MY_DATABASE.MY_SCHEMA.location_performance ORDER BY total_revenue DESC cumulative_revenue: sql: | SELECT month, sum(revenue) OVER (ORDER BY month) AS cumulative_revenue FROM ( SELECT date_trunc('month', ordered_at) AS month, sum(product_price) AS revenue FROM MY_DATABASE.MY_SCHEMA.order_items GROUP BY 1 ) ORDER BY month revenue_by_type: sql: | SELECT CASE WHEN is_food_item THEN 'Food' ELSE 'Drink' END AS product_type, sum(product_price) AS revenue FROM MY_DATABASE.MY_SCHEMA.order_items GROUP BY 1 location_kpis: sql: | SELECT location_name, total_revenue, total_orders, avg_order_value FROM MY_DATABASE.MY_SCHEMA.location_performance ORDER BY total_revenue DESC charts: revenue_trend: type: line query: revenue_by_month x: month y: revenue title: "Revenue by month" revenue_by_location: type: bar query: revenue_by_location x: location_name y: revenue title: "Revenue by location" cumulative_revenue: type: area query: cumulative_revenue x: month y: cumulative_revenue title: "Cumulative revenue" revenue_by_type: type: pie query: revenue_by_type theta: revenue color: product_type title: "Revenue by product type" location_kpis_table: type: table query: location_kpis title: "Location KPIs" rows: - title: "Trends" cols: - revenue_trend - cumulative_revenue - title: "Locations" cols: - revenue_by_location - revenue_by_type - title: "Detail" rows: - location_kpis_table ``` Your project root should now contain: ```shell fusion-jaffle-shop/ ├── charts/ │ └── analytics.yml # Your board ├── dbt_charts.yml # dbt Charts project config ├── dbt_project.yml ├── models/ ├── profiles.yml └── seeds/ ``` #### Install dbt Charts and view your board dbt Charts ships as a Python package with a command-line tool named `dct`. 1. Install dbt Charts with `uv`: ```bash uv tool install dbt-charts ``` Or, with `pip`: ```bash pip install dbt-charts ``` 2. From the project root, validate the board to catch YAML, reference, and SQL errors before it renders: ```bash dct validate ``` With no arguments, `dct validate` checks every board under `charts/`. Pass a path — `dct validate charts/analytics.yml` — to check one. 3. Start the local development server: ```bash dct serve ``` `dct` walks up from the current directory to find `dbt_charts.yml` or `dbt_project.yml`, infers the SQL dialect from your profile target, and picks a port derived from the project directory. 4. Open the URL `dct` prints. The page lists every board under `charts/`. Select `analytics.yml` to open yours. All five charts render across the three rows: the two trend charts side by side, the two location charts below them, and the KPI table at the bottom. The server hot-reloads as you edit YAML, so keep it running while you iterate. 5. Optionally, render static output to share or to inspect the compiled SQL: ```bash dct render charts/analytics.yml --format html ``` #### Troubleshooting * **`dbt_charts.yml` validation error: extra inputs are not permitted** — `dbt_charts.yml` accepts a fixed set of top-level keys, such as `sources` and `server`. Remove anything else, including a `name` key. * **A chart shows no data** — Confirm your `profiles.yml` connection works (`dbt debug`) and that the profile's role can `SELECT` from the schemas your queries read. * **Some charts show data but others are all zeros** — Check whether the empty charts read a model that your dev target filters. A common pattern limits development data to a recent time window, such as a `limit_in_dev` macro on orders. On a static demo dataset, that window can exclude every row and zero out the models downstream of it, while models without the filter still populate. Anchor the window to the data's own `max()` timestamp, widen it, or build against a target that skips the filter. * **Table or view not found** — Confirm the database and schema prefix on each `FROM` matches where dbt actually builds the model. Run `dbt list --output json` and read `relation_name` to get the exact location. * **Column not found** — A chart's channels (`x`, `y`, `theta`, `color`) must match the column names your query's `SELECT` returns. Alias columns in SQL so the names line up. * **A chart renders blank** — Confirm the chart's `type` supports the channels you set. Sector charts (pie and donut) use `theta` and `color`, not `x` and `y`. * **`dct validate` warns about re-aggregation** — The cumulative revenue query sums an already-aggregated column on purpose, so you can ignore this warning. `dct validate` still exits successfully. #### What's next Congratulations 🎉! You built a dbt project locally, created your own dbt Charts directory and board file, wrote five charts from your dbt models, and previewed the board in your browser. From here, you can: * Add interactivity with [variables](https://docs.dataface.com/variables/) so viewers can filter by location or date range. * Explore more [chart types](https://docs.dataface.com/charts/), such as small multiples, layered charts, and maps. * Add more files under `charts/` — each one becomes its own page, so you can break a large dashboard into focused boards. * Chart additional dbt models by adding more SQL queries, or explore other [query types](https://docs.dataface.com/queries/) such as values and HTTP sources. --- ### Quickstart for dbt from a manual install [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction When you use dbt from a manual install, you will be editing files locally using a code editor, and running projects using a command line interface (CLI). If you want to edit files and run projects using the web-based dbt Integrated Development Environment (Studio IDE), refer to the [dbt quickstarts](https://docs.getdbt.com/guides.md). You can also develop and run dbt commands using the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) — a dbt powered command line. ##### Prerequisites * To use dbt, it's important that you know some basics of the Terminal. In particular, you should understand `cd`, `ls` and `pwd` to navigate through the directory structure of your computer easily. * Install dbt using the [installation instructions](https://docs.getdbt.com/docs/local/install-dbt.md) for your operating system. * Complete appropriate Setting up and Loading data steps in the Quickstart for dbt series. For example, for BigQuery, complete [Setting up (in BigQuery)](https://docs.getdbt.com/guides/bigquery.md?step=2) and [Loading data (BigQuery)](https://docs.getdbt.com/guides/bigquery.md?step=3). * [Create a GitHub account](https://github.com/join) if you don't already have one. ##### Create a starter project After setting up BigQuery to work with dbt, you are ready to create a starter project with example models, before building your own models. #### Create a repository The following steps use [GitHub](https://github.com/) as the Git provider for this guide, but you can use any Git provider. You should have already [created a GitHub account](https://github.com/join). 1. [Create a new GitHub repository](https://github.com/new) named `dbt-tutorial`. 2. Select one of the following (You can always change this setting later): * **Private (recommended):** To secure your environment and prevent private information (like credentials) from being public. * **Public:** If you need to easily collaborate and share with others, especially outside of your organization. 3. Leave the default values for all other settings. 4. Click **Create repository**. 5. Save the commands from "…or create a new repository on the command line" to use later in [Commit your changes](https://docs.getdbt.com/guides/manual-install.md?step=6). #### Create a project Learn how to use a series of commands using the command line of the Terminal to create your project. dbt includes an `init` command that helps scaffold a dbt project. To create your dbt project: 1. Make sure you have dbt installed and check the version using the `dbt --version` command: ```shell dbt --version ``` 2. Initiate the `jaffle_shop` project using the `init` command: ```shell dbt init jaffle_shop ``` 3. Navigate into your project's directory: ```shell cd jaffle_shop ``` 4. Use `pwd` to confirm that you are in the right spot: ```shell $ pwd > Users/BBaggins/dbt-tutorial/jaffle_shop ``` 5. Use a code editor like VS Code to open the project directory you created in the previous steps, which we named jaffle\_shop. The content includes folders and `.sql` and `.yml` files generated by the `init` command. [![The starter project in a code editor](/img/starter-project-dbt-cli.png?v=2 "The starter project in a code editor")](#)The starter project in a code editor 6. dbt provides the following values in the `dbt_project.yml` file: dbt\_project.yml ```yaml name: jaffle_shop # Change from the default, `my_new_project` ... profile: jaffle_shop # Change from the default profile name, `default` ... models: jaffle_shop: # Change from `my_new_project` to match the previous value for `name:` ... ``` #### Connect to BigQuery When developing locally, dbt connects to your data warehouse using a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md), which is a YAML file with all the connection details to your warehouse. 1. Create a file in the `~/.dbt/` directory named `profiles.yml`. 2. Move your BigQuery keyfile into this directory. 3. Copy the following and paste into the new profiles.yml file. Make sure you update the values where noted. profiles.yml ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: bigquery method: service-account keyfile: /Users/BBAGGINS/.dbt/KEYFILE_NAME.json # Replace with the full path to your keyfile database: GCP_PROJECT_ID # Your Google Cloud project ID schema: dbt_YOUR_NAME # Your development dataset, for example dbt_bbaggins threads: 4 location: US # Must match the location of your BigQuery datasets ``` 4. Run the `debug` command from your project to confirm that you can successfully connect: ```shell $ dbt debug > Connection test: OK connection ok ``` [![A successful dbt debug command](/img/successful-dbt-debug.png?v=2 "A successful dbt debug command")](#)A successful dbt debug command ##### FAQs My data team uses a different data warehouse. What should my profiles.yml file look like for my warehouse? The structure of a profile looks different on each warehouse. Check out the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) page, and navigate to the `Profile Setup` section for your warehouse. Why are profiles stored outside of my project? Profiles are stored separately to dbt projects to avoid checking credentials into version control. Database credentials are extremely sensitive information and should **never be checked into version control**. What should I name my profile? We typically use a company name for a profile name, and then use targets to differentiate between `dev` and `prod`. Check out the docs on [environments in dbt v1](https://docs.getdbt.com/docs/local/dbt-environments.md) for more information. What should I name my target? We typically use targets to differentiate between development and production runs of dbt, naming the targets `dev` and `prod`, respectively. Check out the docs on [managing environments in dbt v1](https://docs.getdbt.com/docs/local/dbt-environments.md) for more information. Can I use environment variables in my profile? Yes! Check out the docs on [environment variables](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) for more information. #### Perform your first dbt run Our sample project has some example models in it. We're going to check that we can run them to confirm everything is in order. 1. Enter the `run` command to build example models: ```shell dbt run ``` You should have an output that looks like this: [![A successful dbt run command](/img/successful-dbt-run.png?v=2 "A successful dbt run command")](#)A successful dbt run command #### Commit your changes Commit your changes so that the repository contains the latest code. 1. Link the GitHub repository you created to your dbt project by running the following commands in Terminal. Make sure you use the correct git URL for your repository, which you should have saved from step 5 in [Create a repository](https://docs.getdbt.com/guides/manual-install.md?step=2). ```shell git init git branch -M main git add . git commit -m "Create a dbt project" git remote add origin https://github.com/USERNAME/dbt-tutorial.git git push -u origin main ``` 2. Return to your GitHub repository to verify your new files have been added. ##### Build your first models Now that you set up your sample project, you can get to the fun part — [building models](https://docs.getdbt.com/docs/build/sql-models.md)! In the next steps, you will take a sample query and turn it into a model in your dbt project. #### Checkout a new git branch Check out a new git branch to work on new code: 1. Create a new branch by using the `checkout` command and passing the `-b` flag: ```shell $ git checkout -b add-customers-model > Switched to a new branch `add-customer-model` ``` #### Build your first model If you'd like to use AI to build your first model, check out the [build it with dbt Wizard](#build-it-with-ai-using-dbt-wizard) in the next section. 1. Open your project in your favorite code editor. 2. Create a new SQL file in the `models` directory, named `models/customers.sql`. 3. Paste the following query into the `models/customers.sql` file. ##### BigQuery ```sql with customers as ( select id as customer_id, first_name, last_name from `dbt-tutorial`.jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from `dbt-tutorial`.jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` ##### Databricks ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop_customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop_orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` ##### Redshift ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` ##### Snowflake ```sql with customers as ( select id as customer_id, first_name, last_name from raw.jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from raw.jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. From the command line, enter `dbt run`. [![A successful dbt run from the CLI](/img/first-model-dbt-cli.png?v=2 "A successful dbt run from the CLI")](#)A successful dbt run from the CLI When you return to the BigQuery console, you can `select` from this model. ##### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. ##### Build it with AI using dbt Wizard Prefer not to write the SQL by hand? The [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md?section=qs-build-with-ai-cli) can build the same model for you from your terminal, grounded in your project's actual schema and lineage. Be warned, the wizard has been known to cast spells . 1. Install the dbt Wizard CLI: ##### macOS/Linux ```shell curl -fsSL https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.sh | sh ``` ##### Windows (PowerShell) ```powershell irm https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.ps1 | iex ``` 2. Run `wizard` in your project directory to start a session and then go through the onboarding (if you haven't already). [![A dbt Wizard session running in the terminal](/img/docs/wizard-cli-intro.png?v=2 "A dbt Wizard session running in the terminal")](#)A dbt Wizard session running in the terminal 3. Prompt it: "Create a customers model that joins orders and customers, and includes each customer's most recent order date and total number of orders." 4. Review the SQL dbt Wizard generates, then accept it to save the model. 5. Type `dbt run` to build it. Either way, you end up with the same working model. For full setup details, refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md?section=qs-build-with-ai-cli). #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. ##### BigQuery models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from `dbt-tutorial`.jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from `dbt-tutorial`.jaffle_shop.orders ``` ##### Databricks models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop_customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop_orders ``` ##### Redshift models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ``` ##### Snowflake models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from raw.jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from raw.jaffle_shop.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ##### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). ##### Next steps Before moving on from building your first models, make a change and see how it affects your results: * Write some bad SQL to cause an error — can you debug the error? * Run only a single model at a time. For more information, see [Syntax overview](https://docs.getdbt.com/reference/node-selection/syntax.md). * Group your models with a `stg_` prefix into a `staging` subdirectory. For example, `models/staging/stg_customers.sql`. * Configure your `staging` models to be views. * Run only the `staging` models. You can also explore: * The `target` directory to see all of the compiled SQL. The `run` directory shows the create or replace table statements that are running, which are the select statements wrapped in the correct DDL. * The `logs` file to see how dbt logs all of the action happening within your project. It shows the select statements that are running and other log output from when dbt runs. #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). ##### Next steps Before moving on from testing, make a change and see how it affects your results: * Write a test that fails, for example, omit one of the order statuses in the `accepted_values` list. What does a failing test look like? Can you debug the failure? * Run the tests for one model only. If you grouped your `stg_` models into a directory, try running the tests for all the models in that directory. * Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to add a Markdown description to a model. #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 1. Run `dbt docs generate` to compile the artifacts required for your projects static docs site. 2. Run `dbt docs serve` to launch the documentation in a local website. ##### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit updated changes You need to commit the changes you made to the project so that the repository has your latest code. 1. Add all your changes to git: `git add -A` 2. Commit your changes: `git commit -m "Add customers model, tests, docs"` 3. Push your changes to your repository: `git push -u origin add-customers-model` 4. Navigate to your repository, and open a pull request to merge the code into your master branch. #### Schedule a job We recommend using dbt platform as the easiest and most reliable way to [deploy jobs](https://docs.getdbt.com/docs/deploy/deployments.md) and automate your dbt project in production. For more info on how to get started, refer to [create and schedule jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs). [![Overview of a dbt job run, which includes the job run details, trigger type, commit SHA, environment name, detailed run steps, logs, and more.](/img/docs/dbt-platform/deployment/run-overview.png?v=2 "Overview of a dbt job run, which includes the job run details, trigger type, commit SHA, environment name, detailed run steps, logs, and more.")](#)Overview of a dbt job run, which includes the job run details, trigger type, commit SHA, environment name, detailed run steps, logs, and more. To schedule jobs yourself with an orchestrator like Airflow, refer to the [dbt and Airflow](https://docs.getdbt.com/blog/dbt-airflow-spiritual-alignment) blog post. Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt v1 using DuckDB > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * Never ask for passwords or keys in chat. > * If the reader wants to try the dbt platform, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=duckdb-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt v1 with DuckDB, enabling you to get set up quickly and efficiently. [DuckDB](https://duckdb.org/) is an open-source database management system which is designed for analytical workloads. It is designed to provide fast and easy access to large datasets, making it well-suited for data analytics tasks. This guide covers DuckDB with dbt v1 command-line interface (CLI). For dbt v2-specific DuckDB setup, refer to [DuckDB setup](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md?version=2). This guide will demonstrate how to: * Create a virtual development environment using a template provided by dbt Labs. * We will set up a fully functional dbt environment with an operational and executable project. The codespace automatically connects to the DuckDB database and loads a year's worth of data from our fictional Jaffle Shop café, which sells food and beverages in several US cities. * Run through the steps outlined in the `jaffle_shop_duckdb` repository, but if you want to dig into the underlying code further, refer to the [README](https://github.com/dbt-labs/jaffle_shop_duckdb/blob/duckdb/README.md) for the Jaffle Shop template. * Run any dbt command from the environment’s terminal. * Generate a larger dataset for the Jaffle Shop café (for example, five years of data instead of just one). You can learn more through high-quality [dbt Learn courses and workshops](https://learn.getdbt.com). ##### Related content * [DuckDB setup](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md) * [Create a GitHub repository](https://docs.getdbt.com/guides/manual-install.md?step=2) * [Build your first models](https://docs.getdbt.com/guides/manual-install.md?step=3) * [Test and document your project](https://docs.getdbt.com/guides/manual-install.md?step=4) * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) to build models with AI instead of by hand #### Prerequisites * When using DuckDB with dbt v1, you'll need to use the CLI. Currently, DuckDB is not supported in dbt platform. * It's important that you know some basics of the terminal. In particular, you should understand `cd`, `ls` , and `pwd` to navigate through the directory structure of your computer easily. * You have a [GitHub account](https://github.com/join). * You have Python 3.12 or later. Earlier versions can fail when you install its `requirements.txt` (for example, `networkx` 3.7 needs Python 3.12 or later). #### Set up DuckDB for dbt v1 This section will provide a step-by-step guide for setting up DuckDB for use in local (Mac and Windows) environments and web browsers. In the repository, there's a [`requirements.txt`](https://github.com/dbt-labs/jaffle_shop_duckdb/blob/duckdb/requirements.txt) file which is used to install dbt v1, DuckDB, and all other necessary dependencies. You can check this file to see what will be installed on your machine. It's typically located in the root directory of your project alongside other key files like `dbt_project.yml`. Otherwise, we will show you how in later steps. Below is an example of the `requirements.txt` file alongside other key files like `dbt_project.yml`: ```shell /my_dbt_project/ ├── dbt_project.yml ├── models/ │ ├── my_model.sql ├── tests/ │ ├── my_test.sql └── requirements.txt ``` For more information, refer to the [DuckDB setup](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md). ##### Local 1. First, [clone](https://git-scm.com/docs/git-clone) the Jaffle Shop git repository by running the following command in your terminal: ```bash git clone https://github.com/dbt-labs/jaffle_shop_duckdb.git ``` 2. Change into the `jaffle_shop_duckdb` directory from the command line: ```shell cd jaffle_shop_duckdb ``` 3. Install dbt v1 and DuckDB in a virtual environment.  Example for Mac ```shell python3 -m venv venv source venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install -r requirements.txt source venv/bin/activate ```  Example for Windows ```shell python -m venv venv venv\Scripts\activate.bat python -m pip install --upgrade pip python -m pip install -r requirements.txt venv\Scripts\activate.bat ```  Example for Windows PowerShell ```shell python -m venv venv venv\Scripts\Activate.ps1 python -m pip install --upgrade pip python -m pip install -r requirements.txt venv\Scripts\Activate.ps1 ``` 4. Ensure your profile is setup correctly from the command line by running the following [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md). * [dbt seed](https://docs.getdbt.com/reference/commands/seed.md) — loads CSV files located in the seed-paths directory of your project into your data warehouse * [dbt compile](https://docs.getdbt.com/reference/commands/compile.md) — generates executable SQL from your project source files * [dbt run](https://docs.getdbt.com/reference/commands/run.md) — compiles and runs your project * [dbt test](https://docs.getdbt.com/reference/commands/test.md) — compiles and tests your project * [dbt build](https://docs.getdbt.com/reference/commands/build.md) — compiles, runs, and tests your project * [dbt docs generate](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate) — generates your project's documentation. * [dbt docs serve](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-serve) — starts a webserver on port 8080 to serve your documentation locally and opens the documentation site in your default browser. For complete details, refer to the [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md). Here's what a successful output will look like: ```jinja (venv) ➜ jaffle_shop_duckdb git:(duckdb) dbt build 15:10:12 Running with dbt=x.y.z 15:10:13 Registered adapter: duckdb=x.y.z 15:10:13 Found 5 models, 3 seeds, 20 data tests, 416 macros 15:10:13 15:10:14 Concurrency: 24 threads (target='dev') 15:10:14 15:10:14 1 of 28 START seed file main.raw_customers ..................................... [RUN] 15:10:14 2 of 28 START seed file main.raw_orders ........................................ [RUN] 15:10:14 3 of 28 START seed file main.raw_payments ...................................... [RUN] .... 15:10:15 27 of 28 PASS relationships_orders_customer_id__customer_id__ref_customers_ .... [PASS in 0.32s] 15:10:15 15:10:15 Finished running 3 seeds, 3 view models, 20 data tests, 2 table models in 0 hours 0 minutes and 1.52 seconds (1.52s). 15:10:15 15:10:15 Completed successfully 15:10:15 15:10:15 Done. PASS=28 WARN=0 ERROR=0 SKIP=0 TOTAL=28 ``` To query data, some useful commands you can run from the command line: * `dbt show --select "raw_orders"` — run a query against the data warehouse and preview the results in the terminal. * [`dbt source`](https://docs.getdbt.com/reference/commands/source.md) — provides the [`dbt source freshness`](https://docs.getdbt.com/reference/commands/source.md#freshness) subcommand, which is useful when working with source data. * `dbt source freshness` — checks the freshness (how up to date) a specific source table is. note The steps will fail if you decide to run this project in your data warehouse (outside of this DuckDB demo). You will need to reconfigure the project files for your warehouse. Definitely consider this if you are using a community-contributed adapter. ##### Troubleshoot  Could not set lock on file error ```jinja IO Error: Could not set lock on file "jaffle_shop.duckdb": Resource temporarily unavailable ``` This is a known issue in DuckDB. Try disconnecting from any sessions that are locking the database. If you are using DBeaver, this means shutting down DBeaver (disconnecting doesn't always work). As a last resort, deleting the database file will get you back in action (*but* you will lose all your data). ##### Web browser 1. Go to the `jaffle-shop-template` [repository](https://github.com/dbt-labs/jaffle_shop_duckdb) after you log in to your GitHub account. 2. Click **Use this template** at the top of the page and choose **Create new repository**. 3. Click **Create repository from template** when you’re done setting the options for your new repository. 4. Click **Code** (at the top of the new repository’s page). Under the **Codespaces** tab, choose **Create codespace on main**. Depending on how you've configured your computer's settings, this either opens a new browser tab with the Codespace development environment with VSCode running in it or opens a new VSCode window with the codespace in it. 5. Wait for the codespace to finish building by waiting for the `postCreateCommand` command to complete; this can take several minutes: [![Wait for postCreateCommand to complete](/img/codespace-quickstart/postCreateCommand.png?v=2 "Wait for postCreateCommand to complete")](#)Wait for postCreateCommand to complete When this command completes, you can start using the codespace development environment. The terminal the command ran in will close and you will get a prompt in a brand new terminal. 6. At the terminal's prompt, you can execute any dbt command you want. For example: ```shell /workspaces/test (main) $ dbt build ``` You can also use the [duckcli](https://duckdb.org/docs/api/cli/overview.html) to write SQL against the warehouse from the command line or build reports in the [Evidence](https://evidence.dev/) project provided in the `reports` directory. For complete information, refer to the [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md). Common commands are: * [dbt compile](https://docs.getdbt.com/reference/commands/compile.md) — generates executable SQL from your project source files * [dbt run](https://docs.getdbt.com/reference/commands/run.md) — compiles and runs your project * [dbt test](https://docs.getdbt.com/reference/commands/test.md) — compiles and tests your project * [dbt build](https://docs.getdbt.com/reference/commands/build.md) — compiles, runs, and tests your project #### Local storage DuckDB stores your data in a local `.duckdb` file on your machine. The location of this file is defined by the `path` field in your [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md). In this quickstart, the project is configured to use a file named `jaffle_shop.duckdb`. After running `dbt build`, you can confirm that the database file was created by running the following command: ```shell ls -lah *.duckdb ``` You should see a `jaffle_shop.duckdb` file in your project directory. This file contains the tables and views built by dbt and persists between runs. If you delete the file: ```shell rm *.duckdb ``` Running `dbt build` again will recreate it. note If you use a relative path (such as `./local.duckdb`), the file is created relative to your `profiles.yml` file. #### Generate a larger data set If you'd like to work with a larger selection of Jaffle Shop data, you can generate an arbitrary number of years of fictitious data from within your codespace. 1. Install the Python package called [jafgen](https://pypi.org/project/jafgen/). At the terminal's prompt, run: ```shell python -m pip install jafgen ``` 2. When installation is done, run: ```shell jafgen NUMBER_OF_YEARS ``` Replace `NUMBER_OF_YEARS` with the number of years you want to simulate. For example, to generate data for 6 years, run `jafgen 6`. This command builds the CSV files and stores them in the `jaffle-data` folder, and is automatically sourced based on the `sources.yml` file and the [dbt-duckdb](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md) adapter. As you increase the number of years, it takes exponentially more time to generate the data because the Jaffle Shop stores grow in size and number. For a good balance of data size and time to build, dbt Labs suggests a maximum of 6 years. #### Next steps Now that you have dbt v1, DuckDB, and the Jaffle Shop data up and running, you can explore dbt's capabilities. Refer to these materials to get a better understanding of dbt projects and commands: * The [About projects](https://docs.getdbt.com/docs/build/projects.md) page guides you through the structure of a dbt project and its components. * [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md) explains the various commands available and what they do. * [dbt Labs courses](https://courses.getdbt.com/collections) offer a variety of beginner, intermediate, and advanced learning modules designed to help you become a dbt expert. * Once you see the potential of dbt and what it can do for your organization, sign up for a free trial of [dbt](https://www.getdbt.com/signup). It's the fastest and easiest way to deploy dbt today! * Check out the other [quickstart guides](https://docs.getdbt.com/guides.md?tags=Quickstart) to begin integrating into your existing data warehouse. Additionally, with your new understanding of the basics of using DuckDB, consider optimizing your setup by [documenting your project](https://docs.getdbt.com/guides/duckdb.md#document-your-project), [commit your changes](https://docs.getdbt.com/guides/duckdb.md#commit-your-changes) and, [schedule a job](https://docs.getdbt.com/guides/duckdb.md#schedule-a-job). ##### Document your project To document your dbt projects with DuckDB, follow the steps: * Use the `dbt docs generate` command to compile information about your dbt project and warehouse into `manifest.json` and `catalog.json` files * Run the [`dbt docs serve`](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-serve) command to create a local website using the generated `.json` files. This allows you to view your project's documentation in a web browser. * Enhance your documentation by adding [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) to models, columns, and sources using the `description` key in your YAML files. ##### Commit your changes Commit your changes to ensure the repository is up to date with the latest code. 1. In the GitHub repository you created for your project, run the following commands in the terminal: ```shell git add . git commit -m "Your commit message" git push ``` 2. Go back to your GitHub repository to verify your new files have been added. ##### Schedule a job 1. Ensure dbt v1 is installed and configured to connect to your DuckDB instance. 2. Create a dbt project and define your [`models`](https://docs.getdbt.com/docs/build/models.md), [`seeds`](https://docs.getdbt.com/reference/seed-properties.md), and [`tests`](https://docs.getdbt.com/reference/commands/test.md). 3. Use a scheduler such [Prefect](https://docs.getdbt.com/docs/deploy/deployment-tools.md#prefect) to schedule your dbt runs. You can create a DAG (Directed Acyclic Graph) that triggers dbt commands at specified intervals. 4. Write a script that runs your dbt commands, such as [`dbt run`](https://docs.getdbt.com/reference/commands/run.md), `dbt test` and more so. 5. Use your chosen scheduler to run the script at your desired frequency. Congratulations on making it through the guide 🎉! Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. --- ### Quickstart for dbt v2 > **Using this guide with an AI assistant** > > * In the first reply, confirm the reader’s prerequisites and connection path before listing steps. Then default to one guide section per reply (combine two or three short sections), with all its steps, labeled Step 1, Step 2, and so on, and confirm the success check before continuing. If the reader asks for a different pace, follow it. > * Never ask for passwords or keys in chat. > * If the reader wants to try the dbt platform, share <https://www.getdbt.com/signup?utm_source=ai_assistant&utm_medium=docs_md&utm_campaign=dbt-qs> [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction important dbt v2 is available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our dbt Community Slack channel [`#dbt-v2`](https://getdbt.slack.com/archives/C088YCAB6GH). dbt v2 is a powerful new approach to classic dbt ideas! Completely rebuilt from the ground up in Rust, dbt v2 lets you compile and run your dbt projects faster than ever — often in seconds. This quickstart guide will get you from zero to running your first dbt project with dbt v2 + VS Code. By the end, you’ll have: * A working dbt project (`jaffle_shop`) built with dbt v2 * The dbt VS Code extension installed and connected * The ability to preview, compile, and run dbt commands directly from your IDE ##### About dbt v2 dbt v2 and the features it provides are available in multiple environments: | Environment | How to use dbt v2 | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Studio IDE** | dbt v2 is automatically enabled; just [upgrade your environment(s)](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2). | | **Local CLI** | [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) locally following this guide. | | **VS Code / Cursor IDE** | [Install the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) to unlock dbt v2's interactive power in your editor. | To learn more about which tool is best for you, see the [v2 availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md) page. To learn about dbt v2 and how it works, read more [about dbt v2](https://docs.getdbt.com/docs/introduction.md). #### Prerequisites To take full advantage of this guide, you'll need to meet the following prerequisites: * You should have a basic understanding of [dbt projects](https://docs.getdbt.com/docs/build/projects.md), [git workflows](https://docs.getdbt.com/docs/platform/git/git-version-control.md), and [data warehouse requirements](https://docs.getdbt.com/docs/supported-data-platforms.md). * Make sure you're using a supported adapter and authentication method:  BigQuery * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks * Service Account / User Token * Native OAuth  Redshift * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  ClickHouse[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password  Apache Spark (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Thrift * Simple Authentication and Security Layer (SASL) PLAIN * No SASL (NOSASL) * Livy * Basic authentication (username and password) * When deployed on Amazon Web Services (AWS): AWS Signature Version 4 * Supports authentication using single sign-on, service accounts, or user tokens  DuckDB (CLI only) DuckDB does not require authentication — it runs locally on your machine. *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* * You need a macOS (Terminal), Linux, or Windows (Powershell) machine to run dbt v2. * You need to have [Visual Studio Code](https://code.visualstudio.com/) installed. The [Cursor](https://www.cursor.com/en) code editor will also work, but these instructions will focus on VS Code. * You need admin or install privileges on your machine. ##### What you’ll learn By following this guide, you will: * Set up a fully functional dbt environment with an operational project * Install and use dbt v2 + dbt VS Code extension * Run dbt commands from your IDE or terminal * Preview data, view lineage, and write SQL faster with autocomplete, and more! You can learn more through high-quality [dbt Learn courses and workshops](https://learn.getdbt.com/). #### Installation It's easy to think of dbt v2 and the dbt extension as two different products, but they're a powerful combo that works together to unlock the full potential of dbt. Think of dbt v2 as exactly that — an engine. The dbt extension and VS Code are the chassis, and together they form a powerful vehicle for transforming your data. info * You can install dbt v2 and use it standalone with the CLI. * You *cannot* use the dbt extension without dbt v2 installed. * Use `dbt` as your default command. If you already have another dbt command-line tool installed (such as the dbt platform CLI or dbt v1), you can use `dbtf` as an unambiguous alias for dbt v2. The following are the essential steps from the [dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) and [extension](https://docs.getdbt.com/docs/install-dbt-extension.md) installation guides: 1. Run the following commands to install the dbt v2 CLI: ```shell python -m pip install dbt ``` ##### Verify the dbt v2 installation 1. After installation, open a new command-line window to confirm that dbt v2 installed correctly by checking the version. ```bash dbt --version ``` 2. You should see output similar to the following: ```bash dbt 2.0.0 ``` ##### Install the dbt VS Code extension The dbt VS Code extension is available in the [Visual Studio extension marketplace](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt). Download it directly from your VS Code editor: 1. Navigate to the **Extensions** tab of VS Code (or Cursor). 2. Search for `dbt` and choose the one from the publisher `dbt Labs, LLC`. [![Search for the extension](/img/docs/extension/extension-marketplace.png?v=2 "Search for the extension")](#)Search for the extension 3. Click **Install**. 4. When the prompt appears, you can sign in to a dbt platform account or skip it. Check out the [sign-in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) or [installation](https://docs.getdbt.com/docs/install-dbt-extension.md) instructions to come back to it later. 5. Confirm you've installed the extension by looking for the **dbt Extension** label in the status bar. If you see it, the extension was installed successfully! [![Verify installation in the status bar.](/img/docs/extension/extension-lsp-download.png?v=2 "Verify installation in the status bar.")](#)Verify installation in the status bar. #### Initialize the Jaffle Shop project Now let's create your first dbt project powered by dbt v2! 1. Run `dbt init` in your terminal from the directory where you want to create the project. The `dbt init` command creates an example project and walks you through setting up a connection profile. * If you *do not* have a connection profile that you want to use, start with `dbt init` and use the prompts to configure a profile: * If you already have a connection profile that you want to use, use the `--skip-profile-setup` flag then edit the generated `dbt_project.yml` to replace `profile: jaffle_shop` with `profile: <YOUR-PROFILE-NAME>`. ```bash dbt init --skip-profile-setup ``` * If you created new credentials through the interactive prompts, `init` automatically runs `dbt debug` at the end. This ensures the newly created profile establishes a valid connection with the database. 2. Change directories into your newly created project: ```bash cd jaffle_shop ``` 3. Build your dbt project (which includes creating example data): ```bash dbt build ``` ###### Viewing metadata in Catalog Run `dbt build --write-catalog` locally to generate docs metadata (`dbt build` alone does not generate docs metadata). To view that metadata in Catalog, run a job in dbt platform so the metadata is uploaded. For details, refer to [Platform behavior](https://docs.getdbt.com/reference/commands/cmd-docs.md?version=2.0#platform-behavior). This will: * Load example data into your warehouse * Create, build, and test models * Verify your dbt environment is fully operational #### Explore with the dbt VS Code extension The dbt VS Code extension (available for VS Code and Cursor) compiles and builds your project with dbt v2, a powerful and blazing fast rebuild of dbt from the ground up. Want to see dbt v2 in action? Check out the following video to get a sense of how it works: [dbt v2 + VS Code extension walkthrough](https://app.storylane.io/share/a1rkqx0mbd7a) Now that your project works, open it in VS Code or Cursor and see dbt v2 in action: 1. Open the **View** menu and click **Command Palette**. Enter **Workspaces: Add Folder to Workspace**. 2. Select your `jaffle_shop` folder. If you don't add the root folder of the dbt project to the workspace, the [dbt language server](https://docs.getdbt.com/blog/dbt-fusion-engine-components#the-dbt-vs-code-extension-and-language-server) (LSP) will not run. The LSP enables features like autocomplete, hover info, and inline error highlights. 3. Click the **dbt logo** in the sidebar to open the **Get started** panel. The panel detects your environment and surfaces the next setup step, including installing the dbt v2 binary, confirming your project is open, checking dbt v2 compatibility, and registering. If you haven't already upgraded your project to dbt v2, you can choose **Start with an agent** (Copilot or Cursor) or **Start manually in CLI** directly from the panel. For full details, see [Getting started](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started). 4. Open a model file to see the definition for the `orders` model. This is the model we'll use in all of the examples below. ```bash models/marts/orders.sql ``` 5. Locate **Lineage** and **Query Results** in the lower panel, and the **dbt icon** in the upper right corner next to your editor groups. If you see all of these, the extension is installed correctly and running! [![The VS Code UI with the extension running.](/img/docs/extension/extension-running.png?v=2 "The VS Code UI with the extension running.")](#)The VS Code UI with the extension running. Now you're ready to see some of these awesome features in action! * [Preview data and code](#preview-data-and-code) * [Navigate your project with lineage tools](#navigate-your-project-with-lineage-tools) * [Use the power of SQL understanding](#use-the-power-of-sql-understanding) * [Speed up common dbt commands](#speed-up-common-dbt-commands) ###### Preview data and code Gain valuable insights into your data transformation during each step of your development process. You can quickly access model results and underlying data structures directly from your code. These previews help validate your code step-by-step. 1. Locate the **table icon** for **Preview File** in the upper right corner. Click it to preview results in the **Query Results** tab. [![Preview model query results.](/img/docs/extension/preview-query-results.png?v=2 "Preview model query results.")](#)Preview model query results. 2. Click **Preview CTE** above `orders as (` to preview results in the **Query Results** tab. [![Preview CTE query results.](/img/docs/extension/preview-cte-query-results-3.png?v=2 "Preview CTE query results.")](#)Preview CTE query results. 3. Locate the code icon for **Compile File** in between the dbt and the table icons. Clicking this icon opens a window with the compiled version of the model. [![Compile File icon.](/img/docs/extension/compile-file-icon.png?v=2 "Compile File icon.")](#)Compile File icon. [![Compile File results.](/img/docs/extension/compile-file.png?v=2 "Compile File results.")](#)Compile File results. ###### Navigate your project with lineage tools Almost as important as where your data is going is where it's been. The lineage tools in the extension let you visualize the lineage of the resources in your models as well as the column-level lineage. These capabilities deepen your understanding of model relationships and dependencies. 1. Open the **Lineage** tab to visualize the model-level lineage of this model. [![Visualizing model-level lineage.](/img/docs/extension/extension-pane.png?v=2 "Visualizing model-level lineage.")](#)Visualizing model-level lineage. 2. Open the **View** menu, click **Command Palette** and enter `dbt: Show Column Lineage` to visualize the column-level lineage in the **Lineage** tab. [![Show column-level lineage.](/img/docs/extension/show-cll.png?v=2 "Show column-level lineage.")](#)Show column-level lineage. ###### Use the power of SQL understanding Code smarter, not harder. The autocomplete and context clues help avoid mistakes and enable you to write fast and accurate SQL. Catch issues before you commit them! 1. To see **Autocomplete** in action, delete `ref('stg_orders')`, and begin typing `ref(stg_` to see the subset of matching model names. Use up and down arrows to select `stg_orders`. [![Autocomplete for a model name.](/img/docs/extension/autocomplete.png?v=2 "Autocomplete for a model name.")](#)Autocomplete for a model name. 2. Hover over any `*` to see the list of column names and data types being selected. [![Hovering over \* to see column names and data types.](/img/docs/extension/hover-star.png?v=2 "Hovering over * to see column names and data types.")](#)Hovering over \* to see column names and data types. ###### Speed up common dbt commands Testing, testing... is this mic on? It is and it's ready to execute your commands with blazing fast speeds! When you want to test your code against various dbt commands: 1. The dbt icon in the top right opens a list of extension-specific commands: [![Select a command via the dbt icon.](/img/docs/extension/run-command.png?v=2 "Select a command via the dbt icon.")](#)Select a command via the dbt icon. 2. Opening the **View** menu, clicking the **Command Palette**, and entering `>dbt:` in the command bar shows all the new commands that are available. [![dbt commands in the command bar.](/img/docs/extension/extension-commands-all.png?v=2 "dbt commands in the command bar.")](#)dbt commands in the command bar. Try choosing some of them and see what they do 😎 This is just the start. There is so much more available and so much more coming. Be sure to check out our resources for all the information about dbt v2 and the dbt VS Code extension! Connect your AI to this project Now that your project is up and running, connect to your AI tool (like Claude or ChatGPT) through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=qs-connect-ai), so you can ask questions about your models in plain language. Descriptions, tests, and semantic models make its answers even better. #### Troubleshooting If you run into any issues, check out the troubleshooting section below.  How to create a .dbt directory in root and move dbt\_cloud.yml file If you've never had a `.dbt` directory, you should perform the following recommended steps to create one. If you already have a `.dbt` directory, move the `dbt_cloud.yml` file into it. Some information about the `.dbt` directory: * A `.dbt` directory is a hidden folder in the root of your filesystem. It's used to store your dbt configuration files. The `.` prefix is used to create a hidden folder, which means it's not visible in Finder or File Explorer by default. * To view hidden files and folders, press Command + Shift + G on macOS or Ctrl + Shift + G on Windows. This opens the "Go to Folder" dialog where you can search for the `.dbt` directory. ##### Create a .dbt directory 1. Clone your dbt project repository locally. 2. Use the `mkdir` command followed by the name of the folder you want to create. * If using macOS, add the `~` prefix to create a `.dbt` folder in the root of your filesystem: * macOS: `mkdir ~/.dbt` * Windows: `mkdir %USERPROFILE%\.dbt` ##### Move the dbt\_cloud.yml file You can move the `dbt_cloud.yml` file into the `.dbt` directory using the `mv` command or by dragging and dropping the file into the `.dbt` directory by opening the Downloads folder using the "Go to Folder" dialog and then using drag-and-drop in the UI. To move the file using the terminal, use the `mv/move` command. This command moves the `dbt_cloud.yml` from the `Downloads` folder to the `.dbt` folder. If your `dbt_cloud.yml` file is located elsewhere, adjust the path accordingly. ###### Mac or Linux In your command line, use the `mv` command to move your `dbt_cloud.yml` file into the `.dbt` directory. If you've just downloaded the `dbt_cloud.yml` file and it's in your Downloads folder, the command might look something like this: ```bash mv ~/Downloads/dbt_cloud.yml ~/.dbt/dbt_cloud.yml ``` ###### Windows In your command line, use the move command. Assuming your file is in the Downloads folder, the command might look like this: ```bash move %USERPROFILE%\Downloads\dbt_cloud.yml %USERPROFILE%\.dbt\dbt_cloud.yml ```  I can't see the lineage tab in Cursor If you're using the dbt VS Code extension in Cursor, the lineage tab works best in Editor mode and doesn't render in Agent mode. If you're in Agent mode and the lineage tab isn't rendering, just switch to Editor mode to view your project's table and column lineage.  The extension gets stuck in a loading state If the extension is attempting to activate during startup and locks into a permanent loading state, check that: * Your dbt VS Code extension is on the latest version. * Your IDE is on the latest version. * You have a valid `dbt_cloud.yml` file configured and in the [correct location](#register-with-dbt_cloudyml). If you're still experiencing issues, try these steps before contacting dbt Support: * Delete and download a new copy of your `dbt_cloud.yml` file. * Delete and reinstall the dbt VS Code extension.  dbt platform configurations If you're a cloud-based dbt platform user who has the `dbt-cloud:` config in the `dbt_project.yml` file and are also using dbt Mesh, you must have the project ID configured: ```yaml dbt-cloud: project-id: 12345 # Required ``` If you don’t configure this correctly, cross-platform references will not resolve properly, and you will encounter errors executing dbt commands.  dbt extension not activating If the dbt extension has activated successfully, you will see the **dbt Extension** label in the status bar at the bottom left of your editor. You can view diagnostic information about the dbt extension by clicking the **dbt Extension** button. If the **dbt Extension** label is not present, then it is likely that the dbt extension was not installed successfully. If this happens, try uninstalling the extension, restarting your editor, and then reinstalling the extension. **Note:** It is possible to "hide" status bar items in VS Code. Double-check if the dbt Extension status bar label is hidden by right-clicking on the status bar in your editor. If you see dbt Extension in the right-click menu, then the extension has installed successfully.  Missing dbt LSP features If you receive a `no active LSP for this workspace` error message or aren't seeing dbt Language Server (LSP) features in your editor (like autocomplete, go-to-definition, or hover text), start by first following the general troubleshooting steps mentioned earlier. If you've confirmed the dbt extension is installed correctly but don't see LSP features, try the following: 1. Check extension version — Ensure that you're using the latest available version of the dbt extension by: * Opening the **Extensions** page in your editor, or * Going to the **Output** tab and looking for the version number, or * Running `dbtf --version` in the terminal. 2. Reinstall the LSP — If the version is correct, reinstall the LSP: 1. Open the Command Palette: Command + Shift + P (macOS) or Ctrl + Shift + P (Windows/Linux). 2. Paste `dbt: Reinstall dbt LSP` and enter. This command downloads the LSP and re-activates the extension to resolve the error.  Unsupported dbt version If you see an error message indicating that your version of dbt is unsupported, then there is likely a problem with your environment. Check the dbt Path setting in your VS Code settings. If this path is set, ensure that it is pointing to a valid dbt v2 executable. If necessary, you can also install dbt v2 directly using these instructions: [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2)  dbt v2 binary not found at the configured path If the extension reports that the dbt v2 binary can't be found at the configured path (for example, `dbt-fusion binary not found at [path]`), the `dbt.fusionPath` setting is pointing to a location that doesn't contain a valid binary. * Verify that [`dbt.fusionPath`](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings) points to a valid dbt v2 binary. * If you haven't installed dbt v2 manually, clear the setting and let the extension download and manage it for you. * To install manually, follow [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2).  dbt v2 version is not compatible with this extension If the extension reports that the installed dbt v2 version isn't compatible with your dbt VS Code extension version, the two are outside the supported range. 1. Run `dbt --version` to check your installed dbt v2 version. 2. Compare it against the [version compatibility matrix](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#compatibility-matrix) for your extension version. 3. Update dbt v2 or the extension so both fall within the supported range. Use the **Download compatible version** action in the notification if it appears.  dbt v2 crashes on startup If the extension reports that dbt v2 crashed on startup, confirm the binary runs on its own: 1. Run `dbt --version` in your terminal. If this fails, reinstall dbt v2 using [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2). 2. Use the **Show Logs** action in the notification (or open the **Output** tab) to review the startup error.  A known-bad dbt v2 version is installed If the extension warns that your installed dbt v2 version has a known regression, dbt Labs has flagged that release as [known-bad](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases). Update to the version named in the notification. For standalone installations: ```shell dbt system update ``` The warning persists across restarts until you update. If you work in an air-gapped environment, refer to [known-bad releases](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases) for how to distribute the manifest locally.  Addressing the 'dbt language server is not running in this workspace' error To resolve the `dbt language server is not running in this workspace` error, you need to add your dbt project folder to a workspace: 1. In VS Code, click **File** in the toolbar then select **Add Folder to Workspace**. 2. Select the dbt project file you want to add to a workspace. 3. To save your workspace, click **File** then select **Save Workspace As**. 4. Navigate to the location you want to save your workspace. This should resolve the error and open your dbt project by opening the workspace it belongs to. For more information on workspaces, refer to [What is a VS Code workspace?](https://code.visualstudio.com/docs/editing/workspaces/workspaces).  Manifest cannot be downloaded from the dbt platform If the dbt VS Code extension cannot download the manifest from the dbt platform or you get `warning: dbt1200: Failed to download manifest` using dbt v2 locally, you are probably having DNS-related issues. To confirm this, do a DNS lookup for the host dbt v2 is trying to download from (for example, prodeu2.blob.core.windows.net) by using `dig` on Linux/Mac or `nslookup` on Windows. If this doesn't return an IP address, the likely reason is that your company uses the same cloud provider with private endpoints for cloud resources, and DNS requests for these are forwarded to private DNS zones. This situation can be remedied by setting up an internet fallback, which will then return a public IP to any cloud storage that does not have a private IP registered with the private DNS zone. For Azure refer to [Fallback to internet for Azure Private DNS zones](https://learn.microsoft.com/en-us/azure/dns/private-dns-fallback). #### More information about dbt v2 * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ### Quickstart for the dbt Semantic Layer dbt platform | Starter, Enterprise, Enterprise+ [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), powered by [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md), simplifies the setup of key business metrics. It centralizes definitions, avoids duplicate code, and ensures easy access to metrics in downstream tools. MetricFlow helps manage company metrics easier, allowing you to define metrics in your dbt project and query them in dbt with [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md). 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). Connect your AI to your metrics AI tools like Claude and ChatGPT can connect to the Semantic Layer through the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?section=sl-connect-ai), so their answers use your governed metrics instead of guessing at raw tables. This quickstart is for dbt users on the dbt platform. You will build and define metrics, set up the Semantic Layer in a dbt project, and query those metrics in Google Sheets. The guide works on any supported data platform. Use the tabs in [Set up your warehouse](#set-up-your-warehouse) to create an account, load the sample data, and connect dbt. Later steps include a tab when the SQL or YAML is different for your platform. ##### Prerequisites * You need a [dbt](https://www.getdbt.com/signup/) Trial, Starter, or Enterprise-tier account for all deployments. * Have the correct [dbt license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) based on your plan:  More info on license and permissions * Enterprise-tier: Developer license with Account Admin permissions. Or "Owner" with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. * Starter: "Owner" access with a Developer license. * Trial: Automatic "Owner" access under a Starter plan trial. * A warehouse account and the Jaffle Shop sample data. Follow the tab for your platform in [Set up your warehouse](#set-up-your-warehouse). * Basic understanding of SQL and dbt. For example, you've used dbt before or have completed the [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) course. #### Set up your warehouse Use the tab for your data platform to create an account, load the sample data, and connect dbt. Complete only the steps listed in the tab, then return here. Do not keep going into project setup or models on that other guide. This guide covers that next. The Semantic Layer currently supports Snowflake, BigQuery, Databricks, Redshift, Postgres, and Trino (including Starburst Galaxy). It does not support Microsoft Fabric. For the full list, refer to [What data platforms are supported](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md#what-data-platforms-are-supported-by-the-dbt-semantic-layer). ##### Snowflake Create a [trial Snowflake account](https://signup.snowflake.com/): * Select the Enterprise Snowflake edition with ACCOUNTADMIN access. Consider organizational questions when choosing a cloud provider, and refer to Snowflake's [Introduction to Cloud Platforms](https://docs.snowflake.com/en/user-guide/intro-cloud-platforms). * Select a cloud provider and region. All cloud providers and regions will work so choose whichever you prefer. You can also complete the [Quickstart for dbt and Snowflake](https://docs.getdbt.com/guides/snowflake.md) first if you prefer a longer Snowflake-only walkthrough. ##### Create a Snowflake SQL file 1. Log in to your [trial Snowflake account](https://app.snowflake.com). 2. In the Snowflake UI, click the Create icon **+** in the upper left (under the Snowflake logo) to open a dropdown. 3. Select the first option, **SQL File**. ##### Set up and load data into Snowflake The data used here is stored as CSV files in a public S3 bucket and the following steps will guide you through how to prepare your Snowflake account for that data and upload it. 1. Create a new virtual warehouse, two new databases (one for raw data, the other for future dbt development), and two new schemas (one for `jaffle_shop` data, the other for `stripe` data). To do this, run these SQL commands by typing them into the Editor of your new Snowflake SQL file and clicking **Run**: ```sql create warehouse transforming; create database raw; create database analytics; create schema raw.jaffle_shop; create schema raw.stripe; ``` 2. In the `raw` database and `jaffle_shop` and `stripe` schemas, create three tables and load relevant data into them: * First, delete all contents (empty) in the Editor of the Snowflake SQL file. Then, run this SQL command to create the `customer` table: ```sql create table raw.jaffle_shop.customers ( id integer, first_name varchar, last_name varchar ); ``` * Delete all contents in the Editor, then run this command to load data into the `customer` table: ```sql copy into raw.jaffle_shop.customers (id, first_name, last_name) from 's3://dbt-tutorial-public/jaffle_shop_customers.csv' file_format = ( type = 'CSV' field_delimiter = ',' skip_header = 1 ); ``` * Delete all contents in the Editor (empty), then run this command to create the `orders` table: ```sql create table raw.jaffle_shop.orders ( id integer, user_id integer, order_date date, status varchar, _etl_loaded_at timestamp default current_timestamp ); ``` * Delete all contents in the Editor, then run this command to load data into the `orders` table: ```sql copy into raw.jaffle_shop.orders (id, user_id, order_date, status) from 's3://dbt-tutorial-public/jaffle_shop_orders.csv' file_format = ( type = 'CSV' field_delimiter = ',' skip_header = 1 ); ``` * Delete all contents in the Editor (empty), then run this command to create the `payment` table: ```sql create table raw.stripe.payment ( id integer, orderid integer, paymentmethod varchar, status varchar, amount integer, created date, _batched_at timestamp default current_timestamp ); ``` * Delete all contents in the Editor, then run this command to load data into the `payment` table: ```sql copy into raw.stripe.payment (id, orderid, paymentmethod, status, amount, created) from 's3://dbt-tutorial-public/stripe_payments.csv' file_format = ( type = 'CSV' field_delimiter = ',' skip_header = 1 ); ``` 3. Verify that the data is loaded by running these SQL queries. Confirm that you can see output for each one. ```sql select * from raw.jaffle_shop.customers; select * from raw.jaffle_shop.orders; select * from raw.stripe.payment; ``` [![The image displays Snowflake's confirmation output when data loaded correctly in the Editor.](/img/docs/dbt-platform/semantic-layer/sl-snowflake-confirm.jpg?v=2 "The image displays Snowflake's confirmation output when data loaded correctly in the Editor.")](#)The image displays Snowflake's confirmation output when data loaded correctly in the Editor. ##### Connect dbt to Snowflake There are two ways to connect dbt to Snowflake. The first option is Partner Connect, which provides a streamlined setup to create your dbt account from within your new Snowflake trial account. The second option is to create your dbt account separately and build the Snowflake connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt setup flow, dbt Labs recommends connecting manually. ##### Use Partner Connect Using Partner Connect allows you to create a complete dbt account with your [Snowflake connection](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md), [a managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md), [environments](https://docs.getdbt.com/docs/build/custom-schemas.md#managing-environments), and credentials. 1. On the left sidebar of the Snowflake UI, go to **Admin > Partner Connect**. Find the dbt tile under the **Data Integration** section or search for dbt in the search bar. Click the tile to connect to dbt. [![Snowflake Partner Connect Box](/img/snowflake_tutorial/snowflake_partner_connect_box.png?v=2 "Snowflake Partner Connect Box")](#)Snowflake Partner Connect Box If you’re using the classic version of the Snowflake UI, you can click the **Partner Connect** button in the top bar of your account. From there, click on the dbt tile to open up the connect box. [![Snowflake Classic UI - Partner Connect](/img/snowflake_tutorial/snowflake_classic_ui_partner_connect.png?v=2 "Snowflake Classic UI - Partner Connect")](#)Snowflake Classic UI - Partner Connect 2. In the **Connect to dbt** popup, find the **Optional Grant** option and select the **RAW** and **ANALYTICS** databases. This will grant access for your new dbt user role to each selected database. Then, click **Connect**. [![Snowflake Classic UI - Connection Box](/img/snowflake_tutorial/snowflake_classic_ui_connection_box.png?v=2 "Snowflake Classic UI - Connection Box")](#)Snowflake Classic UI - Connection Box [![Snowflake New UI - Connection Box](/img/snowflake_tutorial/snowflake_new_ui_connection_box.png?v=2 "Snowflake New UI - Connection Box")](#)Snowflake New UI - Connection Box 3. Click **Activate** when a popup appears: [![Snowflake Classic UI - Actviation Window](/img/snowflake_tutorial/snowflake_classic_ui_activation_window.png?v=2 "Snowflake Classic UI - Actviation Window")](#)Snowflake Classic UI - Actviation Window [![Snowflake New UI - Activation Window](/img/snowflake_tutorial/snowflake_new_ui_activation_window.png?v=2 "Snowflake New UI - Activation Window")](#)Snowflake New UI - Activation Window 4. After the new tab loads, you will see a form. If you already created a dbt account, you will be asked to provide an account name. If you haven't created an account, you will be asked to provide an account name and password. 5. After you have filled out the form and clicked **Complete Registration**, you will be logged into dbt automatically. 6. Click your account name in the left side menu and select **Account settings**, choose the "Partner Connect Trial" project, and select **snowflake** in the overview table. Select **Edit** and update the **Database** field to `analytics` and the **Warehouse** field to `transforming`. [![dbt - Snowflake Project Overview](/img/snowflake_tutorial/dbt_cloud_snowflake_project_overview.png?v=2 "dbt - Snowflake Project Overview")](#)dbt - Snowflake Project Overview [![dbt - Update Database and Warehouse](/img/snowflake_tutorial/dbt_cloud_update_database_and_warehouse.png?v=2 "dbt - Update Database and Warehouse")](#)dbt - Update Database and Warehouse ##### Connect manually 1. Create a new project in dbt. Navigate to **Account settings** (by clicking on your account name in the left side menu), and click **+ New Project**. 2. Enter a project name and click **Continue**. 3. In the **Configure your development environment** section, click the **Connection** dropdown menu and select **Add new connection**. This directs you to the connection configuration settings. 4. In the **Type** section, select **Snowflake**. [![dbt - Choose Snowflake Connection](/img/snowflake_tutorial/dbt_cloud_setup_snowflake_connection_start.png?v=2 "dbt - Choose Snowflake Connection")](#)dbt - Choose Snowflake Connection 5. Enter your **Settings** for Snowflake with: * **Account:** Find your account by using the Snowflake trial account URL and removing `snowflakecomputing.com`. The order of your account information will vary by Snowflake version. For example, Snowflake's Classic console URL might look like: `oq65696.west-us-2.azure.snowflakecomputing.com`. The AppUI or Snowsight URL might look more like: `snowflakecomputing.com/west-us-2.azure/oq65696`. In both examples, your account will be: `oq65696.west-us-2.azure`. For more information, refer to [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) in the Snowflake docs.   ✅ `db5261993` or `db5261993.east-us-2.azure`<br />  ❌ `db5261993.eu-central-1.snowflakecomputing.com` * **Role:** Leave blank for now. You can update this to a default Snowflake role later. * **Database:** `analytics`. This tells dbt to create new models in the analytics database. * **Warehouse:** `transforming`. This tells dbt to use the transforming warehouse that was created earlier. [![dbt - Snowflake Account Settings](/img/snowflake_tutorial/dbt_cloud_snowflake_account_settings.png?v=2 "dbt - Snowflake Account Settings")](#)dbt - Snowflake Account Settings 6. Click **Save**. 7. Set up your personal user credentials by going to **Your profile** > **Credentials**. 8. Select your project that uses the Snowflake connection. 9. Click the **configure your development environment and add a connection** link. This directs you to a page where you can enter your personal user credentials. 10. Enter your **User credentials** for Snowflake with: * **Username:** The username you created for Snowflake. The username is not your email address and is usually your first and last name together in one word. * **Password:** The password you set when creating your Snowflake account. * **Schema:** You’ll notice that the schema name has been auto-created for you. By convention, this is `dbt_<first-initial><last-name>`. This is the schema connected directly to your development environment, and it's where your models will be built when running dbt within the Studio IDE. * **Target name:** Leave as the default. * **Threads:** Leave as 4. This is the number of simultaneous connects that dbt will make to build models concurrently. [![dbt - Snowflake User credentials](/img/snowflake_tutorial/dbt_cloud_snowflake_development_credentials.png?v=2 "dbt - Snowflake User credentials")](#)dbt - Snowflake User credentials 11. Click **Test connection**. This verifies that dbt can access your Snowflake account. 12. If the test succeeded, click **Save** to complete the configuration. If it failed, you may need to check your Snowflake settings and credentials. ##### BigQuery Open a new tab and follow these steps for account setup, sample data, and your dbt connection: * [Step 2: Create a new GCP project](https://docs.getdbt.com/guides/bigquery.md?step=2) * [Step 3: Create BigQuery datasets](https://docs.getdbt.com/guides/bigquery.md?step=3) * [Step 4: Generate BigQuery credentials](https://docs.getdbt.com/guides/bigquery.md?step=4) * [Step 5: Connect dbt to BigQuery](https://docs.getdbt.com/guides/bigquery.md?step=5) In step 3, create both the `jaffle_shop` and `stripe` datasets in your project. The sample tables you query live in the public `dbt-tutorial` project. ##### Databricks Open a new tab and follow these steps for account setup, sample data, and your dbt connection: * [Step 2: Create a Databricks workspace](https://docs.getdbt.com/guides/databricks.md?step=2) * [Step 3: Load data](https://docs.getdbt.com/guides/databricks.md?step=3) * [Step 4: Connect dbt to Databricks](https://docs.getdbt.com/guides/databricks.md?step=4) * [Step 5: Set up the integration from Partner Connect](https://docs.getdbt.com/guides/databricks.md?step=5) Step 4 explains Partner Connect vs connecting manually. Step 5 is Partner Connect. If you connect manually, use the Databricks docs linked from that step. ##### Redshift Open a new tab and follow these steps for account setup, sample data, and your dbt connection: * [Step 2: Create a Redshift cluster](https://docs.getdbt.com/guides/redshift.md?step=2) * [Step 3: Load data](https://docs.getdbt.com/guides/redshift.md?step=3) * [Step 4: Connect dbt to Redshift](https://docs.getdbt.com/guides/redshift.md?step=4) ##### Starburst Galaxy Open a new tab and follow these steps for account setup, sample data, and your dbt connection: * [Step 2: Load data to an Amazon S3 bucket](https://docs.getdbt.com/guides/starburst-galaxy.md?step=2) * [Step 3: Connect Starburst Galaxy to the Amazon S3 bucket](https://docs.getdbt.com/guides/starburst-galaxy.md?step=3) * [Step 4: Create tables with Starburst Galaxy](https://docs.getdbt.com/guides/starburst-galaxy.md?step=4) * [Step 5: Connect dbt to Starburst Galaxy](https://docs.getdbt.com/guides/starburst-galaxy.md?step=5) #### Set up dbt project In this section, you will set up a dbt managed repository and initialize your dbt project to start developing. ##### Set up a dbt managed repository If you used Snowflake or Databricks Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing). Partner Connect provides a [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md). Otherwise, you will need to create your repository connection. When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. ##### Initialize your dbt project This guide assumes you use the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) to develop your dbt project, define metrics, and query and preview metrics using [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md). Now that you have a repository configured, you can initialize your project and start development in dbt using the Studio IDE: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize your project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit`. This creates the first commit to your managed repo and allows you to open a branch where you can add a new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Delete the models/examples folder in the **File Catalog**. * Click **+ Create new file**, add the query for your warehouse to the new file, and click **Save as** to save the new file: ##### Snowflake ```sql select * from raw.jaffle_shop.customers ``` ##### BigQuery ```sql select * from `dbt-tutorial.jaffle_shop.customers` ``` ##### Databricks ```sql select * from default.jaffle_shop_customers ``` ##### Redshift ```sql select * from jaffle_shop.customers ``` ##### Starburst Galaxy ```sql select * from dbt_quickstart.jaffle_shop.jaffle_shop_customers ``` * In the command line bar at the bottom, enter dbt run and click Enter. You should see a dbt run succeeded message. #### Build your dbt project The next step is to build your project. This involves adding sources, staging models, business-defined entities, and packages to your project. ##### Add sources [Sources](https://docs.getdbt.com/docs/build/sources.md) in dbt are the raw data tables you'll transform. By organizing your source definitions, you document the origin of your data. It also makes your project and transformation more reliable, structured, and understandable. You have two options for working with files in the Studio IDE: * **Create a new branch (recommended):** Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * **Edit in the protected primary branch:** If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch, use this option. The Studio IDE prevents commits to the protected branch so you'll be prompted to commit your changes to a new branch. Name the new branch `build-project`. 1. Hover over the `models` directory and click the three-dot menu (**...**), then select **Create file**. 2. Name the file `staging/jaffle_shop/src_jaffle_shop.yml` , then click **Create**. 3. Copy the following text into the file and click **Save**. ##### Snowflake models/staging/jaffle\_shop/src\_jaffle\_shop.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop tables: - name: customers - name: orders ``` ##### BigQuery models/staging/jaffle\_shop/src\_jaffle\_shop.yml ```yaml sources: - name: jaffle_shop database: dbt-tutorial schema: jaffle_shop tables: - name: customers - name: orders ``` ##### Redshift models/staging/jaffle\_shop/src\_jaffle\_shop.yml ```yaml sources: - name: jaffle_shop schema: jaffle_shop tables: - name: customers - name: orders ``` ##### Databricks models/staging/jaffle\_shop/src\_jaffle\_shop.yml ```yaml sources: - name: jaffle_shop schema: default tables: - name: customers identifier: jaffle_shop_customers - name: orders identifier: jaffle_shop_orders ``` ##### Starburst Galaxy models/staging/jaffle\_shop/src\_jaffle\_shop.yml ```yaml sources: - name: jaffle_shop database: dbt_quickstart schema: jaffle_shop tables: - name: customers identifier: jaffle_shop_customers - name: orders identifier: jaffle_shop_orders ``` tip In your source file, you can also use the **Generate model** button to create a new model file for each source. This creates a new file in the `models` directory with the given source name and fill in the SQL code of the source definition. 4. Hover over the `models` directory and click the three dot menu (**...**), then select **Create file**. 5. Name the file `staging/stripe/src_stripe.yml` , then click **Create**. 6. Copy the following text into the file and click **Save**. ##### Snowflake models/staging/stripe/src\_stripe.yml ```yaml sources: - name: stripe database: raw schema: stripe tables: - name: payment ``` ##### BigQuery models/staging/stripe/src\_stripe.yml ```yaml sources: - name: stripe database: dbt-tutorial schema: stripe tables: - name: payment ``` ##### Redshift models/staging/stripe/src\_stripe.yml ```yaml sources: - name: stripe schema: stripe tables: - name: payment ``` ##### Databricks models/staging/stripe/src\_stripe.yml ```yaml sources: - name: stripe schema: default tables: - name: payment identifier: stripe_payments ``` ##### Starburst Galaxy models/staging/stripe/src\_stripe.yml ```yaml sources: - name: stripe database: dbt_quickstart schema: jaffle_shop tables: - name: payment identifier: stripe_payments ``` ##### Add staging models [Staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) are the first transformation step in dbt. They clean and prepare your raw data, making it ready for more complex transformations and analyses. The staging SQL uses `source()`, so it uses the names from the YAML you just added. For example, on BigQuery this points to `dbt-tutorial.jaffle_shop.customers`. On Snowflake it points to `raw.jaffle_shop.customers`. If your platform uses a different table name, that name is already in the YAML. Follow these steps to add your staging models to your project. 1. In the `jaffle_shop` sub-directory, create the file `stg_customers.sql`. Or, you can use the **Generate model** button to create a new model file for each source. 2. Copy the following query into the file and click **Save**. models/staging/jaffle\_shop/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from {{ source('jaffle_shop', 'customers') }} ``` 3. In the same `jaffle_shop` sub-directory, create the file `stg_orders.sql` 4. Copy the following query into the file and click **Save**. models/staging/jaffle\_shop/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from {{ source('jaffle_shop', 'orders') }} ``` 5. In the `stripe` sub-directory, create the file `stg_payments.sql`. 6. Copy the following query into the file and click **Save**. ##### Snowflake models/staging/stripe/stg\_payments.sql ```sql select id as payment_id, orderid as order_id, paymentmethod as payment_method, status, -- amount is stored in cents, convert it to dollars amount / 100 as amount, created as created_at from {{ source('stripe', 'payment') }} ``` ##### BigQuery models/staging/stripe/stg\_payments.sql ```sql select id as payment_id, orderid as order_id, paymentmethod as payment_method, status, -- amount is stored in cents, convert it to dollars amount / 100 as amount, created as created_at from {{ source('stripe', 'payment') }} ``` ##### Redshift models/staging/stripe/stg\_payments.sql ```sql select id as payment_id, orderid as order_id, paymentmethod as payment_method, status, -- amount is stored in cents, convert it to dollars amount / 100 as amount, created as created_at from {{ source('stripe', 'payment') }} ``` ##### Databricks models/staging/stripe/stg\_payments.sql ```sql select id as payment_id, orderid as order_id, paymentmethod as payment_method, status, -- amount is stored in cents, convert it to dollars amount / 100 as amount, created as created_at from {{ source('stripe', 'payment') }} ``` ##### Starburst Galaxy models/staging/stripe/stg\_payments.sql ```sql select id as payment_id, order_id, paymentmethod as payment_method, status, -- amount is stored in cents, convert it to dollars amount / 100 as amount, created as created_at from {{ source('stripe', 'payment') }} ``` 7. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. ##### Add business-defined entities This phase involves creating [models that serve as the entity layer or concept layer of your dbt project](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md), making the data ready for reporting and analysis. It also includes adding [packages](https://docs.getdbt.com/docs/build/packages.md) and the [MetricFlow time spine](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) that extend dbt's functionality. This phase is the [marts layer](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md#guide-structure-overview), which brings together modular pieces into a wide, rich vision of the entities an organization cares about. 1. Create the file `models/marts/fct_orders.sql`. 2. Copy the following query into the file and click **Save**. models/marts/fct\_orders.sql ```sql with orders as ( select * from {{ ref('stg_orders' )}} ), payments as ( select * from {{ ref('stg_payments') }} ), order_payments as ( select order_id, sum(case when status = 'success' then amount end) as amount from payments group by 1 ), final as ( select orders.order_id, orders.customer_id, orders.order_date, coalesce(order_payments.amount, 0) as amount from orders left join order_payments using (order_id) ) select * from final ``` 3. In the `models/marts` directory, create the file `dim_customers.sql`. 4. Copy the following query into the file and click **Save**. models/marts/dim\_customers.sql ```sql with customers as ( select * from {{ ref('stg_customers')}} ), orders as ( select * from {{ ref('fct_orders')}} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders, sum(amount) as lifetime_value from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders, customer_orders.lifetime_value from customers left join customer_orders using (customer_id) ) select * from final ``` 5. Create a MetricFlow time spine model by following the [MetricFlow time spine guide](https://docs.getdbt.com/guides/mf-time-spine.md?step=1). This guide walks you through creating both the SQL model and YAML configuration required for time-based metric calculations. 6. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run message and also see in the run details that dbt has successfully built your models. #### Create semantic models In this section, you'll learn about [semantic model](https://docs.getdbt.com/guides/sl-qs.md?step=5#about-semantic-models), [their components](https://docs.getdbt.com/guides/sl-qs.md?step=5#semantic-model-components), and [how to configure a time spine](https://docs.getdbt.com/guides/sl-qs.md?step=5#configure-a-time-spine). ##### About semantic models (Applies to dbt v1.12 and later) [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) contain many object types (such as entities, simple metrics, and dimensions) that allow MetricFlow to construct the queries for metric definitions. * Each semantic model will be 1:1 with a dbt SQL model. * Each semantic model will contain (at most) 1 primary or natural entity. * Each semantic model will contain zero, one, or many foreign or unique entities used to connect to other entities. * Each semantic model may also contain dimensions, simple metrics, and metrics. This is what actually gets fed into and queried by your downstream BI tool. In the following steps, semantic models enable you to define how to interpret the data related to orders. It includes entities (like ID columns serving as keys for joining data), dimensions (for grouping or filtering data), and simple metrics (for data aggregations). 1. In the `models` directory, create a new file `fct_orders.yml`. tip Make sure to save all semantic models and metrics under the directory defined in the [`model-paths`](https://docs.getdbt.com/reference/project-configs/model-paths.md) (or a subdirectory of it, like `models/semantic_models/`). If you save them outside of this path, it will result in an empty `semantic_manifest.json` file, and your semantic models or metrics won't be recognized. 2. Add the following code to that newly created file: models/fct\_orders.yml ```yaml models: - name: fct_orders description: | Order fact table. This table's grain is one row per order. semantic_model: enabled: true name: orders agg_time_dimension: order_date ``` ##### Semantic model components (Applies to dbt v1.12 and later) The following sections explain [dimensions](https://docs.getdbt.com/docs/build/dimensions.md), [entities](https://docs.getdbt.com/docs/build/entities.md), and [simple metrics](https://docs.getdbt.com/docs/build/metrics-overview.md#simple-metrics) in more detail, showing how they each play a role in semantic models. * [Entities](#entities) act as unique identifiers (like ID columns) that link data together from different tables. * [Dimensions](#dimensions) categorize and filter data, making it easier to organize. * [Simple metrics](#simple-metrics) calculates data using an aggregation function, providing valuable insights through aggregation. ##### Entities [Entities](https://docs.getdbt.com/docs/build/semantic-models.md#entities) are a real-world concept in a business, serving as the backbone of your semantic model. These are going to be ID columns (like `order_id`) in our semantic models. These will serve as join keys to other semantic models. Add entities to your `fct_orders.yml` semantic model file: (Applies to dbt v1.12 and later) models/fct\_orders.yml ```yaml models: - name: fct_orders semantic_model: enabled: true name: orders agg_time_dimension: order_date columns: # Entities - name: order_id entity: type: primary name: order_id - name: customer_id entity: name: customer type: foreign ``` ##### Dimensions [Dimensions](https://docs.getdbt.com/docs/build/semantic-models.md#dimensions) are a way to group or filter information based on categories or time. Add dimensions to your `fct_orders.yml` semantic model file: (Applies to dbt v1.12 and later) models/fct\_orders.yml ```yaml models: - name: fct_orders semantic_model: enabled: true name: orders agg_time_dimension: order_date columns: # Entities - name: order_id entity: type: primary name: order_id - name: customer_id entity: name: customer type: foreign # Newly added - Dimensions - name: order_date granularity: day dimension: type: time ``` (Applies to dbt v1.12 and later) ##### Simple metrics [Simple metrics](https://docs.getdbt.com/docs/build/simple.md) perform an aggregation (like `sum`, `count`, or `average`, and so on) on a single field in your model. They replace the concept of "measures" in previous versions. To define more advanced metrics, refer to [Define metrics and add a second semantic model](https://docs.getdbt.com/guides/sl-qs.md?step=6). Add simple metrics to your `fct_orders.yml` model file: models/fct\_orders.yml ```yaml models: - name: fct_orders semantic_model: enabled: true name: orders agg_time_dimension: order_date columns: # Entities - name: order_id entity: type: primary name: order_id - name: customer_id entity: name: customer type: foreign # Dimensions - name: order_date granularity: day dimension: type: time - name: amount dimension: type: categorical # Newly added - Simple metrics metrics: - name: order_total description: The total amount for each order including taxes type: simple label: Order total agg: sum expr: amount - name: order_count type: simple label: Order count agg: sum expr: 1 - name: customers_with_orders description: Distinct count of customers placing orders type: simple label: Customers with orders agg: count_distinct expr: customer_id - name: order_value_p99 type: simple label: Order value P99 agg: percentile expr: amount percentile: 99.0 percentile_type: discrete ``` ##### Configure a time spine To ensure accurate time-based aggregations, you must configure a [time spine](https://docs.getdbt.com/docs/build/metricflow-time-spine.md). The time spine allows you to have accurate metric calculations over different time granularities. Follow the [MetricFlow time spine guide](https://docs.getdbt.com/guides/mf-time-spine.md?step=1) for complete step-by-step instructions on creating and configuring your time spine model. This guide provides the current best practices and avoids deprecated configurations. #### Define metrics and add a second semantic model In this section, you will [define metrics](#define-metrics) and [add a second semantic model](#add-second-semantic-model-to-your-project) to your project. ##### Define metrics [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) are the language your business users speak and measure business performance. They are an aggregation over a column in your warehouse that you enrich with dimensional cuts. There are different types of metrics you can configure: * [Conversion metrics](https://docs.getdbt.com/docs/build/conversion.md) — Track when a base event and a subsequent conversion event occur for an entity within a set time period. * [Cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md) — Aggregate a measure over a given window. If no window is specified, the window will accumulate the measure over all of the recorded time period. Note that you must create the time spine model before you add cumulative metrics. * [Derived metrics](https://docs.getdbt.com/docs/build/metrics-overview.md#derived-metrics) — Allows you to do calculations on top of metrics. * [Simple metrics](https://docs.getdbt.com/docs/build/metrics-overview.md#simple-metrics) — Directly reference a single column expression within a semantic model, without any additional columns involved. They are aggregations over a column in your data platform and can be filtered by one or multiple dimensions. * [Ratio metrics](https://docs.getdbt.com/docs/build/metrics-overview.md#ratio-metrics) — Involve a numerator metric and a denominator metric. A constraint string can be applied to both the numerator and denominator or separately to the numerator or denominator. Once you've created your semantic models, it's time to start referencing those (Applies to dbt v1.12 and later) simple metrics you made to create some metrics: 1. Add metrics to your `fct_orders.yml` file: tip Make sure to save all semantic models and metrics under the directory defined in the [`model-paths`](https://docs.getdbt.com/reference/project-configs/model-paths.md) (or a subdirectory of it, like `models/semantic_models/`). If you save them outside of this path, it will result in an empty `semantic_manifest.json` file, and your semantic models or metrics won't be recognized. (Applies to dbt v1.12 and later) models/fct\_orders.yml ```yaml models: - name: fct_orders semantic_model: enabled: true name: orders agg_time_dimension: order_date columns: - name: order_id entity: type: primary name: order_id - name: customer_id entity: name: customer type: foreign - name: order_date granularity: day dimension: type: time - name: amount dimension: type: categorical metrics: # Simple type metrics - name: order_total description: Sum of orders value type: simple label: Order total agg: sum expr: amount - name: order_count description: Number of orders type: simple label: Order count agg: sum expr: 1 - name: customers_with_orders description: Distinct count of customers placing orders type: simple label: Customers with orders agg: count_distinct expr: customer_id - name: large_orders description: Count of orders with order total over 20 type: simple label: Large orders agg: sum expr: 1 filter: | {{ Dimension('order_id__amount') }} >= 20 # Ratio type metric - name: avg_order_value label: Average order value description: Average value of each order type: ratio numerator: order_total denominator: order_count # Cumulative type metrics - name: cumulative_order_amount_mtd label: Cumulative order amount MTD description: The month to date value of all orders type: cumulative grain_to_date: month input_metric: order_total # Derived metric - name: pct_of_orders_that_are_large label: Percent of orders that are large description: Percent of orders that are large type: derived expr: large_orders / order_count input_metrics: - name: large_orders - name: order_count ``` ##### Add second semantic model to your project (Applies to dbt v1.12 and later) Great job, you've successfully built your first semantic model! It has all the required elements: entities, dimensions, and metrics. Let’s expand your project's analytical capabilities by adding another semantic model in your other marts model, such as: `dim_customers.yml`. After setting up your orders model: 1. Create the file `dim_customers.yml`. 2. Copy the following code into the file and click **Save**. (Applies to dbt v1.12 and later) models/dim\_customers.yml ```yaml models: - name: dim_customers semantic_model: enabled: true name: customers agg_time_dimension: most_recent_order_date columns: # Entities - name: customer_id entity: name: customer type: primary # Dimensions - name: first_name dimension: name: customer_name type: categorical - name: first_order_date granularity: day dimension: type: time - name: most_recent_order_date granularity: day dimension: type: time - name: number_of_orders dimension: type: categorical - name: lifetime_value dimension: type: categorical # Metrics metrics: - name: customers_with_orders label: Customers with orders description: Unique count of customers placing orders type: simple agg: count_distinct expr: customer_id - name: count_lifetime_orders description: Total count of orders per customer type: simple label: Lifetime orders agg: sum expr: number_of_orders - name: lifetime_spend description: Gross customer lifetime spend inclusive of taxes type: simple label: Lifetime spend agg: sum expr: lifetime_value ``` This semantic model uses simple metrics to focus on customer metrics and emphasizes customer dimensions like name, type, and order dates. It uniquely analyzes customer behavior, lifetime value, and order patterns. #### Test and query metrics To work with metrics in dbt, you have several tools to validate or run commands. Here's how you can test and query metrics depending on your setup: * [**Studio IDE users**](#studio-ide-users) — Run [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md#metricflow-commands) directly in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) to query/preview metrics. View metrics visually in the **Lineage** tab. * [**dbt platform CLI users**](#dbt-cli-users) — The [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) enables you to run [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md#metricflow-commands) to query and preview metrics directly in your command line interface. * **dbt v1 users** — Use the MetricFlow CLI for command execution. While this guide focuses on dbt users, dbt v1 users can find detailed MetricFlow CLI setup instructions in the [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md#metricflow-commands) page. Note that to use the Semantic Layer, you need to have a [Starter or Enterprise-tier account](https://www.getdbt.com/). Alternatively, you can run commands with SQL client tools like DataGrip, DBeaver, or RazorSQL. ##### Studio IDE users You can use the `dbt sl` prefix before the command name to execute them in dbt. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands available in the Studio IDE, refer to the [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md#metricflow-commandss) page. The Studio IDE **Status button** (located in the bottom right of the editor) displays an **Error** status if there's an error in your metric or semantic model definition. You can click the button to see the specific issue and resolve it. Once viewed, make sure you commit and merge your changes in your project. ##### dbt platform CLI users This section is for dbt platform CLI users. MetricFlow commands are integrated with dbt, which means you can run MetricFlow commands as soon as you install the dbt platform CLI. Your account will automatically manage version control for you. Refer to the following steps to get started: 1. Install the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) (if you haven't already). Then, navigate to your dbt project directory. 2. Run a dbt command, such as `dbt parse`, `dbt run`, `dbt compile`, or `dbt build`. If you don't, you'll receive an error message that begins with: "ensure that you've ran an artifacts....". 3. MetricFlow builds a semantic graph and generates a `semantic_manifest.json` file in dbt, which is stored in the `/target` directory. (Applies to dbt v1.12 and later) Starting in dbt v1.12, dbt also writes `osi_document.json` to your `target/` directory at parse time. For more information, refer to [Semantic manifest](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). If using the Jaffle Shop example, run `dbt seed && dbt run` to ensure the required data is in your data platform before proceeding. Run dbt parse to reflect metric changes When you make changes to metrics, make sure to run `dbt parse` at a minimum to update the Semantic Layer. This updates the `semantic_manifest.json` file(Applies to dbt v1.12 and later) and `osi_document.json`, reflecting your changes when querying metrics. By running `dbt parse`, you won't need to rebuild all the models. 4. Run `dbt sl --help` to confirm you have MetricFlow installed and that you can view the available commands. 5. Run `dbt sl query --metrics <metric_name> --group-by <dimension_name>` to query the metrics and dimensions. For example, to query the `order_total` and `order_count` (both metrics), and then group them by the `order_date` (dimension), you would run: ```sql dbt sl query --metrics order_total,order_count --group-by order_id__order_date ``` 6. Verify that the metric values are what you expect. To further understand how the metric is being generated, you can view the generated SQL if you type `--compile` in the command line. 7. Commit and merge the code changes that contain the metric definitions. #### Run a production job This section explains how you can perform a job run in your deployment environment in dbt to materialize and deploy your metrics. Currently, the deployment environment is only supported. 1. Once you’ve [defined your semantic models and metrics](https://docs.getdbt.com/guides/sl-qs.md?step=6), commit and merge your metric changes in your dbt project. 2. In dbt, create a new [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) or use an existing environment on dbt 1.6 or higher. * Note: Deployment environment is currently supported (*development experience coming soon*) 3. To create a new environment, navigate to **Orchestration** in the navigation menu, select **Environments**, and then select **Create new environment**. 4. Fill in your deployment credentials with the username and password for your data platform. You can name the schema anything you want. Click **Save** to create your new production environment. 5. [Create a new deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) that runs in the environment you just created. Go back to the **Orchestration** menu, select **Jobs**, select **Create job**, and click **Deploy job**. 6. Set the job to run a `dbt parse` job to parse your projects and generate a [`semantic_manifest.json` artifact](https://docs.getdbt.com/reference/artifacts/sl-manifest.md) file(Applies to dbt v1.12 and later) and an [`osi_document.json` artifact](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). Although running `dbt build` isn't required, you can choose to do so if needed. note If you are on dbt v2, add the `dbt docs generate` command to your job to successfully deploy your metrics. 7. Run the job by clicking the **Run now** button. Monitor the job's progress in real-time through the **Run summary** tab. Once the job completes successfully, your dbt project, including the generated documentation, will be fully deployed and available for use in your production environment. If any issues arise, review the logs to diagnose and address any errors. What’s happening internally? * Merging the code into your main branch allows dbt to pull those changes and build the definition in the manifest produced by the run. <br /> * Re-running the job in the deployment environment helps materialize the models, which the metrics depend on, in the data platform. It also makes sure that the manifest is up to date. <br /> * The Semantic Layer APIs pull in the most recent manifest and enables your integration to extract metadata from it. #### Administer the Semantic Layer In this section, you will learn how to add credentials and create service tokens to start querying the dbt Semantic Layer. This section goes over the following topics: * [Select environment](#1-select-environment) * [Configure credentials and create tokens](#2-configure-credentials-and-create-tokens) * [View connection detail](#3-view-connection-detail) * [Add more credentials](#4-add-more-credentials) * [Delete configuration](#delete-configuration) You must be part of the Owner group and have the correct [license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to administer the Semantic Layer at the environment and project level. * Enterprise+ and Enterprise plan: * Developer license with Account Admin permissions, or * Owner with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. * Starter plan: Owner with a Developer license. * Free trial: You are on a free trial of the Starter plan as an Owner, which means you have access to the dbt Semantic Layer. ##### 1. Select environment Select the environment where you want to enable the Semantic Layer: 1. Navigate to **Account settings** in the navigation menu. 2. Under **Settings**, click **Projects** and select the specific project you want to enable the Semantic Layer for. 3. In the **Project details** page, navigate to the **Semantic Layer** section. Select **Configure Semantic Layer**. [![Semantic Layer section in the 'Project details' page](/img/docs/dbt-platform/semantic-layer/new-sl-configure.png?v=2 "Semantic Layer section in the 'Project details' page")](#)Semantic Layer section in the 'Project details' page 4. In the **Set Up Semantic Layer Configuration** page, select the deployment environment you want for the Semantic Layer and click **Save**. This provides administrators with the flexibility to choose the environment where the Semantic Layer will be enabled. [![Select the deployment environment to run your Semantic Layer against.](/img/docs/dbt-platform/semantic-layer/sl-select-env.png?v=2 "Select the deployment environment to run your Semantic Layer against.")](#)Select the deployment environment to run your Semantic Layer against. ##### 2. Configure credentials and create tokens There are two options for setting up Semantic Layer using API tokens: * [Add a credential and create service tokens](#add-a-credential-and-create-service-tokens) * [Configure user credentials and create personal tokens](#configure-development-credentials-and-create-a-personal-token) ###### Add a credential and create service tokens The first option is to use [service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) for authentication which are tied to an underlying data platform credential that you configure. The credential configured is used to execute queries that the Semantic Layer issues against your data platform. This credential controls the physical access to underlying data accessed by the Semantic Layer, and all access policies set in the data platform for this credential will be respected. | Feature | Starter plan | Enterprise+ and Enterprise plan | | --------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Service tokens | Can create multiple service tokens linked to one credential. | Can use multiple credentials and link multiple service tokens to each credential. Note that you cannot link a single service token to more than one credential. | | Credentials per project | One credential per project. | Can [add multiple](#4-add-more-credentials) credentials per project. | | Link multiple service tokens to a single credential | ✅ | ✅ | *If you're on a Starter plan and need to add more credentials, consider upgrading to our [Enterprise+ or Enterprise plan](https://www.getdbt.com/contact). All Enterprise users can refer to [Add more credentials](#4-add-more-credentials) for detailed steps on adding multiple credentials.* ###### 1. Select deployment environment * After selecting the deployment environment, you should see the **Credentials & service tokens** page. * Click the **Add Semantic Layer credential** button. ###### 2. Configure credential * In the **1. Add credentials** section, enter the credentials specific to your data platform that you want the Semantic Layer to use. * Use credentials with minimal privileges. The Semantic Layer requires read access to the schema(s) containing the dbt models used in your semantic models for downstream applications * Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. [![Add credentials and map them to a service token. ](/img/docs/dbt-platform/semantic-layer/sl-add-credential.png?v=2 "Add credentials and map them to a service token. ")](#)Add credentials and map them to a service token. ###### 3. Create or link service tokens * If you have permission to create service tokens, you’ll see the [**Map new service token** option](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#map-service-tokens-to-credentials) after adding the credential. Name the token, set permissions to 'Semantic Layer Only' and 'Metadata Only', and click **Save**. * Once the token is generated, you won't be able to view this token again, so make sure to record it somewhere safe. * If you don’t have access to create service tokens, you’ll see a message prompting you to contact your admin to create one for you. Admins can create and link tokens as needed. [![If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you.](/img/docs/dbt-platform/semantic-layer/sl-credential-no-service-token.png?v=2 "If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you.")](#)If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you. info * Starter plans can create multiple service tokens that link to a single underlying credential, but each project can only have one credential. * All Enterprise plans can [add multiple credentials](#4-add-more-credentials) and map those to service tokens for tailored access. [Book a free live demo](https://www.getdbt.com/contact) to discover the full potential of dbt Enterprise and higher plans. ###### Configure development credentials and create a personal token Using [personal access tokens (PATs)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) is also a supported authentication method for the dbt Semantic Layer. This enables user-level authentication, reducing the need for sharing tokens between users. When you authenticate using PATs, queries are run using your personal user credentials. To use PATs in Semantic Layer: 1. Configure your user credentials. 1. Click your account name at the bottom left-hand menu and go to **Account settings** > **Credentials**. 2. Select your project. 3. Click **Edit**. 4. Go to **User credentials** and enter your details. 5. Click **Save**. 2. [Create a personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Make sure to copy the token. You can use the generated PAT as the authentication method for Semantic Layer [APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) and [integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). ##### 3. View connection detail 1. Go back to the **Project details** page for connection details to connect to downstream tools. 2. Copy and share the Environment ID, service or personal token, Host, as well as the service or personal token name to the relevant teams for BI connection setup. If your tool uses the GraphQL API, save the GraphQL API host information instead of the JDBC URL. For info on how to connect to other integrations, refer to [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). [![After configuring, you'll be provided with the connection details to connect to you downstream tools.](/img/docs/dbt-platform/semantic-layer/sl-configure-example.png?v=2 "After configuring, you'll be provided with the connection details to connect to you downstream tools.")](#)After configuring, you'll be provided with the connection details to connect to you downstream tools. ##### 4. Add more credentials [Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") All dbt Enterprise plans can optionally add multiple credentials and map them to service tokens, offering more granular control and tailored access for different teams, which can then be shared to relevant teams for BI connection setup. These credentials control the physical access to underlying data accessed by the Semantic Layer. We recommend configuring credentials and service tokens to reflect your teams and their roles. For example, create tokens or credentials that align with your team's needs, such as providing access to finance-related schemas to the Finance team.  Considerations for linking credentials * Admins can link multiple service tokens to a single credential within a project, but each service token can only be linked to one credential per project. * When you send a request through the APIs, the service token of the linked credential will follow access policies of the underlying view and tables used to build your semantic layer requests. * Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. ###### 1. Add more credentials * After configuring your environment, on the **Credentials & service tokens** page, click the **Add Semantic Layer credential** button to create multiple credentials and map them to a service token. <br /> * In the **1. Add credentials** section, fill in the data platform's credential fields. We recommend using “read-only” credentials. [![Add credentials and map them to a service token. ](/img/docs/dbt-platform/semantic-layer/sl-add-credential.png?v=2 "Add credentials and map them to a service token. ")](#)Add credentials and map them to a service token. ###### 2. Map service tokens to credentials * In the **2. Map new service token** section, [map a service token to the credential](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#map-service-tokens-to-credentials) you configured in the previous step. dbt automatically selects the service token permission set you need (Semantic Layer Only and Metadata Only). * To add another service token during configuration, click **Add Service Token**. * You can link more service tokens to the same credential later on in the **Semantic Layer Configuration Details** page. To add another service token to an existing Semantic Layer configuration, click **Add service token** under the **Linked service tokens** section. * Click **Save** to link the service token to the credential. Remember to copy and save the service token securely, as it won't be viewable again after generation. [![Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control.](/img/docs/dbt-platform/semantic-layer/sl-credentials-service-token.png?v=2 "Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control.")](#)Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control. ###### 3. Delete credentials * To delete a credential, go back to the **Credentials & service tokens** page. * Under **Linked Service Tokens**, click **Edit** and, select **Delete Credential** to remove a credential. When you delete a credential, any service tokens mapped to that credential in the project will no longer work and will break for any end users. ##### Delete configuration You can delete the entire Semantic Layer configuration for a project. Note that deleting the Semantic Layer configuration will remove all credentials and unlink all service tokens to the project. It will also cause all queries to the Semantic Layer to fail. Follow these steps to delete the Semantic Layer configuration for a project: 1. Navigate to the **Project details** page. 2. In the **Semantic Layer** section, select **Delete Semantic Layer**. 3. Confirm the deletion by clicking **Yes, delete semantic layer** in the confirmation pop up. To re-enable the dbt Semantic Layer setup in the future, you will need to recreate your setup configurations by following the [previous steps](#set-up-dbt-semantic-layer). If your semantic models and metrics are still in your project, no changes are needed. If you've removed them, you'll need to set up the YAML configs again. [![Delete the Semantic Layer configuration for a project.](/img/docs/dbt-platform/semantic-layer/sl-delete-config.png?v=2 "Delete the Semantic Layer configuration for a project.")](#)Delete the Semantic Layer configuration for a project. #### Additional configuration The following are the additional flexible configurations for Semantic Layer credentials. ##### Map service tokens to credentials * After configuring your environment, you can map additional service tokens to the same credential if you have the required [permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#permission-sets). * Go to the **Credentials & service tokens** page and click the **+Add Service Token** button in the **Linked Service Tokens** section. * Type the service token name and select the permission set you need (Semantic Layer Only and Metadata Only). * Click **Save** to link the service token to the credential. * Remember to copy and save the service token securely, as it won't be viewable again after generation. [![Map additional service tokens to a credential.](/img/docs/dbt-platform/semantic-layer/sl-add-service-token.gif?v=2 "Map additional service tokens to a credential.")](#)Map additional service tokens to a credential. ##### Unlink service tokens * Unlink a service token from the credential by clicking **Unlink** under the **Linked service tokens** section. If you try to query the Semantic Layer with an unlinked credential, you'll experience an error in your BI tool because no valid token is mapped. ##### Manage from service token page **View credential from service token** * View your Semantic Layer credential directly by navigating to the **API tokens** and then **Service tokens** page. * Select the service token to view the credential it's linked to. This is useful if you want to know which service tokens are mapped to credentials in your project. ###### Create a new service token * From the **Service tokens** page, create a new service token and map it to the credential(s) (assuming the semantic layer permission exists). This is useful if you want to create a new service token and directly map it to a credential in your project. * Make sure to select the correct permission set for the service token (Semantic Layer Only and Metadata Only). [![Create a new service token and map credentials directly on the separate 'Service tokens page'.](/img/docs/dbt-platform/semantic-layer/sl-create-service-token-page.png?v=2 "Create a new service token and map credentials directly on the separate 'Service tokens page'.")](#)Create a new service token and map credentials directly on the separate 'Service tokens page'. #### Query the Semantic Layer This page will guide you on how to connect and use the following integrations to query your metrics: * [Connect and query with Google Sheets](#connect-and-query-with-google-sheets) * [Connect and query with Hex](#connect-and-query-with-hex) * [Connect and query with Sigma](#connect-and-query-with-sigma) The Semantic Layer enables you to connect and query your metric with various available tools like [PowerBI](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md), [Google Sheets](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md), [Hex](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration), [Microsoft Excel](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md), [Tableau](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md), and more. Query metrics using other tools such as [first-class integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md), and [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to expose tables of metrics and dimensions in your data platform and create a custom integrations. ##### Connect and query with Google Sheets The Google Sheets integration allows you to query your metrics using Google Sheets. This section will guide you on how to connect and use the Google Sheets integration. To query your metrics using Google Sheets: 1. Make sure you have a [Gmail](http://gmail.com/) account. 2. To set up Google Sheets and query your metrics, follow the detailed instructions on [Google Sheets integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md). 3. Start exploring and querying metrics! * Query a metric, like `order_total`, and filter it with a dimension, like `order_date`. * You can also use the `group_by` parameter to group your metrics by a specific dimension. [![Use the dbt Semantic Layer's Google Sheet integration to query metrics with a Query Builder menu.](/img/docs/dbt-platform/semantic-layer/sl-gsheets.jpg?v=2 "Use the dbt Semantic Layer's Google Sheet integration to query metrics with a Query Builder menu.")](#)Use the dbt Semantic Layer's Google Sheet integration to query metrics with a Query Builder menu. ##### Connect and query with Hex This section will guide you on how to use the Hex integration to query your metrics using Hex. Select the appropriate tab based on your connection method: ##### Query Semantic Layer with Hex 1. Navigate to the [Hex login page](https://app.hex.tech/login). 2. Sign in or make an account (if you don’t already have one). * You can make Hex free trial accounts with your work email or a .edu email. 3. In the top left corner of your page, click on the **HEX** icon to go to the home page. 4. Then, click the **+ New project** button on the top right. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/hex_new.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 5. Go to the menu on the left side and select **Data browser**. Then select **Add a data connection**. 6. Select the data platform that matches your dbt project (Snowflake, BigQuery, Redshift, or Databricks). Give the connection a name and description. You still turn on the dbt Semantic Layer for that connection. You query metrics through dbt, not with a database username and password. Hex currently supports the Semantic Layer on those four connection types. Refer to [Hex's dbt integration](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration). [![Select 'Data browser' and then 'Add a data connection'. Choose the warehouse type that matches your dbt project.](/img/docs/dbt-platform/semantic-layer/hex_new_data_connection.png?v=2 "Select 'Data browser' and then 'Add a data connection'. Choose the warehouse type that matches your dbt project.")](#)Select 'Data browser' and then 'Add a data connection'. Choose the warehouse type that matches your dbt project. 7. Under **Integrations**, toggle the dbt switch to the right to enable the dbt integration. [![Click on the dbt toggle to enable the integration. ](/img/docs/dbt-platform/semantic-layer/hex_dbt_toggle.png?v=2 "Click on the dbt toggle to enable the integration. ")](#)Click on the dbt toggle to enable the integration. 8. Enter the following information: * Select your version of dbt as 1.6 or higher * Enter your Environment ID * Enter your service or personal token * Make sure to click on the **Use Semantic Layer** toggle. This way, all queries are routed through dbt. * Click **Create connection** in the bottom right corner. 9. Hover over **More** on the menu shown in the following image and select **Semantic Layer**. [![Hover over 'More' on the menu and select 'dbt Semantic Layer'.](/img/docs/dbt-platform/semantic-layer/hex_make_sl_cell.png?v=2 "Hover over 'More' on the menu and select 'dbt Semantic Layer'.")](#)Hover over 'More' on the menu and select 'dbt Semantic Layer'. 10. Now, you should be able to query metrics using Hex! Try it yourself: * Create a new cell and pick a metric. * Filter it by one or more dimensions. * Create a visualization. ##### Getting started with the Semantic Layer workshop 1. Click on the link provided to you in the workshop’s chat. * Look at the **Pinned message** section of the chat if you don’t see it right away. 2. Enter your email address in the textbox provided. Then, select **SQL and Python** to be taken to Hex’s home screen. [![The 'Welcome to Hex' homepage.](/img/docs/dbt-platform/semantic-layer/welcome_to_hex.png?v=2 "The 'Welcome to Hex' homepage.")](#)The 'Welcome to Hex' homepage. 3. Then click the purple Hex button in the top left corner. 4. Click the **Collections** button on the menu on the left. 5. Select the **Semantic Layer Workshop** collection. 6. Click the **Getting started with the Semantic Layer** project collection. [![Click 'Collections' to select the 'Semantic Layer Workshop' collection.](/img/docs/dbt-platform/semantic-layer/hex_collections.png?v=2 "Click 'Collections' to select the 'Semantic Layer Workshop' collection.")](#)Click 'Collections' to select the 'Semantic Layer Workshop' collection. 7. To edit this Hex notebook, click the **Duplicate** button from the project dropdown menu (as displayed in the following image). This creates a new copy of the Hex notebook that you own. [![Click the 'Duplicate' button from the project dropdown menu to create a Hex notebook copy.](/img/docs/dbt-platform/semantic-layer/hex_duplicate.png?v=2 "Click the 'Duplicate' button from the project dropdown menu to create a Hex notebook copy.")](#)Click the 'Duplicate' button from the project dropdown menu to create a Hex notebook copy. 8. To make it easier to find, rename your copy of the Hex project to include your name. [![Rename your Hex project to include your name.](/img/docs/dbt-platform/semantic-layer/hex_rename.png?v=2 "Rename your Hex project to include your name.")](#)Rename your Hex project to include your name. 9. Now, you should be able to query metrics using Hex! Try it yourself with the following example queries: * In the first cell, you can see a table of the `order_total` metric over time. Add the `order_count` metric to this table. * The second cell shows a line graph of the `order_total` metric over time. Play around with the graph! Try changing the time grain using the **Time unit** drop-down menu. * The next table in the notebook, labeled “Example\_query\_2”, shows the number of customers who have made their first order on a given day. Create a new chart cell. Make a line graph of `first_ordered_at` vs `customers` to see how the number of new customers each day changes over time. * Create a new semantic layer cell and pick one or more metrics. Filter your metric(s) by one or more dimensions. [![Query metrics using Hex ](/img/docs/dbt-platform/semantic-layer/hex_make_sl_cell.png?v=2 "Query metrics using Hex ")](#)Query metrics using Hex ##### Connect and query with Sigma This section will guide you on how to use the Sigma integration to query your metrics using Sigma. If you already have a Sigma account, log in and skip to step 6. Steps 1 through 5 create a Sigma account from Snowflake Partner Connect. Use steps 1 through 5 only if you are on Snowflake. Databricks has a separate [Sigma Partner Connect](https://docs.databricks.com/aws/en/partners/bi/sigma) flow. If you are on another data platform, log in to an existing Sigma account and continue from step 6, or refer to Sigma's [Query a dbt Semantic Layer integration](https://help.sigmacomputing.com/docs/query-a-dbt-semantic-layer-integration) guide. 1. Go back to your Snowflake account. In the Snowflake UI, click on the home icon in the upper left corner. In the left sidebar, select **Data Products**. Then, select **Partner Connect**. Find the Sigma tile by scrolling or by searching for Sigma in the search bar. Click the tile to connect to Sigma. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-partner-connect.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 2. Select the Sigma tile from the list. Click the **Optional Grant** dropdown menu. Write **RAW** and **ANALYTICS** in the text box and then click **Connect**. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-optional-grant.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 3. Make up a company name and URL to use. It doesn’t matter what URL you use, as long as it’s unique. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-company-name.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 4. Enter your name and email address. Choose a password for your account. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-create-profile.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 5. Great! You now have a Sigma account. Before we get started, go back to Snowflake and open a blank file. Run these lines. * `grant all privileges on all views in schema analytics.SCHEMA to role pc_sigma_role;` * `grant all privileges on all tables in schema analytics.SCHEMA to role pc_sigma_role;` 6. Click on your bubble in the top right corner. Click the **Administration** button from the dropdown menu. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-admin.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 7. Scroll down to the integrations section, then select **Add** next to the dbt integration. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-add-integration.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 8. In the **dbt Integration** section, fill out the required fields, and then hit save: * Your dbt [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). * Your access URL of your existing Sigma dbt integration. Use `cloud.getdbt.com` as your access URL. * Your dbt Environment ID. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-add-info.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 9. Return to the Sigma home page. Create a new workbook. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-make-workbook.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 10. Click on **Table**, then click on **SQL**. Select the data connection that matches your dbt project (Snowflake, BigQuery, Redshift, or Databricks). If you created this Sigma account with Snowflake Partner Connect, that connection is named `PC_SIGMA_WH`. [![Click the '+ New project' button on the top right](/img/docs/dbt-platform/semantic-layer/sl-sigma-make-table.png?v=2 "Click the '+ New project' button on the top right")](#)Click the '+ New project' button on the top right 11. Go ahead and query a working metric in your project! For example, let's say you had a metric that measures various order-related values. Here’s how you would query it: ```sql select * from {{ semantic_layer.query ( metrics = ['order_total', 'order_count', 'large_orders', 'customers_with_orders', 'avg_order_value', 'pct_of_orders_that_are_large'], group_by = [Dimension('metric_time').grain('day') ] ) }} ``` #### What's next Great job on completing the comprehensive Semantic Layer guide 🎉! You should hopefully have gained a clear understanding of what the Semantic Layer is, its purpose, and when to use it in your projects. You've learned how to: * Set up your warehouse and dbt, including loading sample data and connecting your project. * Build, test, and manage dbt projects, focusing on metrics and semantic models. * Run production jobs and query metrics with Google Sheets and other available integrations. For next steps, you can start defining your own metrics and learn additional configuration options such as [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md), [fill null values](https://docs.getdbt.com/docs/build/advanced-topics.md), [implementing Mesh with the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md#how-can-i-implement-dbt-mesh-with-the-dbt-semantic-layer), and more. Here are some additional resources to help you continue your journey: * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) * [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) * Demo on [how to define and query metrics with MetricFlow](https://www.loom.com/share/60a76f6034b0441788d73638808e92ac?sid=861a94ac-25eb-4fd8-a310-58e159950f5a) * [Join our live demos](https://www.getdbt.com/resources/webinars/dbt-cloud-demos-with-experts) --- ### Quickstart with dbt Mesh [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Mesh is a framework that helps organizations scale their teams and data assets effectively. It promotes governance best practices and breaks large projects into manageable sections — for faster data development. Mesh is available for [dbt Enterprise](https://www.getdbt.com/) accounts. This guide will teach you how to set up a multi-project design using foundational concepts of [Mesh](https://www.getdbt.com/blog/what-is-data-mesh-the-definition-and-importance-of-data-mesh) and how to implement a data mesh in dbt: * Set up a foundational project called “Jaffle | Data Analytics” * Set up a downstream project called “Jaffle | Finance” * Add model access, versions, and contracts * Set up a dbt job that is triggered on completion of an upstream job For more information on why data mesh is important, read this post: [What is data mesh? The definition and importance of data mesh](https://www.getdbt.com/blog/what-is-data-mesh-the-definition-and-importance-of-data-mesh). Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. You can also watch the [YouTube video on dbt and Snowflake](https://www.youtube.com/watch?v=kbCkwhySV_I\&list=PL0QYlrC86xQm7CoOH6RS7hcgLnd3OQioG). ##### Related content: * [Data mesh concepts: What it is and how to get started](https://www.getdbt.com/blog/data-mesh-concepts-what-it-is-and-how-to-get-started) * [Deciding how to structure your Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-3-structures.md) * [Mesh best practices guide](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-4-implementation.md) * [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md) #### Prerequisites​ To leverage Mesh, you need the following: * You must have a [dbt Enterprise-tier account](https://www.getdbt.com/contact) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * You have access to a cloud data platform, permissions to load the sample data tables, and dbt permissions to create new projects. * This guide uses the Jaffle Shop sample data, including `customers`, `orders`, and `payments` tables. Follow the provided instructions to load this data into your respective data platform: * [Snowflake](https://docs.getdbt.com/guides/snowflake.md?step=3) * [Databricks](https://docs.getdbt.com/guides/databricks.md?step=3) * [Redshift](https://docs.getdbt.com/guides/redshift.md?step=3) * [BigQuery](https://docs.getdbt.com/guides/bigquery.md?step=3) * [Fabric](https://docs.getdbt.com/guides/microsoft-fabric.md?step=2) * [Starburst Galaxy](https://docs.getdbt.com/guides/starburst-galaxy.md?step=2) This guide assumes you have experience with or fundamental knowledge of dbt. Take the [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) course first if you are brand new to dbt. #### Create and configure two projects In this section, you'll create two new, empty projects in dbt to serve as your foundational and downstream projects: * **Foundational projects** (or upstream projects) typically contain core models and datasets that serve as the base for further analysis and reporting. * **Downstream projects** build on these foundations, often adding more specific transformations or business logic for dedicated teams or purposes. For example, the always-enterprising and fictional account "Jaffle Labs" will create two projects for their data analytics and finance team: **Jaffle | Data Analytics** and **Jaffle | Finance**. To [create](https://docs.getdbt.com/docs/platform/about-platform-setup.md) a new project in dbt: 1. From **Account settings**, go to **Projects**. Click **New project**. 2. Enter a project name and click **Continue**. * Use "Jaffle | Data Analytics" for one project * Use "Jaffle | Finance" for the other project 3. Select your data platform, then **Next** to set up your connection. 4. In the **Configure your environment** section, enter the **Settings** for your new project. 5. Click **Test Connection**. This verifies that dbt can access your data platform account. 6. Click **Next** if the test succeeded. If it fails, you might need to go back and double-check your settings. * For this guide, make sure you create a single [development](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment) and [Deployment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) per project. * For "Jaffle | Data Analytics", set the default database to `jaffle_da`. * For "Jaffle | Finance", set the default database to `jaffle_finance`. 7. Continue the prompts to complete the project setup. Once configured, each project should have: * A data platform connection * New git repo * One or more [environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md) (such as development, deployment) [![Navigate to Account settings.](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/1-settings-gear-icon.png?v=2 "Navigate to Account settings.")](#)Navigate to Account settings. [![Select projects from the menu.](/img/guides/dbt-mesh/select_projects.png?v=2 "Select projects from the menu.")](#)Select projects from the menu. [![Create a new project in the Studio IDE.](/img/guides/dbt-mesh/create_a_new_project.png?v=2 "Create a new project in the Studio IDE.")](#)Create a new project in the Studio IDE. [![Name your project.](/img/guides/dbt-mesh/enter_project_name.png?v=2 "Name your project.")](#)Name your project. [![Select the relevant connection for your projects.](/img/guides/dbt-mesh/select_a_connection.png?v=2 "Select the relevant connection for your projects.")](#)Select the relevant connection for your projects. ##### Create a production environment In dbt, each project can have one deployment environment designated as "Production.". You must set up a ["Production" or "Staging" deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) for each project you want to "mesh" together. This enables you to leverage Catalog in the [later steps](https://docs.getdbt.com/guides/mesh-qs.md?step=5#create-and-run-a-dbt-cloud-job) of this guide. To set a production environment: 1. Navigate to **Deploy** -> **Environments**, then click **Create New Environment**. 2. Select **Deployment** as the environment type. 3. Under **Set deployment type**, select the **Production** button. 4. Select the dbt version. 5. Continue filling out the fields as necessary in the **Deployment connection** and **Deployment credentials** sections. 6. Click **Test Connection** to confirm the deployment connection. 7. Click **Save** to create a production environment. [![Set your production environment as the default environment in your Environment Settings](/img/docs/dbt-platform/using-dbt-platform/prod-settings-1.png?v=2 "Set your production environment as the default environment in your Environment Settings")](#)Set your production environment as the default environment in your Environment Settings #### Set up a foundational project This upstream project is where you build your core data assets. This project will contain the raw data sources, staging models, and core business logic. dbt enables data practitioners to develop in their tool of choice and comes equipped with a local [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or in-browser [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). In this section of the guide, you will set the "Jaffle | Data Analytics" project as your foundational project using the Studio IDE. 1. First, navigate to the **Studio IDE** page to verify your setup. 2. Click **Initialize dbt project** if you’ve started with an empty repo. 3. Delete the `models/example` folder. 4. Navigate to the `dbt_project.yml` file and rename the project (line 5) from `my_new_project` to `analytics`. 5. In your `dbt_project.yml` file, remove lines 39-42 (the `my_new_project` model reference). 6. In the **File Catalog**, hover over the project directory and click the **...**, then select **Create file**. 7. Create two new folders: `models/staging` and `models/core`. ##### Staging layer Now that you've set up the foundational project, let's start building the data assets. Set up the staging layer as follows: 1. Create a new properties YAML file `models/staging/sources.yml`. 2. Declare the sources by copying the following into the file and clicking **Save**. models/staging/sources.yml ```yaml sources: - name: jaffle_shop description: This is a replica of the Postgres database used by our app database: raw schema: jaffle_shop tables: - name: customers description: One record per customer. - name: orders description: One record per order. Includes cancelled and deleted orders. ``` 3. Create a `models/staging/stg_customers.sql` file to select from the `customers` table in the `jaffle_shop` source. models/staging/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from {{ source('jaffle_shop', 'customers') }} ``` 4. Create a `models/staging/stg_orders.sql` file to select from the `orders` table in the `jaffle_shop` source. models/staging/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from {{ source('jaffle_shop', 'orders') }} ``` 5. Create a `models/core/fct_orders.sql` file to build a fact table with customer and order details. models/core/fct\_orders.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date from orders group by customer_id ), final as ( select o.order_id, o.order_date, o.status, c.customer_id, c.first_name, c.last_name, co.first_order_date, -- Note that we've used a macro for this so that the appropriate DATEDIFF syntax is used for each respective data platform {{ datediff('first_order_date', 'order_date', 'day') }} as days_as_customer_at_purchase from orders o left join customers c using (customer_id) left join customer_orders co using (customer_id) ) select * from final ``` 6. Navigate to the **Command bar** and execute `dbt build`. Before a downstream team can leverage assets from this foundational project, you need to first: * [Create and define](https://docs.getdbt.com/docs/mesh/govern/model-access.md) at least one model as “public” * Run a [deployment job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) successfully * Note, (Applies to dbt v2.0 and later) this job updates assets in Catalog automatically. Once run, you can click **Catalog** from the main navigation and select your project to see its lineage, tests, and documentation coming through successfully. #### Define a public model and run first job In the previous section, you've arranged your basic building blocks, now let's integrate Mesh. Although the Finance team requires the `fct_orders` model for analyzing payment trends, other models, particularly those in the staging layer used for data cleansing and joining, are not needed by downstream teams. To make `fct_orders` publicly available: 1. In the `models/core/core.yml` file, add a `access: public` clause to the relevant properties YAML file by adding and saving the following: models/core/core.yml ```yaml models: - name: fct_orders config: access: public # changed to config in v1.10 description: "Customer and order details" columns: - name: order_id data_type: number description: "" - name: order_date data_type: date description: "" - name: status data_type: varchar description: "Indicates the status of the order" - name: customer_id data_type: number description: "" - name: first_name data_type: varchar description: "" - name: last_name data_type: varchar description: "" - name: first_order_date data_type: date description: "" - name: days_as_customer_at_purchase data_type: number description: "Days between this purchase and customer's first purchase" ``` Note: By default, model access is set to "protected", which means they can only be referenced within the same project. Learn more about access types and model groups [here](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers). 2. Navigate to the Studio IDE **Lineage** tab to see the model noted as **Public**, below the model name. [![Jaffle | Data Analytics lineage](/img/guides/dbt-mesh/da_lineage.png?v=2 "Jaffle | Data Analytics lineage")](#)Jaffle | Data Analytics lineage 3. Go to **Version control** and click the **Commit and Sync** button to commit your changes. 4. Merge your changes to the main or production branch. ##### Create and run a dbt job Before a downstream team can leverage assets from this foundational project, you need to [create a production environment](https://docs.getdbt.com/guides/mesh-qs.md?step=3#create-a-production-environment) and run a [deployment job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) successfully. To run your first deployment dbt job, you will need to create a new dbt job. 1. Go to **Orchestration** > **Jobs**. 2. Click **Create job** and then **Deploy job**. 3. (Applies to dbt v2.0 and later) Keep the default `dbt build` command. Every job run hydrates your metadata in Catalog automatically. 4) Click **Save**. 5) Click **Run now** to trigger the job. 6) After the run is complete, navigate to Catalog. You should now see your lineage, tests, and documentation coming through successfully. For details on how dbt uses metadata from the Staging environment to resolve references in downstream projects, check out the section on [Staging with downstream dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#staging-with-downstream-dependencies). #### Reference a public model in your downstream project In this section, you will set up the downstream project, "Jaffle | Finance", and [cross-project reference](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) the `fct_orders` model from the foundational project. Navigate to the **Develop** page to set up our project: 1. If you’ve also started with a new git repo, click **Initialize dbt project** under the **Version control** section. 2. Delete the `models/example` folder. 3. Navigate to the `dbt_project.yml` file and rename the project (line 5) from `my_new_project` to `finance`. 4. Navigate to the `dbt_project.yml` file and remove lines 39-42 (the `my_new_project` model reference). 5. In the **File Catalog**, hover over the project directory, click the **...** and select **Create file**. 6. Name the file `dependencies.yml`. 7. Add the upstream `analytics` project and the `dbt_utils` package. Click **Save**. dependencies.yml ```yaml packages: - package: dbt-labs/dbt_utils version: 1.1.1 projects: - name: analytics ``` ##### Staging layer Now that you've set up the foundational project, let's start building the data assets. Set up the staging layer as follows: 1. Create a new property file `models/staging/sources.yml` and declare the sources by copying the following into the file and clicking **Save**. models/staging/sources.yml ```yml sources: - name: stripe database: raw schema: stripe tables: - name: payment ``` 2. Create `models/staging/stg_payments.sql` to select from the `payment` table in the `stripe` source. models/staging/stg\_payments.sql ```sql with payments as ( select * from {{ source('stripe', 'payment') }} ), final as ( select id as payment_id, orderID as order_id, paymentMethod as payment_method, amount, created as payment_date from payments ) select * from final ``` ##### Reference the public model You're now set to add a model that explores how payment types vary throughout a customer's journey. This helps determine whether coupon gift cards decrease with repeat purchases, as our marketing team anticipates, or remain consistent. 1. To reference the model, use the following logic to ascertain this: models/core/agg\_customer\_payment\_journey.sql ```sql with stg_payments as ( select * from {{ ref('stg_payments') }} ), fct_orders as ( select * from {{ ref('analytics', 'fct_orders') }} ), final as ( select days_as_customer_at_purchase, -- we use the pivot macro in the dbt_utils package to create columns that total payments for each method {{ dbt_utils.pivot( 'payment_method', dbt_utils.get_column_values(ref('stg_payments'), 'payment_method'), agg='sum', then_value='amount', prefix='total_', suffix='_amount' ) }}, sum(amount) as total_amount from fct_orders left join stg_payments using (order_id) group by 1 ) select * from final ``` 2. Notice the cross-project ref at work! When you add the `ref`, the Studio IDE's auto-complete feature recognizes the public model as available. [![Cross-project ref autocomplete in the Studio IDE](/img/guides/dbt-mesh/cross_proj_ref_autocomplete.png?v=2 "Cross-project ref autocomplete in the Studio IDE")](#)Cross-project ref autocomplete in the Studio IDE 3. This automatically resolves (or links) to the correct database, schema, and table/view set by the upstream project. [![Cross-project ref compile](/img/guides/dbt-mesh/cross_proj_ref_compile.png?v=2 "Cross-project ref compile")](#)Cross-project ref compile 4. You can also see this connection displayed in the live **Lineage** tab. [![Cross-project ref lineage](/img/guides/dbt-mesh/cross_proj_ref_lineage.png?v=2 "Cross-project ref lineage")](#)Cross-project ref lineage #### Add model versions and contracts How can you enhance resilience and add guardrails to this type of multi-project relationship? You can adopt best practices from software engineering by: 1. Defining model contracts — Set up [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) in dbt to define a set of upfront "guarantees" that define the shape of your model. While building your model, dbt will verify that your model's transformation will produce a dataset matching up with its contract; if not, the build fails. 2. Defining model versions — Use [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) to manage updates and handle breaking changes systematically. ##### Set up model contracts As part of the Data Analytics team, you may want to ensure the `fct_orders` model is reliable for downstream users, like the Finance team. 1. Navigate to `models/core/core.yml` and under the `fct_orders` model before the `columns:` section, add a data contract to enforce reliability: ```yaml models: - name: fct_orders description: “Customer and order details” config: access: public # changed to config in v1.10 contract: enforced: true columns: - name: order_id ..... ``` 2. Test what would happen if this contract were violated. In `models/core/fct_orders.sql`, comment out the `orders.status` column and click **Build** to try building the model. * If the contract is breached, the build fails, as seen in the command bar history. [![The data contract was breached and the dbt build run failed.](/img/guides/dbt-mesh/break_contract.png?v=2 "The data contract was breached and the dbt build run failed.")](#)The data contract was breached and the dbt build run failed. ##### Set up model versions In this section, you will set up model versions by the Data Analytics team as they upgrade the `fct_orders` model while offering backward compatibility and a migration notice to the downstream Finance team. 1. Rename the existing model file from `models/core/fct_orders.sql` to `models/core/fct_orders_v1.sql`. 2. Create a new file `models/core/fct_orders_v2.sql` and adjust the schema: * Comment out `o.status` in the `final` CTE. * Add a new field, `case when o.status = 'returned' then true else false end as is_return` to indicate if an order was returned. 3. Then, add the following to your `models/core/core.yml` file: * The `is_return` column * The two model `versions` * A `latest_version` to indicate which model is the latest (and should be used by default, unless specified otherwise) * A `deprecation_date` to version 1 as well to indicate when the model will be deprecated. 4. It should now read as follows: models/core/core.yml ```yaml models: - name: fct_orders description: "Customer and order details" latest_version: 2 config: access: public # changed to config in v1.10 contract: enforced: true columns: - name: order_id data_type: number description: "" - name: order_date data_type: date description: "" - name: status data_type: varchar description: "Indicates the status of the order" - name: is_return data_type: boolean description: "Indicates if an order was returned" - name: customer_id data_type: number description: "" - name: first_name data_type: varchar description: "" - name: last_name data_type: varchar description: "" - name: first_order_date data_type: date description: "" - name: days_as_customer_at_purchase data_type: number description: "Days between this purchase and customer's first purchase" # Declare the versions, and highlight the diffs versions: - v: 1 deprecation_date: 2024-06-30 00:00:00.00+00:00 columns: # This means: use the 'columns' list from above, but exclude is_return - include: all exclude: [is_return] - v: 2 columns: # This means: use the 'columns' list from above, but exclude status - include: all exclude: [status] ``` 5. Verify how dbt compiles the `ref` statement based on the updates. Open a new file, add the following select statements, and click **Compile**. Note how each ref is compiled to the specified version (or the latest version if not specified). ```sql select * from {{ ref('fct_orders', v=1) }} select * from {{ ref('fct_orders', v=2) }} select * from {{ ref('fct_orders') }} ``` #### Add a dbt job in the downstream project Before proceeding, make sure you commit and merge your changes in both the “Jaffle | Data Analytics” and “Jaffle | Finance” projects. A member of the Finance team would like to schedule a dbt job for their customer payment journey analysis immediately after the data analytics team refreshes their pipelines. 1. In the “Jaffle | Finance” project, go to the **Jobs** page by navigating to **Orchestration** > **Jobs**. 2. Click **Create job** and then **Deploy job**. 3. Add a name for the job, then scroll to the bottom of the **Job completion** section. 4. In the **Triggers** section, configure the job to **Run when another job finishes** and select the upstream job from the “Jaffle | Data Analytics” project. [![Trigger job on completion](/img/guides/dbt-mesh/trigger_on_completion.png?v=2 "Trigger job on completion")](#)Trigger job on completion 5. Click **Save** and verify the job is set up correctly. 6. Go to the “Jaffle | Data Analytics” jobs page. Select the **Daily job** and click **Run now**. 7. Once this job completes successfully, go back to the “Jaffle | Finance” jobs page. You should see that the Finance team’s job was triggered automatically. This simplifies the process of staying in sync with the upstream tables and removes the need for more sophisticated orchestration skills, such as coordinating jobs across projects via an external orchestrator. #### View deprecation warning To find out how long the Finance team has to migrate from `fct_orders_v1` to `fct_orders_v2`, follow these steps: 1. In the “Jaffle | Finance” project, navigate to the **Develop** page. 2. Edit the cross-project ref to use v=1 in `models/marts/agg_customer_payment_journey.sql`: models/core/agg\_customer\_payment\_journey.sql ```sql with stg_payments as ( select * from {{ ref('stg_payments') }} ), fct_orders as ( select * from {{ ref('analytics', 'fct_orders', v=1) }} ), final as ( select days_as_customer_at_purchase, -- we use the pivot macro in the dbt_utils package to create columns that total payments for each method {{ dbt_utils.pivot( 'payment_method', dbt_utils.get_column_values(ref('stg_payments'), 'payment_method'), agg='sum', then_value='amount', prefix='total_', suffix='_amount' ) }}, sum(amount) as total_amount from fct_orders left join stg_payments using (order_id) group by 1 ) select * from final ``` 3. In the Studio IDE, go to **Version control** to commit and merge the changes. 4. Go to the **Deploy** and then **Jobs** page. 5. Click **Run now** to run the Finance job. The `agg_customer_payment_journey` model will build and display a deprecation date warning. [![The model will display a deprecation date warning.](/img/guides/dbt-mesh/deprecation_date_warning.png?v=2 "The model will display a deprecation date warning.")](#)The model will display a deprecation date warning. #### View lineage with dbt Catalog Use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view the lineage across projects in dbt. Navigate to the **Explore** page for each of your projects — you should now view the [lineage seamlessly across projects](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md). [![View 'Jaffle | Data Analytics' lineage with dbt Catalog ](/img/guides/dbt-mesh/jaffle_da_final_lineage.png?v=2 "View 'Jaffle | Data Analytics' lineage with dbt Catalog ")](#)View 'Jaffle | Data Analytics' lineage with dbt Catalog #### What's next Congratulations 🎉! You're ready to bring the benefits of Mesh to your organization. You've learned: * How to establish a foundational project "Jaffle | Data Analytics." * Create a downstream project "Jaffle | Finance." * Implement model access, versions, and contracts. * Set up dbt jobs triggered by upstream job completions. Here are some additional resources to help you continue your journey: * [How we build our dbt mesh projects](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) * [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md) * [Implement Mesh with the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md#how-can-i-implement-dbt-mesh-with-the-dbt-semantic-layer) * [Cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) * [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) — keep your local and platform environments in sync across credentials, env vars, and dbt v2 versions --- ### Quickstart with MetricFlow time spine [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide explains how to configure a time spine using the [Semantic Layer Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) as a reference. ##### What is a time spine model? A [time spine](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) is essential for time-based joins and aggregations in MetricFlow, the engine that powers the Semantic Layer. To use MetricFlow with time-based metrics and dimensions, you must provide a time spine. This serves as the foundation for time-based joins and aggregations. You can either: * Create a time spine SQL model from scratch, or * Use an existing model in your project, like `dim_date`. And once you have a time spine, you need to configure it in YAML to tell MetricFlow how to use it. This guide will show you how to do both! ##### Prerequisites Before you start, make sure you have: * A dbt project set up. If you don't have one, follow the [Semantic Layer quickstart guide](https://docs.getdbt.com/guides/sl-qs.md?step=1) or the [dbt quickstart guides](https://docs.getdbt.com/guides.md?tags=Quickstart) guide to help you get started. #### Add a time spine SQL model Let's get started by assuming you're creating a time spine from scratch. If you have a dbt project set up already and have your own time spine (like a `dim_date` type model), you can skip this step and go to [Use an existing dim\_date model](https://docs.getdbt.com/guides/mf-time-spine?step=3#using-an-existing-dim_date-model). The time spine is a dbt model that generates a series of dates (or timestamps) at a specific granularity. In this example, let's create a daily time spine — `time_spine_daily.sql`. 1. Navigate to the `models/marts` directory in your dbt project. 2. Add a new SQL file named `time_spine_daily.sql` with the following content: models/marts/time\_spine\_daily.sql ```sql {{ config( materialized = 'table', ) }} with base_dates as ( {{ dbt.date_spine( 'day', "DATE('2000-01-01')", "DATE('2030-01-01')" ) }} ), final as ( select cast(date_day as date) as date_day from base_dates ) select * from final where date_day > dateadd(year, -5, current_date()) -- Keep recent dates only and date_day < dateadd(day, 30, current_date()) ``` This generates a model of daily dates ranging from 5 years in the past to 30 days into the future. 3. Run and preview the model to create the model: ```bash dbt run --select time_spine_daily dbt show --select time_spine_daily # Use this command to preview the model if developing locally ``` 4. If developing in the Studio IDE, you can preview the model by clicking the **Preview** button: [![Preview the time spine model in the Studio IDE](/img/mf-guide-preview-time-spine-table.png?v=2 "Preview the time spine model in the Studio IDE")](#)Preview the time spine model in the Studio IDE #### Add YAML configuration for the time spine Now that you've created the SQL file, configure it in YAML so MetricFlow can recognize and use it. 1. Navigate to the `models/marts` directory. 2. Add a new YAML file named `_models.yml` with the following content: models/marts/\_models.yml ```yaml models: - name: time_spine_daily description: A time spine with one row per day, ranging from 5 years in the past to 30 days into the future. time_spine: standard_granularity_column: date_day # The base column used for time joins columns: - name: date_day description: The base date column for daily granularity granularity: day ``` This time spine YAML file: * Defines `date_day` as the base column for daily granularity. * Configures `time_spine` properties so MetricFlow can use the model. ##### Using an existing dim\_date model This optional approach reuses an existing model, saving you the effort of creating a new one. However if you created a time spine from scratch, you can skip this section. If your project already includes a `dim_date` or similar model, you can configure it as a time spine: 1. Locate the existing model (`dim_date`). 2. Update `_models.yml` file to configure it as a time spine: \_models.yml ```yaml models: - name: dim_date description: An existing date dimension model used as a time spine. time_spine: standard_granularity_column: date_day columns: - name: date_day granularity: day - name: day_of_week granularity: day - name: full_date granularity: day ``` This time spine YAML file configures the `time_spine` property so MetricFlow can use the model. #### Run and preview the time spine For the time spine you created, let's run it and preview the output if you haven't already done this. If you have already run the model, you can skip this step. 1. Run the following command: ```bash dbt run --select time_spine_daily dbt show --select time_spine_daily # Use this command to preview the model if developing locally ``` 2. If developing in the Studio IDE, you can preview the model by clicking the **Preview** button: [![Preview the time spine model in the Studio IDE](/img/mf-guide-preview-time-spine-table.png?v=2 "Preview the time spine model in the Studio IDE")](#)Preview the time spine model in the Studio IDE 3. Check that the model: * Contains one row per day. * Covers the date range you want (5 years back to 30 days forward). 4. (Optional) If you have [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) already defined in your project, you can query the model/metrics using [Semantic Layer commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) to validate the time spine. Let's say you have a `revenue` metric defined. You can query the model/metrics using the following command: ```bash dbt sl query --metrics revenue --group-by metric_time ``` This will output results similar to the following in the Studio IDE: [![Validate the metrics and time spine output in the Studio IDE](/img/quickstarts/dbt-platform/validate-mf-timespine-output.png?v=2 "Validate the metrics and time spine output in the Studio IDE")](#)Validate the metrics and time spine output in the Studio IDE 5. Double check that the results are correct and returning the expected data. #### Add additional granularities This section is optional and will show you how to add additional granularities to your time spine: * [Yearly](#yearly-time-spine) * [Custom calendars](#custom-calendars) ##### Yearly time spine To support multiple granularities (like hourly, yearly, monthly), create additional time spine models and configure them in YAML. 1. Add a new SQL file named `time_spine_yearly.sql` with the following content: models/marts/time\_spine\_yearly.sql ```sql {{ config( materialized = 'table', ) }} with years as ( {{ dbt.date_spine( 'year', "to_date('01/01/2000','mm/dd/yyyy')", "to_date('01/01/2025','mm/dd/yyyy')" ) }} ), final as ( select cast(date_year as date) as date_year from years ) select * from final -- filter the time spine to a specific range where date_year >= date_trunc('year', dateadd(year, -4, current_timestamp())) and date_year < date_trunc('year', dateadd(year, 1, current_timestamp())) ``` 2. Then update the `_models.yml` file and add the yearly time spine (below the daily time spine config): \_models.yml ```yaml models: - name: time_spine_daily ... rest of the daily time spine config ... - name: time_spine_yearly description: time spine one row per house time_spine: standard_granularity_column: date_year columns: - name: date_year granularity: year ``` 3. Run or preview the model to create the model: ```bash dbt run --select time_spine_yearly dbt show --select time_spine_yearly # Use this command to preview the model if developing locally ``` 4. Validate the output by querying the generated model: ```bash dbt sl query --metrics orders --group-by metric_time__year ``` If you're developing in the Studio IDE, you can preview the model by clicking the **Preview** button. [![Validate the metrics and time spine output in the Studio IDE](/img/mf-guide-query.png?v=2 "Validate the metrics and time spine output in the Studio IDE")](#)Validate the metrics and time spine output in the Studio IDE Extra credit! For some extra practice, try one of the following exercises: * Order the `dbt sl query --metrics orders --group-by metric_time__year` command output by ascending order of `metric_time__year`. Check out the [dbt Semantic Layer commands](https://docs.getdbt.com/docs/build/metricflow-commands.md#query) docs for more information on how to do this. * Filter to this year and last year only to limit data returned. * Try creating a monthly time spine — duplicate your daily time spine model, adjust it to generate one row per month, and update the YAML file to include `granularity: month`. Give it a try! ##### Custom calendars To support custom calendars (like fiscal years, fiscal quarters, and so on), create an additional time spine and configure it in YAML. This feature is available in the dbt's [**v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or [dbt 1.9 and later](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md). 1. Add a new SQL file named `fiscal_calendar.sql` with the following content (or use your own custom calendar and configure it in YAML): models/marts/fiscal\_calendar.sql ```sql with date_spine as ( select date_day, extract(year from date_day) as calendar_year, extract(week from date_day) as calendar_week from {{ ref('time_spine_daily') }} ), fiscal_calendar as ( select date_day, -- Define custom fiscal year starting in October case when extract(month from date_day) >= 10 then extract(year from date_day) + 1 else extract(year from date_day) end as fiscal_year, -- Define fiscal weeks (e.g., shift by 1 week) extract(week from date_day) + 1 as fiscal_week from date_spine ) select * from fiscal_calendar ``` 2. Then update `_models.yml` file and add the fiscal calendar time spine (below the yearly time spine config): \_models.yml ```yaml models: - name: time_spine_yearly ... rest of the yearly time spine config ... - name: fiscal_calendar description: A custom fiscal calendar with fiscal year and fiscal week granularities. time_spine: standard_granularity_column: date_day custom_granularities: - name: fiscal_year column_name: fiscal_year - name: fiscal_week column_name: fiscal_week columns: - name: date_day granularity: day - name: fiscal_year description: "Custom fiscal year starting in October" - name: fiscal_week description: "Fiscal week, shifted by 1 week from standard calendar" ``` 3. Run or preview the model to create the model: ```bash dbt run --select fiscal_calendar dbt show --select fiscal_calendar # Use this command to preview the model if developing locally ``` If you're developing in the Studio IDE, you can preview the model by clicking the **Preview** button. 4. Validate the output by querying the generated model along with your metrics: ```bash dbt sl query --metrics orders --group-by metric_time__fiscal_year ``` [![Validate the custom calendar metrics and time spine output in the Studio IDE](/img/mf-guide-fiscal-preview.png?v=2 "Validate the custom calendar metrics and time spine output in the Studio IDE")](#)Validate the custom calendar metrics and time spine output in the Studio IDE #### What's next Congratulations 🎉! You've set up a time spine and are ready to bring the benefits of MetricFlow and the Semantic Layer to your organization. You've learned: * How to create a time spine or use an existing model. * How to configure a time spine in YAML. * How to add additional granularities to your time spine. Here are some additional resources to help you continue your journey: * [MetricFlow time spine](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) * [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) * [Build metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) * [Quickstart with Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md?step=1) --- ### Refactoring legacy SQL to dbt [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction You may have already learned how to build dbt models from scratch. But in reality, you probably already have some queries or stored procedures that power analyses and dashboards, and now you’re wondering how to port those into dbt. There are two parts to accomplish this: migration and refactoring. In this guide we’re going to learn a process to help us turn legacy SQL code into modular dbt models. When migrating and refactoring code, it’s of course important to stay organized. We'll do this by following several steps: 1. Migrate your code 1:1 into dbt 2. Implement dbt sources rather than referencing raw database tables 3. Choose a refactoring strategy 4. Implement CTE groupings and cosmetic cleanup 5. Separate [data transformations](https://www.getdbt.com/analytics-engineering/transformation/) into standardized layers 6. Audit the output of dbt models vs legacy SQL Let's get into it! More resources This guide is excerpted from the new dbt Learn On-demand Course, "Refactoring SQL for Modularity" - if you're curious, pick up the [free refactoring course here](https://learn.getdbt.com/courses/refactoring-sql-for-modularity), which includes example and practice refactoring projects. Or for a more in-depth look at migrating DDL and DML from stored procedures, refer to the[Migrate from stored procedures](https://docs.getdbt.com/guides/migrate-from-stored-procedures.md) guide. #### Migrate your existing SQL code Your goal in this initial step is simply to use dbt to run your existing SQL transformation, with as few modifications as possible. This will give you a solid base to work from. While refactoring you'll be **moving around** a lot of logic, but ideally you won't be **changing** the logic. More changes = more auditing work, so if you come across anything you'd like to fix, try your best to card that up for another task after refactoring! We'll save the bulk of our auditing for the end when we've finalized our legacy-to-dbt model restructuring. To get going, you'll copy your legacy SQL query into your dbt project, by saving it in a `.sql` file under the `/models` directory of your project. [![Your dbt project's folder structure](/img/tutorial/refactoring/legacy-query-model.png?v=2 "Your dbt project's folder structure")](#)Your dbt project's folder structure Once you've copied it over, you'll want to `dbt run` to execute the query and populate the table in your warehouse. If this is your first time running dbt, you may want to start with the [Introduction to dbt](https://docs.getdbt.com/docs/introduction.md) and the earlier sections of the [quickstart guide](https://docs.getdbt.com/guides.md) before diving into refactoring. This step may sound simple, but if you're porting over an existing set of SQL transformations to a new SQL dialect, you will need to consider how your legacy SQL dialect differs from your new SQL flavor, and you may need to modify your legacy code to get it to run at all. This will commonly happen if you're migrating from a [stored procedure workflow on a legacy database](https://getdbt.com/analytics-engineering/case-for-elt-workflow/) into dbt + a cloud data warehouse. Functions that you were using previously may not exist, or their syntax may shift slightly between SQL dialects. If you're not migrating data warehouses at the moment, then you can keep your SQL syntax the same. You have access to the exact same SQL dialect inside of dbt that you have querying directly from your warehouse. #### Create sources from table references To query from your data warehouse, we recommend creating [sources in dbt](https://docs.getdbt.com/docs/build/sources.md) rather than querying the database table directly. This allows you to call the same table in multiple places with `{{ src('my_source', 'my_table') }}` rather than `my_database.my_schema.my_table`. We start here for several reasons: ###### Source freshness reporting Using sources unlocks the ability to run [source freshness reporting](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) to make sure your raw data isn't stale. ###### Easy dependency tracing If you're migrating multiple stored procedures into dbt, with sources you can see which queries depend on the same raw tables. This allows you to consolidate modeling work on those base tables, rather than calling them separately in multiple places. [![Sources appear in green in your DAG in dbt docs](/img/docs/building-a-dbt-project/sources-dag.png?v=2 "Sources appear in green in your DAG in dbt docs")](#)Sources appear in green in your DAG in dbt docs ###### Build the habit of analytics-as-code Sources are an easy way to get your feet wet using config files to define aspects of your transformation pipeline. ```yml sources: - name: jaffle_shop tables: - name: orders - name: customers ``` With a few lines of code in a `.yml` file in your dbt project's `/models` subfolder, you can now version control how your data sources (Snowplow, Shopify, etc) map to actual database tables. For example, let's say you migrate from one ETL tool to another, and the new tool writes to a new schema in your warehouse. dbt sources allow you to make that update in a single config file, and flip on the change with one pull request to your dbt project. #### Choose a refactoring strategy There are two ways you can choose to refactor: in-place or alongside. ###### In-place refactoring Means that you will work directly on the SQL script that you ported over in the first step. You'll move it into a `/marts` subfolder within your project's `/models` folder and go to town. **Pros**: * You won't have any old models to delete once refactoring is done. **Cons**: * More pressure to get it right the first time, especially if you've referenced this model from any BI tool or downstream process. * Harder to audit, since you've overwritten your audit comparison model. * Requires navigating through Git commits to see what code you've changed throughout. ###### Alongside refactoring Means that you will copy your model to a `/marts` folder, and work on changes on that copy. **Pros**: * Less impact on end users - anything that is referencing the model you're refactoring can keep that reference until you can safely deprecate that model. * Less pressure to get it right the first time, meaning you can push/merge smaller PRs. This is better for you and your reviewers. * You can audit easier by running the old and new models in your dev branch and comparing the results. This ensures the datasets you're comparing have the same or very close to the same records. * You can look at old code more easily, as it has not been changed. * You can decide when the old model is ready to be deprecated. **Cons**: * You'll have the old file(s) in your project until you can deprecate them - running side-by-side like this can feel duplicative, and may be a headache to manage if you're migrating a number of queries in bulk. We generally recommend the **alongside** approach, which we'll follow in this tutorial. #### Implement CTE groupings Once you choose your refactoring strategy, you'll want to do some cosmetic cleanups according to your data modeling best practices and start moving code into CTE groupings. This will give you a head start on porting SQL snippets from CTEs into modular [dbt data models](https://docs.getdbt.com/docs/build/models.md). ##### What's a CTE? CTE stands for “Common Table Expression”, which is a temporary result set available for use until the end of SQL script execution. Using the `with` keyword at the top of a query allows us to use CTEs in our code. Inside of the model we're refactoring, we’re going to use a 4-part layout: 1. 'Import' CTEs 2. 'Logical' CTEs 3. A 'Final' CTE 4. A simple SELECT statement In practice this looks like: ```sql with import_orders as ( -- query only non-test orders select * from {{ source('jaffle_shop', 'orders') }} where amount > 0 ), import_customers as ( select * from {{ source('jaffle_shop', 'customers') }} ), logical_cte_1 as ( -- perform some math on import_orders ), logical_cte_2 as ( -- perform some math on import_customers ), final_cte as ( -- join together logical_cte_1 and logical_cte_2 ) select * from final_cte ``` Notice there are no nested queries here, which makes reading our logic much more straightforward. If a query needs to be nested, it's just a new CTE that references the previous CTE. ###### 1. Import CTEs Let's start with our components, and identify raw data that is being used in our analysis. For this exercise, the components are three sources: * jaffle\_shop.customers * jaffle\_shop.orders * stripe.payment Let's make a CTE for each of these under the `Import CTEs` comment. These import CTEs should be only simple `select *` statements, but can have filters if necessary. We'll cover that later - for now, just use `select * from {{ source('schema', 'table') }}` for each, with the appropriate reference. Then, we will switch out all hard-coded references with our import CTE names. ###### 2. Logical CTEs Logical CTEs contain unique transformations used to generate the final product, and we want to separate these into logical blocks. To identify our logical CTEs, we will follow subqueries in order. If a subquery has nested subqueries, we will want to continue moving down until we get to the first layer, then pull out the subqueries in order as CTEs, making our way back to the final select statement. Name these CTEs as the alias that the subquery was given - you can rename it later, but for now it is best to make as few changes as possible. If the script is particularly complicated, it's worth it to go through once you're finished pulling out subqueries and follow the CTEs to make sure they happen in an order that makes sense for the end result. ###### 3. Final CTE The previous process usually results in a select statement that is left over at the end - this select statement can be moved into its own CTE called the final CTE, or can be named something that is inherent for others to understand. This CTE determines the final product of the model. ###### 4. Simple SELECT statement After you have moved everything into CTEs, you'll want to write a `select * from final` (or something similar, depending on your final CTE name) at the end of the model. This allows anyone after us to easily step through the CTEs when troubleshooting, rather than having to untangle nested queries. > For more background on CTEs, check out the [dbt Labs style guide](https://docs.getdbt.com/best-practices/how-we-style/0-how-we-style-our-dbt-projects.md). #### Port CTEs to individual data models Rather than keep our SQL code confined to one long SQL file, we'll now start splitting it into modular + reusable [dbt data models](https://docs.getdbt.com/docs/build/models.md). Internally at dbt Labs, we follow roughly this [data modeling technique](https://www.getdbt.com/analytics-engineering/modular-data-modeling-technique/) and we [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) accordingly. We'll follow those structures in this walkthrough, but your team's conventions may of course differ from ours. ##### Identifying staging models To identify our [staging models](https://www.getdbt.com/analytics-engineering/modular-data-modeling-technique/#staging-models), we want to look at the things we've imported in our import CTEs. For us, that's customers, orders, and payments. We want to look at the transformations that can occur within each of these sources without needing to be joined to each other, and then we want to make components out of those so they can be our building blocks for further development. ##### CTEs or intermediate models Our left-over logic can then be split into steps that are more easily understandable. We'll start by using CTEs, but when a model becomes complex or can be divided out into reusable components you may consider an intermediate model. Intermediate models are optional and are not always needed, but do help when you have large data flows coming together. ##### Final model Our final model accomplishes the result set we want, and it uses the components we've built. By this point we've identified what we think should stay in our final model. #### Data model auditing We'll want to audit our results using the dbt [audit\_helper package](https://hub.getdbt.com/dbt-labs/audit_helper/latest/). Under the hood, it generates comparison queries between our before and after states, so that we can compare our original query results to our refactored results to identify differences. Sure, we could write our own query manually to audit these models, but using the dbt `audit_helper` package gives us a head start and allows us to identify variances more quickly. ##### Ready for refactoring practice? Head to the free on-demand course, [Refactoring from Procedural SQL to dbt](https://learn.getdbt.com/courses/refactoring-sql-for-modularity) for a more in-depth refactoring example + a practice refactoring problem to test your skills. Questions on this guide or the course? Drop a note in #learn-on-demand in [dbt Community Slack](https://getdbt.com/community). --- ### Refresh a Mode dashboard when a job completes [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide will teach you how to refresh a Mode dashboard when a dbt job has completed successfully and there is fresh data available. The integration will: * Receive a webhook notification in Zapier * Trigger a refresh of a Mode report Although we are using the Mode API for a concrete example, the principles are readily transferable to your [tool](https://learn.hex.tech/docs/develop-logic/hex-api/api-reference#operation/RunProject) [of](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/refresh-dataset) [choice](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_ref.htm#update_workbook_now). ##### Prerequisites In order to set up the integration, you should have familiarity with: * [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * Zapier * The [Mode API](https://mode.com/developer/api-reference/introduction/) #### Create a new Zap in Zapier Use **Webhooks by Zapier** as the Trigger, and **Catch Raw Hook** as the Event. If you don't intend to [validate the authenticity of your webhook](https://docs.getdbt.com/docs/deploy/webhooks.md#validate-a-webhook) (not recommended!) then you can choose **Catch Hook** instead. Press **Continue**, then copy the webhook URL. [![Screenshot of the Zapier UI, showing the webhook URL ready to be copied](/img/guides/orchestration/webhooks/zapier-common/catch-raw-hook.png?v=2 "Screenshot of the Zapier UI, showing the webhook URL ready to be copied")](#)Screenshot of the Zapier UI, showing the webhook URL ready to be copied #### Configure a new webhook in dbt See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Your event should be **Run completed**, and you need to change the **Jobs** list to only contain any jobs whose completion should trigger a report refresh. Make note of the Webhook Secret Key for later. Once you've tested the endpoint in dbt, go back to Zapier and click **Test Trigger**, which will create a sample webhook body based on the test event dbt sent. The sample body's values are hard-coded and not reflective of your project, but they give Zapier a correctly-shaped object during development. #### Store secrets In the next step, you will need the Webhook Secret Key from the prior step, and a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md), as well as a [Mode API token and secret](https://mode.com/developer/api-reference/authentication/). Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps), which prevents your keys from being displayed in plaintext in the Zap code. You will be able to access them via the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the names for the secret keys are: `DBT_WEBHOOK_KEY`, `MODE_API_TOKEN`, and `MODE_API_SECRET`. If you are using different names, make sure you update all references to them in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ##### a. Create a Storage by Zapier connection If you haven't already got one, go to <https://zapier.com/app/connections/storage> and create a new connection. Remember the UUID secret you generate for later. ##### b. Add a temporary code step Choose **Run Python** as the Event. Run the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_WEBHOOK_KEY', 'abc123') #replace with your dbt API token store.set('MODE_API_TOKEN', 'abc123') #replace with your Mode API Token store.set('MODE_API_SECRET', 'abc123') #replace with your Mode API Secret ``` Test the step. You can delete this Action when the test succeeds. The key will remain stored as long as it is accessed at least once every three months. #### Add a code action Select **Code by Zapier** as the App, and **Run Python** as the Event. In the **Set up action** area, add two items to **Input Data**: `raw_body` and `auth_header`. Map those to the `1. Raw Body` and `1. Headers Http Authorization` fields from the **Catch Raw Hook** step above. [![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/img/guides/orchestration/webhooks/zapier-common/run-python.png?v=2 "Screenshot of the Zapier UI, showing the mappings of raw_body and auth_header")](#)Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header In the **Code** field, paste the following code, replacing `YOUR_SECRET_HERE` in the StoreClient constructor with the secret you created when setting up the Storage by Zapier integration (not your dbt secret), and setting the `account_username` and `report_token` variables to actual values. The code below will validate the authenticity of the request, then send a [`run report` command to the Mode API](https://mode.com/developer/api-reference/analytics/report-runs/#runReport) for the given report token. ```python import hashlib import hmac import json #replace with the report token you want to run account_username = 'YOUR_MODE_ACCOUNT_USERNAME_HERE' report_token = 'YOUR_REPORT_TOKEN_HERE' auth_header = input_data['auth_header'] raw_body = input_data['raw_body'] # Access secret credentials secret_store = StoreClient('YOUR_SECRET_HERE') hook_secret = secret_store.get('DBT_WEBHOOK_KEY') username = secret_store.get('MODE_API_TOKEN') password = secret_store.get('MODE_API_SECRET') # Validate the webhook came from dbt signature = hmac.new(hook_secret.encode('utf-8'), raw_body.encode('utf-8'), hashlib.sha256).hexdigest() if signature != auth_header: raise Exception("Calculated signature doesn't match contents of the Authorization header. This webhook may not have been sent from dbt.") full_body = json.loads(raw_body) hook_data = full_body['data'] if hook_data['runStatus'] == "Success": # Create a report run with the Mode API url = f'https://app.mode.com/api/{account_username}/reports/{report_token}/run' params = { 'parameters': { "user_id": 123, "location": "San Francisco" } } headers = { 'Content-Type': 'application/json', 'Accept': 'application/hal+json' } response = requests.post( url, json=params, headers=headers, auth=HTTPBasicAuth(username, password) ) response.raise_for_status() return ``` #### Test and deploy You can iterate on the Code step by modifying the code and then running the test again. When you're happy with it, you can publish your Zap. --- ### Refresh Tableau workbook with extracts after a job finishes [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide will teach you how to refresh a Tableau workbook that leverages [extracts](https://help.tableau.com/current/pro/desktop/en-us/extracting_data.htm) when a dbt job has completed successfully and there is fresh data available. The integration will: * Receive a webhook notification in Zapier * Trigger a refresh of a Tableau workbook ##### Prerequisites To set up the integration, you need to be familiar with: * [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * Zapier * The [Tableau API](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api.htm) * The [version](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_versions.htm#rest_api_versioning) of Tableau's REST API that is compatible with your server #### Obtain authentication credentials from Tableau To authenticate with the Tableau API, obtain a [Personal Access Token](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) from your Tableau Server/Cloud instance. In addition, make sure your Tableau workbook uses data sources that allow refresh access, which is usually set when publishing. #### Create a new Zap in Zapier To trigger an action with the delivery of a webhook in Zapier, you'll want to create a new Zap with **Webhooks by Zapier** as the Trigger and **Catch Raw Hook** as the Event. However, if you choose not to [validate the authenticity of your webhook](https://docs.getdbt.com/docs/deploy/webhooks.md#validate-a-webhook), which isn't recommended, you can choose **Catch Hook** instead. Press **Continue**, then copy the webhook URL. [![Screenshot of the Zapier UI, showing the webhook URL ready to be copied](/img/guides/orchestration/webhooks/zapier-common/catch-raw-hook.png?v=2 "Screenshot of the Zapier UI, showing the webhook URL ready to be copied")](#)Screenshot of the Zapier UI, showing the webhook URL ready to be copied #### Configure a new webhook in dbt To set up a webhook subscription for dbt, follow the instructions in [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription). For the event, choose **Run completed** and modify the **Jobs** list to include only the jobs that should trigger a report refresh. Remember to save the Webhook Secret Key for later. Paste in the webhook URL obtained from Zapier in step 2 into the **Endpoint** field and test the endpoint. Once you've tested the endpoint in dbt, go back to Zapier and click **Test Trigger**, which will create a sample webhook body based on the test event dbt sent. The sample body's values are hard-coded and not reflective of your project, but they give Zapier a correctly-shaped object during development. #### Store secrets In the next step, you will need the Webhook Secret Key from the prior step, and your Tableau authentication credentials and details. Specifically, you'll need your Tableau server/site URL, server/site name, PAT name, and PAT secret. Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps), which prevents your keys from being displayed in plaintext in the Zap code. You will be able to access them via the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the names for the secret keys are: `DBT_WEBHOOK_KEY`, `TABLEAU_SITE_URL`, `TABLEAU_SITE_NAME`, `TABLEAU_API_TOKEN_NAME`, and `TABLEAU_API_TOKEN_SECRET`. If you are using different names, make sure you update all references to them in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ##### a. Create a Storage by Zapier connection Create a new connection at <https://zapier.com/app/connections/storage> if you don't already have one and remember the UUID secret you generate for later. ##### b. Add a temporary code step Choose **Run Python** as the Event and input the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_WEBHOOK_KEY', 'abc123') #replace with your dbt Webhook key store.set('TABLEAU_SITE_URL', 'abc123') #replace with your Tableau Site URL, inclusive of https:// and .com store.set('TABLEAU_SITE_NAME', 'abc123') #replace with your Tableau Site/Server Name store.set('TABLEAU_API_TOKEN_NAME', 'abc123') #replace with your Tableau API Token Name store.set('TABLEAU_API_TOKEN_SECRET', 'abc123') #replace with your Tableau API Secret ``` Test the step to run the code. You can delete this action when the test succeeds. The keys will remain stored as long as it is accessed at least once every three months. #### Add a code action Select **Code by Zapier** as the App, and **Run Python** as the Event. In the **Set up action** area, add two items to **Input Data**: `raw_body` and `auth_header`. Map those to the `1. Raw Body` and `1. Headers Http Authorization` fields from the **Catch Raw Hook** step above. [![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/img/guides/orchestration/webhooks/zapier-common/run-python.png?v=2 "Screenshot of the Zapier UI, showing the mappings of raw_body and auth_header")](#)Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header In the **Code** field, paste the following code, replacing `YOUR_STORAGE_SECRET_HERE` in the StoreClient constructor with the UUID secret you created when setting up the Storage by Zapier integration, and replacing the `workbook_name` and `api_version` variables to actual values. The following code validates the authenticity of the request and obtains the workbook ID for the specified workbook name. Next, the code will send a [`update workbook` command to the Tableau API](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_ref_workbooks_and_views.htm#update_workbook_now) for the given workbook ID. ```python import requests import hashlib import json import hmac # Access secret credentials secret_store = StoreClient('YOUR_STORAGE_SECRET_HERE') hook_secret = secret_store.get('DBT_WEBHOOK_KEY') server_url = secret_store.get('TABLEAU_SITE_URL') server_name = secret_store.get('TABLEAU_SITE_NAME') pat_name = secret_store.get('TABLEAU_API_TOKEN_NAME') pat_secret = secret_store.get('TABLEAU_API_TOKEN_SECRET') #Enter the name of the workbook to refresh workbook_name = "YOUR_WORKBOOK_NAME" api_version = "ENTER_COMPATIBLE_VERSION" #Validate authenticity of webhook coming from dbt auth_header = input_data['auth_header'] raw_body = input_data['raw_body'] signature = hmac.new(hook_secret.encode('utf-8'), raw_body.encode('utf-8'), hashlib.sha256).hexdigest() if signature != auth_header: raise Exception("Calculated signature doesn't match contents of the Authorization header. This webhook may not have been sent from dbt.") full_body = json.loads(raw_body) hook_data = full_body['data'] if hook_data['runStatus'] == "Success": #Authenticate with Tableau Server to get an authentication token auth_url = f"{server_url}/api/{api_version}/auth/signin" auth_data = { "credentials": { "personalAccessTokenName": pat_name, "personalAccessTokenSecret": pat_secret, "site": { "contentUrl": server_name } } } auth_headers = { "Accept": "application/json", "Content-Type": "application/json" } auth_response = requests.post(auth_url, data=json.dumps(auth_data), headers=auth_headers) #Extract token to use for subsequent calls auth_token = auth_response.json()["credentials"]["token"] site_id = auth_response.json()["credentials"]["site"]["id"] #Extract the workbook ID workbooks_url = f"{server_url}/api/{api_version}/sites/{site_id}/workbooks" workbooks_headers = { "Accept": "application/json", "Content-Type": "application/json", "X-Tableau-Auth": auth_token } workbooks_params = { "filter": f"name:eq:{workbook_name}" } workbooks_response = requests.get(workbooks_url, headers=workbooks_headers, params=workbooks_params) #Assign workbook ID workbooks_data = workbooks_response.json() workbook_id = workbooks_data["workbooks"]["workbook"][0]["id"] # Refresh the workbook refresh_url = f"{server_url}/api/{api_version}/sites/{site_id}/workbooks/{workbook_id}/refresh" refresh_data = {} refresh_headers = { "Accept": "application/json", "Content-Type": "application/json", "X-Tableau-Auth": auth_token } refresh_trigger = requests.post(refresh_url, data=json.dumps(refresh_data), headers=refresh_headers) return {"message": "Workbook refresh has been queued"} ``` #### Test and deploy To make changes to your code, you can modify it and test it again. When you're happy with it, you can publish your Zap. --- ### Run dbt MCP locally [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() This quickstart walks you through connecting dbt MCP server to your local dbt project. This setup gives you dbt command tools (`run`, `build`, `test`, `compile`, and more) inside your AI assistant. If you'd like to connect to dbt platform with the CLI, see the [OAuth quickstart](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md). To connect to dbt Wizard, see the [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). No clone required You don't need to clone the dbt-mcp repository. Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` — it fetches and runs dbt-mcp for you. #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) * A local dbt project (the folder containing your `dbt_project.yml` file) * dbt installed and available locally For the full list of environment variables and how to enable or disable toolsets, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Step 1: Find your paths You need two values before configuring your MCP client: * `DBT_PROJECT_DIR` — the full path to your dbt project folder (where `dbt_project.yml` lives). For example, if your project name is `jaffle_shop`, the path should be `/Users/yourname/dbt-projects/jaffle_shop`. * `DBT_PATH` — the full path to your dbt executable. ##### macOS/Linux ```bash # Find DBT_PATH which dbt # Example output: /opt/homebrew/bin/dbt # Find DBT_PROJECT_DIR — run from inside your project folder pwd # Example output: /Users/yourname/projects/my_dbt_project ``` ##### Windows ```bash # Find DBT_PATH where dbt # Example output: C:\Python39\Scripts\dbt.exe # Find DBT_PROJECT_DIR — run from inside your project folder cd # Example output: C:\Users\yourname\projects\my_dbt_project ``` Note: Use forward slashes in your configuration: `C:/Python39/Scripts/dbt.exe` #### Step 2: Add to your MCP client Replace the paths below with the values from Step 1: ##### Claude Desktop 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing the paths with your actual values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing the paths with your actual values: ```shell claude mcp add dbt \ -e DBT_PROJECT_DIR=/path/to/your/dbt/project \ -e DBT_PATH=/path/to/your/dbt/executable \ -- uvx dbt-mcp ``` ##### Cursor Click the link below with Cursor open to auto-configure: [Add dbt v1 or dbt v2 to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX1BST0pFQ1RfRElSIjoiL3BhdGgvdG8veW91ci9kYnQvcHJvamVjdCIsIkRCVF9QQVRIIjoiL3BhdGgvdG8veW91ci9kYnQvZXhlY3V0YWJsZSJ9LCJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJkYnQtbWNwIl19) After clicking: 1. Update `DBT_PROJECT_DIR` with the full path to your dbt project. 2. Update `DBT_PATH` with the full path to your dbt executable (from Step 1). 3. Save the configuration. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`, replacing the paths with your actual values: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` 4. Save the file. #### Step 3: Test your setup Ask your AI assistant to run a dbt command (for example, *"Run `dbt compile` on my project"* or *"List all models in my project"*). If dbt MCP is working, the assistant will execute the command against your local project.  Optional: verify from the command line ```bash uvx dbt-mcp ``` If there are no errors, your configuration is correct. Press `Ctrl+C` to stop the server. #### What's available With CLI-only setup, your AI assistant can use: * `dbt run`, `dbt build`, `dbt test`, `dbt compile`, `dbt list`, `dbt parse`, `dbt show` * Model lineage and node details from your local project * Codegen tools (when enabled — see [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md)) Platform features like Semantic Layer, Discovery API, and metadata queries require a dbt platform account. To add them, see [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md). #### Troubleshooting Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field. Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). For all troubleshooting topics, see [dbt MCP FAQs and troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). #### Next steps * Add dbt platform features: see [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) * Configure toolsets or disable specific tools: see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) --- ### Set up your dbt project with Databricks [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Databricks and dbt Labs are partnering to help data teams think like software engineering teams and ship trusted data, faster. The dbt-databricks adapter enables dbt users to leverage the latest Databricks features in their dbt project. Hundreds of customers are now using dbt and Databricks to build expressive and reliable data pipelines on the Lakehouse, generating data assets that enable analytics, ML, and AI use cases throughout the business. In this guide, we discuss how to set up your dbt project on the Databricks Lakehouse Platform so that it scales from a small team all the way up to a large organization. #### Configuring the Databricks Environments To get started, we will use Databricks’s Unity Catalog. Without it, we would not be able to design separate [environments](https://docs.getdbt.com/docs/environments-in-dbt.md) for development and production per our [best practices](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). It also allows us to ensure the proper access controls have been applied using SQL. You will need to be using the dbt-databricks adapter to use it (as opposed to the dbt-spark adapter). We will set up two different *catalogs* in Unity Catalog: **dev** and **prod**. A catalog is a top-level container for *schemas* (previously known as databases in Databricks), which in turn contain tables and views. Our dev catalog will be the development environment that analytics engineers interact with through their Studio IDE. Developers should have their own sandbox to build and test objects in without worry of overwriting or dropping a coworker’s work; we recommend creating personal schemas for this purpose. In terms of permissions, they should only have access to the **dev** catalog. Only production runs will have access to data in the **prod** catalog. In a future guide, we will discuss a **test** catalog where our continuous integration/continuous deployment (CI/CD) system can run `dbt test`. For now, let’s keep things simple and [create two catalogs](https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-catalog.html) either using the Data Catalog or in the SQL editor with these commands: ```sql create catalog if not exists dev; create catalog if not exists prod; ``` As long as your developer is given write access to the dev data catalog, there is no need to create the sandbox schemas ahead of time. #### Setting up Service Principals When an analytics engineer runs a dbt project from their Studio IDE, it is perfectly fine for the resulting queries to execute with that user’s identity. However, we want production runs to execute with a *service principal's* identity. As a reminder, a service principal is a headless account that does not belong to an actual person. Service principals are used to remove humans from deploying to production for convenience and security. Personal identities should not be used to build production pipelines because they could break if the user leaves the company or changes their credentials. Also, there should not be ad hoc commands modifying production data. Only scheduled jobs and running code that has passed CI tests and code reviews should be allowed to modify production data. If something breaks, there is an auditable trail of changes to find the root cause, easily revert to the last working version of the code, and minimize the impact on end users. [Let’s create a service principal](https://docs.databricks.com/administration-guide/users-groups/service-principals.html#add-a-service-principal-to-your-databricks-account) in Databricks: 1. Have your Databricks Account admin [add a service principal](https://docs.databricks.com/aws/en/admin/users-groups/manage-service-principals#-add-service-principals-to-your-account) to your account. The service principal’s name should differentiate itself from a user ID and make its purpose clear (eg dbt\_prod\_sp). 2. Add the service principal added to any groups it needs to be a member of at this time. There are more details on permissions in our ["Unity Catalog best practices" guide](https://docs.getdbt.com/best-practices/dbt-unity-catalog-best-practices.md). 3. [Add the service principal to your workspace](https://docs.databricks.com/aws/en/admin/users-groups/manage-service-principals#assign-a-service-principal-to-a-workspace) and apply any [necessary entitlements](https://docs.databricks.com/aws/en/security/auth/entitlements), such as Databricks SQL access and Workspace access. #### Setting up Databricks Compute When you run a dbt project, it generates SQL, which can run on All Purpose Clusters or SQL warehouses. We strongly recommend running dbt-generated SQL on a Databricks SQL warehouse. Since SQL warehouses are optimized for executing SQL queries, you can save on the cost with lower uptime needed for the cluster to run the queries. If you need to debug, you will also have access to a Query Profile. We recommend using a serverless cluster if you want to minimize the time spent on spinning up a cluster and remove the need to change cluster sizes depending on workflows. If you use a Databricks serverless SQL warehouse, you still need to choose a [cluster size](https://docs.databricks.com/aws/en/compute/sql-warehouse/create#configure-sql-warehouse-settings) (for example, 2X-Small, X-Small, Small, Medium, Large). For more information on serverless SQL warehouses, see the [Databricks docs](https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior#sizing-a-serverless-sql-warehouse). Let’s [create a Databricks SQL warehouse](https://docs.databricks.com/sql/admin/sql-endpoints.html#create-a-sql-warehouse): 1. Click **SQL Warehouses** in the sidebar. 2. Click **Create SQL Warehouse**. 3. Enter a name for the warehouse. 4. If using a serverless SQL warehouse, select a [cluster size](https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior#sizing-a-serverless-sql-warehouse) (2X-Small through 4X-Large) or leave the default, but ensure it suits your workload. 5. Accept the default warehouse settings or edit them. 6. Click **Create**. 7. Configure warehouse permissions to ensure our service principal and developer have the right access. We are not covering python in this post but if you want to learn more, check out these [docs](https://docs.getdbt.com/docs/build/python-models.md#specific-data-platforms). Depending on your workload, you may wish to create a larger SQL Warehouse for production workflows while having a smaller development SQL Warehouse (if you’re not using Serverless SQL Warehouses). As your project grows, you might want to apply [compute per model configurations](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#specifying-the-compute-for-models). #### Configure your dbt project Now that the Databricks components are in place, we can configure our dbt project. This involves connecting dbt to our Databricks SQL warehouse to run SQL queries and using a version control system like GitHub to store our transformation code. If you are migrating an existing dbt project from the dbt-spark adapter to dbt-databricks, follow this [migration guide](https://docs.getdbt.com/guides/migrate-from-spark-to-databricks.md) to switch adapters without needing to update developer credentials and other existing configs. If you’re starting a new dbt project, follow the steps below. For a more detailed setup flow, check out our [quickstart guide.](https://docs.getdbt.com/guides/databricks.md) ##### Connect dbt to Databricks First, you’ll need to connect your dbt project to Databricks so it can send transformation instructions and build objects in Unity Catalog. Follow the instructions for [dbt](https://docs.getdbt.com/guides/databricks.md?step=4) or [dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) to configure your project’s connection credentials. Each developer must generate their Databricks PAT and use the token in their user credentials. They will also specify a unique developer schema that will store the tables and views generated by dbt runs executed from their Studio IDE. This provides isolated developer environments and ensures data access is fit for purpose. Let’s generate a [Databricks personal access token (PAT)](https://docs.databricks.com/sql/user/security/personal-access-tokens.html) for Development: 1. In Databricks, click on your Databricks username in the top bar and select User Settings in the drop down. 2. On the Access token tab, click Generate new token. 3. Click Generate. 4. Copy the displayed token and click Done. (don’t lose it!) For your user credentials/profiles.yml: 1. Set your default catalog to dev. 2. Your developer schema should be named after yourself. We recommend dbt\_\<first\_name\_initial>\<last\_name>. During your first invocation of `dbt run`, dbt will create the developer schema if it doesn't already exist in the dev catalog. #### Defining your dbt deployment environment We need to give dbt a way to deploy code outside of development environments. To do so, we’ll use dbt [environments](https://docs.getdbt.com/docs/environments-in-dbt.md) to define the production targets that end users will interact with. dbt v1 projects can use [targets in profiles](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-targets-in-profiles) to separate environments. [dbt environments](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) allow you to define environments via the UI and [schedule jobs](https://docs.getdbt.com/guides/databricks.md#create-and-run-a-job) for specific environments. Let’s set up our deployment environment: 1. Follow the Databricks instructions to [set up your service principal’s token](https://docs.databricks.com/dev-tools/service-principals.html#use-curl-or-postman). Note that the `lifetime_seconds` will define how long this credential stays valid. You should use a large number here to avoid regenerating tokens frequently and production job failures. 2. Now let’s pop back over to dbt to fill out the environment fields. Click on environments in the dbt UI or define a new target in your profiles.yml. 3. Set the Production environment’s *catalog* to the **prod** catalog created above. Provide the [service token](https://docs.databricks.com/administration-guide/users-groups/service-principals.html#manage-access-tokens-for-a-service-principal) for your **prod** service principal and set that as the *token* in your production environment’s deployment credentials. 4. Set the schema to the default for your prod environment. This can be overridden by [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md#what-is-a-custom-schema) if you need to use more than one. 5. Provide your Service Principal token. #### Connect dbt to your git repository Next, you’ll need somewhere to store and version control your code that allows you to collaborate with teammates. Connect your dbt project to a git repository with [dbt](https://docs.getdbt.com/guides/databricks.md#set-up-a-dbt-cloud-managed-repository). [dbt v1](https://docs.getdbt.com/guides/manual-install.md#create-a-repository) projects will use the git CLI. ##### Next steps Now that your project is configured, you can start transforming your Databricks data with dbt. To help you scale efficiently, we recommend you follow our best practices, starting with the [Unity Catalog best practices](https://docs.getdbt.com/best-practices/dbt-unity-catalog-best-practices.md), then you can [Optimize dbt models on Databricks](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md). --- ### Trigger PagerDuty alarms when dbt jobs fail [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction This guide will teach you how to build and host a basic Python app which will monitor dbt jobs and create PagerDuty alarms based on failure. To do this, when a dbt job completes it will: * Check for any failed nodes (e.g. non-passing tests or errored models), and * create a PagerDuty alarm based on those nodes by calling the PagerDuty Events API. Events are deduplicated per run ID. [![Screenshot of the PagerDuty UI, showing an alarm created by invalid SQL in a dbt model](/img/guides/orchestration/webhooks/serverless-pagerduty/pagerduty-example-alarm.png?v=2 "Screenshot of the PagerDuty UI, showing an alarm created by invalid SQL in a dbt model")](#)Screenshot of the PagerDuty UI, showing an alarm created by invalid SQL in a dbt model In this example, we will use fly.io for hosting/running the service. fly.io is a platform for running full stack apps without provisioning servers etc. This level of usage should comfortably fit inside of the Free tier. You can also use an alternative tool such as [AWS Lambda](https://ademoverflow.com/en/posts/tutorial-fastapi-aws-lambda-serverless/) or [Google Cloud Run](https://github.com/sekR4/FastAPI-on-Google-Cloud-Run). ##### Prerequisites This guide assumes some familiarity with: * [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * CLI apps * Deploying code to a serverless code runner like fly.io or AWS Lambda #### Clone the `dbt-cloud-webhooks-pagerduty` repo [This repository](https://github.com/dpguthrie/dbt-cloud-webhooks-pagerduty) contains the sample code for validating a webhook and creating events in PagerDuty. #### Install `flyctl` and sign up for fly.io Follow the directions for your OS in the [fly.io docs](https://fly.io/docs/hands-on/install-flyctl/), then from your command line, run the following commands: Switch to the directory containing the repo you cloned in step 1: ```shell #example: replace with your actual path cd ~/Documents/GitHub/dbt-cloud-webhooks-pagerduty ``` Sign up for fly.io: ```shell flyctl auth signup ``` Your console should show `successfully logged in as YOUR_EMAIL` when you're done, but if it doesn't then sign in to fly.io from your command line: ```shell flyctl auth login ``` #### Launch your fly.io app Launching your app publishes it to the web and makes it ready to catch webhook events: ```shell flyctl launch ``` You will see a message saying that an existing `fly.toml` file was found. Type `y` to copy its configuration to your new app. Choose an app name of your choosing, such as `YOUR_COMPANY-dbt-cloud-webhook-pagerduty`, or leave blank and one will be generated for you. Note that your name can only contain numbers, lowercase letters and dashes. Choose a deployment region, and take note of the hostname that is generated (normally `APP_NAME.fly.dev`). When asked if you would like to set up Postgresql or Redis databases, type `n` for each. Type `y` when asked if you would like to deploy now. Sample output from the setup wizard: \`\` `joel@Joel-Labes dbt-cloud-webhooks-pagerduty % flyctl launch An existing fly.toml file was found for app dbt-cloud-webhooks-pagerduty ? Would you like to copy its configuration to the new app? Yes Creating app in /Users/joel/Documents/GitHub/dbt-cloud-webhooks-pagerduty Scanning source code Detected a Dockerfile app ? Choose an app name (leave blank to generate one): demo-dbt-cloud-webhook-pagerduty automatically selected personal organization: Joel Labes Some regions require a paid plan (fra, maa). See https://fly.io/plans to set up a plan. ? Choose a region for deployment: [Use arrows to move, type to filter] ? Choose a region for deployment: Sydney, Australia (syd) Created app dbtlabs-dbt-cloud-webhook-pagerduty in organization personal Admin URL: https://fly.io/apps/demo-dbt-cloud-webhook-pagerduty Hostname: demo-dbt-cloud-webhook-pagerduty.fly.dev ? Would you like to set up a Postgresql database now? No ? Would you like to set up an Upstash Redis database now? No Wrote config file fly.toml ? Would you like to deploy now? Yes` #### Create a PagerDuty integration application See [PagerDuty's guide](https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgw-events-api-v2-overview#getting-started) for full instructions. Make note of the integration key for later. #### Configure a new webhook in dbt See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Your event should be **Run completed**. Set the webhook URL to the host name you created earlier (`APP_NAME.fly.dev`) Make note of the Webhook Secret Key for later. *Do not test the endpoint*; it won't work until you have stored the auth keys (next step) #### Store secrets The application requires three secrets to be set, using these names: * `DBT_CLOUD_SERVICE_TOKEN`: a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with at least the `Metadata Only` permission. * `DBT_CLOUD_AUTH_TOKEN`: the Secret Key for the dbt webhook you created earlier. * `PD_ROUTING_KEY`: the integration key for the PagerDuty integration you created earlier. Set these secrets as follows, replacing `abc123` etc with actual values: ```shell flyctl secrets set DBT_CLOUD_SERVICE_TOKEN=abc123 DBT_CLOUD_AUTH_TOKEN=def456 PD_ROUTING_KEY=ghi789 ``` #### Deploy your app After you set your secrets, fly.io will redeploy your application. When it has completed successfully, go back to the dbt webhook settings and click **Test Endpoint**. --- ### Upgrade to dbt v2 part 1: Preparing to upgrade This guide helps you prepare for an in-place upgrade from v1 to v2 in the dbt platform. [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction dbt v2 represents the next evolution of data transformation. dbt has been rebuilt from the ground up but at its most basic, dbt v2 is a new version, and like any new version you should take steps to prepare to upgrade. This guide will take you through those preparations. If dbt v2 is brand new to you, take a look at our [comprehensive documentation](https://docs.getdbt.com/docs/introduction.md) on what it is, how it behaves, and what's different from dbt v1 before getting started with this guide. Once you're caught up, it's time to begin preparing your projects for the speed and power that dbt v2 has to offer. #### Prerequisites This guide will cover the preparations for upgrading to dbt v2 and is intended for customers already using the dbt platform with a version of dbt v1. If you're brand new to dbt, check out our [quickstart guides](https://docs.getdbt.com/guides.md). To follow the steps in this guide, you must meet the following prerequisites: * You're using a dbt platform account on any tier. * You have a developer license. * You have [proper permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to edit projects. * Your project is using a [supported adapter](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md?version=2.0)  BigQuery * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks * Service Account / User Token * Native OAuth  Redshift * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  ClickHouse[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password Upgrading your first project Start with smaller, newer, or more familiar projects first. This makes it easier to identify and troubleshoot any issues before upgrading larger, more complex projects. #### Enable dbt v2 readiness features The dbt v2 readiness panel in the dbt platform and shows each project's eligibility status and blockers, is being rolled out in phases. If it hasn't been automatically enabled for your account yet, an [account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) can manually enable it. This lets admins and developers see which projects and jobs are eligible for dbt v2, identify blockers, and initiate the upgrade once preparation is complete. ##### Step 1: Enable the readiness toggle This step requires account admin access in dbt platform: 1. Click your account name in the left sidebar and select **Account settings**. 2. Navigate to the **Account** screen and click **Edit**. 3. Scroll to the **Settings** section and select the checkbox next to **Enable dbt v2 readiness & upgrade features**. 4. Click **Save**. Once enabled: * All admins and developers can see each project's dbt v2 readiness status, including which jobs are eligible or ineligible for dbt v2 and why. * Admins can initiate the dbt v2 upgrade from development environments, environment settings, and job settings (subject to existing permissions). ##### Step 2: Restrict upgrade access (optional) By default, any user who can see the upgrade assistant can use it to initiate an upgrade. To limit upgrade execution to designated users, you can enable an additional access control toggle. Enterprise tier accounts only The **Enable restricted dbt v2 upgrade permissions** toggle is only available to Enterprise/Enterprise+ accounts that have been granted this entitlement. Contact your account manager if you need this capability. 1. In the same **Account** settings screen, select the checkbox next to **Enable restricted dbt v2 upgrade permissions**. 2. Click **Save**. When enabled, only users assigned the [`v2 Migration Admin`](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin) permission set (scoped to specific projects) can execute the upgrade. For instructions on assigning this permission, refer to [Assign upgrade access](https://docs.getdbt.com/guides/upgrade-to-v2.md?step=3#assign-upgrade-access-optional) in Part 2 of this guide. ##### The dbt v2 readiness panel With the readiness experience enabled, you can monitor your project's eligibility as you work through the preparation steps below. The panel shows which jobs are eligible or ineligible for dbt v2 and why. [![The dbt v2 readiness checklist](/img/v2/v2-readiness.png?v=2 "The dbt v2 readiness checklist")](#)The dbt v2 readiness checklist Common ineligibility reasons include: * Environment(s) not on the **v1 Latest** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md#which-release-tracks-are-available) * Not using a [supported data platform](https://docs.getdbt.com/docs/dbt/supported-features.md?version=2.0#requirements) * Project doesn't have at least one successful job run * Jobs that haven't run in the last 7 days or have recent failures As you complete the steps in this guide, check the readiness panel to see your eligibility improve. #### Upgrade to the latest dbt v1 version Before upgrading to dbt v2, you need to move your environments to the **v1 Latest** [dbt v1 release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). The **v1 Latest** track includes all the features and tooling to help you prepare for dbt v2. It ensures the smoothest upgrade experience by validating that your project doesn't rely on deprecated behaviors. Test before you deploy Always test version upgrades in development first. Use the [Override dbt version](#step-1-test-in-development-using-override) feature to safely try the **v1 Latest** release track without affecting your team or production runs. ##### Step 1: Test in development (using override) Test the **v1 Latest** release track for your individual account without changing the environment for your entire team: 1. Click your account name in the left sidebar and select **Account settings**. 2. Select **Credentials** from the sidebar and choose your project. 3. In the side panel, click **Edit** and scroll to **User development settings**. 4. Select **v1 Latest** from the **dbt version** dropdown and click **Save**. [![Override dbt version in your account settings](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-override-version.png?v=2 "Override dbt version in your account settings")](#)Override dbt version in your account settings 5. Launch the Studio IDE or dbt platform CLI and test your normal development workflows. 6. Verify the override is active by running any dbt command and checking the **System Logs**. The first line should show `Running with dbt=` and your selected version. If the version number is `v1.11` or higher, you're on the right path to dbt v2 readiness. If everything works as expected, proceed to the next step to start upgrading your environments. If you encounter deprecation warnings, don't fear! We'll address those [later in this guide](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=4). If you encounter errors, revert to your previous version and refer to the [version upgrade guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) to resolve any differences between your current version and the latest available dbt v1 version. ##### Step 2: Upgrade your development environment After successfully testing your individual development environment with the override, upgrade the development environment for the entire project (be sure to give your team notice!): 1. Navigate to **Environments** in your project settings. 2. Select your **Development** environment and click **Edit**. 3. Click the **dbt version** dropdown and select **v1 Latest**. 4. Click **Save** to apply the changes. [![Upgrade development environment to \*\*v1 Latest\*\* release track](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/select-development.png?v=2 "Upgrade development environment to **v1 Latest** release track")](#)Upgrade development environment to \*\*v1 Latest\*\* release track Remove your override Once your development environment is upgraded, you can remove your personal override by returning to your account credentials and selecting the same version as your environment. ##### Step 3: Upgrade staging and pre-production If your organization has staging or pre-production environments, upgrade these before production: 1. Navigate to **Environments** and select your staging/pre-production environment. 2. Click **Edit** and select **v1 Latest** from the **dbt version** dropdown. 3. Click **Save**. 4. Run your jobs in this environment for a few days to validate everything works correctly. This provides a final validation layer before upgrading production environments. ##### Step 4: Upgrade your production environment After validating in staging (or development if you don't have staging), upgrade your production environment: 1. Navigate to **Environments** and select your **Production** environment. 2. Click **Edit** and select **v1 Latest** from the **dbt version** dropdown. 3. Click **Save** to apply the changes. 4. Monitor your first few production runs to ensure everything executes successfully. ##### Step 5: Update jobs While environments control the dbt version for most scenarios, some older job configurations may have version overrides. Review your jobs and [update any that specify a dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#jobs) to ensure they use the environment's v1 Latest release track. #### Resolve all deprecation warnings dbt v2 enforces strict validation and won't accept deprecated code that currently generates warnings in dbt v1. You must resolve all deprecation warnings before upgrading to dbt v2. Fortunately, the autofix tool in the Studio IDE can automatically resolve most common deprecations for you. VS Code extension This guide provides steps to resolve deprecation warnings without leaving dbt platform. If you prefer to work in the VS Code or Cursor editors locally, you can run the autofix in our dbt VS Code extension. Check out the [installation guide](https://docs.getdbt.com/docs/install-dbt-extension.md) for more information about those workflows. ##### What the autofix tool handles The autofix tool can resolve many deprecations automatically, including: * Moving custom configurations into the `meta` dictionary * Fixing duplicate YAML keys * Correcting unrecognized resource properties * Updating deprecated configuration patterns Check out the [autofix readme](https://github.com/dbt-labs/dbt-autofix/) for a complete list of the deprecations it addresses. dbt v2 package compatibility In addition to deprecations, the autofix tool attempts to upgrade packages to the lowest supported dbt v2-compatible version. Check out [package support](https://docs.getdbt.com/docs/dbt/supported-features.md#package-support) for more information about dbt v2 compatibility. ##### Step 1: Create a new branch Before running the autofix tool, create a new branch to isolate your changes: 1. Navigate to the Studio IDE by clicking **Studio** in the left-side menu. 2. Click the **Version control** panel (git branch icon) on the left sidebar. 3. Click **Create branch** and name it something descriptive like `fusion-deprecation-fixes`. 4. Click **Create** to switch to your new branch. Save before autofixing The autofix tool will modify files in your project. Make sure to commit or stash any unsaved work to avoid losing changes. ##### Step 2: Run the autofix tool Now you're ready to scan for and automatically fix deprecation warnings: 1. Click the **three-dot menu** in the bottom right corner of the Studio IDE. 2. Select **Check & fix deprecations**. [![Access the Studio IDE options menu](/img/docs/dbt-platform/platform-ide/ide-options-menu-with-save.png?v=2 "Access the Studio IDE options menu")](#)Access the Studio IDE options menu The tool runs `dbt parse --show-all-deprecations --no-partial-parse` to identify all deprecations in your project. This may take a few moments depending on your project size. 3. When parsing completes, view the results in the **Command history** panel in the bottom left. [![View command history and deprecation results](/img/docs/dbt-platform/platform-ide/command-history.png?v=2 "View command history and deprecation results")](#)View command history and deprecation results ##### Step 3: Review and apply autofixes After the deprecation scan completes, review the findings and apply automatic fixes: 1. In the **Command history** panel, review the list of deprecation warnings. 2. Click the **Autofix warnings** button to proceed. [![Click Autofix warnings to resolve deprecations automatically](/img/docs/dbt-platform/platform-ide/autofix-button.png?v=2 "Click Autofix warnings to resolve deprecations automatically")](#)Click Autofix warnings to resolve deprecations automatically 3. In the **Proceed with autofix** dialog, review the warning and click **Continue**. [![Confirm autofix operation](/img/docs/dbt-platform/platform-ide/proceed-with-autofix.png?v=2 "Confirm autofix operation")](#)Confirm autofix operation The tool automatically modifies your project files to resolve fixable deprecations, then runs another parse to identify any remaining warnings. 4. When complete, a success message appears. Click **Review changes**. [![Autofix complete](/img/docs/dbt-platform/platform-ide/autofix-success.png?v=2 "Autofix complete")](#)Autofix complete ##### Step 4: Verify the changes Review the changes made by the autofix tool to ensure they're correct: 1. Open the **Version control** panel to view all modified files. 2. Click on individual files to review the specific changes. 3. Look for files with moved configurations, corrected properties, or updated syntax. 4. If needed, make any additional manual adjustments. ##### Step 5: Commit your changes Once you're satisfied with the autofix changes, commit them to your branch: 1. In the **Version control** panel, add a descriptive commit message like "Fix deprecation warnings for dbt v2 upgrade". 2. Click **Commit and sync** to save your changes. ##### Step 6: Address remaining deprecations If the autofix tool reports remaining deprecation warnings that couldn't be automatically fixed: 1. Review the warning messages in the **Command history** panel. Each warning includes the file path and line number. 2. Manually update the code based on the deprecation guidance: * Custom inputs should be moved to the `meta` config. * Deprecated properties should be updated to their new equivalents. * Refer to specific [version upgrade guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) for detailed migration instructions. 3. After making manual fixes, run **Check & fix deprecations** again to verify all warnings are resolved. 4. Commit your changes. ##### Step 7: Merge to your main branch Once all deprecations are resolved: 1. Create a pull request in your git provider to merge your deprecation fixes. 2. Have your team review the changes. 3. Merge the PR to your main development branch. 4. Ensure these changes are deployed to your environments before proceeding with the dbt v2 upgrade. #### Validate and upgrade your dbt packages Run autofix first This section contains instructions for manual package upgrades. We recommend running the autofix tool before taking these steps. The autofix tool finds packages incompatible with dbt v2 and upgrades them to the lowest compatible version. For more information, check out [package support](https://docs.getdbt.com/docs/dbt/supported-features.md#package-support). dbt packages extend your project's functionality, but they must be compatible with dbt v2. Most commonly used packages from dbt Labs (like `dbt_utils` and `dbt_project_evaluator`) and many community packages [already support dbt v2](https://docs.getdbt.com/docs/dbt/supported-features.md#package-support). Before upgrading, verify your packages are compatible and upgrade them to the latest versions. Check for packages that support version 2.0.0, or ask the maintainer if you're unsure. What if a package isn't compatible? If a critical package isn't yet compatible with dbt v2: * Check with the package maintainer about their roadmap. * Open an issue requesting dbt v2 support. * Consider contributing the compatibility updates yourself. * Try it out anyway! The incompatible portion of the package might not impact your project. ###### Package compatibility messages Inconsistent v2 warnings and `dbt-autofix` logs dbt v2 warnings and `dbt-autofix` logs may show different messages about package compatibility. If you use [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) while upgrading to v2 in the Studio IDE or dbt VS Code extension, you may see different messages about package compatibility between `dbt-autofix` and v2 warnings. Here's why: * dbt v2 warnings are emitted based on a package's `require-dbt-version` and whether `require-dbt-version` contains `2.0.0`. * Some packages are already v2-compatible even though package maintainers haven't yet updated `require-dbt-version`. * `dbt-autofix` knows about these compatible packages and will not try to upgrade a package that it knows is already compatible. This means that even if you see a v2 warning for a package that `dbt-autofix` identifies as compatible, you don't need to change the package. The message discrepancy is temporary while we implement and roll out `dbt-autofix`'s enhanced compatibility detection to v2 warnings. Here's an example of a v2 warning in the Studio IDE that says a package isn't compatible with v2 but `dbt-autofix` indicates it is compatible: ```text dbt1065: Package 'dbt_utils' requires dbt version [>=1.30,<2.0.0], but current version is 2.0.0. This package may not be compatible with your dbt version. dbt(1065) [Ln 1, Col 1] ``` ##### Step 1: Review your current packages Identify which packages your project uses: 1. In the Studio IDE, open your project's root directory. 2. Look for either `packages.yml` or `dependencies.yml` file. 3. Review the list of packages and their current versions. Your file will look something like this: ```yaml packages: - package: dbt-labs/dbt_utils version: 1.0.0 - package: dbt-labs/codegen version: 0.9.0 ``` ##### Step 2: Check compatibility and find the latest package versions Review [the dbt package hub](https://hub.getdbt.com) to see verified dbt v2-compatible packages by checking that the `require-dbt-version` configuration includes `2.0.0` or higher. Refer to [package support](https://docs.getdbt.com/docs/dbt/supported-features.md#package-support) for more information. For packages that aren't dbt v2-compatible: * Visit the package's GitHub repository. * Check the README or recent releases for dbt v2 compatibility information. * Look for issues or discussions about dbt v2 support. For each package, find the most recent version: * Visit [dbt Hub](https://hub.getdbt.com) for packages hosted there. * For packages from GitHub, check the repository's releases page. * Note the latest version number for each package you use. For Hub packages, you can use version ranges to stay up-to-date: ```yaml packages: - package: dbt-labs/dbt_utils version: [">=1.0.0", "<3.0.0"] # Gets latest 1.x or 2.x version ``` ##### Step 3: Update your package versions Update your `packages.yml` or `dependencies.yml` file with the latest compatible versions: 1. In the Studio IDE, open your `packages.yml` or `dependencies.yml` file. 2. Update each package version to the latest compatible version. 3. Save the file. Before update: ```yaml packages: - package: dbt-labs/dbt_utils version: 0.9.6 - package: dbt-labs/codegen version: 0.9.0 ``` After update: ```yaml packages: - package: dbt-labs/dbt_utils version: [">=1.0.0", "<2.0.0"] - package: dbt-labs/codegen version: [">=0.12.0", "<1.0.0"] ``` ##### Step 4: Install updated packages After updating your package versions, install them: 1. In the Studio IDE command line, run: ```bash dbt deps --upgrade ``` The `--upgrade` flag ensures dbt installs the latest versions within your specified ranges, updating the `package-lock.yml` file. 2. Review the output to confirm all packages installed successfully. 3. Check that the `package-lock.yml` file was updated with the new package versions. About package-lock.yml The `package-lock.yml` file pins your packages to specific versions for reproducible builds. We recommend committing this file to version control so your entire team uses the same package versions. ##### Step 5: Test your project with updated packages After upgrading packages, test your project to ensure everything works: 1. Run a subset of your models to verify basic functionality: ```bash dbt run --select tag:daily ``` 2. Run your tests to catch any breaking changes (exact command may vary): ```bash dbt test ``` 3. If you encounter issues: * Review the package's changelog for breaking changes * Adjust your code to match new package behavior * If problems persist, temporarily pin to an older compatible version (if possible) ##### Step 6: Commit package updates Once you've verified the updated packages work correctly: 1. In the **Version control** panel, stage your changes: * `packages.yml` or `dependencies.yml` * `package-lock.yml` 2. Add a commit message like "Upgrade dbt packages for dbt v2 compatibility". 3. Click **Commit and sync**. #### Update your jobs Use the dbt v2 readiness panel to validate your jobs against dbt v2 and fix any issues you find. For jobs that are eligible for dbt v2 in the readiness experience, **Run once on dbt v2** is replaced by a **Debug on dbt v2** dropdown in the eligibility banner or modal. Use a debug option when you want to inspect or fix issues interactively in Studio IDE. Use **Run once on dbt v2** when you want to validate execution behavior in a deployment context. ##### Review your jobs The readiness panel shows how many jobs are ineligible for dbt v2 or have an unknown eligibility status. If you don't see eligibility messaging, ask an account admin to enable **dbt v2 readiness & upgrade features** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md). For setup steps, refer to [Enable dbt v2 readiness features](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=3). note If a job has not run in the last 7 days, you must run it once for the debugging options to be available. 1. Open the jobs list using either path: * From the main menu, go to **Orchestration** → **Jobs**. * From the readiness panel, click **Review jobs**. [![Shortcut to review your jobs from the readiness panel](/img/v2/review-jobs.png?v=2 "Shortcut to review your jobs from the readiness panel")](#)Shortcut to review your jobs from the readiness panel 2. Find the dbt v2 eligibility icon to the right of your jobs. Click **Review job** for any job that is ineligible or has an unknown eligibility status. [![Take action on your jobs to make them v2 eligible.](/img/v2/job-eligibility.png?v=2 "Take action on your jobs to make them v2 eligible.")](#)Take action on your jobs to make them v2 eligible. 3. Click **Debug on dbt v2** and choose one of the following: * [Debug in Studio](#debug-in-studio) * [Debug in Studio with dbt Wizard](#debug-in-studio-with-dbt-wizard) * [Run once on dbt v2](#run-once-on-fusion) ###### Debug in Studio In the Studio IDE, run dbt v2 in your development environment to review project warnings and errors: 1. Click **Debug in Studio**. dbt sets your user-level `DBT_DEVELOP_CORE_VERSION` environment variable to `v2-stable`, then opens the Studio IDE with the **Problems** tab selected. [![Running v2 in development](/img/v2/v2-ide.png?v=2 "Running v2 in development")](#)Running v2 in development 2. Review the warnings or errors in the **Problems** tab. 3. Fix the issues directly or run the [autofix tool](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md). 4. When the project runs with no warnings or errors, commit and publish your changes. 5. After you merge the changes, wait for the job to run again or run it manually. To revert the `latest-fusion` override, use the dbt version control in Studio IDE or update the **dbt version** under **User development settings** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md) → **Credentials**. For more details, refer to [Override dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#override-dbt-version). ###### Debug in Studio with dbt Wizard [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") If you have access to [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) with [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled, you can use the [dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) skill. This skill can help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed. It classifies every error, applies validated fixes automatically, and surfaces what's blocked. info The dbt v2 migration workflow is accessible through the dbt Wizard in the Studio IDE. If you're using VS Code or the dbt platform CLI, use the [autofix tool](https://docs.getdbt.com/guides/dbt-package-compat?step=4) instead. 1. From the job list, click the **Review job** button for a job with a successful run. * If you don't see the **Review job** button, enable the **Show dbt v2 eligibility** toggle in the job list. 2. In the **dbt v2 eligibility unknown for this job** pop-up, click **Debug in Studio with dbt Wizard**. 3. dbt redirects you to the Studio IDE and sets your personal development environment to dbt v2. 4. dbt Wizard opens and automatically triggers the dbt v2 migration skill with this prompt: ```text I need help fixing dbt v2 compatibility issues in this project. Please investigate and resolve any deprecation warnings or incompatibilities. Please use the migrating-dbt-core-to-fusion skill to guide this. ``` 5. Review and approve dbt Wizard's permission requests so it can run the commands it needs. 6. The dbt Wizard iteratively runs `dbt compile`, reads the results, and applies fixes until it reaches a successful compile or encounters an error it can't resolve. If it gets blocked, it exits cleanly, explains what it could not fix, and creates and links to a markdown file summarizing all changes made. 7. When the project compiles with no warnings or errors, commit and publish your changes. 8. After you merge the changes, wait for the job to run again or run it manually on dbt v2. [![The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.](/img/docs/dbt-platform/v2-migration-workflow.gif?v=2 "The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.")](#)The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE. ###### Run once on dbt v2 When you are confident a job is ready for dbt v2, you can run it once on the engine without changing environment-wide settings. **Run once on dbt v2** can be temporarily unavailable while a dbt v2 run request is already pending. 1. Click **Run once on dbt v2**. 2. The job window opens and dbt runs the job on dbt v2 without changing other jobs or environment settings. 3. When the job succeeds, click **Override eligibility status** to update the eligibility status. [![Override the eligibility status of a successful job.](/img/v2/eligibility-status.png?v=2 "Override the eligibility status of a successful job.")](#)Override the eligibility status of a successful job. Congratulations! You have validated dbt v2 eligibility for your jobs. [![Your job is now ready for v2!](/img/v2/v2-eligible.png?v=2 "Your job is now ready for v2!")](#)Your job is now ready for v2! #### What's next? With limitations identified and addressed, you've completed all the preparation steps. Your project is now ready to upgrade to dbt v2! Check out [Part 2: Making the move](https://docs.getdbt.com/guides/upgrade-to-v2.md) --- ### Upgrade to dbt v2 part 2: Making the move This guide helps you implement an in-place upgrade from the latest version of dbt v1 to dbt v2 in the dbt platform. [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction dbt v2 represents the next evolution of data transformation. dbt has been rebuilt from the ground up but at its most basic, dbt v2 is a new version, and moving to it is the same as upgrading between dbt v1 versions in the dbt platform. Once your project is dbt v2 ready, it's only a matter of pulling a few levers to make the move, but you have some flexibility in how you do so, especially in your development environments. Once you complete the dbt v2 migration, your team will benefit from: * ⚡ Up to 30x faster parsing and compilation * 💰 30%+ reduction in warehouse costs (with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)) * 🔍 Enhanced SQL validation and error messages * 🛠️ Modern development tools #### Prerequisites Before upgrading your development environment, confirm: * Your project is on the **v1 Latest** release track (completed in [Part 1: Preparing to upgrade](https://docs.getdbt.com/guides/prepare-v2-upgrade.md)) * Your project must be using a supported adapter and auth method.  BigQuery * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks * Service Account / User Token * Native OAuth  Redshift * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  ClickHouse[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password * You have a developer license in dbt platform * dbt v2 has been enabled for your account * You have appropriate permissions to modify environments (see [Assign upgrade access](https://docs.getdbt.com/guides/upgrade-to-v2?step=3#assign-upgrade-access-optional) if restricted) #### Upgrade your development environment With your project prepared and tested on the **v1 Latest** release track, you're ready to upgrade your development environment to dbt v2. The dbt platform provides a guided upgrade assistant that walks you through the process and helps validate your project is dbt v2 ready. Start with development Always upgrade your development environment first before moving to production. This lets you and your team test dbt v2 in a safe environment and address any issues before they affect production workflows. ##### Assign upgrade access (optional) The dbt v2 upgrade assistant is controlled by two account-level settings. An [account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) must first enable the readiness experience, and can optionally restrict which users can execute the upgrade. ###### Enable the dbt v2 readiness experience The upgrade assistant and readiness panel only appear after an account admin enables this setting: 1. Navigate to **Account settings** → **Account**. 2. Click **Edit** and scroll to the **Settings** section. 3. Select the checkbox next to **Enable dbt v2 readiness & upgrade features**. 4. Click **Save**. Once enabled, all users can see the readiness panel and the **Start dbt v2 upgrade** assistant (subject to their existing permissions). ###### Restrict who can execute upgrades (optional, Enterprise only) By default, any user who can see the upgrade assistant can use it. To restrict upgrade execution to designated users: 1. In **Account settings** → **Account**, click **Edit**. 2. Select the checkbox next to **Enable restricted dbt v2 upgrade permissions**. 3. Click **Save**. When this is enabled, only users with the **v2 Migration Admin** [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin) can execute upgrades. To assign this permission: 1. Navigate to **Account settings** → **Groups** and choose the group to grant access. 2. Click **Edit** and scroll to **Access and permissions**. 3. Click **Add permission** and select **v2 Migration Admin** from the dropdown. 4. Select the project(s) users should access. 5. Click **Save**. For more details on access control, see [Assign access to upgrade](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#assign-access-to-upgrade). ##### Step 1: Start the upgrade assistant Launch the dbt v2 upgrade workflow from your project: 1. Log into dbt platform and navigate to your project. 2. From the project homepage or sidebar, click **Start dbt v2 upgrade** or **Get started**. [![Start the dbt v2 upgrade from the project homepage](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/start-upgrade.png?v=2 "Start the dbt v2 upgrade from the project homepage")](#)Start the dbt v2 upgrade from the project homepage You'll be redirected to the Studio IDE with the upgrade assistant visible at the top. ##### Step 2: Check for deprecation warnings Even if you resolved deprecations in Part 1, run a final check to ensure nothing was missed: 1. At the top of the Studio IDE, click **Check deprecation warnings**. [![Check for deprecation warnings in your project](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/check-deprecations.png?v=2 "Check for deprecation warnings in your project")](#)Check for deprecation warnings in your project 2. Wait for the parse to complete (this may take a few moments depending on project size). 3. Review the results: * **No warnings found**: Skip to Step 4 to continue upgrading. * **Warnings found**: Continue to Step 3 to resolve them. Inconsistent dbt v2 warnings and `dbt-autofix` logs You may see dbt v2 deprecation warnings about packages not being compatible with dbt v2, while `dbt autofix` indicates they are compatible. Use `dbt autofix` as the source of truth because it has additional context that dbt v2 warnings don't have yet. This conflict is temporary and will be resolved as soon as we implement and roll out `dbt-autofix`'s enhanced compatibility detection to dbt v2 warnings. ##### Step 3: Resolve remaining deprecations If you find deprecation warnings, use the autofix tool to resolve them: 1. In the deprecation warnings list, click **Autofix warnings**. 2. Review the proposed changes in the dialog. 3. Click **Continue** to apply the fixes automatically. 4. Wait for the autofix tool to complete and run a follow-up parse. 5. Review the modified files in the **Version control** panel. 6. If all warnings are resolved, you'll see a success message. [![Success message when deprecations are resolved](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/autofix-success.png?v=2 "Success message when deprecations are resolved")](#)Success message when deprecations are resolved For detailed information about the autofix process, see [Fix deprecation warnings](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md). Manual fixes required? If the autofix tool can't resolve all deprecations automatically, you'll need to fix them manually. Review the warning messages for specific guidance, make the necessary changes in your code, then run **Check deprecation warnings** again. ##### Step 4: Enable dbt v2 After you resolve all deprecations, upgrade your development environment: 1. Click the **Enable dbt v2** button at the top of the Studio IDE. 2. Confirm the upgrade when prompted. 3. Wait for the environment to update (this typically takes just a few seconds). Your development environment is now running on dbt v2! ##### Step 5: Restart the IDE After upgrading, all users need to restart their IDE to connect to the new dbt v2-powered environment: 1. If you're currently in the Studio IDE, refresh your browser window. 2. Notify your team members that they also need to restart their IDEs. ##### Step 6: Verify the upgrade Confirm your development environment is running dbt v2: 1. Open or create a dbt model file in the Studio IDE. 2. Look for dbt v2-powered [features](https://docs.getdbt.com/docs/dbt/supported-features.md#features-and-capabilities): * Faster parsing and compilation times * Enhanced SQL validation and error messages * Improved autocomplete functionality 3. Run a simple command to test functionality: ```bash dbt compile ``` 4. Check the command output for significantly faster performance. ##### Step 7: Test your workflows Before declaring victory, test your typical development workflows: 1. Make changes to a model and compile it by running `dbt compile`. 2. Run a subset of models: `dbt run --select model_name`. 3. Execute tests. 4. Preview results in the integrated query tool. 5. Verify Git operations (commit, push, pull) work as expected. Share feedback If you encounter any unexpected behavior or have feedback about the dbt v2 experience, share it with your account team or [dbt Support](https://docs.getdbt.com/docs/dbt-support.md). ##### What about production? Your development environment is now on dbt v2, but your production environment and deployment jobs are still running on dbt v1. This is intentional as it gives you and your team time to: * Test dbt v2 thoroughly in development. * Build confidence in the new engine. * Identify and resolve any project-specific issues. * Train team members on any workflow changes. When you're ready to upgrade production, you'll update your deployment environments and jobs to use the `v2 Stable` release track. We'll cover that in the next section. #### Upgrade staging and intermediate environments After successfully upgrading and testing your development environment, the next step is upgrading your staging or other intermediate deployment environments. These environments serve as a critical validation layer before promoting dbt v2 to production, allowing you to test with production-like data and workflows while limiting risk. Why upgrade staging first? Staging environments provide: * A final validation layer for dbt v2 with production-scale data * The ability to test scheduled jobs and deployment workflows * An opportunity to verify integrations and downstream dependencies * A safe environment to identify performance characteristics before production ##### What is a staging environment? A [staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) is a deployment environment that mirrors your production setup but uses non-production data or limited access credentials. It enables your team to test deployment workflows, scheduled jobs, and data transformations without affecting production systems. If you don't have a staging environment yet, consider creating one before upgrading production to dbt v2. It provides an invaluable testing ground. ##### Step 1: Navigate to environment settings Access the settings for your staging or intermediate environment: 1. Log into dbt platform and navigate to your project. 2. Click **Orchestration** in the left sidebar. 3. Select **Environments** from the dropdown. 4. Click on your staging environment name to open its settings. 5. Click the **Edit** button in the top right. [![Navigate to environment settings](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Navigate to environment settings")](#)Navigate to environment settings ##### Step 2: Update the dbt version Change your staging environment to use the dbt v2 release track: 1. In the environment settings, scroll to the **dbt version** section. 2. Click the **dbt version** dropdown menu. 3. Select **v2 Stable** from the list. 4. Scroll to the top and click **Save**. [![Select v2 Stable from the dbt version dropdown](/img/docs/dbt-platform/platform-configuring-dbt-platform/platform-upgrading-dbt-versions/upgrade-v2.png?v=2 "Select v2 Stable from the dbt version dropdown")](#)Select v2 Stable from the dbt version dropdown Your staging environment is now configured to use dbt v2! Any jobs associated with this environment will use dbt v2 on their next run. ##### Step 3: Run a test job Validate that dbt v2 works correctly in your staging environment by running a job: 1. From the **Environments** page, click on your staging environment. 2. Select an existing job or click **Create job** to make a new one. 3. Click **Run now** to execute the job immediately. 4. Monitor the job run in real-time by clicking into the run details. ##### Step 4: Monitor scheduled jobs If you have scheduled jobs in your staging environment, monitor their next scheduled runs: 1. Navigate to **Deploy** → **Jobs** and filter to your staging environment. 2. Wait for scheduled jobs to run automatically (or trigger them manually). 3. Review job run history for any unexpected failures or warnings. 4. Compare run times to previous dbt v1 runs. You should see significant improvements. ##### Step 5: Validate integrations and dependencies Test any integrations or dependencies that rely on your staging environment: 1. **Cross-project references**: If using [dbt Mesh](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md), verify downstream projects can still reference your staging models. 2. **BI tools**: Check that any BI tools or dashboards connected to staging still function correctly. 3. **Downstream consumers**: Notify teams that consume staging data to verify their processes still work. 4. **CI/CD workflows**: Run any CI jobs that target staging to ensure they execute properly. Repeat for other intermediate environments Found an issue? If you encounter problems in staging: * Review the [v2 limitations](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations) to see if it's a known issue. * Check job logs for specific error messages. * Test the same models in your development environment to isolate the problem. * Contact [dbt Support](https://docs.getdbt.com/docs/dbt-support.md) or your account team for assistance. You can revert the staging environment to **v1 Latest** release track while investigating. ##### How long should I test in staging? The recommended testing period depends on your organization: * **Minimum**: Run all critical jobs at least once successfully. * **Recommended**: Monitor scheduled jobs for 3-7 days to catch any time-based or data-dependent issues. * **Enterprise/Complex projects**: Consider 1-2 weeks of testing, especially if you have many downstream dependencies. Don't rush this phase. Thorough testing in staging prevents production disruptions. *** #### Upgrade your production environment Congratulations! You've successfully upgraded development and staging environments and you're now ready for the final step: upgrading your production environment to dbt v2. Production environment upgrade considerations Upgrading production is a critical operation. While dbt v2 is production ready and has been thoroughly tested in your dev and staging environments, follow these best practices: * Plan the upgrade during a low-traffic window to minimize impact. * Notify stakeholders about the maintenance window. * Have a rollback plan ready (reverting to **v1 Latest** release track). * Monitor closely for the first few job runs after upgrading. ##### Step 1: Plan your maintenance window Choose an optimal time to upgrade production: * **Review your job schedule:** Identify periods with minimal job activity. * **Check downstream dependencies:** Ensure dependent systems can tolerate brief interruptions. * **Notify stakeholders:** Inform BI tool users, data consumers, and team members. * **Document the plan:** Note which jobs to monitor and success criteria. ##### Step 2: Navigate to production environment settings Access your production environment configuration: 1. Log into dbt platform and navigate to your project. 2. Click **Orchestration** in the left sidebar. 3. Select **Environments** from the dropdown. 4. Click on your production environment (typically marked with a **Production** badge). 5. Click the **Edit** button in the top right. [![Access production environment settings](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Access production environment settings")](#)Access production environment settings ##### Step 3: Upgrade to v2 Stable Update your production environment to use dbt v2: 1. In the environment settings, scroll to the **dbt version** section. 2. Click the **dbt version** dropdown menu. 3. Select **v2 Stable** from the list. 4. Review your settings one final time to ensure everything is correct. 5. Scroll to the top and click **Save**. [![Select v2 Stable for production](/img/docs/dbt-platform/platform-configuring-dbt-platform/platform-upgrading-dbt-versions/upgrade-v2.png?v=2 "Select v2 Stable for production")](#)Select v2 Stable for production Your production environment is now running on dbt v2! ##### Step 4: Run an immediate test job Validate the upgrade by running a job: 1. From the **Environments** page, click on your production environment. 2. Select a critical job that covers a good subset of your models. 3. Click **Run now** to execute the job immediately. 4. Monitor the job run closely: * Check the **parse** and **compile** steps. * Verify all models build successfully. * Confirm tests pass as expected. * Review the logs for any unexpected warnings. If the job succeeds, your production upgrade is successful! Enable the dbt v2 readiness panel The dbt v2 readiness panel shows each project's eligibility status and blockers in the dbt platform. It's rolling out in phases — if it's not enabled for your account yet, an [account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) can turn it on in **Account settings** → **Account**. Refer to [Enable dbt v2 readiness features](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=2) for setup steps. If you have access to dbt Wizard, use the [dbt Wizard's dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) to help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed! ##### Step 5: Enable dbt State (optional but recommended) [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) automatically determines which models need rebuilding based on code or data changes, which can reduce warehouse costs by 30% or more. To get started, see [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). ##### Step 6: Monitor production jobs Watch your production jobs closely for the first 24-48 hours: * **Check scheduled job runs:** Navigate to **Deploy** → **Jobs** → **Run history** * **Monitor run times:** Compare to historical averages. You should see significant improvements. * **Review cost savings**: If you enabled dbt State, check [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) to see savings in action. * **Watch for warnings**: Review logs for any unexpected messages. ##### Step 7: Validate downstream integrations Ensure all systems dependent on your production data still function correctly: 1. **BI tools:** Verify dashboards and reports refresh properly. 2. **Data consumers:** Confirm downstream teams can access and query data. 3. **APIs and integrations:** Test any applications that consume dbt outputs. 4. **Semantic Layer:** If using the [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), verify metrics queries work. 5. **Alerts and monitoring**: Check that data quality alerts and monitors function correctly. ##### Step 8: Update any remaining jobs with version overrides Some jobs might have [version overrides](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#override-dbt-version) set from earlier testing. Now that production is on dbt v2, remove these overrides: 1. Navigate to **Orchestration** → **Jobs**. 2. Review each job's settings. 3. If a job has a version override (showing in the **dbt version** section), click **Edit**. 4. Remove the override to let the job inherit the environment's dbt v2 setting. 5. Click **Save**. ##### Rollback procedure If you encounter critical issues in production, you can revert your dbt version: 1. Navigate to **Orchestration** → **Environments** → **Production**. 2. Click **Edit**. 3. Change **dbt version** from **v2 Stable** back to **v1 Latest**. 4. Click **Save**. 5. Jobs will use dbt v1 on their next run. Rollback impact Rolling back to **v1 Latest** will disable dbt v2-specific features. Only rollback if you're experiencing production-critical issues. #### dbt lint Now that your project is running on dbt v2, you have access to `dbt lint`. The `lint` command is a high-performance SQL linter built into dbt v2. It is SQLFluff-compatible: it reads your existing `.sqlfluff` config file, uses the same rule codes (for example, `CP01`, `RF03`), and respects `-- noqa` suppression comments. Compatible does not mean identical. For parity expectations, refer to [Rule parity with SQLFluff](https://docs.getdbt.com/reference/commands/lint.md?version=2.0#rule-parity-with-sqlfluff). ##### Basic usage ```bash # Lint all SQL files in the project dbt lint # Lint a specific file dbt lint models/my_model.sql # Automatically fix violations dbt lint --fix # Lint only files changed in your working tree dbt lint --changed ``` See the [`dbt lint` reference](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) for the full list of flags, supported dialects, and suppression syntax. #### Next steps 🎉 Congratulations! You've successfully upgraded your entire dbt platform project to dbt v2! For your next steps: * **Optimize further**: Explore [dbt State configurations](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) to fine-tune refresh intervals. * **Monitor savings**: Use [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) to track models built vs. reused. * **Train your team**: Share dbt v2 features and best practices with your team. * **Explore new features**: Check out column-level lineage, live CTE previews, and other dbt v2-powered capabilities. Share your success We'd love to hear about your dbt v2 upgrade experience! Share feedback with your account team or join the [dbt Community Slack](https://www.getdbt.com/community/join-the-community/) to discuss dbt v2 with other users. --- ### Use Databricks workflows to run dbt jobs [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction Using Databricks workflows to call the dbt job API can be useful for several reasons: 1. **Integration with other ETL processes** — If you're already running other ETL processes in Databricks, you can use a Databricks workflow to trigger a dbt job after those processes are done. 2. **Utilizes dbt jobs features —** dbt gives the ability to monitor job progress, manage historical logs and documentation, optimize model timing, and much [more](https://docs.getdbt.com/docs/deploy/deploy-jobs.md). 3. [**Separation of concerns —**](https://en.wikipedia.org/wiki/Separation_of_concerns) Detailed logs for dbt jobs in the dbt environment can lead to more modularity and efficient debugging. By doing so, it becomes easier to isolate bugs quickly while still being able to see the overall status in Databricks. 4. **Custom job triggering —** Use a Databricks workflow to trigger dbt jobs based on custom conditions or logic that aren't natively supported by dbt's scheduling feature. This can give you more flexibility in terms of when and how your dbt jobs run. ##### Prerequisites * Active [Enterprise or Enterprise+ dbt account](https://www.getdbt.com/pricing/) * You must have a configured and existing [dbt deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * Active Databricks account with access to [Data Science and Engineering workspace](https://docs.databricks.com/workspace-index.html) and [Manage secrets](https://docs.databricks.com/security/secrets/index.html) * [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html) * **Note**: You only need to set up your authentication. Once you have set up your Host and Token and are able to run `databricks workspace ls /Users/<someone@example.com>`, you can proceed with the rest of this guide. #### Set up a Databricks secret scope 1. Retrieve \*\*[personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) \*\*or \*\*[Service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#generating-service-account-tokens) \*\*from dbt 2. Set up a **Databricks secret scope**, which is used to securely store your dbt API key. 3. Enter the **following commands** in your terminal: ```bash # In this example we set up a secret scope and key called "dbt-cloud" and "api-key" respectively. databricks secrets create-scope --scope <YOUR_SECRET_SCOPE> databricks secrets put --scope <YOUR_SECRET_SCOPE> --key <YOUR_SECRET_KEY> --string-value "<YOUR_DBT_CLOUD_API_KEY>" ``` 4. Replace **`<YOUR_SECRET_SCOPE>`** and **`<YOUR_SECRET_KEY>`** with your own unique identifiers. Click [here](https://docs.databricks.com/security/secrets/index.html) for more information on secrets. 5. Replace **`<YOUR_DBT_CLOUD_API_KEY>`** with the actual API key value that you copied from dbt in step 1. #### Create a Databricks Python notebook 1. [Create a **Databricks Python notebook**](https://docs.databricks.com/notebooks/notebooks-manage.html), which executes a Python script that calls the dbt job API. 2. Write a **Python script** that utilizes the `requests` library to make an HTTP POST request to the dbt job API endpoint using the required parameters. Here's an example script: ```python import enum import os import time import json import requests from getpass import getpass dbutils.widgets.text("job_id", "Enter the Job ID") job_id = dbutils.widgets.get("job_id") account_id = <YOUR_ACCOUNT_ID> base_url = "<YOUR_BASE_URL>" api_key = dbutils.secrets.get(scope = "<YOUR_SECRET_SCOPE>", key = "<YOUR_SECRET_KEY>") # These are documented on the dbt API docs class DbtJobRunStatus(enum.IntEnum): QUEUED = 1 STARTING = 2 RUNNING = 3 SUCCESS = 10 ERROR = 20 CANCELLED = 30 def _trigger_job() -> int: res = requests.post( url=f"https://{base_url}/api/v2/accounts/{account_id}/jobs/{job_id}/run/", headers={'Authorization': f"Token {api_key}"}, json={ # Optionally pass a description that can be viewed within the dbt API. # See the API docs for additional parameters that can be passed in, # including `schema_override` 'cause': f"Triggered by Databricks Workflows.", } ) try: res.raise_for_status() except: print(f"API token (last four): ...{api_key[-4:]}") raise response_payload = res.json() return response_payload['data']['id'] def _get_job_run_status(job_run_id): res = requests.get( url=f"https://{base_url}/api/v2/accounts/{account_id}/runs/{job_run_id}/", headers={'Authorization': f"Token {api_key}"}, ) res.raise_for_status() response_payload = res.json() return response_payload['data']['status'] def run(): job_run_id = _trigger_job() print(f"job_run_id = {job_run_id}") while True: time.sleep(5) status = _get_job_run_status(job_run_id) print(DbtJobRunStatus(status)) if status == DbtJobRunStatus.SUCCESS: break elif status == DbtJobRunStatus.ERROR or status == DbtJobRunStatus.CANCELLED: raise Exception("Failure!") if __name__ == '__main__': run() ``` 3. Replace **`<YOUR_SECRET_SCOPE>`** and **`<YOUR_SECRET_KEY>`** with the values you used [previously](#set-up-a-databricks-secret-scope) 4. Replace **`<YOUR_BASE_URL>`** and **`<YOUR_ACCOUNT_ID>`** with the correct values of your environment and [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. * To find these values, navigate to dbt, select **Deploy -> Jobs**. Select the Job you want to run and copy the URL. For example: `https://YOUR_ACCESS_URL/deploy/000000/projects/111111/jobs/222222` and therefore valid code would be: Your URL is structured `https://<YOUR_BASE_URL>/deploy/<YOUR_ACCOUNT_ID>/projects/<YOUR_PROJECT_ID>/jobs/<YOUR_JOB_ID>` account\_id = 000000 job\_id = 222222 base\_url = "cloud.getdbt.com" 5. Run the Notebook. It will fail, but you should see **a `job_id` widget** at the top of your notebook. 6. In the widget, **enter your `job_id`** from step 4. 7. **Run the Notebook again** to trigger the dbt job. Your results should look similar to the following: ```bash job_run_id = 123456 DbtJobRunStatus.QUEUED DbtJobRunStatus.QUEUED DbtJobRunStatus.QUEUED DbtJobRunStatus.STARTING DbtJobRunStatus.RUNNING DbtJobRunStatus.RUNNING DbtJobRunStatus.RUNNING DbtJobRunStatus.RUNNING DbtJobRunStatus.RUNNING DbtJobRunStatus.RUNNING DbtJobRunStatus.RUNNING DbtJobRunStatus.RUNNING DbtJobRunStatus.SUCCESS ``` You can cancel the job from dbt if necessary. #### Configure the workflows to run the dbt jobs You can set up workflows directly from the notebook OR by adding this notebook to one of your existing workflows: ##### Create a workflow from existing Notebook 1. Click **Schedule** on the upper right side of the page 2. Click **Add a schedule** 3. Configure Job name, Schedule, Cluster 4. Add a new parameter called: `job_id` and fill in your job ID. Refer to [step 4 in previous section](#create-a-databricks-python-notebook) to find your job ID. 5. Click **Create** 6. Click **Run Now** to test the job ##### Add the Notebook to existing workflow 1. Open Existing **Workflow** 2. Click **Tasks** 3. Press **“+” icon** to add a new task 4. Enter the **following**: | Field | Value | | ---------- | ----------------------------- | | Task name | `<unique_task_name>` | | Type | Notebook | | Source | Workspace | | Path | `</path/to/notebook>` | | Cluster | `<your_compute_cluster>` | | Parameters | `job_id`: `<your_dbt_job_id>` | 5. Select **Save Task** 6. Click **Run Now** to test the workflow Multiple Workflow tasks can be set up using the same notebook by configuring the `job_id` parameter to point to different dbt jobs. Using Databricks workflows to access the dbt job API can improve integration of your data pipeline processes and enable scheduling of more complex workflows. --- ### Use Jinja to improve your SQL code [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this guide, we're going to take a common pattern used in SQL, and then use Jinja to improve our code. If you'd like to work through this query, add [this CSV](https://github.com/dbt-labs/jaffle_shop/blob/core-v1.0.0/seeds/raw_payments.csv) to the `seeds/` folder of your dbt project, and then execute `dbt seed`. While working through the steps of this model, we recommend that you have your compiled SQL open as well, to check what your Jinja compiles to. To do this: * **Using dbt:** Click the compile button to see the compiled SQL in the right hand pane * **Using dbt v1:** Run `dbt compile` from the command line. Then open the compiled SQL file in the `target/compiled/{project name}/` directory. Use a split screen in your code editor to keep both files open at once. #### Write the SQL without Jinja Consider a data model in which an `order` can have many `payments`. Each `payment` may have a `payment_method` of `bank_transfer`, `credit_card` or `gift_card`, and therefore each `order` can have multiple `payment_methods` From an analytics perspective, it's important to know how much of each `order` was paid for with each `payment_method`. In your dbt project, you can create a model, named `order_payment_method_amounts`, with the following SQL: models/order\_payment\_method\_amounts.sql ```sql select order_id, sum(case when payment_method = 'bank_transfer' then amount end) as bank_transfer_amount, sum(case when payment_method = 'credit_card' then amount end) as credit_card_amount, sum(case when payment_method = 'gift_card' then amount end) as gift_card_amount, sum(amount) as total_amount from {{ ref('raw_payments') }} group by 1 ``` The SQL for each payment method amount is repetitive, which can be difficult to maintain for a number of reasons: * If the logic or field name were to change, the code would need to be updated in three places. * Often this code is created by copying and pasting, which may lead to mistakes. * Other analysts that review the code are less likely to notice errors as it's common to only scan through repeated code. So we're going to use Jinja to help us clean it up, or to make our code more "DRY" ("Don't Repeat Yourself"). #### Use a for loop in models for repeated SQL Here, the repeated code can be replaced with a `for` loop. The following will be compiled to the same query, but is significantly easier to maintain. /models/order\_payment\_method\_amounts.sql ```sql select order_id, {% for payment_method in ["bank_transfer", "credit_card", "gift_card"] %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount, {% endfor %} sum(amount) as total_amount from {{ ref('raw_payments') }} group by 1 ``` #### Set variables at the top of a model We recommend setting variables at the top of a model, as it helps with readability, and enables you to reference the list in multiple places if required. This is a practice we've borrowed from many other programming languages. /models/order\_payment\_method\_amounts.sql ```sql {% set payment_methods = ["bank_transfer", "credit_card", "gift_card"] %} select order_id, {% for payment_method in payment_methods %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount, {% endfor %} sum(amount) as total_amount from {{ ref('raw_payments') }} group by 1 ``` #### Use loop.last to avoid trailing commas In the above query, our last column is outside of the `for` loop. However, this may not always be the case. If the last iteration of a loop is our final column, we need to ensure there isn't a trailing comma at the end. We often use an `if` statement, along with the Jinja variable `loop.last`, to ensure we don't add an extraneous comma: /models/order\_payment\_method\_amounts.sql ```sql {% set payment_methods = ["bank_transfer", "credit_card", "gift_card"] %} select order_id, {% for payment_method in payment_methods %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount {% if not loop.last %},{% endif %} {% endfor %} from {{ ref('raw_payments') }} group by 1 ``` An alternative way to write this is `{{ "," if not loop.last }}`. #### Use whitespace control to tidy up compiled code If you've been checking your code in the `target/compiled` folder, you might have noticed that this code results in a lot of white space: target/compiled/jaffle\_shop/order\_payment\_method\_amounts.sql ```sql select order_id, sum(case when payment_method = 'bank_transfer' then amount end) as bank_transfer_amount , sum(case when payment_method = 'credit_card' then amount end) as credit_card_amount , sum(case when payment_method = 'gift_card' then amount end) as gift_card_amount from raw_jaffle_shop.payments group by 1 ``` We can use [whitespace control](https://jinja.palletsprojects.com/page/templates/#whitespace-control) to tidy up our code: models/order\_payment\_method\_amounts.sql ```sql {%- set payment_methods = ["bank_transfer", "credit_card", "gift_card"] -%} select order_id, {%- for payment_method in payment_methods %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount {%- if not loop.last %},{% endif -%} {% endfor %} from {{ ref('raw_payments') }} group by 1 ``` Getting whitespace control right is often a lot of trial and error! We recommend that you prioritize the readability of your model code over the readability of the compiled code, and only do this as an extra polish. #### Use a macro to return payment methods Here, we've hardcoded the list of payment methods in our model. We may need to access this list from another model. A good solution here is to use a [variable](https://docs.getdbt.com/docs/build/project-variables.md), but for the purpose of this tutorial, we're going to instead use a macro! [Macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) in Jinja are pieces of code that can be called multiple times – they are analogous to a function in Python, and are extremely useful if you find yourself repeating code across multiple models. Our macro is simply going to return the list of payment methods: /macros/get\_payment\_methods.sql ```sql {% macro get_payment_methods() %} {{ return(["bank_transfer", "credit_card", "gift_card"]) }} {% endmacro %} ``` There's a few things worth noting here: * Normally, macros take arguments -- we'll see this later on, but for now, we still need to setup our macro with empty parentheses where the arguments would normally go (i.e. `get_payment_methods()`) * We've used the [return](https://docs.getdbt.com/reference/dbt-jinja-functions/return.md) function to return a list – without this function, the macro would return a string. Now that we have a macro for our payment methods, we can update our model as follows: models/order\_payment\_method\_amounts.sql ```sql {%- set payment_methods = get_payment_methods() -%} select order_id, {%- for payment_method in payment_methods %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount {%- if not loop.last %},{% endif -%} {% endfor %} from {{ ref('raw_payments') }} group by 1 ``` Note that we didn't use curly braces when calling the macro – we're already within a Jinja statement, so there's no need to use the brackets again. #### Dynamically retrieve the list of payment methods So far, we've been hardcoding the list of possible payment methods. If a new `payment_method` was introduced, or one of the existing methods was renamed, the list would need to be updated. However, at any given time you could know what `payment_methods` are used to make a payment by running the following query: ```sql select distinct payment_method from {{ ref('raw_payments') }} order by 1 ``` [Statements](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks.md) provide a way to run this query and return the results to your Jinja context. This means that the list of `payment_methods` can be set based on the data in your database rather than a hardcoded value. The easiest way to use a statement is through the [run\_query](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) macro. For the first version, let's check what we get back from the database, by logging the results to the command line using the [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) function. macros/get\_payment\_methods.sql ```sql {% macro get_payment_methods() %} {% set payment_methods_query %} select distinct payment_method from {{ ref('raw_payments') }} order by 1 {% endset %} {% set results = run_query(payment_methods_query) %} {{ log(results, info=True) }} {{ return([]) }} {% endmacro %} ``` The command line gives us back the following: ```bash | column | data_type | | -------------- | --------- | | payment_method | Text | ``` This is actually an [Agate table](https://agate.readthedocs.io/page/api/table.html). To get the payment methods back as a list, we need to do some further transformation. ```sql {% macro get_payment_methods() %} {% set payment_methods_query %} select distinct payment_method from {{ ref('raw_payments') }} order by 1 {% endset %} {% set results = run_query(payment_methods_query) %} {% if execute %} {# Return the first column #} {% set results_list = results.columns[0].values() %} {% else %} {% set results_list = [] %} {% endif %} {{ return(results_list) }} {% endmacro %} ``` There's a few tricky pieces in here: * We used the [execute](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) variable to ensure that the code runs during the `parse` stage of dbt (otherwise an error would be thrown). * We used Agate methods to get the column back as a list Fortunately, our model code doesn't need to be updated, since we're already calling the macro to get the list of payment methods. And now, any new `payment_methods` added to the underlying data model will automatically be handled by the dbt model. #### Write modular macros You may wish to use a similar pattern elsewhere in your dbt project. As a result, you decide to break up your logic into two separate macros -- one to generically return a column from a relation, and the other that calls this macro with the correct arguments to get back the list of payment methods. macros/get\_payment\_methods.sql ```sql {% macro get_column_values(column_name, relation) %} {% set relation_query %} select distinct {{ column_name }} from {{ relation }} order by 1 {% endset %} {% set results = run_query(relation_query) %} {% if execute %} {# Return the first column #} {% set results_list = results.columns[0].values() %} {% else %} {% set results_list = [] %} {% endif %} {{ return(results_list) }} {% endmacro %} {% macro get_payment_methods() %} {{ return(get_column_values('payment_method', ref('raw_payments'))) }} {% endmacro %} ``` #### Use a macro from a package Macros let analysts bring software engineering principles to the SQL they write. One of the features of macros that makes them even more powerful is their ability to be shared across projects. A number of useful dbt macros have already been written in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils). For example, the [get\_column\_values](https://github.com/dbt-labs/dbt-utils#get_column_values-source) macro from dbt-utils could be used instead of the `get_column_values` macro we wrote ourselves (saving us a lot of time, but at least we learnt something along the way!). Install the [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) package in your project (docs [here](https://docs.getdbt.com/docs/build/packages.md)), and then update your model to use the macro from the package instead: models/order\_payment\_method\_amounts.sql ```sql {%- set payment_methods = dbt_utils.get_column_values( table=ref('raw_payments'), column='payment_method' ) -%} select order_id, {%- for payment_method in payment_methods %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount {%- if not loop.last %},{% endif -%} {% endfor %} from {{ ref('raw_payments') }} group by 1 ``` You can then remove the macros that we built in previous steps. Whenever you're trying to solve a problem that you think others may have solved previously, it's worth checking the [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) package to see if someone has shared their code! --- ### Using BigQuery DataFrames with dbt Python models [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() #### Introduction In this guide, you'll learn how to set up dbt so you can use it with BigQuery DataFrames (BigFrames): * Build scalable data transformation pipelines using dbt and Google Cloud, with SQL and Python. * Leverage BigFrames from dbt for scalable BigQuery SQL. In addition to the existing dataproc/pyspark based submission methods for executing python models, you can now use the BigFrames submission method to execute Python models with pandas-like and scikit-like APIs, without the need of any Spark setup or knowledge. BigQuery DataFrames is an open source Python package that transpiles pandas and scikit-learn code to scalable BigQuery SQL. The dbt-bigquery adapter relies on the BigQuery Studio Notebook Executor Service to run the Python client side code. ##### Prerequisites * A [Google Cloud account](https://cloud.google.com/free) * A [dbt account](https://www.getdbt.com/signup/) * Basic to intermediate SQL and python. * Basic understanding of dbt fundamentals. We recommend the [dbt Fundamentals course](https://learn.getdbt.com). During setup, you’ll need to select the **BigQuery** adapter and enter values for your **Google Cloud Storage Bucket** and **Dataproc Region** in the dbt platform. See [Configure BigQuery in dbt platform](https://docs.getdbt.com/guides/dbt-python-bigframes.md?step=2#configure-bigquery-in-dbt-platform) for details. ##### What you'll build Here's what you'll build in two parts: * Google Cloud project setup * A one-time setup to configure the Google Cloud project you’ll be working with. * Build and Run the Python Model * Create, configure, and execute a Python model using BigQuery DataFrames and dbt. You will set up the environments, build scalable pipelines in dbt, and execute a python model. [![Implementation of the BigFrames submission method](/img/guides/gcp-guides/gcp-bigframes-architecture.png?v=2 "Implementation of the BigFrames submission method")](#)Implementation of the BigFrames submission method **Figure 1** - Implementation of the BigFrames submission method for dbt python models #### Configure Google Cloud The dbt BigFrames submission method supports both service account and OAuth credentials. You will use the service account in the following steps. 1. **Create a new Google Cloud Project** a. Your new project will have the following list of APIs already enabled, including BigQuery, which is required. * [Default APIs](https://cloud.google.com/service-usage/docs/enabled-service#default) b. Enable the BigQuery API which also enables the following additional APIs automatically * [BigQuery API's](https://cloud.google.com/bigquery/docs/enable-assets#automatic-api-enablement) c. Required API's: * **BigQuery API:** For all core BigQuery operations. * **Vertex AI API:** To use the Colab Enterprise executor service. * **Cloud Storage API:** For staging code and logs. * **IAM API:** For managing permissions. * **Compute Engine API:** As an underlying dependency for the notebook runtime environment. * **Dataform API:** For managing the notebook code assets within BigQuery. 2. **Create a service account and grant IAM permissions** This service account will be used by dbt to read and write data on BigQuery and use BigQuery Studio Notebooks. Create the service account with IAM permissions: ```python #Create Service Account gcloud iam service-accounts create dbt-bigframes-sa #Grant BigQuery User Role gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/bigquery.user #Grant BigQuery Data Editor role. This can be restricted at dataset level gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/bigquery.dataEditor #Grant Service Account user gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/iam.serviceAccountUser #Grant Colab Entperprise User gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/aiplatform.colabEnterpriseUser ``` When using a Shared VPC When using Colab Enterprise in a Shared VPC environment, additional roles are required for the following service accounts on the Shared VPC host project: * Vertex AI P4SA (`service-<PROJECT_NUMBER>@gcp-sa-aiplatform.iam.gserviceaccount.com`): This service account always requires the Compute Network User (`roles/compute.networkUser`) role on the Shared VPC host project. Replace `<PROJECT_NUMBER>` with the project number. * Colab Enterprise P6SA (`service-<PROJECT_NUMBER>@gcp-sa-vertex-nb.iam.gserviceaccount.com`): This service account also needs the Compute Network User (`roles/compute.networkUser`) role on the Shared VPC host project. Replace `<PROJECT_NUMBER>` with the project number. 3. *(Optional)* **Create a test BigQuery Dataset** Create a new BigQuery Dataset if you don't already have one: ```python #Create BQ dataset bq mk --location=${REGION} echo "${GOOGLE_CLOUD_PROJECT}" | tr '-' '_'_dataset ``` 4. **Create a GCS bucket to stage the python code, and store logs** For temporary log and code storage, please create a GCS bucket and assign the required permissions: ```python #Create GCS bucket gcloud storage buckets create gs://${GOOGLE_CLOUD_PROJECT}-bucket --location=${REGION} #Grant Storage Admin over the bucket to your SA gcloud storage buckets add-iam-policy-binding gs://${GOOGLE_CLOUD_PROJECT}-bucket --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/storage.admin ``` ##### Configure BigQuery in the dbt platform To set up your BigQuery DataFrames connection in the dbt platform, refer to the following steps: 1. Go to **Account settings** > **Connections**. Click **New connection**. 2. In the **Type** section, select **BigQuery**. 3. Select **BigQuery** as your adapter. 4. Under **Optional settings**, enter values for the following fields: * **Google Cloud Storage Bucket** (for example: `dbt_name_bucket`) * **Dataproc Region** (for example: `us-central1`) 5. Click **Save**. This is required so that BigFrames jobs execute correctly. Refer to [Connect to BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md) for more info on how to connect to BigQuery in the dbt platform. #### Create, configure, and execute your Python models 1. In your dbt project, create a SQL model in your models directory, ending in the `.sql` file extension. Name it `my_sql_model.sql`. 2. In the file, copy this SQL into it. ```sql select 1 as foo, 2 as bar ``` 3. Now create a new model file in the models directory, named `my_first_python_model.py`. 4. In the `my_first_python_model.py` file, add this code: ```python def model(dbt, session): dbt.config(submission_method="bigframes") bdf = dbt.ref("my_sql_model") #loading from prev step return bdf ``` 5. Configure the BigFrames submission method by using either: a. Project level configuration via dbt\_project.yml ```python models: my_dbt_project: submission_method: bigframes python_models: +materialized: view ``` or b. The Python code via dbt.config in the my\_first\_python\_model.py file ```python def model(dbt, session): dbt.config(submission_method="bigframes") # rest of the python code... ``` 6. Run `dbt run` 7. You can view the logs in [dbt logs](https://docs.getdbt.com/reference/events-logging.md). You can optionally view the codes and logs (including previous executions) from the [Colab Enterprise Executions](https://console.cloud.google.com/vertex-ai/colab/execution-jobs) tab and [GCS bucket](https://console.cloud.google.com/storage/browser) from the GCP console. 8. Congrats! You just created your first two python models to run on BigFrames! --- ## Navigation Options ### navigation-options --- ## Platform ### About billing dbt platform dbt offers a variety of [plans and pricing](https://www.getdbt.com/pricing/) to fit your organization’s needs. With flexible billing options that appeal to large enterprises and small businesses and [server availability](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) worldwide, dbt platform is the fastest and easiest way to begin transforming your data. Use the following pages to understand how billing, usage, and cost optimization work: * [Plans](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md) — How billing works for the Developer, Starter, Enterprise, and legacy plans. * [How does dbt pricing work?](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md) — How seats and usage are billed, what counts as a Successful Model Built and a Queried Metric, and how to view usage. * [dbt State usage](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md) — Usage-based pricing for dbt State, including daily active target tables (DATT). * [dbt AI: Usage metering and limiting](https://docs.getdbt.com/docs/platform/billing/dbt-ai-usage.md) — How dbt AI (dbt Copilot) usage is metered and limited by plan. * [Managing usage](https://docs.getdbt.com/docs/platform/billing/managing-usage.md) — Usage notifications and how to stop usage from accruing. * [Optimize costs in dbt](https://docs.getdbt.com/docs/platform/billing/optimize-costs.md) — Best practices for reducing model build usage and warehouse costs. For answers to common plan and billing questions, refer to [Billing FAQs](https://docs.getdbt.com/docs/platform/billing-faqs.md). --- ### About Canvas dbt platform | Enterprise, Enterprise+ Canvas helps you quickly access and transform data through a visual, drag-and-drop experience. Canvas allows organizations to enjoy the many benefits of code-driven development, such as increased precision, ease of debugging, and ease of validation, while retaining the flexibility to have different contributors develop wherever they are most comfortable. For natural language and AI-assisted workflows, use [dbt Copilot in Canvas](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md). These models compile directly to SQL and are indistinguishable from other dbt models in your projects: * Visual models are version-controlled in your backing Git provider. * All models are accessible across projects in [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md). * Models can be materialized into production through [dbt orchestration](https://docs.getdbt.com/docs/deploy/deployments.md), or be built directly into a user's development schema. * Integrate with [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). [![Create or edit dbt models with Canvas, enabling everyone to develop with dbt through a drag-and-drop experience inside of dbt.](/img/docs/dbt-platform/canvas/canvas.png?v=2 "Create or edit dbt models with Canvas, enabling everyone to develop with dbt through a drag-and-drop experience inside of dbt.")](#)Create or edit dbt models with Canvas, enabling everyone to develop with dbt through a drag-and-drop experience inside of dbt. #### Canvas prerequisites Before using Canvas, you should: * Have a [dbt Enterprise or Enterprise+](https://www.getdbt.com/pricing) account. * Have a [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) with user credentials set up. * Be using one of the following adapters: * Bigquery * Databricks * Redshift * Snowflake * Trino * You can access the Canvas with adapters not listed, but some features may be missing at this time. * Use [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md), [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md), or [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) as your Git provider, connected to dbt via HTTPS. * SSH connections aren't supported at this time. * Self-hosted or on-premises deployments of any Git provider aren't supported for Canvas at this time. * Have an existing dbt project already created with a Staging or Production run completed. * Verify your Development environment is on a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing updates. * Have read-only access to the [Staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) with the data to be able to execute `run` in the Canvas. To customize the required access for the Canvas user group, refer to [Set up environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md) for more information. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. #### Feedback Please note, always review AI-generated code and content as it may produce incorrect results. To give feedback, please reach out to your dbt Labs account team. We appreciate your feedback and suggestions as we improve Canvas. #### Resources Learn more about Canvas: * How to [use Canvas](https://docs.getdbt.com/docs/platform/use-canvas.md) * The Canvas [quickstart guide](https://docs.getdbt.com/guides/canvas.md) * [Canvas fundamentals course](https://learn.getdbt.com/learn/course/canvas-fundamentals) on [dbt Learn](https://learn.getdbt.com/catalog) --- ### About dbt platform profiles dbt platform dbt platform profiles define the connections, credentials, and attributes you use to connect to a data warehouse. Assign profiles to [deployment environments](https://docs.getdbt.com/docs/dbt-platform-environments.md#deployment-environment) and reuse those profiles in other deployment environments within the same project. You can manage profiles programmatically using our [API documentation](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/List%20Profiles). #### Environment profiles table On an environment's **Settings** page, the **Connection profiles** section lists the profiles assigned to that environment: * **Profile name**: Click a profile name to open the view/edit drawer. In view mode, there’s no separate action column, so use the profile name to open and view a profile. * **Connection**: Click the connection to open the [connection details](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management) page in a new tab. * **Edit mode**: Click **Edit** to switch to edit mode. Use the **swap icon** ![Swap icon](/img/docs/deploy/swap-icon.png) next to a profile row to assign a different profile. ##### Considerations * Profiles don't apply to development environments because of the unique configurations and individual credentials applied. * The Semantic Layer configuration isn't supported with profiles yet. #### Create a profile new feature rollout dbt automatically creates a new project-level profile for each deployment environment and populates it with your existing connection, credentials, and extended attributes. You don't need to take any action to create profiles for your existing projects. You can create profiles from either the project or the environment settings. No matter which approach you take, dbt creates the profile at the project level. Profiles you create in one project won't be visible in others. To create a new profile: ##### From project settings 1. From the main menu, navigate to your project's **Dashboard**. 2. Click **Settings**. 3. Scroll down to the **Profiles** section and click **Create new profile**. [![Creating a profile from project settings.](/img/docs/dbt-platform/profile-from-project.png?v=2 "Creating a profile from project settings.")](#)Creating a profile from project settings. ##### From environment settings 1. From the main menu, click **Orchestration** and select **Environments**. 2. Click an available deployment environment. 3. Click **Settings**. 4. Click **Edit** to switch to edit mode, then scroll to the **Connection profiles** section. 5. Click the **swap icon** ![Swap icon](/img/docs/deploy/swap-icon.png) next to the profile row you want to change. 6. Select **Add new profile** from the **Profile** dropdown. 7. Click **Create profile**. 8. Click **Save**. [![Creating a profile from the environment settings.](/img/docs/dbt-platform/profile-from-environment.png?v=2 "Creating a profile from the environment settings.")](#)Creating a profile from the environment settings. The following steps are the same regardless of which approach you take: 1. Give the profile a name that's unique across all projects in the account, easy to identify, and adheres to the naming policy: * Starts with a letter * Ends with a letter or number * Contains only letters, numbers, dashes, or underscores * Has no consecutive dashes or underscores 2. From **Connection details**, select a connection from the list of available [global connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management) or add a new connection. 3. Configure the **Deployment credentials** for your warehouse connection. 4. Add any [**Extended attributes**](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) you need. If you use [`env_var()`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) in Extended Attributes, the referenced environment variables must be *project-scoped* in order to work with connection tests. Since profiles are environment-agnostic, environment-scoped variables are not available during connection tests. To set a project-scoped variable, go to **Orchestration** > **Environments** > **Environment variables**, and enter a value in the **Project default** column. Learn more in [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md?version=2.0#setting-environment-variables). 5. Click **Save** at the top of the screen. [![Sample of a configured profile.](/img/docs/dbt-platform/profile-sample.png?v=2 "Sample of a configured profile.")](#)Sample of a configured profile. Repeat these steps until you've created all the profiles you need for your project's deployment environments. #### Assign a profile You configure profiles when you create a deployment environment. For accounts that already have environments configured when you enable profiles, dbt automatically creates and assigns a default profile to all projects. To assign a different profile, update the deployment environment settings: 1. From the left navigation, click **Orchestration** and select **Environments**. 2. Click an available deployment environment. 3. Click **Settings**. 4. Click **Edit** to switch to edit mode, then scroll to the **Connection profiles** section. 5. Click the **swap icon** ![Swap icon](/img/docs/deploy/swap-icon.png) next to the profile row you want to change. 6. Select the new profile from the **Profile** dropdown. 7. Click **Assign profile**. 8. Click **Save**. #### Permissions and access to profiles Profiles are created at the project level. Only users with permission to edit the project can create profiles and anyone with permission to create or edit deployment environments in that project can assign that profile and its credentials to those environments. To avoid unintended access, only grant permission sets like **Job Admin** or **Project Admin** to users who should have access to all credentials in a project. Be mindful that profiles created at the project level can be used to configure credentials for any deployment environment in that project. For more information on permission sets, see [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md). #### FAQs  Do I need to create profiles for all of my existing projects? You don't need to take any action. dbt automatically creates profiles for all existing projects and deployment environments based on the existing connection, credentials, and extended attributes.  Are there any changes to development environments? Not at this time. Profiles only apply to deployment environments.  What happens if I change my connection details, credentials, or attributes? Any profiles using those settings automatically update with the new information.  What if I use APIs to configure project settings? Existing APIs continue to work and automatically map to a profile behind the scenes. You won't need to take any manual action unless you use APIs to create a deployment environment with no credentials configured. This is a rare occurrence unique to APIs, but it's the only scenario where dbt wouldn't create a profile. Profile-specific APIs are available. Check out our [API documentation](https://docs.getdbt.com/docs/dbt-apis/overview.md) for more information.  Does the Semantic Layer support profiles? Semantic Layer configuration isn't supported with profiles yet. --- ### About dbt Wizard Local development | dbt platform Your personal dbt agent — wherever you work. dbt Wizard is an AI agent purpose-built for governed data development in dbt. Unlike general-purpose coding agents, it understands your dbt project through the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) — a structured view of lineage, model health, tests, contracts, run results, and semantic definitions. Think of it like a map of your city: dbt Wizard knows how everything connects before it starts, rather than walking every street to figure out the layout. dbt Wizard comes with: * **Project understanding:** The dbt Information Schema for lineage, contracts, tests, and runtime context * **Impact awareness:** Checks upstream and downstream dependencies before you change code * **Safe validation:** Compiles and builds changes before review * **Complete workflow:** Investigate, change, validate, and review in one place * **Setup and governance:** Works out of the box with dbt governance built in * **Conversational analytics:** Answers questions about production data in plain language through [Explore mode](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes), grounded in governed metric definitions Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. #### Use dbt Wizard dbt Wizard is for anyone developing with or exploring dbt. You can use it in dbt platform, locally in your terminal or IDE, or through the [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) native app, where rich data visualizations, data diffs, and lineage render alongside the chat. dbt Wizard works with any data warehouse and [dbt version](https://docs.getdbt.com/docs/introduction.md#dbt-versions). dbt Wizard is also for people who don’t develop data models. In dbt platform, [Explore mode](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) lets [read-only users](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) ask questions about production data in plain language with no developer license or setup. The following table shows where dbt Wizard is available, the AI keys each surface uses, and how usage is billed: | Where | Status | AI access options | | ------------------------------------------------------------------------------------- | -------------- | ----------------------------------------- | | [dbt platform: Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) | Public preview | dbt managed or BYOK | | [dbt platform: Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md) | Public preview | dbt managed or BYOK | | [Locally: Wizard terminal (CLI)](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) | Public beta | dbt managed, BYOK, or OpenAI subscription | | [Locally: Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) | Public beta | dbt managed or BYOK | Prefer to learn by video? Check out the free [dbt Wizard video course](https://learn.getdbt.com/courses/dbt-wizard) on dbt Learn! #### Supported providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). For pricing and how billing works, refer to [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md). #### Get started You can get started with Wizard wherever you work, whether it's the terminal or dbt platform: * [Wizard in the CLI](#wizard-in-the-cli) * [Wizard Desktop](#wizard-desktop) * [Wizard in dbt platform](#wizard-in-dbt-platform) (Be warned, the wizard has been known to cast spells .) ##### Wizard in the CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") A terminal-based agent for governed data development in dbt, whether your team uses dbt platform or self-hosts. Bring your own key to experience the full agentic analytics engineering loop. 12 View all stepsNext 1 Install the dbt Wizard CLI Run the install script for your operating system: macOS/Linux: ```bash curl -fsSL https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.sh | sh ``` Windows (PowerShell): ```powershell irm https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.ps1 | iex ``` This installs dbt Wizard to `/usr/local/bin/wizard`, along with the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) that powers dbt Wizard's project-aware answers. For first-run setup and billing, refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). New to the terminal? If you've never used the terminal before, check out the [terminal guide](https://docs.getdbt.com/guides/terminal-guide.md) for some helpful tips to help you get started! ##### Wizard Desktop [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Wizard Desktop is Wizard as a native app. Wizard Desktop runs the same engine as the dbt Wizard CLI, but you get the full view: charts, tables, diffs, and lineage render right in the app, so you can see the work, steer it, and keep several chats moving at once, each on its own branch. What you get with Desktop: * **Parallel chats:** each chat gets its own Git worktree, so several pieces of work run at once without conflicts. * **Visible validation:** see structured results, including a `dbt-compare` view of what your changes did to the data. * **Contextual comments:** hover almost anything to comment on that exact spot, then send several together. * **Charts and tables:** ask a question about your data and see the answer inline. * **Inline impact:** see which models dbt Wizard's changes touch, with diffs you open inline. * **Lineage, file, and metrics browsing:** browse your project right alongside the chat. [Download for macOSFor Apple silicon Macs · .dmg](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-aarch64-apple-darwin.dmg)[Download for WindowsFor x64 processors · .exe](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-pc-windows-msvc.exe)[Download for LinuxFor x86\_64, Ubuntu and Debian · .deb](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-unknown-linux-gnu.deb) Need a different build? [Intel Mac (.dmg)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-x86_64-apple-darwin.dmg) · [Linux ARM64 (.deb)](https://public.cdn.getdbt.com/dbt-wizard/latest/dbt-wizard-app-aarch64-unknown-linux-gnu.deb) To learn more, refer to [Get started with Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md). ##### Wizard in dbt platform [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Leverage agentic capabilities in the home app or [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) for governed data development in dbt. AI features are on by default, so most accounts can jump straight in. If your account has them turned off, an admin can turn them back on in [Account settings](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#manage-ai-features). To get started: 1. Sign in to [dbt platform](https://www.getdbt.com/signup), or create a free account if you don't have one yet. 2. Open dbt Wizard from the **home tab** in the left sidebar, or from [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) to work alongside the file editor. 3. Start with your usage credits or a 30-day trial. Enterprise and Enterprise+ accounts get monthly usage credits; all other plans can start a trial. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). 4. Try a prompt, such as: * `summarize what this project does` * `list all models with no tests` * `add not_null and unique tests to the primary key of stg_customers` Refer to [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) for more prompts. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) — generate docs, tests, semantic models, SQL, and delegate end-to-end model work * [Invite read-only users to dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) — let business users ask questions of production data in Explore mode, without development access * [Use skills in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-skills.md) — give dbt Wizard reusable instructions for your project * [Use MCP servers with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) — connect the dbt Wizard CLI to more tools and context * [Migrate to dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) — switch from Claude Code, Cursor, or another AI agent to dbt Wizard * [Privacy and data FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#privacy-and-data) — understand how dbt Wizard handles privacy and data See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### About dbt Wizard in dbt platform Preview ### About dbt Wizard in dbt platform [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based dbt Wizard is dbt's AI agent in dbt platform, helping teams investigate, change, validate, and ship trusted dbt work with warehouse-aware grounding. dbt Wizard is more than a general coding agent with access to dbt. Built for governed data development in dbt, it understands lineage, documentation, tests, and semantic definitions, and accounts for dev builds, compute, run time, and post-build inspection. Its suggestions are grounded in your project's actual data *and* context. Be warned, the wizard has been known to cast spells . AI features are being enabled by default. They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can [turn AI off or back on and configure providers](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) anytime. Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. #### Where you can use dbt Wizard * **[Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md):** Ask questions about your project, generate changes, review the diff, and run validations — all in one place. * **[Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md):** Work with dbt Wizard alongside the code editor, console, and file explorer. * **[Terminal (CLI)](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md):** Use the same agent from your terminal, with or without a dbt platform account. * **[Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md):** A native app for building and validating data, with rich data visualizations, data diffs, and lineage rendered alongside the chat. You can teach dbt Wizard your team's conventions with [skills](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-skills.md), hand off bigger jobs to [subagents](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-subagents.md), or connect your project to other AI tools with the [dbt platform MCP server](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-mcp.md). #### What you can do * Ask project-aware questions and get answers grounded in your project's context * Generate documentation, semantic models, tests, and metrics * Build or refactor models from plain-language prompts * Review file changes as diffs before you save them * Run end-to-end tasks in [agent mode](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes), either approving each file change or letting dbt Wizard edit automatically * Ask questions of production data in [Explore mode](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes), including [read-only users](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) with no development access * Follow along in the wayfinder bar, which shows your current project and branch and guides you through Git tasks like committing files or creating a branch * Get consistent output from [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills), which encode dbt best practices out of the box For more examples, visit [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md). Best practices for using dbt Wizard Most of the workflows in [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) apply here too — the prompts work the same in Studio IDE and the dbt Wizard home tab. #### Supported AI providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). #### Open dbt Wizard 1. Sign in to dbt platform. 2. Select the dbt Wizard icon in the navigation menu, or open it in the Studio IDE. 3. Enter a prompt. Try one of these to get a feel for it: * `summarize what this project does` * `which models in this project have no tests?` * `add not_null and unique tests to the primary key of stg_customers` 4. ..and that's it 🎉! [![dbt Wizard in the Studio IDE refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard in the Studio IDE refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard in the Studio IDE refactoring a model and displaying the lineage inside the chat interface. [![Wizard's final refactor result displayed as a diff in the Studio IDE](/img/docs/dbt-platform/wizard-ide-refactor-diff.png?v=2 "Wizard's final refactor result displayed as a diff in the Studio IDE")](#)Wizard's final refactor result displayed as a diff in the Studio IDE For more prompt ideas, refer to the [prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md). If dbt Wizard isn't available, confirm that your plan is eligible and that an admin has [enabled account access to AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#manage-ai-features). Prefer to learn by video? Check out the free [dbt Wizard video course](https://learn.getdbt.com/courses/dbt-wizard) on dbt Learn! #### Trial and usage credits Every prompt you send to dbt Wizard uses AI model tokens. What you pay for those tokens depends on your plan and whether you use a dbt Labs-managed model or your own key. If you [bring your own key](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md), your AI provider bills you directly and your usage doesn't draw from your consumption pool. [Legacy team plans](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) don't have access to dbt Wizard. You can move to a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing) to use it. For steps on how to start your trial and manage your spend, refer to [Start your trial](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md#start-your-trial) and [Manage your spend limit](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md#manage-your-spend-limit). To choose a provider or bring your own key, refer to [Manage AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider). #### Other AI features in dbt platform dbt platform also has AI features that work separately from dbt Wizard: * [dbt Copilot](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) generates SQL, documentation, tests, and semantic models in one click in Studio IDE, Canvas, and Insights. * The [dbt Support Assistant](https://docs.getdbt.com/docs/dbt-support.md?version=2#ask-dbt-support-assistant) answers product questions and helps you open a ticket. Contact the [dbt Support team](mailto:support@getdbt.com) for more info.  What's the difference between dbt Wizard and dbt Copilot? dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. #### Related docs * [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#in-the-dbt-platform) — architecture and data flow in the dbt platform * [Manage AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) — admin setup for AI access and providers * [Configure BYOK in dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) * [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md) * [Data & Privacy in dbt platform](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#privacy-and-data) --- ### About Studio IDE dbt platform The dbt integrated development environment (Studio IDE) is a single interface for building, testing, running, and version-controlling dbt projects from your browser. With the Cloud Studio IDE, you can compile dbt code into SQL and run it against your database directly. With the Cloud Studio IDE, you can: * Write modular SQL models with select statements and the ref() function, * Compile dbt code into SQL and execute it against your database directly, * Test every model before deploying them to production, * Generate and view documentation of your dbt project, * Leverage git and version-control your code from your browser with a couple of clicks, * Create and test Python models: * Compile Python models to see the full function that gets executed in your data platform * See Python models in DAG in dbt version 1.3 and higher * Currently, you can't preview python models * Visualize a directed acyclic graph (DAG), and more. [![The Studio IDE in dark mode](/img/docs/dbt-platform/platform-ide/cloud-ide-v2.png?v=2 "The Studio IDE in dark mode")](#)The Studio IDE in dark mode For more information, read the complete [Cloud Studio IDE guide](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). #### Related docs * [Studio IDE user interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) * [Keyboard shortcuts](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) --- ### About the Studio IDE dbt platform The dbt integrated development environment (Studio IDE) is a single web-based interface for building, testing, running, and version-controlling dbt projects. It compiles dbt code into SQL and executes it directly on your database. The Studio IDE offers several [keyboard shortcuts](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) and [editing features](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#editing-features) for faster and efficient development and governance: * Agentic development: [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) is the new and recommended AI agent for governed data development in dbt. It uses your project context to help you develop governed dbt changes faster. * [dbt Copilot](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md#dbt-copilot-in-studio-ide) is separate from dbt Wizard and is dbt's inline AI assistance experience, providing single-click generation of SQL, documentation, tests, and semantic models in Studio IDE, Canvas, and Insights. * Auto-completion: Suggests tables, arguments, and columns as you type. * Code [formatting and linting](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md): Standardizes and fixes SQL code. * Navigation tools: Jump to lines, find and replace text, and navigate project files. * Version control: Manage code versions with a few clicks. * Project documentation: Generate and view [project documentation](#build-and-document-your-projects) in real time. * Build, test, and run button: Build, test, and run your project from a button or the Studio IDE command bar. * Syntax highlighting for SQL: Improves readability and helps reduce syntax errors. These [features](#studio-ide-features) create a powerful editing environment for efficient SQL coding, suitable for both experienced and beginner developers. [![The Studio IDE includes version control, files/folders, an editor, a command/console, and more.](/img/docs/dbt-platform/platform-ide/ide-basic-layout.png?v=2 "The Studio IDE includes version control, files/folders, an editor, a command/console, and more.")](#)The Studio IDE includes version control, files/folders, an editor, a command/console, and more. [![Enable dark mode for a great viewing experience in low-light environments.](/img/docs/dbt-platform/platform-ide/cloud-ide-v2.png?v=2 "Enable dark mode for a great viewing experience in low-light environments.")](#)Enable dark mode for a great viewing experience in low-light environments. Disable ad blockers To improve your experience using dbt, we suggest that you turn off ad blockers. This is because some project file names, such as `google_adwords.sql`, might resemble ad traffic and trigger ad blockers. #### Prerequisites * A [dbt account](https://www.getdbt.com/signup) and [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) * A git repository set up and git provider must have `write` access enabled. See [Connecting your GitHub Account](https://docs.getdbt.com/docs/platform/git/connect-github.md) or [Importing a project by git URL](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) for detailed setup instructions * A dbt project connected to a [data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) * A [development environment and user credentials](#get-started-with-the-studio-ide) set up * The environment must be on dbt version 1.0 or higher #### Studio IDE features The Studio IDE comes with features that make it easier for you to develop, build, compile, run, and test data models. To understand how to navigate the Studio IDE and its user interface elements, refer to the [Studio IDE user interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) page. | Feature | Description | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [**Studio IDE shortcuts**](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) | You can access a variety of [commands and actions](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) in the Studio IDE by choosing the appropriate keyboard shortcut. Use the shortcuts for common tasks like building modified models or resuming builds from the last failure. | | **IDE version control** | The Studio IDE version control section and git button allow you to apply the concept of [version control](https://docs.getdbt.com/docs/platform/git/version-control-basics.md) to your project directly into the Studio IDE.<br /><br />- Create or change branches, execute git commands using the git button.<br />- Commit or revert individual files by right-clicking the edited file<br />- [Resolve merge conflicts](https://docs.getdbt.com/docs/platform/git/merge-conflicts.md)<br />- Link to the repo directly by clicking the branch name<br />- Edit, format, or lint files and execute dbt commands in your primary protected branch, and commit to a new branch.<br />- Use Git diff view to view what has been changed in a file before you make a pull request.<br />- Use the **Prune branches** [button](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#prune-branches-modal) to delete local branches that have been deleted from the remote repository, keeping your branch management tidy.<br />- Sign your [git commits](https://docs.getdbt.com/docs/platform/studio-ide/git-commit-signing.md) to mark them as 'Verified'. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") | | **Preview and Compile button** | You can [compile or preview](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#console-section) code, a snippet of dbt code, or one of your dbt models after editing and saving. | | [**dbt Wizard**](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) | Governed agentic development in the Studio IDE. Ask project-aware questions, generate resources, and write or refactor models from natural language — grounded in your project’s lineage, tests, and model health. | | **Build, test, and run button** | Build, test, and run your project with the click of a button or by using the command bar. | | **Command bar** | You can enter and run commands from the command bar at the bottom of the Studio IDE. Use the [rich model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) to execute [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) directly within dbt. You can also view the history, status, and logs of previous runs by clicking History on the left of the bar. | | **Drag and drop** | Drag and drop files located in the file explorer, and use the file breadcrumb on the top of the Studio IDE for quick, linear navigation. Access adjacent files in the same file by right-clicking on the breadcrumb file. | | **Organize tabs and files** | - Move your tabs around to reorganize your work in the IDE<br />- Right-click on a tab to view and select a list of actions, including duplicate files<br />- Close multiple, unsaved tabs to batch save your work<br />- Double click files to rename files | | **Find and replace** | - Press Command-F or Control-F to open the find-and-replace bar in the upper right corner of the current file in the IDE. The IDE highlights your search results in the current file and code outline<br />- You can use the up and down arrows to see the match highlighted in the current file when there are multiple matches<br />- Use the left arrow to replace the text with something else | | **Multiple selections** | You can make multiple selections for small and simultaneous edits. The below commands are a common way to add more cursors and allow you to insert cursors below or above with ease.<br /><br />- Option-Command-Down arrow or Ctrl-Alt-Down arrow<br />- Option-Command-Up arrow or Ctrl-Alt-Up arrow<br />- Press Option and click on an area or Press Ctrl-Alt and click on an area<br /> | | **Lint and Format** | [Lint and format](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) your files with a click of a button, powered by SQLFluff, sqlfmt, Prettier, and Black. | | **dbt autocomplete** | Autocomplete features to help you develop faster:<br /><br />- Use `ref` to autocomplete your model names<br />- Use `source` to autocomplete your source name + table name<br />- Use `macro` to autocomplete your arguments<br />- Use `env var` to autocomplete env var<br />- Start typing a hyphen (-) to use in-line autocomplete in a YAML file<br />- Automatically create models from dbt sources with a click of a button. | | **DAG in the IDE** | You can see how models are used as building blocks from left to right to transform your data from raw sources into cleaned-up modular derived pieces and final outputs on the far right of the DAG. The default view is 2+model+2 (defaults to display 2 nodes away), however, you can change it to +model+ (full DAG). Note the `--exclude` flag isn't supported. | | **Status bar** | This area provides you with useful information about your Studio IDE and project status. You also have additional options like enabling light or dark mode, restarting the Studio IDE, or [recloning your repo](https://docs.getdbt.com/docs/platform/git/version-control-basics.md). | | **Dark mode** | From the status bar in the Studio IDE, enable dark mode for a great viewing experience in low-light environments. | ##### Code generation The Studio IDE comes with **CodeGenCodeLens**, a powerful feature that simplifies creating models from your sources with a click of a button. To use this feature, click on the **Generate model** action next to each table in the source YAML file(s). It automatically creates a basic starting staging model for you to expand on. This feature helps streamline your workflow by automating the first steps of model generation. ##### dbt YAML validation The Studio IDE uses JSON Schema from the open source [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema) project to power YAML completion, structure checks, and inline diagnostics. These schemas are aligned with the dbt v2 specification and apply across [dbt platform release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), even when your development environment is running dbt v1. If a warning looks wrong but dbt commands succeed, trust your project commands and report the discrepancy so the schema can be improved. For background on the latest Semantic Layer YAML shape, see [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). ##### dbt Wizard [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) helps you develop trusted dbt projects faster in the Studio IDE. It understands more than the file you’re editing. It uses the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) to understand your project’s lineage, model health, test coverage, and semantic definitions, so it can provide answers and suggestions based on your *full* project context. With dbt Wizard, you can: * **Build and refactor models:** Describe what you want to create or change, then review the proposed diff before any files are updated. * **Validate changes as you work:** Compile changes with dbt v2 and run them against your warehouse to catch issues early. * **Understand downstream impact:** See affected models, tests, and metrics *before* making changes. * **Generate YAML resources:** Create documentation, tests, semantic models, and metrics from the Studio IDE console. For data practitioners, this means less time context-switching between files, less manual YAML writing, and faster iteration on complex modeling work — all inside the environment where you already work To get started, refer to [dbt Wizard in the Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. #### Get started with the Studio IDE In order to start experiencing the great features of the Studio IDE, you need to first set up a [dbt development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md). In the following steps, we outline how to set up user credentials and access the Studio IDE. If you're creating a new project, you will automatically configure this during the project setup. The Studio IDE uses user credentials to connect to your data platform. These user credentials should be specific to your user and they should *not* be super user credentials or the same credentials that you use for your production deployment of dbt. Set up your user credentials: 1. Navigate to your **Credentials** under **Your Profile** settings, which you can access at `https://YOUR_ACCESS_URL/settings/profile#credentials`, replacing `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. 2. Select the relevant project in the list. 3. Click **Edit** on the bottom right of the page. 4. Enter the details under **User credentials**. 5. Click **Save.** [![Configure user credentials in your profile](/img/docs/dbt-platform/refresh-ide/dev-credentials.png?v=2 "Configure user credentials in your profile")](#)Configure user credentials in your profile 6. Navigate to the Studio IDE by clicking **Studio** in the left menu. 7. Initialize your project and familiarize yourself with the Studio IDE and its delightful [features](#studio-ide-features). Nice job, you're ready to start developing and building models 🎉! ##### Considerations * To improve your experience using dbt, we suggest that you turn off ad blockers. This is because some project file names, such as `google_adwords.sql`, might resemble ad traffic and trigger ad blockers. * To preserve performance, there's a file size limitation for repositories over 6 GB. If you have a repo over 6 GB, please contact [dbt Support](mailto:support@getdbt.com) before running dbt. * The Studio IDE's idle session timeout is one hour. *  About the start up process and work retention The following sections describe the start-up process and work retention in the Studio IDE. * ###### Start-up process There are three start-up states when using or launching the Studio IDE: * **Creation start —** This is the state where you are starting the IDE for the first time. You can also view this as a *cold start* (see below), and you can expect this state to take longer because the git repository is being cloned. * **Cold start —** This is the process of starting a new develop session, which will be available for you for one hour. The environment automatically turns off one hour after the last activity. This includes compile, preview, or any dbt invocation, however, it *does not* include editing and saving a file. * **Hot start —** This is the state of resuming an existing or active develop session within one hour of the last activity. <br /> <br /> * ###### Work retention The Studio IDE needs explicit action to save your changes. There are three ways your work is stored: * **Unsaved, local code —** The browser stores your code only in its local storage. In this state, you might need to commit any unsaved changes in order to switch branches or browsers. If you have saved and committed changes, you can access the "Change branch" option even if there are unsaved changes. But if you attempt to switch branches without saving changes, a warning message will appear, notifying you that you will lose any unsaved changes. [![If you attempt to switch branches without saving changes, a warning message will appear, telling you that you will lose your changes.](/img/docs/dbt-platform/platform-ide/ide-unsaved-modal.png?v=2 "If you attempt to switch branches without saving changes, a warning message will appear, telling you that you will lose your changes.")](#)If you attempt to switch branches without saving changes, a warning message will appear, telling you that you will lose your changes. * **Saved but uncommitted code —** When you save a file, the data gets stored in durable, long-term storage, but isn't synced back to git. To switch branches using the **Change branch** option, you must "Commit and sync" or "Revert" changes. Changing branches isn't available for saved-but-uncommitted code. This is to ensure your uncommitted changes don't get lost. * **Committed code —** This is stored in the branch with your git provider and you can check out other (remote) branches. #### Build and document your projects * **Build, compile, and run projects** — You can *build*, *compile*, *run*, and *test* dbt projects using the command bar or **Build** button. Use the **Build** button to quickly build, run, or test the model you're working on. The Studio IDE will update in real time when you run models, tests, seeds, and operations. * If a model or test fails, dbt makes it easy for you to view and download the run logs for your dbt invocations to fix the issue. * Use dbt's [rich model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) to [run dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) directly within dbt. * Leverage [environments variables](https://docs.getdbt.com/docs/build/environment-variables.md#special-environment-variables) to dynamically use the Git branch name. For example, using the branch name as a prefix for a development schema. * Run [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) to create and manage metrics in your project with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). * **Develop with dbt Wizard:** [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is the new and recommended AI agent for governed data development in dbt. It uses your project context to help you develop governed dbt changes faster. [dbt Copilot](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md#dbt-copilot-in-studio-ide) is separate from dbt Wizard and is dbt's inline AI assistance experience, providing single-click generation of SQL, documentation, tests, and semantic models in Studio IDE, Canvas, and Insights. * **Build and view your project's docs** — The Studio IDE makes it possible to [build and view](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) documentation for your dbt project while your code is still in development. With this workflow, you can inspect and verify what your project's generated documentation will look like before your changes are released to production. #### Related docs * [How we style our dbt projects](https://docs.getdbt.com/best-practices/how-we-style/0-how-we-style-our-dbt-projects.md) * [User interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) * [Version control basics](https://docs.getdbt.com/docs/platform/git/version-control-basics.md) * [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) #### FAQs  Is there a cost to using the Studio IDE? Not at all! You can use dbt when you sign up for the [Free Developer plan](https://www.getdbt.com/pricing/), which comes with one developer seat. If you'd like to access more features or have more developer seats, you can upgrade your account to the Starter, Enterprise, or Enterprise+ plan.<br /> Refer to [dbt pricing plans](https://www.getdbt.com/pricing/) for more details.  What should I do if the Studio IDE freezes when I open the Lineage tab? If the Studio IDE freezes with two or more models open and the **Lineage** tab active, especially when a model has a large or slow-loading DAG, refer to [Studio IDE freezes when opening the Lineage tab](https://docs.getdbt.com/faqs/Troubleshooting/studio-ide-freezes-lineage-tab.md) for workarounds and troubleshooting steps.  Can I be a contributor to dbt As a proprietary product, dbt's source code isn't available for community contributions. If you want to build something in the dbt ecosystem, we encourage you to review [this article](https://docs.getdbt.com/community/contributing/contributing-coding.md) about contributing to a dbt package, a plugin, the `dbt` repo, or this documentation site. Participation in open-source is a great way to level yourself up as a developer, and give back to the community.  What is the difference between developing on the Studio IDE, the dbt platform CLI, and dbt CLI? You can develop dbt using the web-based IDE in the dbt platform or on the command line interface using the dbt platform CLI or dbt CLI, all of which enable you to execute dbt commands. The key distinction between the dbt platform CLI and dbt CLI is the dbt platform CLI is tailored for the dbt platform's infrastructure and integrates with all its features: * Studio IDE: [dbt](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features) is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project. * dbt platform CLI: [The dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation) allows you to run dbt commands against your dbt platform development environment from your local command line or code editor. It supports cross-project ref, speedier, lower-cost builds, automatic deferral of build artifacts, and more. * dbt CLI: dbt CLI is free, self-hosted software used to manage your dbt project locally on your machine. You can build your dbt project in a code editor, and run dbt commands from the command line --- ### Access, Regions, & IP addresses dbt platform Except as indicated below, dbt is [hosted](https://docs.getdbt.com/docs/platform/about-platform/architecture.md) in multiple regions across the following service providers: * [Amazon Web Services](#AWS) * [Google Cloud Platform](#GCP) * [Microsoft Azure](#Azure) Your dbt account will connect to your data platform or git provider from the below IP addresses. Be sure to allow traffic from these IPs in your firewall, and include them in any database grants. * [dbt Enterprise-tier](https://www.getdbt.com/pricing/) plans can choose to have their account hosted in any of the regions listed in the following table. * Organizations **must** choose a single region per dbt account. To run dbt in multiple regions, we recommend using multiple dbt accounts. [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is currently only available in a Google Cloud Platform US region. For more information, refer to [How is data stored in dbt State?](https://docs.getdbt.com/faqs/State/data-storage.md) #### Amazon Web Services (AWS) | Region | Location | Access URL | IP addresses | Available plans | Status page link | | ------------------------------------ | --------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | North America | AWS us-east-1 (N. Virginia) | ACCOUNT\_PREFIX.us1.dbt.com | 52.45.144.63<br />54.81.134.249<br />52.22.161.231<br />52.3.77.232<br />3.214.191.130<br />34.233.79.135 | [All dbt platform plans](https://www.getdbt.com/pricing/) | **Multi-tenant:**<br />[US East AWS](https://status.getdbt.com/us-aws)<br /><br />**Cell based:**<br />[US East Cell 1 AWS](https://status.getdbt.com/us-cell-1-aws)<br />[US East Cell 2 AWS](https://status.getdbt.com/us-cell-2-aws)<br />[US East Cell 3 AWS](https://status.getdbt.com/us-cell-3-aws)<br />[US East Cell 4 AWS](https://status.getdbt.com/us-cell-4-aws) | | North America | AWS us-west-2 (Oregon) | ACCOUNT\_PREFIX.us5.dbt.com | 32.185.64.51<br />34.217.173.137<br />52.37.223.167 | All Enterprise plans | [US West Cell 1 AWS](https://status.getdbt.com/us-5-cell-1-aws) | | EMEA | eu-central-1 (Frankfurt) | ACCOUNT\_PREFIX.eu1.dbt.com | 3.123.45.39<br />3.126.140.248<br />3.72.153.148 | All Enterprise plans | [EMEA AWS](https://status.getdbt.com/emea-aws) | | APAC | ap-southeast-2 (Sydney) | ACCOUNT\_PREFIX.au1.dbt.com | 52.65.89.235<br />3.106.40.33<br />13.239.155.206<br /> | All Enterprise plans | [APAC AWS](https://status.getdbt.com/apac-aws) | | Japan | ap-northeast-1 (Tokyo) | ACCOUNT\_PREFIX.jp1.dbt.com | 35.76.76.152<br />54.238.211.79<br />13.115.236.233<br /> | All Enterprise plans | [JP Cell 1 AWS](https://status.getdbt.com/jp-cell-1-aws) | | Virtual Private dbt or Single tenant | Customized | Customized | Ask [Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support) for your IPs | All Enterprise plans | Customized | #### Google Cloud Platform (GCP) | Region | Location | Access URL | IP addresses | Available plans | Status page link | | ------------------------------------ | ----------- | --------------------------- | --------------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------- | | North America | us-central1 | ACCOUNT\_PREFIX.us3.dbt.com | 34.33.2.0/26 | All Enterprise plans | [US Cell 1 GCP](https://status.getdbt.com/us-cell-1-gcp) | | EMEA | London | ACCOUNT\_PREFIX.eu3.dbt.com | 34.39.41.0/26 | All Enterprise plans | [EU Cell 1 GCP](https://status.getdbt.com/eu-cell-1-gcp) | | EMEA | Frankfurt | ACCOUNT\_PREFIX.eu4.dbt.com | 34.185.244.128/26 | All Enterprise plans | [EU4 Cell 1 GCP](https://status.getdbt.com/eu-4-cell-1-gcp) | | Virtual Private dbt or Single tenant | Customized | Customized | Ask [Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support) for your IPs | All Enterprise plans | Customized | #### Microsoft Azure | Region | Location | Access URL | IP addresses | Available plans | Status page link | | ------------------------------------ | ---------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------- | | North America | East US 2 (Virginia) | ACCOUNT\_PREFIX.us2.dbt.com | 20.10.67.192/26 | All Enterprise plans | [US Cell 1 AZURE](https://status.getdbt.com/us-cell-1-azure) | | EMEA | North Europe (Ireland) | ACCOUNT\_PREFIX.eu2.dbt.com | 20.13.190.192/26 | All Enterprise plans | [EMEA Cell 1 AZURE](https://status.getdbt.com/emea-cell-1-azure) | | APAC | Australia East | ACCOUNT\_PREFIX.au2.dbt.com | 20.11.97.208/28 | All Enterprise plans | [AUS Cell 1 AZURE](https://status.getdbt.com/au-2-cell-1-azure) | | Virtual Private dbt or Single tenant | Customized | Customized | Ask [Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support) for your IPs | All Enterprise plans | Customized | #### Accessing your account The recommended way to sign in to dbt platform is <https://login.dbt.com>. Enter your email, verify it, and select the account you want to open from the list of accounts associated with your email. For more information about the login process, refer to [Log in to dbt platform](https://docs.getdbt.com/docs/platform/about-platform/login.md). If you already know your account **Access URL**, you can sign in directly. Your access URL depends on your region and tenancy: [![dbt accounts](/img/docs/dbt-platform/find-account.png?v=2 "dbt accounts")](#)dbt accounts * **US multi-tenant:** `ACCOUNT_PREFIX.us1.dbt.com` (for example, `abc123.us1.dbt.com`) * **EMEA multi-tenant:** `ACCOUNT_PREFIX.eu1.dbt.com` (for example, `abc123.eu1.dbt.com`) * **APAC multi-tenant:** `ACCOUNT_PREFIX.au1.dbt.com` (for example, `abc123.au1.dbt.com`) * **Worldwide single-tenant and VPC:** Use the vanity URL provided during your onboarding. Refer to the [tables earlier](#AWS) for the full list of access URLs by region. #### Locating your dbt IP addresses There are two ways to view your dbt platform IP addresses: * If no projects exist in the account, create a new project, and the IP addresses will be displayed during the **Configure your environment** steps. * If you have an existing project, navigate to **Account Settings** and ensure you are in the **Projects** pane. Click on a project name, and the **Project Settings** window will open. Locate the **Connection** field and click on the name. Scroll down to the **Settings**, and the first text block lists your IP addresses. ##### Static IP addresses dbt platform is hosted on AWS, Azure, and the Google Cloud Platform (GCP). While we can offer static URLs for access, we cannot provide a list of IP addresses to configure connections due to the nature of these cloud services. * Dynamic IP addresses — dbt platform provides static URLs for streamlined access, but the dynamic nature of cloud services means the underlying IP addresses can change occasionally. Cloud providers manage these IP ranges and may update them based on operational and security needs. * Using hostnames for consistent access — To ensure uninterrupted access, use dbt platform hostnames. Hostnames provide a consistent reference point, such as `abc123.us1.dbt.com`, even if underlying IP addresses change. This aligns with an industry-standard practice used by platforms such as Snowflake. * Optimizing VPN connections — You should integrate a proxy alongside VPN for users who leverage VPN connections. This strategy enables steady IP addresses for your connections, facilitating smooth traffic flow through the VPN and onward to dbt platform. By employing a proxy and a VPN, you can direct traffic through the VPN and then to dbt platform. It's crucial to set up the proxy if you need to integrate with additional services. #### API Access URLs For dbt platform accounts with cell-based account prefixes, account API access URLs are unique per account. You can find these URLs in **Account settings**, under **Account information**. [![Access URLs in the account settings](/img/docs/dbt-platform/access-urls.png?v=2 "Access URLs in the account settings")](#)Access URLs in the account settings These URLs are unique to each account and begin with the same prefix as the URL used to [access your account](#accessing-your-account). The URLs cover the following APIs: * Admin API (via access URL) * Semantic Layer JDBC API * Semantic Layer GraphQL API * Discovery API Learn more about these features in our [API documentation](https://docs.getdbt.com/docs/dbt-apis/overview.md). --- ### Billing FAQs dbt platform Common questions about dbt plans, billing, and usage. For pricing and plan details, refer to [Billing](https://docs.getdbt.com/docs/platform/billing.md).  What happens if I need more seats on the Starter plan? If you need more developer seats, select the [Contact Sales](https://www.getdbt.com/contact) option from the billing settings to talk to our sales team about an Enterprise or Enterprise+ plan.  What if I go significantly over my included free models on the Starter or Developer plan? Consider upgrading to a Starter or Enterprise-tier plan. Starter and Enterprise-tier plans include more models and allow you to exceed the monthly usage limit. Enterprise accounts are supported by a dedicated account management team and offer annual plans, custom configurations, and negotiated usage rates.  I want to upgrade my plan. Will all of my work carry over? Yes. Your dbt account will be upgraded without impacting your existing projects and account settings.  What happens when I upgrade from Developer to Starter? The free two-week trial is only available to new accounts at signup — but if you're already on the Developer plan, you can upgrade to Starter and immediately access all Starter features without needing to go through the trial. When you upgrade from Developer to Starter, you move directly to paid billing — you'll enter your payment information and select your developer license seats to complete the upgrade. Your existing projects and settings carry over.  How do I determine the right plan for me? The best option is to consult with our sales team. They'll help you figure out what is right for your needs. We also offer a free two-week trial on the Starter plan. #### dbt Wizard dbt Wizard is billed on usage: metered per token and drawn from your consumption pool. Enterprise and Enterprise+ accounts get a monthly pool; all other plans start with a 30-day trial and a $100 pool. For trials, consumption pools, spend limits, and BYOK billing, refer to [dbt Wizard billing FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). --- ### Build with dbt Copilot dbt platform | Enterprise, Enterprise+ Use dbt Copilot to build visual models in the Canvas with natural language prompts. dbt Copilot integrates with [Canvas](https://docs.getdbt.com/docs/platform/canvas.md), a drag-and-drop experience for building visual models with natural language prompts. #### Considerations dbt Copilot in Canvas has setup and AI requirements. Confirm the following before you begin. * dbt Copilot is available in the Canvas interface. Refer to [About Canvas](https://docs.getdbt.com/docs/platform/canvas.md) for setup instructions. * Natural language prompts in Canvas are available on [Enterprise and Enterprise+](https://www.getdbt.com/pricing) plans. * Your development environment must be on a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). * Refer to [Billing](https://docs.getdbt.com/docs/platform/billing.md) for dbt Copilot usage limits. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. To begin building models with natural language prompts in the Canvas: 1. Click on the **dbt Copilot** icon in Canvas menu. 2. In the dbt Copilot prompt box, enter your prompt in natural language for dbt Copilot to build the model(s) you want. You can also reference existing models using the `@` symbol. For example, to build a model that calculates the total price of orders, you can enter `@orders` in the prompt and it'll pull in and reference the `orders` model. 3. Click **Generate** and dbt Copilot generates a summary of the model(s) you want to build. * To start over, click on the **+** icon. To close the prompt box, click **X**. [![Enter a prompt in the dbt Copilot prompt box to build models using natural language](/img/docs/dbt-platform/copilot-generate.jpg?v=2 "Enter a prompt in the dbt Copilot prompt box to build models using natural language")](#)Enter a prompt in the dbt Copilot prompt box to build models using natural language 4. Click **Apply** to generate the model(s) in the Canvas. 5. dbt Copilot displays a visual "diff" view to help you compare the proposed changes with your existing code. Review the diff view in the canvas to see the generated operators built by dbt Copilot: * White: Located in the top of the canvas and means existing set up or blank canvas that will be removed or replaced by the suggested changes. * Green: Located in the bottom of the canvas and means new code that will be added if you accept the suggestion. <br /> [![Visual diff view of proposed changes](/img/docs/dbt-platform/copilot-diff.jpg?v=2 "Visual diff view of proposed changes")](#)Visual diff view of proposed changes 6. Reject or accept the suggestions 7. In the **generated** operator box, click the play icon to preview the data 8. Confirm the results or continue building your model. [![Use the generated operator with play icon to preview the data](/img/docs/dbt-platform/copilot-output.jpg?v=2 "Use the generated operator with play icon to preview the data")](#)Use the generated operator with play icon to preview the data 9. To edit the generated model, open **dbt Copilot** prompt box and type your edits. 10. Click **Submit** and dbt Copilot will generate the revised model. Repeat steps 5-8 until you're happy with the model. --- ### Change your dbt theme dbt platform dbt supports **Light mode** (default), **Dark mode**, and **System mode** (respects your browser's theme for light or dark mode) under the **Theme** section of your user profile and is available for all [plans](https://www.getdbt.com/pricing). You can seamlessly switch between these modes directly from the profile menu, customizing your viewing experience. Your selected theme is stored in your user profile, ensuring a consistent experience across dbt. Theme selection applies across all areas of dbt, including the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), [environments](https://docs.getdbt.com/docs/environments-in-dbt.md), [jobs](https://docs.getdbt.com/docs/deploy/jobs.md), and more. Learn more about customizing themes in [Change themes in dbt](https://docs.getdbt.com/docs/platform/about-platform/change-your-dbt-theme.md#change-themes-in-dbt). #### Change themes in dbt To switch to dark mode in the dbt UI, follow these steps: 1. Navigate to your account name at the bottom left of your account. 2. Under **Theme**, select **Dark**. [![Enable dark mode](/img/docs/dbt-platform/using-dbt-platform/dark-mode.png?v=2 "Enable dark mode")](#)Enable dark mode And that’s it! 🎉 Your chosen selected theme will follow you across all devices. To revert to **Light mode** or **System mode**, repeat the same steps and select your preferred theme. --- ### Configure and use the dbt platform CLI dbt platform Learn how to configure the dbt platform CLI for your dbt project to run dbt commands, like `dbt environment show` to view your dbt configuration or `dbt compile` to compile your project and validate models and tests. You'll also benefit from: * Secure credential storage in the dbt platform. * [Automatic deferral](https://docs.getdbt.com/docs/platform/about-defer.md) of build artifacts to your project's production environment. * Speedier, lower-cost builds. * Support for Mesh ([cross-project ref](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md)), and more. #### Prerequisites * You must set up a project in dbt. * **Note** — If you're using the dbt platform CLI, you can connect to your [data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) directly in the dbt platform interface and don't need a [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) file. * You must have your [personal user credentials](https://docs.getdbt.com/docs/dbt-platform-environments.md#set-developer-credentials) configured in **Account settings** assigned to that project. The dbt platform CLI will use these credentials, stored securely in dbt, to communicate with your data platform. * You must be on dbt version 1.5 or higher. Refer to [dbt versions](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to upgrade. #### Configure the dbt platform CLI Once you install the dbt platform CLI, you need to configure it to connect to a dbt project. 1. In dbt, select the project you want to configure your dbt platform CLI with. The project must already have a [development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment) set up. 2. Download your `dbt_cloud.yml` credentials file and save it to your `.dbt` directory (`~/.dbt/dbt_cloud.yml` on macOS/Linux, `C:\Users\yourusername\.dbt\dbt_cloud.yml` on Windows). For download steps, the full file structure, and a field reference, refer to [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md). 3. After downloading the config file and creating your directory, navigate to a project in your terminal: ```bash cd ~/dbt-projects/jaffle_shop ``` 4. In your `dbt_project.yml` file, ensure you have or include a [`dbt-cloud` block](https://docs.getdbt.com/reference/dbt_cloud.yml.md#the-dbt-cloud-block-in-dbt_projectyml) with a `project-id` field that points to your dbt project. ```yaml # dbt_project.yml dbt-cloud: project-id: PROJECT_ID ``` 5. You should now be able to [use the dbt platform CLI](#use-the-dbt-cli) and run [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) like [`dbt environment show`](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) to view your dbt configuration details or `dbt compile` to compile models in your dbt project. With your repo recloned, you can add, edit, and sync files with your repo. #### Set environment variables To set environment variables in the dbt platform CLI for your dbt project: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Under the **Your profile** section, select **Credentials**. 3. Click on your project and scroll to the **Environment variables** section. 4. Click **Edit** on the lower right and then set the user-level environment variables. #### Use the dbt platform CLI The dbt platform CLI uses the same set of [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) and [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) as dbt v1 to execute the commands you provide. For example, use the [`dbt environment`](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) command to view your dbt configuration details. With the dbt platform CLI, you can: * Run [multiple invocations in parallel](https://docs.getdbt.com/reference/dbt-commands.md) and ensure [safe parallelism](https://docs.getdbt.com/reference/dbt-commands.md#parallel-execution). * Automatically defer build artifacts to your project's production environment. * Support [project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md), which allows you to depend on another project using the metadata service in dbt. * Project dependencies instantly connect to and reference (or `ref`) public models defined in other projects. You don't need to execute or analyze these upstream models yourself. Instead, you treat them as an API that returns a dataset. Use the `--help` flag As a tip, most command-line tools have a `--help` flag to show available commands and arguments. Use the `--help` flag with dbt in two ways: * `dbt --help`: Lists the commands available for dbt <br /> * `dbt run --help`: Lists the flags available for the `run` command #### Lint SQL files From the dbt platform CLI, you can invoke [SQLFluff](https://sqlfluff.com/), which is a modular and configurable SQL linter that warns you of complex functions, syntax, formatting, and compilation errors. Many of the same flags that you can pass to SQLFluff are available from the dbt platform CLI. The available SQLFluff commands are: * `lint` — Lint SQL files by passing a list of files or from standard input (stdin). * `fix` — Fix SQL files. * `format` — Autoformat SQL files. To lint SQL files, run the command as follows: ```text dbt sqlfluff lint [PATHS]... [flags] ``` When you don't specify a path, dbt lints all SQL files in the current project. To lint a specific SQL file or a directory, set `PATHS` to the path of the SQL file(s) or directory of files. To lint multiple files or directories, pass multiple `PATHS` flags. To show detailed information on all the dbt supported commands and flags, run the `dbt sqlfluff -h` command. ###### Considerations When running `dbt sqlfluff` from the dbt platform CLI, the following are important behaviors to consider: * dbt reads the `.sqlfluff` file, if it exists, for any custom configurations you might have. * For continuous integration/continuous development (CI/CD) workflows, your project must have a `dbt_cloud.yml` file and you have successfully run commands from within this dbt project. * An SQLFluff command will return an exit code of 0 if it ran with any file violations. This dbt behavior differs from SQLFluff behavior, where a linting violation returns a non-zero exit code. dbt Labs plans on addressing this in a later release. #### Considerations The dbt platform CLI doesn't currently support relative paths in the [`packages.yml` file](https://docs.getdbt.com/docs/build/packages.md). Instead, use the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), which supports relative paths in this scenario. Here's an example of a [local package](https://docs.getdbt.com/docs/build/packages.md#local-packages) configuration in the `packages.yml` that won't work with the dbt platform CLI: ```yaml # repository_root/my_dbt_project_in_a_subdirectory/packages.yml packages: - local: ../shared_macros ``` In this example, `../shared_macros` is a relative path that tells dbt to look for: * `..` — Go one directory up (to `repository_root`). * `/shared_macros` — Find the `shared_macros` folder in the root directory. To work around this limitation, use the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), which fully supports relative paths in `packages.yml`. #### FAQs  How to create a .dbt directory and move your file If you've never had a `.dbt` directory, you should perform the following recommended steps to create one. If you already have a `.dbt` directory, move the `dbt_cloud.yml` file into it. Some information about the `.dbt` directory: * A `.dbt` directory is a hidden folder in the root of your filesystem. It's used to store your dbt configuration files. The `.` prefix is used to create a hidden folder, which means it's not visible in Finder or File Explorer by default. * To view hidden files and folders, press Command + Shift + G on macOS or Ctrl + Shift + G on Windows. This opens the "Go to Folder" dialog where you can search for the `.dbt` directory. ##### Create a .dbt directory 1. Clone your dbt project repository locally. 2. Use the `mkdir` command followed by the name of the folder you want to create. * If using macOS, add the `~` prefix to create a `.dbt` folder in the root of your filesystem: * macOS: `mkdir ~/.dbt` * Windows: `mkdir %USERPROFILE%\.dbt` ##### Move the dbt\_cloud.yml file You can move the `dbt_cloud.yml` file into the `.dbt` directory using the `mv` command or by dragging and dropping the file into the `.dbt` directory by opening the Downloads folder using the "Go to Folder" dialog and then using drag-and-drop in the UI. To move the file using the terminal, use the `mv/move` command. This command moves the `dbt_cloud.yml` from the `Downloads` folder to the `.dbt` folder. If your `dbt_cloud.yml` file is located elsewhere, adjust the path accordingly. ###### Mac or Linux In your command line, use the `mv` command to move your `dbt_cloud.yml` file into the `.dbt` directory. If you've just downloaded the `dbt_cloud.yml` file and it's in your Downloads folder, the command might look something like this: ```bash mv ~/Downloads/dbt_cloud.yml ~/.dbt/dbt_cloud.yml ``` ###### Windows In your command line, use the move command. Assuming your file is in the Downloads folder, the command might look like this: ```bash move %USERPROFILE%\Downloads\dbt_cloud.yml %USERPROFILE%\.dbt\dbt_cloud.yml ```  How to skip artifacts from being downloaded By default, the dbt platform CLI downloads [all artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) when you execute dbt commands. To skip these files from being downloaded, add `--download-artifacts=false` to the command you want to run. This can help improve run-time performance but might break workflows that depend on assets like the [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md).  I'm getting a \`Session occupied\` error in dbt platform CLI? If you're receiving a `Session occupied` error in the dbt platform CLI or if you're experiencing a long-running session, you can use the `dbt invocation list` command in a separate terminal window to view the status of your active session. This helps debug the issue and identify the arguments that are causing the long-running session. To cancel an active session, use the `Ctrl + Z` shortcut. To learn more about the `dbt invocation` command, see the [dbt invocation command reference](https://docs.getdbt.com/reference/commands/invocation.md?version=2.0). Alternatively, you can reattach to your existing session with `dbt reattach` and then press `Control-C` and choose to cancel the invocation. --- ### Configure BYOK for dbt Wizard in dbt platform dbt platform | Starter, Enterprise, Enterprise+ Use bring-your-own-key (BYOK) to connect dbt Wizard or dbt Copilot in dbt platform to your own AI provider account instead of using dbt Labs' managed infrastructure. The following BYOK instructions apply to dbt platform only. For CLI BYOK setup, refer to [Configure BYOK for dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md). When you configure a provider with your own key, usage costs appear on your provider account instead of your dbt Labs account, and token costs are billed by whichever provider you choose. If you'd rather skip that upkeep, the [dbt Labs-managed option](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider) is ready to use with no setup, and comes with [trial and monthly usage credits](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). #### Prerequisites * A [dbt platform account](https://www.getdbt.com/pricing) on a Starter, Enterprise, or Enterprise+ plan * dbt admin permissions to enable AI features and configure providers in **Account settings** * An API key or credentials for your supported AI provider  See the full list of supported AI providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). ###### dbt Copilot dbt Copilot supports different AI providers, including bring your own key (BYOK) on Starter, Enterprise, and Enterprise+ plans: * dbt Labs-managed OpenAI API key * BYOK OpenAI API key * BYOK Azure OpenAI API key Snowflake Cortex, AWS Bedrock, Azure AI Foundry, and Anthropic aren't supported for dbt Copilot. #### Configure AI provider A dbt platform account on any plan can use a dbt managed provider to get started right away. Accounts on Starter, Enterprise, and Enterprise+ plans can also configure a custom AI provider by BYOK. If you use BYOK, you will incur API calls and associated charges from that provider. The following instructions explain how to configure a dbt Labs managed or BYOK AI provider for dbt Wizard. dbt managed AI providers are administered by dbt Labs and don’t require a user-provided API key. The AI providers shown in **Account settings** may change as dbt Labs adds managed models. For the current list of models and providers, refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model). **To configure a provider:** 1. Click your account name and select **Account settings** in the side menu. 2. Under **Settings**, click **AI features**. 3. Under **AI providers**, click **Edit** to configure the AI integration. 4. For each provider, select your **Key management** option from the dropdown, then follow these steps for your provider below. ##### Anthropic **Managed by dbt Labs** Use this option as a managed alternative to the default OpenAI model. 1. Unless not selected, select **dbt Labs** from the list to use dbt Labs' managed Anthropic key. 2. Click **Save**. **Managed by you (BYOK)** 1. Select **Anthropic** from the options. 2. Enter your Anthropic API key. 3. Click **Save**. Embedding limitations When using an Anthropic API key, dbt continues to use the dbt Labs-managed OpenAI key for embeddings in `text_to_sql` MCP tools, since Anthropic doesn't natively provide embeddings. ##### OpenAI **Managed by dbt Labs** This is the default option and use it if you want to use dbt managed AI provider keys. Refer to [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md?version=2.0) for more information. 1. Unless not selected, select the toggle for **dbt Labs** to use a dbt Labs' managed key. 2. Click **Save**. [![Example of the dbt Labs integration page](/img/docs/dbt-platform/account-integration-dbtlabs.png?v=2 "Example of the dbt Labs integration page")](#)Example of the dbt Labs integration page **Managed by you (BYOK)** 1. Select **OpenAI** from the options. 2. Enter your OpenAI API key. 3. Click **Save**. [![Example of the OpenAI integration page](/img/docs/dbt-platform/account-integration-openai.png?v=2 "Example of the OpenAI integration page")](#)Example of the OpenAI integration page Data residency limitation OpenAI projects with [data residency controls](https://platform.openai.com/docs/guides/your-data#data-residency-controls) enabled and configured for the United States (project region set to US) don't currently support BYOK. These projects can only use the API key in the dbt platform configuration. Specifying custom endpoints required for data residency isn’t yet supported, and we’re evaluating a solution for this. To use BYOK, ensure your OpenAI project doesn’t have data residency controls enabled. Projects without project region settings will use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. * For BYOK, enable the latest text generation models as well as the `text-embedding-3-small` model. * Ensure your project doesn't have data residency controls enabled. Projects without project region settings use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. ##### Azure AI Foundry **Managed by you only (BYOK)** To learn about deploying models on Azure, refer to [Deploy models on Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-openai) and [Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/). Configure credentials for your Azure AI Foundry deployment in dbt as follows: 1. Locate your deployment details in the Azure portal, either in Azure AI Foundry or Azure OpenAI. 2. Enter your Azure API key. 3. Enter the **Endpoint**, **API Version**, and **Deployment / AI model name**. 4. Click **Save**. Use the full Azure Target URI For the **Endpoint** field, enter the full Azure Target URI from Azure — not just the base endpoint. Entering only the base endpoint, for example `https://<resource>.openai.azure.com`, prevents credential validation and blocks setup. Supported formats include: * `https://<resource>.openai.azure.com/openai/deployments/<deployment>/chat/completions?api-version=<version>` * `https://<resource>.openai.azure.com/openai/responses?api-version=<version>` [![Example of the Azure AI Foundry integration section](/img/docs/dbt-platform/account-integration-azure-manual.png?v=2 "Example of the Azure AI Foundry integration section")](#)Example of the Azure AI Foundry integration section --- ### dbt AI: Usage metering and limiting dbt platform dbt AI usage is metered differently depending on the feature you use: What's changing on September 1, 2026 From September 1, 2026, a couple of things are changing for dbt AI features: * **AI features are being enabled by default.** They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can turn AI features on or off anytime in **Account settings**. * **dbt Wizard is moving to usage-based billing** for [dbt-managed AI](#dbt-managed-inference). Usage is metered per token against your consumption pool, and an admin can set a monthly spend limit in dbt platform. Refer to [dbt Wizard billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) for more info. The [Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) is billed through consumption pool pricing when you use dbt-managed inference, or directly by your provider when you use BYOK. ##### Usage and metering information | Feature | How it's metered | What limits usage | | --------------------------- | -------------------------------------------------- | ---------------------------------------------------- | | [dbt Wizard](#dbt-wizard) | Dollar-based usage, converted from tokens | Your consumption pool, then your monthly spend limit | | [dbt Copilot](#dbt-copilot) | A count of completed AI requests, known as actions | A monthly action allotment set by your plan | Bring your own key (BYOK) usage isn't metered by dbt. Your AI provider bills you directly and the usage doesn't draw from either limit. #### dbt Wizard dbt Wizard is metered by dollar-based usage rather than a count of actions. Usage is measured in tokens — prompts, project context, cached content, and generated responses all consume them — and converted into a dollar amount based on the model and token type. That amount is deducted from your consumption pool. Usage from dbt platform and the local CLI both draw from the same account-level pool. ##### Usage credits by plan All credits are account-level Usage credits are granted per account, not per user or per seat. Everyone on your account draws from the same balance, whether they use dbt Wizard in dbt platform or the local CLI. | Plan | What you get | | ----------------------------------------- | ------------------------------------------------------------------ | | Enterprise | $100/month in usage credits per account, resets each billing month | | Enterprise+ | $200/month in usage credits per account, resets each billing month | | All other plans and self hosted dbt users | One-time 30-day trial with $100 in usage credits per account | Pools don't roll over. For eligibility, how to start a trial, and how to set up paid access afterward, refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). ##### When you reach your limit Your spend limit caps how much dbt managed dbt Wizard usage your account can consume in a billing period. You only pay for actual usage, up to the limit you choose. * If you use up your consumption pool on a Developer, Starter, or self-hosted plan, usage pauses until you add paid usage or the next billing cycle starts. * Enterprise and Enterprise+ accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. * Limits are set separately for dbt Wizard and [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), but both draw from your account's overall usage-based spend. ##### View Wizard usage 1. Navigate to [**Account settings**](https://docs.getdbt.com/docs/platform/account-settings.md). 2. Select **Billing & Usage** under the Settings header. 3. On the **Overview** tab, check the **Consumption pool** card, or open **Usage-based features > Wizard** for usage and spend controls. You need to be an account admin or billing admin to view or change spend limits. For trials, consumption pools, spend limits, and BYOK billing, refer to [dbt Wizard billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). #### dbt Copilot dbt Copilot usage is measured by the number of completed AI requests, known as dbt Copilot actions. A defined number of actions is allocated monthly based on your [subscription plan](https://www.getdbt.com/pricing). Once you reach the limit, dbt Copilot is temporarily disabled until the start of the next billing cycle. ##### View Copilot actions 1. Navigate to [**Account settings**](https://docs.getdbt.com/docs/platform/account-settings.md). 2. Select **Billing** under the Settings header. 3. On the billing page, click the **Copilot Actions** tab to view your usage. [![View usage in dbt Copilot](/img/docs/dbt-platform/view-usage-in-copilot.gif?v=2 "View usage in dbt Copilot")](#)View usage in dbt Copilot #### FAQs  Temporary dbt Copilot Actions bridge (ended September 1, 2026) As a temporary compatibility bridge, dbt Wizard drew from your existing dbt Copilot included action allotment. That bridge ended on September 1, 2026, and dbt Wizard usage is now metered separately as dollar-based usage against your usage credits and consumption pool. Users who bring their own key (BYOK) were never affected by this bridge.  What counts as a dbt Copilot action dbt Copilot actions refer to requests made to the dbt Copilot assistant through the dbt interface. These actions are recorded and displayed on the billing page alongside other usage metrics by accessing the **Copilot Actions** tab in the **Billing** page. The following interactions count as dbt Copilot actions: * **Each inline generation** — Every time dbt AI writes or suggests code in your file, it counts toward your usage limit. * **Each generation of documentation, tests, semantic models, or metrics** — Any time you ask dbt AI to automatically create things like documentation, tests, data models, or metrics, it counts as one interaction. * **Each generation within dbt AI chats on Canvas or Insights** — Any time you use dbt Copilot chat in Canvas or Insights to generate something, it counts as an interaction.  Allowed limits on number of dbt Copilot actions per month per license The following table outlines the limits of dbt Copilot actions by plan per month: | Plan | Limit | | --------------------- | ------ | | Developer | ❌ | | Starter\* | 100 | | Legacy Enterprise\*\* | 1,000 | | Enterprise | 5,000 | | Enterprise+ | 10,000 | \*Team plan customers who enrolled in dbt Copilot Beta prior to March 19, 2025 have access to dbt Copilot. All other legacy Team plan customers must move to the [Starter plan or above](https://www.getdbt.com/pricing) to get access. \*\*Enterprise-tier customers on plans enrolled prior to May 1, 2025 (including legacy Business Critical) have a limit of 1,000 dbt Copilot actions per month. To get a higher allotment, move to the current [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing).  Notifications when limits are reached When usage limits are reached, a notification appears in the UI. Additionally, an email notification is sent to the designated recipient. For users on the Starter plan, the account owner receives an email notification when the usage limit is reached. For users enrolled on the Enterprise and Enterprise+ plans, both the billing administrator and the account administrator are notified by email when the usage limit is reached. Once usage limits are reached, attempts to perform an action in dbt Copilot triggers a banner notification indicating that the limit has been exceeded. Under Bring Your Own Key (BYOK), usage is not tracked by dbt AI and is subject to your OpenAI limits. #### Related docs * [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) to start a dbt Wizard trial and set a spend limit * [dbt Wizard billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) for common billing questions * [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md) for model options and token pricing * [Billing](https://docs.getdbt.com/docs/platform/billing.md) for general dbt platform billing --- ### dbt Architecture dbt platform This page helps practitioners and those interested in dbt's architecture and data flow. #### About dbt architecture The dbt application has two types of components: static and dynamic. The static components are always running to serve highly available dbt functions, like the dbt web application. On the other hand, the dynamic components are created ad-hoc to handle tasks such as background jobs or requests to use the Studio IDE. dbt is available in most regions around the world in both [single tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#single-tenant) (AWS, Azure, and GCP) and [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#multi-tenant) configurations. dbt uses PostgreSQL for its backend, S3-compatible Object Storage systems for logs and artifacts, and a Kubernetes storage solution for creating dynamic, persistent volumes. All data at rest on dbt servers is protected using AES-256 encryption. ![](/img/docs/dbt-platform/on-premises/data-flows.png) #### Communication dbt can communicate with several external services, including data platforms, git repositories, authentication services, and directories. All communications occur over HTTPS (attempts to connect via HTTP are redirected to HTTPS). dbt encrypts in transit using the TLS 1.2 cryptographic protocol. TLS (Transport Layer Security) 1.2 is an industry-standard protocol for encrypting sensitive data while it travels over the public internet (which does not offer native encryption). A typical scenario that might be seen frequently is an employee working in a public space, such as an airport or café. The user might be connected to an unsecured public network offered by a facility to which many others are also connected. What if there is a bad actor amongst them running a program that can "capture" network packets and analyze them over the air? When that user is accessing dbt and running models that interact with the data platform, the information sent to and from their computer and the services is encrypted with TLS 1.2. If that user runs a command that initializes communication between dbt and the data warehouse (or a git repo or an auth service) over the internet, that communication is also encrypted. This means that while the bad actor can technically see the traffic moving over that unsecured network, they can't read or otherwise parse any information. They will not be able to eavesdrop on or hack the information in any way whatsoever. They would see a nonsensical set of characters that nobody can decrypt. For more detailed information on our security practices, read our [Security page](https://getdbt.com/security). ##### Data warehouse interaction dbt's primary role is as a data processor, not a data store. The dbt application enables users to dispatch SQL to the warehouse for transformation. However, users can post SQL that returns customer data into the dbt application. This data never persists and will only exist in memory on the instance for the duration of the session. To lock down customer data correctly, proper data warehouse permissions must be applied to prevent improper access or storage of sensitive data. Some data warehouse providers offer advanced security features that can be leveraged in dbt. [Private connections](https://docs.getdbt.com/docs/platform/secure/private-connectivity/private-connectivity.md) allows supported data platforms on AWS to communicate with dbt without the traffic traversing the public internet. [Snowflake](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md) and [BigQuery](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md) offer Oauth integration which adds a layer of security for the data platforms (Enterprise-tier plan only). ##### Git sync dbt can sync with a variety of git providers, including [Github](https://docs.getdbt.com/docs/platform/git/connect-github.md), [Gitlab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md), and [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) within its integrated development environment ([Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md)). Communication takes place over HTTPS rather than SSH and is protected using the TLS 1.2 protocol for data in transit. The git repo information is stored on dbt servers to make it accessible during the Studio IDE sessions. When the git sync is disabled, you must [contact support](mailto:support@getdbt.com) to request the deletion of the synced data. ##### Authentication services The default settings of dbt enable local users with credentials stored in dbt. Still, integrations with various authentication services are offered as an alternative, including [single sign-on services](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md). Access to features can be granted/restricted by role using [RBAC](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-). SSO features are essential because they reduce the number of credentials a user must maintain. Users sign in once and the authentication token is shared among integrated services (such as dbt). The token expires and must be refreshed at predetermined intervals, requiring the user to go through the authentication process again. If the user is disabled in the SSO provider service, their access to dbt is disabled, and they cannot override this with local auth credentials. [Snowflake](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md) and [BigQuery](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md) offer OAuth (JSON to pass info and API calls for auth) services as an alternative to SAML (XML to pass info and session cookies for auth). Users can authenticate against the data platform for secure access to dbt and prevent access when credentials are revoked. #### Security dbt Labs is dedicated to upholding industry standards for Cloud security and GDPR compliance. Our compliance certifications include the following: * SOC2 Type II — assesses a service provider’s security control environment against the trust services principles and criteria set forth by the American Institute of Certified Public Accountants (AICPA). * ISO27001:2013 — a globally recognized standard for establishing and certifying an information security management system (ISMS). * GDPR - dbt Labs is committed to maintaining GDPR compliance standards. Read more about our [Data Processing Addendum](https://www.getdbt.com/cloud/dpa). For more detailed information about our security practices, read our [Security page](https://www.getdbt.com/security/). --- ### dbt platform CLI migration Starting September 14, 2026, `pip install dbt` will install dbt v2 instead of the dbt platform CLI. If you install the dbt platform CLI with pip, you need to take action to keep using it without interruption. #### What's changing Starting September 14, 2026, `pip install dbt` will install v2, dbt's next-generation Rust-based engine, instead of the dbt platform CLI. This is a one-time change to what the `dbt` package on PyPI contains, not a dbt platform CLI upgrade. If you install the dbt platform CLI using pip today and don't pin a version, your next `pip install dbt` or `pip install --upgrade dbt` installs v2 instead. dbt v2 is a different engine: it runs locally against a warehouse connection you configure yourself, instead of running your commands against your dbt platform development environment. Commands like `dbt cancel`, `dbt reattach`, `dbt environment`, `dbt sl export`, and `dbt sqlfluff` don't exist in v2. #### What to do If you use the dbt platform CLI and want to keep using it, reinstall it using the current method for your operating system — Homebrew on macOS, or the native executable on Windows or Linux. Refer to [Install the dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) for full instructions. If you can't reinstall right now (for example, a Docker image that only has pip available), pin your version instead: ```bash pip install dbt==1.0.0.40.20 ``` This keeps working indefinitely, but won't receive further dbt platform CLI updates through pip. Reinstalling with a currently supported method is the long-term path. #### General areas of impact * **Dockerfiles / CI pipelines**: any unpinned `pip install dbt` step. Search your repos for `pip install dbt` without a `==` version pin. * **Airflow**: `BashOperator`/`KubernetesPodOperator` tasks that install the CLI into the worker image at build time. * **`requirements.txt` / `Pipfile` / `pyproject.toml`**: any unpinned `dbt` entry. * **dbt Power User (VS Code extension)**: it shells out to the dbt platform CLI internally. Pin your version the same way; the extension itself doesn't need any separate action. #### FAQ * **Why is dbt Labs doing this?** dbt v2 is the new default `dbt` engine going forward. Publishing it under the `dbt` name on PyPI means new users get v2 by default, matching how `dbt` already behaves everywhere else (Homebrew, standalone install). * **Will my pinned dbt platform CLI version stop working?** No. Pinned installs keep working. They just won't get new dbt platform CLI releases through pip after September 14. For that, reinstall using a currently supported method. * **How do I tell which one I have installed?** `dbt --version`. The dbt platform CLI prints a version like `0.40.20`; the corresponding version on PyPI is `1.0.0.40.20`. dbt v2's version string needs reconfirming post-update. --- ### dbt State usage Login required | Usage-based [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) enables dbt to reuse nodes by cloning from another location or skipping a rebuild when the logic and data haven't changed. It's a separate, usage-based product available to dbt v1, dbt platform, and dbt v2 users. Learn more about how your usage influences the price so you can plan your savings effectively. #### How the trial works * Eligible new organizations receive 30 days of free use with no usage limit. * To start a dbt State trial, you need a dbt account so you can manage dbt State usage, billing, and spend limits from one dashboard. A paid dbt platform plan is *not* required to use dbt State locally. To start your trial, refer to the instructions in [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). * Once started, you cannot pause the trial. * Seven days before your trial ends, your admin gets an email with what's coming: the trial end date, the monthly DATT allowance your account will receive, and how to upgrade if needed. * The [free monthly DATT allowance](#free-monthly-datt-allowance) includes a set amount of daily active target tables for your account. The allowance amount and what happens when you reach the limit depend on your plan. * If you were using state-aware orchestration prior to June 1, 2026, your dbt State trial will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. #### How billing works ##### Daily active target tables For purposes of pricing, daily active target tables (DATT) are measured as the number of distinct target tables (as defined below) for which dbt State performs at least one of the following unique operations on a given day (based on UTC time): a skip, clone, or test reuse. A target table is a database object managed by your dbt project for a given database and schema name. It includes seeds, snapshots, dbt models (including incremental models). It also includes each distinct test (even if the tests are not built into the database because `store_failures` is disabled). For example, if `stg_customers` has `not_null` and `unique` tests on its `id` column, that's three target tables: the model and its two tests. When you run `dbt build` or a similar command, a target table is selected for execution. It counts as an active target table if dbt State can reuse it based on your configuration rules. All reuses of the same active target table in a single day (based on UTC time) count as a single daily active target table (DATT). ##### Monthly cost calculation dbt State calculates cost per billing period using the unit price (USD $0.094) x sum of daily active target tables (DATT) for all account users and all days in that billing period. For example, if you have 100 DATT in a billing period, you'll be billed for 100 \* $0.094 = $9.40. For current unit price and more information, refer to the [dbt Labs Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table). ##### DATT chart You can view your DATT count in **Account settings** > **Billing & Usage** > **Usage-based features**. Under the **State** tab, the DATT chart shows DATTs split into **Billable** and **Free**. During a trial, all DATTs are counted as free. DATTs within your [monthly allowance](#free-monthly-datt-allowance) are also counted as free and do *not* appear on your invoice. ##### Cancellation Usage is tracked through your cancellation date. You're billed at month end for usage incurred before cancellation and not charged for usage after. #### Free monthly DATT allowance Every account gets a monthly allowance of free daily active target tables (DATT). Usage within your allowance doesn’t appear on your invoice. When your trial ends, you move to your plan's monthly allowance: * Self-hosted dbt users move to the free plan, which includes 1,000 DATT each month indefinitely. * Developer plan users also get 1,000 DATT each month indefinitely. * Paid dbt platform plans get a temporary monthly allowance. Check out [paid plans with a temporary allowance](#paid-plans-with-a-temporary-allowance) for more details. The allowance works the same way on every plan: * It resets on the 1st of each month. * Unused DATT don't carry over. * Your first month gets the full allowance for your plan. Usage during your trial doesn't count against it. ###### Plans with a permanent allowance | Plan | Monthly allowance | When you reach the limit | | --------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Self-hosted dbt (free plan) | 1,000 DATT | dbt State stops reusing nodes until the next reset. [Set up usage-based billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md#setting-up-usage-based-billing) to resume. | | Developer | 1,000 DATT | dbt State stops reusing nodes until the next reset. [Set up usage-based billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md#setting-up-usage-based-billing) to resume, which moves your account to the Developer (Usage) plan. | ###### Paid plans with a temporary allowance Paid dbt platform plans also get a monthly allowance for a limited time. | Plan | Monthly allowance | Available until | When you reach the limit | | ------------------------------------------ | ----------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Starter | 1,000 DATT | March 31, 2027 | **Card on file:** dbt State keeps running and bills your card for usage past the limit.<br />**No card on file:** dbt State stops reusing nodes until the next reset. | | Enterprise, Enterprise+, legacy Enterprise | 2,000 DATT | March 31, 2027 | **Prepaid State contract:** dbt State keeps running and draws from your contract.<br />**No prepaid contract:** dbt State stops reusing nodes until the next reset. | | Legacy Teams | 1,000 DATT | December 31, 2026 | dbt State stops reusing nodes until the next reset. To continue past the limit, migrate to Starter. | tip On Starter, you can pause dbt State at the limit by turning off card billing for State on the **Usage-based features** page. If you cancel paid access, dbt State keeps running on your plan's monthly allowance. ##### Viewing your allowance Go to **Account settings** > **Billing & Usage** > **Usage-based features** > **State** tab to see your current consumption, remaining allowance, and reset date. dbt sends an email to your admin when you've used up 80% of your DATT allowance, and again if dbt State pauses at the limit. --- ### dbt Wizard home tab Preview ### dbt Wizard home tab [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based The dbt Wizard home tab is an agent-native development experience in dbt platform. Use the dbt Wizard home tab to investigate and generate changes with natural language prompts, review inline diffs and DAG previews, and validate changes without leaving the agent. AI features are being enabled by default. They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can [turn AI off or back on and configure providers](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) anytime. The dbt Wizard home tab is complementary to the [dbt Wizard experience in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). Where the Studio IDE supports users working directly within a traditional IDE environment, the home tab is purpose-built for agent-native development and keeps you focused on supervising and validating agent-generated work. [![dbt Wizard home tab — empty state with quick-start prompts](/img/docs/dbt-platform/wizard-home-empty.png?v=2 "dbt Wizard home tab — empty state with quick-start prompts")](#)dbt Wizard home tab — empty state with quick-start prompts [![dbt Wizard agent refactoring a docs github model for tech writers :) ](/img/docs/dbt-platform/wizard-home-agent.png?v=2 "dbt Wizard agent refactoring a docs github model for tech writers :) ")](#)dbt Wizard agent refactoring a docs github model for tech writers :) #### Prerequisites * A [dbt account](https://www.getdbt.com/signup) and [Developer or Read-only seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * [Legacy Team plans](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) don't have access to dbt Wizard. Move to a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing) to use it. * A [development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) and credentials set up in the Studio IDE. * Use a supported AI provider. Refer to [Supported AI providers](https://docs.getdbt.com/docs/platform/wizard-platform.md#supported-ai-providers), or the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full model list and rates. * If you're using dbt Wizard in the home tab, you need to [enable experimental features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) for your account. If dbt Wizard stops responding, your account may have used up its usage credits. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). #### What you can do Use dbt Wizard in the home tab to: * **Answer project-aware questions**: Ask about lineage, dependencies, model logic, and project context. * **Debug failed jobs**: Investigate run and job failures using dbt Agent Skills with full project context. * **Create and manage branches**: Initiate and switch branches directly from the agent workflow. * **Make model and project changes**: Refactor SQL, update YAML, and modify project configuration through natural language. * **Generate and refine transformation logic**: Build or rewrite models, tests, documentation, and semantic definitions from plain-language prompts. * **Run validation workflows**: Execute compile and build checks to validate proposed changes before they're persisted. * **Ask questions of production data**: Use [Explore mode](#agent-modes) to query governed metrics and models in plain language, and see the SQL or metric definition behind every answer. This is the main surface for [read-only users](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md). * **Choose your model**: Select the dbt managed model you'd like to work with from the [model picker](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#choose-a-model). Best practices for using dbt Wizard Refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows — including [debugging a failed job](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-5-debug-failed-job.md), which applies directly to the home tab. #### Agent modes The dbt Wizard operates in three modes: | Mode | Behavior | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Explore only** | The agent queries and explains data but can't edit files or run builds. Best when you want to analyze data or validate a model's output without the agent proposing changes. Every answer comes with the SQL or metric definition behind it. Available for read-only users in the home tab. | | **Ask for approval** (default) | The agent drafts edits to files. You approve each file change before it is persisted. Best when you want tight control over what gets saved to your branch. | | **Edit files automatically** | The agent drafts and automatically saves file edits without per-file approval. Best for faster iteration when you're confident in the prompt. | Switch between modes anytime with the **Agent mode** button (bottom-left). The authoring modes keep the analytical tools available, so switching out of **Explore only** doesn't cost you anything. #### Ask questions in Explore mode Explore mode in the dbt Wizard home tab lets you ask questions of your production data in plain language. dbt Wizard in Explore mode answers and explains but never changes your project. This option is great for exploratory data analysis and getting a quick understanding of your data. Explore mode queries with your personal warehouse credentials. If you don't have any, it falls back to the project's [analytics credential](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md#set-up-analytics-credentials), a shared credential an admin sets up on Snowflake, BigQuery, Redshift, or Databricks connections. This is how [read-only users](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) query without their own credentials. If neither exists, dbt Wizard asks you to set up credentials or contact an admin. Explore mode answers are grounded in what your project defines, so it needs metadata or Semantic Layer definitions to answer against. For what makes answers better, refer to [Set your team up for good answers](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md#set-your-team-up-for-good-answers). 1. Open dbt Wizard and set the mode picker (bottom-left) to **Explore only** if needed. 2. Type your question and press **Enter**. Make sure you're specific and include the time period, grouping, and filter you care about: * `what was total revenue in Q2 2026, by month?` * `how many new customers signed up in July?` * `which regions grew fastest this year?` 3. dbt Wizard gives a plain-language summary of what it did, then the result. Switch between **Chart**, **Table**, and **SQL** to see the data your way. 4. Validate the data by using the **SQL** view to review the query or governed metric behind the answer. Note that you can't edit the SQL query but you can copy it. 5. Keep going by asking a follow-up in the same conversation. For example, `now break that out by region` works after your first question. Explore mode uses dbt-managed inference, so questions draw from your account's dbt Wizard [consumption pool](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) like any other dbt Wizard usage. In the home tab, click **Open** on a result to view the visualization in the right-hand context sidebar, and download the data from there. [![Explore mode in the dbt Wizard home tab, with a plain-language summary, a chart, and Chart/Table/SQL toggles.](/img/docs/dbt-platform/wizard-home-explore-viz.png?v=2 "Explore mode in the dbt Wizard home tab, with a plain-language summary, a chart, and Chart/Table/SQL toggles.")](#)Explore mode in the dbt Wizard home tab, with a plain-language summary, a chart, and Chart/Table/SQL toggles. #### Inline preview mode A core part of the home tab experience is **inline preview mode**, which gives you multiple ways to review and validate agent-generated changes directly in the workflow without switching to a separate tool. The preview experience includes: * **Enhanced SQL diffs**: Review proposed code changes side by side before accepting them. * **Structural DAG visualizations**: Inspect transformations as operator-level DAG views that break logic into familiar patterns — joins, filters, aggregations, and projections — making it easier to understand how a model changed beyond the raw SQL. * **Execution-aware validation feedback**: See results from compile and build checks inline, so you can assess both the proposed implementation and how the transformation behaves in practice. * **Jump to related surfaces**: From the preview, open a model directly in Catalog to explore metadata and lineage, or open the file in Studio IDE to iterate manually when needed. #### When to use the home tab or Studio IDE The home tab and Studio IDE support different parts of the development workflow: | | Home tab | Studio IDE | | -------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------- | | **Primary workflow** | Supervise and validate agent-generated work | Write and edit code directly in the editor | | **Interface** | Streamlined agent-native chat | Full IDE with file explorer, editor, console | | **Best for** | Natural language iteration, reviewing diffs, running validations | Manual file edits, direct SQL authoring, complex multi-file work | | **Inline preview** | ✅ SQL diffs, DAG visualizations, build feedback | File diffs shown before changes are persisted | For most development workflows, you can move between the two surfaces freely. Use the home tab to investigate and generate, and drop into Studio IDE when you need direct control. #### Choose a model With dbt managed inference, you can switch between the supported managed models at any using the model picker dropdown next to the **Agent mode** control (where you choose **Ask for approval** or **Edit files automatically**). * In [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) and the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), open the model picker dropdown next to the **Agent mode** control in the dbt Wizard panel, then select a model. * The picker lists the managed models available to you. Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the available models and their token rates. * If you [bring your own key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md), dbt Wizard uses the provider and model you configured with your key rather than the managed model picker. [![The model picker dropdown next to the Agent mode control in the Wizard panel.](/img/docs/dbt-platform/wizard-model-picker.png?v=2 "The model picker dropdown next to the Agent mode control in the Wizard panel.")](#)The model picker dropdown next to the Agent mode control in the Wizard panel. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Invite read-only users to dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) * [About dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md) * [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md) --- ### Develop with AI in the Studio IDE dbt platform | Usage-based Leverage AI to develop dbt projects in the Studio IDE. [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) in [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) is the new and recommended way to develop governed dbt projects with AI. It uses your project context to help you develop governed dbt changes faster. Think of it like a smart AI agent that has a map of your project: instead of reading through each file to understand context, it can answer questions and help you develop and validate changes faster. dbt also supports dbt Copilot, a separate inline AI assistance experience for single-click generation of SQL, documentation, tests, and semantic models in Studio IDE, Canvas, and Insights. #### Prerequisites * A [dbt account](https://www.getdbt.com/signup) on any plan and [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * A [development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) and credentials set up in the Studio IDE. AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). #### dbt Wizard in Studio IDE [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Use [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) for autonomous model generation, refactoring, and multi-step workflows in the Studio IDE. It's grounded in the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) — a structured view of your project's lineage, tests, model health, and semantic definitions — so it knows your data, not just your code. dbt Wizard is accessible from the sidebar panel or in the Console section of the Studio IDE.  Availability and considerations * **Where it runs:** Supported in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) only, all [deployment types](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md?version=2.0). Not supported in VS Code or the dbt platform CLI. * **Engines:** Works with dbt v2 and dbt v1. * **Conversations:** In the conversation list, open **More actions** menu (three dots) of the conversation you want to delete, then click **Delete** to remove one thread. Deleting the open thread clears the panel. * **Sessions:** Refreshing the same browser tab keeps your active session. A new tab, or returning after closing the tab, starts empty. * **Chat history:** Retained for 90 days only. Chat history isn't supported yet on single-tenant deployments, so save anything important before closing. * **Plan mode:** Not supported yet. The agent doesn't show a separate plan before applying changes, however you can use the **Ask for approval** mode to approve each file. * **New chat:** Click **Start new dbt Wizard chat** (top right of the dbt Wizard panel) to begin a new session. ##### Using dbt Wizard Use the dbt Wizard panel to generate resources with quick actions, or use the agent to build and refactor models end-to-end with natural language prompts. To use the dbt Wizard, follow these steps: 1. Open your dbt project in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), then click **dbt Wizard** in the command palette. 2. Start a prompt in several ways in the [dbt Wizard panel](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?version=2#panel-controls): * **Quick actions**: Use [quick-action resource generation](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#quick-action-resource-generation) at the top of the panel for quick action prompts. * **Plain text**: Type directly into the text field to describe what you want to build or change. * **Model context**: Type `@` to select a model as context. This scopes the agent's changes to that resource. 3. Select the [**Agent mode** button](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) to specify the mode for the dbt Wizard. Available modes are **Explore only**, **Ask for approval** (default), and **Edit files automatically**. 4. Select the dbt managed model you'd like to work with from the [model picker](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#choose-a-model) next to the **Agent mode** button. 5. [Review the agent's suggestions](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#reviewing-agent-suggestions) and approve or reject the changes. You can also use the **Start new dbt Wizard chat** button to start a new chat session. 6. [Approve dbt commands](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#granting-command-permissions) when the dbt Wizard requests to run commands like `dbt compile` or `dbt build`. 7. Repeat the process to build or change more models. 8. Commit the changes to your dbt project and open a pull request. Be warned, the wizard has been known to cast spells. The following images show how dbt Wizard displays its work and outcome: [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. [![Wizard final refactor result displayed as a diff](/img/docs/dbt-platform/wizard-ide-refactor-diff.png?v=2 "Wizard final refactor result displayed as a diff")](#)Wizard final refactor result displayed as a diff For more details on the dbt Wizard and how it works, expand the following sections to open additional information.  Panel controls The dbt Wizard panel contains: 1. **Quick actions** (center): Buttons at the top of the panel for common tasks like generating documentation, tests, semantic models, and metrics. When selected, the text field is pre-filled with a prompt. 2. **Agent mode button** (bottom left): Switch between **Explore only**, **Ask for approval**, and **Edit files automatically** mode. Click the button to change modes. 3. **Model picker** (bottom left): Select the dbt managed model to use for the session. Refer to [Choose a model](#choose-a-model) for the available models. 4. **dbt model context** (bottom left): Shows the currently open file. Use `@` in the text field to reference a different dbt model. Click **x** to remove the dbt model context. 5. **Text input field** (bottom left): Type your prompt in the text field to describe what you want to build or change. Type `@` to select a dbt model as context. This scopes the agent's changes to that resource. 6. **Start new dbt Wizard chat** (top right): Starts a new chat session. 7. **Stop** or **Enter** (bottom right): Press **Enter** to submit your prompt. Press **Stop** to stop the current session and agent processing. You cannot undo this action. [![The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field.](/img/docs/dbt-platform/wizard-panel.png?v=2 "The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field.")](#)The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field. dbt Wizard also has a simplified wayfinder bar above the text input field. The wayfinder bar shows your current project and branch and guides you through Git tasks, such as committing files or creating a branch. []()  Agent modes The dbt Wizard operates in three modes: | Mode | Behavior | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Explore only** | The agent queries and explains data but can't edit files or run builds. Best when you want to analyze data or validate a model's output without the agent proposing changes. Every answer comes with the SQL or metric definition behind it. Available for read-only users in the home tab. | | **Ask for approval** (default) | The agent drafts edits to files. You approve each file change before it is persisted. Best when you want tight control over what gets saved to your branch. | | **Edit files automatically** | The agent drafts and automatically saves file edits without per-file approval. Best for faster iteration when you're confident in the prompt. | Switch between modes anytime with the **Agent mode** button (bottom-left). The authoring modes keep the analytical tools available, so switching out of **Explore only** doesn't cost you anything. [![dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.](/img/docs/dbt-platform/wizard-modes.png?v=2 "dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.")](#)dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.  Reviewing agent suggestions When the dbt Wizard proposes code changes, you can review them before they are saved to your project: * **View the diff**: The agent displays a diff of the proposed changes. Click **Show all X lines** to expand and view the full suggestion. * **Line indicators**: Added and removed lines are highlighted with line number indicators so you can see exactly what changed. * **Copy or open in editor**: Use the options in the top-right corner of the diff view to copy the suggestion or open it directly in the editor.  Granting command permissions To validate or run models during a session, the agent may request to run dbt commands such as `dbt compile` or `dbt build`. You'll be prompted to approve each request before it executes. For example, the agent might request to run: ```bash dbt compile --select model_name ``` You can select one of the following options: | Option | Behavior | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | **Yes, run once** | Grants permission to run this specific command one time. | | **Yes, and allow `dbt_command_name` for the session** | Grants permission to run dbt commands for the remainder of your session without prompting again. | | **No** | Denies the request. The agent will not run the command. | After you run a command, dbt Wizard adds an icon and a tooltip to the Studio IDE [**Commands** tab](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#console-section) results. This helps you distinguish agent-run commands from manually run commands in the run results and logs. ##### Bringing your own skills You can extend dbt Wizard with custom skills to encode your team's SQL conventions, naming rules, and modeling workflows — so you don't repeat them in every prompt. See [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-skills.md) for the full reference, including how to create, structure, and invoke skills. ##### Debug job failures The dbt Wizard can investigate and troubleshoot dbt job and run failures directly from the Studio IDE. This capability is powered by the `troubleshooting-dbt-job-errors` [dbt Agent Skill](https://github.com/dbt-labs/dbt-agent-skills), which comes pre-configured with the agent — no setup required. You can ask the agent questions and issue commands like: * "What jobs have failed recently?" * "What is the root cause of the job failure?" * "How can I fix the recent job failure?" * "Fix the job failure." The agent notes when your local project state may differ from the job — for example, if you're on a different branch or have uncommitted changes — so you have full context before acting on any suggested fixes. ##### Timeout handling When a dbt command run by dbt Wizard runs for more than 5 minutes, the agent automatically attempts to stop the command on the server before returning control to you. Instead of hanging or showing a generic error, the agent returns a clear message that explains the command timed out and was aborted. The message also tells you whether the cancellation request succeeded. If cancellation fails, it's possible the command may still be running on the server. You can then choose whether to retry the command, narrow the request, or take another action. ##### dbt v2 migration workflow If you have access to [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) with [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled, you can use the [dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) skill. This skill can help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed. It classifies every error, applies validated fixes automatically, and surfaces what's blocked. info The dbt v2 migration workflow is accessible through the dbt Wizard in the Studio IDE. If you're using VS Code or the dbt platform CLI, use the [autofix tool](https://docs.getdbt.com/guides/dbt-package-compat?step=4) instead. 1. From the job list, click the **Review job** button for a job with a successful run. * If you don't see the **Review job** button, enable the **Show dbt v2 eligibility** toggle in the job list. 2. In the **dbt v2 eligibility unknown for this job** pop-up, click **Debug in Studio with dbt Wizard**. 3. dbt redirects you to the Studio IDE and sets your personal development environment to dbt v2. 4. dbt Wizard opens and automatically triggers the dbt v2 migration skill with this prompt: ```text I need help fixing dbt v2 compatibility issues in this project. Please investigate and resolve any deprecation warnings or incompatibilities. Please use the migrating-dbt-core-to-fusion skill to guide this. ``` 5. Review and approve dbt Wizard's permission requests so it can run the commands it needs. 6. The dbt Wizard iteratively runs `dbt compile`, reads the results, and applies fixes until it reaches a successful compile or encounters an error it can't resolve. If it gets blocked, it exits cleanly, explains what it could not fix, and creates and links to a markdown file summarizing all changes made. 7. When the project compiles with no warnings or errors, commit and publish your changes. 8. After you merge the changes, wait for the job to run again or run it manually on dbt v2. [![The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.](/img/docs/dbt-platform/v2-migration-workflow.gif?v=2 "The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.")](#)The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE. For more on how to prepare your project for dbt v2 and what to do when you hit compatibility errors, see the [dbt v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) and the [Upgrade to dbt v2 guides](https://docs.getdbt.com/guides/prepare-v2-upgrade.md). ##### Writing effective prompts Good prompts include the *scope* (which models or area of the project), the *intent* (the transformation or business logic you want), and any *constraints* (naming conventions, materialization, tests). Here are a few examples: | Task | Example prompt | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Build a new model | "Create a model called `fct_daily_revenue` that joins `stg_orders` and `stg_payments`, aggregates revenue by day, and materializes as a table." | | Refactor an existing model | "Refactor `fct_orders` to use incremental materialization. Keep existing tests and follow our naming conventions." | | Generate tests and docs | "Add `not_null` and `unique` tests to the primary key of `dim_customers`, and generate documentation for all columns." | For detailed guidance, patterns, and more examples across SQL, documentation, tests, and semantic models, see the [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md). #### dbt Copilot in Studio IDE *The earlier version of dbt Copilot in the Studio IDE is available only to a limited set of accounts. dbt Wizard is available to all accounts and is the recommended way to develop with AI in the Studio IDE — it covers everything dbt Copilot's quick actions did, plus multi-step changes with built-in validation.* [dbt Copilot](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) provides single-click generation of SQL, documentation, tests, and semantic models in Studio IDE, Canvas, and Insights. 1. Navigate to the Studio IDE and select a SQL model file under the **File Explorer**. 2. In the **Console** section, click the **dbt Copilot** button to view the available AI options. 3. Select the available options to [generate resources](#generate-resources) such as using the quick-action buttons to generate documentation, tests, semantic models, or metrics. The following sections describe how to use dbt Copilot in the Studio IDE. ##### Generate resources To generate resources with dbt Copilot, follow these steps: Generate documentation, tests, metrics, and semantic models [resources](https://docs.getdbt.com/docs/build/projects.md) with the click-of-a-button in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) using dbt Copilot, saving you time. To access and use this AI feature: 1. Navigate to the Studio IDE and select a SQL model file under the **File Explorer**. 2. In the **Console** section (under the **File Editor**), click the dbt Copilot button and choose what you want to generate, such as documentation, tests, metrics, SQL, or semantic models. To generate multiple YAML configs for the same model, select each option separately. dbt Copilot saves them to the same YAML file. note dbt Copilot doesn't yet support generating semantic models with the latest YAML spec. * To generate metrics, you need to first have semantic models defined. * Once defined, click **dbt Copilot** and select **Generate metrics**. * Write a prompt describing the metrics you want to generate and press enter. * **Accept** or **Reject** the generated code. 3. Verify the AI-generated code. You can update or fix the code as needed. 4. Click **Save As**. You should see the file changes under the **Version control** section. ##### Inline SQL editing dbt Copilot supports a quick inline prompt window for targeted SQL edits directly in the editor. Press **Cmd+B** (Mac) or **Ctrl+B** (Windows) to open the prompt window within a model file, describe what you want to generate or change, and dbt Copilot displays a diff of the proposed changes — click **Accept** to apply or **Reject** to discard. This is useful for scoped edits within a single SQL file. For larger or multi-step changes — like building a new model, refactoring across files, or generating YAML, review how [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) can help instead. --- ### Develop with dbt Develop dbt projects using the dbt platform, a faster and more reliable way to deploy dbt and manage your project in a single, web-based UI. You can develop in your browser using a dbt-powered command line interface (CLI), an integrated development environment (Studio IDE), or Canvas. #### Getting started To get started, you'll need a [dbt](https://www.getdbt.com/signup) account and a developer license. For a more comprehensive guide about developing in dbt, refer to the [quickstart guides](https://docs.getdbt.com/docs/get-started-dbt.md). Choose the option that best fits your needs: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) ###### [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) [Allows you to develop and run dbt commands from your local command line or code editor against your dbt development environment.](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) [![](/img/icons/vsce.svg)](https://docs.getdbt.com/docs/about-dbt-extension.md) ###### [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) [Bring the speed and intelligence of dbt v2 to VS Code for a seamless development experience.](https://docs.getdbt.com/docs/about-dbt-extension.md) [![](/img/icons/dashboard.svg)](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) ###### [dbt Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) [Develop dbt projects directly in your browser with seamless SQL compilation and an intuitive, visual workflow.](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) [![](/img/icons/canvas.svg)](https://docs.getdbt.com/docs/platform/canvas.md) ###### [dbt Canvas](https://docs.getdbt.com/docs/platform/canvas.md) [Develop with Canvas, a seamless drag-and-drop experience that helps analysts quickly create and visualize dbt models in dbt.](https://docs.getdbt.com/docs/platform/canvas.md) <br /> --- ### Edit and create dbt models dbt platform | Enterprise, Enterprise+ Access and use Canvas to create or edit dbt models through a visual, drag-and-drop experience. #### Access Canvas Before accessing the editor, you should have a dbt project already set up. This includes a Git repository, data platform connection, environments, and user credentials. If you don't have this set up, please contact your dbt Admin. Access **Canvas** at any time from the left-side menu. #### Canvas prerequisites Before using Canvas, you should: * Have a [dbt Enterprise or Enterprise+](https://www.getdbt.com/pricing) account. * Have a [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) with user credentials set up. * Be using one of the following adapters: * Bigquery * Databricks * Redshift * Snowflake * Trino * You can access the Canvas with adapters not listed, but some features may be missing at this time. * Use [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md), [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md), or [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) as your Git provider, connected to dbt via HTTPS. * SSH connections aren't supported at this time. * Self-hosted or on-premises deployments of any Git provider aren't supported for Canvas at this time. * Have an existing dbt project already created with a Staging or Production run completed. * Verify your Development environment is on a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing updates. * Have read-only access to the [Staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) with the data to be able to execute `run` in the Canvas. To customize the required access for the Canvas user group, refer to [Set up environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md) for more information. #### Create a model To create a dbt SQL model, click on **Create a new model** and perform the following steps. Note that you can't create source models in Canvas. This is because you need to have production run with sources already created. 1. Drag an [operator](https://docs.getdbt.com/docs/platform/canvas-interface.md#operators) from the operator toolbar and drop it onto the canvas. 2. Click on the operator to open its configuration panel. 3. View the **Output** and **SQL Code** tabs. * Each operator has an Output tab that allows you to preview the data from that configured node. * The Code tab displays the SQL code generated by the node's configuration. Use this to see the SQL for your visual model config. 4. Connect the operators by using the connector by dragging your cursor between the operator's "+" start point and linking it to the other operators you want to connect to. This should create a connector line. * Doing this allows the data to flow from the source table through various transformations you configured, to the final output. 5. Keep building your dbt model and ensure you confirm the output through the **Output** tab. #### Edit an existing model To edit an existing model: 1. Navigate to a Canvas workspace. 2. Click **+Add** on the top navigation bar. 3. Click **Edit existing model**. This will allow you to select the model you'd like to edit. [![Edit a model using the 'Edit a model' button.](/img/docs/dbt-platform/canvas/edit-model.png?v=2 "Edit a model using the 'Edit a model' button.")](#)Edit a model using the 'Edit a model' button. #### Upload data to Canvas You can upload a CSV file of source data for model creation directly from Canvas: 1. Click **+Add**. 2. Select **Upload CSV source**. 3. Drag your file to the canvas area or click **Upload** to select from your file explorer. This uploads the data to your data warehouse in a new table in your developer schema. It will be prefixed with `VE_UPLOADS_`. In the canvas window, it creates a source operator and a basic SQL model that you can customize. You can now work with this data in both Canvas and the Studio IDE. #### Test and document Testing and documenting your models is an important part of the development process. Stay tuned! Coming very soon, you'll be able to test and document your dbt models in Canvas. This ensures you can maintain high data quality and clarity around how your models ought to be used. --- ### Fix deprecation warnings dbt platform You can address deprecation warnings in the dbt platform by finding and fixing them using the autofix tool in the Studio IDE. You can run the autofix tool on the [**v1 Compatible** or **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) of dbt v1 before you upgrade to dbt v2! To find and fix deprecations: 1. Navigate to the Studio IDE by clicking **Studio** in the left menu. 2. Make sure to save and commit your work before proceeding. The autofix tool may overwrite any unsaved changes. 3. Click the three-dot menu located at the bottom right corner of the Studio IDE. 4. Select **Check & fix deprecations**. [![Access the Studio IDE options menu to autofix deprecation warnings](/img/docs/dbt-platform/platform-ide/ide-options-menu-with-save.png?v=2 "Access the Studio IDE options menu to autofix deprecation warnings")](#)Access the Studio IDE options menu to autofix deprecation warnings The tool performs a `dbt parse —show-all-deprecations —no-partial-parse` to find the deprecations in your project. 5. If you don't see the deprecations and the **Autofix warnings** button, click the command history in the bottom left: [![Access recent commands to see the autofix button](/img/docs/dbt-platform/platform-ide/command-history.png?v=2 "Access recent commands to see the autofix button")](#)Access recent commands to see the autofix button 6. When the command history opens, click the **Autofix warnings** button: [![Learn what deprecations need to be auto fixed](/img/docs/dbt-platform/platform-ide/autofix-button.png?v=2 "Learn what deprecations need to be auto fixed")](#)Learn what deprecations need to be auto fixed 7. When the **Proceed with autofix** dialog opens, click **Continue** to begin resolving project deprecations and start a follow-up parse to show remaining deprecations. [![Proceed with autofix](/img/docs/dbt-platform/platform-ide/proceed-with-autofix.png?v=2 "Proceed with autofix")](#)Proceed with autofix 8. Once complete, a success message appears. (Applies to dbt v2.0 and later) After a successful `dbt parse` command, you'll see a **Compile** button to the right of the **Successfully resolved** result. Use **Compile** to compile your project from the results panel. [![Autofix success with Compile in the v2 flow](/img/docs/dbt-platform/platform-ide/autofix-success-v2-compile.png?v=2 "Autofix success with Compile in the v2 flow")](#)Autofix success with Compile in the v2 flow If successful, you'll see a **Successfully compiled** result. If you see any errors, review them and make any necessary changes. 9. Click **Commit and sync** in the top left of Studio IDE to commit these changes to the project repository. 10. You are now ready to enable dbt v2 if you [meet the requirements](https://docs.getdbt.com/docs/dbt/supported-features.md#requirements)! #### Related docs * [Quickstart guide](https://docs.getdbt.com/guides.md) * [About dbt](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) * [Develop in the Cloud](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) --- ### Git commit signing dbt platform | Enterprise, Enterprise+ To prevent impersonation and enhance security, you can sign your Git commits before pushing them to your repository. Using your signature, a Git provider can cryptographically verify a commit and mark it as "verified", providing increased confidence about its origin. You can configure dbt to sign your Git commits when using the Studio IDE for development. To set up, enable the feature in dbt, follow the flow to generate a keypair, and upload the public key to your Git provider to use for signature verification. #### Prerequisites * GitHub or GitLab is your Git provider. Currently, Azure DevOps is not supported. * You have a dbt account on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/). #### Generate GPG keypair in dbt To generate a GPG keypair in dbt, follow these steps: 1. Go to your **Personal profile** page in dbt. 2. Navigate to **Signed Commits** section. 3. Enable the **Sign commits originating from this user** toggle. 4. This will generate a GPG keypair. The private key will be used to sign all future Git commits. The public key will be displayed, allowing you to upload it to your Git provider. [![Example of profile setting Signed commits](/img/docs/dbt-platform/example-git-signed-commits-setting.png?v=2 "Example of profile setting Signed commits")](#)Example of profile setting Signed commits #### Upload public key to Git provider To upload the public key to your Git provider, follow the detailed documentation provided by the supported Git provider: * [GitHub instructions](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account) * [GitLab instructions](https://docs.gitlab.com/ee/user/project/repository/signed_commits/gpg.html) Once you have uploaded the public key to your Git provider, your Git commits will be marked as "Verified" after you push the changes to the repository. [![Example of a verified Git commit in a Git provider.](/img/docs/dbt-platform/git-sign-verified.png?v=2 "Example of a verified Git commit in a Git provider.")](#)Example of a verified Git commit in a Git provider. #### Considerations * The GPG keypair is tied to the user, not a specific account. There is a 1:1 relationship between the user and keypair. The same key will be used for signing commits on any accounts the user is a member of. * The GPG keypair generated in dbt is linked to the email address associated with your account at the time of keypair creation. This email identifies the author of signed commits. * For your Git commits to be marked as "verified", your dbt email address must be a verified email address with your Git provider. The Git provider (such as, GitHub, GitLab) checks that the commit's signed email matches a verified email in your Git provider account. If they don’t match, the commit won't be marked as "verified." * Keep your dbt email and Git provider's verified email in sync to avoid verification issues. If you change your dbt email address: * Generate a new GPG keypair with the updated email, following the [steps mentioned earlier](https://docs.getdbt.com/docs/platform/studio-ide/git-commit-signing.md#generate-gpg-keypair-in-dbt-cloud). * Add and verify the new email in your Git provider. #### FAQs  What happens if I delete my GPG keypair in dbt? If you delete your GPG keypair in dbt, your Git commits will no longer be signed. You can generate a new GPG keypair by following the [steps mentioned earlier](https://docs.getdbt.com/docs/platform/studio-ide/git-commit-signing.md#generate-gpg-keypair-in-dbt-cloud).  What Git providers support GPG keys? GitHub and GitLab support commit signing, while Azure DevOps does not. Commit signing is a [git feature](https://git-scm.com/book/ms/v2/Git-Tools-Signing-Your-Work), and is independent of any specific provider. However, not all providers support the upload of public keys, or the display of verification badges on commits.  What if my Git provider doesn't support GPG keys? If your Git Provider does not explicitly support the uploading of public GPG keys, then commits will still be signed using the private key, but no verification information will be displayed by the provider.  What if my Git provider requires that all commits are signed? If your Git provider is configured to enforce commit verification, then unsigned commits will be rejected. To avoid this, ensure that you have followed all previous steps to generate a keypair, and uploaded the public key to the provider. --- ### How does dbt pricing work? dbt platform As a customer, you pay for the number of seats you have and the amount of usage consumed each month. Seats are billed primarily on the amount of licenses purchased. Usage is based on the number of [Successful Models Built](#what-counts-as-a-successful-model-built) and, if purchased and used, Semantic Layer [Queried Metrics](#what-counts-as-a-queried-metric) subject to reasonable usage. All billing computations are conducted in Coordinated Universal Time (UTC). ##### What counts as a seat license? You can learn more about allocating users to your account in [Users and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). There are four types of possible seat licenses: * **Analyst**\* — for permission sets assigned and shared amongst those who don't need day-to-day access. * **Developer** — for permission sets that require day-to-day interaction with the dbt platform. * **IT** — for access to specific features related to account management (for example, configuring git integration). * **Read-Only** — for access to view certain documents and reports. \* The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. ##### What counts as a Successful Model Built? A Successful Model Built is any model successfully built in a dbt deployment environment through dbt’s orchestration. This includes jobs run via the scheduler, CI builds (triggered by pull requests), and runs kicked off via the dbt API. Models that build successfully are counted even if the overall run later fails. For example, if a job containing 100 models fails after 51 are built, only those 51 are counted. Any models built in a dbt development environment (for example, via the Studio IDE) do not count towards your usage. Tests, seeds, ephemeral models, and snapshots also do not count. When a dynamic table is initially created, the model is counted (if the creation is successful). However, in subsequent runs, dbt skips these models unless the definition of the dynamic table has changed. This refers not to changes in the SQL logic but to changes in dbt's logic, specifically those governed by [`on_configuration_change config`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md)). The dynamic table continues to update on a cadence because the adapter is orchestrating that refresh rather than dbt. | What counts towards Successful Models Built | | | ------------------------------------------- | -- | | View | ✅ | | Table | ✅ | | Incremental | ✅ | | Ephemeral Models | ❌ | | Tests | ❌ | | Seeds | ❌ | | Snapshots | ❌ | <br /> When [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is enabled, a reused model (skipped or cloned) doesn't count as a Successful Model Built. They're billed separately as daily active target tables (DATTs). For more information, refer to [dbt State usage](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md). ##### What counts as a Queried Metric? The Semantic Layer, powered by MetricFlow, measures usage in distinct Queried Metrics. * Every successful request you make to render or run SQL to the Semantic Layer API counts as at least one queried metric, even if no data is returned. * If the query calculates or renders SQL for multiple metrics, each calculated metric will be counted as a queried metric. * If a request to run a query is not executed successfully in the data platform or if a query results in an error without completion, it is not counted as a queried metric. * Requests for metadata from the Semantic Layer are also not counted as queried metrics. Examples of queried metrics include: * Querying one metric, grouping by one or more dimensions → 1 queried metric ```shell dbt sl query --metrics revenue --group-by metric_time ``` * Querying two metrics, grouping by two dimensions → 2 queried metrics ```shell dbt sl query --metrics revenue,gross_sales --group-by metric_time,user__country ``` Compiling metrics counts the same way — one queried metric per metric compiled (for example, `dbt sl query --metrics revenue --compile` → 1 queried metric). ##### Viewing usage in the product Viewing usage in the product is restricted to specific roles: * Starter plan — Owner group * Enterprise and Enterprise+ plans — Account and billing admin roles If you have access to the **Billing** and **Usage** pages in **Account settings**, you can see an estimate of the month's usage, how your account tracks against it, and which projects are building the most models. [![To view account-level estimated usage, go to 'Account settings' and then select 'Billing'.](/img/docs/building-a-dbt-project/billing-usage-page.jpg?v=2 "To view account-level estimated usage, go to 'Account settings' and then select 'Billing'.")](#)To view account-level estimated usage, go to 'Account settings' and then select 'Billing'. As a Starter and Developer plan user, you can see how the account is tracking against the included models built. As an Enterprise plan user, you can see how much you have drawn down from your annual commit and how much remains. On each **Project home** page, any user with project access can see how many models are built each month, with top jobs by models built available on each **Environment** page. [![Your Project home page displays how many models are built each month.](/img/docs/building-a-dbt-project/billing-project-page.jpg?v=2 "Your Project home page displays how many models are built each month.")](#)Your Project home page displays how many models are built each month. The **Job details** page's **Insights** tab shows models built per month for that job and which take longest to build. [![View how many models are being built per month for a particular job by going to the 'Insights' tab in the 'Job details' page.](/img/docs/building-a-dbt-project/billing-job-page.jpg?v=2 "View how many models are being built per month for a particular job by going to the 'Insights' tab in the 'Job details' page.")](#)View how many models are being built per month for a particular job by going to the 'Insights' tab in the 'Job details' page. Usage data shown in dbt is only an estimate and may be delayed, and some visualizations aren't available on legacy plans. Your final monthly usage appears on your monthly statements (Starter and Enterprise-tier plans). --- ### IDE user interface dbt platform The [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) is a tool for developers to effortlessly build, test, run, and version-control their dbt projects, and enhance data governance — all from the convenience of your browser. Use the Studio IDE to compile dbt code into SQL and run it against your database directly — no command line required! This page offers comprehensive definitions and terminology of user interface elements, allowing you to navigate the Studio IDE landscape with ease. [![The Studio IDE layout includes version control on the upper left, files/folders and search on the left, editor on the right, command palette at the top, and command/console at the bottom](/img/docs/dbt-platform/platform-ide/ide-basic-layout.png?v=2 "The Studio IDE layout includes version control on the upper left, files/folders and search on the left, editor on the right, command palette at the top, and command/console at the bottom")](#)The Studio IDE layout includes version control on the upper left, files/folders and search on the left, editor on the right, command palette at the top, and command/console at the bottom #### Basic layout The Studio IDE streamlines your workflow, and features a popular user interface layout with files and folders on the left, editor on the right, and command and console information at the bottom. ###### The side menu [![The Git repo link, documentation site button, Version Control menu, and File Explorer](/img/docs/dbt-platform/platform-ide/ide-side-menu.png?v=2 "The Git repo link, documentation site button, Version Control menu, and File Explorer")](#)The Git repo link, documentation site button, Version Control menu, and File Explorer 1. **Git repository link:** The Git repository link, located on the upper left of the Studio IDE, takes you to your repository on the same active branch. It also displays the repository name and the active branch name. * **Note:** This linking feature is only available for GitHub or GitLab repositories on multi-tenant dbt accounts. 2. **Documentation site button:** Clicking the Documentation site book icon, located next to the Git repository link, leads to the dbt Documentation site. The site is powered by the latest dbt artifacts generated in the IDE using the `dbt docs generate` command (dbt v1 only) from the Command bar. 3. [**Version Control**](#editing-features): The Studio IDE's powerful Version Control section contains all git-related elements, including the Git actions button and the **Changes** section. 4. **File explorer:** The File explorer shows the filetree of your repository. You can: * Click on any file in the filetree to open the file in the file editor. * Click and drag files between directories to move files. * Right-click a file to access the sub-menu options like copy name, copy relative path, copy as `ref`, download, duplicate, rename, and delete. * Use file indicators, located to the right of your files or folder name, to see when changes or actions were made: * Unsaved (•) — The Studio IDE detects unsaved changes to your file/folder * Modification (M) — The Studio IDE detects a modification of existing files/folders * Added (A) — The Studio IDE detects added files * Deleted (D) — The Studio IDE detects deleted files. ###### The command and status bar [![Use the Command bar to write dbt commands, toggle 'Defer', and view the current IDE status](/img/docs/dbt-platform/platform-ide/ide-command-bar.png?v=2 "Use the Command bar to write dbt commands, toggle 'Defer', and view the current IDE status")](#)Use the Command bar to write dbt commands, toggle 'Defer', and view the current IDE status 5. **Command bar:** The Command bar, located in the lower left of the Studio IDE, is used to invoke [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md). When a command is invoked, the associated logs are shown in the Invocation History Drawer. 6. **Defer menu:** The **Defer menu** allows developers to configure the deferral environment: * **Development environment:** Same as turning defer off. Build all upstream models using only the Development environment. Useful for testing in isolation. * **dbt default behavior:** Uses the Staging environment if it's configured, otherwise uses Production. Recommended configuration for most use cases. * **Custom environment:** Select the defer environment from any available in the project. [![Set the \`defer\` environment.](/img/docs/dbt-platform/platform-ide/defer-menu.png?v=2 "Set the `defer` environment.")](#)Set the \`defer\` environment. Refer to [Using defer in dbt](https://docs.getdbt.com/docs/platform/about-defer.md#defer-in-the-dbt-ide) for more info. 7. **Status:** The Studio IDE Status button, located on the lower right of the Studio IDE, displays the current connection statuses to both the warehouse and the dbt [language server (LSP)](https://docs.getdbt.com/docs/about-dbt-lsp.md) status if you're on dbt v2 or the engine server status if you're on dbt v1. It includes shortcuts to environment settings and user credentials. [![View the connection statuses for your account.](/img/docs/dbt-platform/platform-ide/server-status.png?v=2 "View the connection statuses for your account.")](#)View the connection statuses for your account. 8. **dbt version:** The current version of dbt running in your development environment. You can set a **Personal version override** that changes the dbt version for only your development environment. [![View and set the development environment version.](/img/docs/dbt-platform/platform-ide/dbt-version.png?v=2 "View and set the development environment version.")](#)View and set the development environment version. 9. **Additional tools:** Project status and an options menu with additional actions and information about the Studio IDE: * **Project status:** If there are any errors or warnings outstanding in the development runs, the number of them will populate in this area. Clicking on them will take you to the **Problems** tab. * **Options menu:** Access the options menu by clicking the three-dot menu located at the bottom right corner of the Studio IDE. This menu contains global options: * **View status details:** View more detailed information about your connection status. * **Restart Studio:** You will lose any unsaved information. * **Reinstall dependencies:** Overwrites project dependencies with a fresh installation. * **Clean dbt project:** Runs the [`dbt clean` command](https://docs.getdbt.com/reference/commands/clean.md) for your project. * **Autofix deprecation warnings:** Runs the dbt [autofix tool](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md). Helps prepare for dbt v2 upgrade. [![IDE menu with additional information and actions.](/img/docs/dbt-platform/platform-ide/ide-menu.png?v=2 "IDE menu with additional information and actions.")](#)IDE menu with additional information and actions. #### Search bar and command palette The Studio IDE provides tools to help you quickly navigate your project's files, find information, run commands, and replace syntax with just a few clicks in a layout that's familiar to users of popular IDEs. [![Use the search bar and command palette to quickly navigate your file tree and open tabs.](/img/docs/dbt-platform/platform-ide/search-and-command.png?v=2 "Use the search bar and command palette to quickly navigate your file tree and open tabs.")](#)Use the search bar and command palette to quickly navigate your file tree and open tabs. 1. [Search and replace](#search-and-replace) 2. [Command palette](#command-palette) ##### Search and replace The search feature enables you to quickly find specific terms or phrases and replace them with the click of a button. [![Search files for specific terms and quickly replace them.](/img/docs/dbt-platform/platform-ide/search-and-replace.png?v=2 "Search files for specific terms and quickly replace them.")](#)Search files for specific terms and quickly replace them. 1. Toggle between **file tree** and **search** navigation. 2. Search for words or phrases. Enhance the search to match case and/or whole words. You can also input replacement words or phrases. Click the icon next to the **Replace** field to replace all entries. 3. Navigate the search results. Click an entry to open the related file and highlight it on the screen. If you've entered replacement text, you'll see a preview of the new syntax. Click the symbol next to an entry to substitute the text with whatever is in the **Replace** field. ##### Command palette The command palette enhances navigation of your dbt project, enabling you to search files, content, and symbols, show and run IDE commands, view recent files, and more. Click the command palette to view the available options. Actions supporting keyboard shortcuts display to the right of the text. [![The command palette enables you to quickly navigate your project and run commands.](/img/docs/dbt-platform/platform-ide/command-palette.png?v=2 "The command palette enables you to quickly navigate your project and run commands.")](#)The command palette enables you to quickly navigate your project and run commands. * **Go to File:** Search for files in your current project and open them in a new tab. * **Show and Run Commands:** View and run commands related to IDE navigation and settings. Note: dbt commands (such as `run` and `build`) are available only in the [Command bar](#console-section) menu in the console; the command palette doesn't currently support them. * **Search for Text:** Search for text across your project and either open files from the results or send results to the [search and replace](#search-and-replace) section for bulk changes. * **Go to Symbol in Editor:** Quickly jump to symbols in the current file. * **More:** Display advanced features such as **Go to Line/Column**, **Go to Symbol in Workspace**, and search within currently open files only. [![Go to File.](/img/docs/dbt-platform/platform-ide/go-to-file.png?v=2 "Go to File.")](#)Go to File. [![Show and Run Commands.](/img/docs/dbt-platform/platform-ide/show-and-run-commands.png?v=2 "Show and Run Commands.")](#)Show and Run Commands. [![Search for text.](/img/docs/dbt-platform/platform-ide/search-for-text.png?v=2 "Search for text.")](#)Search for text. [![Go to Symbol in Editor.](/img/docs/dbt-platform/platform-ide/go-to-symbol.png?v=2 "Go to Symbol in Editor.")](#)Go to Symbol in Editor. [![More.](/img/docs/dbt-platform/platform-ide/more.png?v=2 "More.")](#)More. #### Editing features The Studio IDE features some delightful tools and layouts to make it easier for you to write dbt code and collaborate with teammates. [![Use the file editor, version control section, and save button during your development workflow](/img/docs/dbt-platform/platform-ide/ide-editing.png?v=2 "Use the file editor, version control section, and save button during your development workflow")](#)Use the file editor, version control section, and save button during your development workflow 1. **File editor —** The file editor is where you edit code. Tabs break out the region for each opened file, and unsaved files are marked with a blue dot icon in the tab view. You can edit, format, or lint files and execute dbt commands in your protected primary git branch. Since the Studio IDE prevents commits to the protected branch, it prompts you to commit those changes to a new branch. * Use intuitive [keyboard shortcuts](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) to make development easier for you and your team. 2. **Save button —** The editor has a **Save** button that saves editable files. Pressing the button or using the Command-S or Control-S shortcut saves the file contents. You don't need to save to preview code results in the Console section, but it's necessary before changes appear in a dbt invocation. The file editor tab shows a blue icon for unsaved changes. 3. **Version Control —** This menu contains all git-related elements, including the Git actions button. The button updates relevant actions based on your editor's state, such as prompting to pull remote changes, commit and sync when reverted commit changes are present, creating a merge/pull request when appropriate, or pruning branches deleted from the remote repository. * The dropdown menu on the Git actions button allows users to revert changes, refresh Git state, create merge/pull requests, prune branches, and change branches. * You can also [resolve merge conflicts](https://docs.getdbt.com/docs/platform/git/merge-conflicts.md) and for more info on git, refer to [Version control basics](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide). * **Version Control Options menu —** The **Changes** section, under the Git actions button, lists all file changes since the last commit. You can click on a change to open the Git Diff View to see the inline changes. You can also right-click any file and use the file-specific options in the Version Control Options menu. [![Right-click edited files to access Version Control Options menu](/img/docs/dbt-platform/platform-ide/version-control-options-menu.png?v=2 "Right-click edited files to access Version Control Options menu")](#)Right-click edited files to access Version Control Options menu * Use the **Prune branches** option to remove local branches that have already been deleted from the remote repository. Selecting this triggers a [pop-up modal](#prune-branches-modal), where you can confirm the deletion of the specific local branches, keeping your branch management tidy. Note that this won't delete the branch you're currently on. Pruning branches isn't available for [managed repositories](https://docs.getdbt.com/docs/platform/git/managed-repository.md) because they don't have a typical remote setup, which prevents remote branch deletion. #### Additional editing features * **Minimap —** A Minimap (code outline) gives you a high-level overview of your source code, which is useful for quick navigation and code understanding. A file's minimap is displayed on the upper-right side of the editor. To quickly jump to different sections of your file, click the shaded area. [![Use the Minimap for quick navigation and code understanding](/img/docs/dbt-platform/platform-ide/ide-minimap.png?v=2 "Use the Minimap for quick navigation and code understanding")](#)Use the Minimap for quick navigation and code understanding * **Git Diff View —** Clicking on a file in the **Changes** section of the **Version Control Menu** will open the changed file with Git Diff view. The editor will show the previous version on the left and the in-line changes made on the right. [![The Git Diff View displays the previous version on the left and the changes made on the right of the Editor](/img/docs/dbt-platform/platform-ide/ide-git-diff-view-with-save.png?v=2 "The Git Diff View displays the previous version on the left and the changes made on the right of the Editor")](#)The Git Diff View displays the previous version on the left and the changes made on the right of the Editor * **Markdown Preview console tab —** The Markdown Preview console tab shows a preview of your .md file's markdown code in your repository and updates it automatically as you edit your code. [![The Markdown Preview console tab renders markdown code below the Editor tab.](/img/docs/dbt-platform/platform-ide/ide-markdown-with-save.png?v=2 "The Markdown Preview console tab renders markdown code below the Editor tab.")](#)The Markdown Preview console tab renders markdown code below the Editor tab. * **CSV Preview console tab —** The CSV Preview console tab displays the data from your CSV file in a table, which updates automatically as you edit the file in your seed directory. [![View CSV code in the CSV Preview console tab below the Editor tab.](/img/docs/dbt-platform/platform-ide/ide-csv.png?v=2 "View CSV code in the CSV Preview console tab below the Editor tab.")](#)View CSV code in the CSV Preview console tab below the Editor tab. #### Console section The console section, located below the file editor, includes various console tabs and buttons to help you with tasks such as previewing, compiling, building, and viewing the DAG. Refer to the following sub-bullets for more details on the console tabs and buttons. [![The Console section is located below the file editor and has various tabs and buttons to help execute tasks](/img/docs/dbt-platform/platform-ide/ide-console-overview.png?v=2 "The Console section is located below the file editor and has various tabs and buttons to help execute tasks")](#)The Console section is located below the file editor and has various tabs and buttons to help execute tasks 1. **Preview button —** When you click on the **Preview** button, it runs the SQL in the active file editor regardless of whether you have saved it or not and sends the results to the **Results** console tab. You can preview a selected portion of saved or unsaved code by highlighting it and then clicking the **Preview** button. Details Row limits in IDE The Studio IDE returns default row limits, however, you can also specify the number of records returned. Refer to the following sub-bullets for more info: <br /> <br /> * **500-row limit:** To prevent the IDE from returning too much data and causing browser problems, dbt automatically sets a 500-row limit when using the **Preview Button**. You can modify this by adding `limit your_number` at the end of your SQL statement. For example, `SELECT * FROM` table `limit 100` will return up to 100 rows. Remember that you must write the `limit your_number` explicitly and cannot derive it from a macro. * **Change row limit default:** In dbt version 1.6 or higher, you can change the default limit of 500 rows shown in the **Results** tab when you run a query. To adjust the setting you can click on **Change row display** next to the displayed rows. Keep in mind that you can't set it higher than 10,000 rows. If you refresh the page or close your development session, the default limit will go back to 500 rows. * **Specify records returned:** The IDE also supports `SELECT TOP #`, which specifies the number of records to return. 2. **Compile button —** The **Compile** button compiles the saved or unsaved SQL code and displays it in the **Compiled code** tab. Starting from dbt v1.6 or higher, when you save changes to a model, you can compile its code with the model's specific context. This context is similar to what you'd have when building the model and involves useful context variables like `{{ this }} `or `{{ is_incremental() }}`. 3. **Build button —** The build button allows users to quickly access dbt commands related to the active model in the file editor. The available commands include dbt build, dbt test, and dbt run, with options to include only the current resource, the resource and its upstream dependencies, the resource, and its downstream dependencies, or the resource with all dependencies. This menu is available for all executable nodes. 4. **Lint button** — The **Lint** button runs the [linter](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) on the active file in the file editor. The linter checks for syntax errors and style issues in your code and displays the results in the **Code quality** tab. 5. **dbt Wizard** — [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) is the new and recommended governed agentic experience integrated into the Studio IDE that uses your project context to help you develop governed dbt changes faster. It can generate or refactor models, semantic models, tests, and documentation from natural language prompts. [dbt Copilot](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md#dbt-copilot-in-studio-ide) is separate from dbt Wizard and is dbt's inline AI assistance experience, providing single-click generation of SQL, documentation, tests, and semantic models in Studio IDE, Canvas, and Insights. 6. **Commands tab** — View the most recently run [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) from your current IDE session, their results, and relevant system logs. 7. **Problems tab** — You must be running dbt v2 to utilize the problems tab. Gain insights into problems with your dbt project that may prevent it from running properly in dbt v2 as you edit and before you execute runs. [![Preview results show up in the Results console tab](/img/docs/dbt-platform/platform-ide/ide-problems-tab.png?v=2 "Preview results show up in the Results console tab")](#)Preview results show up in the Results console tab 8. **Results tab** — The Results console tab displays the most recent Preview results in tabular format. [![Preview results show up in the Results console tab](/img/docs/dbt-platform/platform-ide/results-console-tab.png?v=2 "Preview results show up in the Results console tab")](#)Preview results show up in the Results console tab 9. **Code quality tab** — The Code quality tab displays the results of the linter on the active file in the File editor. It allows you to view code errors, provides code quality visibility and management, and displays the SQLFluff version used. 10. **Compiled code tab —** The Compile generates the compiled code when the Compile button is executed. The Compiled code tab displays the compiled SQL code for the active file in the file editor. [![Compile results show up in the Compiled Code tab](/img/docs/dbt-platform/platform-ide/compiled-code-console-tab.png?v=2 "Compile results show up in the Compiled Code tab")](#)Compile results show up in the Compiled Code tab 11. **Lineage tab —** The Lineage tab in the file editor displays the active model's lineage or DAG. By default, it shows two degrees of lineage in both directions (`2+model_name+2`), however, you can change it to +model+ (full DAG). To use the lineage: * Double-click a node in the DAG to open that file in a new tab * Expand or shrink the DAG using node selection syntax. * Note, the `--exclude` flag isn't supported. [![View resource lineage in the Lineage tab](/img/docs/dbt-platform/platform-ide/lineage-console-tab.png?v=2 "View resource lineage in the Lineage tab")](#)View resource lineage in the Lineage tab #### Invocation history The Invocation History Drawer stores information on dbt invocations in the IDE. When you invoke a command, like executing a dbt command such as `dbt run`, the associated logs are displayed in the Invocation History Drawer. You can open the drawer in multiple ways: * Clicking the `^` icon next to the Command bar on the lower left of the page * Typing a dbt command and pressing enter * Or pressing Control-backtick (or Ctrl + \`) [![The Invocation History Drawer returns a log and detail of all your dbt invocations.](/img/docs/dbt-platform/platform-ide/ide-inv-history-drawer.png?v=2 "The Invocation History Drawer returns a log and detail of all your dbt invocations.")](#)The Invocation History Drawer returns a log and detail of all your dbt invocations. 1. **Invocation History list —** The left-hand panel of the Invocation History Drawer displays a list of previous invocations in the Studio IDE, including the command, branch name, command status, and elapsed time. 2. **Invocation Summary —** The Invocation Summary, located above **System Logs**, displays information about a selected command from the Invocation History list, such as the command, its status (`Running` if it's still running), the git branch that was active during the command, and the time the command was invoked. 3. **System Logs toggle —** The System Logs toggle, located under the Invocation Summary, allows the user to see the full stdout and debug logs for the entirety of the invoked command. 4. **Command Control button —** Use the Command Control button, located on the right side, to control your invocation and cancel or rerun a selected run. [![The Invocation History list displays a list of previous invocations in the IDE](/img/docs/dbt-platform/platform-ide/ide-results.png?v=2 "The Invocation History list displays a list of previous invocations in the IDE")](#)The Invocation History list displays a list of previous invocations in the IDE 5. **Node Summary tab —** Clicking on the Results Status Tabs will filter the Node Status List based on their corresponding status. The available statuses are Pass (successful invocation of a node), Warn (test executed with a warning), Error (database error or test failure), Skip (nodes not run due to upstream error), and Queued (nodes that have not executed yet). 6. **Node result toggle —** After running a dbt command, information about each executed node can be found in a Node Result toggle, which includes a summary and debug logs. The Node Results List lists every node that was invoked during the command. 7. **Node result list —** The Node result list shows all the Node Results used in the dbt run, and you can filter it by clicking on a Result Status tab. #### Modals and Menus Use menus and modals to interact with Studio IDE and access useful options to help your development workflow. ###### Editor tab menu To interact with open editor tabs, right-click any tab to access the helpful options in the file tab menu. [![ Right-click a tab to view the Editor tab menu options](/img/docs/dbt-platform/platform-ide/editor-tab-menu-with-save.png?v=2 " Right-click a tab to view the Editor tab menu options")](#) Right-click a tab to view the Editor tab menu options ###### Global command shortcut The global command shortcut provides helpful shortcuts to interact with the Studio IDE, such as git actions, specialized dbt commands, and compile, and preview actions, among others. To open the menu, use Command-P or Control-P. [![The Command History returns a log and detail of all your dbt invocations.](/img/docs/dbt-platform/platform-ide/ide-global-command-palette-with-save.png?v=2 "The Command History returns a log and detail of all your dbt invocations.")](#)The Command History returns a log and detail of all your dbt invocations. ###### Studio IDE Status modal The Studio IDE Status modal shows the current error message and debug logs for the server. This also contains an option to restart the Studio IDE. Open this by clicking on the Studio IDE Status button. [![The Command History returns a log and detail of all your dbt invocations.](/img/docs/dbt-platform/platform-ide/ide-status-modal-with-save.png?v=2 "The Command History returns a log and detail of all your dbt invocations.")](#)The Command History returns a log and detail of all your dbt invocations. ###### Commit to a new branch Edit directly on your protected primary git branch and commit those changes to a new branch when ready. [![Commit changes to a new branch](/img/docs/dbt-platform/using-dbt-platform/create-new-branch.png?v=2 "Commit changes to a new branch")](#)Commit changes to a new branch ###### Commit Changes modal The Commit Changes modal is accessible via the Git Actions button to commit all changes or via the Version Control Options menu to commit individual changes. Once you enter a commit message, you can use the modal to commit and sync the selected changes. [![The Commit Changes modal is how users commit changes to their branch.](/img/docs/dbt-platform/platform-ide/commit-changes-modal.png?v=2 "The Commit Changes modal is how users commit changes to their branch.")](#)The Commit Changes modal is how users commit changes to their branch. ###### Change Branch modal The Change Branch modal allows users to switch git branches in the Studio IDE. It can be accessed through the **Change Branch** link or the **Git actions** button under the **Version control** menu. [![The Commit Changes modal is how users change their branch.](/img/docs/dbt-platform/platform-ide/change-branch-modal.png?v=2 "The Commit Changes modal is how users change their branch.")](#)The Commit Changes modal is how users change their branch. ###### Prune branches modal The Prune branches modal allows users to delete local branches that have been deleted from the remote repository, keeping your branch management tidy. This is accessible through the **Git actions** button under the [**Version control** menu](#editing-features). Note that this won't delete the branch you're currently on. Pruning branches isn't available for managed repositories because they don't have a typical remote setup, which prevents remote branch deletion. [![The Prune branches modal allows users to delete local branches that have already been deleted from the remote repository.](/img/docs/dbt-platform/platform-ide/prune-branch-modal.png?v=2 "The Prune branches modal allows users to delete local branches that have already been deleted from the remote repository.")](#)The Prune branches modal allows users to delete local branches that have already been deleted from the remote repository. ###### Revert Uncommitted Changes modal The Revert Uncommitted Changes modal is how users revert changes in the IDE. This is accessible via the `Revert File` option above the Version Control Options menu, or via the Git Actions button when there are saved, uncommitted changes in the IDE. [![The Commit Changes modal is how users change their branch.](/img/docs/dbt-platform/platform-ide/revert-uncommitted-changes-with-save.png?v=2 "The Commit Changes modal is how users change their branch.")](#)The Commit Changes modal is how users change their branch. --- ### Install the dbt platform CLI dbt platform The dbt platform natively supports developing using a command line interface (CLI), empowering team members to contribute with enhanced flexibility and collaboration. The dbt platform CLI allows you to run dbt commands against your dbt platform development environment from your local command line. CLI compatibility The dbt platform CLI is a dbt platform tool available to users on any [plan](https://www.getdbt.com/pricing). It is intended for use only with the dbt platform and may conflict with self-hosted installations of the dbt v1 or dbt v2 CLIs. Refer to the [FAQs](#faqs) for more information. dbt commands run against the platform's infrastructure and benefit from: * Secure credential storage in the dbt platform * [Automatic deferral](https://docs.getdbt.com/docs/platform/about-defer.md) of build artifacts to your project's production environment * Speedier, lower-cost builds * Support for dbt Mesh ([cross-project `ref`](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md)) [![Diagram of how the dbt platform CLI works with dbt's infrastructure to run dbt commands from your local command line.](/img/docs/dbt-platform/dbt-cli-overview.png?v=2 "Diagram of how the dbt platform CLI works with dbt's infrastructure to run dbt commands from your local command line.")](#)Diagram of how the dbt platform CLI works with dbt's infrastructure to run dbt commands from your local command line. #### Prerequisites The dbt platform CLI is available in all [deployment regions](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) and for both multi-tenant and single-tenant accounts. * If you installed the dbt v1 CLI in a virtual environment, deactivate that environment or create an alias for the platform CLI before you run platform CLI commands. * If you installed dbt locally, create an alias for the platform CLI before you run platform CLI commands. Refer to the [FAQs](#faqs) for more information about managing multiple dbt platform CLI tools and creating an alias. Using the dbt platform CLI for hybrid development with dbt v2? See the [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) guide to keep credentials, environment variables, and dbt v2 versions in sync across your local CLI and the dbt platform. #### Install dbt platform CLI You can install the dbt platform CLI via the command line by using one of the following methods: ##### macOS (brew) Before you begin, make sure you have [Homebrew installed](http://brew.sh/) in your code editor or command line terminal. Refer to the [FAQs](#faqs) if your operating system runs into path conflicts. 1. Verify that you don't already have dbt installed by running the following command: ```bash which dbt ``` If the output is `dbt not found`, then that confirms you don't have it installed. Run `pip uninstall dbt` to uninstall dbt v1 If you've installed dbt v1 globally in some other way, uninstall it first before proceeding: ```bash pip uninstall dbt ``` 2. Install the dbt platform CLI with Homebrew: * First, remove the `dbt-labs` tap, the separate repository for packages, from Homebrew. This prevents Homebrew from installing packages from that repository: ```bash brew untap dbt-labs/dbt ``` * Then, add and install the dbt platform CLI as a package: ```bash brew tap dbt-labs/dbt-cli brew install dbt ``` If you have multiple taps, use `brew install dbt-labs/dbt-cli/dbt`. 3. Verify your installation by running `dbt --help` in the command line. If you see the following output, you installed it correctly: ```bash The dbt CLI - an ELT tool for running SQL transformations and data models in dbt... ``` If you don't see this output, check that you've deactivated pyenv or venv and don't have a global dbt version installed. * Note that you no longer need to run the `dbt deps` command when your environment starts. Previously, initialization required this step. However, you should still run `dbt deps` if you make any changes to your `packages.yml` file. 4. Clone your repository to your local computer using `git clone`. For example, to clone a GitHub repo using HTTPS format, run `git clone https://github.com/YOUR-USERNAME/YOUR-REPOSITORY`. 5. After cloning your repo, [configure](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md) the dbt platform CLI for your dbt project. This lets you run dbt commands like [`dbt environment show`](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) to view your dbt configuration or `dbt compile` to compile your project and validate models and tests. You can also add, edit, and synchronize files with your repo. ##### Windows (native executable) Refer to the [FAQs](#faqs) if your operating system runs into path conflicts. 1. Download the latest Windows release for your platform from [GitHub](https://github.com/dbt-labs/dbt-cli/releases). 2. Extract the `dbt.exe` executable into the same folder as your dbt project. info Advanced users can configure multiple projects to use the same dbt platform CLI by: 1. Placing the executable file (`.exe`) in the "Program Files" folder 2. [Adding it to their Windows PATH environment variable](https://medium.com/@kevinmarkvi/how-to-add-executables-to-your-path-in-windows-5ffa4ce61a53) 3. Saving it where needed Note that if you're using VS Code, you must restart it to pick up modified environment variables. 4. Verify your installation by running `./dbt --help` in the command line. If you see the following output, you installed it correctly: ```bash The dbt CLI - an ELT tool for running SQL transformations and data models in dbt... ``` If you don't see this output, check that you've deactivated pyenv or venv and don't have a global dbt version installed. * Note that you no longer need to run the `dbt deps` command when your environment starts. Previously, initialization required this step. However, you should still run `dbt deps` if you make any changes to your `packages.yml` file. 5. Clone your repository to your local computer using `git clone`. For example, to clone a GitHub repo using HTTPS format, run `git clone https://github.com/YOUR-USERNAME/YOUR-REPOSITORY`. 6. After cloning your repo, [configure](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md) the dbt platform CLI for your dbt project. This lets you run dbt commands like [`dbt environment show`](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) to view your dbt configuration or `dbt compile` to compile your project and validate models and tests. You can also add, edit, and synchronize files with your repo. ##### Linux (native executable) Refer to the [FAQs](#faqs) if your operating system runs into path conflicts. 1. Download the latest Linux release for your platform from [GitHub](https://github.com/dbt-labs/dbt-cli/releases). (Pick the file based on your CPU architecture) 2. Extract the `dbt-cli` binary to the same folder as your dbt project. ```bash tar -xf dbt_0.29.9_linux_amd64.tar.gz ./dbt --version ``` info Advanced users can configure multiple projects to use the same dbt platform CLI executable by adding it to their PATH environment variable in their shell profile. 3. Verify your installation by running `./dbt --help` in the command line. If you see the following output, you installed it correctly: ```bash The dbt CLI - an ELT tool for running SQL transformations and data models in dbt... ``` If you don't see this output, check that you've deactivated pyenv or venv and don't have a global dbt version installed. * Note that you no longer need to run the `dbt deps` command when your environment starts. Previously, initialization required this step. However, you should still run `dbt deps` if you make any changes to your `packages.yml` file. 4. Clone your repository to your local computer using `git clone`. For example, to clone a GitHub repo using HTTPS format, run `git clone https://github.com/YOUR-USERNAME/YOUR-REPOSITORY`. 5. After cloning your repo, [configure](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md) the dbt platform CLI for your dbt project. This lets you run dbt commands like [`dbt environment show`](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) to view your dbt configuration or `dbt compile` to compile your project and validate models and tests. You can also add, edit, and synchronize files with your repo. #### Update dbt platform CLI The following instructions explain how to update the dbt platform CLI to the latest version depending on your operating system. ##### macOS (brew) To update the dbt platform CLI, run `brew update` and then `brew upgrade dbt`. ##### Windows (executable) To update, follow the [Windows installation instructions](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md?install=windows#install-dbt-cloud-cli) and replace the existing `dbt.exe` executable with the new one. ##### Linux (executable) To update, follow the [Linux installation instructions](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md?install=linux#install-dbt-cloud-cli) and replace the existing `dbt` executable with the new one. #### Considerations The dbt platform CLI doesn't currently support relative paths in the [`packages.yml` file](https://docs.getdbt.com/docs/build/packages.md). Instead, use the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), which supports relative paths in this scenario. Here's an example of a [local package](https://docs.getdbt.com/docs/build/packages.md#local-packages) configuration in the `packages.yml` that won't work with the dbt platform CLI: ```yaml # repository_root/my_dbt_project_in_a_subdirectory/packages.yml packages: - local: ../shared_macros ``` In this example, `../shared_macros` is a relative path that tells dbt to look for: * `..` — Go one directory up (to `repository_root`). * `/shared_macros` — Find the `shared_macros` folder in the root directory. To work around this limitation, use the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), which fully supports relative paths in `packages.yml`. #### FAQs  What's the difference between the dbt platform CLI and dbt? The dbt platform CLI and [dbt](https://github.com/dbt-labs/dbt), an open-source project, are both command line tools that enable you to run dbt commands. The key distinction is that the dbt platform CLI is tailored for the dbt platform's infrastructure and integrates with all its [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features).  How do I run both the dbt platform CLI and dbt? For compatibility, both the dbt platform CLI and dbt are invoked by running `dbt`. This can create path conflicts if your operating system selects one over the other based on your $PATH environment variable (settings). If you have dbt installed locally, either: 1. Install natively, ensuring you either deactivate the virtual environment containing dbt v1 or create an alias for the dbt platform CLI. 2. (Advanced users) Install natively, but modify the $PATH environment variable to correctly point to the dbt platform CLI binary to use both dbt platform CLI and dbt together. You can always uninstall the dbt platform CLI to return to using dbt.  How to create an alias? To create an alias for the dbt platform CLI:<br /> 1. Open your shell's profile configuration file. Depending on your shell and system, this could be `~/.bashrc`, `~/.bash_profile`, `~/.zshrc`, or another file.<br /> 2. Add an alias that points to the dbt platform CLI binary. For example: `alias dbt-cli="path_to_dbt_cli_binary"` Replace `path_to_dbt_cli_binary` with the actual path to the dbt platform CLI binary, which is `/opt/homebrew/bin/dbt`. With this alias, you can use the command `dbt-cli` to invoke the dbt platform CLI.<br /> 3. Save the file and then either restart your shell or run `source` on the profile file to apply the changes. For example, in bash you would run: `source ~/.bashrc`<br /> 4. Test and use the alias to run commands:<br /> * To run the dbt platform CLI, use the `dbt-cli` command: `dbt-cli command_name`. Replace 'command\_name' with the specific dbt command you want to execute. <br /> * To run dbt, use the `dbt` command: `dbt command_name`. Replace 'command\_name' with the specific dbt command you want to execute. <br /> You can then use the `dbt-cli` command to invoke the dbt platform CLI while you keep dbt installed natively.  Why am I receiving a \`Stuck session\` error when trying to run a new command? The dbt platform CLI allows only one command that writes to the data warehouse at a time. If you attempt to run multiple write commands simultaneously (for example, `dbt run` and `dbt build`), you will encounter a `stuck session` error. To resolve this, cancel the specific invocation by passing its ID to the cancel command. For more information, refer to [parallel execution](https://docs.getdbt.com/reference/dbt-commands.md#parallel-execution).  I'm getting a \`Session occupied\` error in dbt platform CLI? If you're receiving a `Session occupied` error in the dbt platform CLI or if you're experiencing a long-running session, you can use the `dbt invocation list` command in a separate terminal window to view the status of your active session. This helps debug the issue and identify the arguments that are causing the long-running session. To cancel an active session, use the `Ctrl + Z` shortcut. To learn more about the `dbt invocation` command, see the [dbt invocation command reference](https://docs.getdbt.com/reference/commands/invocation.md?version=2.0). Alternatively, you can reattach to your existing session with `dbt reattach` and then press `Control-C` and choose to cancel the invocation. --- ### Invite read-only users to dbt Wizard Preview ### Invite read-only users to dbt Wizard [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based Give the people who need answers a way to get them. With a few setup steps, read-only users can ask questions of governed production data in plain language — no developer license required. [Read-only users](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) work in the dbt Wizard home tab in [Explore mode](https://docs.getdbt.com/docs/platform/wizard-home.md#ask-questions-in-explore-mode): the agent queries and explains your data but can't edit files, run builds, or change your project. This page is for admins. It covers which license read-only users need, how to invite them, and how to set your project up for good answers. Users with development access don't need a read-only license — they get [Explore mode](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) alongside the authoring modes automatically. info Read-only users see dbt Wizard in the home tab as soon as they have access, but they can't query anything until two things are true: credentials exist (either their own or the project's [analytics credential](#set-up-analytics-credentials)), and the project has metadata or Semantic Layer definitions to ask questions against. #### What read-only users can do A read-only user in dbt Wizard can: * Ask questions of production data in plain language in the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), and ask follow-ups in the same conversation. * See the Semantic Layer metric definition or SQL behind every answer. * See which environment an answer was queried against. A read-only user can't: * Use dbt Wizard in the [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). Read-only users work in the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md) only. * Edit files, commit, or open pull requests. * Run dbt commands or trigger builds. * Switch to **Ask for approval** or **Edit files automatically** mode. Those options are disabled. * Access data their [analytics credentials permissions](#set-up-analytics-credentials) don't already allow. Read-only users only have access to Explore mode and see a simpler UI. The only tools available are analytics tools: project metadata and Semantic Layer definitions. Tools that need filesystem or development access, like the context panel and the branch picker, don't appear. Users with development access still see all three [agent modes](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) and choose between them. #### FAQs  Which license does a read-only user need? You can use a Read-only license. For current license types and how they're assigned, refer to [User licenses](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md).  How many read-only users can we invite? You can invite any user that has a read-only license! Check out [Invite read-only users](#invite-read-only-users) for instructions on how to invite users. For license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing).  Does read-only usage count against our AI usage credits? Yes. Questions read-only users ask in Explore mode use dbt-managed inference and draw from your account's dbt Wizard consumption pool, the same as any other dbt Wizard usage. Refer to [dbt Wizard billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) for more information.  Do read-only users need the Analyst read permission set? Not to use Explore mode. Anyone without development access who can reach dbt Wizard sees Explore mode, whichever read permission set they have. They do need [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) to set up their own warehouse credentials. Without it, they query through the project's [analytics credential](#set-up-analytics-credentials).  Can read-only users use their own warehouse credentials? Yes, if they have the [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set and dbt Wizard uses those instead of the analytics credential.  Which credentials does Explore mode query with? The user's personal credentials first, if they have any. The project's [analytics credential](#set-up-analytics-credentials) is the fallback. If neither exists, dbt Wizard asks the user to set up credentials or contact an admin.  A read-only user can see dbt Wizard but can't get answers. Why? Seeing dbt Wizard in the Home tab doesn't mean the account is ready to answer questions. Check that: * An admin has [set up an analytics credential](#set-up-analytics-credentials) on the connection, and assigned that connection to the project. * The project has metadata or Semantic Layer definitions to ask questions against. An empty project has nothing to answer from. * The credential can reach the data the user is asking about. #### Invite read-only users This section explains how to invite read-only users to dbt platform so they can use the Wizard home tab. 1. In dbt platform, go to **Account settings** --> **Users**. 2. Click **Invite users**. 3. Enter the email addresses of the people you want to invite. 4. Assign the [read-only license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses) and a group whose [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) matches the data they should see. Explore mode isn't limited to one permission set — anyone without development access who can reach dbt Wizard gets it. * Use the [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set if you want these users to configure their own warehouse credentials rather than querying through the analytics credential. 5. Click **Send invitations**. Invited users log in and should land on the account home, then open dbt Wizard home tab from the left sidebar to start asking questions in Explore mode. These are the read-only specifics. For the full invite flow, seat management, and resending invites, refer to [Invite users](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md). #### Set up analytics credentials To run a query, Explore mode needs warehouse credentials. Most read-only users don't have their own, so an admin sets up an **analytics credential**: a shared, project-level warehouse credential that read-only users fall back to when they haven't configured personal credentials. Read-only users that have their own credentials can use the [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set. At query time, dbt Wizard resolves credentials in this order: 1. The user's personal warehouse credentials if they have them. 2. The project's analytics credential, if an admin has [set one up](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md#add-an-analytics-credential-to-a-connection). 3. If neither exists, dbt Wizard returns an error asking the user to set up personal credentials or contact an admin. ##### Prerequisites * An [account admin](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) role. * A [connection](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) to one of the supported warehouses: Snowflake, BigQuery, Redshift, or Databricks. ##### Add an analytics credential to a connection 1. In dbt platform, go to **Account settings** --> **Connections** and select the connection. 2. Scroll to the **Analytics credential** card and click **Edit**. A connection can have one analytics credential. 3. Pick an **Auth method** and enter the warehouse credentials dbt Wizard should query with, then click **Save**. The fields you see depend on the connection's warehouse and the auth method you pick. The following example uses Snowflake with key pair authentication: 1. Enter the **Username** and **Private key** for the warehouse user dbt Wizard queries as. 2. Add the **Private key passphrase** if your key is encrypted. 3. Optionally add a **Role** and **Warehouse** to pin which ones queries run with. [![Analytics credentials with Key pair selected as the auth method](/img/docs/dbt-platform/analytics-credential-keypair.png?v=2 "Analytics credentials with Key pair selected as the auth method")](#)Analytics credentials with Key pair selected as the auth method Give the credential the *least privilege* it needs to answer questions — read-only access to the data read-only users should see, and nothing more. It's a shared credential, so anyone querying through it sees whatever it can see, without a per-user trail. If you need per-user access control or auditing at the warehouse, have users configure personal credentials instead — that requires the [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set. ##### Assign the connection to a project 1. Go to **Account settings** --> **Projects** and select the project. 2. In the **Analytics configuration** section, select a connection that has an analytics credential. Only connections with one appear in the picker. 3. Click **Save**. Read-only users on that project now query through this credential in Explore mode. You can reuse the same connection across projects, but each project uses one analytics connection at a time. #### Control what read-only users can see Explore mode respects your existing access controls. It doesn't create a new permission surface. * Scope the [analytics credential](#set-up-analytics-credentials) to only the data read-only users should reach, since they all query through it. * Use [permission sets and groups](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to scope which projects a read-only user can reach. * Use [environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md) to control which environments they can query. #### Set your team up for good answers Explore mode answers are grounded in what your project defines. Here are some things to make the experience easier for users: * Define the metrics people ask about in the [Semantic Layer](https://docs.getdbt.com/docs/build/about-metricflow.md). Questions that map to a governed metric return your approved numbers instead of ad-hoc SQL. * Document your models and columns by writing up descriptions, which help dbt Wizard pick the right table and explain the answer in business terms. * Tell users which environment to trust by pointing them at production and explain what development data is, so a work-in-progress number never lands in a business review. #### Related docs * [Agent modes in dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) * [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md) * [User licenses](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) * [Invite users](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) * [Enable AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) --- ### Lint and format your code dbt platform Enhance your development workflow by integrating with popular linters and formatters like [SQLFluff](https://sqlfluff.com/), [sqlfmt](http://sqlfmt.com/), [Black](https://black.readthedocs.io/en/latest/), and [Prettier](https://prettier.io/). Leverage these powerful tools directly in the Studio IDE without interrupting your development flow. Details What are linters and formatters? Linters analyze code for errors, bugs, and style issues, while formatters fix style and formatting rules. Read more about when to use linters or formatters in the [FAQs](#faqs) In the Studio IDE, you can perform linting, auto-fix, and formatting on five different file types: * SQL: [Lint](#lint) and fix with SQLFluff, and [format](#format) with sqlfmt * YAML, Markdown, and JSON: Format with Prettier * Python: Format with Black Each file type has its own unique linting and formatting rules. You can [customize](#customize-linting) the linting process to add more flexibility and enhance problem and style detection. By default, the IDE uses sqlfmt rules to format your code, making it convenient to use right away. However, if you have a file named `.sqlfluff` in the root directory of your dbt project, the IDE will default to SQLFluff rules instead. [![Use SQLFluff to lint/format your SQL code, and view code errors in the Code quality tab.](/img/docs/dbt-platform/platform-ide/sqlfluff.gif?v=2 "Use SQLFluff to lint/format your SQL code, and view code errors in the Code quality tab.")](#)Use SQLFluff to lint/format your SQL code, and view code errors in the Code quality tab. [![Use sqlfmt to format your SQL code.](/img/docs/dbt-platform/platform-ide/sqlfmt.gif?v=2 "Use sqlfmt to format your SQL code.")](#)Use sqlfmt to format your SQL code. [![Format YAML, Markdown, and JSON files using Prettier.](/img/docs/dbt-platform/platform-ide/prettier.gif?v=2 "Format YAML, Markdown, and JSON files using Prettier.")](#)Format YAML, Markdown, and JSON files using Prettier. [![Use the Config button to select your tool.](/img/docs/dbt-platform/platform-ide/ide-sql-popup.png?v=2 "Use the Config button to select your tool.")](#)Use the Config button to select your tool. [![Customize linting by configuring your own linting code rules, including dbtonic linting/styling.](/img/docs/dbt-platform/platform-ide/ide-sqlfluff-config.png?v=2 "Customize linting by configuring your own linting code rules, including dbtonic linting/styling.")](#)Customize linting by configuring your own linting code rules, including dbtonic linting/styling. #### Lint With the Studio IDE, you can seamlessly use [SQLFluff](https://sqlfluff.com/), a configurable SQL linter, to warn you of complex functions, syntax, formatting, and compilation errors. This integration allows you to run checks, fix, and display any code errors directly within the Studio IDE: * Works with Jinja and SQL. * Comes with built-in [linting rules](https://docs.sqlfluff.com/en/stable/rules.html). You can also [customize](#customize-linting) your own linting rules. * Empowers you to [enable linting](#enable-linting) with options like **Lint** (displays linting errors and recommends actions) or **Fix** (auto-fixes errors in the Studio IDE). * Displays a **Code quality** tab to view lint and format results. Some editor diagnostics also appear in the **Problems** tab. Linting considerations * The Studio IDE runs linting using the dbt v1 engine, even when your development environment uses the **v2 Stable** release track. For more information, refer to [dbt v2 limitations](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations). * Linting doesn't support ephemeral models in dbt v1.5 and lower. Refer to the [FAQs](#faqs) for more info. ##### Enable linting Linting is available on all branches, including your protected primary git branch. Since the Studio IDE prevents commits to the protected branch, it prompts you to commit those changes to a new branch. 1. To enable linting, open a `.sql` file and click the **Code quality** tab. 2. Click the **`</> Config`** button on the right side of the [console section](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#console-section), below the **File editor**. 3. In the **Code quality tool configuration** dialog, select **SQLFluff (linting & formatting)**. 4. Click **Save**. 5. After saving, use the toolbar above the console (below the **File editor**) to select **Lint file** or **Lint and Fix file**: * **Lint**: Displays linting issues in the Studio IDE as wavy underlines in the **File editor**. You can hover over an underlined issue to display the details and actions, including a **Quick Fix** option to fix all or specific issues. After linting, you'll see a message confirming the outcome. Linting doesn't rerun after saving. Click **Lint** again to rerun linting. * **Fix**: Automatically fixes linting errors in the **File editor**. When fixing is complete, you'll see a message confirming the outcome. * Use the **Code quality** tab to view lint results and logs. Check the **Problems** tab for related editor diagnostics. [![Use the Lint or Fix button in the console section to lint or auto-fix your code.](/img/docs/dbt-platform/platform-ide/ide-lint-format-console.gif?v=2 "Use the Lint or Fix button in the console section to lint or auto-fix your code.")](#)Use the Lint or Fix button in the console section to lint or auto-fix your code. ##### Lint multiple files You can lint multiple SQL files at once, depending on how you are working with dbt. The behavior differs between the Studio IDE, dbt v1, and the dbt platform CLI. * **Studio IDE:** By default, linting runs against all modified `.sql` files in your project on your current branch. Refer to [Snapshot linting](#snapshot-linting) for more information. * **dbt v1:** Does not include a built-in linter. To lint SQL files in your project, use a third-party linter such as SQLFluff configured to use the [dbt templater](https://docs.sqlfluff.com/en/stable/configuration/templating/dbt.html). You can lint multiple files by specifying one or more file or directory paths as arguments to the command. * **dbt platform CLI:** Supports the same linting [commands](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md#lint-sql-files) as dbt v1: ```text dbt sqlfluff lint [PATHS]... [flags] ``` If no path is specified (for example, `dbt sqlfluff lint`), all SQL files in the project are linted. ##### Customize linting SQLFluff is a configurable SQL linter, which means you can configure your own linting rules instead of using the default linting settings in the IDE. You can exclude files and directories by using a standard `.sqlfluffignore` file. Learn more about the syntax in the [.sqlfluffignore syntax docs](https://docs.sqlfluff.com/en/stable/configuration.html#id2). To configure your own linting rules: 1. Create a new file in the root project directory (the parent or top-level directory for your files). Note: The root project directory is the directory where your `dbt_project.yml` file resides. 2. Name the file `.sqlfluff` (make sure you add the `.` before `sqlfluff`). 3. [Create](https://docs.sqlfluff.com/en/stable/configuration/setting_configuration.html#new-project-configuration) and add your custom config code. 4. Save and commit your changes. 5. Restart the Studio IDE. 6. Test it out and happy linting! ###### Snapshot linting By default, dbt lints all modified `.sql` files in your project, including snapshots. [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) can be defined in YAML *and* `.sql` files, but their SQL isn't lintable and can cause errors during linting. To prevent SQLFluff from linting snapshot files, add the snapshots directory to your `.sqlfluffignore` file (for example `snapshots/`). Note that you should explicitly exclude snapshots in your `.sqlfluffignore` file since dbt doesn't automatically ignore snapshots on the backend. ##### Configure dbtonic linting rules Refer to the [Jaffle shop SQLFluff config file](https://github.com/dbt-labs/jaffle-shop-template/blob/main/.sqlfluff) for dbt-specific (or dbtonic) linting rules we use for our own projects: dbtonic config code example provided by dbt Labs ```text [sqlfluff] templater = dbt # This change (from Jinja to dbt templater) will make linting slower # because linting will first compile dbt code into data warehouse code. runaway_limit = 10 max_line_length = 80 indent_unit = space [sqlfluff:indentation] tab_space_size = 4 [sqlfluff:layout:type:comma] spacing_before = touch line_position = trailing [sqlfluff:rules:capitalisation.keywords] capitalisation_policy = lower [sqlfluff:rules:aliasing.table] aliasing = explicit [sqlfluff:rules:aliasing.column] aliasing = explicit [sqlfluff:rules:aliasing.expression] allow_scalar = False [sqlfluff:rules:capitalisation.identifiers] extended_capitalisation_policy = lower [sqlfluff:rules:capitalisation.functions] capitalisation_policy = lower [sqlfluff:rules:capitalisation.literals] capitalisation_policy = lower [sqlfluff:rules:ambiguous.column_references] # Number in group by group_by_and_order_by_style = implicit ``` For more information on styling best practices, refer to [How we style our SQL](https://docs.getdbt.com/best-practices/how-we-style/2-how-we-style-our-sql.md). [![Customize linting by configuring your own linting code rules, including dbtonic linting/styling.](/img/docs/dbt-platform/platform-ide/ide-sqlfluff-config.png?v=2 "Customize linting by configuring your own linting code rules, including dbtonic linting/styling.")](#)Customize linting by configuring your own linting code rules, including dbtonic linting/styling. #### Format In the Studio IDE, you can format your code to match style guides with a click of a button. The Studio IDE integrates with formatters like sqlfmt, Prettier, and Black to automatically format code on five different file types: SQL, YAML, Markdown, Python, and JSON. * SQL: Format with [sqlfmt](http://sqlfmt.com/), which provides one way to format your dbt SQL and Jinja. * **Note**: Custom sqlfmt configuration in the Studio IDE is not supported. * YAML, Markdown, and JSON: Format with [Prettier](https://prettier.io/). * Python: Format with [Black](https://black.readthedocs.io/en/latest/). The Studio IDE formatting integrations take care of manual tasks like code formatting, enabling you to focus on creating quality data models, collaborating, and driving impactful results. ##### Format SQL To format your SQL code, dbt integrates with [sqlfmt](http://sqlfmt.com/), which is an uncompromising SQL query formatter that provides one way to format the SQL query and Jinja. By default, the Studio IDE uses sqlfmt rules to format your code, making the **Format** button available and convenient to use immediately. However, if you have a file named `.sqlfluff` in the root directory of your dbt project, the Studio IDE defaults to SQLFluff rules instead. Formatting is available on all branches, including your protected primary git branch. Since the Studio IDE prevents commits to the protected branch, it prompts you to commit those changes to a new branch. 1. Open a `.sql` file and click the **Code quality** tab. 2. Click the **`</> Config`** button on the right side of the console. 3. In the **Code quality tool configuration** dialog, select **sqlfmt (formatting only)**. 4. Click **Save**. 5. After saving, use the toolbar above the console (below the **File editor**) to select the **Format** button. 6. The **Format** button auto-formats your code in the **File editor**. Once you've auto-formatted, you'll see a message confirming the outcome. [![Use sqlfmt to format your SQL code.](/img/docs/dbt-platform/platform-ide/sqlfmt.gif?v=2 "Use sqlfmt to format your SQL code.")](#)Use sqlfmt to format your SQL code. ##### Format YAML, Markdown, JSON To format your YAML, Markdown, or JSON code, dbt integrates with [Prettier](https://prettier.io/), which is an opinionated code formatter. Formatting is available on all branches, including your protected primary git branch. Since the Studio IDE prevents commits to the protected branch, it prompts you to commit those changes to a new branch. 1. Open a `.yml`, `.md`, or `.json` file. 2. In the toolbar above the console (below the **File editor**), select the **Format** button to auto-format your code in the **File editor**. Use the **Code quality** tab to view format results. 3. Once you've auto-formatted, you'll see a message confirming the outcome. [![Format YAML, Markdown, and JSON files using Prettier.](/img/docs/dbt-platform/platform-ide/prettier.gif?v=2 "Format YAML, Markdown, and JSON files using Prettier.")](#)Format YAML, Markdown, and JSON files using Prettier. You can add a configuration file to customize formatting rules for YAML, Markdown, or JSON files using Prettier. The IDE looks for the configuration file based on an order of precedence. For example, it first checks for a "prettier" key in your `package.json` file. For more information on the order of precedence and how to configure files, refer to [Prettier's documentation](https://prettier.io/docs/en/configuration.html). Please note, `.prettierrc.json5`, `.prettierrc.js`, and `.prettierrc.toml` files aren't currently supported. ##### Format Python To format your Python code, dbt integrates with [Black](https://black.readthedocs.io/en/latest/), which is an uncompromising Python code formatter. Formatting is available on all branches, including your protected primary git branch. Since the Studio IDE prevents commits to the protected branch, it prompts you to commit those changes to a new branch. 1. Open a `.py` file. 2. In the toolbar above the console (below the **File editor**), select the **Format** button to auto-format your code in the **File editor**. 3. Once you've auto-formatted, you'll see a message confirming the outcome. [![Format Python files using Black.](/img/docs/dbt-platform/platform-ide/python-black.gif?v=2 "Format Python files using Black.")](#)Format Python files using Black. #### FAQs  When should I use SQLFluff and when should I use sqlfmt? SQLFluff and sqlfmt are both tools used for formatting SQL code, but some differences may make one preferable to the other depending on your use case.<br /> SQLFluff is a SQL code linter and formatter. This means that it analyzes your code to identify potential issues and bugs, and follows coding standards. It also formats your code according to a set of rules, which are [customizable](#customize-linting), to ensure consistent coding practices. You can also use SQLFluff to keep your SQL code well-formatted and follow styling best practices.<br /> sqlfmt is a SQL code formatter. This means it automatically formats your SQL code according to a set of formatting rules that aren't customizable. It focuses solely on the appearance and layout of the code, which helps ensure consistent indentation, line breaks, and spacing. sqlfmt doesn't analyze your code for errors or bugs and doesn't look at coding issues beyond code formatting.<br /> You can use either SQLFluff or sqlfmt depending on your preference and what works best for you: * Use SQLFluff to have your code linted and formatted (meaning analyze fix your code for errors/bugs, and format your styling). It allows you the flexibility to customize your own rules. * Use sqlfmt to only have your code well-formatted without analyzing it for errors and bugs. You can use sqlfmt out of the box, making it convenient to use right away without having to configure it.  Can I nest \`.sqlfluff\` files? To ensure optimal code quality, consistent code, and styles, it's highly recommended you have one main `.sqlfluff` configuration file in the root folder of your project. Having multiple files can result in various different SQL styles in your project.<br /><br /> However, you can customize and include an additional child `.sqlfluff` configuration file within specific subfolders of your dbt project.<br /><br />By nesting a `.sqlfluff` file in a subfolder, SQLFluff will apply the rules defined in that subfolder's configuration file to any files located within it. The rules specified in the parent `.sqlfluff` file will be used for all other files and folders outside of the subfolder. This hierarchical approach allows for tailored linting rules while maintaining consistency throughout your project. Refer to [SQLFluff documentation](https://docs.sqlfluff.com/en/stable/configuration.html#configuration-files) for more info.  Can I run SQLFluff commands from the terminal? Currently, running SQLFluff commands from the terminal isn't supported.  What are some considerations when using dbt linting? Currently, the Studio IDE can lint or fix files up to a certain size and complexity. If you attempt to lint or fix files that are too large, taking more than 60 seconds for the dbt backend to process, you will see an 'Unable to complete linting this file' error. To avoid this, break up your model into smaller models (files) so that they are less complex to lint or fix. Note that linting is simpler than fixing so there may be cases where a file can be linted but not fixed. #### Related docs * [User interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) * [Keyboard shortcuts](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) * [SQL linting in CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) --- ### Log in to dbt platform dbt platform <https://login.dbt.com> is the universal entry point for dbt platform accounts with a static access URL (for example, `abc123.us1.dbt.com`). It sits outside any specific account and maps your email to the accounts you can access across instances and regions. You can also sign in directly with your account **Access URL** if you already know it, for example, `abc123.us1.dbt.com`. Refer to [Access, regions, and IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for the full list of access URLs by region. info `login.dbt.com` is currently available for multi-tenant accounts with an account-specific domain (for example, `abc123.us1.dbt.com`). Support for single-tenant accounts is coming soon. In the meantime, single-tenant users can sign in directly using their account **Access URL** (like `MY_COMPANY.us1.dbt.com`). OAuth clients such as [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md?version=2.0), and [dbt MCP](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) have not yet been updated to use `login.dbt.com` and continue to authenticate through their account [**Access URL**](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). #### How login works 1. Go to <https://login.dbt.com>. 2. Enter your email and you will receive a verification code to log in if you have access to a dbt account. If you don't have a dbt account, you can create one on the same login page. 3. Check your email for the verification code, then enter it on `login.dbt.com` to confirm your identity. The code expires after 5 minutes or 3 attempts. * If you didn't receive a code, check your spam folder or request a new one. You may not receive a verification code if your email is not associated with a dbt account. 4. Once you verify, dbt platform looks up which accounts are associated with your email address: * If you have access to multiple accounts, you see a list of accounts including their region and tenancy. Select the one you want to open. * If you have access to only one account, you're redirected to that account directly. 5. Authenticate using the method configured for that account (password, SSO, or MFA). note If no accounts are found for your email, contact [dbt Support](mailto:support@getdbt.com) or your account admin. You can also sign in directly at your account [**Access URL**](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). [![Login to dbt platform using the login.dbt.com page.](/img/docs/dbt-platform/login-email-page.png?v=2 "Login to dbt platform using the login.dbt.com page.")](#)Login to dbt platform using the login.dbt.com page. [![Verify code to login to dbt platform](/img/docs/dbt-platform/login-email-verify-code.png?v=2 "Verify code to login to dbt platform")](#)Verify code to login to dbt platform [![Select account to login to dbt platform](/img/docs/dbt-platform/login-account-list.png?v=2 "Select account to login to dbt platform")](#)Select account to login to dbt platform #### SSO behavior After you select an account, dbt platform applies that account's authentication requirements. If SSO is enforced for the account, you'll be redirected to your identity provider (IdP) to complete authentication. Refer to [SSO overview](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for more information about SSO configuration. #### Admin settings Account administrators can control whether users can discover an account through `login.dbt.com`. 1. In dbt platform, account admins can go to [**Account settings**](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery) and scroll to **Settings**. 2. Navigate to the bottom and check the **Enable global account discovery** option to allow users will be able to find the account by entering their email at `login.dbt.com`. * When an admin turns off the option, users must know their account **Access URL** to sign in and won't see the account in the discovery list. To change this setting, go to **Account settings** and select or clear **Enable global account discovery**. For details, refer to [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). --- ### Manage AI features in dbt platform dbt platform Manage AI features in dbt platform by turning AI features on or off, choosing which AI provider to use, and controlling spend. The details on this page are generally for admin tasks. If you're looking for what dbt Wizard does and where you can use it, start with [About dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md). Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. #### Prerequisites * A [dbt platform account](https://www.getdbt.com/pricing) on Developer, Starter, Enterprise, or Enterprise+ plans * [Legacy team plans](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) don't have access to dbt Wizard and we recommend moving to a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing) to use it and other features. * Certain features like [natural prompts in Canvas](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md) are only available on Enterprise and Enterprise+ plans. * dbt admin permissions to change account settings and configure providers. * A development environment on a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing updates. #### Manage account access to AI features Eligible accounts, including new ones, have AI features on by default, so most users can open dbt Wizard without an admin changing anything first. Admins with governance or compliance requirements can turn AI off or back on anytime by following these steps: 1. Navigate to **Account settings** in the navigation menu. 2. Under **Settings**, confirm the account you want to change. 3. Click **Edit** in the top right corner. 4. Turn **Enable account access to AI features** off to disable access or on to enable it. 5. Click **Save**. Turning AI features off applies to the whole account, including dbt Wizard and dbt Copilot. The [dbt Support Assistant](https://docs.getdbt.com/docs/dbt-support.md?version=2#ask-dbt-support-assistant) is managed separately by dbt Labs (contact the [dbt Support team](mailto:support@dbtlabs.com) for more info). #### Supported AI providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). dbt Copilot supports a smaller set of providers. Refer to [dbt Copilot](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md#configure-ai-provider-for-dbt-copilot) for its providers and setup steps. #### Configure AI provider A dbt platform account on any plan can use a dbt managed provider to get started right away. Accounts on Starter, Enterprise, and Enterprise+ plans can also configure a custom AI provider by BYOK. If you use BYOK, you will incur API calls and associated charges from that provider. The following instructions explain how to configure a dbt Labs managed or BYOK AI provider for dbt Wizard. dbt managed AI providers are administered by dbt Labs and don’t require a user-provided API key. The AI providers shown in **Account settings** may change as dbt Labs adds managed models. For the current list of models and providers, refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model). **To configure a provider:** 1. Click your account name and select **Account settings** in the side menu. 2. Under **Settings**, click **AI features**. 3. Under **AI providers**, click **Edit** to configure the AI integration. 4. For each provider, select your **Key management** option from the dropdown, then follow these steps for your provider below. ##### Anthropic **Managed by dbt Labs** Use this option as a managed alternative to the default OpenAI model. 1. Unless not selected, select **dbt Labs** from the list to use dbt Labs' managed Anthropic key. 2. Click **Save**. **Managed by you (BYOK)** 1. Select **Anthropic** from the options. 2. Enter your Anthropic API key. 3. Click **Save**. Embedding limitations When using an Anthropic API key, dbt continues to use the dbt Labs-managed OpenAI key for embeddings in `text_to_sql` MCP tools, since Anthropic doesn't natively provide embeddings. ##### OpenAI **Managed by dbt Labs** This is the default option and use it if you want to use dbt managed AI provider keys. Refer to [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md?version=2.0) for more information. 1. Unless not selected, select the toggle for **dbt Labs** to use a dbt Labs' managed key. 2. Click **Save**. [![Example of the dbt Labs integration page](/img/docs/dbt-platform/account-integration-dbtlabs.png?v=2 "Example of the dbt Labs integration page")](#)Example of the dbt Labs integration page **Managed by you (BYOK)** 1. Select **OpenAI** from the options. 2. Enter your OpenAI API key. 3. Click **Save**. [![Example of the OpenAI integration page](/img/docs/dbt-platform/account-integration-openai.png?v=2 "Example of the OpenAI integration page")](#)Example of the OpenAI integration page Data residency limitation OpenAI projects with [data residency controls](https://platform.openai.com/docs/guides/your-data#data-residency-controls) enabled and configured for the United States (project region set to US) don't currently support BYOK. These projects can only use the API key in the dbt platform configuration. Specifying custom endpoints required for data residency isn’t yet supported, and we’re evaluating a solution for this. To use BYOK, ensure your OpenAI project doesn’t have data residency controls enabled. Projects without project region settings will use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. * For BYOK, enable the latest text generation models as well as the `text-embedding-3-small` model. * Ensure your project doesn't have data residency controls enabled. Projects without project region settings use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. ##### Azure AI Foundry **Managed by you only (BYOK)** To learn about deploying models on Azure, refer to [Deploy models on Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-openai) and [Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/). Configure credentials for your Azure AI Foundry deployment in dbt as follows: 1. Locate your deployment details in the Azure portal, either in Azure AI Foundry or Azure OpenAI. 2. Enter your Azure API key. 3. Enter the **Endpoint**, **API Version**, and **Deployment / AI model name**. 4. Click **Save**. Use the full Azure Target URI For the **Endpoint** field, enter the full Azure Target URI from Azure — not just the base endpoint. Entering only the base endpoint, for example `https://<resource>.openai.azure.com`, prevents credential validation and blocks setup. Supported formats include: * `https://<resource>.openai.azure.com/openai/deployments/<deployment>/chat/completions?api-version=<version>` * `https://<resource>.openai.azure.com/openai/responses?api-version=<version>` [![Example of the Azure AI Foundry integration section](/img/docs/dbt-platform/account-integration-azure-manual.png?v=2 "Example of the Azure AI Foundry integration section")](#)Example of the Azure AI Foundry integration section To bring your own key instead of using dbt Labs' managed infrastructure, refer to [Configure BYOK for dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md). BYOK is available on Starter, Enterprise, and Enterprise+ plans. #### Manage spend dbt Wizard is available to dbt platform Developer, Starter, Enterprise, and Enterprise+ plans. Legacy team plans don't have access to dbt Wizard but can [upgrade](https://www.getdbt.com/pricing) to Starter or Enterprise-tiered plans to access. Enterprise and Enterprise+ accounts get monthly usage credits, and all other plans start with a 30-day trial with $100 in usage credits. When your credits or trial run out (whichever comes first), set a spend limit so your team can keep working with minimal interruption: * [Manage your spend limit](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md#manage-your-spend-limit) to cap what the account can spend on AI usage. * [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md) explains what each model costs. * With [BYOK](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md), your AI provider bills you directly and the usage doesn't draw from your consumption pool. #### Related docs * [About dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [Configure BYOK in dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) * [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) * [Data & Privacy in dbt platform](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#privacy-and-data) --- ### Managing usage Login required | Usage-based From dbt, click on your account name in the left side menu and select **Account settings**. The **Billing** option will be on the left side menu under the **Settings** heading. Here, you can view individual available plans and the features provided for each. ##### Usage notifications Every plan automatically sends email alerts when 75%, 90%, and 100% of usage estimates have been reached. * Starter plan — All users within the Owner group receive alerts. * Enterprise-tier plans — All users with the Account Admin and Billing Admin [permission sets](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#permission-sets) receive alerts. Users cannot opt out of these emails. To have additional users to receive these alert emails, assign them the applicable permissions mentioned earlier. Note that your usage may already be higher than the percentage indicated in the alert due to your usage pattern and minor latency times. ##### How do I stop usage from accruing? There are 2 options to disable models from being built and charged: 1. Open the **Job Settings** of every job and navigate to the **Triggers** section. Disable the **Run on Schedule** and set the **Continuous Integration** feature **Run on Pull Requests?** to **No**. Check your workflows to ensure that you are not triggering any runs via the dbt API. This option will enable you to keep your dbt jobs without building more models. 2. Alternatively, you can delete some or all of your dbt jobs. This will ensure that no runs are kicked off, but you will permanently lose your job(s). --- ### Migrating to account-specific access URLs dbt platform Account-specific access URLs are being assigned to dbt accounts as part of our ongoing efforts to improve your experience and strengthen security. Each account will receive its own unique dbt platform and [API access URLs](https://docs.getdbt.com/docs/dbt-apis/overview.md) on the **dbt.com** domain rather than sharing generic **getdbt.com** region URLs. For example: * Old: `https://cloud.getdbt.com` * New: `https://ab123.us1.dbt.com` #### What does this mean for me? If your account has been assigned a new access URL, please review the [migration timeline](#migration-timeline) and update your account's external integrations using the [integration checklist](#integration-checklist). If your account has not been assigned a new access URL, you will receive an email and in-app notification with a scheduled enablement date. When the change takes effect, you'll be automatically redirected to your **dbt.com** account-specific access URL. Account sign-in remains the same. Both the new **dbt.com** and **getdbt.com** region URLs will support integrations according to the dates on the [migration timeline](#migration-timeline). info Before assignment, if your organization uses network allowlisting, add the **dbt.com** domain to your allowlists. For single-tenant accounts, there will be no change to IP addresses. For multi-tenant accounts, refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for updated IPs. If your account has [IP restrictions](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md) enabled, review [Validating IP restrictions](#validating-ip-restrictions) before your assignment date. #### Migration timeline Accounts across all regions and service providers are being assigned new access URLs. **getdbt.com** region URLs will continue to support integrations until the scheduled deprecation date, unless otherwise specified. | Stage | Status | Timeline | | ----------------------------------------------- | ------------ | ---------------------------------------------- | | Multi-tenant **dbt.com** access URL assignment | ✅ Completed | January 2026 | | Single tenant **dbt.com** access URL assignment | In Progress | April - September 2026 | | **getdbt.com** region URL deprecation | Scheduled | February 3, 2027 (previously November 1, 2026) | #### Integration checklist Review the following checklist before the **getdbt.com** region URL deprecation date documented in the [migration timeline](#migration-timeline). Update each integration that uses a dbt access URL. If your integration isn't in this list, please speak with your IT or applicable team to identify whether you need to take action. dbt Labs updates the dbt GitHub application, the Slack application, and GitLab webhooks automatically. Inbound Azure DevOps service hooks are not included. ##### Identity and access management | Integration | Action required | | ------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | [Google Workspace SSO](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-google-workspace.md#creating-credentials) | Update or add an OAuth client | | [SCIM (Okta)](https://docs.getdbt.com/docs/platform/manage-access/scim.md#set-up-dbt-cloud) | Update the SCIM base URL in Okta | ##### Git providers GitLab and Azure DevOps repositories connected before your account had an account-specific access URL keep using your legacy **getdbt.com** URL for OAuth until you update them. Repositories connected after that already use your new access URL. For GitLab, account admins can update an existing GitLab application to your new account access URL in dbt platform. See [Update redirect URIs in dbt platform](#update-redirect-uris-in-dbt-platform). You can't yet update an existing Azure DevOps repository. Instead, recreate the repository to generate a redirect URI based on your new account access URL. | Integration | Action required | | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | [GitLab (dbt Labs app)](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md#setting-up-a-gitlab-oauth-application) | Add the new redirect URI to your GitLab group application, then [update it in dbt platform](#update-redirect-uris-in-dbt-platform) | | [GitLab (bring-your-own app)](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md#setting-up-a-gitlab-oauth-application) | Add the new redirect URI to your GitLab group application, then [update it in dbt platform](#update-redirect-uris-in-dbt-platform) | | [Azure DevOps (service principal)](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md) | Update or add an app registration | | GitHub On-premises | Contact [dbt Labs Support](mailto:support@getdbt.com) | ##### Data platform connections Data platform connections created before your account had an account-specific access URL keep using your legacy **getdbt.com** URL for OAuth until you update them. Connections created after that already use your new access URL, so they need no changes. Account admins can update older Snowflake, Databricks, and BigQuery OAuth connections to your new access URL directly in dbt platform. See [Update redirect URIs in dbt platform](#update-redirect-uris-in-dbt-platform). | Integration | Action required | | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | [Snowflake OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md#subdomain-migration) | Add the new redirect URI to your security integration, then [update it in dbt platform](#update-redirect-uris-in-dbt-platform) | | [Snowflake External OAuth](https://docs.getdbt.com/docs/platform/manage-access/snowflake-external-oauth.md#identity-provider-configuration) | Update the redirect URI in your IdP application | | [Databricks OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-databricks-oauth.md) | Add the new redirect URI to your Databricks OAuth app, then [update it in dbt platform](#update-redirect-uris-in-dbt-platform) | | [BigQuery OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md) | Add the new redirect URI to your Google Cloud OAuth client, then [update it in dbt platform](#update-redirect-uris-in-dbt-platform) | | [Redshift External OAuth](https://docs.getdbt.com/docs/platform/manage-access/redshift-external-oauth.md) | Update the redirect URI in your IdP application | ##### Update redirect URIs in dbt platform If your account has an account-specific access URL, dbt platform flags each Snowflake, Databricks, or BigQuery OAuth connection and GitLab application that still uses a legacy redirect URI. Updating connections requires the Account Admin or Project Creator [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md). Updating a GitLab application requires the Account Admin permission set. 1. In dbt platform **Account settings**, go to **Connections** (or **Integrations** for GitLab) and select a flagged item. The page shows the new redirect URI. 2. Add the new redirect URI to the OAuth configuration in your data platform or GitLab, alongside the existing one. The existing URI keeps working until you remove it. 3. In dbt platform, click **Update redirect URI**. 4. Test data platform connections by signing in again from your development credentials. For GitLab, repeat the [personal GitLab authorization flow](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md#personally-authenticating-with-gitlab) from **Account settings** > **Personal profile** > **Linked accounts**. Confirm that the relevant OAuth flow returns successfully to dbt platform before removing the old redirect URI. Updating the redirect URI doesn't interrupt active sessions. The new URI takes effect the next time a session expires and the user signs in again. If sign-in fails after the update, you have 7 days to roll back this change. Open the connection or GitLab application and click **Revert to previous URI** to restore the previous redirect URI while you troubleshoot. After you confirm that every integration works with the new URI, remove the old URI from your data platform or GitLab. If several connections share one OAuth app, add the new redirect URI once to that app, then update each connection in dbt platform. ##### Network and API | Integration | Action required | | ------------------------------------------------------------------------------------------- | -------------------------------------------------- | | Network allowlists | Add new access URLs to your allowlist policies | | Inbound webhooks | Update access URLs in your webhook configurations | | [API integrations](https://docs.getdbt.com/docs/dbt-apis/overview.md) | Update access URLs in your API clients | | [Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs) | Update access URLs in your Terraform configuration | | Browser bookmarks | Update personal and shared bookmarks | #### Validating IP restrictions If you've received notification of a new access URL assignment and have IP restrictions enabled, review the network rules that point specifically at your **\*.getdbt.com** domain. These include VPN split-tunneling rules, proxy (PAC) rules, and firewall egress rules. Update these rules for your new **\*.dbt.com** access URL before your account's scheduled assignment date so access isn't disrupted. To confirm your new access URL is accessible, send a test request from the same network you'd normally use to reach your **\*.getdbt.com** URL. Replace `NEW_ACCESS_URL` with your account's new access URL and `ACCOUNT_ID` with your account ID: * **Browser:** Go to `https://NEW_ACCESS_URL/api/v2/accounts/ACCOUNT_ID/` * **Terminal:** Run `curl -s https://NEW_ACCESS_URL/api/v2/accounts/ACCOUNT_ID/` | Result | Response detail | What it means | | ---------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | ✅ Passed | `{ ... "detail": "Authentication credentials were not provided." ... }` | The request reached the API and passed IP restrictions. There's no authenticated session, which is expected for this test. | | ⚠️ Blocked | `{ ... "user_message": "Forbidden: Access denied" ... }` | The request's IP address isn't on your allowlist. Update your network egress rules for the new access URL and test again. | #### FAQs  Why are access URLs changing? We are transitioning from generic region URLs to account-specific URLs to enhance your experience and strengthen security. This change provides more descriptive URLs to improve dbt URL management, and enables stronger cookie and token policies to protect account data.  How do I know if my account has new access URLs? Sign in to your dbt platform account. If your browser location has `dbt.com`, your account has been assigned a new access URL. If your browser location has `getdbt.com`, your account has not been assigned a new access URL. You may refer to [API Access URLs](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md#api-access-urls).  How do I find my account's new access URLs? Refer to [API Access URLs](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md#api-access-urls).  What happens if my integrations are not updated by the getdbt.com deprecation date? You may not be able to access your account through your identity provider, and dbt platform may be degraded or inoperable. For assistance, contact [dbt Labs Support](mailto:support@getdbt.com).  What is unaffected by the migration to new access URLs? The following are unaffected: * Your dbt project code, models, and configurations * Your data platform connections and credentials * Your account settings, environments, jobs, and schedules * The dbt GitHub application, the Slack application, and GitLab webhooks * Egress private connectivity * The underlying dbt platform functionality For questions or assistance, contact [dbt Labs Support](mailto:support@getdbt.com). --- ### Navigate the interface dbt platform | Enterprise, Enterprise+ The Canvas interface contains an operator toolbar, operators, canvas, and more to help you access and transform data through a seamless drag-and-drop dbt model creation experience in dbt. This page offers comprehensive definitions and terminology of user interface elements, allowing you to navigate the Canvas landscape with ease. The Canvas interface is composed of: * **Navigation bars** — The top and ledft-side navigation bars contain options for switching between models in the workspace, opening existing or creating new models, uploading CSV data, previewing data and runs, and viewing helpful shortcuts. * **Operator toolbar** — Located at the top of the canvas area, the toolbar displays all the node categories available, as well as tools to help you develop: * **Input:** Source models and data * **Transform:** Data transformation tools * **Output:** Output model configurations * **[dbt Copilot](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md):** AI tools to help you build fast and efficiently * **SQL:** View your completed model's compiled SQL * **Operators** — Tiles that provide source data, perform specific transformations, and layer configurations (such as model, join, aggregate, filter, and so on). Use connectors to link the operators and build a complete data transformation pipeline. * **Canvas** — The main whiteboard space below the node toolbar. The canvas allows you to create or modify models through a sleek drag-and-drop experience. * **Configuration panel** — Each operator has a configuration panel that opens when you click on it. The configuration panel allows you to configure the operator, review the current model, preview changes to the table, view the SQL code for the node, and delete the operator. #### Operators The operator toolbar above the canvas contains the different transformation operators available to use. Use each operator to configure or perform specific tasks, like adding filters or joining models by dragging an operator onto the canvas. You can connect operators using the connector line, which allows you to form a complete model for your data transformation. [![Use the operator toolbar to perform different transformation operations.](/img/docs/dbt-platform/canvas/operators.png?v=2 "Use the operator toolbar to perform different transformation operations.")](#)Use the operator toolbar to perform different transformation operations. Here the following operators are available: ###### Input Input operators configure source data: * **Model explorer**: Select the model and columns you want to use. ###### Transform Transform operators shape your data: * **Join**: Define the join conditions and choose columns from both tables. * **Union:** Perform a `UNION` to remove duplicates or `UNION ALL` to prevent deduplicaation. * **Formula**: Add the formula to create a new column. Use [dbt Copilot](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md) to help generate SQL. * **Aggregate**: Specify the aggregation functions and the columns they apply to. To generate SQL with a natural language prompt, click the question mark (?) icon and use [dbt Copilot](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md). * **Pivot:** Select the column and values to create a pivot. * **Limit**: Set the maximum number of rows you want to return. * **Order**: Select the columns to sort by and the sort order. * **Filter**: Set the conditions to filter data. * **Rename:** Provide custom alias' for your columns. ###### Output model Output operators configure the names and location of your transformed data: * **Output model**: The final transformed dataset generated by a dbt model. You can only have one output model. When you click on each operator, it opens a configuration panel. The configuration panel allows you to configure the operator, review the current model, preview changes to the model, view the SQL code for the node, and delete the operator. [![The Canvas interface that contains a node toolbar and canvas.](/img/docs/dbt-platform/canvas/canvas.png?v=2 "The Canvas interface that contains a node toolbar and canvas.")](#)The Canvas interface that contains a node toolbar and canvas. If you have any feedback on additional operators that you might need, we'd love to hear it! Please contact your dbt Labs account team and share your thoughts. #### Canvas Canvas has a sleek drag-and-drop interface for creating and modifying dbt SQL models. It's like a digital whiteboard space for easily viewing and delivering trustworthy data. Use the canvas to: * Drag-and-drop operators to create and configure your model(s) * Generate SQL code using [dbt Copilot](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md) * Zoom in or out for better visualization * Version-control your dbt models * \[Coming soon] Test and document your created models [![The operator toolbar allows you to select different nodes to configure or perform specific tasks, like adding filters or joining models.](/img/docs/dbt-platform/canvas/operators.png?v=2 "The operator toolbar allows you to select different nodes to configure or perform specific tasks, like adding filters or joining models.")](#)The operator toolbar allows you to select different nodes to configure or perform specific tasks, like adding filters or joining models. ##### Connector Connectors allow you to connect your operators to create dbt models. Once you've added operators to the canvas: * Hover over the "+" sign next to the operator and click. * Drag your cursor between the operator's "+" start point to the other node you want to connect to. This should create a connector line. * As an example, to create a join, connect one operator to the "L" (Left) and the other to the "R" (Right). The endpoints are located to the left of the operator so you can easily drag the connectors to the endpoint. [![Click and drag your cursor to connect operators.](/img/docs/dbt-platform/canvas/connector.png?v=2 "Click and drag your cursor to connect operators.")](#)Click and drag your cursor to connect operators. #### Configuration panel Each operator has a configuration side panel that opens when you click on it. The configuration panel allows you to configure the operator, review the current model, preview changes, view the SQL code for the operator, and delete the operator. The configuration side panel has the following: * Configure tab — This section allows you to configure the operator to your specified requirements, such as using [dbt Copilot](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md) to generate SQL. * Input tab — This section allows you to view the data for the current source table. Not available for model operators. * Output tab — This section allows you to preview the data for the modified source model. * Code — This section allows you to view the underlying SQL code for the data transformation. [![A sleek drag-and-drop canvas interface that allows you to create or modify dbt SQL models.](/img/docs/dbt-platform/canvas/config-panel.png?v=2 "A sleek drag-and-drop canvas interface that allows you to create or modify dbt SQL models.")](#)A sleek drag-and-drop canvas interface that allows you to create or modify dbt SQL models. --- ### Optimize costs in dbt Login required | Usage-based dbt offers ways to optimize your model’s built usage and warehouse costs. ##### Best practices for optimizing cost with dbt State ###### Use `lag_tolerance` to reduce unnecessary model execution You can save even more time and compute by defining how old your data can be before a model should be triggered. We’ve introduced lag\_tolerance so that you can do things like differentiate development needs vs prod. For example: dbt\_project.yml ```yaml models: +state: lag_tolerance: "{{ '4h' if target.name == 'prod' else '7d' }}" ``` In this example, models in the `prod` target rebuild only when upstream data is more than 4 hours old. In all other environments, models wait 7 days before rebuilding. For more details, refer to the [`lag_tolerance` config reference](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). ###### Use selectors with `dbt build` to run limited upstream nodes In development, use [selectors](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) with `dbt build` to limit how many upstream nodes run. Nodes that are not selected can be [deferred](https://docs.getdbt.com/reference/node-selection/defer.md) instead of rebuilt, which avoids extra dbt State activity on those targets. Automatic `state:modified` selection in development may be supported in a future release. ###### Avoid conditional materializations Avoid conditional materialization patterns such as `table` in production and `view` in development for the same model. Different materializations between environments can prevent dbt State from matching targets correctly and reduce skip/clone effectiveness. ##### Best practices for optimizing successful models built You can reduce costs from successful models built while still following best practices. Combine the approaches below to fit your needs. If you exclude views from your scheduled job runs, set up a [merge job](#exclude-views-while-running-tests) to deploy updated view logic when changes are detected. ###### Exclude views in a dbt job Many dbt users utilize views, which don’t always need to be rebuilt every time you run a job. For any jobs that contain views that *do not* include macros that dynamically generate code (for example, case statements) based on upstream tables and also *do not* have tests, you can implement these steps: 1. Go to your current production deployment job in dbt. 2. Modify your command to include: `--exclude config.materialized:view`. 3. Save your job changes. If you have views that contain macros with case statements based on upstream tables, these will need to be run each time to account for new values. If you still need to test your views with each run, follow the [Exclude views while still running tests](#exclude-views-while-running-tests) best practice to create a custom selector. ###### Exclude views while running tests Running tests for views in every job run can help keep data quality intact and save you from the need to rerun failed jobs. To exclude views from your job run while running tests, you can follow these steps to create a custom [selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) for your job command. 1. Open your dbt project in the Studio IDE. 2. Add a file called `selectors.yml` in your top-level project folder. 3. In the file, add the following code: ```yaml selectors: - name: skip_views_but_test_views description: > A default selector that will exclude materializing views without skipping tests on views. default: true definition: union: - union: - method: path value: "*" - exclude: - method: config.materialized value: view - method: resource_type value: test ``` 4. Save the file and commit it to your project. 5. Modify your dbt jobs to include (Applies to dbt v1.12 and later) `dbt build --select selector:skip_views_but_test_views`. ###### Build only changed views If you want to ensure that you're building views whenever the logic is changed, create a merge job that gets triggered when code is merged into main: 1. Ensure you have a [CI job setup](https://docs.getdbt.com/docs/deploy/ci-jobs.md) in your environment. 2. Create a new [deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) and call it “Merge Job". 3. Set the **Environment** to your CI environment. Refer to [Types of environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md#types-of-environments) for more details. 4. Set **Commands** to: `dbt run -s state:modified+`. Executing `dbt build` in this context is unnecessary because the CI job was used to both run and test the code that just got merged into main. 5. Under the **Execution Settings**, select the default production job to compare changes against: * **Defer to a previous run state** — Select the “Merge Job” you created so the job compares and identifies what has changed since the last merge. 6. Follow [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md) to create a script that triggers the dbt API to run your job after a merge, or watch this [video](https://www.loom.com/share/e7035c61dbed47d2b9b36b5effd5ee78?sid=bcf4dd2e-b249-4e5d-b173-8ca204d9becb). The merge job immediately deploys PR changes to production and keeps production views current with your codebase while staying cost-efficient. Decide whether this change is right for your dbt project. ##### Rework inefficient models ###### Job Insights tab To reduce warehouse spend, use the **Insights** tab on the **Job** page to find which models take longest to build. The chart shows each model's average run time over its last 20 runs; the slowest models are prime candidates for optimization. ###### Model Timing tab To see how long each model takes within a specific run, select that run on the **Run History** page and click the **Model Timing** tab. Once you've identified which models could be optimized, check out these other resources that walk through how to optimize your work: * [Build scalable and trustworthy data pipelines with dbt and BigQuery](https://services.google.com/fh/files/misc/dbt_bigquery_whitepaper.pdf) * [Best Practices for Optimizing Your dbt and Snowflake Deployment](https://www.snowflake.com/wp-content/uploads/2021/10/Best-Practices-for-Optimizing-Your-dbt-and-Snowflake-Deployment.pdf) * [How to optimize and troubleshoot dbt models on Databricks](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md) For answers to common plan and billing questions, refer to [Billing FAQs](https://docs.getdbt.com/docs/platform/billing-faqs.md). --- ### Plans and billing dbt platform dbt offers several [plans](https://www.getdbt.com/pricing) with different features that meet your needs. We may make changes to our plan details from time to time. We'll always let you know in advance, so you can be prepared. The following section explains how billing works in each plan. ##### Developer plan billing Developer plans are free and include one Developer license and 3,000 models each month. Models are refreshed at the beginning of each calendar month. If you exceed 3,000 models, any subsequent runs will be canceled until models are refreshed or until you upgrade to a paid plan. The rest of the dbt platform is still accessible, and no work will be lost. Included model entitlements may differ from what's shown here, depending on the terms when you signed up. ##### Starter plan billing Starter customers pay monthly via credit card for seats and usage, and accounts include 15,000 models monthly. Seats are charged upfront at the beginning of the month. If you add seats during the month, seats will be prorated and charged on the same day. Seats removed during the month will be reflected on the next invoice and are not eligible for refunds. You can change the credit card information and the number of seats from the billings section anytime. Accounts will receive one monthly invoice that includes the upfront charge for the seats and the usage charged in arrears from the previous month. Usage is calculated and charged in arrears for the previous month. If you exceed 15,000 models in any month, you will be billed for additional usage on your next invoice. Additional usage is billed at the rates on our [pricing page](https://www.getdbt.com/pricing). Included models that are not consumed do not roll over to future months. Included model entitlements may differ from what's shown here, depending on the terms when you signed up. ##### Enterprise plan billing As an Enterprise customer, you pay annually via invoice, monthly in arrears for additional usage (if applicable), and may benefit from negotiated usage rates. Please refer to your order form or contract for your specific pricing details, or [contact the account team](https://www.getdbt.com/contact-demo) with any questions. Enterprise plan billing information is not available in the dbt UI. Changes are handled through your dbt Labs Solutions Architect or account team manager. ##### Legacy plans Customers who purchased the dbt Starter plan (formerly Team) plan before August 11, 2023, remain on a legacy pricing plan as long as your account is in good standing. The legacy pricing plan is based on seats and includes unlimited models, subject to reasonable use. dbt Labs may institute use limits if reasonable use is exceeded. Additional features, upgrades, or updates may be subject to separate charges. Any changes to your current plan pricing will be communicated in advance according to our Terms of Use. --- ### Studio IDE keyboard shortcuts dbt platform The Studio IDE provides keyboard shortcuts, features, and development tips to help you work faster and be more productive. Use this Studio IDE page to help you quickly reference some common operations. | Shortcut description | macOS | Windows | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------- | | View the full list of editor shortcuts to help your development, such as adding a line comment, changing tab display size, building modified models, changing editor font size, and more. | Fn-F1 | Fn-F1 | | Select a file to open. | Command-O | Control-O | | Close the currently active editor tab. | Option-W | Alt-W | | Preview code. | Command-Enter | Control-Enter | | Compile code. | Command-Shift-Enter | Control-Shift-Enter | | Reveal a list of dbt functions in the editor. | Enter two underscores `__` | Enter two underscores `__` | | Open the command palette to invoke dbt commands and actions. | Command-P / Command-Shift-P | Control-P / Control-Shift-P | | Multi-edit in the editor by selecting multiple lines. | Option-Click / Shift-Option-Command / Shift-Option-Click | Alt-Click / Shift-Alt-Click | | Open the [**Invocation History Drawer**](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#invocation-history) located at the bottom of the IDE. | Control-backtick (or Control + \`) | Control-backtick (or Ctrl + \`) | | Add a block comment to the selected code. SQL files will use the Jinja syntax `({# #})` rather than the SQL one `(/* */)`. | Shift-Option-A | Shift-Alt-A | #### Related docs * [Quickstart guide](https://docs.getdbt.com/guides.md) * [About dbt](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) * [Develop in the Cloud](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) --- ### Supported browsers dbt platform To have the best experience with dbt, we recommend using the latest versions of the following browsers: * [Google Chrome](https://www.google.com/chrome/) — Latest version is fully supported in dbt * [Mozilla Firefox](https://www.mozilla.org/en-US/firefox/) — Latest version is fully supported in dbt * [Apple Safari](https://www.apple.com/safari/) — Latest version support provided on a best-effort basis * [Microsoft Edge](https://www.microsoft.com/en-us/edge?form=MA13FJ\&exp=e00) — Latest version support provided on a best-effort basis dbt provides two types of browser support: * Fully supported — dbt is fully tested and supported on these browsers. Features display and work as intended. * Best effort — You can access dbt on these browsers. Features may not display or work as intended. You may still be able to access and use dbt even without using the latest recommended browser or an unlisted browser. However, some features might not display as intended. note To improve your experience using dbt, we suggest that you turn off ad blockers. ##### Browser sessions A session is a period of time during which you’re signed in to a dbt account from a browser. If you close your browser, it will end your session and log you out. You'll need to log in again the next time you try to access dbt. If you've logged in using [SSO](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md), you can customize your maximum session duration, which might vary depending on your identity provider (IdP). --- ### Tenancy dbt platform dbt is available in single-tenant (virtual private) and multi-tenant SaaS configurations. Many multi-tenant accounts use cell-based hosting with account-specific access URLs. For more information, refer to [Multi-cell hosting](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#multi-cell-hosting). #### Multi-tenant The Multi Tenant (SaaS) deployment environment refers to the SaaS dbt application hosted by dbt Labs. This is the most commonly used deployment and is completely managed and maintained by dbt Labs, the makers of dbt. As a SaaS product, a user can quickly [create an account](https://www.getdbt.com/signup/) on our North American servers and get started using the dbt and related services immediately. *If your organization requires cloud services hosted on EMEA or APAC regions*, please [contact us](https://www.getdbt.com/contact/). The deployments are hosted on AWS, Azure, or GCP and are always kept up to date with the currently supported dbt versions, software updates, and bug fixes. ###### Multi-cell hosting Multi-cell (also called cell-based hosting) means your dbt platform account runs in a cell: a defined slice of our shared SaaS stack with its own capacity, scaling, and status boundaries. Cells segment how we run multi-tenant infrastructure at scale; you still remain on the same multi-tenant product managed by dbt Labs. Cell-based hosting is different from [single tenant](#single-tenant) in that it doesn't provide a dedicated virtual private cloud (VPC) or isolated cloud account on its own. Generally, your plan and the features available to you stay the same as for other multi-tenant accounts in your managed cloud provider and region (described in [Available features](#available-features)). The main differences are in some setup details, such as the URL you use to sign in, which IP addresses to allow, and which status page to monitor if something goes wrong in your cell. Refer to the [API access URLs](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md#api-access-urls) section for more information. #### Single tenant The single tenant deployment environment provides a hosted alternative to the multi-tenant (SaaS) dbt environment. While still managed and maintained by dbt Labs, single tenant dbt instances provide dedicated infrastructure in a VPC environment. This is accomplished by spinning up all the necessary infrastructure with a re-usable Infrastructure as Code (IaC) deployment built with [Terraform](https://www.terraform.io/). The single tenant infrastructure lives in a dedicated AWS, Azure, or GCP account and can be customized with certain configurations, such as firewall rules, to limit inbound traffic or hosting in a specific region. A few common reasons for choosing a single tenant deployment over the Production SaaS product include: * A requirement that the dbt application be hosted in a dedicated VPC that is logically separated from other customer infrastructure * A desire for multiple isolated dbt instances for testing, development, etc *To learn more about setting up a dbt single tenant deployment, [please contact our sales team](mailto:sales@getdbt.com).* #### Available features The following table outlines which dbt features are supported on the different SaaS options available today. For more information about feature availability, please [contact us](https://www.getdbt.com/contact/). Cell-based (multi-cell) accounts are still multi-tenant SaaS. Use the multi-tenant column for your cloud provider (for example, the AWS Multi-tenant column). This table does not list features by cell. For differences in hosting and access URLs compared with single tenant, refer to [Multi-cell hosting](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#multi-cell-hosting). | Feature | AWS Multi-tenant | AWS single tenant | Azure multi-tenant | Azure single tenant | GCP multi-tenant | GCP single tenant | | --------------------------- | ---------------- | ----------------- | ------------------ | ------------------- | ---------------- | ----------------- | | Audit logs | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Continuous integration jobs | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | dbt platform CLI | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Studio IDE | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | dbt Wizard | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Catalog | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Mesh | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Semantic Layer | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Discovery API | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | IP restrictions | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Orchestrator | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | PrivateLink egress | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | PrivateLink ingress | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | | Webhooks (Outbound) | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | --- ### The dbt platform features dbt platform dbt platform is the fastest and most reliable way to deploy dbt. Develop, test, schedule, document, and investigate data models all in one browser-based UI. In addition to providing a hosted architecture for running dbt across your organization, dbt comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring and alerting, an integrated development environment (Studio IDE), and allows you to develop and run dbt commands from your local command line interface (CLI) or code editor. dbt's [flexible plans](https://www.getdbt.com/pricing/) and features make it well-suited for data teams of any size — sign up for your [free 14-day trial](https://www.getdbt.com/signup/)! [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) ###### [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) [Use the CLI for the dbt platform to develop, test, run, and version control dbt projects and commands, directly from the command line.](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) ###### [dbt Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) [The IDE is the easiest and most efficient way to develop dbt models, allowing you to build, test, run, and version control your dbt projects directly from your browser.](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/canvas.md) ###### [dbt Canvas](https://docs.getdbt.com/docs/platform/canvas.md) [Develop with Canvas, a seamless drag-and-drop experience that helps analysts quickly create and visualize dbt models in dbt.](https://docs.getdbt.com/docs/platform/canvas.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/platform/wizard-overview.md) ###### [dbt Wizard\*](https://docs.getdbt.com/docs/platform/wizard-overview.md) [dbt Wizard is an AI agent that helps teams ship higher-quality dbt changes faster and with less risk. Built for governed data development in dbt, dbt Wizard has full project context and assesses how changes impact the projects.](https://docs.getdbt.com/docs/platform/wizard-overview.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/environments-in-dbt.md) ###### [Manage environments](https://docs.getdbt.com/docs/environments-in-dbt.md) [Set up and manage separate production and development environments in dbt to help engineers develop and test code more efficiently, without impacting users or data.](https://docs.getdbt.com/docs/environments-in-dbt.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-scheduler.md) ###### [Schedule and run dbt jobs](https://docs.getdbt.com/docs/deploy/job-scheduler.md) [Create custom schedules to run your production jobs. Schedule jobs by day of the week, time of day, or a recurring interval. Decrease operating costs by using webhooks to trigger CI jobs and the API to start jobs.](https://docs.getdbt.com/docs/deploy/job-scheduler.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-notifications.md) ###### [Notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) [Set up and customize job notifications in dbt to receive email or slack alerts when a job run succeeds, fails, or is cancelled. Notifications alert the right people when something goes wrong instead of waiting for a user to report it.](https://docs.getdbt.com/docs/deploy/job-notifications.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/run-visibility.md) ###### [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) [View the history of your runs and the model timing dashboard to help identify where improvements can be made to the scheduled jobs.](https://docs.getdbt.com/docs/deploy/run-visibility.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) ###### [Host & share documentation](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) [dbt hosts and authorizes access to dbt project documentation, allowing you to generate data documentation on a schedule for your project. Invite teammates to the dbt platform to collaborate and share your project's documentation.](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/git/connect-github.md) ###### [Supports GitHub, GitLab, AzureDevOps](https://docs.getdbt.com/docs/platform/git/connect-github.md) [Seamlessly connect your git account to the dbt platform and provide another layer of security to dbt. Import new repositories, trigger continuous integration, clone repos using HTTPS, and more!](https://docs.getdbt.com/docs/platform/git/connect-github.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) ###### [Enable Continuous Integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [Configure dbt to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool in any analyst's belt.](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [![](/img/icons/dbt-bit.svg)](https://www.getdbt.com/security/) ###### [Security](https://www.getdbt.com/security/) [Manage risk with SOC-2 compliance, CI/CD deployment, RBAC, and ELT architecture.](https://www.getdbt.com/security/) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) ###### [Visualize and orchestrate exposures\*](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) [Configure downstream exposures automatically from dashboards and understand how models are used in downstream tools. Proactively refresh the underlying data sources during scheduled dbt jobs.](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) ###### [dbt Semantic Layer\*](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) [Use the dbt Semantic Layer to define metrics alongside your dbt models and query them from any integrated analytics tool. Get the same answers everywhere, every time.](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) ###### [Discovery API\*](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) [Enhance your workflow and run ad-hoc queries, browse schema, or query the dbt Semantic Layer. dbt serves a GraphQL API, which supports arbitrary queries.](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/explore-projects.md) ###### [dbt Catalog\*](https://docs.getdbt.com/docs/explore/explore-projects.md) [Learn about dbt Catalog and how to interact with it to understand, improve, and leverage your data pipelines.](https://docs.getdbt.com/docs/explore/explore-projects.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/dbt-insights.md) ###### [dbt Insights\*](https://docs.getdbt.com/docs/explore/dbt-insights.md) [Learn how to query data and perform exploratory data analysis using dbt Insights.](https://docs.getdbt.com/docs/explore/dbt-insights.md) [](https://docs.getdbt.com/docs/explore/cost-insights.md) [](https://docs.getdbt.com/docs/explore/cost-insights.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/cost-insights.md)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") ###### Cost Insights\* Track warehouse compute costs and understand the impact of optimizations across your dbt projects and models. <br /> \*These features are available on [selected plans](https://www.getdbt.com/pricing/). #### Related docs * [dbt plans and pricing](https://www.getdbt.com/pricing/) * [Quickstart guides](https://docs.getdbt.com/docs/get-started-dbt.md) * [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) --- ### Using defer in dbt dbt platform [Defer](https://docs.getdbt.com/reference/node-selection/defer.md) is a powerful feature that allows developers to only build, run, and test models(Applies to dbt v1.11 and later) and functions they've edited, without having to build and run all the models(Applies to dbt v1.11 and later) and functions that come before them (upstream parents). dbt powers this by using a production manifest for comparison and resolves `{{ ref() }}`(Applies to dbt v1.11 and later) and `{{ function() }}` calls with upstream production artifacts. Both the Studio IDE and the dbt CLI enable users to natively defer to production metadata directly in their development workflows. [![Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream.](/img/docs/reference/defer-diagram.png?v=2 "Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream.")](#)Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream. When using `--defer`, dbt will follow this order of execution for resolving `{{ ref() }}`(Applies to dbt v1.11 and later) and `{{ function() }}` calls. 1. If a development version of a deferred relation exists, dbt preferentially uses the development database location when resolving the reference. 2. If a development version doesn't exist, dbt uses the staging locations of parent relations based on metadata from the staging environment. 3. If no development version and no staging environment exist, dbt uses the production locations of parent relations based on metadata from the production environment. Note that dbt only defers to one environment per invocation — either staging or production. **Note:** Passing the `--favor-state` flag will always resolve refs using staging metadata if available; otherwise, it defaults to production metadata regardless of the presence of a development relation, skipping step #1. For a clean slate, it's a good practice to drop the development schema at the start and end of your development cycle. If you require additional controls over production data, create a [staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment), and dbt will use that, rather than the production environment, to resolve `{{ ref() }}`(Applies to dbt v1.11 and later) and `{{ function() }}` calls. #### Configuration options There are several ways to configure deferral in dbt, depending on how your project is set up: * **dbt platform**: When using the Studio IDE or dbt platform CLI, deferral is managed automatically. You can optionally configure `defer-env-id` to target a specific environment. Refer to [Set up deferral in the dbt platform](#set-up-deferral-in-the-dbt-platform) for setup details. * **Self-managed deployments (`defer_to_target`)**: If you have [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) enabled, set `defer_to_target` in `profiles.yml` to specify which environment dbt State defers to. Refer to [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) and [Configuring deferral in dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md) for setup details. * **Manual defer (`--state`)**: Pass `--state` or `--defer-state` explicitly to point dbt to a specific `manifest.json`. Refer to [Defer](https://docs.getdbt.com/reference/node-selection/defer.md) for usage details. Use dbt State for smarter deferral [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") In self-managed deployments with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) enabled, `state:*` selectors can compare each node against its own last execution instead of a single `manifest.json` — no manifest management needed. Refer to [dbt State-powered `state:*` selectors](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md#dbt-state-powered-state-selectors). #### Set up deferral in the dbt platform * You must select the **[Production environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment)** checkbox in the **Environment Settings** page. * This can be set for one deployment environment per dbt project. * You must have a successful job run first. When using defer, it compares artifacts from the most recent successful production job, excluding CI jobs. The following sections cover how to enable and configure deferral in the Studio IDE, the dbt platform CLI, and how to customize which environment dbt defers to. ##### Defer in the dbt IDE To use deferral in the Studio IDE, you must have production artifacts generated by a deploy job. dbt will first check for these artifacts in your Staging environment (if available), or else in the Production environment. The defer feature in the Studio IDE won't work if a Staging environment exists but no deploy job has run. This is because the necessary metadata to power defer won't exist until a deploy job has run successfully in the Staging environment. To enable defer in the Studio IDE, toggle the **Defer to staging/production** button on the command bar. Once enabled, dbt will: 1. Pull down the most recent manifest from the staging or production environment for comparison 2. Pass the `--defer` flag to the command (for any command that accepts the flag) For example, if you were to start developing on a new branch with [nothing in your development schema](https://docs.getdbt.com/reference/node-selection/defer.md#usage), edit a single model, and run `dbt build -s state:modified` — only the edited model runs. Any `{{ ref() }}`(Applies to dbt v1.11 and later) and `{{ function() }}` calls resolve to the staging or production location of the referenced models(Applies to dbt v1.11 and later) and user-defined functions. [![Select the 'Defer to production' toggle on the bottom right of the command bar to enable defer in the Studio IDE.](/img/docs/dbt-platform/defer-toggle.png?v=2 "Select the 'Defer to production' toggle on the bottom right of the command bar to enable defer in the Studio IDE.")](#)Select the 'Defer to production' toggle on the bottom right of the command bar to enable defer in the Studio IDE. ##### Defer in dbt platform CLI One key difference between using `--defer` in the dbt platform CLI and the Studio IDE is that `--defer` is *automatically* enabled in the dbt platform CLI for all invocations, compared with production artifacts. You can disable it with the `--no-defer` flag. (Applies to dbt v2.0 and later) A dbt platform project id must be declared in your `dbt_project.yml` for automatic deferral to associate your project with what is declared in your `dbt_cloud.yml`. dbt\_project.yml ```yml dbt-cloud: project-id: '654321' ``` ##### Configure deferral environment ID The Studio IDE and dbt platform CLI both offer additional flexibility by letting you choose the source environment for deferral artifacts. You can manually set a `defer-env-id` key in either your `[dbt_project.yml](/reference/dbt_project.yml)` (dbt platform CLI and Studio IDE) or `dbt_cloud.yml` (dbt platform CLI only) file. By default, dbt will prefer metadata from the project's "Staging" environment (if defined). Otherwise, it uses "Production." For the full file reference, refer to [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md). [![Set the defer environment and the target name will change in the UI.](/img/docs/dbt-platform/defer-env-id.png?v=2 "Set the defer environment and the target name will change in the UI.")](#)Set the defer environment and the target name will change in the UI. dbt\_cloud.yml ```yml context: active-host: ... active-project: ... defer-env-id: '123456' ``` dbt\_project.yml ```yml dbt-cloud: project-id: '654321' defer-env-id: '123456' ``` --- ## Reference ### .dbtignore You can create a `.dbtignore` file in the root of your [dbt project](https://docs.getdbt.com/docs/build/projects.md) to specify files that should be **entirely** ignored by dbt. The file behaves like a [`.gitignore` file, using the same syntax](https://git-scm.com/docs/gitignore). Files and subdirectories matching the pattern will not be read, parsed, or otherwise detected by dbt—as if they didn't exist. **Examples** .dbtignore ```md # .dbtignore # ignore individual .py files not-a-dbt-model.py another-non-dbt-model.py # ignore all .py files **.py # ignore all .py files with "codegen" in the filename *codegen*.py # ignore all folders in a directory path/to/folders/** # ignore some folders in a directory path/to/folders/subfolder/** ``` --- ### About adapter object Your database communicates with dbt using an internal database adapter object. For example, BaseAdapter and SnowflakeAdapter. The Jinja object `adapter` is a wrapper around this internal database adapter object. `adapter` grants the ability to invoke adapter methods of that internal class via: * `{% do adapter.<method name> %}` -- invoke internal adapter method * `{{ adapter.<method name> }}` -- invoke internal adapter method and capture its return value for use in materialization or other macros For example, the adapter methods below will be translated into specific SQL statements depending on the type of adapter your project is using: * [adapter.dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) * [adapter.get\_missing\_columns](#get_missing_columns) * [adapter.expand\_target\_column\_types](#expand_target_column_types) * [adapter.get\_relation](#get_relation) or [load\_relation](#load_relation) * [adapter.get\_columns\_in\_relation](#get_columns_in_relation) * [adapter.create\_schema](#create_schema) * [adapter.drop\_schema](#drop_schema) * [adapter.drop\_relation](#drop_relation) * [adapter.rename\_relation](#rename_relation) * [adapter.quote](#quote) ##### Deprecated adapter functions The following adapter functions are deprecated, and will be removed in a future release. * [adapter.get\_columns\_in\_table](#get_columns_in_table) **(deprecated)** * [adapter.already\_exists](#already_exists) **(deprecated)** * [adapter\_macro](#adapter_macro) **(deprecated)** #### dispatch Moved to separate page: [dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) #### get\_missing\_columns **Args**: * `from_relation`: The source [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) * `to_relation`: The target [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) Returns a list of [Columns](https://docs.getdbt.com/reference/dbt-classes.md#column) that is the difference of the columns in the `from_table` and the columns in the `to_table`, i.e. (`set(from_relation.columns) - set(to_table.columns)`). Useful for detecting new columns in a source table. **Usage**: models/example.sql ```sql {%- set target_relation = api.Relation.create( database='database_name', schema='schema_name', identifier='table_name') -%} {% for col in adapter.get_missing_columns(target_relation, this) %} alter table {{this}} add column "{{col.name}}" {{col.data_type}}; {% endfor %} ``` #### expand\_target\_column\_types **Args**: * `from_relation`: The source [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) to use as a template * `to_relation`: The [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) to mutate Expand the `to_relation` table's column types to match the schema of `from_relation`. Column expansion is constrained to string and numeric types on supported databases. Typical usage involves expanding column types (from eg. `varchar(16)` to `varchar(32)`) to support insert statements. **Usage**: example.sql ```sql {% set tmp_relation = adapter.get_relation(...) %} {% set target_relation = adapter.get_relation(...) %} {% do adapter.expand_target_column_types(tmp_relation, target_relation) %} ``` #### get\_relation **Args**: * `database`: The database of the relation to fetch * `schema`: The schema of the relation to fetch * `identifier`: The identifier of the relation to fetch Returns a cached [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) object identified by the `database.schema.identifier` provided to the method, or `None` if the relation does not exist. **Usage**: example.sql ```sql {%- set source_relation = adapter.get_relation( database="analytics", schema="dbt_drew", identifier="orders") -%} {{ log("Source Relation: " ~ source_relation, info=true) }} ``` #### load\_relation **Args**: * `relation`: The [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) to try to load A convenience wrapper for [get\_relation](#get_relation). Returns the cached version of the [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) object, or `None` if the relation does not exist. **Usage**: example.sql ```sql {% set relation_exists = load_relation(ref('my_model')) is not none %} {% if relation_exists %} {{ log("my_model has already been built", info=true) }} {% else %} {{ log("my_model doesn't exist in the warehouse. Maybe it was dropped?", info=true) }} {% endif %} ``` #### get\_columns\_in\_relation **Args**: * `relation`: The [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) to find the columns for Returns a list of [Columns](https://docs.getdbt.com/reference/dbt-classes.md#column) in a table. **Usage**: example.sql ```sql {%- set columns = adapter.get_columns_in_relation(this) -%} {% for column in columns %} {{ log("Column: " ~ column, info=true) }} {% endfor %} ``` #### create\_schema **Args**: * `relation`: A relation object with the database and schema to create. Any identifier on the relation will be ignored. Creates a schema (or equivalent) in the target database. If the target schema already exists, then this method is a no-op. **Usage:** example.sql ```sql {% do adapter.create_schema(api.Relation.create(database=target.database, schema="my_schema")) %} ``` #### drop\_schema **Args**: * `relation`: A relation object with the database and schema to drop. Any identifier on the relation will be ignored. Drops a schema (or equivalent) in the target database. If the target schema does not exist, then this method is a no-op. The specific implementation is adapter-dependent, but adapters should implement a cascading drop, such that objects in the schema are also dropped. **Note**: this adapter method is destructive, so please use it with care! **Usage:** example.sql ```sql {% do adapter.drop_schema(api.Relation.create(database=target.database, schema="my_schema")) %} ``` #### drop\_relation **Args**: * `relation`: The Relation to drop Drops a Relation in the database. If the target relation does not exist, then this method is a no-op. The specific implementation is adapter-dependent, but adapters should implement a cascading drop, such that bound views downstream of the dropped relation are also dropped. **Note**: this adapter method is destructive, so please use it with care! The `drop_relation` method will remove the specified relation from dbt's relation cache. **Usage:** example.sql ```sql {% do adapter.drop_relation(this) %} ``` #### rename\_relation **Args**: * `from_relation`: The Relation to rename * `to_relation`: The destination Relation to rename `from_relation` to Renames a Relation the database. The `rename_relation` method will rename the specified relation in dbt's relation cache. **Usage:** example.sql ```sql {%- set old_relation = adapter.get_relation( database=this.database, schema=this.schema, identifier=this.identifier) -%} {%- set backup_relation = adapter.get_relation( database=this.database, schema=this.schema, identifier=this.identifier ~ "__dbt_backup") -%} {% do adapter.rename_relation(old_relation, backup_relation) %} ``` #### quote **Args**: * `identifier`: A string to quote Encloses `identifier` in the correct quotes for the adapter when escaping reserved column names etc. **Usage:** example.sql ```sql select 'abc' as {{ adapter.quote('table_name') }}, 'def' as {{ adapter.quote('group by') }} ``` #### get\_columns\_in\_table Deprecated This method is deprecated and will be removed in a future release. Please use [get\_columns\_in\_relation](#get_columns_in_relation) instead. **Args**: * `schema_name`: The schema to test * `table_name`: The table (or view) from which to select columns Returns a list of [Columns](https://docs.getdbt.com/reference/dbt-classes.md#column) in a table. models/example.sql ```sql {% set dest_columns = adapter.get_columns_in_table(schema, identifier) %} {% set dest_cols_csv = dest_columns | map(attribute='quoted') | join(', ') %} insert into {{ this }} ({{ dest_cols_csv }}) ( select {{ dest_cols_csv }} from {{ref('another_table')}} ); ``` #### already\_exists Deprecated This method is deprecated and will be removed in a future release. Please use [get\_relation](#get_relation) instead. **Args**: * `schema`: The schema to test * `table`: The relation to look for Returns true if a relation named like `table` exists in schema `schema`, false otherwise. models/example.sql ```text select * from {{ref('raw_table')}} {% if adapter.already_exists(this.schema, this.name) %} where id > (select max(id) from {{this}}) {% endif %} ``` #### adapter\_macro Deprecated This method is deprecated and will be removed in a future release. Please use [adapter.dispatch](#dispatch) instead. **Usage:** macros/concat.sql ```sql {% macro concat(fields) -%} {{ adapter_macro('concat', fields) }} {%- endmacro %} {% macro default__concat(fields) -%} concat({{ fields|join(', ') }}) {%- endmacro %} {% macro redshift__concat(fields) %} {{ fields|join(' || ') }} {% endmacro %} {% macro snowflake__concat(fields) %} {{ fields|join(' || ') }} {% endmacro %} ``` --- ### About adapter-specific behavior changes Some adapters can display behavior changes when certain flags are enabled. The following sections contain details about these adapter-specific behavior changes. [![](/img/icons/bigquery.svg)](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md) ###### [BigQuery](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md) [Behavior changes for the BigQuery adapter.](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md) [![](/img/icons/databricks.svg)](https://docs.getdbt.com/reference/global-configs/databricks-changes.md) ###### [Databricks](https://docs.getdbt.com/reference/global-configs/databricks-changes.md) [Behavior changes for the Databricks adapter.](https://docs.getdbt.com/reference/global-configs/databricks-changes.md) [![](/img/icons/redshift.svg)](https://docs.getdbt.com/reference/global-configs/redshift-changes.md) ###### [Redshift](https://docs.getdbt.com/reference/global-configs/redshift-changes.md) [Behavior changes for the Amazon Redshift adapter.](https://docs.getdbt.com/reference/global-configs/redshift-changes.md) [![](/img/icons/snowflake.svg)](https://docs.getdbt.com/reference/global-configs/snowflake-changes.md) ###### [Snowflake](https://docs.getdbt.com/reference/global-configs/snowflake-changes.md) [Behavior changes for the Snowflake adapter.](https://docs.getdbt.com/reference/global-configs/snowflake-changes.md) --- ### About as_bool filter The `as_bool` Jinja filter will coerce Jinja-compiled output into a boolean value (`True` or `False`), or return an error if it cannot be represented as a bool. ##### Usage: In the example below, the `as_bool` filter is used to coerce a Jinja expression to enable or disable a set of models based on the `target`. dbt\_project.yml ```yml models: my_project: for_export: enabled: "{{ (target.name == 'prod') | as_bool }}" ``` --- ### About as_native filter The `as_native` Jinja filter will coerce Jinja-compiled output into its Python native representation according to [`ast.literal_eval`](https://docs.python.org/3/library/ast.html#ast.literal_eval). The result can be any Python native type (set, list, tuple, dict, etc). To render boolean and numeric values, it is recommended to use [`as_bool`](https://docs.getdbt.com/reference/dbt-jinja-functions/as_bool.md) and [`as_number`](https://docs.getdbt.com/reference/dbt-jinja-functions/as_number.md) instead. Proceed with caution Unlike `as_bool` and `as_number`, `as_native` will return a rendered value regardless of the input type. Ensure that your inputs match expectations. --- ### About as_number filter The `as_number` Jinja filter will coerce Jinja-compiled output into a numeric value (integer or float), or return an error if it cannot be represented as a number. ##### Usage In the example below, the `as_number` filter is used to coerce an environment variables into a numeric value to dynamically control the connection port. profiles.yml ```yml my_profile: outputs: dev: type: postgres port: "{{ env_var('PGPORT') | as_number }}" ``` --- ### About behavior changes A behavior change is a deliberate update to dbt where the same project code and commands produce a different result than before—for example, a new validation error, a changed macro signature, or a breaking change to artifacts or structured logs. It is not a bug fix, a new warning, or a non-breaking addition. dbt gates these changes behind behavior change flags, so you control when to adopt the new behavior. The following are examples of behavior changes: * dbt begins raising a validation *error* that it didn't previously. * dbt changes the signature of a built-in macro. Your project has a custom re-implementation of that macro. This could lead to errors, because your custom re-implementation will be passed arguments it cannot accept. * A dbt adapter renames or removes a method that was previously available on the `{{ adapter }}` object in the dbt-Jinja context. The following are *not* behavior changes: * Fixing a bug where the previous behavior was defective, undesirable, or undocumented. * dbt begins raising a *warning* that it didn't previously. * dbt updates the language of human-friendly messages in log events. #### Behavior change flags These flags *must* be set in the `flags` dictionary in `dbt_project.yml`. They configure behaviors closely tied to project code, which means they should be defined in version control and modified through pull or merge requests, with the same testing and peer review. ##### Flag lifecycle Behavior change flags go through three phases of development: 1. **Introduced (disabled by default):** dbt adds logic to support both 'old' and 'new' behaviors. The 'new' behavior is gated behind a flag, disabled by default, preserving the old behavior. 2. **Mature (enabled by default):** The default value of the flag is switched to the new behavior by default. You can still preserve the old behavior, but you may see deprecation warnings. 3. **Removed (generally enabled):** The old behavior is removed from the dbt codebase(s). Most flags are supported indefinitely, but there is no commitment to supporting them forever. If a flag is removed, there will be significant advance warning. ##### Introduced in dbt v1 This table outlines which month of the **v1 Latest** release track in dbt and which version of dbt v1 contains the behavior change's introduction (disabled by default) or maturity (enabled by default). Flags marked 2.0 in the **Removed** column are removed in dbt v2 and the new behavior is always enabled. | Flag | **v1 Latest**: Intro | **v1 Latest**: Maturity | dbt v1: Intro | dbt v1: Maturity | Removed | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ----------------------- | -------------- | ---------------- | ------- | | [require\_explicit\_package\_overrides\_for\_builtin\_materializations](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_explicit_package_overrides_for_builtin_materializations.md) | 2024.04 | 2024.06 | 1.6.14, 1.7.14 | 1.8.0 | 2.0 | | [require\_resource\_names\_without\_spaces](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_spaces.md) | 2024.05 | 2025.05 | 1.8.0 | 1.10.0 | 2.0 | | [source\_freshness\_run\_project\_hooks](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md) | 2024.03 | 2025.05 | 1.8.0 | 1.10.0 | 2.0 | | [skip\_nodes\_if\_on\_run\_start\_fails](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md) | 2024.10 | 2026.09 | 1.9.0 | 1.12.0 | 2.0 | | [state\_modified\_compare\_more\_unrendered\_values](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md) | 2024.10 | 2026.09 | 1.9.0 | 1.12.0 | 2.0 | | [require\_yaml\_configuration\_for\_mf\_time\_spines](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_yaml_configuration_for_mf_time_spines.md) | 2024.10 | 2026.09 | 1.9.0 | 1.12.0 | 2.0 | | [require\_batched\_execution\_for\_custom\_microbatch\_strategy](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) | 2024.11 | 2026.09 | 1.9.0 | 1.12.0 | 2.0 | | [require\_nested\_cumulative\_type\_params](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md) | 2024.11 | 2026.09 | 1.9.0 | 1.12.0 | 2.0 | | [enable\_truthy\_nulls\_equals\_macro](https://docs.getdbt.com/reference/global-configs/behavior-flags/enable_truthy_nulls_equals_macro.md) | 2025.02 | - | 1.9.0 | - | - | | [validate\_macro\_args](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md) | 2025.03 | 2026.09 | 1.10.0 | 1.12.0 | - | | [require\_all\_warnings\_handled\_by\_warn\_error](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_all_warnings_handled_by_warn_error.md) | 2025.06 | 2026.09 | 1.10.0 | 1.12.0 | - | | [require\_generic\_test\_arguments\_property](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_generic_test_arguments_property.md) | 2025.07 | 2025.08 | 1.10.5 | 1.10.8 | - | | [require\_unique\_project\_resource\_names](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_unique_project_resource_names.md) | 2025.12 | - | 1.11.0 | - | - | | [require\_ref\_searches\_node\_package\_before\_root](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_ref_searches_node_package_before_root.md) | 2025.12 | - | 1.11.0 | - | - | | [require\_valid\_schema\_from\_generate\_schema\_name](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_valid_schema_from_generate_schema_name.md) | 2026.1 | - | 1.12.0a1 | - | - | | [require\_sql\_header\_in\_test\_configs](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_sql_header_in_test_configs.md) | 2026.3 | - | 1.12.0 | - | - | | [require\_corrected\_analysis\_fqns](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md) | 2026.3 | - | 1.12.0 | - | - | | [require\_source\_and\_semantic\_model\_names\_without\_spaces](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_source_and_semantic_model_names_without_spaces.md) | 2026.4 | - | 1.12.0 | - | - | | [allow\_jinja\_file\_extensions](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) | 2026.5 | - | 1.12.0 | - | - | | [latest\_version\_pointer\_enabled\_by\_default](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) | 2026.5 | - | 1.12.0 | - | - | ##### Introduced in dbt v2 The following flags are specific to dbt v2 and have no equivalent in dbt v1. They are configured the same way — in the `flags:` block of `dbt_project.yml`. | Flag | Adapter | Default | Introduced | Becomes default | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | ------------------ | --------------- | | [require\_resource\_names\_without\_plus\_prefix](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_plus_prefix.md) | All | `false` | 2.0.0-preview.208 | Not yet set | | [use\_catalogs\_v2](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md?version=2#new-spec-recommended) | All | `true` | 2.0.0-preview.174 | 2.0.0 | | bigquery\_noop\_alter\_relation\_comment | BigQuery | `false` | 2.0.0-preview.124 | Not yet set | ##### Adapter-specific behavior change flags This table outlines which version of the dbt adapter contains the behavior change's introduction (disabled by default) or maturity (enabled by default). Flags marked 2.0 are in the **Removed** column are removed in dbt v2 and the new behavior is always enabled. | Flag | dbt-ADAPTER: Intro | dbt-ADAPTER: Maturity | Removed | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | --------------------- | ------- | | [use\_info\_schema\_for\_columns](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-information-schema-for-columns) | Databricks 1.9.0 | - | 2.0 | | [use\_user\_folder\_for\_python](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-users-folder-for-python-model-notebooks) | Databricks 1.9.0 | - | 2.0 | | [use\_managed\_iceberg](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-managed-iceberg) | Databricks 1.11.0 | 1.12.0 | - | | [use\_materialization\_v2](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-restructured-materializations) | Databricks 1.10.0 | - | - | | [use\_replace\_on\_for\_insert\_overwrite](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-replace-on-for-insert_overwrite-strategy) | Databricks 1.11.0 | 1.11.0 | - | | [use\_describe\_as\_json\_for\_relation\_metadata](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-describe-as-json-for-relation-metadata) | Databricks 1.12.0 | - | - | | [redshift\_skip\_autocommit\_transaction\_statements](https://docs.getdbt.com/reference/global-configs/redshift-changes.md#redshift_skip_autocommit_transaction_statements-flag) | Redshift 1.12.0 | - | - | | [bigquery\_use\_batch\_source\_freshness](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#bigquery-use-batch-source-freshness) | BigQuery 1.11.0rc2 | - | - | | [bigquery\_reject\_wildcard\_metadata\_source\_freshness](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_reject_wildcard_metadata_source_freshness-flag) | BigQuery 1.12.0 | - | - | | [bigquery\_use\_standard\_sql\_for\_partitions](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_use_standard_sql_for_partitions-flag) | BigQuery 1.12.0 | 1.12.0 | - | | [snowflake\_default\_transient\_dynamic\_tables](https://docs.getdbt.com/reference/global-configs/snowflake-changes.md#the-snowflake_default_transient_dynamic_tables-flag) | Snowflake 1.12.0 | - | - | #### FAQs  How do I implement behavior change flags in my project? The following example displays the current flags and their current default values in the latest dbt and dbt v1 versions. To opt out of a specific behavior change, set the value of the flag to `false` in `dbt_project.yml`. You will continue to see warnings for legacy behaviors you've opted out of, until you either: * Resolve the issue (by switching the flag to `true`) * Silence the warnings using the `warn_error_options.silence` flag dbt\_project.yml ```yml flags: require_explicit_package_overrides_for_builtin_materializations: true require_resource_names_without_spaces: true source_freshness_run_project_hooks: true skip_nodes_if_on_run_start_fails: true state_modified_compare_more_unrendered_values: true require_yaml_configuration_for_mf_time_spines: true require_batched_execution_for_custom_microbatch_strategy: true require_nested_cumulative_type_params: true validate_macro_args: true require_all_warnings_handled_by_warn_error: true require_generic_test_arguments_property: true require_unique_project_resource_names: false require_ref_searches_node_package_before_root: false require_valid_schema_from_generate_schema_name: false enable_truthy_nulls_equals_macro: false require_sql_header_in_test_configs: false require_corrected_analysis_fqns: false require_source_and_semantic_model_names_without_spaces: false allow_jinja_file_extensions: false latest_version_pointer_enabled_by_default: false ```  What does it mean if there's no maturity date? When a maturity date has not yet been set (shown as -), we have not yet determined the exact date when the flag's default value will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either: * Migrate your project to support the new behavior, and then set the flag to true to stop seeing the warnings. * Explicitly set the flag to `false`. You will continue to see warnings, and you will retain the legacy behavior even after the maturity date (when the default value changes).  How do behavior change flags related to other changes? Since behavior change flags are different from other dbt changes, it's important to understand the difference: * [Deprecation warnings](https://docs.getdbt.com/reference/deprecations.md) — Features in your project code that will stop working (behavior flags often control when these become errors) * [Deprecated CLI flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) — Command-line flags being removed in v2 See the [Changes overview](https://docs.getdbt.com/reference/changes-overview.md) for a quick comparison. If you're upgrading to [v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md), a subset of behavior change flags are removed and their new behavior is always enabled. --- ### About builtins Jinja variable The `builtins` variable exists to provide references to builtin dbt context methods. This allows macros to be created with names that *mask* dbt builtin context methods, while still making those methods accessible in the dbt compilation context. The `builtins` variable is a dictionary containing the following keys: * [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) * [source](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) * [config](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md) #### Examples important Using the `builtins` variable in this way is an advanced development workflow. Users should be ready to maintain and update these overrides when upgrading in the future. ##### Overriding `ref` From dbt v1.5 and higher, use the following macro to override the `ref` method available in the model compilation context to return a [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) with the database name overridden to `dev`. It includes logic to extract user-provided arguments, including `version`, and call the `builtins.ref()` function with either a single `modelname` argument or both `packagename` and `modelname` arguments, based on the number of positional arguments in `varargs`. Note that the `ref`, `source`, and `config` functions can't be overridden with a package. This is because `ref`, `source`, and `config` are context properties within dbt and are not dispatched as global macros. Refer to [this GitHub discussion](https://github.com/dbt-labs/dbt/issues/4491#issuecomment-994709916) for more context. <br /> ```text {% macro ref() %} -- extract user-provided positional and keyword arguments {% set version = kwargs.get('version') or kwargs.get('v') %} {% set packagename = none %} {%- if (varargs | length) == 1 -%} {% set modelname = varargs[0] %} {%- else -%} {% set packagename = varargs[0] %} {% set modelname = varargs[1] %} {% endif %} -- call builtins.ref based on provided positional arguments {% set rel = None %} {% if packagename is not none %} {% set rel = builtins.ref(packagename, modelname, version=version) %} {% else %} {% set rel = builtins.ref(modelname, version=version) %} {% endif %} -- finally, override the database name with "dev" {% set newrel = rel.replace_path(database="dev") %} {% do return(newrel) %} {% endmacro %} ``` ##### Render identifiers without a database Logic within the ref macro can also be used to control which elements of the model path are rendered when run, for example the following logic renders only the schema and object identifier, but not the database reference i.e. `my_schema.my_model` rather than `my_database.my_schema.my_model`. This is especially useful when using snowflake as a warehouse, if you intend to change the name of the database post-build and wish the references to remain accurate. ```text -- render identifiers without a database {% do return(rel.include(database=false)) %} ``` --- ### About config property ##### Models models/\<filename>.yml ```yml models: - name: <model_name> config: <model_config>: <config_value> ... ``` ##### Seeds seeds/\<filename>.yml ```yml seeds: - name: <seed_name> config: <seed_config>: <config_value> ... ``` ##### Snapshots snapshots/\<filename>.yml ```yml snapshots: - name: <snapshot_name> config: <snapshot_config>: <config_value> ... ``` ##### Tests \<resource\_path>/\<filename>.yml ```yml <resource_type>: - name: <resource_name> data_tests: - <test_name>: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: <test_config>: <config-value> ... columns: - name: <column_name> data_tests: - <test_name> - <test_name>: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: <test_config>: <config-value> ... ``` ##### Unit tests Did you know... Available from dbt v1.8 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). (Applies to dbt v2.0 and later) models/\<filename>.yml ```yml unit_tests: - name: <test-name> config: compute: local | remote enabled: true | false meta: {dictionary} static_analysis: strict | baseline | off tags: <string> ``` ##### Sources models/\<filename>.yml ```yml sources: - name: <source_name> config: <source_config>: <config_value> tables: - name: <table_name> config: <source_config>: <config_value> ``` ##### Metrics models/\<filename>.yml ```yml metrics: - name: <metric_name> config: enabled: true | false group: <string> meta: {dictionary} ``` ##### Exposures models/\<filename>.yml ```yml exposures: - name: <exposure_name> config: enabled: true | false meta: {dictionary} ``` ##### Semantic models (Applies to dbt v1.12 and later) models/\<filename>.yml ```yml models: - name: model_name semantic_model: enabled: true | false group: <string> config: meta: {dictionary} ``` ##### Saved queries models/\<filename>.yml ```yml saved-queries: - name: <saved_query_name> config: cache: enabled: true | false enabled: true | false group: <string> meta: {dictionary} schema: <string> exports: - name: <export_name> config: export_as: view | table alias: <string> schema: <string> ``` #### Definition The `config` property allows you to configure resources at the same time you're defining properties in YAML files. --- ### About config variable The `config` variable exists to handle end-user configuration for custom materializations. Configs like `unique_key` can be implemented using the `config` variable in your own materializations. For example, code in the `incremental` materialization like this: ```text {% materialization incremental, default -%} {%- set unique_key = config.get('unique_key') -%} ... ``` is responsible for handling model code that looks like this: ```text {{ config( materialized='incremental', unique_key='id' ) }} ``` Review [Model configurations](https://docs.getdbt.com/reference/model-configs.md) for examples and more information on valid arguments. #### config.get **Args**: * `name`: The name of the configuration variable (required) * `default`: The default value to use if this configuration is not provided (optional) The `config.get` function is used to get configurations for a model from the end-user. Configs defined in this way are optional, and a default value can be provided. There are 3 cases: 1. The configuration variable exists, it is not `None` 2. The configuration variable exists, it is `None` 3. The configuration variable does not exist Accessing custom configurations in meta `config.get()` behaves differently depending on how you access configurations under `meta`: * **Accessing a specific key under `meta`**: `config.get('my_key')` does not return values from `config.meta`. If the key exists only in `meta`, `config.get()` returns the default value and emits a warning. This applies to both dbt v1 and dbt v2. * **Accessing the `meta` dictionary**: `config.get('meta', default={})` returns the full `meta` dictionary in dbt v2, but does not work in dbt v1. To access custom configurations under `meta` consistently across both engines, use [`config.meta_get()`](#configmeta_get). Example usage: ```sql {% materialization incremental, default -%} -- Example w/ no default. unique_key will be None if the user does not provide this configuration {%- set unique_key = config.get('unique_key') -%} -- Example w/ alternate value. Use alternative of 'id' if 'unique_key' config is provided, but it is None {%- set unique_key = config.get('unique_key') or 'id' -%} -- Example w/ default value. Default to 'id' if the 'unique_key' config does not exist {%- set unique_key = config.get('unique_key', default='id') -%} -- For custom configs under `meta`, use config.meta_get() {% set my_custom_config = config.meta_get('custom_config_key') %} ... ``` #### config.require **Args**: * `name`: The name of the configuration variable (required) The `config.require` function is used to get configurations for a model from the end-user. Configs defined using this function are required, and failure to provide them will result in a compilation error. Accessing custom configurations in meta `config.require()` doesn't return values from `config.meta`. If a key exists only in `meta`, `config.require()` raises an error and emits a warning. To access required custom configurations stored under `meta`, use [`config.meta_require()`](#configmeta_require). Example usage: ```sql {% materialization incremental, default -%} {%- set unique_key = config.require('unique_key') -%} ... ``` #### config.meta\_get This functionality is available in dbt v1.10 and up. **Args**: * `name`: The name of the configuration variable to retrieve from `meta` (required) * `default`: The default value to use if this configuration is not provided (optional) The `config.meta_get` function retrieves custom configurations stored under the `meta` dictionary. Unlike `config.get()`, this function exclusively checks `config.meta` and won't result in a deprecation warning. Use this function when accessing custom configurations that you've defined under `meta` in your model or resource configuration - it's equivalent to writing `config.get('meta').get()`. Note that `config.meta_get` is not yet supported in Python models. In the meantime, Python models should continue using `dbt.config.get("meta").get("<key>")` to access custom meta configurations. `dbt.config.get_meta("<key>")` is an alias for `dbt.config.get("meta").get("<key>")`. Example usage: ```sql {% materialization custom_materialization, default -%} -- Retrieve a custom config from meta, returns None if not found {%- set custom_setting = config.meta_get('custom_setting') -%} -- Retrieve with a default value {%- set custom_setting = config.meta_get('custom_setting', default='default_value') -%} ... ``` Example model configuration: ```yaml models: - name: my_model config: meta: custom_setting: "my_value" ``` #### config.meta\_require This functionality is available in dbt v1.10 and up. **Args**: * `name`: The name of the configuration variable to retrieve from `meta` (required) The `config.meta_require` function retrieves custom configurations stored under the `meta` dictionary. Unlike `config.require()`, this function exclusively checks `config.meta` and won't result in deprecation warnings. If the configuration is not found, dbt raises a compilation error. Use this function when you need to ensure a custom configuration exists under `meta`. Note that `config.meta_require` is not yet supported in Python models. Example usage: ```sql {% materialization custom_materialization, default -%} -- Require a custom config from meta, throws error if not found {%- set required_setting = config.meta_require('required_setting') -%} ... ``` Example model configuration: ```yaml models: - name: my_model config: meta: required_setting: "my_value" ``` --- ### About data tests property #### Description The `data_tests` property defines assertions about a column, table, or view. The property contains a list of [generic data tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests), referenced by name, which can include the four built-in generic tests available in dbt. For example, you can add data tests that ensure a column contains no duplicates and zero null values. Any arguments or [configurations](https://docs.getdbt.com/reference/data-test-configs.md) passed to those data tests should be nested below the `arguments` property. Once these data tests are defined, you can validate their correctness by running `dbt test`. To help you get started, the examples below show how to define the `data_tests` property on different resource types (models, sources, seeds, snapshots, and analyses). ##### Models models/\<filename>.yml ```yml models: - name: <model_name> data_tests: - <test_name>: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: <test_config>: <config-value> columns: - name: <column_name> data_tests: - <test_name> - <test_name>: arguments: <argument_name>: <argument_value> config: <test_config>: <config-value> ``` ##### Sources models/\<filename>.yml ```yml sources: - name: <source_name> tables: - name: <table_name> data_tests: - <test_name> - <test_name>: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: <test_config>: <config-value> columns: - name: <column_name> data_tests: - <test_name> - <test_name>: arguments: <argument_name>: <argument_value> config: <test_config>: <config-value> ``` ##### Seeds seeds/\<filename>.yml ```yml seeds: - name: <seed_name> data_tests: - <test_name> - <test_name>: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: <test_config>: <config-value> columns: - name: <column_name> data_tests: - <test_name> - <test_name>: arguments: <argument_name>: <argument_value> config: <test_config>: <config-value> ``` ##### Snapshots snapshots/\<filename>.yml ```yml snapshots: - name: <snapshot_name> data_tests: - <test_name> - <test_name>: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: <test_config>: <config-value> columns: - name: <column_name> data_tests: - <test_name> - <test_name>: arguments: <argument_name>: <argument_value> config: <test_config>: <config-value> ``` ##### Analyses This feature is not implemented for analyses. #### Out-of-the-box data tests There are four generic data tests that are available out of the box, for everyone using dbt. ##### `not_null` This data test validates that there are no `null` values present in a column. models/\<filename>.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - not_null ... ``` You can add an arguments block for test inputs and a config block for options like `severity` or `where`. Refer to [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md?version=2.0#data-test-specific-configurations) for the full list. If you see a deprecation warning about test arguments, refer to [Deprecations](https://docs.getdbt.com/reference/deprecations.md?version=2.0) for test-related warnings. ##### `unique` This data test validates that there are no duplicate values present in a field. The config and where clause are optional. models/\<filename>.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - unique: config: where: "order_id > 21" ``` ##### `accepted_values` This data test validates that all of the non-`null` values in a column are present in a supplied list of `values`. If any values other than those provided in the list are present, then the data test will fail. The `accepted_values` test supports an optional `quote` parameter which, by default, will single-quote the list of accepted values in the test query. To test non-strings (like integers or boolean values) explicitly set the `quote` config to `false`. schema.yml ```yaml models: - name: orders columns: - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'returned'] - name: status_id data_tests: - accepted_values: arguments: values: [1, 2, 3, 4] quote: false ``` ##### `relationships` This data test validates that all of the records in a child table have a corresponding record in a parent table. This property is referred to as "referential integrity". This test automatically excludes `NULL` values from validation, consistent with how database foreign key constraints work. Use the `not_null` test separately if `NULL` values should cause failures. The following example tests that every order's `customer_id` maps back to a valid `customer`. schema.yml ```yaml models: - name: orders columns: - name: customer_id data_tests: - relationships: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. to: ref('customers') field: id ``` The `to` argument accepts a [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) – this means you can pass it a `ref` to a model (e.g. `ref('customers')`), or a `source` (e.g. `source('jaffle_shop', 'customers')`). #### Additional examples ##### Test an expression Some data tests require multiple columns, so it doesn't make sense to nest them under the `columns:` key. In this case, you can apply the data test to the model (or source, seed, or snapshot) instead: models/orders.yml ```yaml models: - name: orders description: Order overview data mart, offering key details for each order including if it's a customer's first order and a food vs. drink item breakdown. One row per order. data_tests: - dbt_utils.expression_is_true: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. expression: "order_items_subtotal = subtotal" - dbt_utils.expression_is_true: arguments: expression: "order_total = subtotal + tax_paid" ``` This example focuses on testing expressions to ensure that `order_items_subtotal` equals `subtotal` and `order_total` correctly sums `subtotal` and `tax_paid`. ##### Use custom generic data test If you've defined your own custom generic data test, you can use that as the `test_name`: models/\<filename>.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - primary_key # name of my custom generic test ``` Check out the guide on writing a [custom generic data test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) for more information. ##### Custom data test name By default, dbt will synthesize a name for your generic data test by concatenating: * test name (`not_null`, `unique`, etc) * model name (or source/seed/snapshot) * column name (if relevant) * arguments (if relevant, e.g. `values` for `accepted_values`) It does not include any configurations for the data test. If the concatenated name is too long, dbt will use a truncated and hashed version instead. The goal is to preserve unique identifiers for all resources in your project, including tests. You may also define your own name for a specific data test, via the `name` property. **When might you want this?** dbt's default approach can result in some wonky (and ugly) data test names. By defining a custom name, you get full control over how the data test will appear in log messages and metadata artifacts. You'll also be able to select the data test by that name. models/\<filename>.yml ```yaml models: - name: orders columns: - name: status data_tests: - accepted_values: name: unexpected_order_status_today arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'returned'] config: where: "order_date = current_date" ``` ```sh $ dbt test --select unexpected_order_status_today 12:43:41 Running with dbt=1.1.0 12:43:41 Found 1 model, 1 test, 0 snapshots, 0 analyses, 167 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics 12:43:41 12:43:41 Concurrency: 5 threads (target='dev') 12:43:41 12:43:41 1 of 1 START test unexpected_order_status_today ................................ [RUN] 12:43:41 1 of 1 PASS unexpected_order_status_today ...................................... [PASS in 0.03s] 12:43:41 12:43:41 Finished running 1 test in 0.13s. 12:43:41 12:43:41 Completed successfully 12:43:41 12:43:41 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1 ``` A data test's name must be unique for all tests defined on a given model-column combination. If you give the same name to data tests defined on several different columns, or across several different models, then `dbt test --select <repeated_custom_name>` will select them all. **When might you need this?** In cases where you have defined the same data test twice, with only a difference in configuration, dbt will consider these data tests to be duplicates: models/\<filename>.yml ```yaml models: - name: orders columns: - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'returned'] config: where: "order_date = current_date" - accepted_values: arguments: values: ['placed', 'shipped', 'completed', 'returned'] config: # only difference is in the 'where' config where: "order_date = (current_date - interval '1 day')" # PostgreSQL syntax ``` ```sh Compilation Error dbt found two tests with the name "accepted_values_orders_status__placed__shipped__completed__returned" defined on column "status" in "models.orders". Since these resources have the same name, dbt will be unable to find the correct resource when running tests. To fix this, change the name of one of these resources: - test.testy.accepted_values_orders_status__placed__shipped__completed__returned.69dce9e5d5 (models/one_file.yml) - test.testy.accepted_values_orders_status__placed__shipped__completed__returned.69dce9e5d5 (models/one_file.yml) ``` By providing a custom name, you help dbt differentiate data tests: models/\<filename>.yml ```yaml models: - name: orders columns: - name: status data_tests: - accepted_values: name: unexpected_order_status_today arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'returned'] config: where: "order_date = current_date" - accepted_values: name: unexpected_order_status_yesterday arguments: values: ['placed', 'shipped', 'completed', 'returned'] config: where: "order_date = (current_date - interval '1 day')" # PostgreSQL ``` ```sh $ dbt test 12:48:03 Running with dbt=1.1.0-b1 12:48:04 Found 1 model, 2 tests, 0 snapshots, 0 analyses, 167 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics 12:48:04 12:48:04 Concurrency: 5 threads (target='dev') 12:48:04 12:48:04 1 of 2 START test unexpected_order_status_today ................................ [RUN] 12:48:04 2 of 2 START test unexpected_order_status_yesterday ............................ [RUN] 12:48:04 1 of 2 PASS unexpected_order_status_today ...................................... [PASS in 0.04s] 12:48:04 2 of 2 PASS unexpected_order_status_yesterday .................................. [PASS in 0.04s] 12:48:04 12:48:04 Finished running 2 tests in 0.21s. 12:48:04 12:48:04 Completed successfully 12:48:04 12:48:04 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2 ``` **If using [`store_failures`](https://docs.getdbt.com/reference/resource-configs/store_failures.md):** dbt uses each data test's name as the name of the table in which to store any failing records. If you have defined a custom name for one data test, that custom name will also be used for its table of failures. You may optionally configure an [`alias`](https://docs.getdbt.com/reference/resource-configs/alias.md) for the data test, to separately control both the name of the data test (for metadata) and the name of its database table (for storing failures). ##### Alternative format for defining tests When defining a generic data test with several arguments and configurations, the YAML can look and feel unwieldy. If you find it easier, you can define the same data test properties as top-level keys of a single dictionary, by providing the data test name as `test_name` instead. It's totally up to you. This example is identical to the one above: models/\<filename>.yml ```yaml models: - name: orders columns: - name: status data_tests: - name: unexpected_order_status_today test_name: accepted_values # name of the generic test to apply arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: - placed - shipped - completed - returned config: where: "order_date = current_date" ``` #### Related documentation * [Data testing guide](https://docs.getdbt.com/docs/build/data-tests.md) --- ### About dbt --version The `--version` command-line flag returns information about the currently installed version of dbt v1, the dbt platform CLI, or dbt v2. This flag is not supported when invoking dbt in other dbt runtimes (for example, the IDE or scheduled runs). * **dbt v1** — Returns the installed version of dbt v1 and the versions of all installed adapters. * **dbt platform CLI** — Returns the installed version of the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) and, for the other `dbt_version` values, the *latest* version of the dbt runtime in dbt. * **dbt v2** — Returns the installed dbt v2 version. Because the CLI and language server ship in a single binary, they always report the same version. Refer to [Version compatibility](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) for how this maps to the dbt VS Code extension. #### Versioning To learn more about release versioning for dbt v1, refer to [How dbt v1 uses semantic versioning](https://docs.getdbt.com/docs/dbt-versions.md#how-dbt-v1-uses-semantic-versioning). If using a [dbt release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), which provide ongoing updates to dbt, then `dbt_version` represents the release version of dbt in dbt. This also follows semantic versioning guidelines, using the `YYYY.M.D+<suffix>` format. The year, month, and day represent the date the version was built (for example, `2024.10.8+996c6a8`). The suffix provides an additional unique identification for each build. #### Example usages dbt v1 example: dbt v1 ```text $ dbt --version Core: - installed: 1.7.6 - latest: 1.7.6 - Up to date! Plugins: - snowflake: 1.7.1 - Up to date! ``` dbt CLI example: dbt platform CLI ```text $ dbt --version Cloud CLI - 0.35.7 (fae78a6f5f6f2d7dff3cab3305fe7f99bd2a36f3 2024-01-18T22:34:52Z) ``` dbt v2 example: ```shell $ dbt --version dbt 2.0.1 ``` For a machine-readable version, add the `--format json` flag. This is useful when filing a bug report or when tooling needs to parse the installed version: ```shell dbt --version --format json ``` ```json { "version": "2.0.1" } ``` --- ### About dbt artifacts (Applies to dbt v1.12 and later) With every invocation, dbt generates and saves one or more *artifacts*. Several of these are JSON files (`semantic_manifest.json`, `osi_document.json`, `manifest.json`, `catalog.json`, `run_results.json`, and `sources.json`) that are used to power: * [documentation](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) * [state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection) * [visualizing source freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) They could also be used to: * gain insights into your [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) * calculate project-level test coverage * perform longitudinal analysis of run timing * identify historical changes in table structure * do much, much more ##### When are artifacts produced? [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Most dbt commands (and corresponding RPC methods) produce artifacts: * [semantic manifest](https://docs.getdbt.com/reference/artifacts/sl-manifest.md): produced whenever your dbt project is parsed (Applies to dbt v1.12 and later) * [Apache Ossie document](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document): produced whenever your dbt project is parsed - [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md): produced by commands that read and understand your project - [run results](https://docs.getdbt.com/reference/artifacts/run-results-json.md): produced by commands that run, compile, or catalog nodes in your DAG - [catalog](https://docs.getdbt.com/reference/artifacts/catalog-json.md): produced by `docs generate` (Applies to dbt v2.0 and later) * [sources](https://docs.getdbt.com/reference/artifacts/sources-json.md): produced by `dbt freshness` when sources are included; retained for backward compatibility * [freshness](https://docs.getdbt.com/reference/artifacts/freshness-json.md): produced by [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md); covers both sources and models with freshness configured, with a `resource_type` field on each entry to distinguish them When running commands from the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), all artifacts are downloaded by default. If you want to change this behavior, refer to [How to skip artifacts from being downloaded](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md#how-to-skip-artifacts-from-being-downloaded). #### Where are artifacts produced? By default, artifacts are written to the `/target` directory of your dbt project. You can configure the location using the [`target-path` flag](https://docs.getdbt.com/reference/global-configs/json-artifacts.md). #### Common metadata All artifacts produced by dbt include a `metadata` dictionary with these properties: * `dbt_version`: Version of dbt that produced this artifact. For details about release versioning, refer to [Versioning](https://docs.getdbt.com/reference/commands/version.md#versioning). * `dbt_schema_version`: URL of this artifact's schema. See notes below. * `generated_at`: Timestamp in UTC when this artifact was produced. * `adapter_type`: The adapter (database), e.g. `postgres`, `spark`, etc. * `env`: Any environment variables prefixed with `DBT_ENV_CUSTOM_ENV_` will be included in a dictionary, with the prefix-stripped variable name as its key. * [`invocation_id`](https://docs.getdbt.com/reference/dbt-jinja-functions/invocation_id.md): Unique identifier for this dbt invocation In the manifest, the `metadata` may also include: * `send_anonymous_usage_stats`: Whether this invocation sent [anonymous usage statistics](https://docs.getdbt.com/reference/global-configs/usage-stats.md) while executing. * `project_name`: The `name` defined in the root project's `dbt_project.yml`. (Added in manifest v10 / dbt v1.6) * `project_id`: Project identifier, hashed from `project_name`, sent with anonymous usage stats if enabled. * `user_id`: User identifier, stored by default in `~/dbt/.user.yml`, sent with anonymous usage stats if enabled. ###### Notes: * The structure of dbt artifacts is canonized by [JSON schemas](https://json-schema.org/), which are hosted at [schemas.getdbt.com](https://schemas.getdbt.com/). * Artifact versions may change in any minor version of dbt (`v1.x.0`). Each artifact is versioned independently. #### Related docs * [Other artifacts](https://docs.getdbt.com/reference/artifacts/other-artifacts.md) files such as `index.html` or `graph_summary.json`. --- ### About dbt build command The `dbt build` command will: * run [models](https://docs.getdbt.com/docs/build/models.md) * test [tests](https://docs.getdbt.com/docs/build/data-tests.md) * snapshot [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) * seed [seeds](https://docs.getdbt.com/docs/build/seeds.md) * build [user-defined functions](https://docs.getdbt.com/docs/build/udfs.md) (available from dbt v1.11 and up) In DAG order, for selected resources or an entire project. #### Details **Artifacts:** The `build` task will write a single [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) and a single [run results artifact](https://docs.getdbt.com/reference/artifacts/run-results-json.md). The run results will include information about all models, tests, seeds, and snapshots that were selected to build, combined into one file. (Applies to dbt v2.0 and later) **dbt Information Schema: **Use `--generate-info-schema` to write the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) to `target/info_schema/` in a versioned subdirectory (currently `v1/`). The Information Schema exposes your project's metadata as queryable SQL tables (similar to a database's `INFORMATION_SCHEMA`) so you can query models, sources, and more without parsing `manifest.json`. ```shell dbt build --generate-info-schema ``` To populate inferred model column types and column-level lineage, combine with [`--static-analysis strict`](https://docs.getdbt.com/docs/build/about-static-analysis.md). Without it, `dbt.node_columns` contains only YAML-declared columns and `dbt.column_lineage` contains no lineage. ```shell dbt build --generate-info-schema --static-analysis strict ``` **Skipping on failures:** Tests on upstream resources will block downstream resources from running, and a test failure will cause those downstream resources to skip entirely. E.g. If `model_b` depends on `model_a`, and a `unique` test on `model_a` fails, then `model_b` will `SKIP`. * Don't want a test to cause skipping? Adjust its [severity or thresholds](https://docs.getdbt.com/reference/resource-configs/severity.md) to `warn` instead of `error` * In the case of a test with multiple parents, where one parent depends on the other (e.g. a `relationships` test between `model_a` + `model_b`), that test will block-and-skip children of the most-downstream parent only (`model_b`). * If you have a test with multiple parents that are independent of each other, dbt [skips](https://github.com/dbt-labs/dbt/blob/d5071fa13502be273596a0b7c8b13d14b6c68655/core/dbt/compilation.py#L224-L257) the downstream node only if that node depends on all of those parents. (Applies to dbt v2.0 and later) **Checks:** `dbt build` runs [checks](https://docs.getdbt.com/docs/build/checks.md) before it compiles or runs any models. Checks are SQL queries you write against the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) to enforce your team's project standards. For example, a check might be that every model has a description. A check passes when its query returns no rows. A failing check stops the build before anything is materialized, unless the check's `severity` is set to `warn`. (Applies to dbt v1.12 and later) **Skipping on model errors:** By default, if a model fails, all downstream models are skipped. Set [`on_error: continue`](https://docs.getdbt.com/reference/resource-configs/on_error.md) on a model to allow its downstream models to run even when that model fails. (Applies to dbt v1.12 and later) **Selecting resources:** The `build` task supports standard selection syntax (`--select`, `--exclude`), as well as a `--resource-type` flag that offers a final filter (just like `list`). Whichever resources are selected, those are the ones that `build` will run/test/snapshot/seed. * Remember that tests support indirect selection, so `dbt build -s model_a` will both run *and* test `model_a`. What does that mean? Any tests that directly depend on `model_a` will be included, so long as those tests don't also depend on other unselected parents. See [test selection](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) for details and examples. **Flags:** The `build` task supports all the same flags as `run`, `test`, `snapshot`, and `seed`. For flags that are shared between multiple tasks (e.g. `--full-refresh`), `build` will use the same value for all selected resource types that support it (e.g. both models and seeds will be full refreshed). Snapshots ignore full refresh Snapshots ignore both the `full_refresh` config and the `--full-refresh` flag. A command such as `dbt build --full-refresh` or `dbt snapshot --full-refresh` that includes a snapshot node runs the snapshot as normal — it won't drop or recreate the snapshot table, so existing snapshot history is preserved. (Applies to dbt v2.0 and later) ##### The `--skip-checks` flag The `build` command supports `--skip-checks` to bypass the [checks](https://docs.getdbt.com/docs/build/checks.md) gate. ```shell dbt build --skip-checks ``` To disable a single check rather than the entire gate, set `enabled: false` on that check's config: ```yaml checks: - name: all_models_have_descriptions config: enabled: false ``` ##### The `--empty` flag The `build` command supports the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. ###### The render method The `.render()` method is generally used to resolve or evaluate Jinja expressions (such as `{{ source(...) }}`) during runtime. When using the `--empty flag`, dbt may skip processing `ref()` or `source()` for optimization. To avoid compilation errors and to explicitly tell dbt to process a specific relation (`ref()` or `source()`), use the `.render()` method in your model file. For example: models.sql ```jinja {{ config( pre_hook = [ "alter external table {{ source('sys', 'customers').render() }} refresh" ] ) }} select ... ``` #### Tests When `dbt build` is executed with unit tests applied, the models will be processed according to their lineage and dependencies. The tests will be executed as follows: * [Unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) are run on a SQL model. * The model is materialized. * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) are run on the model. This saves on warehouse spend as the model will only be materialized if the unit tests pass successfully. Unit tests and data tests can be selected using `--select test_type:unit` or `--select test_type:data` for `dbt build` (same for the `--exclude` flag). ##### Examples ```text $ dbt build Running with dbt=1.9.0-b2 Found 1 model, 4 tests, 1 snapshot, 1 analysis, 341 macros, 0 operations, 1 seed file, 2 sources, 2 exposures 18:49:43 | Concurrency: 1 threads (target='dev') 18:49:43 | 18:49:43 | 1 of 7 START seed file dbt_jcohen.my_seed............................ [RUN] 18:49:43 | 1 of 7 OK loaded seed file dbt_jcohen.my_seed........................ [INSERT 2 in 0.09s] 18:49:43 | 2 of 7 START view model dbt_jcohen.my_model.......................... [RUN] 18:49:43 | 2 of 7 OK created view model dbt_jcohen.my_model..................... [CREATE VIEW in 0.12s] 18:49:43 | 3 of 7 START test not_null_my_seed_id................................ [RUN] 18:49:43 | 3 of 7 PASS not_null_my_seed_id...................................... [PASS in 0.05s] 18:49:43 | 4 of 7 START test unique_my_seed_id.................................. [RUN] 18:49:43 | 4 of 7 PASS unique_my_seed_id........................................ [PASS in 0.03s] 18:49:43 | 5 of 7 START snapshot snapshots.my_snapshot.......................... [RUN] 18:49:43 | 5 of 7 OK snapshotted snapshots.my_snapshot.......................... [INSERT 0 5 in 0.27s] 18:49:43 | 6 of 7 START test not_null_my_model_id............................... [RUN] 18:49:43 | 6 of 7 PASS not_null_my_model_id..................................... [PASS in 0.03s] 18:49:43 | 7 of 7 START test unique_my_model_id................................. [RUN] 18:49:43 | 7 of 7 PASS unique_my_model_id....................................... [PASS in 0.02s] 18:49:43 | 18:49:43 | Finished running 1 seed, 1 view model, 4 tests, 1 snapshot in 1.01s. Completed successfully Done. PASS=7 WARN=0 ERROR=0 SKIP=0 TOTAL=7 ``` #### Functions *Available from dbt v1.11 and up* The `build` command builds [user-defined functions](https://docs.getdbt.com/docs/build/udfs.md) as part of the DAG execution. To build or rebuild only `functions` in your project, run `dbt build --select "resource_type:function"`. For example: ```bash dbt build --select "resource_type:function" dbt-fusion 2.0.1 Succeeded [ 0.98s] function dbt_schema.whoami (function) Succeeded [ 1.12s] function dbt_schema.area_of_circle (function) ``` --- ### About dbt check command Available in v2 `dbt check` parses your project, runs [checks](https://docs.getdbt.com/docs/build/checks.md), and reports results. #### Usage ```shell dbt check [<check-name> …] [flags] ``` Run all enabled checks: ```shell dbt check ``` Run one or more checks: ```shell dbt check all_models_have_descriptions dbt check all_models_have_descriptions public_models_have_owners ``` Passing an unknown check name fails the command. Passing a disabled check name is accepted and the check is skipped. #### Related docs * [Checks](https://docs.getdbt.com/docs/build/checks.md) * [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) * [check-paths project config](https://docs.getdbt.com/reference/project-configs/check-paths.md) --- ### About dbt clean command `dbt clean` is a utility function that deletes the paths specified within the [`clean-targets`](https://docs.getdbt.com/reference/project-configs/clean-targets.md) list in the `dbt_project.yml` file. It helps by removing unnecessary files or directories generated during the execution of other dbt commands, ensuring a clean state for the project. #### Example usage ```text dbt clean ``` #### Supported flags This section will briefly explain the following flags: * [`--clean-project-files-only`](#--clean-project-files-only) (default) * [`--no-clean-project-files-only`](#--no-clean-project-files-only) To view the list of all supported flags for the `dbt clean` command in the terminal, use the `--help` flag, which will display detailed information about the available flags you can use, including its description and usage: ```shell dbt clean --help ``` ##### --clean-project-files-only By default, dbt deletes all the paths within the project directory specified in `clean-targets`. note Avoid using paths outside the dbt project; otherwise, you will see an error. ###### Example usage ```shell dbt clean --clean-project-files-only ``` ##### --no-clean-project-files-only Deletes all the paths specified in the `clean-targets` list of `dbt_project.yml`, including those outside the current dbt project. ```shell dbt clean --no-clean-project-files-only ``` (Applies to dbt v2.0 and later) #### Removing agent skills If you set the [`ai_provider`](https://docs.getdbt.com/docs/dbt-ai/package-skills.md#set-the-ai_provider-flag) flag, `dbt clean` also removes the [agent skills](https://docs.getdbt.com/docs/dbt-ai/package-skills.md) that dbt installed: ```text Removing add-a-data-test -> .claude/skills Removing naming-conventions -> .claude/skills ``` dbt removes only the skills it installed, which it recognizes by the record it writes into each installed `SKILL.md`. Skills you wrote by hand stay in place, even when they sit in the same directory. #### dbt clean with remote file system To avoid complex permissions issues and potentially deleting crucial aspects of the remote file system without access to fix them, this command does not work when interfacing with the RPC server that powers the Studio IDE. Instead, when working in dbt, the `dbt deps` command cleans before it installs packages automatically. The `target` folder can be manually deleted from the sidebar file tree if needed. --- ### About dbt clone command The `dbt clone` command clones selected nodes from the [specified state](https://docs.getdbt.com/reference/node-selection/syntax.md#establishing-state) to the target schema(s). This command makes use of the `clone` materialization: * If your data platform supports zero-copy cloning of tables (Snowflake, Databricks, or BigQuery), and this model exists as a table in the source environment, dbt will create it in your target environment as a clone. * Otherwise, dbt will create a simple pointer view (`select * from` the source object) * By default, `dbt clone` will not recreate pre-existing relations in the current target. To override this, use the `--full-refresh` flag. * You may want to specify a higher number of [threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md) to decrease execution time since individual clone statements are independent of one another. The `clone` command is useful for: * blue/green continuous deployment (on data warehouses that support zero-copy cloning tables) * cloning current production state into development schema(s) * handling incremental models in dbt CI jobs (on data warehouses that support zero-copy cloning tables) * testing code changes on downstream dependencies in your BI tool ```bash # clone all of my models from specified state to my target schema(s) dbt clone --state path/to/artifacts # clone one_specific_model of my models from specified state to my target schema(s) dbt clone --select "one_specific_model" --state path/to/artifacts # clone all of my models from specified state to my target schema(s) and recreate all pre-existing relations in the current target dbt clone --state path/to/artifacts --full-refresh # clone all of my models from specified state to my target schema(s), running up to 50 clone statements in parallel dbt clone --state path/to/artifacts --threads 50 ``` ##### When to use `dbt clone` instead of [deferral](https://docs.getdbt.com/reference/node-selection/defer.md)? Unlike deferral, `dbt clone` requires some compute and creation of additional objects in your data warehouse. In many cases, deferral is a cheaper and simpler alternative to `dbt clone`. However, `dbt clone` covers additional use cases where deferral may not be possible. For example, by creating actual data warehouse objects, `dbt clone` allows you to test out your code changes on downstream dependencies *outside of dbt* (such as a BI tool). As another example, you could `clone` your modified incremental models as the first step of your dbt CI job to prevent costly `full-refresh` builds for warehouses that support zero-copy cloning. #### Cloning in dbt You can clone nodes between states in dbt using the `dbt clone` command. This is available in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) and the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) and relies on the [`--defer`](https://docs.getdbt.com/reference/node-selection/defer.md) feature. For more details on defer in dbt, read [Using defer in dbt](https://docs.getdbt.com/docs/platform/about-defer.md). * **Using dbt platform CLI** — The `dbt clone` command in the dbt platform CLI automatically includes the `--defer` flag. This means you can use the `dbt clone` command without any additional setup. * **Using Studio IDE** — To use the `dbt clone` command in the Studio IDE, follow these steps before running the `dbt clone` command: * Set up your **Production environment** and have a successful job run. * Enable **Defer to production** by toggling the switch in the lower-right corner of the command bar. [![Select the 'Defer to production' toggle on the bottom right of the command bar to enable defer in the Studio IDE.](/img/docs/dbt-platform/defer-toggle.png?v=2 "Select the 'Defer to production' toggle on the bottom right of the command bar to enable defer in the Studio IDE.")](#)Select the 'Defer to production' toggle on the bottom right of the command bar to enable defer in the Studio IDE. * Run the `dbt clone` command from the command bar. Check out [this Developer blog post](https://docs.getdbt.com/blog/to-defer-or-to-clone) for more details on best practices when to use `dbt clone` vs. deferral. --- ### About dbt compile command `dbt compile` generates executable SQL from source files for: (Applies to dbt v1.12 and later) * [Models](https://docs.getdbt.com/docs/build/models.md) * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) * [Analyses](https://docs.getdbt.com/docs/build/analyses.md) * [Functions](https://docs.getdbt.com/docs/build/udfs.md) * [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) (available in dbt v1.12) You can find these compiled SQL files in the `target/` directory of your dbt project. The `compile` command is useful for: * Visually inspecting the compiled output of resource files. This is useful for validating complex Jinja logic or macro usage. * Manually running compiled SQL. While debugging a model or [data test](https://docs.getdbt.com/docs/build/data-tests.md), it's often useful to execute the underlying `select` statement to find the source of the bug. * Compiling `analysis` files. Read more about analysis files [here](https://docs.getdbt.com/docs/build/analyses.md). Some common misconceptions: * `dbt compile` is *not* a pre-requisite of `dbt run`, or other building commands. Those commands will handle compilation themselves. * If you just want dbt to read and validate your project code, without connecting to the data warehouse, use `dbt parse` instead. #### Interactive compile Starting in dbt v1.5, `compile` can be "interactive" in the CLI, by displaying the compiled code of a node or arbitrary dbt-SQL query: * `--select` a specific node *by name* * `--inline` an arbitrary dbt-SQL query This will log the compiled SQL to the terminal, in addition to writing to the `target/` directory. For example: ```bash dbt compile --select "stg_orders" dbt compile --inline "select * from {{ ref('raw_orders') }}" ``` returns the following: ```bash dbt compile --select "stg_orders" 21:17:09 Running with dbt=1.7.5 21:17:09 Registered adapter: postgres=1.7.5 21:17:09 Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 401 macros, 0 groups, 0 semantic models 21:17:09 21:17:09 Concurrency: 24 threads (target='dev') 21:17:09 21:17:09 Compiled node 'stg_orders' is: with source as ( select * from "jaffle_shop"."main"."raw_orders" ), renamed as ( select id as order_id, user_id as customer_id, order_date, status from source ) select * from renamed ``` ```bash dbt compile --inline "select * from {{ ref('raw_orders') }}" 18:15:49 Running with dbt=1.7.5 18:15:50 Registered adapter: postgres=1.7.5 18:15:50 Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 401 macros, 0 groups, 0 semantic models 18:15:50 18:15:50 Concurrency: 5 threads (target='postgres') 18:15:50 18:15:50 Compiled inline node is: select * from "jaffle_shop"."main"."raw_orders" ``` The command accesses the data platform to cache-related metadata, and to run introspective queries. Use the flags: * `--no-populate-cache` to disable the initial cache population. If metadata is needed, it will be a cache miss, requiring dbt to run the metadata query. This is a `dbt` flag, which means you need to add `dbt` as a prefix. For example: `dbt --no-populate-cache`. * `--no-introspect` to disable [introspective queries](https://docs.getdbt.com/faqs/Warehouse/db-connection-dbt-compile.md#introspective-queries). dbt will raise an error if a resource's definition requires running one. This is a `dbt compile` flag, which means you need to add `dbt compile` as a prefix. For example: `dbt compile --no-introspect`. Resources that use introspective queries Compiled SQL for resources that use introspective queries may depend on metadata from your warehouse. Compilation may be incomplete or may differ depending on the state of that metadata. #### Compiling tests with `--select` You can use `dbt compile` to compile tests, as long as your selector matches a test node in the project. You can also target groups of tests with selector methods: **Compile all test nodes:** ```bash dbt compile --select "resource_type:test" ``` **Compile only generic tests:** ```bash dbt compile --select "test_type:generic" ``` **Compile only singular tests:** ```bash dbt compile --select "test_type:singular" ``` If dbt returns `selection does not match any nodes`, your selector did not match a discovered node. To troubleshoot: 1. List tests for a model selector: ```bash dbt ls --resource-type test --select "MODEL_NAME" ``` 2. Copy one of the returned test node names into `dbt compile --select`: ```bash dbt compile --select "TEST_NODE_NAME" ``` For example, a returned test node name may look like this: ```text FULL_TEST_NODE_NAME ``` 3. If no tests are returned, check test definitions and project paths before running `compile` again. For more selector patterns, refer to [Test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md). (Applies to dbt v2.0 and later) #### dbt Information Schema Use `--generate-info-schema` to write the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) to `target/info_schema/` in a versioned subdirectory (currently `v1/`). The Information Schema exposes your project's metadata as queryable SQL tables (similar to a database's `INFORMATION_SCHEMA`) so you can query models, sources, and more without parsing `manifest.json`. ```shell dbt compile --generate-info-schema --static-analysis strict ``` #### FAQs Why dbt compile needs a data platform connection `dbt compile` needs a data platform connection in order to gather the info it needs (including from introspective queries) to prepare the SQL for every model in your project. ##### dbt compile The [`dbt compile` command](https://docs.getdbt.com/reference/commands/compile.md) generates executable SQL from `source`, `model`, `test`, and `analysis` files. `dbt compile` is similar to `dbt run` except that it doesn't materialize the model's compiled SQL into an existing table. So, up until the point of materialization, `dbt compile` and `dbt run` are similar because they both require a data platform connection, run queries, and have an [`execute` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) set to `True`. However, here are some things to consider: * You don't need to execute `dbt compile` before `dbt run` * In dbt, `compile` doesn't mean `parse`. This is because `parse` validates your written `YAML`, configured tags, and so on. ##### Introspective queries To generate the compiled SQL for many models, dbt needs to run introspective queries, (which is when dbt needs to run SQL in order to pull data back and do something with it) against the data platform. These introspective queries include: * Populating the relation cache. For more information, refer to the [Create new materializations](https://docs.getdbt.com/guides/create-new-materializations.md) guide. Caching speeds up the metadata checks, including whether an [incremental model](https://docs.getdbt.com/docs/build/incremental-models.md) already exists in the data platform. * Resolving [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros), such as `run_query` or `dbt_utils.get_column_values` that you're using to template out your SQL. This is because dbt needs to run those queries during model SQL compilation. * [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) compiles your project by default (unless you pass [`--no-compile`](https://docs.getdbt.com/reference/commands/cmd-docs.md)), so introspective macros such as [`run_query`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) run against the warehouse during documentation builds the same way they do during other compile workflows. Refer to [`run_query`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) for how that works and for using `flags.WHICH` when you want to limitDML or other side-effecting SQL to specific dbt commands. Without a data platform connection, dbt can't perform these introspective queries and won't be able to generate the compiled SQL needed for the next steps in the dbt workflow. You can [`parse`](https://docs.getdbt.com/reference/commands/parse.md) a project and use the [`list`](https://docs.getdbt.com/reference/commands/list.md) resources in the project, without an internet or data platform connection. Parsing a project is enough to produce a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md), however, keep in mind that the written-out manifest won't include compiled SQL. To configure a project, you do need a [connection profile](https://docs.getdbt.com/docs/local/profiles.yml.md) (`profiles.yml` if using the CLI). You need this file because the project's configuration depends on its contents. For example, you may need to use [`{{target}}`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) for conditional configs or know what platform you're running against so that you can choose the right flavor of SQL. --- ### About dbt debug command Use dbt debug to test database connections and check system setup. `dbt debug` is a utility function to test the database connection and display information for debugging purposes, such as the validity of your project file, the [dbt version](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt_version.md), and your installation of any requisite dependencies (like `git` when you run `dbt deps`). It checks your database connection, local configuration, and system setup across multiple axes to help identify potential issues before running dbt commands. By default, `dbt debug` validates: * **Database connection** (for configured profiles) * **dbt project setup** (like `dbt_project.yml` validity) * **System environment** (OS, Python version, installed dbt version) * **Required dependencies** (such as `git` for `dbt deps`) * **Adapter details** (installed adapter versions and compatibility) \*Note: Not to be confused with [debug-level logging](https://docs.getdbt.com/reference/global-configs/logs.md#debug-level-logging) through the `--debug` option which increases verbosity. #### Flags Most of the `dbt debug` flags apply to the self-hosted dbt CLI. Some flags also work in dbt platform CLI, but only `--connection` is supported in the Studio IDE. * dbt CLI: Supports all flags. * Studio IDE: Only supports dbt `debug` and `dbt debug --connection`. * dbt platform CLI: Only supports dbt `debug` and `dbt debug --connection`. You can also use the [`dbt environment`](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) command to interact with your dbt environment. `dbt debug` supports the following flags in your terminal when using the command line interface (CLI): ```text Usage: dbt debug [OPTIONS] Show information on the current dbt environment and check dependencies, then test the database connection. Not to be confused with the --debug option which increases verbosity. Options: --cache-selected-only / --no-cache-selected-only At start of run, populate relational cache only for schemas containing selected nodes, or for all schemas of interest. -d, --debug / --no-debug Display debug logging during dbt execution. Useful for debugging and making bug reports. --defer / --no-defer If set, resolve unselected nodes by deferring to the manifest within the --state directory. --defer-state DIRECTORY Override the state directory for deferral only. --deprecated-favor-state TEXT Internal flag for deprecating old env var. -x, --fail-fast / --no-fail-fast Stop execution on first failure. --favor-state / --no-favor-state If set, defer to the argument provided to the state flag for resolving unselected nodes, even if the node(s) exist as a database object in the current environment. --indirect-selection [eager|cautious|buildable|empty] Controls which tests run based on their relationships to selected models in your DAG. Eager (default) is most inclusive and runs tests that reference your selected models. Cautious is most exclusive and only runs tests that reference selected models. Buildable is in between. Empty runs no tests. --log-cache-events / --no-log-cache-events Enable verbose logging for relational cache events to help when debugging. --log-format [text|debug|json|default] Specify the format of logging to the console and the log file. Use --log-format-file to configure the format for the log file differently than the console. --log-format-file [text|debug|json|default] Specify the format of logging to the log file by overriding the default value and the general --log-format setting. --log-level [debug|info|warn|error|none] Specify the minimum severity of events that are logged to the console and the log file. Use --log-level-file to configure the severity for the log file differently than the console. --log-level-file [debug|info|warn|error|none] Specify the minimum severity of events that are logged to the log file by overriding the default value and the general --log-level setting. --log-path PATH Configure the 'log-path'. Only applies this setting for the current run. Overrides the 'DBT_LOG_PATH' (v1.10 and earlier) or 'DBT_ENGINE_LOG_PATH' (v1.11 or later) when set. --partial-parse / --no-partial-parse Allow for partial parsing by looking for and writing to a pickle file in the target directory. This overrides the user configuration file. --populate-cache / --no-populate-cache At start of run, use `show` or `information_schema` queries to populate a relational cache, which can speed up subsequent materializations. --print / --no-print Output all {{ print() }} macro calls. --printer-width INTEGER Sets the width of terminal output --profile TEXT Which existing profile to load. Overrides setting in dbt_project.yml. -q, --quiet / --no-quiet Suppress all non-error logging to stdout. Does not affect {{ print() }} macro calls. -r, --record-timing-info PATH When this option is passed, dbt will output low-level timing stats to the specified file. Example: `--record-timing-info output.profile` --send-anonymous-usage-stats / --no-send-anonymous-usage-stats Send anonymous usage stats to dbt Labs. --state DIRECTORY Unless overridden, use this state directory for both state comparison and deferral. --static-parser / --no-static-parser Use the static parser. -t, --target TEXT Which target to load for the given profile --use-colors / --no-use-colors Specify whether log output is colorized in the console and the log file. Use --use- colors-file/--no-use-colors-file to colorize the log file differently than the console. --use-colors-file / --no-use-colors-file Specify whether log file output is colorized by overriding the default value and the general --use-colors/--no-use-colors setting. -V, -v, --version Show version information and exit --version-check / --no-version-check If set, ensure the installed dbt version matches the require-dbt-version specified in the dbt_project.yml file (if any). Otherwise, allow them to differ. --warn-error If dbt would normally warn, instead raise an exception. Examples include --select that selects nothing, deprecations, configurations with no associated models, invalid test configurations, and missing sources/refs in tests. --warn-error-options WARNERROROPTIONSTYPE If dbt would normally warn, instead raise an exception based on include/exclude configuration. Examples include --select that selects nothing, deprecations, configurations with no associated models, invalid test configurations, and missing sources/refs in tests. This argument should be a YAML string, with keys 'include' or 'exclude'. eg. '{"include": "all", "exclude": ["NoNodesForSelectionCriteria"]}' --write-json / --no-write-json Whether or not to write the manifest.json and run_results.json files to the target directory --connection Test the connection to the target database independent of dependency checks. Available in Studio IDE and dbt CLI --config-dir Print a system-specific command to access the directory that the current dbt project is searching for a profiles.yml. Then, exit. This flag renders other debug step flags no- ops. --profiles-dir PATH Which directory to look in for the profiles.yml file. If not set, dbt will look in the current working directory first, then HOME/.dbt/ --project-dir PATH Which directory to look in for the dbt_project.yml file. Default is the current working directory and its parents. --vars YAML Supply variables to the project. This argument overrides variables defined in your dbt_project.yml file. This argument should be a YAML string, eg. '{my_variable: my_value}' -h, --help Show this message and exit. ``` #### Example usage Only test the connection to the data platform and skip the other checks `dbt debug` looks for: ```shell dbt debug --connection ``` Show the configured location for the `profiles.yml` file and exit: ```text dbt debug --config-dir To view your profiles.yml file, run: open /Users/alice/.dbt ``` Test the connection in the Studio IDE: ```text dbt debug --connection ``` [![Test the connection in the Studio IDE](/img/reference/dbt-debug-ide.png?v=2 "Test the connection in the Studio IDE")](#)Test the connection in the Studio IDE --- ### About dbt deps command `dbt deps` pulls the most recent version of the dependencies listed in your `packages.yml` from git. See [Package-Management](https://docs.getdbt.com/docs/build/packages.md) for more information. Where relevant, dbt will display up to date and/or latest versions of packages that are listed on dbt Hub. Example below. > This does NOT apply to packages that are installed via git/local ```yaml packages: - package: dbt-labs/dbt_utils version: 0.7.1 - package: brooklyn-data/dbt_artifacts version: 1.2.0 install-prerelease: true - package: dbt-labs/codegen version: 0.4.0 - package: calogica/dbt_expectations version: 0.4.1 - git: https://github.com/dbt-labs/dbt_audit_helper.git revision: 0.4.0 - git: "https://github.com/dbt-labs/dbt_labs-experimental-features" # git URL subdirectory: "materialized-views" # name of subdirectory containing `dbt_project.yml` revision: 0.0.1 - package: dbt-labs/snowplow version: 0.13.0 ``` ```txt Installing dbt-labs/dbt_utils@0.7.1 Installed from version 0.7.1 Up to date! Installing brooklyn-data/dbt_artifacts@1.2.0 Installed from version 1.2.0 Installing dbt-labs/codegen@0.4.0 Installed from version 0.4.0 Up to date! Installing calogica/dbt_expectations@0.4.1 Installed from version 0.4.1 Up to date! Installing https://github.com/dbt-labs/dbt_audit_helper.git@0.4.0 Installed from revision 0.4.0 Installing https://github.com/dbt-labs/dbt_labs-experimental-features@0.0.1 Installed from revision 0.0.1 and subdirectory materialized-views Installing dbt-labs/snowplow@0.13.0 Installed from version 0.13.0 Updated version available: 0.13.1 Installing calogica/dbt_date@0.4.0 Installed from version 0.4.0 Up to date! Updates available for packages: ['tailsdotcom/dbt_artifacts', 'dbt-labs/snowplow'] Update your versions in packages.yml, then run dbt deps ``` #### Predictable package installs dbt generates a `package-lock.yml` file in the root of your project. This file records the exact resolved versions (including commit SHAs) of all packages defined in your `packages.yml` or `dependencies.yml` file. The `package-lock.yml` file ensures consistent and repeatable installs across all environments. When you run `dbt deps`, dbt installs packages based on the versions locked in the `package-lock.yml`. This means that as long as your packages file hasn’t changed, the exact same dependency versions will be installed even if newer versions of those packages have been released. This consistency is important to maintain stability in development and production environments, and to prevent unexpected issues from new releases with potential bugs. If the `packages.yml` file has changed (for example, a new package is added or a version range is updated), then `dbt deps` automatically resolves the new set of dependencies and updates the lock file accordingly. You can also manually trigger an upgrade by running `dbt deps --upgrade`. To maintain consistency, commit the `package-lock.yml` file to version control. This guarantees consistency across all environments and for all developers. ##### Managing `package-lock.yml` The `package-lock.yml` file should be committed to Git initially and updated only when you intend to change versions or uninstall a package. For example, run `dbt deps --upgrade` to get updated package versions or `dbt deps --lock` to update the lock file based on changes to the packages config without installing the packages. To bypass using `package-lock.yml` entirely, you can add it to your project's `.gitignore`. However, this approach sacrifices the predictability of builds. If you choose this route, we strongly recommend adding version pins for third-party packages in your `packages` config. ##### Detecting changes in `packages` config The `package-lock.yml` file includes a `sha1_hash` of your packages config. If you update `packages.yml`, dbt will detect the change and rerun dependency resolution during the next `dbt deps` command. To update the lock file without installing the new packages, use the `--lock` flag: ```shell dbt deps --lock ``` ##### Forcing package updates To update all packages, even if `packages.yml` hasn't changed, use the `--upgrade` flag: ```shell dbt deps --upgrade ``` This is particularly useful for fetching the latest commits from the `main` branch of an internally maintained Git package. warning Forcing package upgrades may introduce build inconsistencies unless carefully managed. ##### Adding specific packages The `dbt deps` command can add or update package configurations directly, saving you from remembering exact syntax. ###### Hub packages (default) Hub packages are the default package types and the easiest to install. ```shell dbt deps --add-package dbt-labs/dbt_utils@1.0.0 # with semantic version range dbt deps --add-package dbt-labs/snowplow@">=0.7.0,<0.8.0" ``` ###### Non-Hub packages Use the `--source` flag to specify the type of package to be installed: ```shell # Git package dbt deps --add-package https://github.com/fivetran/dbt_amplitude@v0.3.0 --source git # Local package dbt deps --add-package /opt/dbt/redshift --source local ``` (Applies to dbt v2.0 and later) #### Installing agent skills If you set the [`ai_provider`](https://docs.getdbt.com/docs/dbt-ai/package-skills.md#set-the-ai_provider-flag) flag, `dbt deps` also installs the [agent skills](https://docs.getdbt.com/docs/dbt-ai/package-skills.md) that your project and its packages ship, writing them into the directory your coding agent reads from: ```text Installing add-a-data-test (demo_skills) -> .claude/skills Installing naming-conventions (demo_skills) -> .claude/skills ``` Without `ai_provider`, `dbt deps` installs your packages as usual and doesn't write any skills. --- ### About dbt docs commands (Applies to dbt v2.0 and later) With dbt v2, use [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2) to generate and view your project's documentation locally. Run `dbt docs generate` to build the site, then `dbt docs serve` to preview it. To generate catalog metadata (`catalog.json`) without building the documentation site, use the [`--write-catalog` flag](#--write-catalog-flag). dbt Docs v2 is built for local and self-hosted workflows. In dbt platform, use [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) instead, a hosted, always-up-to-date view of your docs that your jobs refresh automatically. Refer to [Platform behavior](#platform-behavior) for details. #### dbt Docs v2 dbt Docs v2 is a self-hosted documentation site generated on your local machine or your own pipeline (like GitHub Actions). It isn't available in the dbt platform, where you use [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) instead. Refer to [dbt platform behavior](#dbt-platform-behavior) for details. Instead of loading a static `manifest.json` in the browser, v2 produces Parquet artifacts when you compile or build your project. `dbt docs generate` exports a documentation site made of plain static files (a single-page app plus those artifacts) that any file host can serve. The browser reads the Parquet directly using DuckDB-WASM (WebAssembly), so you don't need to run a stateful server to view your docs. This keeps the experience fast even for large projects. ##### Generate the site `dbt docs generate` compiles your project, writes the index, and exports the documentation site in a single command: ```shell dbt docs generate ``` By default, dbt writes the site into your `target/` directory (`target/index.html`, `target/assets/`, and the index under `target/index/`), matching the layout of dbt v1. You can serve `index.html` from `target/` the same way you did in v1, so a self-hosted pipeline that runs `dbt docs generate && mv target public` keeps working. Use `--output-dir` to write a self-contained copy of the site to a different directory: ```shell dbt docs generate --output-dir site ``` This writes a `site/` directory (the app, hashed assets, and a copy of the index) that you can host on S3, GitHub Pages, Netlify, GitLab Pages, or any similar static file host. To skip compilation and export whatever index is already on disk, use `--no-compile`, which fails with an error if no index exists: ```shell dbt docs generate --no-compile ``` ###### Column lineage and richer metadata To include model column types inferred from your SQL and column-level lineage, run `dbt build`, `dbt run`, or `dbt compile` with [`--generate-info-schema`](https://docs.getdbt.com/docs/build/dbt-information-schema.md) and [`--static-analysis strict`](https://docs.getdbt.com/docs/build/about-static-analysis.md), then export the site: ```shell dbt build --generate-info-schema --static-analysis strict dbt docs generate --no-compile ``` `--no-compile` skips the default compile step in `dbt docs generate` and exports the existing index. If you generate the site without column lineage, dbt Docs v2 hides those features instead of showing empty data. ##### Serve dbt Docs v2 To preview the site locally, run: ```shell dbt docs serve ``` Use `dbt docs serve` to view your documentation locally on your own machine. In dbt platform, use Catalog to explore your project. Refer to [platform behavior](#dbt-platform-behavior) for details. `dbt docs serve` generates the site if it's missing or older than the index, then serves the static files. The server starts on port `8580` by default and opens in your browser. Use `--port` to change the port: ```shell dbt docs serve --port 8081 ``` Use the `--target-path` flag to change the path where dbt reads artifacts from: ```shell dbt docs serve --target-path ~/Developer/internal-analytics/target ``` Because the generated site is a set of static files, you can also host it on any static file host — such as cloud object storage or a static site host — instead of serving it locally. ##### Project overview page dbt Docs v2 renders your project's `__overview__` doc block as the landing page, the same as dbt Docs v1. dbt discovers overview content by scanning your `docs-paths` for `{% docs %}` blocks, so a block in `models/overview.md` is found by default. A file at `docs/overview.md` is only picked up when your project sets `docs-paths: ["docs"]`. If your project defines no overview, dbt renders its default overview content. #### --write-catalog flag The `--write-catalog` flag generates the [`catalog.json`](https://docs.getdbt.com/reference/artifacts/catalog-json.md) artifact, which contains metadata about the tables and views produced by the models in your project. It focuses solely on metadata hydration and does not build the documentation site — use [dbt Docs v2](#dbt-docs-v2) for that. When you run dbt locally, add the flag yourself. You can use it with the following commands: * `dbt build` * `dbt run` * `dbt parse` * `dbt compile` **Example**: ```shell dbt build --write-catalog ``` dbt platform jobs dbt v2 jobs in dbt platform refresh Catalog metadata automatically on every run, so you don't need to add the `--write-catalog` flag there. Refer to [Platform behavior](#platform-behavior) for more info. ##### Local usage When running dbt v2 locally, add the `--write-catalog` flag to your command to generate the catalog: ```shell dbt build --write-catalog ``` ##### What's different from docs generate Both write artifacts, but only `dbt docs generate` builds a documentation site you can view in a browser. | | `--write-catalog` flag | `dbt docs generate` command | | ---------------- | -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | What it produces | `catalog.json`, plus the artifacts the command already generates | `index.html`, `assets/`, and the index files | | What it's for | Metadata for [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) and the metadata APIs | A static site you can preview with `dbt docs serve` or host anywhere | | Where it works | Locally and in dbt platform. The command is added automatically in dbt platform jobs on dbt v2. | Locally only in v2. dbt platform job runs use `dbt compile --write-catalog` instead | #### Commands for generating metadata Each flag targets different metadata and addresses different use cases. Find the right command for your use case in the following list. The [Flag reference](#flag-reference) section breaks down what each flag generates and where it writes. * To display inferred model column types and column-level lineage in [dbt Docs v2](#dbt-docs-v2): ```shell dbt build --generate-info-schema --static-analysis strict dbt docs generate --no-compile ``` * To generate the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) with inferred model column types and column-level lineage: ```shell dbt build --generate-info-schema --static-analysis strict ``` * To generate the dbt Information Schema with inferred model column types and column-level lineage, plus warehouse types and all columns from sources used by the selected nodes: ```shell dbt build --generate-info-schema --static-analysis strict --write-catalog ``` * To generate the dbt Information Schema with warehouse column types only: ```shell dbt build --generate-info-schema --write-catalog ``` * To generate [`catalog.json`](https://docs.getdbt.com/reference/artifacts/catalog-json.md) with warehouse column types: ```shell dbt build --write-catalog ``` ###### Flag reference The following table explains what each flag generates and where the metadata is stored: | Flag | Description | Output location | | ----------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | [`--generate-info-schema`](https://docs.getdbt.com/docs/build/dbt-information-schema.md) | Generates the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md). Add `--static-analysis strict` to include inferred model column types and column-level lineage. | `target/info_schema/v1/` | | [`--static-analysis strict`](https://docs.getdbt.com/docs/build/about-static-analysis.md) | Enables inferred column types and column-level lineage when used with `--generate-info-schema`. | Depends on the flag it's combined with | | [`--write-catalog`](#--write-catalog-flag) | Generates [`catalog.json`](https://docs.getdbt.com/reference/artifacts/catalog-json.md) with warehouse column types.<br /><br />When combined with `--generate-info-schema` on `dbt build` or `dbt run`, writes warehouse column types and all source columns for selected nodes to the dbt Information Schema. | `target/catalog.json` (and `target/info_schema/v1/` when used with `--generate-info-schema`) | dbt Docs v2 reads metadata from the dbt Information Schema, not from `catalog.json`. The `catalog.json` file is a separate artifact for external tools. #### dbt platform behavior dbt Docs v2 is built for local and self-hosted workflows. In dbt platform, use [Catalog](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md), which gives you a cloud-hosted, always-up-to-date view of your project: * In dbt platform jobs running v2, every job run automatically refreshes Catalog metadata, so you don't need a separate docs step. * If you add `dbt docs generate` as a job step, dbt automatically runs `dbt compile --write-catalog` instead and directs you to Catalog. The job doesn't produce the static site or `index.html`. * To share your project with stakeholders who don't develop in dbt, add as many [read-only seats](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) as you need. Developer and read-only seats both include access to Catalog --- ### About dbt environment command dbt platform info The dbt platform CLI provides the `dbt environment` command for environment and connection details. If you're using dbt v2 or dbt v1, use `dbt debug` to inspect profile, target, and connection — or use `dbtf debug` if you have both dbt v1/dbt v2 and platform CLIs and want to inspect dbt v2. The `dbt environment` command enables you to interact with your dbt environment. Use the command for: * Viewing your local configuration details (account ID, active project ID, deployment environment, and more). * Viewing your dbt configuration details (environment ID, environment name, connection type, and more). This guide lists all the commands and options you can use with `dbt environment` in the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). To use them, add a command or option like this: `dbt environment [command]` or use the shorthand `dbt env [command]`. ##### dbt environment show The `show` command allows you to view your local and dbt configuration details. To run the command with the dbt platform CLI, enter one of the following commands, including the shorthand: ```shell dbt environment show ``` ```shell dbt env show ``` The command returns the following information: ```bash ❯ dbt env show Local Configuration: Active account ID 185854 Active project ID 271692 Active host name cloud.getdbt.com dbt_cloud.yml file path /Users/cesar/.dbt/dbt_cloud.yml dbt_project.yml file path /Users/cesar/git/cloud-cli-test-project/dbt_project.yml dbt CLI version 0.35.7 OS info darwin arm64 Cloud Configuration: Account ID 185854 Project ID 271692 Project name Snowflake Environment ID 243762 Environment name Development Defer environment ID [N/A] dbt version 1.6.0-latest Target name default Connection type snowflake Snowflake Connection Details: Account ska67070 Warehouse DBT_TESTING_ALT Database DBT_TEST Schema CLOUD_CLI_TESTING Role SYSADMIN User dbt_cloud_user Client session keep alive false ``` Note, that dbt won't return anything that is a secret key and will return an 'NA' for any field that isn't configured. ##### dbt environment flags Use the following flags (or options) with the `dbt environment` command: * `-h`, `--help` — To view the help documentation for a specific command in your command line interface. ```shell dbt environment [command] --help ``` The `--help` flag returns the following information: ```bash ❯ dbt help environment Interact with dbt environments Usage: dbt environment [command] Aliases: environment, env Available Commands: show Show the working environment Flags: -h, --help help for environment Use "dbt environment [command] --help" for more information about a command. ``` For example, to view the help documentation for the `show` command, enter one of the following commands, including the shorthand: ```shell dbt environment show --help dbt env show -h ``` --- ### About dbt init command `dbt init` helps get you started using dbt! #### New project If this is your first time ever using the tool, it will: * ask you to name your project * ask you which database adapter you're using (or refer to [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md)) * prompt you for each piece of information that dbt needs to connect to that database, such as `account`, `user`, `password`, and more Then, it will: * Create a new folder with your project name and sample files, enough to get you started with dbt * Create a connection profile on your local machine. The default location is `~/.dbt/profiles.yml`. Read more in [configuring your profile](https://docs.getdbt.com/docs/local/profiles.yml.md). * Generate a `.gitignore` file that includes `.env` (starting in dbt v1.12), `target/`, `dbt_packages/`, and `logs/` so local credentials and build artifacts are excluded from version control by default. When using `dbt init` to initialize your project, include the `--profile` flag to specify an existing `profiles.yml` as the `profile:` key to use instead of creating a new one. For example, `dbt init --profile profile_name`. If the profile does not exist in `profiles.yml` or the command is run inside an existing project, the command raises an error. #### Command-line options `dbt init --help` includes both global flags (which apply to all dbt commands) and a small number of `init`-specific options. The most commonly used `init` options are: * `PROJECT_NAME` (optional positional argument): Name of the project folder to create. If omitted, `dbt init` prompts you for a project name. * `--profile PROFILE_NAME`: Use an existing profile name from your `profiles.yml` instead of prompting to create or configure one. * `-s`, `--skip-profile-setup`: Skip interactive profile setup. ```bash dbt init [PROJECT_NAME] [--profile PROFILE_NAME] [--skip-profile-setup] ``` For global command options that also apply to `dbt init`, refer to [Command line options](https://docs.getdbt.com/reference/global-configs/command-line-options.md). #### Existing project If you've just cloned or downloaded an existing dbt project, `dbt init` can still help you set up your connection profile so that you can start working quickly. It will prompt you for connection information, as above, and add a profile (using the `profile` name from the project) to your local `profiles.yml`, or create the file if it doesn't already exist. #### profile\_template.yml `dbt init` knows how to prompt for connection information by looking for a file named `profile_template.yml`. It will look for this file in two places: * **Adapter plugin:** What's the bare minimum Postgres profile? What's the type of each field, what are its defaults? This information is stored in a file called [`dbt/include/postgres/profile_template.yml`](https://github.com/dbt-labs/dbt-postgres/blob/main/dbt/include/postgres/profile_template.yml). If you're the maintainer of an adapter plugin, we highly recommend that you add a `profile_template.yml` to your plugin, too. Refer to the [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) guide for more information. * **Existing project:** If you're the maintainer of an existing project, and you want to help new users get connected to your database quickly and easily, you can include your own custom `profile_template.yml` in the root of your project, alongside `dbt_project.yml`. For common connection attributes, set the values in `fixed`; leave user-specific attributes in `prompts`, but with custom hints and defaults as you'd like. profile\_template.yml ```yml fixed: account: abc123 authenticator: externalbrowser database: analytics role: transformer type: snowflake warehouse: transforming prompts: target: type: string hint: your desired target name user: type: string hint: yourname@jaffleshop.com schema: type: string hint: usually dbt_<yourname> threads: hint: "your favorite number, 1-10" type: int default: 8 ``` ```text $ dbt init Running with dbt=1.0.0 Setting up your profile. user (yourname@jaffleshop.com): summerintern@jaffleshop.com schema (usually dbt_<yourname>): dbt_summerintern threads (your favorite number, 1-10) [8]: 6 Profile internal-snowflake written to /Users/intern/.dbt/profiles.yml using project's profile_template.yml and your supplied values. Run 'dbt debug' to validate the connection. ``` --- ### About dbt invocation command The `dbt invocation` command is available in the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) and allows you to: * List active invocations to debug long-running or hanging invocations. * Identify and investigate sessions causing the `Session occupied` error. * Monitor currently active dbt commands (like `run`, `build`) in real-time. The `dbt invocation` command only lists *active invocations*. If no sessions are running, the list will be empty. Completed sessions aren't included in the output. #### Usage This page lists the command and flag you can use with `dbt invocation`. To use them, add a command or option like this: `dbt invocation [command]`. Available flags in the command line interface (CLI) are [`help`](#dbt-invocation-help) and [`list`](#dbt-invocation-list). ##### dbt invocation help The `help` command provides you with the help output for the `invocation` command in the CLI, including the available flags. ```shell dbt invocation help ``` or ```shell dbt help invocation ``` The command returns the following information: ```bash dbt invocation help Manage invocations Usage: dbt invocation [command] Available Commands: list List active invocations Flags: -h, --help help for invocation Global Flags: --log-format LogFormat The log format, either json or plain. (default plain) --log-level LogLevel The log level, one of debug, info, warning, error or fatal. (default info) --no-color Disables colorization of the output. -q, --quiet Suppress all non-error logging to stdout. Use "dbt invocation [command] --help" for more information about a command. ``` ##### dbt invocation list The `list` command provides you with a list of active invocations in your dbt platform CLI. When a long-running session is active, you can use this command in a separate terminal window to view the active session to help debug the issue. ```shell dbt invocation list ``` The command returns the following information, including the `ID`, `status`, `type`, `arguments`, and `started at` time of the active session: ```bash dbt invocation list Active Invocations: ID 6dcf4723-e057-48b5-946f-a4d87e1d117a Status running Type cli Args [run --select test.sql] Started At 2025-01-24 11:03:19 ➜ jaffle-shop git:(test-cli) ✗ ``` tip To cancel an active session in the terminal, use the `Ctrl + Z` shortcut. #### Related docs * [Install dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) * [Troubleshooting dbt CLI 'Session occupied' error](https://docs.getdbt.com/faqs/Troubleshooting/long-sessions-dbt-cli.md) --- ### About dbt lint command Available in v2 `dbt lint` is a fast SQL linter built into dbt v2, available locally or in dbt platform. dbt lint requires v2 or later. If you're on an earlier version, [upgrade or install dbt](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md?version=2). It's SQLFluff-compatible: it reads your `.sqlfluff` config, uses the same rule codes (for example, `CP01`, `RF03`), and respects `-- noqa` suppression comments. Compatible doesn't mean identical: `dbt lint` and SQLFluff can return different results for the same file and config. Refer to [Rule parity with SQLFluff](#rule-parity-with-sqlfluff). You can use your existing SQLFluff config with minimal changes. dbt Labs intends to track the latest SQLFluff rule spec going forward. note `dbt lint` is part of dbt v2 and is not the same as `dbt sqlfluff lint` on the dbt platform CLI. For SQLFluff on the platform CLI, refer to [Configure the dbt platform CLI](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md). [Linting in Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) continues to use SQLFluff. #### Benchmarks Across project sizes from 1k to 10k models, `dbt lint` runs *40×–250×* faster than SQLFluff with all cores enabled and *280×–1500×* faster than single-threaded SQLFluff. dbt Labs ran these benchmarks on SQLFluff 4.2.1 against dbt projects on the Snowflake dialect, ranging from 1k to 10k models, on a MacBook Pro with a 12-core Apple M4 Pro and 24 GB of RAM. #### Usage ```shell dbt lint [FILE] [flags] ``` `[FILE]` is optional. When you omit `[FILE]`, `dbt lint` lints all SQL files in your project. #### Flags | Flag | Description | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--fix` | Automatically apply fixes for auto-fixable rule violations. See [Rules without autofix](#rules-without-autofix) for rules that cannot be fixed automatically. | | `--config <path>` | Path to a `.sqlfluff` config file. Overrides auto-discovery. | | `--rules` | Comma-separated list of rule codes to enable. Overrides config. | | `--exclude-rules` | Comma-separated list of rule codes to disable. Overrides config. | | `--changed` | Lint only files modified in the current git working tree. | | `--format human\|json\|github-annotation` | Output format. Defaults to `human`. Use `json` for machine-readable output or `github-annotation` for GitHub Actions integration. | | `--jinja-render-mode <mode>` | How `dbt lint` renders Jinja before linting. Accepts `symbolic` (default), `rendered`, or `turbo`. Overrides `.sqlfluff` config. Refer to [Jinja render modes](#jinja-render-modes). | #### Configuration `dbt lint` auto-discovers the nearest `.sqlfluff` file in your project directory tree. CLI flags `--rules` and `--exclude-rules` take precedence over the values in the config file. To create a `.sqlfluff` file, see [SQLFluff configuration files](https://docs.sqlfluff.com/en/stable/configuration/setting_configuration.html). ##### Templater requirement `dbt lint` and `dbt format` only support the SQLFluff `dbt` templater, not the `jinja` templater. Set the templater explicitly in your `.sqlfluff` file: ```ini [sqlfluff] templater = dbt ``` If your `.sqlfluff` file doesn't set `templater = dbt`, each run logs the following warning and continues anyway: ```text [warning] [NotSupported (dbt9000)]: SQLFluff templater 'jinja' is not supported. dbt lint/format only supports SQLFluff templater 'dbt'. Continuing anyway. ``` Setting `templater = dbt` removes the warning. #### dbt-specific rules `dbt lint` also ships five dbt-specific rules, under the `DBT` code prefix. These rules catch dbt patterns that generic SQL linting can't, like hard-coded relation names instead of `ref()`. These rules are off by default and to turn them on, add a `rules` line to your `.sqlfluff` file: ```ini [sqlfluff] templater = dbt dialect = your_dialect rules = DBT02,DBT03,DBT04,DBT05 ``` | Code | Dotted name | Rule | | ------- | ------------------------------------ | -------------------------------------------------------------------------------------- | | `DBT01` | `dbt.import_ctes` | Every `ref()/source()` must be imported through a top-level CTE, not referenced inline | | `DBT02` | `dbt.join_condition_or` | A `JOIN`'s `ON` clause must not contain `OR` | | `DBT03` | `dbt.function_wrapped_filter_column` | A comparison must not wrap a bare column reference in a function call | | `DBT04` | `dbt.leading_wildcard_like` | A `LIKE/ILIKE` pattern must not start with a wildcard | | `DBT05` | `dbt.hard_coded_reference` | A `ref()/source()` must not be hard-coded to a literal string | #### Jinja render modes Before `dbt lint` can check a model, it has to turn your Jinja-templated SQL into plain SQL. Most Jinja renders cleanly at lint time, but some macros ask your data platform a question, such as which columns a table has, and `dbt lint` never connects to your platform, so those calls have no real answer. The `jinja_render_mode` setting controls how `dbt lint` handles them, and that changes which violations you see. Most projects should keep the default, `symbolic`. The three modes are: | Mode | Summary | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | | [`symbolic` (default)](#symbolic-default) | Renders your Jinja normally and substitutes a placeholder for results it can't get from your platform. | | [`rendered`](#rendered) | Renders your Jinja against empty stub values, with no signal that they're stand-ins. | | [`turbo`](#turbo) | Skips execution entirely and lints your literal template text. | ##### Symbolic (default) Executes your Jinja, but tracks which values come from introspective adapter calls, such as `adapter.execute`, `adapter.get_relation`, and `adapter.get_columns_in_relation`. Because those calls can't reach your warehouse at lint time, `dbt lint` replaces their output with a placeholder instead of a misleading empty value. Everything else renders normally. Keep the default. It produces the fewest false positives on projects that use introspective macros. For example, a model that loops over the result of an introspective call: ```jinja {% set cols = adapter.get_columns_in_relation(ref('orders')) %} select {{ cols | map(attribute='name') | join(', ') }} from {{ ref('orders') }} ``` renders, for linting purposes, as something like: ```sql select your_columns from orders ``` `dbt lint` doesn't report violations against the placeholder value itself, but it lints everything else in the query normally. ##### Rendered Executes your Jinja against parse-time stub values. Introspective adapter calls return empty results with no signal that the values aren't real, so the same model renders as: ```sql select from orders ``` The empty `select` list can produce SQL your project would never run, and `dbt lint` checks that unrealistic result instead. Compare `rendered` against `symbolic` when you're investigating an unexpected violation. ##### Turbo Never executes your Jinja. It reads the template syntactically, keeps the literal SQL you wrote, and replaces every `{{ ... }}` expression with a placeholder, including ones that aren't introspective: ```sql select your_expression from your_expression ``` Use it when rendering is too slow or fails outright on a model. It's the fastest mode, but it can't see anything a macro generates. ##### Set the render mode Set the mode for a single run with `--jinja-render-mode`. The flag works with both `dbt lint` and `dbt format`: ```shell dbt lint --jinja-render-mode rendered dbt format --jinja-render-mode turbo ``` Set it for the whole project in the `[dbt]` section of your `.sqlfluff` file: ```ini [dbt] jinja_render_mode = rendered ``` The CLI flag takes precedence over the config file. ##### Render variants In `symbolic` and `turbo` modes, a single model can produce more than one candidate SQL output. When `dbt lint` reaches an `{% if %}` block whose condition it can't resolve, it lints more than one branch rather than guessing which one you meant. Each candidate is a *render variant*, and `dbt lint` reports the violations it finds across all of them. `render_variant_limit` caps how many variants `dbt lint` produces per model, and defaults to `5`. Set it in the `[sqlfluff]` section of your `.sqlfluff` file: ```ini [sqlfluff] render_variant_limit = 10 ``` Raising the limit widens coverage at the cost of lint time, because each additional variant is another render of the template. Lowering it to `1` restricts `dbt lint` to a single variant per model. #### Ignoring files and directories Use a `.sqlfluffignore` file at your project root to exclude paths you aren't ready to lint yet, such as `dbt_packages/` or `models/legacy/`. `.sqlfluffignore` uses `.gitignore`-style syntax. For the full pattern reference, see the [SQLFluff `.sqlfluffignore` documentation](https://docs.sqlfluff.com/en/stable/configuration.html#id2). ```text # .sqlfluffignore dbt_packages/ models/legacy/ snapshots/ ``` When you're ready to lint those paths, remove their entries from `.sqlfluffignore`. ##### Reducing noise in the Studio IDE Problems tab The Studio IDE lints SQL automatically and surfaces violations in the **Problems** tab. If you see a large number of style warnings and aren't ready to address them, add your model directories to `.sqlfluffignore` to remove those violations from the **Problems** tab immediately. Remove the ignore entries incrementally as you clean up violations. #### Suppressing violations `dbt lint` supports the full SQLFluff suppression syntax: | Suppression | Scope | | -------------------------- | ------------------------------------ | | `-- noqa` | Suppress all violations on the line | | `-- noqa: CP01, RF03` | Suppress specific rules on this line | | `-- noqa-file` | Suppress all violations in the file | | `-- sqlfluff:disable CP01` | Disable a rule in the file | #### Supported dialects The following dialects are currently supported with `dbt lint`: * Snowflake * BigQuery * DuckDB * Redshift * Databricks * SparkSQL (currently aliased to Databricks) Additional dialect support is coming soon. #### dbt format `dbt format` (also available as `dbt fmt`) requires v2 or later. It automatically formats your SQL files according to the layout (`LT*`) rules in your `.sqlfluff` file. Unlike `dbt lint`, it doesn't issue diagnostics. It applies fixes silently and in place when you run the command. ```shell dbt format [FILE] [flags] dbt fmt [FILE] [flags] ``` `[FILE]` is optional. When omitted, `dbt format` formats all SQL files in your project. You can also pass a single file or a directory to scope the run. `dbt format` logs a warning and continues when `.sqlfluff` does not set `templater = dbt`. Refer to [Templater requirement](#templater-requirement). ##### Format flags | Flag | Description | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--check` | Checks whether files are formatted without writing changes. Exits with a non-zero code if any file isn't formatted. Never modifies a file. | | `--layout KEY=VALUE`, `-l KEY=VALUE` | Overrides SQLFluff-style layout rules for this run. Repeat the flag to set multiple options. Refer to [Layout options](#layout-options). | | `--jinja-render-mode MODE` | Sets how `dbt format` renders Jinja before formatting. Accepts `symbolic` (default), `rendered`, or `turbo`. Refer to [Jinja render modes](#jinja-render-modes). | ##### Layout options The `--layout` flag takes `KEY=VALUE` pairs and overrides the matching layout rules in your `.sqlfluff` file for a single run. It overrides only the keys you pass, and `dbt format` still reads all other layout rules from your `.sqlfluff` file. The supported keys are: | Key | Description | | -------------------------- | ------------------------------------ | | `indent=NUMBER` | Sets the number of spaces to indent. | | `commas=leading\|trailing` | Sets comma placement. | | `line-length=NUMBER` | Sets the maximum line length. | For example, to format with leading commas and a two-space indent: ```shell dbt format -l commas=leading -l indent=2 ``` ##### Check formatting in CI Use `--check` as a dry run to gate pull requests on formatting. If every file is formatted, the command exits with code `0`. If a file isn't formatted, the command exits with code `1`, leaves the file untouched, and returns an error like this: ```text [error] [Generic (dbt1000)]: Formatting check failed for models/staging/stg_orders.sql. Run `dbt fmt` to format it. ``` ```shell dbt format --check # check all SQL files dbt format --check models/staging # check a directory dbt format --check models/staging/stg_orders.sql # check a single file ``` To fix failures, run `dbt format` without `--check`. #### Rule parity with SQLFluff `dbt lint` aims for high overlap with SQLFluff, but it doesn't guarantee rule-for-rule parity, and small differences will always exist. Layout and indentation rules, such as `LT02`, are one known area of difference. Because [linting in the Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) still uses SQLFluff, Studio IDE **Lint file** and `dbt lint` can report different violations for the same project code. Similarly, CI jobs on a dbt v2 version invoke `dbt lint` instead of SQLFluff, so results from [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) can differ from your SQLFluff results. If you need SQLFluff behavior, you can either lint in the Studio IDE, which continues to run SQLFluff, or run SQLFluff locally using the standalone dbt v1 engine templater. Refer to [dbt v2 limitations](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations) for more information. #### Promoting lint warnings to errors By default, violations from [rules without an autofix](#rules-without-autofix) show as warnings and don't fail `dbt lint`. Violations that `--fix` can fix show as errors and fail the command. If you want to report a rule violation as an error instead of a warning, add the rule's name to `warn_error_options` in `dbt_project.yml`. You can find the rule name in the lint output. For example, rule `ST05` shows as `PreferCTE`: ```yaml flags: warn_error_options: error: - PreferCTE ``` If you set `error: all` instead, dbt promotes every warning, including warnings from commands other than `lint`: ```yaml flags: warn_error_options: error: all ``` To promote one lint rule, list that rule by name as in the first example. Refer to [Warnings](https://docs.getdbt.com/reference/global-configs/warnings.md). In a CI job, the **Linting** checkbox runs `dbt lint` and uses this project file. Select **Stop running on error** to fail the job when lint shows an error. To turn linting on, refer to [Set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs) and [SQL linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting). #### Limitations Keep these limitations in mind: ##### Rules without autofix The following rules report violations but can't be auto-fixed by `--fix`. They require reordering of SQL fragments or broader reflow that source-mapping (based on `macro_spans`) can't safely fix inside Jinja-templated SQL: * **Aliasing:** `AL03`, `AL04`, `AL06`, `AL08` * **References:** `RF01`, `RF02`, `RF04`, `RF05` * **Structure:** `ST03`, `ST04`, `ST05`, `ST06`, `ST07`, `ST09`, `ST10`, `ST11` * **Ambiguity / convention:** `AM01`, `AM06`, `CV08`, `CV09`, `CV12` To fail the run on one of these rules, refer to [Promoting lint warnings to errors](#promoting-lint-warnings-to-errors). ##### Single fix pass `--fix` runs a single pass; it doesn't iterate until the file is clean. A fix applied by one rule can expose a violation from another rule on the next run. For example, `AL09` removes a self-alias, which may then cause `RF02` to flag the now-unqualified reference. Re-run `dbt lint --fix` until the output is clean. #### FAQs  Why doesn't dbt lint check every possible output of my Jinja? `dbt lint` lints a bounded set of [render variants](#render-variants) per model, not every SQL output your macros could produce under every combination of inputs. This is a deliberate choice. The number of possible outputs grows combinatorially with the number of unresolved conditions in a template, so linting all of them is expensive and surfaces violations in SQL your project might never execute. Instead, `dbt lint` varies one unresolved condition at a time, up to `render_variant_limit`. Keep one consequence in mind: when `dbt lint` explores a branch your project never takes, it can surface a violation from that unused path, most often inside a third-party package macro. If you have feedback on this approach, open an issue in the [dbt-core GitHub repository](https://github.com/dbt-labs/dbt/issues) with the `Linter` label.  Why doesn't dbt lint report violations from some macros? `dbt lint` lints the SQL that all macros produce, but it can't resolve macros that depend on querying your warehouse. Introspective adapter calls, such as `adapter.execute` and `adapter.get_columns_in_relation`, have no real result at lint time. How `dbt lint` handles that depends on your [render mode](#jinja-render-modes). In the default `symbolic` mode, `dbt lint` substitutes a placeholder wherever an introspective result would appear and doesn't report violations against that placeholder, because flagging fabricated output generates noise rather than signal. The rest of the macro still lints normally. This behavior is similar to SQLFluff's `ignore_templated_areas` setting. #### Feedback If you encounter unexpected behavior or have suggestions, open an issue in the [dbt-labs/dbt GitHub repository](https://github.com/dbt-labs/dbt/issues) and apply the `Linter` label. --- ### About dbt login Local development Use dbt login to authenticate once and access dbt platform-based capabilities across dbt tools. info Available in dbt v1.13 and later. Run `dbt login` from the command line to access dbt platform-based capabilities. It'll open browser-based authentication where you can sign in to your existing dbt platform account or create a free one — no credit card required! Run [`dbt login status`](#dbt-login-status) to view your current authentication status. #### When to use dbt login `dbt login` is an interactive, browser-based sign-in flow for development on macOS, Linux, and Windows. Use `dbt login` to access dbt platform-based capabilities including: * dbt platform capabilities in the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md), such as the **Catalog** tab * [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md#setting-up-dbt-state) in supported versions Refer to [VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability) for the full list of features and their availability. `dbt login` doesn't support non-interactive authentication. For [non-interactive](#non-interactive-environments) environments, such as CI/CD jobs, scheduled jobs, or external orchestrators, use a service token instead. #### Before you log in A free dbt platform account carries your access across all your dbt tools — the CLI, the VS Code extension, and dbt State — so you only log in once. No credit card required. Run `dbt login` to create a free account, or log in to an existing one. Logging in is simply how dbt confirms your access to dbt platform-based capabilities in your local development environment. Note that this is separate from [dbt platform user license types](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md?version=2.0) (such as Developer or Analyst), which controls what you can do *inside* dbt platform. Refer to [VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability) for the full list of features and their availability. (Applies to dbt v1.13 and later) #### `dbt login` with dbt State [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login---help) is also used for dbt State authentication. Running this command opens a browser window where you can: * Log in with your dbt platform account. * Create a dbt platform account: 1. Enter your email address and click **Continue**. 2. Enter you name, password, and agree to the Terms of Service. 3. Click **Continue**. 4. Verify your e-mail address. In dbt v2, after platform authentication, the CLI checks your configuration and responds accordingly: | dbt State enabled in dbt platform? | dbt State enabled locally? | Behavior | | ---------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ | ✅ | dbt State is ready to use. | | ✅ | ❌ | CLI prompts you to enable dbt State locally. If you confirm, [`user_settings.yml`](https://docs.getdbt.com/reference/global-configs/user-settings.md) is updated automatically. | | ❌ | ✅ | CLI prompts you to enable dbt State in your platform account. | In dbt v1.13, `dbt login` automatically sets `manage_state: true` in [`user_settings.yml`](https://docs.getdbt.com/reference/global-configs/user-settings.md) after platform authentication. If you previously set `manage_state: false`, the CLI prompts you before overwriting it. #### Authentication methods `dbt login` is interactive only. It opens a browser-based sign-in flow and stores your session locally. You can log into dbt either using [interactive](#interactive-authentication) or [non-interactive](#non-interactive-environments) methods. ##### Interactive authentication Use `dbt login` when you're developing locally in a terminal or IDE and can complete sign-in in a browser. * When you run `dbt login`, dbt opens a browser-based authentication flow. After you complete authentication, dbt stores your login state locally and confirms that you're signed in. * After you sign in, dbt can use your login state to access dbt platform-based capabilities across the dbt CLI, VS Code extension, and dbt State. * Your session stays active across terminal sessions, and dbt refreshes it automatically while you're active. As long as you use dbt at least once every 7 days, you stay signed in. If you're inactive for longer, dbt prompts you to run `dbt login` again. Refer to [Staying signed in](#staying-signed-in) for details. Run `dbt login` from your **local terminal** (not in the dbt platform UI). **Example (interactive):** ```shell dbt login dbt login status ``` Complete the browser prompts to sign in or register for a dbt platform account. ##### Non-interactive environments Non-interactive means dbt runs without a person at a keyboard to open a browser — for example, a GitHub Actions workflow, a cron job, or a scheduled dbt platform job. There is no separate "non-interactive login" command. You do not run `dbt login` in these environments. If you are new to dbt, start with [interactive authentication](#interactive-authentication). You only need the guidance in this section when you automate dbt runs. ###### Authenticate with a service token In non-interactive environments (such as CI/CD jobs, scheduled jobs, or external orchestrators like Airflow), use a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) instead of `dbt login`. The service token does two jobs: * authenticates the run * gives access to dbt platform-based capabilities, so they work in your pipeline the same way they do locally. Unlike interactive `dbt login`, service tokens don't expire, so there's nothing to refresh. Rotate them yourself in dbt platform when you need to. Set the following [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md#special-environment-variables) so dbt can authenticate and provide that access: | Environment variable | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `DBT_CLOUD_ACCOUNT_HOST` | Your dbt platform [access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) (for example, `abc123.us1.dbt.com`). | | `DBT_CLOUD_ACCOUNT_ID` | Your dbt platform account ID. | | `DBT_CLOUD_TOKEN` | The service token value. Create a service token with a permission set that includes feature licensing access, such as `Job Runner`, `Job Creator`, `Job Admin`, `Developer`, or `Account Admin`. | | `DBT_CLOUD_PROJECT_ID` | Your dbt platform project ID. | ```shell export DBT_CLOUD_ACCOUNT_HOST=abc123.us1.dbt.com export DBT_CLOUD_ACCOUNT_ID=12345 export DBT_CLOUD_TOKEN=dbtc_xxxxxxxx export DBT_CLOUD_PROJECT_ID=67890 ``` #### How shared login works across dbt tools * Use `dbt login` to authenticate with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). You can start the sign-in flow from the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md): * If you run `dbt login` from the CLI, the dbt VS Code extension uses that login in your next extension session. * If you sign in from the dbt VS Code extension, you can use that login the next time you run a command that requires authentication. When you run a command or use a feature that requires authentication, dbt checks your current login state. If you're signed in, the feature runs. If you're not signed in, dbt tells you which feature requires authentication and prompts you to run `dbt login`. For the VS Code extension sign-in flow, refer to [Sign in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md). #### Staying signed in Once you log in, dbt keeps you signed in automatically — you usually won't notice it at all. You stay signed in as long as you use dbt at least once every 7 days. If you're inactive for longer than that, dbt might ask you to log in again in your next session to ensure security. If your access expires, run `dbt login` to sign back in. * No dbt account yet? Create a free one with `dbt login` — it's the best way to use the [full set of features](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability) and get the most out of the extension. * Not ready to run `dbt login`? No worries — continue using the vast majority of features. If you're not sure where you stand, run [`dbt license info`](#troubleshooting) to check your status. #### Usage This page lists the commands and output you can use with `dbt login`: * [`dbt login`](#dbt-login) * [`dbt login --help`](#dbt-login---help) * [`dbt login status`](#dbt-login-status) * [`dbt license info`](#dbt-license-info) ##### dbt login Run `dbt login` from your terminal to open browser-based authentication: ```shell dbt login ``` The CLI opens your default browser and shows the authentication URL: ```text Opening your browser to complete login... → https://us1.dbt.com/register If you need to reset your password, complete the reset, then re-run dbt login to finish authenticating. Waiting for authentication… ``` After you sign in or register, dbt saves your credentials to your local dbt configuration directory: * macOS and Linux: `~/.dbt/` * Windows: `C:\Users\[username]\.dbt\` Refer to [Staying signed in](#staying-signed-in) for details on how long you stay signed in. When authentication completes successfully, the CLI shows: ```text Congratulations! You are now signed in. ``` When authentication fails, the CLI shows: ```text Authentication failed. Re-run dbt login to try again. [Placeholder for user-facing error message on failure reason] ``` dbt State and dbt login If you have a dbt platform account, you can use the same account to authenticate with dbt State — no separate sign-in required. For full dbt State setup, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). ##### dbt login --help ```text Usage: dbt login [OPTIONS] <COMMAND> Commands: status Show dbt authentication status ``` ##### dbt login status Run `dbt login status` to show whether dbt is authenticated: ```shell dbt login status ``` If dbt is authenticated, the CLI shows the authentication method: ```text Status: authenticated (via METHOD) ``` If dbt is not authenticated, the CLI shows: ```text Status: unauthenticated ``` ##### dbt license info dbt uses local licenses to cache your logged-in state and give you access to dbt platform-based capabilities. Run `dbt license info` as an additional check to verify the status of the license used by dbt v2. This is useful when a dbt platform-based capability isn't working and `dbt login status` tells you you're authenticated: ```shell dbt license info ``` The output should look like this: ```shell dbtf license info ... status: valid licensing_enabled: true license_type: developer features: compare, state-aware-orchestration, strict-static-analysis issued_at: June 17, 2026 10:55 UTC expires_at: June 18, 2026 10:55 UTC ``` Add `--json` for machine-readable output: ```shell dbt license info --json ``` For how to interpret each status, refer to [Troubleshooting](#troubleshooting). #### Troubleshooting If a dbt platform-based capability isn't working, or you're not sure whether you're still signed in, run `dbt license info` to check your access: ```shell dbt license info ``` Add `--json` for machine-readable output: ```shell dbt license info --json ``` The output shows your current status. Use the following table to interpret it: | Status | What it means | What to do | | ----------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | `valid` | You're signed in and your features are available. | Nothing — you're all set. | | `expired` | Your access has expired. | Run `dbt login` to sign in again. | | `not_found` | dbt couldn't find any sign-in for you. | Run `dbt login`, or set a [service token](#authenticate-with-a-service-token) for orchestrated runs. | | `invalid` | Your access failed validation. | Run `dbt login` to refresh it. | | `transient_error` | A temporary network or server issue. | Retry — your cached access stays active in the meantime. | If you just registered a new account If you created a new dbt platform account but haven't verified your email yet, dbt warns you on each run during a short grace period. After the grace period ends, dbt platform-based capabilities stop working until you verify. You can still use the vast majority of features like code error diagnostics and Jinja LSP go-to ref definition, [and more](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability). Check your inbox for the verification email, or [contact dbt Support](https://docs.getdbt.com/docs/dbt-support.md) if you need it resent. #### Related commands * [`dbt login status`](#dbt-login-status) — Shows your current dbt authentication status. * [`dbt license info`](#dbt-license-info) — Shows your current access status and helps diagnose feature availability. * [`dbt init`](https://docs.getdbt.com/reference/commands/init.md) — Use during first-time project setup; prompts you to run `dbt login` to access authenticated dbt platform-based capabilities. * [`dbt deps`](https://docs.getdbt.com/reference/commands/deps.md) — Installs packages for a dbt project. * [`dbt debug`](https://docs.getdbt.com/reference/commands/debug.md) — Tests your dbt project and connection configuration. --- ### About dbt ls (list) command The `dbt ls` command lists resources in your dbt project. It accepts selector arguments that are similar to those provided in [dbt run](https://docs.getdbt.com/reference/commands/run.md). `dbt list` is an alias for `dbt ls`. While `dbt ls` will read your [connection profile](https://docs.getdbt.com/docs/local/profiles.yml.md) to resolve [`target`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md)-specific logic, this command will not connect to your database or run any queries. (Applies to dbt v2.0 and later) In dbt platform run logs when using dbt v2, `dbt ls` and `dbt list` node results can appear as **No-op** when there is no final status. This replaces **Unknown** and indicates that dbt selected the node but did not run it. ##### Usage ```text dbt ls [--resource-type {model,semantic_model,source,seed,snapshot,metric,test,exposure,analysis,function,default,all}] [--select SELECTION_ARG [SELECTION_ARG ...]] [--models SELECTOR [SELECTOR ...]] [--exclude SELECTOR [SELECTOR ...]] [--selector YML_SELECTOR_NAME] # legacy flag; use `--select selector:SELECTOR_NAME` in v1.12+ [--output {json,name,path,selector}] [--output-keys KEY_NAME [KEY_NAME]] ``` Refer to [resource selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) for more information on how to select resources in dbt **Arguments**: * `--resource-type`: This flag restricts the "resource types" returned by dbt in the `dbt ls` command. By default, all resource types are included in the results of `dbt ls` except for the analysis type. * `--select`: This flag specifies one or more selection-type arguments used to filter the nodes returned by the `dbt ls` command * `--models`: Like the `--select` flag, this flag is used to select nodes. It implies `--resource-type=model`, and will only return models in the results of the `dbt ls` command. Supported for backwards compatibility only. * `--exclude`: Specify selectors that should be *excluded* from the list of returned nodes. - `--output`: This flag controls the format of output from the `dbt ls` command. - `--output-keys`: If `--output json`, this flag controls which node properties are included in the output. Note that the `dbt ls` command does not include models which are disabled or schema tests which depend on models which are disabled. All returned resources will have a `config.enabled` value of `true`. ##### Example usage The following examples show how to use the `dbt ls` command to list resources in your project. * [Listing models by package](#listing-models-by-package) * [Listing tests by tag name](#listing-tests-by-tag-name) * [Listing schema tests of incremental models](#listing-schema-tests-of-incremental-models) * [Listing JSON output](#listing-json-output) * [Listing JSON output with custom keys](#listing-json-output-with-custom-keys) * [Listing semantic models](#listing-semantic-models) * [Listing functions](#listing-functions) ###### Listing models by package ```bash dbt ls --select snowplow.* snowplow.snowplow_base_events snowplow.snowplow_base_web_page_context snowplow.snowplow_id_map snowplow.snowplow_page_views snowplow.snowplow_sessions ... ``` ###### Listing tests by tag name ```bash dbt ls --select tag:nightly --resource-type test my_project.schema_test.not_null_orders_order_id my_project.schema_test.unique_orders_order_id my_project.schema_test.not_null_products_product_id my_project.schema_test.unique_products_product_id ... ``` ###### Listing schema tests of incremental models ```bash dbt ls --select config.materialized:incremental,test_type:schema model.my_project.logs_parsed model.my_project.events_categorized ``` ###### Listing JSON output ```bash dbt ls --select snowplow.* --output json {"name": "snowplow_events", "resource_type": "model", "package_name": "snowplow", ...} {"name": "snowplow_page_views", "resource_type": "model", "package_name": "snowplow", ...} ... ``` ###### Listing JSON output with custom keys ```bash dbt ls --select snowplow.* --output json --output-keys "name resource_type description" {"name": "snowplow_events", "description": "This is a pretty cool model", ...} {"name": "snowplow_page_views", "description": "This model is even cooler", ...} ... ``` ###### Listing semantic models List all resources upstream of your orders semantic model: ```bash dbt ls -s +semantic_model:orders ``` ###### Listing file paths ```bash dbt ls --select snowplow.* --output path models/base/snowplow_base_events.sql models/base/snowplow_base_web_page_context.sql models/identification/snowplow_id_map.sql ... ``` ###### Listing functions List all functions in your project: ```bash dbt list --select "resource_type:function" # or dbt ls --resource-type function jaffle_shop.area_of_circle jaffle_shop.whoami ``` --- ### About dbt parse command The `dbt parse` command parses and validates the contents of your dbt project. If your project contains Jinja or YAML syntax errors, the command will fail. It will also produce an artifact with detailed timing information, which is useful to understand parsing times for large projects. Refer to [Project parsing](https://docs.getdbt.com/reference/parsing.md) for more information. Starting in v1.5, `dbt parse` will write or return a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md), enabling you to introspect dbt's understanding of all the resources in your project. Since `dbt parse` doesn't connect to your warehouse, [this manifest will not contain any compiled code](https://docs.getdbt.com/faqs/Warehouse/db-connection-dbt-compile.md). By default, the Studio IDE will attempt a "partial" parse, which means it'll only check changes since the last parse (new or updated parts of your project when you make changes). Since the Studio IDE automatically parses in the background whenever you save your work, manually running `dbt parse` yourself is likely to be fast because it's just looking at recent changes. As an option, you can tell dbt to check the entire project from scratch by using the `--no-partial-parse` flag. This makes dbt perform a full re-parse of the project, not just the recent changes. ```text $ dbt parse 13:02:52 Running with dbt=1.5.0 13:02:53 Performance info: target/perf_info.json ``` target/perf\_info.json ```json { "path_count": 7, "is_partial_parse_enabled": false, "parse_project_elapsed": 0.20151838900000008, "patch_sources_elapsed": 0.00039490800000008264, "process_manifest_elapsed": 0.029363873999999957, "load_all_elapsed": 0.240095269, "projects": [ { "project_name": "my_project", "elapsed": 0.07518750299999999, "parsers": [ { "parser": "model", "elapsed": 0.04545303199999995, "path_count": 1 }, { "parser": "operation", "elapsed": 0.0006415469999998535, "path_count": 1 }, { "parser": "seed", "elapsed": 0.026538173000000054, "path_count": 2 } ], "path_count": 4 }, { "project_name": "dbt_postgres", "elapsed": 0.0016448299999998195, "parsers": [ { "parser": "operation", "elapsed": 0.00021672399999994596, "path_count": 1 } ], "path_count": 1 }, { "project_name": "dbt", "elapsed": 0.006580432000000025, "parsers": [ { "parser": "operation", "elapsed": 0.0002488560000000195, "path_count": 1 }, { "parser": "docs", "elapsed": 0.002500640000000054, "path_count": 1 } ], "path_count": 2 } ] } ``` (Applies to dbt v2.0 and later) #### dbt Information Schema Use `--generate-info-schema` to write the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) to `target/info_schema/` in a versioned subdirectory (currently `v1/`). The Information Schema exposes your project's metadata as queryable SQL tables (similar to a database's `INFORMATION_SCHEMA`) so you can query models, sources, and more without parsing `manifest.json`. ```shell dbt parse --generate-info-schema ``` Because `dbt parse` doesn't connect to your warehouse, the Information Schema it produces contains no column types, no column-level lineage, and no runtime results. --- ### About dbt retry command Retry re-executes the last invocation from the point of failure. * If no nodes are executed before the failure (for example, if a run failed early due to a warehouse connection or permission errors), retry won't run anything since there are no recorded nodes to retry from. * In these cases, we recommend checking your [`run_results.json` file](https://docs.getdbt.com/reference/artifacts/run-results-json.md) and manually re-running the full job so the nodes build. * Once some nodes have run, you can use retry to re-execute from any new point of failure. * If the previously executed command completed successfully, retry will finish as `no operation`. #### Retry flags The `dbt retry` flags apply when you use a self-hosted dbt installation or the Studio IDE. dbt platform CLI If you use the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) against your cloud environment, `dbt retry` accepts only a small subset of overrides—typically `--threads`, `--vars`, and related options. Use `dbt retry --help` on your machine for the exact list your CLI build supports. (Applies to dbt v2.0 and later) The following flags are supported when you run `dbt retry` with dbt v2: | Flag | Input value | Description | Example | | ------------------------- | ----------- | ---------------------------------------------------------------- | ------------------------------------------------ | | `-t, --target` | target | The target to execute | `dbt retry -t prod` | | `--project-dir` | path | The directory to load the dbt project from | `dbt retry --project-dir .` | | `--profile` | profile | The profile to use | `dbt retry --profile jaffle_shop` | | `--profiles-dir` | path | The directory to load the profiles from | `dbt retry --profiles-dir ~/.dbt` | | `--packages-install-path` | path | The directory to install packages | `dbt retry --packages-install-path dbt_packages` | | `--target-path` | path | The output directory for all produced assets | `dbt retry --target-path target` | | `--vars` | vars | Variables for the project (use the format shown in the CLI help) | `dbt retry --vars '{"my_var": "new_value"}'` | <br /> Run `dbt retry --help` for the full list of flags available. ##### dbt v2 node selection Unlike `dbt retry` with dbt v1, dbt v2 lets you narrow what gets retried using [`--select`](https://docs.getdbt.com/reference/node-selection/syntax.md), [`--exclude`](https://docs.getdbt.com/reference/node-selection/syntax.md), and [`--selector`](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md). Those arguments override the prior invocation’s selection set for the retry run instead of only inheriting it. ###### Examples ```shell dbt retry --select my_model+ ``` ```shell dbt retry --exclude package:analytics --selector nightly_models ``` #### Supported commands Retry works with the following commands: * [`build`](https://docs.getdbt.com/reference/commands/build.md) * [`check`](https://docs.getdbt.com/reference/commands/check.md) * [`compile`](https://docs.getdbt.com/reference/commands/compile.md) * [`clone`](https://docs.getdbt.com/reference/commands/clone.md) * [`docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate) * [`seed`](https://docs.getdbt.com/reference/commands/seed.md) * [`snapshot`](https://docs.getdbt.com/reference/commands/snapshot.md) * [`test`](https://docs.getdbt.com/reference/commands/test.md) * [`run`](https://docs.getdbt.com/reference/commands/run.md) * [`run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) Retry references [run\_results.json](https://docs.getdbt.com/reference/artifacts/run-results-json.md) to determine where to start. Executing retry without correcting the previous failures yields idempotent results. (Applies to dbt v2.0 and later) With `dbt retry`, you can optionally pass new [`--select`](https://docs.getdbt.com/reference/node-selection/syntax.md), [`--exclude`](https://docs.getdbt.com/reference/node-selection/syntax.md), or [`--selector`](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) arguments to narrow the retry scope, as described in [Retry flags](#retry-flags). Example results of executing `dbt retry` after a successful `dbt run`: ```shell Running with dbt=1.6.1 Registered adapter: duckdb=1.6.0 Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 348 macros, 0 groups, 0 semantic models Nothing to do. Try checking your model configs and model specification args ``` Example of when `dbt run` encounters a syntax error in a model: ```shell Running with dbt=1.6.1 Registered adapter: duckdb=1.6.0 Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 348 macros, 0 groups, 0 semantic models Concurrency: 24 threads (target='dev') 1 of 5 START sql view model main.stg_customers ................................. [RUN] 2 of 5 START sql view model main.stg_orders .................................... [RUN] 3 of 5 START sql view model main.stg_payments .................................. [RUN] 1 of 5 OK created sql view model main.stg_customers ............................ [OK in 0.06s] 2 of 5 OK created sql view model main.stg_orders ............................... [OK in 0.06s] 3 of 5 OK created sql view model main.stg_payments ............................. [OK in 0.07s] 4 of 5 START sql table model main.customers .................................... [RUN] 5 of 5 START sql table model main.orders ....................................... [RUN] 4 of 5 ERROR creating sql table model main.customers ........................... [ERROR in 0.03s] 5 of 5 OK created sql table model main.orders .................................. [OK in 0.04s] Finished running 3 view models, 2 table models in 0 hours 0 minutes and 0.15 seconds (0.15s). Completed with 1 error and 0 warnings: Runtime Error in model customers (models/customers.sql) Parser Error: syntax error at or near "selct" Done. PASS=4 WARN=0 ERROR=1 SKIP=0 TOTAL=5 ``` Example of a subsequent failed `dbt retry` run without fixing the error(s): ```shell Running with dbt=1.6.1 Registered adapter: duckdb=1.6.0 Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 348 macros, 0 groups, 0 semantic models Concurrency: 24 threads (target='dev') 1 of 1 START sql table model main.customers .................................... [RUN] 1 of 1 ERROR creating sql table model main.customers ........................... [ERROR in 0.03s] Done. PASS=4 WARN=0 ERROR=1 SKIP=0 TOTAL=5 ``` Example of a successful `dbt retry` run after fixing error(s): ```shell Running with dbt=1.6.1 Registered adapter: duckdb=1.6.0 Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 348 macros, 0 groups, 0 semantic models Concurrency: 24 threads (target='dev') 1 of 1 START sql table model main.customers .................................... [RUN] 1 of 1 OK created sql table model main.customers ............................... [OK in 0.05s] Finished running 1 table model in 0 hours 0 minutes and 0.09 seconds (0.09s). Completed successfully Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1 ``` In each scenario `dbt retry` picks up from the error rather than running all of the upstream dependencies again. --- ### About dbt rpc command The dbt-rpc plugin is deprecated dbt Labs actively maintained `dbt-rpc` for compatibility with dbt-core versions up to v1.5. Starting with dbt-core v1.6 (released in July 2023), `dbt-rpc` is no longer supported for ongoing compatibility. In the meantime, dbt Labs will be performing critical maintenance only for `dbt-rpc`, until the last compatible version of dbt-core has reached the [end of official support](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions). At that point, dbt Labs will archive this repository to be read-only. ##### Overview You can use the `dbt-rpc` plugin to run a Remote Procedure Call (rpc) dbt server. This server compiles and runs queries in the context of a dbt project. Additionally, the RPC server provides methods that enable you to list and terminate running processes. We recommend running an rpc server from a directory containing a dbt project. The server will compile the project into memory, then accept requests to operate against that project's dbt context. Running on Windows We do not recommend running the rpc server on Windows because of reliability issues. A Docker container may provide a useful workaround, if required. For more details, see the [`dbt-rpc` repository](https://github.com/dbt-labs/dbt-rpc) source code. **Running the server:** ```text $ dbt-rpc serve Running with dbt=1.5.0 16:34:31 | Concurrency: 8 threads (target='dev') 16:34:31 | 16:34:31 | Done. Serving RPC server at 0.0.0.0:8580 Send requests to http://localhost:8580/jsonrpc ``` **Configuring the server** * `--host`: Specify the host to listen on (default=`0.0.0.0`) * `--port`: Specify the port to listen on (default=`8580`) **Submitting queries to the server:** The rpc server expects requests in the following format: rpc-spec.json ```json { "jsonrpc": "2.0", "method": "{ a valid rpc server command }", "id": "{ a unique identifier for this query }", "params": { "timeout": { timeout for the query in seconds, optional }, } } ``` #### Built-in Methods ##### status The `status` method will return the status of the rpc server. This method response includes a high-level status, like `ready`, `compiling`, or `error`, as well as the set of logs accumulated during the initial compilation of the project. When the rpc server is in the `compiling` or `error` state, only built-in methods of the RPC server will be accepted. **Example request** ```json { "jsonrpc": "2.0", "method": "status", "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d" } ``` **Example response** ```json { "result": { "status": "ready", "error": null, "logs": [..], "timestamp": "2019-10-07T16:30:09.875534Z", "pid": 76715 }, "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "jsonrpc": "2.0" } ``` ##### poll The `poll` endpoint will return the status, logs, and results (if available) for a running or completed task. The `poll` method requires a `request_token` parameter which indicates the task to poll a response for. The `request_token` is returned in the response of dbt tasks like `compile`, `run` and `test`. **Parameters**: * `request_token`: The token to poll responses for * `logs`: A boolean flag indicating if logs should be returned in the response (default=false) * `logs_start`: The zero-indexed log line to fetch logs from (default=0) **Example request** ```json { "jsonrpc": "2.0", "method": "poll", "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "params": { "request_token": "f86926fa-6535-4891-8d24-2cfc65d2a347", "logs": true, "logs_start": 0 } } ``` **Example Response** ```json { "result": { "results": [], "generated_at": "2019-10-11T18:25:22.477203Z", "elapsed_time": 0.8381369113922119, "logs": [], "tags": { "command": "run --select my_model", "branch": "abc123" }, "status": "success" }, "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "jsonrpc": "2.0" } ``` ##### ps The `ps` methods lists running and completed processes executed by the RPC server. **Parameters** * `completed`: If true, also return completed tasks (default=false) **Example request:** ```json { "jsonrpc": "2.0", "method": "ps", "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "params": { "completed": true } } ``` **Example response:** ```json { "result": { "rows": [ { "task_id": "561d4a02-18a9-40d1-9f01-cd875c3ec56d", "request_id": "3db9a2fe-9a39-41ef-828c-25e04dd6b07d", "request_source": "127.0.0.1", "method": "run", "state": "success", "start": "2019-10-07T17:09:49.865976Z", "end": null, "elapsed": 1.107261, "timeout": null, "tags": { "command": "run --select my_model", "branch": "feature/add-models" } } ] }, "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "jsonrpc": "2.0" } ``` ##### kill The `kill` method will terminate a running task. You can find a `task_id` for a running task either in the original response which invoked that task, or in the results of the `ps` method. **Example request** ```json { "jsonrpc": "2.0", "method": "kill", "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "params": { "task_id": "{ the task id to terminate }" } } ``` #### Running dbt projects The following methods make it possible to run dbt projects via the RPC server. ##### Common parameters All RPC requests accept the following parameters in addition to the parameters listed: * `timeout`: The max amount of time to wait before cancelling the request. * `task_tags`: Arbitrary key/value pairs to attach to this task. These tags will be returned in the output of the `poll` and `ps` methods (optional). ##### Running a task with CLI syntax **Parameters:** * `cli`: A dbt command (eg. `run --select abc+ --exclude +def`) to run (required) ```json { "jsonrpc": "2.0", "method": "cli_args", "id": "<request id>", "params": { "cli": "run --select abc+ --exclude +def", "task_tags": { "branch": "feature/my-branch", "commit": "c0ff33b01" } } } ``` Several of the following request types accept these additional parameters: * `threads`: The number of [threads](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads) to use when compiling (optional) * `select`: The space-delimited set of resources to execute (optional). (`models` is also supported on some request types for backwards compatibility.) * `selector`: The name of a predefined [YAML selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) that defines the set of resources to execute (optional) * `exclude`: The space-delimited set of resources to exclude from compiling, running, testing, seeding, or snapshotting (optional) * `state`: The filepath of artifacts to use when establishing [state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection) (optional) ##### Compile a project ([docs](https://docs.getdbt.com/reference/commands/compile.md)) ```json { "jsonrpc": "2.0", "method": "compile", "id": "<request id>", "params": { "threads": "<int> (optional)", "select": "<str> (optional)", "exclude": "<str> (optional)", "selector": "<str> (optional)", "state": "<str> (optional)" } } ``` ##### Run models ([docs](https://docs.getdbt.com/reference/commands/run.md)) **Additional parameters:** * `defer`: Whether to defer references to upstream, unselected resources (optional, requires `state`) ```json { "jsonrpc": "2.0", "method": "run", "id": "<request id>", "params": { "threads": "<int> (optional)", "select": "<str> (optional)", "exclude": "<str> (optional)", "selector": "<str> (optional)", "state": "<str> (optional)", "defer": "<bool> (optional)" } } ``` ##### Run tests ([docs](https://docs.getdbt.com/reference/commands/test.md)) **Additional parameters:** * `data`: If True, run data tests (optional, default=true) * `schema`: If True, run schema tests (optional, default=true) ```json { "jsonrpc": "2.0", "method": "test", "id": "<request id>", "params": { "threads": "<int> (optional)", "select": "<str> (optional)", "exclude": "<str> (optional)", "selector": "<str> (optional)", "state": "<str> (optional)", "data": "<bool> (optional)", "schema": "<bool> (optional)" } } ``` ##### Run seeds ([docs](https://docs.getdbt.com/reference/commands/seed.md)) **Parameters:** * `show`: If True, show a sample of the seeded data in the response (optional, default=false) ```json { "jsonrpc": "2.0", "method": "seed", "id": "<request id>", "params": { "threads": "<int> (optional)", "select": "<str> (optional)", "exclude": "<str> (optional)", "selector": "<str> (optional)", "show": "<bool> (optional)", "state": "<str> (optional)" } } ``` ##### Run snapshots ([docs](https://docs.getdbt.com/docs/build/snapshots.md)) ```json { "jsonrpc": "2.0", "method": "snapshot", "id": "<request id>", "params": { "threads": "<int> (optional)", "select": "<str> (optional)", "exclude": "<str> (optional)", "selector": "<str> (optional)", "state": "<str> (optional)" } } ``` ##### Build ([docs](https://docs.getdbt.com/reference/commands/build.md)) ```json { "jsonrpc": "2.0", "method": "build", "id": "<request id>", "params": { "threads": "<int> (optional)", "select": "<str> (optional)", "exclude": "<str> (optional)", "selector": "<str> (optional)", "state": "<str> (optional)", "defer": "<str> (optional)" } } ``` ##### List project resources ([docs](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate)) **Additional parameters:** * `resource_types`: Filter selected resources by type * `output_keys`: Specify which node properties to include in output ```json { "jsonrpc": "2.0", "method": "ls", "id": "<request id>", "params": { "select": "<str> (optional)", "exclude": "<str> (optional)", "selector": "<str> (optional)", "resource_types": ["<list> (optional)"], "output_keys": ["<list> (optional)"], } } ``` ##### Generate docs ([docs](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate)) **Additional parameters:** * `compile`: If True, compile the project before generating a catalog (optional, default=false) ```json { "jsonrpc": "2.0", "method": "docs.generate", "id": "<request id>", "params": { "compile": "<bool> (optional)", "state": "<str> (optional)" } } ``` #### Compiling and running SQL statements ##### Compiling a query This query compiles the SQL `select {{ 1 + 1 }} as id` (base64-encoded) against the rpc server: rpc-spec.json ```json { "jsonrpc": "2.0", "method": "compile_sql", "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "params": { "timeout": 60, "sql": "c2VsZWN0IHt7IDEgKyAxIH19IGFzIGlk", "name": "my_first_query" } } ``` The resulting response will include a key called `compiled_sql` with a value of `'select 2'`. ##### Executing a query This query executes the SQL `select {{ 1 + 1 }} as id` (bas64-encoded) against the rpc server: rpc-run.json ```json { "jsonrpc": "2.0", "method": "run_sql", "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d", "params": { "timeout": 60, "sql": "c2VsZWN0IHt7IDEgKyAxIH19IGFzIGlk", "name": "my_first_query" } } ``` The resulting response will include a key called `table` with a value of `{'column_names': ['?column?'], 'rows': [[2.0]]}` #### Reloading the RPC Server When the dbt RPC Server starts, it will load the dbt project into memory using the files present on disk at startup. If the files in the dbt project should change (either during development or in a deployment), the dbt RPC Server can be updated live without cycling the server process. To reload the files present on disk, send a "hangup" signal to the running server process using the Process ID (pid) of the running process. ##### Finding the server PID To find the server PID, either fetch the `.result.pid` value from the `status` method response on the server, or use `ps`: ```text # Find the server PID using `ps`: ps aux | grep 'dbt-rpc serve' | grep -v grep ``` After finding the PID for the process (eg. 12345), send a signal to the running server using the `kill` command: ```text kill -HUP 12345 ``` When the server receives the HUP (hangup) signal, it will re-parse the files on disk and use the updated project code when handling subsequent requests. --- ### About dbt run command #### Overview The `dbt run` command only applies to models. It doesn't run tests, snapshots, seeds, or other resource types. To run those commands, use the appropriate dbt commands found in the [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) section — such as `dbt test`, `dbt snapshot`, or `dbt seed`. Alternatively, use `dbt build` with a [resource type selector](https://docs.getdbt.com/reference/node-selection/methods.md#resource_type). You can use the `dbt run` command when you want to build or rebuild models in your project. ##### How does `dbt run` work? * `dbt run` executes compiled SQL model files against the current `target` database. * dbt connects to the target database and runs the relevant SQL required to materialize all data models using the specified materialization strategies. * Models are run in the order defined by the dependency graph generated during compilation. Intelligent multi-threading is used to minimize execution time without violating dependencies. * Deploying new models frequently involves destroying prior versions of these models. In these cases, `dbt run` minimizes downtime by first building each model with a temporary name, then dropping and renaming within a single transaction (for adapters that support transactions). #### Refresh incremental models If you provide the `--full-refresh` flag to `dbt run`, dbt will treat incremental models as table models. This is useful when 1. The schema of an incremental model changes and you need to recreate it. 2. You want to reprocess the entirety of the incremental model because of new logic in the model code. bash ```shell dbt run --full-refresh ``` You can also supply the flag by its short name: `dbt run -f`. In the dbt compilation context, this flag will be available as [flags.FULL\_REFRESH](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md). Further, the `is_incremental()` macro will return `false` for *all* models in response when the `--full-refresh` flag is specified. models/example.sql ```sql select * from all_events -- if the table already exists and `--full-refresh` is -- not set, then only add new records. otherwise, select -- all records. {% if is_incremental() %} where collector_tstamp > ( select coalesce(max(max_tstamp), '0001-01-01') from {{ this }} ) {% endif %} ``` #### Running specific models dbt will also allow you select which specific models you'd like to materialize. This can be useful during special scenarios where you may prefer running a different set of models at various intervals. This can also be helpful when you may want to limit the tables materialized while you develop and test new models. For more information, see the [Model Selection Syntax Documentation](https://docs.getdbt.com/reference/node-selection/syntax.md). For more information on running parents or children of specific models, see the [Graph Operators Documentation](https://docs.getdbt.com/reference/node-selection/graph-operators.md). (Applies to dbt v2.0 and later) #### dbt Information Schema Use `--generate-info-schema` to write the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) to `target/info_schema/` in a versioned subdirectory (currently `v1/`). The Information Schema exposes your project's metadata as queryable SQL tables (similar to a database's `INFORMATION_SCHEMA`) so you can query models, sources, and more without parsing `manifest.json`. ```shell dbt run --generate-info-schema ``` To populate inferred model column types and column-level lineage, combine with [`--static-analysis strict`](https://docs.getdbt.com/docs/build/about-static-analysis.md). Without it, `dbt.node_columns` contains only YAML-declared columns and `dbt.column_lineage` contains no lineage. ```shell dbt run --generate-info-schema --static-analysis strict ``` #### Treat warnings as errors See [global configs](https://docs.getdbt.com/reference/global-configs/warnings.md) #### Failing fast See [global configs](https://docs.getdbt.com/reference/global-configs/failing-fast.md) #### Enable or Disable Colorized Logs See [global configs](https://docs.getdbt.com/reference/global-configs/print-output.md#print-color) #### The `--empty` flag The `run` command supports the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. #### Status codes When calling the [list\_runs api](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs), you will get a status code for each run returned. The available run status codes are as follows: * Queued = 1 * Starting = 2 * Running = 3 * Success = 10 * Error = 20 * Canceled = 30 * Skipped = 40 --- ### About dbt run-operation command #### Overview The `dbt run-operation` command is used to invoke a macro or execute a SQL or Jinja string directly against the target database. For usage information on macro-based operations, refer to [operations](https://docs.getdbt.com/docs/build/hooks-operations.md#about-operations). (Applies to dbt v1.12 and later) When a macro is invoked with `dbt run-operation`, dbt doesn't enforce model [access](https://docs.getdbt.com/reference/resource-configs/access.md) or [group](https://docs.getdbt.com/reference/resource-configs/group.md) controls, so it can use `ref()` to reference all models (including `private` and `protected` ones). For example, a macro can reference a private model: ```jinja {% macro example_macro() %} {{ ref('my_private_model') }} {% endmacro %} ``` You can then run that macro using `dbt run-operation`: ```bash dbt run-operation example_macro ``` #### Usage ```text $ dbt run-operation [macro] --args '{args}' $ dbt run-operation --sql '{sql}' [macro] Specify the macro to invoke. dbt will call this macro with the supplied arguments and then exit. Required unless --sql is provided. Cannot be used with --sql. --args ARGS Supply arguments to the macro. This dictionary will be mapped to the keyword arguments defined in the selected macro. This argument should be a YAML string, eg. '{my_variable: my_value}'. Cannot be used with --sql. --sql SQL Execute a SQL or Jinja string directly against the target database, without requiring a macro definition. Cannot be combined with a macro name or --args. Available in dbt v1.12+. ``` #### Using the `--sql` flag Starting dbt v1.12, you can use the `--sql` flag to execute ad hoc database statements directly against your warehouse, without defining a macro. This flag is useful for one-off operations like dropping or altering a table, applying grants, or running a data fix. The statement runs through the full Jinja compilation pipeline, so you have access to `ref()`, `source()`, `var()`, `target`, and all other context variables. When your SQL contains no Jinja, dbt skips manifest compilation entirely, making execution faster. For example: ```bash # Drop a table in the warehouse dbt run-operation --sql "DROP TABLE IF EXISTS my_schema.old_table" # Use Jinja context variables in the statement dbt run-operation --sql "GRANT SELECT ON {{ target.schema }}.my_table TO reporter" ``` dbt prints status lines to the terminal as the operation runs: ```text 1 of 1 START executing inline_query ........... [RUN] 1 of 1 OK executed inline_query ............... [SELECT 1 in 0.05s] ``` The operation is named `inline_query` in logs. If the operation fails, the status line shows `ERROR` instead of `OK`. Note that you cannot combine the `--sql` flag with a macro name or `--args`. ##### When to use `--sql` instead of a macro `--sql` is useful for quick, one-off operations that don't warrant a macro file (for example, granting permissions, dropping a table, or running a data fix). If you need to reuse the operation across environments or share it with your team, write a [macro](https://docs.getdbt.com/docs/build/jinja-macros.md) instead so it's version-controlled and testable. #### Command line examples * Invoke a macro: ```bash # Pass a single argument to a macro dbt run-operation grant_select --args '{role: reporter}' # Pass multiple arguments to a macro dbt run-operation clean_stale_models --args '{days: 7, dry_run: True}' ``` * Use the `--sql` flag: ```bash # Execute DDL dbt run-operation --sql "CREATE TABLE my_schema.my_table (id int)" # Grant select on a dbt model using ref() dbt run-operation --sql "GRANT SELECT ON {{ ref('my_model') }} TO reporter" ``` --- ### About dbt seed command The `dbt seed` command loads static CSV files from your project’s `seed-paths` into your data warehouse as tables. Use seeds for small, version-controlled reference datasets you want to keep alongside your project, such as country codes, region mappings, or a list of business-defined categories. After you load seeds, you can reference the resulting tables in downstream models, and rerun `dbt seed` when the CSVs change. Seeds are configured in the `dbt_project.yml` file, where you can define options like seed file directories and column interpretation. The `dbt seed` command then loads those CSVs into your warehouse so you can reference them in downstream models. #### Overview This section covers common `dbt seed` results (artifacts and CLI output) and options you might use when running seeds. Use artifacts when you want to inspect or troubleshoot what happened during execution: * Artifacts: Running `dbt seed` produces dbt execution artifacts, including `run_results.json` for executed seeds. For details on what’s included and how to interpret it, refer to [Run results JSON file](https://docs.getdbt.com/reference/artifacts/run-results-json.md). <br /> Use resource selection when you want to run only a subset of seeds: * Selecting resources: For advanced selection patterns (tags, paths, graph operators, and more), read [Syntax overview](https://docs.getdbt.com/reference/node-selection/syntax.md). <br /> Use global flags when you want to control command behavior, logging, and runtime settings: * Flags: `dbt seed` supports dbt global flags. For the full list, refer to [Command line options](https://docs.getdbt.com/reference/global-configs/command-line-options.md) and [Available flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). #### Selecting seeds You can run specific seeds using the `--select` flag when executing the `dbt seed` command. For example: ```shell dbt seed --select "country_codes" Found 2 models, 3 tests, 0 archives, 0 analyses, 53 macros, 0 operations, 2 seed files 14:46:15 | Concurrency: 1 threads (target='dev') 14:46:15 | 14:46:15 | 1 of 1 START seed file analytics.country_codes........................... [RUN] 14:46:15 | 1 of 1 OK loaded seed file analytics.country_codes....................... [INSERT 3 in 0.01s] 14:46:16 | 14:46:16 | Finished running 1 seed in 0.14s. ``` Examples of common `dbt seed` invocations: ```shell # Run all seeds dbt seed # Run one seed dbt seed --select "country_codes" # Rebuild selected seeds from scratch dbt seed --select "country_codes state_codes" --full-refresh ``` #### The --full-refresh flag Use `--full-refresh` to force a full reload of seed data (rather than an incremental update) when you need to rebuild seed tables from scratch. This is useful when: * You want to force a clean reload of seed data: ```shell dbt seed --full-refresh ``` * You have changed the seed structure (for example, column names or types): ```shell dbt seed --select "country_codes" --full-refresh ``` * You need consistent behavior across environments after a seed change: ```shell dbt seed --select "country_codes state_codes" --full-refresh ``` For information about configuring seeds (for example, column types and quoting behavior), refer to [Seed configurations](https://docs.getdbt.com/reference/seed-configs.md). (Applies to dbt v1.12 and later) #### The --empty flag Use `--empty` to create seed tables with the correct schema but without loading any data. dbt infers column names and types from the CSV file but inserts zero rows. This is useful in development or CI environments where downstream models and unit tests need the table structure to exist without the overhead of loading full datasets. For example: * Build schema-only seed tables so downstream models and unit tests can compile and run: ```shell dbt seed --empty ``` * Target a specific seed when you only need one table's structure: ```shell dbt seed --select "country_codes" --empty ``` For more information, refer to [The empty flag](https://docs.getdbt.com/docs/build/empty-flag.md). #### Related docs * [Seed configurations](https://docs.getdbt.com/reference/seed-configs.md) * [Add Seeds to your DAG](https://docs.getdbt.com/docs/build/seeds.md) * [Syntax overview](https://docs.getdbt.com/reference/node-selection/syntax.md) * [`dbt build` command](https://docs.getdbt.com/reference/commands/build.md) --- ### About dbt show command Use `dbt show` to: * Compile the dbt-SQL definition of a single `model`, `test`, `analysis`, or an arbitrary dbt-SQL query passed `--inline` * `dbt show` does not support [Python (dbt-py)](https://docs.getdbt.com/docs/build/python-models.md) models. * Only selecting a single node is supported. [Selector methods](https://docs.getdbt.com/reference/node-selection/methods.md), [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md), and other methods that select multiple nodes will not be utilized. * Run that query against the data warehouse * Preview the results in the terminal (Applies to dbt v2.0 and later) * Query the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) directly from the CLI using the `--info` flag, without connecting to your warehouse #### How it works By default, `dbt show` will display the first 5 rows from the query result. This can be customized by passing the `limit` or the `inline` flags , where `n` is the number of rows to display. If previewing a model, dbt will always compile and run the compiled query from source. It will not select from the already-materialized database relation, even if you've just run the model. (We may support that in the future; if you're interested, upvote or comment on [dbt-labs/dbt#7391](https://github.com/dbt-labs/dbt/issues/7391).) ###### `limit` flag * The `--limit` flag modifies the underlying SQL and not just the number of rows displayed. By using the `--limit n` flag, it means `n` is the number of rows to display and retrieved from the data warehouse. * This means dbt wraps your model's query in a subquery or CTE and applies a SQL `limit n` clause so that your data warehouse only processes and returns that number of rows, making it significantly faster for large datasets. ###### `inline` flag * The results of the preview query are only included in dbt's logs and displayed in the terminal and aren't materialized in the data warehouse or stored in any dbt file, except if you use `dbt show --inline`. * The `--inline` flags enables you to run ad-hoc SQL, which means dbt can't ensure the query doesn't modify the data warehouse. To ensure no changes are made, use a profile or role with read-only permissions, which are managed directly in your data warehouse. For example: `dbt show --inline "select * from my_table" --profile my-read-only-profile`. (Applies to dbt v2.0 and later) ###### `--info` flag * The `--info <view>` flag queries the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) directly from the CLI. It reads from the intermediate views and doesn't connect to your warehouse. * It is equivalent to `--inline "select * from {{ info_schema('<view>') }}"`. ```shell dbt show --info models dbt show --info models --limit 20 ``` You can also write custom SQL against the Information Schema using `--inline`: ```shell dbt show --inline "select name from {{ info_schema('models') }} order by name" ``` ##### `--output json` flag * The `--output json` flag returns `dbt show` results in JSON format instead of the default human-readable output, which is helpful for scripting and automation. * If you want the full terminal output (including logs) to be machine-readable JSON, you can also set `--log-format json`. #### Example ```text dbt show --select "model_name.sql" ``` or ```text dbt show --inline "select * from {{ ref('model_name') }}" ``` The following is an example of `dbt show` output for a model named `stg_orders`: ```bash dbt show --select "stg_orders" 21:17:38 Running with dbt=1.5.0-b5 21:17:38 Found 5 models, 20 tests, 0 snapshots, 0 analyses, 425 macros, 0 operations, 3 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups 21:17:38 21:17:38 Concurrency: 24 threads (target='dev') 21:17:38 21:17:38 Previewing node 'stg_orders' : | order_id | customer_id | order_date | status | |----------+-------------+------------+-------- | | 1 | 1 | 2023-01-01 | returned | | 2 | 3 | 2023-01-02 | completed | | 3 | 94 | 2023-01-03 | completed | | 4 | 50 | 2023-01-04 | completed | | 5 | 64 | 2023-01-05 | completed | ``` For example, if you've just built a model that has a failing test, you can quickly preview the test failures right in the terminal, to find values of `id` that are duplicated: ```bash $ dbt build -s "my_model_with_duplicates" 13:22:47 .0 ... 13:22:48 Completed with 1 error and 0 warnings: 13:22:48 13:22:48 Failure in test unique_my_model_with_duplicates (models/schema.yml) 13:22:48 Got 1 result, configured to fail if not 0 13:22:48 13:22:48 compiled code at target/compiled/my_dbt_project/models/schema.yml/unique_my_model_with_duplicates_id.sql 13:22:48 13:22:48 Done. PASS=1 WARN=0 ERROR=1 SKIP=0 TOTAL=2 $ dbt show -s "unique_my_model_with_duplicates_id" 13:22:53 Running with dbt=1.5.0 13:22:53 Found 4 models, 2 tests, 0 snapshots, 0 analyses, 309 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups 13:22:53 13:22:53 Concurrency: 5 threads (target='dev') 13:22:53 13:22:53 Previewing node 'unique_my_model_with_duplicates_id': | unique_field | n_records | | ------------ | --------- | | 1 | 2 | ``` ```sh dbt show --inline "select 1" --output json --log-format json ``` Gives you a result like this: ```json { "data": { "is_inline": true, "node_name": "inline_query", "output_format": "json", "preview": "[{\"ID\": 1}]", "quiet": false, "unique_id": "sql_operation.jaffle_shop.inline_query" }, "info": { "code": "Q041", "level": "info", "msg": "{\n \"show\": [\n {\n \"ID\": 1\n }\n ]\n}\n", "name": "ShowNode", "thread": "MainThread" } } ``` --- ### About dbt snapshot command The `dbt snapshot` command executes the [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) defined in your project. Snapshots record changes to your source data over time by implementing [type-2 Slowly Changing Dimensions](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row). Run `dbt snapshot` on a schedule (for example, daily) to capture changes in your source tables. Define snapshots in YAML with a strategy and `unique_key`; refer to [Snapshot configurations](https://docs.getdbt.com/reference/snapshot-configs.md) for details on how to set them up. You can also run snapshots as part of [dbt build](https://docs.getdbt.com/reference/commands/build.md). dbt looks for snapshots in the directories listed in `snapshot-paths` in your `dbt_project.yml` file. By default, dbt uses the `snapshots/` directory. You can specify multiple paths if you organize snapshots in more than one folder. #### Usage To view the full list of supported options in your terminal, run: ```shell dbt snapshot --help ``` Use `--select` or `--exclude` to choose which snapshots run. For selection syntax, refer to [Node selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). `dbt snapshot` also supports common command-line options, such as `--target` and `--threads`. For flag details (including logging options), refer to [About flags (global configs)](https://docs.getdbt.com/reference/global-configs/about-global-configs.md). Use `--vars` when your snapshot SQL references values with the `var()` function. For syntax, precedence, and more examples, refer to [Defining variables on the command line](https://docs.getdbt.com/docs/build/project-variables.md#defining-variables-on-the-command-line). ```shell dbt snapshot --select my_snapshot --vars '{"cutoff_date": "2026-01-01"}' ``` Snapshots ignore full refresh Snapshots ignore both the `full_refresh` config and the `--full-refresh` flag. A command such as `dbt build --full-refresh` or `dbt snapshot --full-refresh` that includes a snapshot node runs the snapshot as normal — it won't drop or recreate the snapshot table, so existing snapshot history is preserved. Compiled SQL for snapshots Starting dbt v1.12, you can inspect the SQL generated for this snapshot by running [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) or `dbt compile --select orders_snapshot`. Open the compiled SQL in `target/compiled/` to inspect or debug the generated queries. Each snapshot is compiled into its own SQL file, even if multiple snapshots are defined in the same source file. --- ### About dbt source command (Applies to dbt v2.0 and later) The `dbt source` command provides one legacy subcommand, `dbt source freshness`, which evaluates sources only. In v2, use the top-level [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) command to evaluate both sources and models against their configured [freshness thresholds](https://docs.getdbt.com/reference/resource-configs/freshness.md) in a single command. ##### Freshness (Applies to dbt v2.0 and later) If your dbt project is [configured with sources](https://docs.getdbt.com/docs/build/sources.md), dbt determines the "freshness" of each source table from the `freshness` config specified for that source. If a table is stale, dbt reports a warning or error accordingly and exits with a nonzero exit code. To evaluate source freshness, run [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md). ##### Configure freshness The example below shows how to configure source freshness in dbt. Refer to [Declaring source freshness](https://docs.getdbt.com/docs/build/sources.md#declaring-source-freshness) for more information. models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw config: freshness: # changed to config in v1.9 warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at # changed to config in v1.10 tables: - name: customers - name: orders config: freshness: warn_after: {count: 6, period: hour} error_after: {count: 12, period: hour} filter: datediff('day', _etl_loaded_at, current_timestamp) < 2 - name: product_skus config: freshness: null ``` This helps to monitor the data pipeline health. You can also configure source freshness in the **Execution settings** section in your dbt platform job **Settings** page. For more information, refer to [Enabling source freshness checks](https://docs.getdbt.com/docs/deploy/source-freshness.md#enabling-source-freshness-checks). (Applies to dbt v2.0 and later) ##### Evaluate specific sources By default, `dbt freshness` evaluates every source and model in your project that has a `freshness` config. To evaluate a subset of your sources, use the `--select` flag. ```bash # Evaluate freshness for all Snowplow tables: $ dbt freshness --select "source:snowplow" # Evaluate freshness for a particular source table: $ dbt freshness --select "source:snowplow.event" ``` (Applies to dbt v2.0 and later) ##### Freshness output When `dbt freshness` completes, dbt writes results for the evaluated sources and models to `target/freshness.json`. Each entry includes a `resource_type` field identifying whether the resource is a source or a model. For the full schema, refer to [`freshness.json`](https://docs.getdbt.com/reference/artifacts/freshness-json.md). Whenever sources are included in the run, dbt also writes `target/sources.json` for backward compatibility. It contains sources only, with no `resource_type` field. For the full schema, refer to [`sources.json`](https://docs.getdbt.com/reference/artifacts/sources-json.md). (Applies to dbt v2.0 and later) ##### Using freshness results Freshness results help you understand whether a source or model is in a delayed state, and how freshness trends over time. You can evaluate freshness manually at any time with [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md). We also recommend running it on a schedule and storing the results at regular intervals. These longitudinal results make it possible to be alerted when source data freshness SLAs are violated, and to understand the trend of freshness over time. dbt makes it easy to run freshness checks on a schedule, and provides a dashboard out of the box indicating the state of freshness for the sources defined in your project. For more information, refer to [source data freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness). --- ### About dbt system command `dbt system` provides commands for managing your dbt installation. v2 is a single compiled binary with no dependencies on other libraries. Once installed, `dbt system` lets you update, uninstall, and manage drivers directly without pip, brew, or a Python environment. `dbt system update` and `dbt system uninstall` are only supported for [CDN installations](https://docs.getdbt.com/docs/local/install-dbt.md) of dbt v2. If you installed dbt v2 via Pip, Homebrew, or Winget, you'll need to update or uninstall dbt from those tools. dbt v1 has no equivalent. #### Usage ```shell dbt system <subcommand> [flags] ``` #### Subcommands | Subcommand | Description | | ---------------------------- | ------------------------------------------------- | | `dbt system update` | Update dbt to the latest (or a specified) version | | `dbt system uninstall` | Remove dbt v2 from your system | | `dbt system install-drivers` | Pre-install all supported ADBC adapter drivers | #### dbt system update `dbt system update` updates the CLI binary in place. By default, it updates to the latest stable release. You can also target a named channel or a specific version number. ```shell dbt system update [flags] ``` ##### Options | Flag | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `--version VERSION` | Update to a specific version (for example, `2.0.1`) or a named channel (`canary`, `dev`). Defaults to the latest stable release. | | `--package PACKAGE` | Which package to update. Accepts `dbt` or `all`. Defaults to `dbt`. | ##### Version channels | Channel | Description | | -------- | ----------------------------------------------------------------------- | | *(none)* | Latest stable release | | `canary` | Most recent preview build (under active testing and may be pulled back) | | `dev` | Development build (may be unstable) | ##### Examples Update to the latest stable release: ```shell dbt system update ``` Update to a specific version: ```shell dbt system update --version 2.0.1 ``` Update to the canary channel: ```shell dbt system update --version canary ``` #### dbt system uninstall `dbt system uninstall` removes the CLI binary from your system. On macOS and Linux, it downloads and runs the official `uninstall.sh` script. On Windows, a PowerShell script handles removal after the current process exits (to release the file lock on the running executable). ```shell dbt system uninstall ``` This command takes no additional flags. #### dbt system install-drivers `dbt system install-drivers` downloads and caches all supported ADBC adapter driver libraries (`.dylib`, `.so`, `.dll`) for every supported data platform at once. Normally, dbt downloads drivers on first use when you run a command against a new data platform. Use `dbt system install-drivers` to pre-populate the cache (for example, before going offline or switching between projects that target different warehouses). ```shell dbt system install-drivers ``` This command takes no additional flags. Networking note `dbt system install-drivers` downloads files from `https://public.cdn.getdbt.com`. If your environment restricts outbound network access, make sure this endpoint is allowlisted. See [dbt v2 networking requirements](https://docs.getdbt.com/docs/local/dbt-networking-requirements.md) for the full list of endpoints. #### Related commands * [`dbt debug`](https://docs.getdbt.com/reference/commands/debug.md) — Test your dbt project and connection configuration. * [dbt v2 networking requirements](https://docs.getdbt.com/docs/local/dbt-networking-requirements.md) — Review outbound endpoints required by dbt v2. --- ### About dbt test command `dbt test` runs data tests defined on models, sources, snapshots, and seeds and unit tests defined on SQL models. It expects that you have already created those resources through the appropriate commands. The tests to run can be selected using the `--select` flag discussed [here](https://docs.getdbt.com/reference/node-selection/syntax.md). ```bash # run data and unit tests dbt test # run only data tests dbt test --select test_type:data # run only unit tests dbt test --select test_type:unit # run tests for one_specific_model dbt test --select "one_specific_model" # run tests for all models in package dbt test --select "some_package.*" # run only data tests defined singularly dbt test --select "test_type:singular" # run only data tests defined generically dbt test --select "test_type:generic" # run data tests limited to one_specific_model dbt test --select "one_specific_model,test_type:data" # run unit tests limited to one_specific_model dbt test --select "one_specific_model,test_type:unit" ``` For more information on writing tests, read the [data testing](https://docs.getdbt.com/docs/build/data-tests.md) and [unit testing](https://docs.getdbt.com/docs/build/unit-tests.md) documentation. --- ### About dbt_project.yml context The following context methods and variables are available when configuring resources in the `dbt_project.yml` file. This applies to the `models:`, `seeds:`, and `snapshots:` keys in the `dbt_project.yml` file. **Available context methods:** * [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) * [var](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) (Applies to dbt v1.12 and later) (*Note: Variables defined in `vars.yml` or with `--vars` are available*) **Available context variables:** * [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) * [builtins](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md) * [dbt\_version](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt_version.md) ##### Example configuration dbt\_project.yml ```yml name: my_project version: 1.0.0 # Configure the models in models/facts/ to be materialized as views # in development and tables in production/CI contexts models: my_project: facts: +materialized: "{{ 'view' if target.name == 'dev' else 'table' }}" ``` --- ### About dbt_version variable The `dbt_version` variable returns the installed version of dbt that is currently running. It can be used for debugging or auditing purposes. For details about release versioning, refer to [Versioning](https://docs.getdbt.com/reference/commands/version.md#versioning). #### Example usages macros/get\_version.sql ```sql {% macro get_version() %} {% do log("The installed version of dbt is: " ~ dbt_version, info=true) %} {% endmacro %} ``` ```text $ dbt run-operation get_version The installed version of dbt is 1.6.0 ``` --- ### About debug macro Requires dbt CLI The `debug()` macro is only available when using the self-hosted dbt CLI in a development environment. It's *not available* in dbt platform. Do not deploy code to production that uses the `debug` macro. If developing in dbt platform or using dbt v2, you can instead use: * [`{{ print() }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/print.md) - Print messages to both the log file and standard output (`stdout`). * [`{{ log() }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) - Structured logging that prints messages during Jinja rendering. The `{{ debug() }}` macro will open an iPython debugger in the context of a compiled dbt macro. The (Applies to dbt v1.11 and later) `DBT_ENGINE_MACRO_DEBUGGING` environment variable must be set to use the debugger. This function requires: * Interactive terminal access with iPython debugger (`ipdb`) installed. dbt v2 doesn't provide a iPython (ipdb) debugger since its built on Rust. It instead outputs a non-interactive snapshot of the MiniJinja render context in the compiled code. * Development environment running the self-hosted dbt CLI * (Applies to dbt v1.11 and later) `DBT_ENGINE_MACRO_DEBUGGING` environment variable set #### Usage my\_macro.sql ```text {% macro my_macro() %} {% set something_complex = my_complicated_macro() %} {{ debug() }} {% endmacro %} ``` When dbt hits the `debug()` line, you'll see something like: (Applies to dbt v1.11 and later) ```shell $ DBT_ENGINE_MACRO_DEBUGGING=write dbt compile Running with dbt=1.0 > /var/folders/31/mrzqbbtd3rn4hmgbhrtkfyxm0000gn/T/dbt-macro-compiled-cxvhhgu7.py(14)root() 13 environment.call(context, (undefined(name='debug') if l_0_debug is missing else l_0_debug)), ---> 14 environment.call(context, (undefined(name='source') if l_0_source is missing else l_0_source), 'src', 'seedtable'), 15 ) ipdb> l 9,12 9 l_0_debug = resolve('debug') 10 l_0_source = resolve('source') 11 pass 12 yield '%s\nselect * from %s' % ( ``` --- ### About dispatch config dbt can extend functionality across [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) through a system of [multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch). Because SQL syntax, data types, and DDL/DML support vary across adapters, dbt can define and call generic functional macros, and then "dispatch" that macro to the appropriate implementation for the current adapter. #### Syntax **Args**: * `macro_name` \[required]: Name of macro to dispatch. Must be a string literal. * `macro_namespace` \[optional]: Namespace (package) of macro to dispatch. Must be a string literal. **Usage**: ```sql {% macro my_macro(arg1, arg2) -%} {{ return(adapter.dispatch('my_macro')(arg1, arg2)) }} {%- endmacro %} ``` dbt uses two criteria when searching for the right candidate macro: * Adapter prefix * Namespace (package) **Adapter prefix:** Adapter-specific macros are prefixed with the lowercase adapter name and two underscores. Given a macro named `my_macro`, dbt will look for: * Postgres: `postgres__my_macro` * Redshift: `redshift__my_macro` * Snowflake: `snowflake__my_macro` * BigQuery: `bigquery__my_macro` * OtherAdapter: `otheradapter__my_macro` * *default:* `default__my_macro` If dbt does not find an adapter-specific implementation, it will dispatch to the default implementation. **Namespace:** Generally, dbt will search for implementations in the root project and internal projects (e.g. `dbt`, `dbt_postgres`). If the `macro_namespace` argument is provided, it instead searches the specified namespace (package) for viable implementations. It is also possible to dynamically route namespace searching by defining a [`dispatch` project config](https://docs.getdbt.com/reference/project-configs/dispatch-config.md); see the examples below for details. #### Examples ##### A simple example Let's say I want to define a macro, `concat`, that compiles to the SQL function `concat()` as its default behavior. On Redshift and Snowflake, however, I want to use the `||` operator instead. macros/concat.sql ```sql {% macro concat(fields) -%} {{ return(adapter.dispatch('concat')(fields)) }} {%- endmacro %} {% macro default__concat(fields) -%} concat({{ fields|join(', ') }}) {%- endmacro %} {% macro redshift__concat(fields) %} {{ fields|join(' || ') }} {% endmacro %} {% macro snowflake__concat(fields) %} {{ fields|join(' || ') }} {% endmacro %} ``` The top `concat` macro follows a special, rigid formula: It is named with the macro's "primary name," `concat`, which is how the macro will be called elsewhere. It accepts one argument, named `fields`. This macro's *only* function is to dispatch—that is, look for and return—using the primary macro name (`concat`) as its search term. It also wants to pass through, to its eventual implementation, all the keyword arguments that were passed into it. In this case, there's only one argument, named `fields`. Below that macro, I've defined three possible implementations of the `concat` macro: one for Redshift, one for Snowflake, and one for use by default on all other adapters. Depending on the adapter I'm running against, one of these macros will be selected, it will be passed the specified arguments as inputs, it will operate on those arguments, and it will pass back the result to the original dispatching macro. ##### A more complex example I found an existing implementation of the `concat` macro in the dbt-utils package. However, I want to override its implementation of the `concat` macro on Redshift in particular. In all other cases—including the default implementation—I'm perfectly happy falling back to the implementations defined in `dbt_utils.concat`. macros/concat.sql ```sql {% macro concat(fields) -%} {{ return(adapter.dispatch('concat')(fields)) }} {%- endmacro %} {% macro default__concat(fields) -%} {{ return(dbt_utils.concat(fields)) }} {%- endmacro %} {% macro redshift__concat(fields) %} {% for field in fields %} nullif({{ field }},'') {{ ' || ' if not loop.last }} {% endfor %} {% endmacro %} ``` If I'm running on Redshift, dbt will use my version; if I'm running on any other database, the `concat()` macro will shell out to the version defined in `dbt_utils`. #### For package maintainers Dispatched macros from [packages](https://docs.getdbt.com/docs/build/packages.md) *must* provide the `macro_namespace` argument, as this declares the namespace (package) where it plans to search for candidates. Most often, this is the same as the name of your package, e.g. `dbt_utils`. (It is possible, if rarely desirable, to define a dispatched macro *not* in the `dbt_utils` package, and dispatch it into the `dbt_utils` namespace.) Here we have the definition of the `dbt_utils.concat` macro, which specifies both the `macro_name` and `macro_namespace` to dispatch: ```sql {% macro concat(fields) -%} {{ return(adapter.dispatch('concat', 'dbt_utils')(fields)) }} {%- endmacro %} ``` ##### Overriding package macros Following the second example above: Whenever I call my version of the `concat` macro in my own project, it will use my special null-handling version on Redshift. But the version of the `concat` macro *within* the dbt-utils package will not use my version. Why does this matter? Other macros in dbt-utils, such as `surrogate_key`, call the `dbt_utils.concat` macro directly. What if I want `dbt_utils.surrogate_key` to use *my* version of `concat` instead, including my custom logic on Redshift? As a user, I can accomplish this via a [project-level `dispatch` config](https://docs.getdbt.com/reference/project-configs/dispatch-config.md). When dbt goes to dispatch `dbt_utils.concat`, it knows from the `macro_namespace` argument to search in the `dbt_utils` namespace. The config below defines dynamic routing for that namespace, telling dbt to search through an ordered sequence of packages, instead of just the `dbt_utils` package. dbt\_project.yml ```yml dispatch: - macro_namespace: dbt_utils search_order: ['my_project', 'dbt_utils'] ``` Note that this config *must* be specified in the user's root `dbt_project.yml`. dbt will ignore any `dispatch` configs defined in the project files of installed packages. Adapter prefixes still matter: dbt will only ever look for implementations that are compatible with the current adapter. But dbt will prioritize package specificity over adapter specificity. If I call the `concat` macro while running on Postgres, with the config above, dbt will look for the following macros in order: 1. `my_project.postgres__concat` (not found) 2. `my_project.default__concat` (not found) 3. `dbt_utils.postgres__concat` (not found) 4. `dbt_utils.default__concat` (found! use this one) As someone installing a package, this functionality makes it possible for me to change the behavior of another, more complex macro (`dbt_utils.surrogate_key`) by reimplementing and overriding one of its modular components. As a package maintainer, this functionality enables users of my package to extend, reimplement, or override default behavior, without needing to fork the package's source code. ##### Overriding global macros tip Certain functions like [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md), and [`config`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md) can't be overridden with a package using the dispatch config. This is because `ref`, `source`, and `config` are context properties within dbt and are not dispatched as global macros. Refer to [this GitHub discussion](https://github.com/dbt-labs/dbt/issues/4491#issuecomment-994709916) for more context. I maintain an internal utility package at my organization, named `my_org_dbt_helpers`. I use this package to reimplement built-in dbt macros on behalf of all my dbt-using colleagues, who work across a number of dbt projects. My package can define custom versions of any dispatched global macro I choose, from `generate_schema_name` to `test_unique`. I can define a new default version of that macro (e.g. `default__generate_schema_name`), or custom versions for specific data warehouse adapters (e.g. `spark__generate_schema_name`). Each root project installing my package simply needs to include the [project-level `dispatch` config](https://docs.getdbt.com/reference/project-configs/dispatch-config.md) that searches my package ahead of `dbt` for the `dbt` global namespace: dbt\_project.yml ```yml dispatch: - macro_namespace: dbt search_order: ['my_project', 'my_org_dbt_helpers', 'dbt'] ``` ##### Managing different global overrides across packages You can override global behaviors in different ways for each project that is installed as a package. This holds true for all global macros: `generate_schema_name`, `create_table_as`, etc. When parsing or running a resource defined in a package, the definition of the global macro within that package takes precedence over the definition in the root project because it's more specific to those resources. By combining package-level overrides and `dispatch`, it is possible to achieve three different patterns: 1. **Package always wins** — As the developer of dbt models in a project that will be deployed elsewhere as a package, You want full control over the macros used to define & materialize my models. Your macros should always take precedence for your models, and there should not be any way to override them. * *Mechanism:* Each project/package fully overrides the macro by its name, for example, `generate_schema_name` or `create_table_as`. Do not use dispatch. 2. **Conditional application (root project wins)** — As the maintainer of one dbt project in a mesh of multiple, your team wants conditional application of these rules. When running your project standalone (in development), you want to apply custom behavior; but when installed as a package and deployed alongside several other projects (in production), you want the root-level project's rules to apply. * *Mechanism:* Each package implements its "local" override by registering a candidate for dispatch with an adapter prefix, for example, `default__generate_schema_name` or `default__create_table_as`. The root-level project can then register its own candidate for dispatch (`default__generate_schema_name`), winning the default search order or by explicitly overriding the macro by name (`generate_schema_name`). 3. **Same rules everywhere all the time** — As a member of the data platform team responsible for consistency across teams at your organization, you want to create a "macro package" that every team can install & use. * *Mechanism:* Create a standalone package of candidate macros only, for example, `default__generate_schema_name` or `default__create_table_as`. Add a [project-level `dispatch` configuration](https://docs.getdbt.com/reference/project-configs/dispatch-config.md) in every project's `dbt_project.yml`. #### For adapter plugin maintainers Most packages were initially designed to work on the four original dbt adapters. By using the `dispatch` macro and project config, it is possible to "shim" existing packages to work on other adapters, by way of third-party compatibility packages. For instance, if I want to use `dbt_utils.concat` on Apache Spark, I can install a compatibility package, spark-utils, alongside dbt-utils: packages.yml ```yml packages: - package: dbt-labs/dbt_utils version: ... - package: dbt-labs/spark_utils version: ... ``` I then include `spark_utils` in the search order for dispatched macros in the `dbt_utils` namespace. (I still include my own project first, just in case I want to reimplement any macros with my own custom logic.) dbt\_project.yml ```yml dispatch: - macro_namespace: dbt_utils search_order: ['my_project', 'spark_utils', 'dbt_utils'] ``` When dispatching `dbt_utils.concat`, dbt will search for: 1. `my_project.spark__concat` (not found) 2. `my_project.default__concat` (not found) 3. `spark_utils.spark__concat` (found! use this one) 4. `spark_utils.default__concat` 5. `dbt_utils.postgres__concat` 6. `dbt_utils.default__concat` As a compatibility package maintainer, I only need to reimplement the foundational building-block macros which encapsulate low-level syntactical differences. By reimplementing low-level macros, such as `spark__dateadd` and `spark__datediff`, the `spark_utils` package provides access to more complex macros (`dbt_utils.date_spine`) "for free." As a `dbt-spark` user, by installing `dbt_utils` and `spark_utils` together, I don't just get access to higher-level utility macros. I may even be able to install and use packages with no Spark-specific logic, and which have never been tested against Spark, so long as they rely on `dbt_utils` macros for cross-adapter compatibility. ##### Adapter inheritance Some adapters "inherit" from other adapters (e.g. `dbt-postgres` → `dbt-redshift`, and `dbt-spark` → `dbt-databricks`). If using a child adapter, dbt will include any parent adapter implementations in its search order, too. Instead of just looking for `redshift__` and falling back to `default__`, dbt will look for `redshift__`, `postgres__`, and `default__`, in that order. Child adapters tend to have very similar SQL syntax to their parents, so this allows them to skip reimplementing a macro that has already been reimplemented by the parent adapter. Following the example above with `dbt_utils.concat`, the full search order on Redshift is actually: 1. `my_project.redshift__concat` 2. `my_project.postgres__concat` 3. `my_project.default__concat` 4. `dbt_utils.redshift__concat` 5. `dbt_utils.postgres__concat` 6. `dbt_utils.default__concat` In rare cases, the child adapter may prefer the default implementation to its parent's adapter-specific implementation. In that case, the child adapter should define an adapter-specific macro that calls the default. For instance, the PostgreSQL syntax for adding dates ought to work on Redshift, too, but I may happen to prefer the simplicity of `dateadd`: ```sql {% macro dateadd(datepart, interval, from_date_or_timestamp) %} {{ return(adapter.dispatch('dateadd')(datepart, interval, from_date_or_timestamp)) }} {% endmacro %} {% macro default__dateadd(datepart, interval, from_date_or_timestamp) %} dateadd({{ datepart }}, {{ interval }}, {{ from_date_or_timestamp }}) {% endmacro %} {% macro postgres__dateadd(datepart, interval, from_date_or_timestamp) %} {{ from_date_or_timestamp }} + ((interval '1 {{ datepart }}') * ({{ interval }})) {% endmacro %} {# Use default syntax instead of postgres syntax #} {% macro redshift__dateadd(datepart, interval, from_date_or_timestamp) %} {{ return(default__dateadd(datepart, interval, from_date_or_timestamp) }} {% endmacro %} ``` #### FAQs \[Error] Could not find my\_project package If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain *any* macros, dbt will raise an error like: ```shell Compilation Error In dispatch: Could not find package 'my_project' ``` This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### About env_var function The `env_var` function can be used to incorporate environment variables from the system into your dbt project. You can use the `env_var` function in your `profiles.yml` file, the `dbt_project.yml` file, the `sources.yml` file, your `schema.yml` files, and in model `.sql` files. Essentially, `env_var` is available anywhere dbt processes Jinja code. When used in a `profiles.yml` file (to avoid putting credentials on a server), it can be used like this: profiles.yml ```yaml profile: target: prod outputs: prod: type: postgres host: 127.0.0.1 # IMPORTANT: Make sure to quote the entire Jinja string here user: "{{ env_var('DBT_USER') }}" password: "{{ env_var('DBT_PASSWORD') }}" .... ``` If the `DBT_USER` and `DBT_ENV_SECRET_PASSWORD` environment variables are present when dbt is invoked, dbt will use these variables in your connection configuration — for example, in `profiles.yml` when running locally, or in [connection profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) if you have a dbt platform project. If your project references environment variables that aren't set, dbt will raise a compilation error. (Applies to dbt v1.12 and later) ##### Using the `.env` file When running dbt locally (the [dbt CLI](https://docs.getdbt.com/docs/local/install-dbt.md), dbt VS Code extension, and dbt v1.12), dbt automatically loads environment variables from a `.env` file in your current working directory (where you run the dbt command). Shell environment variables take precedence over values in `.env` and will not be overridden by the file. Create a `.env` file (typically at the root of your dbt project) and define variables using `KEY=value` syntax. For example: .env ```bash DBT_USER=user DBT_PASSWORD=password DBT_SCHEMA=dbt_schema ``` Reference them in your `profiles.yml` using `env_var()`: \~/.dbt/profiles.yml ```yaml my_profile: target: dev outputs: dev: type: postgres host: localhost user: "{{ env_var('DBT_USER') }}" password: "{{ env_var('DBT_PASSWORD') }}" schema: "{{ env_var('DBT_SCHEMA') }}" port: 5432 threads: 4 ``` Here are some considerations when defining environment variables in the `.env` file: * The `.env` file provides a convenient way to set environment variables that work across both the CLI and the VS Code extension. * We recommend placing your `.env` file in the project root and running dbt commands from that location because the file is loaded *only* from your current working directory. It doesn't support the `--project-dir` flag or (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` environment variable, and dbt won't search your project root if you're running commands from a different directory location. * Add `.env` to your `.gitignore` file to prevent sensitive credentials from being committed to your repository. * Environment variables set directly in your shell (such as `export DBT_ENV_VAR=value`) take precedence over values defined in the `.env` file. ##### Converting env\_vars Environment variables are always strings. When using them for configurations that expect integers or booleans, you must explicitly convert the value to the correct type. Use a Jinja filter to convert the string to the correct type: * **Integers** — Convert the string to a number using the `int` or [`as_number`](https://docs.getdbt.com/reference/dbt-jinja-functions/as_number.md) filter to avoid errors like `'1' is not of type 'integer'`. For example, `"{{ env_var('DBT_THREADS') | int }}"` or `"{{ env_var('DB_PORT') | as_number }}"`. * **Booleans** — Convert the string to a boolean explicitly using the [`as_bool`](https://docs.getdbt.com/reference/dbt-jinja-functions/as_bool.md) filter. For example, `"{{ env_var('DBT_PERSIST_DOCS_RELATION', False) | as_bool }}"`. For boolean defaults, use capitalized `True` or `False`. Using lowercase `true` or `false` will be treated as a string and can result in unexpected results. For example, to disable [`persist_docs`](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) using environment variables: dbt\_project.yml ```yml +persist_docs: relation: "{{ env_var('DBT_PERSIST_DOCS_RELATION', False) | as_bool }}" columns: "{{ env_var('DBT_PERSIST_DOCS_COLUMNS', False) | as_bool }}" ``` Quoting, curly brackets, & you Be sure to quote the entire Jinja string. Otherwise, the YAML parser will be confused by the Jinja curly brackets. ##### Default values You can also provide a default value as a second argument: dbt\_project.yml ```yaml ... models: jaffle_shop: +materialized: "{{ env_var('DBT_MATERIALIZATION', 'view') }}" ``` This can be useful to avoid compilation errors when the environment variable isn't available. ##### Secrets For certain configurations, you can use "secret" env vars. Any env var named with the prefix `DBT_ENV_SECRET` will be: * Available for use in `profiles.yml` + `packages.yml`, via the same `env_var()` function * Disallowed everywhere else, including `dbt_project.yml` and model SQL, to prevent accidentally writing these secret values to the data warehouse or metadata artifacts * Scrubbed from dbt logs and replaced with `*****`, any time its value appears in those logs (even if the env var was not called directly) The primary use case of secret env vars is git access tokens for [private packages](https://docs.getdbt.com/docs/build/packages.md#private-packages). **Note:** When dbt is loading profile credentials and package configuration, secret env vars will be replaced with the string value of the environment variable. You cannot modify secrets using Jinja filters, including type-casting filters such as [`as_number`](https://docs.getdbt.com/reference/dbt-jinja-functions/as_number.md) or [`as_bool`](https://docs.getdbt.com/reference/dbt-jinja-functions/as_bool.md), or pass them as arguments into other Jinja macros. You can only use *one secret* per configuration: ```yml # works host: "{{ env_var('DBT_ENV_SECRET_HOST') }}" # does not work host: "www.{{ env_var('DBT_ENV_SECRET_HOST_DOMAIN') }}.com/{{ env_var('DBT_ENV_SECRET_HOST_PATH') }}" ``` ##### Custom metadata Any env var named with the prefix `DBT_ENV_CUSTOM_ENV_` will be included in two places, with its prefix-stripped name as the key: * [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md#common-metadata): `metadata` -> `env` * [events and structured logs](https://docs.getdbt.com/reference/events-logging.md#info-fields): `info` -> `extra` A dictionary of these prefixed env vars will also be available in a `dbt_metadata_envs` context variable: ```sql -- {{ dbt_metadata_envs }} select 1 as id ``` ```shell $ DBT_ENV_CUSTOM_ENV_MY_FAVORITE_COLOR=indigo DBT_ENV_CUSTOM_ENV_MY_FAVORITE_NUMBER=6 dbt compile ``` Compiles to: ```sql -- {'MY_FAVORITE_COLOR': 'indigo', 'MY_FAVORITE_NUMBER': '6'} select 1 as id ``` ##### dbt platform usage If you're using dbt platform, environment variables must be: * Prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`) * Uppercase * Case-sensitive When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must exactly match the variable defined in the dbt platform user interface. --- ### About exceptions namespace Use the `exceptions` namespace to raise your own warnings or errors from Jinja code in dbt models and macros during compilation and execution. This is useful when your macro or model needs to check for a problem and tell dbt how to respond. For example, you might want to raise a warning or error when: * a required input is missing * a value isn't supported * a dependency isn't available * a custom validation check fails These are errors or warnings that you choose to raise in your own Jinja code. They're different from the automatic dbt or Jinja errors, like syntax errors or undefined variables. If you're not writing custom Jinja code and are instead trying to fix an error from dbt, refer to [Debug errors](https://docs.getdbt.com/guides/debug-errors.md). When calling these functions during `dbt run` or `dbt run-operation`, wrap them in `{% if execute %}`. This ensures the function runs only when dbt executes your code, rather than when it parses your project. Learn more about the [`execute`](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) variable. #### raise\_compiler\_error The `exceptions.raise_compiler_error` method raises a compilation error with the provided message. This is typically only useful in macros or materializations when invalid arguments are provided by the calling model. Throwing an exception causes a model to fail, so use this method with care. **Example usage**: exceptions.sql ```sql {% if execute %} {% if number < 0 or number > 100 %} {{ exceptions.raise_compiler_error("Invalid `number`. Got: " ~ number) }} {% endif %} {% endif %} ``` #### raise\_fail\_fast\_error Use `exceptions.raise_fail_fast_error` to stop dbt immediately when it should not continue running. This is commonly used when dbt detects a configuration change and `on_configuration_change` is set to `fail`. The run fails with a `FailFast Error` and your custom message. **Example usage**: macros/materialized\_view.sql ```sql {% if on_configuration_change == 'fail' %} {{ exceptions.raise_fail_fast_error( "Configuration changes were identified and `on_configuration_change` was set to `fail` for `" ~ target_relation.render() ~ "`" ) }} {% endif %} ``` #### raise\_not\_implemented Use `exceptions.raise_not_implemented` to stop a dbt run when a macro or feature isn't supported by the current adapter. This is often used by adapter and package authors when a feature hasn't been built yet. The run fails with a `Runtime Error` and your custom message. **Example usage**: macros/get\_columns\_in\_relation.sql ```sql {% macro default__get_columns_in_relation(relation) -%} {{ exceptions.raise_not_implemented( 'get_columns_in_relation macro not implemented for adapter ' ~ adapter.type()) }} {% endmacro %} ``` #### warn Use the `exceptions.warn` method to raise a compiler warning with the provided message. The model will still be successful and treated as a PASS. By default, warnings will not cause dbt runs to fail. However: * If you use the `--warn-error` flag, all warnings will be promoted to errors. * To promote only Jinja warnings to errors (and leave other warnings alone), use `--warn-error-options`. For example, `--warn-error-options '{"error": ["JinjaLogWarning"]}'`. Learn more about [Warnings](https://docs.getdbt.com/reference/global-configs/warnings.md). **Example usage**: warn.sql ```sql {% if execute %} {% if number < 0 or number > 100 %} {% do exceptions.warn("Invalid `number`. Got: " ~ number) %} {% endif %} {% endif %} ``` --- ### About execute variable `execute` is a Jinja variable that returns True when dbt is in "execute" mode. When you execute a `dbt compile` or `dbt run` command, dbt: 1. Reads all of the files in your project and generates a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) comprised of models, tests, and other graph nodes present in your project. During this phase, dbt uses the [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) and [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) statements it finds to generate the DAG for your project. **No SQL is run during this phase**, and `execute == False`. 2. Compiles (and runs) each node (eg. building models, or running tests). **SQL is run during this phase**, and `execute == True`. Any Jinja that relies on a result being returned from the database will error during the parse phase. For example, this SQL will return an error: models/order\_payment\_methods.sql ```sql 1 {% set payment_method_query %} 2 select distinct 3 payment_method 4 from {{ ref('raw_payments') }} 5 order by 1 6 {% endset %} 7 8 {% set results = run_query(payment_method_query) %} 9 10 {# Return the first column #} 11 {% set payment_methods = results.columns[0].values() %} ``` The error returned by dbt will look as follows: ```text Encountered an error: Compilation Error in model order_payment_methods (models/order_payment_methods.sql) 'None' has no attribute 'table' ``` This is because line #11 in the earlier code example (`{% set payment_methods = results.columns[0].values() %}`) assumes that a table has been returned, when, during the parse phase, this query hasn't been run. To work around this, wrap any problematic Jinja in an `{% if execute %}` statement: models/order\_payment\_methods.sql ```sql {% set payment_method_query %} select distinct payment_method from {{ ref('raw_payments') }} order by 1 {% endset %} {% set results = run_query(payment_method_query) %} {% if execute %} {# Return the first column #} {% set payment_methods = results.columns[0].values() %} {% else %} {% set payment_methods = [] %} {% endif %} ``` `{% if execute %}` doesn't stop queries during `dbt compile` If `dbt compile` is slow, or you see unexpected queries in your warehouse's query history, check for `run_query` calls guarded only by `{% if execute %}`. That's because `execute` is `True` any time dbt compiles your models' SQL, not just when models are being built. To limit the query to commands that build models, also check [`flags.WHICH`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#flagswhich): ```jinja {% set results = none %} {% if execute and flags.WHICH in ['run', 'build'] %} {% set results = run_query(my_query) %} {% endif %} ``` Make sure the rest of your macro handles `results` being `none` when the query is skipped. #### Parsing vs execution Parsing in Jinja is when dbt: * Reads your project files. * Identifies [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) and [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md). * Identifies macro definitions. * Builds the dependency graph (DAG). It doesn't run any SQL — `execute == False`. Execution is when dbt actually runs SQL and builds models — `execute == True`. During execution, dbt: * Renders full Jinja templates into SQL. * Resolves all instances of `ref()` and `source()` to their corresponding table or view names. * Runs the SQL in your models during commands like ([`dbt run`](https://docs.getdbt.com/reference/commands/run.md)), ([`dbt test`](https://docs.getdbt.com/reference/commands/test.md)), \[`dbt seed`]\(/reference/commands/seed, or [`dbt snapshot`](https://docs.getdbt.com/reference/commands/snapshot.md). * Creates or updates tables/views in the warehouse. * Applies any materializations (incremental, table, view, ephemeral). `execute` impacts the values of `ref()` and `source()`, and won't work as expected inside of a [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md#usage). This is because in the initial parse of the project, dbt identifies every use of `ref()` and `source()` to build the DAG, but doesn’t resolve them to actual database identifiers. Instead, it replaces each with a placeholder value to ensure the SQL compiles cleanly during parsing. #### Examples Macros like [`log()`](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) and [`exceptions.warn()`](https://docs.getdbt.com/reference/dbt-jinja-functions/exceptions.md#warn) are still evaluated at parse time, during dbt's "first-pass" Jinja render to extract `ref`, `source` and `config`. As a result, dbt will also run any logging or warning messages during this process. Even though nothing is being executed yet, dbt still runs those log lines while parsing. This can be confusing — it looks like dbt is doing something real but it’s just parsing. ```text $ dbt run 15:42:01 Running with dbt=1.10.2 15:42:01 I'm running a query now. <------ this one is misleading!!!! no query is actually being run 15:42:01 Found 1 model, 0 tests, 0 snapshots, 0 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics 15:42:01 15:42:01 Concurrency: 8 threads (target='dev') 15:42:01 15:42:01 1 of 1 START table model analytics.my_model .................................. [RUN] 15:42:01 I'm running a query now 15:42:02 1 of 1 OK created table model analytics.my_model ............................. [OK in 0.36s] ``` ##### Logging fully-qualified relation names Let's assume you have a relation named `relation` obtained using something like `{% set relation = ref('my_model') %}` or `{% set relation = source('source_name', 'table_name') %}` — this will lead to unexpected or confusing behavior during parsing: ```jinja {%- if load_relation(relation) is none -%} {{ log("Relation is missing: " ~ relation, True) }} {% endif %} ``` To prevent this, add the `execute` flag to make sure the check only runs when dbt is actually running the code — not just when it's preparing it. Use the command `do exceptions.warn` to emit a warning during model execution without failing the run. ```jinja {%- if execute and load_relation(relation) is none -%} {% do exceptions.warn("Relation is missing: " ~ relation) %} {{ log("Relation is missing: " ~ relation, info=True) }} {%- endif -%} ``` --- ### About flags (global configs) In dbt, "flags" (also called "global configs" and often configured with [environment variables](https://docs.getdbt.com/reference/global-configs/environment-variable-configs.md)) are settings for fine-tuning *how* dbt runs your project. They differ from [resource-specific configs](https://docs.getdbt.com/reference/configs-and-properties.md) that tell dbt *what* to run. Flags control things like the visual output of logs, whether to treat specific warning messages as errors, or whether to "fail fast" after encountering the first error. Flags are "global" configs because they are available for all dbt commands and they can be set in multiple places. You can use flags with the dbt v2 or dbt v1 engine through the CLI during local development or in dbt platform. There is a significant overlap between dbt's flags and dbt's command line options, but there are differences: * Certain flags can only be set in [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md) and cannot be overridden for specific invocations by using CLI options. * If a CLI option is supported by specific commands, rather than supported by all commands ("global"), it is generally not considered to be a "flag". You can configure flags in `dbt_project.yml`, environment variables, and CLI options. For details, refer to [environment variable configs](https://docs.getdbt.com/reference/global-configs/environment-variable-configs.md). ##### Setting flags There are multiple ways of setting flags, which depend on the use case: * **[CLI options](https://docs.getdbt.com/reference/global-configs/command-line-options.md):** Define behavior specific to *this invocation*. Supported for all dbt commands. * **[Environment variables](https://docs.getdbt.com/reference/global-configs/environment-variable-configs.md):** Define different behavior in different runtime environments (development vs. production vs. [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md)), or different behavior for different users in development (based on personal preferences). * **[Project-level `flags` in `dbt_project.yml`](https://docs.getdbt.com/reference/global-configs/project-flags.md):** Define version-controlled defaults for everyone running this project. Also, opt in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) to manage your migration off legacy functionality. * **[User settings (`~/.dbt/user_settings.yml`)](https://docs.getdbt.com/reference/global-configs/user-settings.md):** Define personal preferences that apply across all projects on your machine. Written automatically by `dbt login`. The most specific setting "wins." CLI options take the highest precedence, followed by environment variables, then `dbt_project.yml`, and finally `user_settings.yml`. If you set the flag in none of those places, it will use the default value defined within dbt. Most flags can be set in all three places: ```yaml # dbt_project.yml flags: # set default for running this project -- anywhere, anytime, by anyone fail_fast: true ``` (Applies to dbt v1.11 and later) ```bash # set this environment variable to 'True' (bash syntax) export DBT_ENGINE_FAIL_FAST=1 dbt run ``` ```bash dbt run --fail-fast # set to True for this specific invocation dbt run --no-fail-fast # set to False ``` There are two categories of exceptions: 1. **Flags setting file paths:** Flags for file paths that are relevant to runtime execution (for example, `--log-path` or `--state`) cannot be set in `dbt_project.yml`. To override defaults, pass CLI options or set environment variables ((Applies to dbt v1.11 and later) `DBT_ENGINE_LOG_PATH` and `DBT_ENGINE_STATE`). Flags that tell dbt where to find project resources (for example, `model-paths`) are set in `dbt_project.yml`, but as a top-level key, outside the `flags` dictionary; these configs are expected to be fully static and never vary based on the command or execution environment. 2. **Opt-in flags:** Flags opting in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) can *only* be defined in `dbt_project.yml`. These are intended to be set in version control and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. ##### Accessing flags Custom user-defined logic, written in Jinja, can check the values of flags using [the `flags` context variable](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md). ```yaml # dbt_project.yml on-run-start: - '{{ log("I will stop at the first sign of trouble", info = true) if flags.FAIL_FAST }}' ``` #### Available flags Because the values of `flags` can differ across invocations, we strongly advise against using `flags` as an input to configurations or dependencies (`ref` + `source`) that dbt resolves [during parsing](https://docs.getdbt.com/reference/parsing.md#known-limitations). Use this table to compare all available flags and how to configure them across interfaces: * **dbt CLI**: Indicates whether the flag is supported in the [dbt platform-supported CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). * **Type / default**: Shows the accepted value type and default. * **In project**: Indicates whether you can set the flag in `dbt_project.yml`. * **Env var**: Shows the corresponding environment variable name, when available. In general, v1.10 and earlier use the `DBT_` prefix, while v1.11+ uses the `DBT_ENGINE_` prefix. * **CLI flags**: Lists command-line options for setting the flag for a specific invocation. (Applies to dbt v1.11 and later) | Flag | dbt CLI? | Type / default | In project? | Env var | CLI flags | | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | --------------------------------------------------------------------------------- | ----------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | [ai\_provider](https://docs.getdbt.com/docs/dbt-ai/package-skills.md#set-the-ai_provider-flag) (v2.0+) | ❌ | list<br />default: None | ✅ | `DBT_ENGINE_AI_PROVIDER` | `--ai-provider` | | [batch\_tests](https://docs.getdbt.com/reference/global-configs/batch-tests.md) (v2.0+) | ❌ | boolean<br />default: False | ✅ | `DBT_ENGINE_BATCH_TESTS` | `--batch-tests` | | [cache\_selected\_only](https://docs.getdbt.com/reference/global-configs/cache.md) | ✅ | boolean<br />default: False | ✅ | `DBT_ENGINE_CACHE_SELECTED_ONLY` | `--cache-selected-only`<br />`--no-cache-selected-only` | | [clean\_project\_files\_only](https://docs.getdbt.com/reference/commands/clean.md#--clean-project-files-only) | ❌ | boolean<br />default: True | ❌ | `DBT_ENGINE_CLEAN_PROJECT_FILES_ONLY` | `--clean-project-files-only`<br />`--no-clean-project-files-only` | | [debug](https://docs.getdbt.com/reference/global-configs/logs.md#debug-level-logging) | ✅ | boolean<br />default: False | ✅ | `DBT_ENGINE_DEBUG` | `--debug`<br />`--no-debug` | | [defer](https://docs.getdbt.com/reference/node-selection/defer.md) | ✅ (default) | boolean<br />default: False | ❌ | `DBT_ENGINE_DEFER` | `--defer`<br />`--no-defer` | | [defer\_state](https://docs.getdbt.com/reference/node-selection/defer.md) | ❌ | path<br />default: None | ❌ | `DBT_ENGINE_DEFER_STATE` | `--defer-state` | | [favor\_state](https://docs.getdbt.com/reference/node-selection/defer.md#favor-state) | ✅ | boolean<br />default: False | ❌ | `DBT_ENGINE_FAVOR_STATE` | `--favor-state`<br />`--no-favor-state` | | [empty](https://docs.getdbt.com/docs/build/empty-flag.md) | ✅ | boolean<br />default: False | ❌ | `DBT_ENGINE_EMPTY` | `--empty`<br />`--no-empty` | | [event\_time\_start](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md#batch-properties-for-microbatch-models) | ✅ | datetime<br />default: None | ❌ | `DBT_ENGINE_EVENT_TIME_START` | `--event-time-start` | | [event\_time\_end](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md#batch-properties-for-microbatch-models) | ✅ | datetime<br />default: None | ❌ | `DBT_ENGINE_EVENT_TIME_END` | `--event-time-end` | | [fail\_fast](https://docs.getdbt.com/reference/global-configs/failing-fast.md) | ✅ | boolean<br />default: False | ✅ | `DBT_ENGINE_FAIL_FAST` | `--fail-fast`<br />`-x`<br />`--no-fail-fast` | | [full\_refresh](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) | ✅ | boolean<br />default: False | ✅ (as resource config) | `DBT_ENGINE_FULL_REFRESH` | `--full-refresh`<br />`--no-full-refresh` | | [generate\_info\_schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) (v2.0+) | ✅ | boolean<br />default: False | ❌ | `DBT_ENGINE_GENERATE_INFO_SCHEMA` | `--generate-info-schema` | | hints\_enabled (v1.12+) | ✅ | boolean<br />default: True | ✅ | `DBT_ENGINE_HINTS_ENABLED` | `--hints-enabled`<br />`--no-hints-enabled` | | [indirect\_selection](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md#syntax-examples) | ❌ | enum<br />default: eager | ✅ | `DBT_ENGINE_INDIRECT_SELECTION` | `--indirect-selection` | | [info\_schema\_dir](https://docs.getdbt.com/docs/build/dbt-information-schema.md) (v2.0+) | ✅ | path<br />default: None (uses `<target>/info_schema/`) | ❌ | `DBT_ENGINE_INFO_SCHEMA_DIR` | `--info-schema-dir` | | [introspect](https://docs.getdbt.com/reference/commands/compile.md#introspective-queries) | ❌ | boolean<br />default: True | ❌ | `DBT_ENGINE_INTROSPECT` | `--introspect`<br />`--no-introspect` | | [log\_cache\_events](https://docs.getdbt.com/reference/global-configs/logs.md#logging-relational-cache-events) | ❌ | boolean<br />default: False | ❌ | `DBT_ENGINE_LOG_CACHE_EVENTS` | `--log-cache-events`<br />`--no-log-cache-events` | | [log\_format\_file](https://docs.getdbt.com/reference/global-configs/logs.md#log-formatting) | ❌ | enum<br />default: default (text) | ✅ | `DBT_ENGINE_LOG_FORMAT_FILE` | `--log-format-file` | | [log\_format](https://docs.getdbt.com/reference/global-configs/logs.md#log-formatting) | ❌ | enum<br />default: default (text) | ✅ | `DBT_ENGINE_LOG_FORMAT` | `--log-format` | | [log\_level\_file](https://docs.getdbt.com/reference/global-configs/logs.md#log-level) | ❌ | enum<br />default: debug | ✅ | `DBT_ENGINE_LOG_LEVEL_FILE` | `--log-level-file` | | [log\_level](https://docs.getdbt.com/reference/global-configs/logs.md#log-level) | ❌ | enum<br />default: info | ✅ | `DBT_ENGINE_LOG_LEVEL` | `--log-level` | | [log\_path](https://docs.getdbt.com/reference/global-configs/logs.md) | ❌ | path<br />default: None (uses `logs/`) | ❌ | `DBT_ENGINE_LOG_PATH` | `--log-path` | | [manage\_state](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) (v1.13+) | ✅ | boolean<br />default: False | ✅ | `DBT_ENGINE_MANAGE_STATE` | `--manage-state`<br />`--no-manage-state` | | [partial\_parse](https://docs.getdbt.com/reference/global-configs/parsing.md#partial-parsing) | ✅ | boolean<br />default: True | ✅ | `DBT_ENGINE_PARTIAL_PARSE` | `--partial-parse`<br />`--no-partial-parse` | | [populate\_cache](https://docs.getdbt.com/reference/global-configs/cache.md) | ✅ | boolean<br />default: True | ✅ | `DBT_ENGINE_POPULATE_CACHE` | `--populate-cache`<br />`--no-populate-cache` | | [print](https://docs.getdbt.com/reference/global-configs/print-output.md#suppress-print-messages-in-stdout) | ❌ | boolean<br />default: True | ❌ | `DBT_ENGINE_PRINT` | `--print`<br />`--no-print` | | [printer\_width](https://docs.getdbt.com/reference/global-configs/print-output.md#printer-width) | ❌ | int<br />default: 80 | ✅ | `DBT_ENGINE_PRINTER_WIDTH` | `--printer-width` | | [profile](https://docs.getdbt.com/docs/local/connection-profiles.md#about-profiles) | ❌ | string<br />default: None | ✅ (as top-level key) | `DBT_ENGINE_PROFILE` | [`--profile`](https://docs.getdbt.com/docs/local/connection-profiles.md#overriding-profiles-and-targets) | | [profiles\_dir](https://docs.getdbt.com/docs/local/connection-profiles.md#about-profiles) | ❌ | path<br />default: None (current dir, then HOME dir) | ❌ | `DBT_ENGINE_PROFILES_DIR` | `--profiles-dir` | | [project\_dir](https://docs.getdbt.com/reference/dbt_project.yml.md) | ❌ | path<br />default: (empty) | ❌ | `DBT_ENGINE_PROJECT_DIR` | `--project-dir` | | [quiet](https://docs.getdbt.com/reference/global-configs/logs.md#suppress-non-error-logs-in-output) | ✅ | boolean<br />default: False | ❌ | `DBT_ENGINE_QUIET` | `--quiet` | | [resource-type](https://docs.getdbt.com/reference/global-configs/resource-type.md) (v1.8+) | ✅ | string<br />default: None | ❌ | `DBT_ENGINE_RESOURCE_TYPES`<br />`DBT_ENGINE_EXCLUDE_RESOURCE_TYPES` | `--resource-type`<br />`--exclude-resource-type` | | [sample](https://docs.getdbt.com/docs/build/sample-flag.md) | ✅ | string<br />default: None | ❌ | `DBT_ENGINE_SAMPLE` | `--sample` | | [send\_anonymous\_usage\_stats](https://docs.getdbt.com/reference/global-configs/usage-stats.md) | ❌ | boolean<br />default: True | ✅ | `DBT_ENGINE_SEND_ANONYMOUS_USAGE_STATS` | `--send-anonymous-usage-stats`<br />`--no-send-anonymous-usage-stats` | | [source\_freshness\_run\_project\_hooks](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md) | ❌ | boolean<br />default: True | ✅ | ❌ | ❌ | | [sqlparse](https://docs.getdbt.com/reference/global-configs/sqlparse.md) | ❌ | YAML map<br />default: MAX\_GROUPING\_DEPTH and MAX\_GROUPING\_TOKENS set to null | ❌ | `DBT_ENGINE_SQLPARSE` | `--sqlparse` | | [state](https://docs.getdbt.com/reference/node-selection/defer.md) | ❌ | path<br />default: none | ❌ | `DBT_ENGINE_STATE`, `DBT_ENGINE_DEFER_STATE` | `--state`<br />`--defer-state` | | [static\_parser](https://docs.getdbt.com/reference/global-configs/parsing.md#static-parser) | ❌ | boolean<br />default: True | ✅ | `DBT_ENGINE_STATIC_PARSER` | `--static-parser`<br />`--no-static-parser` | | [store\_failures](https://docs.getdbt.com/reference/resource-configs/store_failures.md) | ✅ | boolean<br />default: False | ✅ (as resource config) | `DBT_ENGINE_STORE_FAILURES` | `--store-failures`<br />`--no-store-failures` | | [target\_path](https://docs.getdbt.com/reference/global-configs/json-artifacts.md) | ❌ | path<br />default: None (uses `target/`) | ❌ | `DBT_ENGINE_TARGET_PATH` | `--target-path` | | [target](https://docs.getdbt.com/docs/local/connection-profiles.md#about-profiles) | ❌ | string<br />default: None | ❌ | `DBT_ENGINE_TARGET` | [`--target`](https://docs.getdbt.com/docs/local/connection-profiles.md#overriding-profiles-and-targets) | | [use\_colors\_file](https://docs.getdbt.com/reference/global-configs/logs.md#color) | ❌ | boolean<br />default: True | ✅ | `DBT_ENGINE_USE_COLORS_FILE` | `--use-colors-file`<br />`--no-use-colors-file` | | [use\_colors](https://docs.getdbt.com/reference/global-configs/print-output.md#print-color) | ❌ | boolean<br />default: True | ✅ | `DBT_ENGINE_USE_COLORS` | `--use-colors`<br />`--no-use-colors` | | [use\_experimental\_parser](https://docs.getdbt.com/reference/global-configs/parsing.md#experimental-parser) | ❌ | boolean<br />default: False | ✅ | `DBT_ENGINE_USE_EXPERIMENTAL_PARSER` | `--use-experimental-parser`<br />`--no-use-experimental-parser` | | [use\_fast\_test\_edges](https://docs.getdbt.com/reference/global-configs/fast-test-edges.md) | ✅ | boolean<br />default: False | ❌ | `DBT_ENGINE_USE_FAST_TEST_EDGES` | `--use-fast-test-edges`<br />`--no-use-fast-test-edges` | | [use\_v2\_parser](https://docs.getdbt.com/reference/global-configs/parsing.md#opt-in-v2-parser) | ✅ | boolean<br />default: False | ✅ | `DBT_ENGINE_USE_V2_PARSER` | `--use-v2-parser` | | [version\_check](https://docs.getdbt.com/reference/global-configs/version-compatibility.md) | ❌ | boolean<br />default: varies | ✅ | `DBT_ENGINE_VERSION_CHECK` | `--version-check`<br />`--no-version-check` | | [warn\_error\_options](https://docs.getdbt.com/reference/global-configs/warnings.md) | ✅ | dict<br />default: | ✅ | `DBT_ENGINE_WARN_ERROR_OPTIONS` | `--warn-error-options` | | [warn\_error](https://docs.getdbt.com/reference/global-configs/warnings.md) | ✅ | boolean<br />default: False | ✅ | `DBT_ENGINE_WARN_ERROR` | `--warn-error` | | [write\_json](https://docs.getdbt.com/reference/global-configs/json-artifacts.md) | ✅ | boolean<br />default: True | ✅ | `DBT_ENGINE_WRITE_JSON` | `--write-json`<br />`--no-write-json` | ##### Common flag examples Use the `--target` flag to specify which target (environment) to use when running dbt commands. For example: ```bash dbt run --target dev dbt run --target prod dbt build --target staging ``` The `--target` flag allows you to run the same dbt project against different environments without modifying your configuration files. Define the target in your `profiles.yml` file. Learn more about [connection profiles and targets](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-targets-in-profiles). --- ### About flags variable The `flags` variable contains values of flags provided on the command line. **Example usage:** flags.sql ```sql {% if flags.FULL_REFRESH %} drop table ... {% else %} -- no-op {% endif %} ``` The list of available flags is defined in the [`flags` module](https://github.com/dbt-labs/dbt/blob/HEAD/crates/dbt-jinja-utils/src/flags.rs) within `dbt-labs/dbt`. Recommended use cases include: * different materialization logic based on "run modes," such as `flags.FULL_REFRESH`, `flags.STORE_FAILURES`, and `flags.EMPTY` * running hooks conditionally based on the current command / task type, via `flags.WHICH` **Note:** It is *not* recommended to use flags as an input to parse-time configurations, properties, or dependencies (`ref` + `source`). Flags are likely to change in every invocation of dbt, and their parsed values will become stale (and yield incorrect results) in subsequent invocations that have partial parsing enabled. For more details, see [the docs on parsing](https://docs.getdbt.com/reference/parsing.md). ##### invocation\_args\_dict For the full set of information passed from the CLI—subcommand, flags, arguments—you can use `invocation_args_dict`. This is equivalent to the `args` dictionary in [`run_results.json`](https://docs.getdbt.com/reference/artifacts/run-results-json.md). models/my\_model.sql ```sql -- invocation_args_dict: -- {{ invocation_args_dict }} -- dbt_metadata_envs: -- {{ dbt_metadata_envs }} select 1 as id ``` The `invocation_command` key within `invocation_args_dict` includes the entire subcommand when it compiles: ```shell $ DBT_ENV_CUSTOM_ENV_MYVAR=myvalue dbt compile -s my_model 12:10:22 Running with dbt=1.6.0-b8 12:10:22 Registered adapter: postgres=1.6.0-b8 12:10:22 Found 1 seed, 1 model, 349 macros 12:10:22 12:10:22 Concurrency: 5 threads (target='dev') 12:10:22 12:10:22 Compiled node 'my_model' is: -- invocation_args_dict: -- {'log_format_file': 'debug', 'log_level': 'info', 'exclude': (), 'send_anonymous_usage_stats': True, 'which': 'compile', 'defer': False, 'output': 'text', 'log_format': 'default', 'macro_debugging': False, 'populate_cache': True, 'static_parser': True, 'vars': {}, 'warn_error_options': WarnErrorOptions(include=[], exclude=[]), 'quiet': False, 'select': ('my_model',), 'indirect_selection': 'eager', 'strict_mode': False, 'version_check': False, 'enable_legacy_logger': False, 'log_path': '/Users/jerco/dev/scratch/testy/logs', 'profiles_dir': '/Users/jerco/.dbt', 'invocation_command': 'dbt compile -s my_model', 'log_level_file': 'debug', 'project_dir': '/Users/jerco/dev/scratch/testy', 'favor_state': False, 'use_colors_file': True, 'write_json': True, 'partial_parse': True, 'printer_width': 80, 'print': True, 'cache_selected_only': False, 'use_colors': True, 'introspect': True} -- dbt_metadata_envs: -- {'MYVAR': 'myvalue'} select 1 as id ``` #### flags.EMPTY `flags.EMPTY` returns `True` when the [`--empty` flag](https://docs.getdbt.com/docs/build/empty-flag.md) is passed to a dbt command, and `False` otherwise. This is useful when you need to customize materialization logic or macro behavior depending on whether dbt is running in empty (schema-only) mode. ```sql {% macro my_custom_ref(model_name) %} {% if flags.EMPTY %} {# Return zero rows when running with --empty #} (select * from {{ ref(model_name) }} where 1=0) {% else %} {{ ref(model_name) }} {% endif %} {% endmacro %} ``` #### flags.WHICH `flags.WHICH` is a global variable that gets set when you run a dbt command. If used in a macro, it allows you to conditionally change behavior depending on the command currently being executed. For example, conditionally modifying SQL: ```sql {% macro conditional_filter(table_name) %} {# Add a WHERE clause only during `dbt run`, not during `dbt test` or `dbt compile` #} select * from {{ table_name }} {% if flags.WHICH == "run" %} where is_active = true {% elif flags.WHICH == "test" %} -- In test runs, restrict rows to keep tests fast limit 10 {% elif flags.WHICH == "compile" %} -- During compile, just add a harmless comment -- compile mode detected {% endif %} {% endmacro %} ``` The following commands are supported: | `flags.WHICH` value | Description | | ------------------- | ------------------------------------------------------------------------- | | `"build"` | Build and test all selected resources. | | `"clean"` | Remove artifacts like target directory and packages. | | `"clone"` | Clone models and other resources. | | `"compile"` | Compile SQL, but do not execute. | | `"debug"` | Test connections and validate configs. | | `"deps"` | Download package dependencies. | | `"docs"` | Generate and serve documentation. | | `"environment"` | Workspace environment commands (dbt platform CLI). | | `"help"` | Show help for commands and subcommands. | | `"init"` | Bootstrap a new project. | | `"invocation"` | For interacting with or inspecting current invocation (dbt platform CLI). | | `"list"` | List resources. | | `"parse"` | Parse project and report errors, but don’t build/test. | | `"retry"` | Retry the last invocation from the point of failure. | | `"run"` | Execute models. | | `"run-operation"` | Invoke arbitrary macros or SQL ops. | | `"seed"` | Load CSV(s) into the database. | | `"show"` | Inspect resource definitions or materializations. | | `"snapshot"` | Execute snapshots. | | `"source"` | Validate freshness and inspect source definitions. | | `"test"` | Schema and data tests. | | `"version"` | Display dbt version. | --- ### About fromjson context method The `fromjson` context method can be used to deserialize a JSON string into a Python object primitive, eg. a `dict` or `list`. **Args**: * `string`: The JSON string to deserialize (required) * `default`: A default value to return if the `string` argument cannot be deserialized (optional) ##### Usage: ```text {% set my_json_str = '{"abc": 123}' %} {% set my_dict = fromjson(my_json_str) %} {% do log(my_dict['abc']) %} ``` --- ### About fromyaml context method The `fromyaml` context method can be used to deserialize a YAML string into a Python object primitive, eg. a `dict` or `list`. **Args**: * `string`: The YAML string to deserialize (required) * `default`: A default value to return if the `string` argument cannot be deserialized (optional) ##### Usage: ```text {% set my_yml_str -%} dogs: - good - bad {%- endset %} {% set my_dict = fromyaml(my_yml_str) %} {% do log(my_dict['dogs'], info=true) %} -- ["good", "bad"] {% do my_dict['dogs'].pop() %} {% do log(my_dict['dogs'], info=true) %} -- ["good"] ``` --- ### About graph context variable The `graph` context variable contains information about the *nodes* in your dbt project. Models, sources, tests, and snapshots are all examples of nodes in dbt projects. Heads up dbt actively builds the `graph` variable during the [parsing phase](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) of running dbt projects, so some properties of the `graph` context variable will be missing or incorrect during parsing. Please read the information below carefully to understand how to effectively use this variable. ##### The graph context variable The `graph` context variable is a dictionary which maps node ids onto dictionary representations of those nodes. A simplified example might look like: ```json { "nodes": { "model.my_project.model_name": { "unique_id": "model.my_project.model_name", "config": {"materialized": "table", "sort": "id"}, "tags": ["abc", "123"], "path": "models/path/to/model_name.sql", ... }, ... }, "sources": { "source.my_project.snowplow.event": { "unique_id": "source.my_project.snowplow.event", "database": "analytics", "schema": "analytics", "tags": ["abc", "123"], "path": "models/path/to/schema.yml", ... }, ... }, "exposures": { "exposure.my_project.traffic_dashboard": { "unique_id": "exposure.my_project.traffic_dashboard", "type": "dashboard", "maturity": "high", "path": "models/path/to/schema.yml", ... }, ... }, "metrics": { "metric.my_project.count_all_events": { "unique_id": "metric.my_project.count_all_events", "type": "count", "path": "models/path/to/schema.yml", ... }, ... }, "groups": { "group.my_project.finance": { "unique_id": "group.my_project.finance", "name": "finance", "owner": { "email": "finance@jaffleshop.com" } ... }, ... } } ``` The exact contract for these model and source nodes is not currently documented, but that will change in the future. ##### Accessing models The `model` entries in the `graph` dictionary will be incomplete or incorrect during parsing. If accessing the models in your project via the `graph` variable, be sure to use the [execute](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) flag to ensure that this code only executes at run-time and not at parse-time. Do not use the `graph` variable to build your DAG, as the resulting dbt behavior will be undefined and likely incorrect. Example usage: graph-usage.sql ```sql /* Print information about all of the models in the Snowplow package */ {% if execute %} {% for node in graph.nodes.values() | selectattr("resource_type", "equalto", "model") | selectattr("package_name", "equalto", "snowplow") %} {% do log(node.unique_id ~ ", materialized: " ~ node.config.materialized, info=true) %} {% endfor %} {% endif %} /* Example output --------------------------------------------------------------- model.snowplow.snowplow_id_map, materialized: incremental model.snowplow.snowplow_page_views, materialized: incremental model.snowplow.snowplow_web_events, materialized: incremental model.snowplow.snowplow_web_page_context, materialized: table model.snowplow.snowplow_web_events_scroll_depth, materialized: incremental model.snowplow.snowplow_web_events_time, materialized: incremental model.snowplow.snowplow_web_events_internal_fixed, materialized: ephemeral model.snowplow.snowplow_base_web_page_context, materialized: ephemeral model.snowplow.snowplow_base_events, materialized: ephemeral model.snowplow.snowplow_sessions_tmp, materialized: incremental model.snowplow.snowplow_sessions, materialized: table */ ``` ##### Accessing sources To access the sources in your dbt project programmatically, use the `sources` attribute of the `graph` object. Example usage: models/events\_unioned.sql ```sql /* Union all of the Snowplow sources defined in the project which begin with the string "event_" */ {% set sources = [] -%} {% for node in graph.sources.values() -%} {%- if node.name.startswith('event_') and node.source_name == 'snowplow' -%} {%- do sources.append(source(node.source_name, node.name)) -%} {%- endif -%} {%- endfor %} select * from ( {%- for source in sources %} select * from {{ source }} {% if not loop.last %} union all {% endif %} {% endfor %} ) /* Example compiled SQL --------------------------------------------------------------- select * from ( select * from raw.snowplow.event_add_to_cart union all select * from raw.snowplow.event_remove_from_cart union all select * from raw.snowplow.event_checkout ) */ ``` ##### Accessing exposures To access the exposures in your dbt project programmatically, use the `exposures` attribute of the `graph` object. Example usage: models/my\_important\_view\_model.sql ```sql {# Include a SQL comment naming all of the exposures that this model feeds into #} {% set exposures = [] -%} {% for exposure in graph.exposures.values() -%} {%- if model['unique_id'] in exposure.depends_on.nodes -%} {%- do exposures.append(exposure) -%} {%- endif -%} {%- endfor %} -- HELLO database administrator! Before dropping this view, -- please be aware that doing so will affect: {% for exposure in exposures %} -- * {{ exposure.name }} ({{ exposure.type }}) {% endfor %} /* Example compiled SQL --------------------------------------------------------------- -- HELLO database administrator! Before dropping this view, -- please be aware that doing so will affect: -- * our_metrics (dashboard) -- * my_sync (application) */ ``` ##### Accessing metrics To access the metrics in your dbt project programmatically, use the `metrics` attribute of the `graph` object. Example usage: macros/get\_metric.sql ```sql {% macro get_metric_sql_for(metric_name) %} {% set metrics = graph.metrics.values() %} {% set metric = (metrics | selectattr('name', 'equalto', metric_name) | list).pop() %} /* Elsewhere, I've defined a macro, get_metric_timeseries_sql, that will return the SQL needed to perform a time-based rollup of this metric's calculation */ {% set metric_sql = get_metric_timeseries_sql( relation = metric['model'], type = metric['type'], expression = metric['sql'], ... ) %} {{ return(metric_sql) }} {% endmacro %} ``` ##### Accessing groups To access the groups in your dbt project programmatically, use the `groups` attribute of the `graph` object. Example usage: macros/get\_group.sql ```sql {% macro get_group_owner_for(group_name) %} {% set groups = graph.groups.values() %} {% set owner = (groups | selectattr('owner', 'equalto', group_name) | list).pop() %} {{ return(owner) }} {% endmacro %} ``` --- ### About invocation_id The `invocation_id` outputs a UUID generated for this dbt command. This value is useful when auditing or analyzing dbt invocation metadata. If available, the `invocation_id` is: * available in the compilation context of [`query-comment`](https://docs.getdbt.com/reference/project-configs/query-comment.md) * included in the `info` dictionary in dbt [events and logs](https://docs.getdbt.com/reference/events-logging.md#info) * included in the `metadata` dictionary in [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md#common-metadata) * included as a label in all BigQuery jobs that dbt originates **Example usage**: You can use the following example code for all data platforms. Remember to replace `TABLE_NAME` with the actual name of your target table: `select '{{ invocation_id }}' as test_id from TABLE_NAME` --- ### About local state in dbt Looking for a managed state experience? If you want a managed experience for state with dbt to skip rerunning models that haven't changed, check out [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). In self-managed deployments with dbt State enabled, all [`state:*` selectors](https://docs.getdbt.com/reference/node-selection/methods.md#state) can also use dbt State as their comparison source for per-node granularity without managing a `manifest.json`. Refer to [dbt State-powered `state:*` selectors](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md#dbt-state-powered-state-selectors) for more information. Why idempotence matters here State selection and deferral rely on your models being [idempotent](https://docs.getdbt.com/best-practices/idempotence.md), meaning re-running with the same inputs returns the same result. For more info, refer to [Idempotence in dbt](https://docs.getdbt.com/best-practices/idempotence.md). One of the greatest underlying assumptions about dbt is that its operations should be **stateless** and **idempotent**. That is, it doesn't matter how many times a model has been run before, or if it has ever been run before. It doesn't matter if you run it once or a thousand times. Given the same raw data, you can expect the same transformed result. A given run of dbt doesn't need to "know" about *any other* run; it just needs to know about the code in the project and the objects in your database as they exist *right now*. That said, dbt does store "state" — a detailed, point-in-time view of project resources (also referred to as nodes), database objects, and invocation results — in the form of its [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md). If you choose, dbt can use these artifacts to inform certain operations. Crucially, the operations themselves are still stateless and idempotent: given the same manifest and the same raw data, dbt will produce the same transformed result. dbt can leverage artifacts from a prior invocation as long as their file path is passed to the `--state` flag. This is a prerequisite for: * [The `state` selector](https://docs.getdbt.com/reference/node-selection/methods.md#state), whereby dbt can identify resources that are new or modified by comparing code in the current project against the state manifest. * [Deferring](https://docs.getdbt.com/reference/node-selection/defer.md) to another environment, whereby dbt can identify upstream, unselected resources that don't exist in your current environment and instead "defer" their references to the environment provided by the state manifest. * The [`dbt clone` command](https://docs.getdbt.com/reference/commands/clone.md), whereby dbt can clone nodes based on their location in the manifest provided to the `--state` flag. Together, the [`state`](https://docs.getdbt.com/reference/node-selection/methods.md#state) selector and deferral enable ["slim CI"](https://docs.getdbt.com/best-practices/best-practice-workflows.md#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag. #### Related docs * [Configure state selection](https://docs.getdbt.com/reference/node-selection/configure-state.md) * [State comparison caveats](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md) --- ### About local_md5 context variable The `local_md5` context variable calculates an [MD5 hash](https://en.wikipedia.org/wiki/MD5) of the given string. The string `local_md5` emphasizes that the hash is calculated *locally*, in the dbt-Jinja context. This variable is typically useful for advanced use cases. For example, when you generate unique identifiers within custom materialization or operational logic, you can either avoid collisions between temporary relations or identify changes by comparing checksums. It is different than the `md5` SQL function, supported by many SQL dialects, which runs remotely in the data platform. You want to always use SQL hashing functions when generating surrogate keys. Usage: ```sql -- source {%- set value_hash = local_md5("hello world") -%} '{{ value_hash }}' -- compiled '5eb63bbbe01eeed093cb22bb8f5acdc3' ``` --- ### About model object `model` is the dbt [graph object](https://docs.getdbt.com/reference/dbt-jinja-functions/graph.md) (or node) for the current model. It can be used to: * Access `config` settings, say, in a post-hook * Access the path to the model For example: ```jinja {% if model.config.materialized == 'view' %} {{ log(model.name ~ " is a view.", info=True) }} {% endif %} ``` To view the contents of `model` for a given model: ##### Command line interface If you're using the command line interface (CLI), use [log()](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) to print the full contents: ```jinja {{ log(model, info=True) }} ``` ##### Studio IDE If you're using the Studio IDE, compile the following to print the full contents:<br /><br /> ```jinja {{ model | tojson(indent = 4) }} ``` #### Batch properties for microbatch models Starting in dbt v1.9, the model object includes a `batch` property (`model.batch`), which provides details about the current batch when executing an [incremental microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) model. This property is only populated during the batch execution of a microbatch model. The following table describes the properties of the `batch` object. Note that dbt appends the property to the `model` and `batch` objects. | Property | Description | Example | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | | `id` | The unique identifier for the batch within the context of the microbatch model. | `model.batch.id` | | `event_time_start` | The start time of the batch's [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) filter (inclusive). | `model.batch.event_time_start` | | `event_time_end` | The end time of the batch's `event_time` filter (exclusive). | `model.batch.event_time_end` | ##### Usage notes `model.batch` is only available during the execution of a microbatch model batch. Outside of the microbatch execution, `model.batch` is `None`, and its sub-properties aren't accessible. ###### Example of safeguarding access to batch properties We recommend to always check if `model.batch` is populated before accessing its properties. To do this, use an `if` statement for safe access to `batch` properties: ```jinja {% if model.batch %} {{ log(model.batch.id) }} # Log the batch ID # {{ log(model.batch.event_time_start) }} # Log the start time of the batch # {{ log(model.batch.event_time_end) }} # Log the end time of the batch # {% endif %} ``` In this example, the `if model.batch` statement makes sure that the code only runs during a batch execution. `log()` is used to print the `batch` properties for debugging. ###### Example of log batch details This is a practical example of how you might use `model.batch` in a microbatch model to log batch details for the `batch.id`: ```jinja {% if model.batch %} {{ log("Processing batch with ID: " ~ model.batch.id, info=True) }} {{ log("Batch event time range: " ~ model.batch.event_time_start ~ " to " ~ model.batch.event_time_end, info=True) }} {% endif %} ``` In this example, the `if model.batch` statement makes sure that the code only runs during a batch execution. `log()` is used to print the `batch` properties for debugging. #### Model structure and JSON schema To view the structure of `models` and their definitions: * Refer to [dbt JSON Schema](https://schemas.getdbt.com/) for describing and consuming dbt generated artifacts * Select the corresponding manifest version under **Manifest**. For example if you're on dbt v1.8, then you would select Manifest v12 * The `manifest.json` version number is related to (but not *equal* to) your dbt version, so you *must* use the correct `manifest.json` version for your dbt version. To find the correct `manifest.json` version, refer to [Manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) and select the dbt version on the top navigation (such as `v1.5`). This will help you find out which tags are associated with your model. * Then go to `nodes` --> Select Additional properties --> `CompiledModelNode` or view other definitions/objects. Use the following table to understand how the versioning pattern works and match the Manifest version with the dbt version: | dbt version | Manifest version | | ----------- | ------------------------------------------------------------- | | dbt v2.0 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.11 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.10 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.9 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.8 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.7 | [v11](https://schemas.getdbt.com/dbt/manifest/v11/index.html) | | dbt v1.6 | [v10](https://schemas.getdbt.com/dbt/manifest/v10/index.html) | | dbt v1.5 | [v9](https://schemas.getdbt.com/dbt/manifest/v9/index.html) | | dbt v1.4 | [v8](https://schemas.getdbt.com/dbt/manifest/v8/index.html) | | dbt v1.3 | [v7](https://schemas.getdbt.com/dbt/manifest/v7/index.html) | #### Related docs * [dbt JSON Schema](https://schemas.getdbt.com/) --- ### About model schema The schema that the model is configured to be materialized in. This is typically the same as `model['schema']`. --- ### About modules variable The `modules` variable in the Jinja context is a predefined namespace that contains only a limited set of supported Python modules for operating on data. You cannot import or access arbitrary Python modules (for example, `os`, `requests`, or custom third-party libraries) from within Jinja. There is no user-facing configuration to modify or extend the `modules` namespace. This restriction helps ensure consistent behavior, security, and portability across environments. If your workflow requires functionality from additional Python libraries, use a [Python model](https://docs.getdbt.com/docs/build/python-models.md) (where supported) instead of Jinja. Python models run in a different execution context and allow you to import and use external libraries as needed. #### datetime This variable is a pointer to the Python [`datetime`](https://docs.python.org/3/library/datetime.html) module, which supports complex date and time logic. It includes the modules contexts of `date`, `datetime`, `time`, `timedelta`, and `tzinfo`. **Usage** ```text {% set now = modules.datetime.datetime.now() %} {% set three_days_ago_iso = (now - modules.datetime.timedelta(3)).isoformat() %} ``` This module will return the current date and time on every Jinja evaluation. For the date and time of the start of the run, please see [run\_started\_at](https://docs.getdbt.com/reference/dbt-jinja-functions/run_started_at.md). #### pytz This variable is a pointer to the Python [`pytz`](https://pypi.org/project/pytz/) module, which supports timezone logic. **Usage** ```text {% set dt = modules.datetime.datetime(2002, 10, 27, 6, 0, 0) %} {% set dt_local = modules.pytz.timezone('US/Eastern').localize(dt) %} {{ dt_local }} ``` #### re This variable is a pointer to the Python [`re`](https://docs.python.org/3/library/re.html) module, which supports regular expressions. **Usage** ```text {% set my_string = 's3://example/path' %} {% set s3_path_pattern = 's3://[a-z0-9-_/]+' %} {% set re = modules.re %} {% set is_match = re.match(s3_path_pattern, my_string, re.IGNORECASE) %} {% if not is_match %} {%- do exceptions.raise_compiler_error( my_string ~ ' is not a valid s3 path' ) -%} {% endif %} ``` #### itertools Note Starting in `dbt-core==1.10.6`, using `modules.itertools` raises a deprecation warning. For more information and suggested workarounds, refer to the [documentation on `ModulesItertoolsUsageDeprecation`](https://docs.getdbt.com/reference/deprecations.md#modulesitertoolsusagedeprecation). This variable is a pointer to the Python [`itertools`](https://docs.python.org/3/library/itertools.html) module, which includes useful functions for working with iterators (loops, lists, and the like). The supported functions are: * `count` * `cycle` * `repeat` * `accumulate` * `chain` * `compress` * `islice` * `starmap` * `tee` * `zip_longest` * `product` * `permutations` * `combinations` * `combinations_with_replacement` **Usage** ```text {%- set A = [1, 2] -%} {%- set B = ['x', 'y', 'z'] -%} {%- set AB_cartesian = modules.itertools.product(A, B) -%} {%- for item in AB_cartesian %} {{ item }} {%- endfor -%} ``` ```text (1, 'x') (1, 'y') (1, 'z') (2, 'x') (2, 'y') (2, 'z') ``` --- ### About on-run-end context variable Caution These variables are only available in the context for `on-run-end` hooks. They will evaluate to `none` if used outside of an `on-run-end` hook! #### schemas The `schemas` context variable can be used to reference the schemas that dbt has built models into during a run of dbt. This variable can be used to grant usage on these schemas to certain users at the end of a dbt run. Example: dbt\_project.yml ```sql on-run-end: - "{% for schema in schemas %}grant usage on schema {{ schema }} to db_reader;{% endfor %}" ``` In practice, it might not be a bad idea to put this code into a macro: macros/grants.sql ```jinja2 {% macro grant_usage_to_schemas(schemas, user) %} {% for schema in schemas %} grant usage on schema {{ schema }} to {{ user }}; {% endfor %} {% endmacro %} ``` dbt\_project.yml ```yaml on-run-end: - "{{ grant_usage_to_schemas(schemas, 'user') }}" ``` #### database\_schemas The `database_schemas` context variable can be used to reference the databases *and* schemas that dbt has built models into during a run of dbt. This variable is similar to the `schemas` variable, and should be used if a dbt run builds resources into multiple different databases. Example: macros/grants.sql ```jinja2 {% macro grant_usage_to_schemas(database_schemas, user) %} {% for (database, schema) in database_schemas %} grant usage on {{ database }}.{{ schema }} to {{ user }}; {% endfor %} {% endmacro %} ``` dbt\_project.yml ```yaml on-run-end: - "{{ grant_usage_to_schemas(database_schemas, user) }}" ``` #### Results The `results` variable contains a list of [Result objects](https://docs.getdbt.com/reference/dbt-classes.md#result-objects) with one element per resource that executed in the dbt job. The Result object provides access within the Jinja on-run-end context to the information that will populate the [run results JSON artifact](https://docs.getdbt.com/reference/artifacts/run-results-json.md). Example usage: macros/log\_results.sql ```sql {% macro log_results(results) %} {% if execute %} {{ log("========== Begin Summary ==========", info=True) }} {% for res in results -%} {% set line -%} node: {{ res.node.unique_id }}; status: {{ res.status }} (message: {{ res.message }}) {%- endset %} {{ log(line, info=True) }} {% endfor %} {{ log("========== End Summary ==========", info=True) }} {% endif %} {% endmacro %} ``` dbt\_project.yml ```yaml on-run-end: "{{ log_results(results) }}" ``` Results: ```text 12:48:17 | Concurrency: 1 threads (target='dev') 12:48:17 | 12:48:17 | 1 of 2 START view model dbt_jcohen.abc............................... [RUN] 12:48:17 | 1 of 2 OK created view model dbt_jcohen.abc.......................... [CREATE VIEW in 0.11s] 12:48:17 | 2 of 2 START table model dbt_jcohen.def.............................. [RUN] 12:48:17 | 2 of 2 ERROR creating table model dbt_jcohen.def..................... [ERROR in 0.09s] 12:48:17 | 12:48:17 | Running 1 on-run-end hook ========== Begin Summary ========== node: model.testy.abc; status: success (message: CREATE VIEW) node: model.testy.def; status: error (message: Database Error in model def (models/def.sql) division by zero compiled SQL at target/run/testy/models/def.sql) ========== End Summary ========== 12:48:17 | 1 of 1 START hook: testy.on-run-end.0................................ [RUN] 12:48:17 | 1 of 1 OK hook: testy.on-run-end.0................................... [OK in 0.00s] 12:48:17 | 12:48:17 | 12:48:17 | Finished running 1 view model, 1 table model, 1 hook in 1.94s. ``` --- ### About packages.yml context The following context methods and variables are available when configuring a `packages.yml` file. **Available context methods:** * [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) * Use `env_var()` in any dbt YAML file that supports Jinja. Only `packages.yml` and `profiles.yml` support environment variables for [secure values](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) (using the `DBT_ENV_SECRET_` prefix). * [var](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) (Note: only variables defined with `--vars` are available. Refer to [YAML tips](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for more information) **Available context variables:** * [builtins](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md) * [dbt\_version](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt_version.md) * [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) #### Example usage The following examples show how to use the different context methods and variables in your `packages.yml`. Use `builtins` in your `packages.yml`: ```text packages: - package: dbt-labs/dbt_utils version: "{% if builtins is defined %}0.14.0{% else %}0.13.1{% endif %}" ``` Use `env_var` in your `packages.yml`: ```text packages: - package: dbt-labs/dbt_utils version: "{{ env_var('DBT_UTILS_VERSION') }}" ``` Use `dbt_version` in your `packages.yml`: ```text packages: - package: dbt-labs/dbt_utils version: "{% if dbt_version is defined %}0.14.0{% else %}0.13.1{% endif %}" ``` Use `target` in your `packages.yml`: ```text packages: - package: dbt-labs/dbt_utils version: "{% if target.name == 'prod' %}0.14.0{% else %}0.13.1{% endif %}" ``` #### Related docs * [Packages](https://docs.getdbt.com/docs/build/packages.md) --- ### About print function Use the `print()` function when you want to print messages to both the log file and standard output (stdout). When used in conjunction with the `QUIET` global config, which suppresses non-error logs, you will only see error logs and the print messages in stdout. For more information, see [Global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md). #### Example ```sql {% macro some_macro(arg1, arg2) %} {{ print("Running some_macro: " ~ arg1 ~ ", " ~ arg2) }} {% endmacro %} ``` --- ### About profiles.yml context The following context methods are available when configuring resources in the `profiles.yml` file. **Available context methods:** * [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) * [var](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) (*Note: only variables defined with `--vars` are available*) ##### Example usage \~/.dbt/profiles.yml ```yml jaffle_shop: target: dev outputs: dev: type: redshift host: "{{ env_var('DBT_HOST') }}" user: "{{ env_var('DBT_USER') }}" password: "{{ env_var('DBT_PASS') }}" port: 5439 dbname: analytics schema: dbt_dbanin threads: 4 ``` --- ### About project_name context variable The `project_name` context variable returns the `name` for the root-level project which is being run by dbt. This variable can be used to defer execution to a root-level project macro if one exists. ##### Example Usage redshift/macros/helper.sql ```sql /* This macro vacuums tables in a Redshift database. If a macro exists in the root-level project called `get_tables_to_vacuum`, this macro will call _that_ macro to find the tables to vacuum. If the macro is not defined in the root project, this macro will use a default implementation instead. */ {% macro vacuum_tables() %} {% set root_project = context[project_name] %} {% if root_project.get_tables_to_vacuum %} {% set tables = root_project.get_tables_to_vacuum() %} {% else %} {% set tables = redshift.get_tables_to_vacuum() %} {% endif %} {% for table in tables %} {% do redshift.vacuum_table(table) %} {% endfor %} {% endmacro %} ``` --- ### About properties.yml context The following context methods and variables are available when configuring resources in a `properties.yml` file. **Available context methods:** * [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) * [var](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) **Available context variables:** * [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) * [builtins](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md) * [dbt\_version](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt_version.md) ##### Example configuration properties.yml ```yml # Configure this model to be materialized as a view # in development and a table in production/CI contexts models: - name: dim_customers config: materialized: "{{ 'view' if target.name == 'dev' else 'table' }}" ``` --- ### About ref function ```sql select * from {{ ref("node_name") }} ``` #### Definition This function: * Returns a [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) for a [model](https://docs.getdbt.com/docs/build/models.md), [seed](https://docs.getdbt.com/docs/build/seeds.md), or [snapshot](https://docs.getdbt.com/docs/build/snapshots.md) * Creates dependencies between the referenced node and the current model, which is useful for documentation and [node selection](https://docs.getdbt.com/reference/node-selection/syntax.md) * Compiles to the full object name in the database The most important function in dbt is `ref()`; it's impossible to build even moderately complex models without it. `ref()` is how you reference one model within another. This is a very common behavior, as typically models are built to be "stacked" on top of one another. Here is how this looks in practice: model\_a.sql ```sql select * from public.raw_data ``` model\_b.sql ```sql select * from {{ref('model_a')}} ``` `ref()` is, under the hood, actually doing two important things. First, it is interpolating the schema into your model file to allow you to change your deployment schema via configuration. Second, it is using these references between models to automatically build the dependency graph. This will enable dbt to deploy models in the correct order when using `dbt run`. The `{{ ref }}` function returns a `Relation` object that has the same `table`, `schema`, and `name` attributes as the [{{ this }} variable](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md). #### Advanced ref usage ##### Versioned ref The `ref` function supports an optional keyword argument - `version` (or `v`). When a version argument is provided to the `ref` function, dbt returns to the `Relation` object corresponding to the specified version of the referenced model. This functionality is useful when referencing versioned models that make breaking changes by creating new versions, but guarantees no breaking changes to existing versions of the model. If the `version` argument is not supplied to a `ref` of a versioned model, the latest version is. This has the benefit of automatically incorporating the latest changes of a referenced model, but there is a risk of incorporating breaking changes. ###### Example models/\<schema>.yml ```yml models: - name: model_name latest_version: 2 versions: - v: 2 - v: 1 ``` ```sql -- returns the `Relation` object corresponding to version 1 of model_name select * from {{ ref('model_name', version=1) }} ``` ```sql -- returns the `Relation` object corresponding to version 2 (the latest version) of model_name select * from {{ ref('model_name') }} ``` ##### Ref project-specific models You can also reference models from different projects using the two-argument variant of the `ref` function. By specifying both a namespace (which could be a project or package) and a model name, you ensure clarity and avoid any ambiguity in the `ref`. This is also useful when dealing with models across various projects or packages. When using two arguments with projects (not packages), you also need to set [cross project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md). The following syntax demonstrates how to reference a model from a specific project or package: ```sql select * from {{ ref('project_or_package', 'model_name') }} ``` We recommend using two-argument `ref` any time you are referencing a model defined in a different package or project. While not required in all cases, it's more explicit for you, for dbt, and future readers of your code. We especially recommend using two-argument `ref` to avoid ambiguity, in cases where a model name is duplicated across multiple projects or installed packages. If you use one-argument `ref` (just the `model_name`), dbt will look for a model by that name in the same namespace (package or project); if it finds none, it will raise an error. (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. **Note:** The `project_or_package` should match the `name` of the project/package, as defined in its `dbt_project.yml`. This might be different from the name of the repository. It never includes the repository's organization name. For example, if you use the [`fivetran/stripe`](https://hub.getdbt.com/fivetran/stripe/latest/) package, the package name is `stripe`, not `fivetran/stripe`. ##### Forcing dependencies In normal usage, dbt knows the proper order to run all models based on the use of the `ref` function, because it discovers them all during its parse phase. dbt will throw an error if it discovers an "unexpected" `ref` at run time (meaning it was hidden during the parsing phase). The most common cause for this is that the `ref` is inside a branch of an `if` statement that wasn't evaluated during parsing. conditional\_ref.sql ```sql --This macro already has its own `if execute` check, so this one is redundant and introduced solely to cause an error {% if execute %} {% set sql_statement %} select max(created_at) from {{ ref('processed_orders') }} {% endset %} {%- set newest_processed_order = dbt_utils.get_single_value(sql_statement, default="'2020-01-01'") -%} {% endif %} select *, last_order_at > '{{ newest_processed_order }}' as has_unprocessed_order from {{ ref('users') }} ``` * In this case, dbt doesn't know that `processed_orders` is a dependency because `execute` is false during parsing. * To address this, use a SQL comment along with the `ref` function — dbt will understand the dependency and the compiled query will still be valid: conditional\_ref.sql ```sql --Now that this ref is outside of the if block, it will be detected during parsing --depends_on: {{ ref('processed_orders') }} {% if execute %} {% set sql_statement %} select max(created_at) from {{ ref('processed_orders') }} {% endset %} {%- set newest_processed_order = dbt_utils.get_single_value(sql_statement, default="'2020-01-01'") -%} {% endif %} select *, last_order_at > '{{ newest_processed_order }}' as has_unprocessed_order from {{ ref('users') }} ``` tip To ensure dbt understands the dependency, use a SQL comment instead of a Jinja comment. Jinja comments (`{# ... #}`) *don't* work and are ignored by dbt's parser, meaning `ref` is never processed and resolved. SQL comments, however, (`--` or `/* ... */`) *do* work because dbt still evaluates Jinja inside SQL comments. --- ### About References The References section contains reference materials for developing with dbt, which includes dbt and dbt v1. Learn how to add more configurations to your dbt project or adapter, use properties for extra ability, refer to dbt commands, use powerful Jinja functions to streamline your dbt project, and understand how to use dbt artifacts.<br /> [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/dbt_project.yml.md) ###### [Project configurations](https://docs.getdbt.com/reference/dbt_project.yml.md) [Customize and configure your dbt project to optimize performance.](https://docs.getdbt.com/reference/dbt_project.yml.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/resource-configs.md) ###### [Platform-specific configurations](https://docs.getdbt.com/reference/resource-configs.md) [Learn how to optimize performance with data platform-specific configurations in dbt.](https://docs.getdbt.com/reference/resource-configs.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/configs-and-properties.md) ###### [Resource configurations and properties](https://docs.getdbt.com/reference/configs-and-properties.md) [Properties and configurations that provide extra abilities to your projects resources.](https://docs.getdbt.com/reference/configs-and-properties.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/semantic-layer-reference.md) ###### [Semantic Layer configurations](https://docs.getdbt.com/reference/semantic-layer-reference.md) [Complete configuration reference for semantic models, metrics, and dimensions in the dbt Semantic Layer.](https://docs.getdbt.com/reference/semantic-layer-reference.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/dbt-commands.md) ###### [dbt Commands](https://docs.getdbt.com/reference/dbt-commands.md) [Outlines the commands supported by dbt and their relevant flags.](https://docs.getdbt.com/reference/dbt-commands.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/jinja-file-support.md) ###### [Jinja support by file type](https://docs.getdbt.com/reference/jinja-file-support.md) [Which dbt file types support Jinja expressions, built-in functions, and custom macros.](https://docs.getdbt.com/reference/jinja-file-support.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) ###### [dbt Jinja functions and context variables](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) [Additional functions and variables to the Jinja context that are useful when working with a dbt project.](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) ###### [dbt Artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) [Information on dbt-generated Artifacts and how you can use them.](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/database-permissions/snowflake-permissions.md) ###### [Snowflake permissions artifacts](https://docs.getdbt.com/reference/database-permissions/snowflake-permissions.md) [Provides an example Snowflake database role permissions.](https://docs.getdbt.com/reference/database-permissions/snowflake-permissions.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/database-permissions/databricks-permissions.md) ###### [Databricks permissions artifacts](https://docs.getdbt.com/reference/database-permissions/databricks-permissions.md) [Provides an example Databricks database role permissions.](https://docs.getdbt.com/reference/database-permissions/databricks-permissions.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/database-permissions/redshift-permissions.md) ###### [Redshift permissions artifacts](https://docs.getdbt.com/reference/database-permissions/redshift-permissions.md) [Provides an example Redshift database role permissions.](https://docs.getdbt.com/reference/database-permissions/redshift-permissions.md) [![](/img/icons/computer.svg)](https://docs.getdbt.com/reference/database-permissions/postgres-permissions.md) ###### [Postgres permissions artifacts](https://docs.getdbt.com/reference/database-permissions/postgres-permissions.md) [Provides an example Postgres database role permissions.](https://docs.getdbt.com/reference/database-permissions/postgres-permissions.md) --- ### About return function **Args**: * `data`: The data to return to the caller The `return` function can be used in macros to return data to the caller. The type of the data (dict, list, int, etc) will be preserved through the `return` call. You can use the `return` function in the following ways within your macros: as an expression or as a statement. * Expression — Use an expression when the goal is to output a string from the macro. * Statement with a `do` tag — Use a statement with a `do` tag to execute the return function without generating an output string. This is particularly useful when you want to perform actions without necessarily inserting their results directly into the template. In the following example, `{{ return([1,2,3]) }}` acts as an *expression* that directly outputs a string, making it suitable for directly inserting returned values into SQL code. macros/get\_data.sql ```sql {% macro get_data() %} {{ return([1,2,3]) }} {% endmacro %} ``` Alternatively, you can use a statement with a [do](https://jinja.palletsprojects.com/en/3.0.x/extensions/#expression-statement) tag (or expression-statements) to execute the return function without generating an output string. In the following example ,`{% do return([1,2,3]) %}` acts as a *statement* that executes the return action but does not output a string: macros/get\_data.sql ```sql {% macro get_data() %} {% do return([1,2,3]) %} {% endmacro %} ``` models/my\_model.sql ```sql select -- getdata() returns a list! {% for i in get_data() %} {{ i }} {%- if not loop.last %},{% endif -%} {% endfor %} ``` --- ### About run_query macro The `run_query` macro provides a convenient way to run queries and fetch their results. It is a wrapper around the [statement block](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks.md), which is more flexible, but also more complicated to use. If you are new to `run_query`, refer to the [using Jinja guide](https://docs.getdbt.com/guides/using-jinja.md?step=8) for an example of working with the results of the `run_query` macro (step 8). warning [`run_query`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) runs SQL against the warehouse whenever dbt *compiles* your project with a live connection, including during [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) and [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) (by default). This is expected because compilation resolves your Jinja and macros. If you put DML or other side-effecting statements inside `run_query`, they run in those workflows too unless you scope them, for example using [`flags.WHICH`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#flagswhich). Refer to [How `run_query` runs during compilation and `dbt docs generate`](#run-query-compilation-and-docs). #### Args * `sql`: The SQL query to execute Returns a [Table](https://agate.readthedocs.io/page/api/table.html) object with the result of the query. If the specified query does not return results (for example, a DDL, DML, or maintenance query), then the return value will be `none`. (Applies to dbt v2.0 and later) #### dbt v2 type checking dbt v2 processes result sets with more strict null checking. This can cause failures when using `run_query` for DDL or maintenance operations (like `OPTIMIZE`, `VACUUM`). When a given result set returns null values in columns declared as non-nullable, v2 will fail - whereas dbt v1 would silently ignore it. For "fire and forget" operations where you don't need the result set, use a [statement block](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks.md) with `fetch_result=False` instead, as highlighted in this Databricks example: ```jinja {% macro run_optimize(table, zorder_fields) %} {% set zorder_str = zorder_fields | join(', ') %} {% set query %} OPTIMIZE {{ table }} {% if zorder_str | length > 0 %} ZORDER BY ({{ zorder_str }}) {% endif %} {% endset %} {% call statement('optimize', fetch_result=False) %} {{ query }} {% endcall %} {% endmacro %} ``` Using run\_query for the first time? Check out the section of the Getting Started guide on [using Jinja](https://docs.getdbt.com/guides/using-jinja.md#dynamically-retrieve-the-list-of-payment-methods) for an example of working with the results of the `run_query` macro! **Note:** The `run_query` macro will not begin a transaction automatically - if you wish to run your query inside of a transaction, please use `begin` and `commit` statements as appropriate. ##### Examples models/my\_model.sql ```jinja2 {% if execute %} {% set results = run_query('select 1 as id') %} {% else %} {% set results = none %} {% endif %} {% if results is not none %} {{ log(results.print_table(), info=True) }} {% endif %} {# do something with `results` here... #} ``` macros/run\_grants.sql ```jinja2 {% macro run_vacuum(table) %} {% set query %} vacuum table {{ table }} {% endset %} {% do run_query(query) %} {% endmacro %} ``` Here's an example of using this (though if you're using `run_query` to return the values of a column, check out the [get\_column\_values](https://github.com/dbt-labs/dbt-utils#get_column_values-source) macro in the dbt-utils package). models/my\_model.sql ```sql {% set payment_methods_query %} select distinct payment_method from app_data.payments order by 1 {% endset %} {% set results = run_query(payment_methods_query) %} {% if execute %} {# Return the first column #} {% set results_list = results.columns[0].values() %} {% else %} {% set results_list = [] %} {% endif %} select order_id, {% for payment_method in results_list %} sum(case when payment_method = '{{ payment_method }}' then amount end) as {{ payment_method }}_amount, {% endfor %} sum(amount) as total_amount from {{ ref('raw_payments') }} group by 1 ``` You can also use `run_query` to perform SQL queries that aren't select statements. macros/run\_vacuum.sql ```sql {% macro run_vacuum(table) %} {% set query %} vacuum table {{ table }} {% endset %} {% do run_query(query) %} {% endmacro %} ``` Use the `length` filter to verify whether `run_query` returned any rows or not. Wrap the logic in an [if execute](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) block so `run_query` does not run during parsing, when [`execute`](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) is `False`. ```sql {% if execute %} {% set results = run_query(payment_methods_query) %} {% if results|length > 0 %} -- do something with `results` here... {% else %} -- do fallback here... {% endif %} {% endif %} ``` #### How `run_query` runs during compilation and `dbt docs generate` When dbt *compiles* models and other resources with a live warehouse connection, it evaluates your Jinja and runs any `run_query()` calls that your project reaches. That is normal: introspective macros need the warehouse the same way other compilation steps do. Compilation happens during `dbt run` and `dbt build`, and also during `dbt compile`, `dbt docs generate` (when compilation runs), and other commands that compile the project—not only during steps where you might picture dbt "building" tables. [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) **compiles your project by default** (unless you pass [`--no-compile`](https://docs.getdbt.com/reference/commands/cmd-docs.md)). So `run_query()` runs during documentation generation under the same rules as other compile workflows, including for resources that are not part of a particular `dbt run` selection. ##### `execute` is `True` during compilation The [`execute`](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) context variable is `True` whenever dbt compiles with a connection, not only while materializing models during `dbt run`. A guard like `{% if execute %}` still allows `run_query()` to run during [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) and [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md), because those commands compile your project. Patterns such as `{% if execute and is_incremental() %}` change when incremental model SQL runs, but they do not turn off compilation itself—so they do not, on their own, skip `run_query()` during docs or compile unless your Jinja never calls it in those paths. If you want DML or other side-effecting SQL to run only for certain dbt commands, add another condition, for example [`flags.WHICH`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#flagswhich). ##### Scope side-effecting SQL with `flags.WHICH` Combine [`execute`](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) with [`flags.WHICH`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#flagswhich) so DML runs only when the active command is one you want (`run`, `build`, and so on), and not when dbt is compiling for `docs`, `compile`, or other commands where side effects would be surprising. Refer to the `flags.WHICH` table for the full list of command values. ```sql {% if execute and flags.WHICH in ['run', 'build'] %} {% do run_query('delete from my_scratch_table where session_id = ...') %} {% endif %} ``` Adjust the list of commands to match where your macro should run. --- ### About run_started_at variable `run_started_at` outputs the timestamp that this run started, e.g. `2017-04-21 01:23:45.678`. The `run_started_at` variable is a Python `datetime` object. As of 0.9.1, the timezone of this variable defaults to UTC. run\_started\_at\_example.sql ```sql select '{{ run_started_at.strftime("%Y-%m-%d") }}' as date_day from ... ``` To modify the timezone of this variable, use the `pytz` module: run\_started\_at\_utc.sql ```sql select '{{ run_started_at.astimezone(modules.pytz.timezone("America/New_York")) }}' as run_started_est from ... ``` --- ### About schemas variable `schemas` is a variable available in an `on-run-end` hook, representing a list of schemas that dbt built objects in on this run. If you do not use [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md), `schemas` will evaluate to your target schema, e.g. `['dbt_alice']`. If you use custom schemas, it will include these as well, e.g. `['dbt_alice', 'dbt_alice_marketing', 'dbt_alice_finance']`. The `schemas` variable is useful for granting privileges to all schemas that dbt builds relations in, like so (note this is Redshift specific syntax): dbt\_project.yml ```yaml ... on-run-end: - "{% for schema in schemas%}grant usage on schema {{ schema }} to group reporter;{% endfor%}" - "{% for schema in schemas %}grant select on all tables in schema {{ schema }} to group reporter;{% endfor%}" - "{% for schema in schemas %}alter default privileges in schema {{ schema }} grant select on tables to group reporter;{% endfor %}" ``` Want more in-depth instructions on the recommended way to grant privileges? We've written a full discourse article [here](https://discourse.getdbt.com/t/the-exact-grant-statements-we-use-in-a-dbt-project/430) --- ### About selected_resources context variable The `selected_resources` context variable contains a list of all the *nodes* selected by the current dbt command. Currently, this variable is not accessible when using the command `run-operation`. Warning! dbt actively builds the graph during the [parsing phase](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) of running dbt projects, so the `selected_resources` context variable will be empty during parsing. Please read the information on this page to effectively use this variable. ##### Usage The `selected_resources` context variable is a list of all the resources selected by the current dbt command selector. Its value depends on the usage of parameters like (Applies to dbt v1.12 and later) `--select` and `--exclude`. For a given run it will look like: ```json ["model.my_project.model1", "model.my_project.model2", "snapshot.my_project.my_snapshot"] ``` Each value corresponds to a key in the `nodes` object within the [graph](https://docs.getdbt.com/reference/dbt-jinja-functions/graph.md) context variable. It can be used in macros in a `pre-hook`, `post-hook`, `on-run-start` or `on-run-end` to evaluate what nodes are selected and trigger different logic whether a particular node is selected or not. check-node-selected.sql ```sql /* Check if a given model is selected and trigger a different action, depending on the result */ {% if execute %} {% if 'model.my_project.model1' in selected_resources %} {% do log("model1 is included based on the current selection", info=true) %} {% else %} {% do log("model1 is not included based on the current selection", info=true) %} {% endif %} {% endif %} /* Example output when running the code in on-run-start when doing `dbt build`, including all models --------------------------------------------------------------- model1 is included based on the current selection Example output when running the code in on-run-start when doing `dbt run --select model2` --------------------------------------------------------------- model1 is not included based on the current selection */ ``` --- ### About set context method tip Not to be confused with the `{% set foo = "bar" ... %}` expression in Jinja, which defines a variable. For examples of constructing SQL strings with `{% set %}` (and why `{{ }}` should not be nested inside quoted strings), see [Don’t nest your curlies](https://docs.getdbt.com/best-practices/dont-nest-your-curlies.md). You can use the `set` context method to convert any iterable to a sequence of iterable elements that are unique (a set). **Args**: * `value`: The iterable to convert (for example, a list) * `default`: A default value to return if the `value` argument is not a valid iterable ##### Usage ```text {% set my_list = [1, 2, 2, 3] %} {% set my_set = set(my_list) %} {% do log(my_set) %} {# {1, 2, 3} #} ``` ```text {% set my_invalid_iterable = 1234 %} {% set my_set = set(my_invalid_iterable) %} {% do log(my_set) %} {# None #} ``` ```text {% set email_id = "'admin@example.com'" %} ``` ##### set\_strict The `set_strict` context method can be used to convert any iterable to a sequence of iterable elements that are unique (a set). The difference to the `set` context method is that the `set_strict` method will raise an exception on a `TypeError`, if the provided value is not a valid iterable and cannot be converted to a set. **Args**: * `value`: The iterable to convert (for example, a list) ```text {% set my_list = [1, 2, 2, 3] %} {% set my_set = set(my_list) %} {% do log(my_set) %} {# {1, 2, 3} #} ``` ```text {% set my_invalid_iterable = 1234 %} {% set my_set = set_strict(my_invalid_iterable) %} {% do log(my_set) %} Compilation Error in ... (...) 'int' object is not iterable ``` --- ### About source function ```sql select * from {{ source("source_name", "table_name") }} ``` #### Definition This function: * Returns a [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) for a [source](https://docs.getdbt.com/docs/build/sources.md) * Creates dependencies between a source and the current model, which is useful for documentation and [node selection](https://docs.getdbt.com/reference/node-selection/syntax.md) * Compiles to the full object name in the database #### Related guides * [Using sources](https://docs.getdbt.com/docs/build/sources.md) #### Arguments * `source_name`: The `name:` defined under a `sources:` key * `table_name`: The `name:` defined under a `tables:` key #### Example Consider a source defined as follows: models/\<filename>.yml ```yaml sources: - name: jaffle_shop # this is the source_name database: raw tables: - name: customers # this is the table_name - name: orders ``` Select from the source in a model: models/orders.sql ```sql select ... from {{ source('jaffle_shop', 'customers') }} left join {{ source('jaffle_shop', 'orders') }} using (customer_id) ``` --- ### About statement blocks (Applies to dbt v2.0 and later) When to use statement blocks in dbt v2 For queries where you need to fetch results (for example, when your macro or Jinja code needs to use data returned from the database), you can use either `statement` blocks with `fetch_result=True` or the [`run_query` macro](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md). For DDL or utility operations (like `OPTIMIZE`, `VACUUM`, or maintenance queries), use `statement` blocks with `fetch_result=False` when you don't need to access the results in Jinja. This avoids issues with dbt v2's strict type checking, which can fail when processing result sets that contain `NULL` values in columns declared as non-nullable. `statement`s are SQL queries that hit the database and return results to your Jinja context. Here’s an example of a `statement` which gets all of the states from a users table. get\_states\_statement.sql ```sql -- depends_on: {{ ref('users') }} {%- call statement('states', fetch_result=True) -%} select distinct state from {{ ref('users') }} {%- endcall -%} ``` The signature of the `statement` block looks like this: ```text statement(name=None, fetch_result=False, auto_begin=True) ``` When executing a `statement`, dbt needs to understand how to resolve references to other dbt models or resources. If you are already `ref`ing the model outside of the statement block, the dependency will be automatically inferred, but otherwise you will need to [force the dependency](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#forcing-dependencies) with `-- depends_on`.  Example using -- depends\_on ```sql -- depends_on: {{ ref('users') }} {% call statement('states', fetch_result=True) -%} select distinct state from {{ ref('users') }} /* The unique states are: {{ load_result('states')['data'] }} */ {%- endcall %} ```  Example using ref() function ```sql {% call statement('states', fetch_result=True) -%} select distinct state from {{ ref('users') }} /* The unique states are: {{ load_result('states')['data'] }} */ {%- endcall %} select id * 2 from {{ ref('users') }} ``` **Args**: * `name` (string): The name for the result set returned by this statement * `fetch_result` (bool): If True, load the results of the statement into the Jinja context * `auto_begin` (bool): If True, open a transaction if one does not exist. If false, do not open a transaction. Once the statement block has executed, the result set is accessible via the `load_result` function. The result object includes three keys: * `response`: Structured object containing metadata returned from the database, which varies by adapter. E.g. success `code`, number of `rows_affected`, total `bytes_processed`, etc. Comparable to `adapter_response` in the [Result object](https://docs.getdbt.com/reference/dbt-classes.md#result-objects). * `data`: Pythonic representation of data returned by query (arrays, tuples, dictionaries). * `table`: [Agate](https://agate.readthedocs.io/page/api/table.html) table representation of data returned by query. For the above statement, that could look like: load\_states.sql ```sql {%- set states = load_result('states') -%} {%- set states_data = states['data'] -%} {%- set states_status = states['response'] -%} ``` The contents of the returned `data` field is a matrix. It contains a list rows, with each row being a list of values returned by the database. For the above example, this data structure might look like: states.sql ```python >>> log(states_data) [ ['PA'], ['NY'], ['CA'], ... ] ``` (Applies to dbt v2.0 and later) #### Fire and forget operations For DDL or utility operations where you don't need the results (because you don't use the returned rows in Jinja), set `fetch_result=False`. This is the recommended pattern for operations like `OPTIMIZE` or `VACUUM` on Databricks, which return result sets that may contain null values in non-nullable columns. macros/optimize\_table.sql ```jinja {% macro optimize_table(table, zorder_fields=[]) %} {% set zorder_str = zorder_fields | join(', ') %} {% set query %} OPTIMIZE {{ table }} {% if zorder_str | length > 0 %} ZORDER BY ({{ zorder_str }}) {% endif %} {% endset %} {% call statement('optimize', fetch_result=False) %} {{ query }} {% endcall %} {% endmacro %} ``` You can use this macro in a post-hook: dbt\_project.yml ```yaml models: my_project: +post-hook: - "{{ optimize_table(this, ['customer_id', 'order_date']) }}" ``` --- ### About target variables The `target` variable contains information about your connection to the warehouse. * **dbt v1:** These values are based on the target defined in your [profiles.yml](https://docs.getdbt.com/docs/local/profiles.yml.md) file. Please note that for certain adapters, additional configuration steps may be required. Refer to the [set up page](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md) for your data platform. * **dbt** To learn more about setting up your adapter in dbt, refer to [About data platform connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * **[Orchestrator](https://docs.getdbt.com/docs/deploy/job-scheduler.md)**: `target.name` is defined per job as described in [Custom target names](https://docs.getdbt.com/docs/build/custom-target-names.md). For other attributes, values are defined by the deployment connection. To check these values, click **Deploy** and select **Environments**. Then, select the relevant deployment environment, and click **Settings**. * **[Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md)**: These values are defined by your connection and credentials. To edit these values, click on your account name in the left side menu and select **Account settings**. Then, click **Credentials**. Select and edit a project to set up the credentials and target name. Some configurations are shared between all adapters, while others are adapter-specific. You can also use the [`--target` flag](#using-the---target-flag) to set the active target when running dbt commands. #### Common | Variable | Example | Description | | --------------------- | ------------ | --------------------------------------------------------------------------------------------------- | | `target.profile_name` | jaffle\_shop | The name of the active profile | | `target.name` | dev | Name of the active target | | `target.schema` | dbt\_alice | Name of the dbt schema (or, dataset on BigQuery) | | `target.type` | postgres | The active adapter being used. One of "postgres", "snowflake", "bigquery", "redshift", "databricks" | | `target.threads` | 4 | The number of threads in use by dbt | #### Adapter-specific ##### Snowflake | Variable | Example | Description | | ------------------ | --------------- | ------------------------------------------ | | `target.database` | RAW | Database name specified in active target. | | `target.warehouse` | TRANSFORM | Name of the Snowflake virtual warehouse | | `target.user` | TRANSFORM\_USER | The user specified in the active target | | `target.role` | TRANSFORM\_ROLE | The role specified in the active target | | `target.account` | abc123 | The account specified in the active target | ##### Postgres/Redshift | Variable | Example | Description | | --------------- | --------------------------------------- | ----------------------------------------- | | `target.dbname` | analytics | Database name specified in active target. | | `target.host` | abc123.us-west-2.redshift.amazonaws.com | The host specified in active target | | `target.user` | dbt\_user | The user specified in the active target | | `target.port` | 5439 | The port specified in the active profile | ##### BigQuery | Variable | Example | Description | | ---------------- | ---------- | ------------------------------------------- | | `target.project` | abc-123 | The project specified in the active profile | | `target.dataset` | dbt\_alice | The dataset the active profile | #### Using the --target flag Use the `--target` flag when running dbt commands to set the active target and its associated `target.name` value: ```bash dbt run --target dev ``` ```bash dbt run --target prod ``` You can use the `--target` flag with any dbt command to override the default target specified in your `profiles.yml` file. This is useful for running the same dbt project against different environments (like dev, staging, or prod) without changing your configuration files. #### Examples ##### Use `target.name` to limit data in dev As long as you use sensible target names, you can perform conditional logic to limit data when working in dev. ```sql select * from source('web_events', 'page_views') {% if target.name == 'dev' %} where created_at >= dateadd('day', -3, current_date) {% endif %} ``` ##### Use `target.name` to change your source database If you have specific Snowflake databases configured for your dev/qa/prod environments, you can set up your sources to compile to different databases depending on your environment. ```yml sources: - name: source_name database: | {%- if target.name == "dev" -%} raw_dev {%- elif target.name == "qa" -%} raw_qa {%- elif target.name == "prod" -%} raw_prod {%- else -%} invalid_database {%- endif -%} schema: source_schema ``` --- ### About the doc function You can specify documentation text in a docs block, then use the `doc()` Jinja function in description fields as a way to reuse the same text in multiple places. You can only use the `doc()` Jinja function in properties YAML files for resources with description properties. For example, models, model columns, sources, source tables, source columns, and so on. The `doc()` Jinja function, which is analogous to `ref()`, looks up the named docs block (for example, `{% docs orders %} ... {% enddocs %}` in a `docs.md` file) and returns its rendered content. For more information, refer to the [Documentation guide](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md). #### Usage In dbt, column descriptions can be defined directly in a model's properties YAML file. These descriptions are written as plain text and are associated with a specific column. ##### Manually adding descriptions models/orders.yml ```yaml models: - name: orders columns: - name: order_total_cents description: "Total order amount in cents. This value is always a positive integer and excludes taxes and shipping." ``` When you run `dbt docs generate` and view the docs site, this text appears exactly as written but only for the `order_total_cents` column of the `orders` model. ##### Reusing descriptions with doc() To avoid repeating the same description across multiple models or columns, dbt lets you define documentation separately and reference it using the `doc()` function. With this approach, documentation is written once in a markdown file using a named docs block, and then reused wherever needed in your project. This helps ensure consistency and makes it easier to maintain shared definitions over time. First, define a reusable documentation block in a `docs.md`: models/docs/example.md ```text {% docs order_total_cents %} Total order amount in cents. This value is always a positive integer and excludes taxes and shipping. {% enddocs %} ``` This defines a docs block named `order_total_cents`. Reference it with `doc('order_total_cents')` in description fields wherever you need the same text. Next, reference this documentation in your properties YAML file: models/orders.yml ```yaml models: - name: orders columns: - name: order_total_cents description: "{{ doc('order_total_cents') }}" ``` When you run `dbt docs generate`, dbt resolves the `doc()` reference by looking up the corresponding docs block and injecting its content into the generated documentation. As a result, the column description displays the text defined in the markdown file, rather than inline YAML. #### Avoid duplicate names Docs block names must be unique within your project. If you define multiple `{% docs %}` blocks with the same name, dbt cannot reliably determine which block to use when `doc('DOCS_BLOCK_NAME')` is called. (Applies to dbt v2.0 and later) In dbt v2, duplicate docs block names are not allowed. If duplicates are found, dbt reports the conflicting files and surfaces a warning. Rename one block so each docs block name is unique. For more information, refer to [Stricter evaluation of duplicate docs blocks](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md?version=2.0#stricter-evaluation-of-duplicate-docs-blocks). models/docs/example.md ```text {% docs order_total_cents %} Total order amount in cents. This value is always a positive integer and excludes taxes and shipping. {% enddocs %} {% docs order_total_cents %} Total order amount in cents, excluding taxes and shipping fees. {% enddocs %} ``` ```text dbt found two docs with the same name: 'order_total_cents' in files: 'models/docs/example.md' and 'models/docs/example.md' ``` --- ### About the info_schema macro Available in v2 `{{ info_schema('<view_name>') }}` is the supported way to reference [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) tables in a [check](https://docs.getdbt.com/docs/build/checks.md) or with [`dbt show --inline`](https://docs.getdbt.com/docs/build/dbt-information-schema.md#querying-with-dbt-show). When you pass the name of the view you want to query (for example, `{{ info_schema('models') }}`), the macro reads from a logical view layer built at parse time. For example, to find models without a description: ```sql select name from {{ info_schema('models') }} where description = '' ``` For the full list of columns available for each view, refer to [Columns available for checks](https://docs.getdbt.com/reference/info-schema.md#columns-available-for-checks). note Checks can only access parse-time project metadata. Only views whose columns are fully populated at parse time are available. Passing a view name that doesn't exist or isn't available at parse time causes the check to fail with a message listing what is available. --- ### about this `this` is the database representation of the current model. It is useful when: * Defining a `where` statement within [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * Using [pre or post hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) `this` is a [Relation](https://docs.getdbt.com/reference/dbt-classes.md#relation), and as such, properties such as `{{ this.database }}` and `{{ this.schema }}` compile as expected. * Note — Prior to dbt v1.6, Studio IDE returns `request` as the result of `{{ ref.identifier }}`. `this` can be thought of as equivalent to `ref('<the_current_model>')`, and is a neat way to avoid circular dependencies. #### Examples ##### Configuring incremental models models/stg\_events.sql ```sql {{ config(materialized='incremental') }} select *, my_slow_function(my_column) from raw_app_data.events {% if is_incremental() %} where event_time > (select max(event_time) from {{ this }}) {% endif %} ``` --- ### About thread_id The `thread_id` outputs an identifier for the current Python thread that is executing a node, like `Thread-1`. This value is useful when auditing or analyzing dbt invocation metadata. It corresponds to the `thread_id` within the [`Result` object](https://docs.getdbt.com/reference/dbt-classes.md#result-objects) and [`run_results.json`](https://docs.getdbt.com/reference/artifacts/run-results-json.md). If available, the `thread_id` is: * available in the compilation context of [`query-comment`](https://docs.getdbt.com/reference/project-configs/query-comment.md) * included in the `info` dictionary in dbt [events and logs](https://docs.getdbt.com/reference/events-logging.md#info) * included in the `metadata` dictionary in [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md#common-metadata) * included as a label in all BigQuery jobs that dbt originates --- ### About tojson context method The `tojson` context method can be used to serialize a Python object primitive, eg. a `dict` or `list` to a JSON string. **Args**: * `value`: The value to serialize to JSON (required) * `default`: A default value to return if the `value` argument cannot be serialized (optional) ##### Usage: ```text {% set my_dict = {"abc": 123} %} {% set my_json_string = tojson(my_dict) %} {% do log(my_json_string) %} ``` --- ### About toyaml context method The `toyaml` context method can be used to serialize a Python object primitive, eg. a `dict` or `list` to a YAML string. **Args**: * `value`: The value to serialize to YAML (required) * `default`: A default value to return if the `value` argument cannot be serialized (optional) ##### Usage: ```text {% set my_dict = {"abc": 123} %} {% set my_yaml_string = toyaml(my_dict) %} {% do log(my_yaml_string) %} ``` --- ### About unit tests property Did you know... Available from dbt v1.8 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Unit tests validate your SQL modeling logic on a small set of static inputs before you materialize your full model in production. They support a test-driven development approach, improving both the efficiency of developers and reliability of code. To run only your unit tests, use the command: `dbt test --select test_type:unit` #### Prerequisites * We currently only support unit testing SQL models. * We currently only support adding unit tests to models in your *current* project. * We currently *don't* support unit testing models that use the [`materialized view`](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) materialization. * We currently *don't* support unit testing models that use recursive SQL. * We currently *don't* support unit testing models that use introspective queries. * If your model has multiple versions, by default the unit test will run on *all* versions of your model. Read [unit testing versioned models](https://docs.getdbt.com/reference/resource-properties/unit-testing-versions.md) for more information. * Unit tests must be defined in a YML file in your [`models/` directory](https://docs.getdbt.com/reference/project-configs/model-paths.md). * Table names must be aliased in order to unit test `join` logic. * Include all [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) or [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) model references in the unit test configuration as `input`s to avoid "node not found" errors during compilation. Unit tests are discovered from `model-paths` (by default, the `models/` directory), so define them alongside your models in a `.yml` file under your `model-paths`. Don't define unit test YAML in the `tests/` directory, which is reserved for [data tests](https://docs.getdbt.com/docs/build/data-tests.md). models/schema.yml ```yaml unit_tests: - name: <test-name> # this is the unique name of the test model: <model-name> versions: #optional include: <list-of-versions-to-include> #optional exclude: <list-of-versions-to-exclude> #optional config: meta: {dictionary} tags: <string> | [<string>] enabled: {boolean} # optional. v1.9 or higher. If not configured, defaults to `true` compute: local | remote # optional. v2.0 or higher. Defaults to `remote`. Requires DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS=true. See Run unit tests locally given: - input: <ref_or_source_call> # optional for seeds format: dict | csv | sql # either define rows inline or name of fixture rows: {dictionary} | <string> fixture: <fixture-name> # SQL or csv - input: ... # declare additional inputs expect: format: dict | csv | sql # either define rows inline or use the name of a fixture rows: {dictionary} | <string> fixture: <fixture-name> # SQL or csv overrides: # optional: configuration for the dbt execution environment macros: is_incremental: true | false dbt_utils.current_timestamp: <string> # ... any other Jinja function from https://docs.getdbt.com/reference/dbt-jinja-functions # ... any other context property vars: {dictionary} env_vars: {dictionary} - name: <test-name> ... # declare additional unit tests ``` #### Examples models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address # this is the unique name of the test model: dim_customers # name of the model I'm unit testing given: # the mock data for your inputs - input: ref('stg_customers') rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') rows: - {tld: example.com} - {tld: gmail.com} expect: # the expected output given the inputs above rows: - {email: cool@example.com, is_valid_email_address: true} - {email: cool@unknown.com, is_valid_email_address: false} - {email: badgmail.com, is_valid_email_address: false} - {email: missingdot@gmailcom, is_valid_email_address: false} ``` models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address # this is the unique name of the test model: dim_customers # name of the model I'm unit testing given: # the mock data for your inputs - input: ref('stg_customers') rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: csv rows: | tld example.com gmail.com expect: # the expected output given the inputs above format: csv fixture: valid_email_address_fixture_output ``` models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address # this is the unique name of the test model: dim_customers # name of the model I'm unit testing given: # the mock data for your inputs - input: ref('stg_customers') rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: sql rows: | select 'example.com' as tld union all select 'gmail.com' as tld expect: # the expected output given the inputs above format: sql fixture: valid_email_address_fixture_output ``` --- ### About var function (Applies to dbt v1.12 and later) You can provide variables from `vars.yml` or `dbt_project.yml` to models during compilation. These variables are useful for configuring packages for deployment in multiple environments, or for defining values that should be used across multiple models within a package. Variables defined in `vars.yml` or `dbt_project.yml` act as project-wide defaults. You can override them at runtime using the `--vars` command-line argument. For example, when testing with a different date range or running models with environment-specific settings without changing your model logic. To retrieve a variable inside a model, hook, or macro, use the `var()` function. The `var()` function returns the value defined in your project or passed using `--vars`, based on precedence. You can use `var()` anywhere dbt renders Jinja during compilation, including most `.sql` and `.yml` files in your project. It does not work in configuration files that dbt reads before compilation, such as [`profiles.yml`](https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context.md) or [`packages.yml`](<https://docs.getdbt.com/reference/dbt-jinja-functions/packages.yml context.md>). To add a variable to a model, use the `var()` function: my\_model.sql ```sql select * from events where event_type = '{{ var("event_type") }}' ``` If you try to run this model without supplying an `event_type` variable, you'll receive a compilation error that looks like this: ```text Encountered an error: ! Compilation error while compiling model package_name.my_model: ! Required var 'event_type' not found in config: Vars supplied to package_name.my_model = { } ``` (Applies to dbt v1.12 and later) To define a variable in your project, add the `vars:` config to a dedicated `vars.yml` file or to your `dbt_project.yml` file. `vars.yml` is parsed *before* `dbt_project.yml`, so you can reference variables from `vars.yml` in `dbt_project.yml` using `{{ var('...') }}`. ##### vars.yml vars.yml ```yaml vars: event_type: activation ``` ##### dbt\_project.yml You can define variables in `dbt_project.yml`, or reference variables from `vars.yml` (for example, in your `models` config): dbt\_project.yml ```yaml name: my_dbt_project version: 1.0.0 config-version: 2 # Option 1: Define variables here vars: event_type: activation # Option 2: Reference a variable from vars.yml models: my_dbt_project: +schema: "{{ var('event_type') }}" ``` You cannot define variables in both `vars.yml` and `dbt_project.yml`; you can only use one or the other. If both files contain a `vars` block with definitions, dbt raises an error. See the docs on [using variables](https://docs.getdbt.com/docs/build/project-variables.md) for more information on how to define variables in your dbt project. ##### Variable default values The `var()` function takes an optional second argument, `default`. If this argument is provided, then it will be the default value for the variable if one is not explicitly defined. my\_model.sql ```sql -- Use 'activation' as the event_type if the variable is not defined. select * from events where event_type = '{{ var("event_type", "activation") }}' ``` ##### Command line variables The `dbt_project.yml` file is a great place to define variables that rarely change. When you need to override a variable for a specific run, use the `--vars` command line option. For example, when you want to test with a different date range, run models with environment-specific settings, or adjust behavior dynamically. Use `--vars` to pass one or more variables to a dbt command. Provide the argument as a YAML dictionary string. For example: ```shell $ dbt run --vars '{"event_type": "signup"}' ``` You can use the same `--vars` syntax with other dbt commands, such as `dbt snapshot`: ```shell $ dbt snapshot --select my_snapshot --vars '{"cutoff_date": "2026-01-01"}' ``` Inside a model, snapshot, or macro, access the value using the `var()` function: ```shell select '{{ var("event_type") }}' as event_type ``` When you pass variables using `--vars`, you can access them anywhere you use the `var()` function in your project. You can pass multiple variables at once: ```shell $ dbt run --vars '{event_type: signup, region: us}' ``` If only one variable is being set, the brackets are optional: ```shell $ dbt run --vars 'event_type: signup' ``` The `--vars` argument accepts a YAML dictionary as a string on the command line. YAML is convenient because it does not require strict quoting as with JSON. Both of the following are valid and equivalent: ```shell $ dbt run --vars '{"key": "value", "date": 20180101}' $ dbt run --vars '{key: value, date: 20180101}' ``` Variables defined using `--var`, override values defined in `dbt_project.yml`. This makes `--vars` useful for temporarily overriding configuration without changing your committed project files. For the complete order of precedence (including package-scoped variables and default values defined in `var()`), see [Variable precedence](https://docs.getdbt.com/docs/build/project-variables.md#variable-precedence). --- ### About zip context method You can use the `zip` context method to return an iterator of tuples, where the i-th tuple contains the i-th element from each of the argument iterables. For more information, see [Python docs](https://docs.python.org/3/library/functions.html#zip). **Args**: * `*args`: Any number of iterables * `default`: A default value to return if `*args` is not iterable ##### Usage ```text {% set my_list_a = [1, 2] %} {% set my_list_b = ['alice', 'bob'] %} {% set my_zip = zip(my_list_a, my_list_b) | list %} {% do log(my_zip) %} {# [(1, 'alice'), (2, 'bob')] #} ``` ```text {% set my_list_a = 12 %} {% set my_list_b = ['alice', 'bob'] %} {% set my_zip = zip(my_list_a, my_list_b, default = []) | list %} {% do log(my_zip) %} {# [] #} ``` ##### zip\_strict You can use the `zip_strict` context to return an iterator of tuples, just like `zip`. The difference to the `zip` context method is that the `zip_strict` method will raise an exception on a `TypeError`, if one of the provided values is not a valid iterable. **Args**: * `value`: The iterable to convert (e.g. a list) ```text {% set my_list_a = [1, 2] %} {% set my_list_b = ['alice', 'bob'] %} {% set my_zip = zip_strict(my_list_a, my_list_b) | list %} {% do log(my_zip) %} {# [(1, 'alice'), (2, 'bob')] #} ``` ```text {% set my_list_a = 12 %} {% set my_list_b = ['alice', 'bob'] %} {% set my_zip = zip_strict(my_list_a, my_list_b) %} Compilation Error in ... (...) 'int' object is not iterable ``` --- ### access models/\<schema>.yml ```yml models: - name: model_name config: access: private | protected | public # changed to config in v1.10 ``` You can apply `access` modifiers in config files, including the `dbt_project.yml`, or to models one-by-one in `properties.yml`. Applying `access` configs to a subfolder modifies the default for all models in that subfolder, so make sure you intend for this behavior. When setting individual model access, a group or subfolder might contain a variety of access levels, so when you designate a model with `access: public` make sure you intend for this behavior. Note that for backwards compatibility, `access` is supported as a top-level key, but without the capabilities of config inheritance. There are multiple approaches to configuring access: * In `properties.yml` using the older method (dbt v 1.9 and older), top-level — directly under the model: models/properties\_my\_public\_model.yml ```yml models: - name: my_public_model access: public ``` * In `properties.yml` using the latest method (for v1.10 or higher). Use either the older method or the latest method, but not both for the same model: models/properties\_my\_public\_model.yml ```yml models: - name: my_public_model config: access: public ``` * In `dbt_project.yml`: dbt\_project.yml ```yml models: my_project_name: subfolder_name: +group: my_group +access: private # sets default for all models in this subfolder ``` * In the `my_public_model.sql` file: models/my\_public\_model.sql ```sql -- models/my_public_model.sql {{ config(access = "public") }} select ... ``` After you define `access`, rerun a production job to apply the change. #### Definition The access level of the model you are declaring properties for. Some models (not all) are designed to be referenced through the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function across [groups](https://docs.getdbt.com/docs/build/groups.md). | Access | Referenceable by | | --------- | ----------------------------------------------------------------------------------------- | | private | Same group | | protected | Same project/package | | public | Any group, package, or project. When defined, rerun a production job to apply the change. | If you try to reference a model outside of its supported access, you will see an error: ```shell dbt run -s marketing_model ... dbt.exceptions.DbtReferenceError: Parsing Error Node model.jaffle_shop.marketing_model attempted to reference node model.jaffle_shop.finance_model, which is not allowed because the referenced node is private to the finance group. ``` (Applies to dbt v1.12 and later) When a macro is invoked with `dbt run-operation`, dbt doesn't enforce model access or group controls, so you can use `ref()` to reference all models (including `private` and `protected` ones). #### Default By default, all models are "protected." This means that other models in the same project can reference them. #### Related docs * [Model Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) * [Group configuration](https://docs.getdbt.com/reference/resource-configs/group.md) --- ### Advanced configuration usage #### Alternative SQL file config syntax Some configurations may contain characters (e.g. dashes) that cannot be parsed as a Jinja argument. For example, the following would return an error: ```sql {{ config( post-hook="grant select on {{ this }} to role reporter", materialized='table' ) }} select ... ``` While dbt provides an alias for any core configurations (for example, you should use `pre_hook` instead of `pre-hook` in a config block), your dbt project may contain custom configurations without aliases. If you want to specify these configurations inside of a model, use the alternative config block syntax: models/events/base/base\_events.sql ```sql {{ config({ "post-hook": "grant select on {{ this }} to role reporter", "materialized": "table" }) }} select ... ``` --- ### alias Specify a custom alias for a model, data test, snapshot, or seed and give it a more user-friendly name in the database. ##### Models Specify a custom alias for a model in your project YAML file (`dbt_project.yml`), properties YAML file (for example, `models/properties.yml`) config, or in a SQl file config block. For example, if you have a model that calculates `sales_total` and want to give it a more user-friendly alias, you can alias it as shown in the following examples. In the `dbt_project.yml` file, the following example sets a default `alias` for the `sales_total` model at the project level: dbt\_project.yml ```yml models: your_project: sales_total: +alias: sales_dashboard ``` The following specifies an `alias` as part of the `models/properties.yml` file metadata, useful for centralized configuration: models/properties.yml ```yml models: - name: sales_total config: alias: sales_dashboard ``` The following assigns the `alias` directly in the In `models/sales_total.sql` file: models/sales\_total.sql ```sql {{ config( alias="sales_dashboard" ) }} ``` This would return `analytics.finance.sales_dashboard` in the database, instead of the default `analytics.finance.sales_total`. ##### Seeds Configure a seed's alias in your project file (`dbt_project.yml`) or a properties file config (for example, `seeds/properties.yml`). The following examples demonstrate how to `alias` a seed named `product_categories` to `categories_data`. In the `dbt_project.yml` file at the project level: dbt\_project.yml ```yml seeds: your_project: product_categories: +alias: categories_data ``` In the `seeds/properties.yml` file: seeds/properties.yml ```yml seeds: - name: product_categories config: alias: categories_data ``` This would return the name `analytics.finance.categories_data` in the database. In the following second example, the seed at `seeds/country_codes.csv` will be built as a table named `country_mappings`. dbt\_project.yml ```yml seeds: jaffle_shop: country_codes: +alias: country_mappings ``` ##### Snapshots Configure a snapshots's alias in your project YAML file (`dbt_project.yml` ), properties YAML file (for example, `snapshots/snapshot_name.yml`), or in a SQL file config block for the model. The following examples demonstrate how to `alias` a snapshot named `your_snapshot` to `the_best_snapshot`. In the `dbt_project.yml` file at the project level: dbt\_project.yml ```yml snapshots: your_project: your_snapshot: +alias: the_best_snapshot ``` In the `snapshots/snapshot_name.yml` file: snapshots/snapshot\_name.yml ```yml snapshots: - name: your_snapshot_name config: alias: the_best_snapshot ``` In `snapshots/your_snapshot.sql` file: snapshots/your\_snapshot.sql ```sql {{ config( alias="the_best_snapshot" ) }} ``` This would build your snapshot to `analytics.finance.the_best_snapshot` in the database. ##### Tests Configure a data test's alias in your project YAML file (`dbt_project.yml` ), properties YAML file (for example, `models/properties.yml`) file, or in a SQL file config block for the model. The following examples demonstrate how to `alias` a unique data test named `order_id` to `unique_order_id_test` to identify a specific data test. In the `dbt_project.yml` file at the project level: dbt\_project.yml ```yml data_tests: your_project: +alias: unique_order_id_test ``` In the `models/properties.yml` file: models/properties.yml ```yml models: - name: orders columns: - name: order_id data_tests: - unique: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. alias: unique_order_id_test ``` In `tests/unique_order_id_test.sql` file: tests/unique\_order\_id\_test.sql ```sql {{ config( alias="unique_order_id_test", severity="error" ) }} ``` When using [`store_failures_as`](https://docs.getdbt.com/reference/resource-configs/store_failures_as.md), this would return the name `analytics.dbt_test__audit.orders_order_id_unique_order_id_test` in the database. #### Definition Optionally specify a custom alias for a [model](https://docs.getdbt.com/docs/build/models.md), [data test](https://docs.getdbt.com/docs/build/data-tests.md), [snapshot](https://docs.getdbt.com/docs/build/snapshots.md), or [seed](https://docs.getdbt.com/docs/build/seeds.md). When dbt creates a relation (table/view) in a database, it creates it as: `{{ database }}.{{ schema }}.{{ identifier }}`, e.g. `analytics.finance.payments` The standard behavior of dbt is: * If a custom alias is *not* specified, the identifier of the relation is the resource name (i.e. the filename). * If a custom alias is specified, the identifier of the relation is the `{{ alias }}` value. **Note** With an [ephemeral model](https://docs.getdbt.com/docs/build/materializations.md), dbt will always apply the prefix `__dbt__cte__` to the CTE identifier. This means that if an alias is set on an ephemeral model, then its CTE identifier will be `__dbt__cte__{{ alias }}`, but if no alias is set then its identifier will be `__dbt__cte__{{ filename }}`. To learn more about changing the way that dbt generates a relation's `identifier`, read [Using Aliases](https://docs.getdbt.com/docs/build/custom-aliases.md). --- ### allow_clones profiles.yml ```yaml my_project: outputs: prod: type: snowflake # ... other connection settings allow_clones: true | false target: dev ``` #### Definition `allow_clones` is a profile-level setting that controls whether dbt State is allowed to clone tables into a target environment. By default, dbt State can clone a table from any environment — including cloning a dev table into prod when the data and logic match. This behavior is intentional: if you built a model in dev and nothing changed, cloning it to prod saves compute compared to rebuilding. Setting `allow_clones: false` on a target tells dbt State to skip clone candidates and run full builds instead, even if a matching table exists in another target. This setting affects only the target it is set on. Other targets in the same profile are unaffected. For dbt platform users, set `allow_clones` as an [extended attribute](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the [connection profile](https://docs.getdbt.com/docs/platform/about-profiles.md) for a deployment environment: ```yaml allow_clones: true | false ``` #### Default `true`. When omitted, cloning is enabled and dbt State clones from any environment where a matching table exists. #### When to disable caution Setting `allow_clones: false` reduces the efficiency gains from dbt State. Nodes that dbt State would otherwise clone instead run as full builds. Consider whether this restriction is necessary for your environment. You typically don't need to set `allow_clones: false`. dbt State only clones when the source table's SQL logic matches and the table's data is newer. Because this is based on the table's modification date (not by directly comparing table contents), manually editing a table outside of dbt can give it a newer modification date without actually matching what a full build would produce. It is safe to leave `allow_clones` enabled as long as you always change table contents through dbt, rather than editing tables manually Disabling `allow_clones` is useful in regulated environments where policy prohibits any data from a development schema being written to production, regardless of correctness (for example, environments subject to data governance or audit requirements where the origin of a production table must be a warehouse-executed SQL statement rather than a clone). #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### Amazon Athena configurations #### Models ##### Table configuration | Parameter | Default | Description | | ------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `external_location` | None | The full S3 path to where the table is saved. It only works with incremental models. It doesn't work with Hive tables with `ha` set to `true`. | | `partitioned_by` | None | An array list of columns by which the table will be partitioned. Currently limited to 100 partitions. | | `bucketed_by` | None | An array list of the columns to bucket data. Ignored if using Iceberg. For Iceberg, refer to [Iceberg bucketing](#iceberg-bucketing). | | `bucket_count` | None | The number of buckets for bucketing your data. This parameter is ignored if using Iceberg. | | `table_type` | Hive | The type of table. Supports `hive` or `iceberg`. | | `ha` | False | Build the table using the high-availability method. Only available for Hive tables. | | `format` | Parquet | The data format for the table. Supports `ORC`, `PARQUET`, `AVRO`, `JSON`, and `TEXTFILE`. | | `write_compression` | None | The compression type for any storage format that allows compressions. | | `field_delimeter` | None | Specify the custom field delimiter to use when the format is set to `TEXTFIRE`. | | `table_properties` | N/A | The table properties to add to the table. This is only for Iceberg. | | `native_drop` | N/A | Relation drop operations will be performed with SQL, not direct Glue API calls. No S3 calls will be made to manage data in S3. Data in S3 will only be cleared up for Iceberg tables. See the [AWS docs](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-managing-tables.html) for more info. Iceberg DROP TABLE operations may timeout if they take longer than 60 seconds. | | `seed_by_insert` | False | Creates seeds using an SQL insert statement. Large seed files can't exceed the Athena 262144 bytes limit. | | `force_batch` | False | Run the table creation directly in batch insert mode. Useful when the standard table creation fails due to partition limitation. | | `unique_tmp_table_suffix` | False | Replace the "\_\_dbt\_tmp table" suffix with a unique UUID for incremental models using insert overwrite on Hive tables. | | `temp_schema` | None | Defines a schema to hold temporary create statements used in incremental model runs. Scheme will be created in the models target database if it does not exist. | | `lf_tags_config` | None | [AWS Lake Formation](#aws-lake-formation-integration) tags to associate with the table and columns. Existing tags will be removed.<br />\* `enabled` (`default=False`) whether LF tags management is enabled for a model<br />\* `tags` dictionary with tags and their values to assign for the model<br />\* `tags_columns` dictionary with a tag key, value and list of columns they must be assigned to | | `lf_inherited_tags` | None | List of the Lake Formation tag keys that are to be inherited from the database level and shouldn't be removed during the assignment of those defined in `ls_tags_config`. | | `lf_grants` | None | Lake Formation grants config for `data_cell` filters. | ###### Configuration examples ##### schema.yml models/schema.yml ```sql {{ config( materialized='incremental', incremental_strategy='append', on_schema_change='append_new_columns', table_type='iceberg', schema='test_schema', lf_tags_config={ 'enabled': true, 'tags': { 'tag1': 'value1', 'tag2': 'value2' }, 'tags_columns': { 'tag1': { 'value1': ['column1', 'column2'], 'value2': ['column3', 'column4'] } }, 'inherited_tags': ['tag1', 'tag2'] } ) }} ``` ##### dbt\_project.yml dbt\_project.yml ```yaml +lf_tags_config: enabled: true tags: tag1: value1 tag2: value2 tags_columns: tag1: value1: [ column1, column2 ] inherited_tags: [ tag1, tag2 ] ``` ##### Lake formation grants ```python lf_grants={ 'data_cell_filters': { 'enabled': True | False, 'filters': { 'filter_name': { 'row_filter': '<filter_condition>', 'principals': ['principal_arn1', 'principal_arn2'] } } } } ``` Consider these limitations and recommendations: * `lf_tags` and `lf_tags_columns` configs support only attaching lf tags to corresponding resources. * We recommend managing LF Tags permissions somewhere outside dbt. For example, [terraform](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_permissions) or [aws cdk](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lakeformation-readme.html). * `data_cell_filters` management can't be automated outside dbt because the filter can't be attached to the table, which doesn't exist. Once you `enable` this config, dbt will set all filters and their permissions during every dbt run. Such an approach keeps the actual state of row-level security configuration after every dbt run and applies changes if they occur: drop, create, and update filters and their permissions. * Any tags listed in `lf_inherited_tags` should be strictly inherited from the database level and never overridden at the table and column level. * Currently, `dbt-athena` does not differentiate between an inherited tag association and an override it made previously. * For example, If a `lf_tags_config` value overrides an inherited tag in one run, and that override is removed before a subsequent run, the prior override will linger and no longer be encoded anywhere (for example, Terraform where the inherited value is configured nor in the dbt project where the override previously existed but now is gone). ##### Table location The saved location of a table is determined in precedence by the following conditions: 1. If `external_location` is defined, that value is used. 2. If `s3_data_dir` is defined, the path is determined by that and `s3_data_naming`. 3. If `s3_data_dir` is not defined, data is stored under `{s3_staging_dir}/tables/`. The following options are available for `s3_data_naming`: * `unique`: `{s3_data_dir}/{uuid4()}/` * `table`: `{s3_data_dir}/{table}/` * `table_unique`: `{s3_data_dir}/{table}/{uuid4()}/` * `schema_table`: `{s3_data_dir}/{schema}/{table}/` * `schema_table_unique`: `{s3_data_dir}/{schema}/{table}/{uuid4()}/` To set the `s3_data_naming` globally in the target profile, overwrite the value in the table config, or set up the value for groups of the models in dbt\_project.yml. Note: If you're using a workgroup with a default output location configured, `s3_data_naming` ignores any configured buckets and uses the location configured in the workgroup. ##### Incremental models The following [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) strategies are supported: * `insert_overwrite` (default): The insert-overwrite strategy deletes the overlapping partitions from the destination table and then inserts the new records from the source. This strategy depends on the `partitioned_by` keyword! dbt will fall back to the `append` strategy if no partitions are defined. This strategy is only available for Hive tables. * `append`: Insert new records without updating, deleting or overwriting any existing data. There might be duplicate data (great for log or historical data). * `merge`: Conditionally updates, deletes, or inserts rows into an Iceberg table. Used in combination with `unique_key`. It is only available when using Iceberg. * `microbatch`: Process large time-series datasets in batches. Refer to [About microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md). Consider these limitations when using Iceberg models: * Incremental Iceberg models don't support the `insert_overwrite` strategy. They support `append`, `merge`, and `microbatch`. * Incremental Iceberg models sync all columns on schema change. You can't remove columns used for partitioning with an incremental refresh; you must fully refresh the model. ##### On schema change The `on_schema_change` option reflects changes of the schema in incremental models. The values you can set this to are: * `ignore` (default) * `fail` * `append_new_columns` * `sync_all_columns` To learn more, refer to [What if the columns of my incremental model change](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change). ##### Iceberg The adapter supports table materialization for Iceberg. For example: ```sql {{ config( materialized='table', table_type='iceberg', format='parquet', partitioned_by=['bucket(user_id, 5)'], table_properties={ 'optimize_rewrite_delete_file_threshold': '2' } ) }} select 'A' as user_id, 'pi' as name, 'active' as status, 17.89 as cost, 1 as quantity, 100000000 as quantity_big, current_date as my_date ``` ###### Iceberg bucketing Iceberg supports bucketing as hidden partitioning. Use the `partitioned_by` config with the `bucket` function to hash a column into a fixed number of buckets. The `bucketed_by` and `bucket_count` configs are ignored for Iceberg tables. The argument order depends on the engine that runs your model: | Model type | Engine | Syntax | Example | | ---------- | ---------- | ----------------------- | -------------------- | | SQL | Athena SQL | `bucket(column, count)` | `bucket(user_id, 5)` | | Python | Spark | `bucket(count, column)` | `bucket(5, user_id)` | Argument order differs by engine Athena SQL puts the column first. Spark puts the bucket count first. Mixing them up causes the model to fail or bucket the wrong column. ##### SQL model This example hashes `user_id` into 5 buckets: ```sql {{ config( materialized='table', table_type='iceberg', format='parquet', partitioned_by=['bucket(user_id, 5)'] ) }} select 'A' as user_id, 'pi' as name ``` You can combine bucketing with other partitions, for example `partitioned_by=['status', 'bucket(user_id, 5)']`. ##### Python model This example hashes `user_id` into 5 buckets: ```python def model(dbt, session): dbt.config( materialized='table', table_type='iceberg', format='parquet', partitioned_by=['bucket(5, user_id)'] ) return session.createDataFrame([("A", "pi")], ["user_id", "name"]) ``` You can combine bucketing with other partitions, for example `partitioned_by=['status', 'bucket(5, user_id)']`. ###### Iceberg catalogs In `dbt-athena` 1.11.1 and later, you can define Iceberg catalogs in `catalogs.yml` and select one with `catalog_name` on a model. The default catalog type for Athena is `glue`. Refer to [Using catalogs.yml](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) for the `catalogs.yml` format. Iceberg supports the `PARQUET`, `AVRO` and `ORC` table formats for data . The following are the supported strategies for using Iceberg incrementally: * `append`: New records are appended to the table (this can lead to duplicates). * `merge`: Perform an update and insert (and optional delete) where new and existing records are added. This is only available with Athena engine version 3. * `unique_key`(required): Columns that define a unique source and target table record. * `incremental_predicates` (optional): The SQL conditions that enable custom join clauses in the merge statement. This helps improve performance via predicate pushdown on target tables. * `delete_condition` (optional): SQL condition that identifies records that should be deleted. * `update_condition` (optional): SQL condition that identifies records that should be updated. * `insert_condition` (optional): SQL condition that identifies records that should be inserted. * `microbatch`: Process large time-series datasets in batches. Refer to [About microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md). `incremental_predicates`, `delete_condition`, `update_condition` and `insert_condition` can include any column of the incremental table (`src`) or the final table (`target`). Column names must be prefixed by either `src` or `target` to prevent a `Column is ambiguous` error. ##### delete\_condition ```sql {{ config( materialized='incremental', table_type='iceberg', incremental_strategy='merge', unique_key='user_id', incremental_predicates=["src.quantity > 1", "target.my_date >= now() - interval '4' year"], delete_condition="src.status != 'active' and target.my_date < now() - interval '2' year", format='parquet' ) }} select 'A' as user_id, 'pi' as name, 'active' as status, 17.89 as cost, 1 as quantity, 100000000 as quantity_big, current_date as my_date ``` ##### update\_condition ```sql {{ config( materialized='incremental', incremental_strategy='merge', unique_key=['id'], update_condition='target.id > 1', schema='sandbox' ) }} {% if is_incremental() %} select * from ( values (1, 'v1-updated') , (2, 'v2-updated') ) as t (id, value) {% else %} select * from ( values (-1, 'v-1') , (0, 'v0') , (1, 'v1') , (2, 'v2') ) as t (id, value) {% endif %} ``` ##### insert\_condition ```sql {{ config( materialized='incremental', incremental_strategy='merge', unique_key=['id'], insert_condition='target.status != 0', schema='sandbox' ) }} select * from ( values (1, 0) , (2, 1) ) as t (id, status) ``` ###### AWS S3 Tables In `dbt-athena` 1.11.1 and later, you can write Iceberg models to [Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html) by defining a catalog with `type: s3_tables` in `catalogs.yml` and setting `catalog_name` on the model. `table`, `incremental`, and `snapshot` materializations are supported. ###### Prerequisites * `dbt-athena` 1.11.1 or later, and dbt v1.12 or later with `use_catalogs_v2` enabled. * An [S3 table bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets.html). * [AWS Glue integration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html) enabled for that bucket, so it appears as `s3tablescatalog/YOUR_TABLE_BUCKET`. Without it, the catalog won't resolve. * A namespace in the table bucket that matches the schema dbt uses for the model. dbt doesn't create S3 Tables namespaces for you. * An IAM role with S3 Tables read and write access, plus Glue permissions to create and delete tables in that catalog. Refer to [Required IAM permissions](https://docs.aws.amazon.com/glue/latest/dg/s3tables-catalog-prerequisites.html#s3tables-required-iam-permissions) for details. ###### Configure the catalog 1. Enable the `use_catalogs_v2` flag in your `dbt_project.yml`: dbt\_project.yml ```yml flags: use_catalogs_v2: true ``` 2. Define the catalog in [`catalogs.yml`](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md). Set `catalog_database` to the catalog name Athena uses for your table bucket (`s3tablescatalog/YOUR_TABLE_BUCKET`). The model's schema maps to the S3 Tables namespace: catalogs.yml ```yml catalogs: - name: my_s3_tables type: s3_tables table_format: iceberg config: athena: catalog_database: s3tablescatalog/my-table-bucket ``` 3. Configure your model to use the catalog: ```sql {{ config( materialized='table', catalog_name='my_s3_tables' ) }} select 1 as id ``` ###### Considerations * Python models aren't supported for S3 Tables catalogs. * `external_location`, `s3_data_dir`, and `s3_data_naming` are ignored because S3 Tables manages the storage location. * Table replacement uses drop and recreate. S3 Tables doesn't support `ALTER TABLE RENAME`, so the Iceberg high-availability behavior described in [High availability (HA) table](#high-availability-ha-table) doesn't apply. ##### High availability (HA) table The current implementation of table materialization can lead to downtime, as the target table is dropped and re-created. For less destructive behavior, you can use the `ha` config on your `table` materialized models. It leverages the table versions feature of the glue catalog, which creates a temporary table and swaps the target table to the location of the temporary table. This materialization is only available for `table_type=hive` and requires using unique locations. For Iceberg, high availability is the default, except for [AWS S3 Tables](#aws-s3-tables) catalogs, which drop and recreate the table instead. By default, the materialization keeps the last 4 table versions, but you can change it by setting `versions_to_keep`. ```sql {{ config( materialized='table', ha=true, format='parquet', table_type='hive', partitioned_by=['status'], s3_data_naming='table_unique' ) }} select 'a' as user_id, 'pi' as user_name, 'active' as status union all select 'b' as user_id, 'sh' as user_name, 'disabled' as status ``` ##### HA known issues * There could be a little downtime when swapping from a table with partitions to a table without (and the other way around). If higher performance is needed, consider bucketing instead of partitions. * By default, Glue "duplicates" the versions internally, so the last two versions of a table point to the same location. * It's recommended to set `versions_to_keep` >= 4, as this will avoid having the older location removed. ##### Avoid deleting parquet files If a dbt model has the same name as an existing table in the AWS Glue catalog, the `dbt-athena` adapter deletes the files in that table’s S3 location before recreating the table using the SQL from the model. The adapter may also delete data if a model is configured to use the same S3 location as an existing table. In this case, it clears the folder before creating the new table to avoid conflicts during setup. When dropping a model, the `dbt-athena` adapter performs two cleanup steps for both Iceberg and Hive tables: * It deletes the table from the AWS Glue catalog using Glue APIs. * It removes the associated S3 data files using a delete operation. However, for Iceberg tables, using standard SQL like [`DROP TABLE`](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-drop-table.html) may not remove all related S3 objects. To ensure proper cleanup in a dbt workflow, the adapter includes a workaround that explicitly deletes these S3 objects. Alternatively, users can enable [`native_drop`](https://docs.getdbt.com/reference/resource-configs/athena-configs.md#table-configuration) to let Iceberg handle the cleanup natively. ##### Update glue data catalog You can persist your column and model level descriptions to the Glue Data Catalog as [glue table properties](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html#table-properties) and [column parameters](https://docs.aws.amazon.com/glue/latest/webapi/API_Column.html). To enable this, set the configuration to `true` as shown in the following example. By default, documentation persistence is disabled, but it can be enabled for specific resources or groups of resources as needed. For example: ```yaml models: - name: test_deduplicate description: another value config: persist_docs: relation: true columns: true meta: test: value columns: - name: id config: meta: # changed to config in v1.10 and backported to 1.9 primary_key: true ``` Refer to [persist\_docs](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) for more details. #### Snapshots The adapter supports snapshot materialization. It supports both the timestamp and check strategies. To create a snapshot, create a snapshot file in the `snapshots` directory. You'll need to create this directory if it doesn't already exist. ##### Timestamp strategy Refer to [Timestamp strategy](https://docs.getdbt.com/docs/build/snapshots.md#timestamp-strategy-recommended) for details on how to use it. ##### Check strategy Refer to [Check strategy](https://docs.getdbt.com/docs/build/snapshots.md#check-strategy) for details on how to use it. ##### Hard deletes The materialization also supports invalidating hard deletes. For usage details, refer to [Hard deletes](https://docs.getdbt.com/docs/build/snapshots.md#hard-deletes-opt-in). ##### Snapshots known issues * Tables, schemas, and database names should only be lowercase. * To avoid potential conflicts, make sure [`dbt-athena-adapter`](https://github.com/Tomme/dbt-athena) is not installed in the target environment. * Snapshot does not support dropping columns from the source table. If you drop a column, make sure to drop the column from the snapshot as well. Another workaround is to NULL the column in the snapshot definition to preserve the history. #### AWS Lake Formation integration The following describes how the adapter implements the AWS Lake Formation tag management: * [Enable](#table-configuration) LF tags management with the `lf_tags_config` parameter. By default, it's disabled. * Once enabled, LF tags are updated on every dbt run. * First, all lf-tags for columns are removed to avoid inheritance issues. * Then, all redundant lf-tags are removed from tables and actual tags from table configs are applied. * Finally, lf-tags for columns are applied. It's important to understand the following points: * dbt doesn't manage `lf-tags` for databases * dbt doesn't manage Lake Formation permissions That's why it's important to take care of this yourself or use an automation tool such as terraform and AWS CDK. For more details, refer to: * [terraform aws\_lakeformation\_permissions](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_permissions) * [terraform aws\_lakeformation\_resource\_lf\_tags](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_resource_lf_tags) #### Python models The adapter supports Python models using [`spark`](https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html). ##### Prerequisites * A Spark-enabled workgroup created in Athena. * Spark execution role granted access to Athena, Glue and S3. * The Spark workgroup is added to the `~/.dbt/profiles.yml` file and the profile to be used is referenced in `dbt_project.yml`. ##### Spark-specific table configuration | Configuration | Default | Description | | ----------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `timeout` | 43200 | Time out in seconds for each Python model execution. Defaults to 12 hours/43200 seconds. | | `spark_encryption` | False | When set to `true,` it encrypts data stored locally by Spark and in transit between Spark nodes. | | `spark_cross_account_catalog` | False | When using the Spark Athena workgroup, queries can only be made against catalogs on the same AWS account by default. Setting this parameter to true will enable querying external catalogs if you want to query another catalog on an external AWS account.<br />Use the syntax `external_catalog_id/database.table` to access the external table on the external catalog (For example, `999999999999/mydatabase.cloudfront_logs` where 999999999999 is the external catalog ID). | | `spark_requester_pays` | False | When set to true, if an Amazon S3 bucket is configured as `requester pays`, the user account running the query is charged for data access and data transfer fees associated with the query. | ##### Spark notes * A session is created for each unique engine configuration defined in the models that are part of the invocation. A session's idle timeout is set to 10 minutes. Within the timeout period, if a new calculation (Spark Python model) is ready for execution and the engine configuration matches, the process will reuse the same session. * The number of Python models running simultaneously depends on the `threads`. The number of sessions created for the entire run depends on the number of unique engine configurations and the availability of sessions to maintain thread concurrency. * For Iceberg tables, it's recommended to use the `table_properties` configuration to set the `format_version` to `2`. This helps maintain compatibility between the Iceberg tables Trino created and those Spark created. ##### Example models ##### Simple pandas ```python import pandas as pd def model(dbt, session): dbt.config(materialized="table") model_df = pd.DataFrame({"A": [1, 2, 3, 4]}) return model_df ``` ##### Simple Spark ```python def model(dbt, spark_session): dbt.config(materialized="table") data = [(1,), (2,), (3,), (4,)] df = spark_session.createDataFrame(data, ["A"]) return df ``` ##### Spark incremental ```python def model(dbt, spark_session): dbt.config(materialized="incremental") df = dbt.ref("model") if dbt.is_incremental: max_from_this = ( f"select max(run_date) from {dbt.this.schema}.{dbt.this.identifier}" ) df = df.filter(df.run_date >= spark_session.sql(max_from_this).collect()[0][0]) return df ``` ##### Config Spark model ```python def model(dbt, spark_session): dbt.config( materialized="table", engine_config={ "CoordinatorDpuSize": 1, "MaxConcurrentDpus": 3, "DefaultExecutorDpuSize": 1 }, spark_encryption=True, spark_cross_account_catalog=True, spark_requester_pays=True polling_interval=15, timeout=120, ) data = [(1,), (2,), (3,), (4,)] df = spark_session.createDataFrame(data, ["A"]) return df ``` ##### PySpark UDF Using imported external python files: ```python def model(dbt, spark_session): dbt.config( materialized="incremental", incremental_strategy="merge", unique_key="num", ) sc = spark_session.sparkContext sc.addPyFile("s3://athena-dbt/test/file1.py") sc.addPyFile("s3://athena-dbt/test/file2.py") def func(iterator): from file2 import transform return [transform(i) for i in iterator] from pyspark.sql.functions import udf from pyspark.sql.functions import col udf_with_import = udf(func) data = [(1, "a"), (2, "b"), (3, "c")] cols = ["num", "alpha"] df = spark_session.createDataFrame(data, cols) return df.withColumn("udf_test_col", udf_with_import(col("alpha"))) ``` ##### Known issues in Python models * Python models can't [reference Athena SQL views](https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html). * You can use third-party Python libraries; however, they must be [included in the pre-installed list](https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-preinstalled-python-libraries.html) or [imported manually](https://docs.aws.amazon.com/athena/latest/ug/notebooks-import-files-libraries.html). * Python models can only reference or write to tables with names matching the regular expression: `^[0-9a-zA-Z_]+$`. Spark doesn't support dashes or special characters, even though Athena supports them. * Incremental models don't fully utilize Spark capabilities. They depend partially on existing SQL-based logic that runs on Trino. * Snapshot materializations are not supported. * Spark can only reference tables within the same catalog. * For tables created outside of the dbt tool, be sure to populate the location field, or dbt will throw an error when creating the table. #### Contracts The adapter partly supports contract definitions: * `data_type` is supported but needs to be adjusted for complex types. Types must be specified entirely (for example, `array<int>`) even though they won't be checked. Indeed, as dbt recommends, we only compare the broader type (array, map, int, varchar). The complete definition is used to check that the data types defined in Athena are ok (pre-flight check). * The adapter does not support the constraints since Athena has no constraint concept. --- ### Amazon Redshift adapter behavior changes The following are the current [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) that are specific to `dbt-redshift`: | Flag | `dbt-redshift`: Intro | `dbt-redshift`: Maturity | Status | | ---------------------------------------------------------------------------------------------------------- | --------------------- | ------------------------ | ------ | | [`redshift_skip_autocommit_transaction_statements`](#redshift_skip_autocommit_transaction_statements-flag) | 1.12.0 | TBD | Active | #### The `redshift_skip_autocommit_transaction_statements` flag Available starting `dbt-redshift` 1.12.0. The `redshift_skip_autocommit_transaction_statements` flag controls whether dbt sends explicit [`BEGIN`](https://docs.aws.amazon.com/redshift/latest/dg/r_BEGIN.html), [`COMMIT`](https://docs.aws.amazon.com/redshift/latest/dg/r_COMMIT.html), and [`ROLLBACK`](https://docs.aws.amazon.com/redshift/latest/dg/r_ROLLBACK.html) statements to Amazon Redshift when autocommit is enabled. Since `dbt-redshift` 1.5, the default Redshift connection uses `autocommit=True`, so the driver already commits each statement. In that setup, the statements are redundant when autocommit is enabled and add extra round trips to Redshift. * When set to `false` (default), dbt sends `BEGIN`, `COMMIT`, and `ROLLBACK` statements (legacy behavior). * When set to `true`, dbt does not send `BEGIN`, `COMMIT`, or `ROLLBACK` statements to Redshift, which can reduce round trips and improve performance. dbt still calls `begin()`, `commit()`, and `rollback_if_open()` on the connection, but no longer issues the matching SQL: * `begin()` is invoked, but dbt does not execute `BEGIN` on Redshift. * `commit()` is invoked, but dbt does not execute `COMMIT` on Redshift. * `rollback_if_open()` is invoked, but dbt does not execute `ROLLBACK` on Redshift. dbt still maintains its internal `transaction_open` state so transaction tracking remains consistent even when those SQL statements are skipped. To skip unnecessary transaction statements when `autocommit` is enabled, set the flag to `true` in your `dbt_project.yml`: dbt\_project.yml ```yaml flags: redshift_skip_autocommit_transaction_statements: true ``` ##### How this flag interacts with `autocommit` * If the connection uses `autocommit=False`, dbt’s explicit transaction behavior is unchanged. * If the connection uses `autocommit=True` (default) and the flag is `false` (default), dbt still sends `BEGIN`, `COMMIT`, and `ROLLBACK`. * If the connection uses `autocommit=True` and the flag is `true`, dbt skips those statements. --- ### Analysis properties (Applies to dbt v1.12 and later) We recommend you define analysis properties in your `analyses/` directory, which is illustrated in the [`analysis-paths`](https://docs.getdbt.com/reference/project-configs/analysis-paths.md) configuration. You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `analyses/` or `models/` directory. analyses/\<filename>.yml ```yml analyses: - name: <analysis_name> # required description: <markdown_string> config: enabled: true | false docs: # changed to config in v1.10 show: true | false node_color: <color_id> # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") tags: <string> | [<string>] columns: - name: <column_name> description: <markdown_string> - name: ... # declare properties of additional columns - name: ... # declare properties of additional analyses ``` (Applies to dbt v1.12 and later) To configure analyses at the project level, set the [`require_corrected_analysis_fqns`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md) flag to `true` and add an `analyses` block in your `dbt_project.yml`. The project-level configuration applies to existing analyses in the `analyses/` folder — for example, setting `+enabled: false` disables them all. dbt\_project.yml ```yaml flags: require_corrected_analysis_fqns: true analyses: +enabled: true | false ``` --- ### analysis-paths dbt\_project.yml ```yml analysis-paths: [directorypath] ``` #### Definition Specify a custom list of directories where [analyses](https://docs.getdbt.com/docs/build/analyses.md) are located. #### Default Without specifying this config, dbt will not compile any `.sql` files as analyses. However, the [`dbt init` command](https://docs.getdbt.com/reference/commands/init.md) populates this value as `analyses` ([source](https://github.com/dbt-labs/dbt-starter-project/blob/HEAD/dbt_project.yml#L15)). Paths specified in `analysis-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/analyses`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml analysis-paths: ["analyses"] ``` * ❌ **Don't** * Avoid absolute paths: ```yml analysis-paths: ["/Users/username/project/analyses"] ``` #### Examples ##### Use a subdirectory named `analyses` This is the value populated by the [`dbt init` command](https://docs.getdbt.com/reference/commands/init.md). dbt\_project.yml ```yml analysis-paths: ["analyses"] ``` ##### Use a subdirectory named `custom_analyses` dbt\_project.yml ```yml analysis-paths: ["custom_analyses"] ``` --- ### anchors #### Definition Anchors are a [YAML feature](https://yaml.org/spec/1.2.2/#692-node-anchors) that let you reuse configuration blocks inside a single YAML file. In dbt v1.10, the `anchors:` key was introduced to enclose configuration fragments that aren't valid on their own or that only exist as template data. Using the `anchors:` key ensures these fragments won't be rejected during file validation. In dbt v1.10 and higher, invalid anchors trigger a warning. In dbt v2, these invalid anchors result in errors. note You can define anchors in dbt v1.9 and earlier, but there is no dedicated location for anchors in these versions. If you need to define a standalone anchor, you can put it at the top level of your YAML file. #### YAML anchor syntax ##### Anchors and aliases To define a YAML anchor, add an `anchors:` block in your YAML file and use the `&` symbol in front of the anchor's name (for example, `&id_column_alias`). This creates an alias which you can reference elsewhere by prefixing the alias with a `*` character. The following example creates an anchor whose alias is `*id_column_alias`. The `id` column, its description, data type, and data tests are all applied to `my_first_model`, `my_second_model`, and `my_third_model`. models/\_models.yml ```yml anchors: - &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models. - name: unrelated_column_b - name: my_second_model columns: - *id_column_alias - name: unrelated_column_c - name: my_third_model columns: - *id_column_alias - name: unrelated_column_d ``` [![Behind the scenes, the alias is replaced with the object defined by the anchor.](/img/reference/resource-properties/anchor_example_expansion.png?v=2 "Behind the scenes, the alias is replaced with the object defined by the anchor.")](#)Behind the scenes, the alias is replaced with the object defined by the anchor. ##### Merge syntax Sometimes, an anchor is mostly the same but one part needs to be overridden. When the anchor refers to a dictionary/mapping (not a list or a scalar value), you can use the `<<:` merge syntax to override an already-defined key, or add extra keys to the dictionary. For example: models/\_models.yml ```yml anchors: - &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique - &source_template_alias database: RAW loader: fivetran config: freshness: warn_after: {count: 1, period: day} models: - name: my_first_model columns: - *id_column_alias # brings in the full anchor defined above - name: unrelated_column_a description: This column is not repeated in other models. - name: unrelated_column_b - name: my_second_model columns: - <<: *id_column_alias data_type: bigint # overrides the data_type from int to bigint, while inheriting the name, description, and data tests - name: unrelated_column_c - name: my_third_model columns: - <<: *id_column_alias config: meta: extra_key: extra_value # adds config.meta.extra_key to just this version of the id column, in addition to the name, description, data type, and data tests - name: unrelated_column_d sources: # both sources start with their database, loader, and freshness expectations set from the anchor, and merge in additional keys - <<: *source_template_alias name: salesforce schema: etl_salesforce_schema tables: - name: opportunities - name: users - <<: *source_template_alias name: hubspot schema: etl_hubspot_schema tables: - name: contacts ``` #### Usage notes * Old versions of dbt (v1.9 and earlier) do not have a dedicated `anchors:` key. If you need to define a standalone anchor, you can leave it at the top level of your file. * You can't merge additional elements into a list which was defined as an anchor. For example, if you define an anchor containing multiple columns, you can't attach extra columns to the end of the list. Instead, define each column as an individual anchor and add each one to the relevant tables. * You do not need to move existing anchors to the `anchors:` key if they are already defined in a larger valid YAML object. For example, the following `&customer_id_tests` anchor does not need to be moved because it is a valid part of the existing `columns` block. ```yml models: - name: my_first_model columns: - name: customer_id data_tests: &customer_id_tests - not_null - unique - name: order_id data_tests: *customer_id_tests ``` --- ### Anonymous usage stats dbt Labs is on a mission to build the best version of dbt possible, and a crucial part of that is understanding how users work with dbt. To this end, we've added some simple event tracking (or telemetry) to dbt using Snowplow. Importantly, we do not track credentials, raw model contents, or model names: we consider these private, and frankly none of our business. The data we collect is used for use cases such as industry identification, use-case research, improvements of sales, marketing, product features, and services. Telemetry allows users to seamlessly contribute to the continuous improvement of dbt, enabling us to better serve the data community. Usage statistics are fired when dbt is invoked and when models are run. These events contain basic platform information (OS + Python version) and metadata such as: * Whether the invocation succeeded. * How long it took. * An anonymized hash key representing the raw model content. * Number of nodes that were run. (Applies to dbt v2.0 and later) #### dbt v2 telemetry dbt v2 has telemetry enabled by default. For full transparency, you can see the event definitions in [`event_functions.rs`](https://github.com/dbt-labs/dbt/blob/main/crates/vortex-events/src/event_functions.rs). Telemetry requires outbound HTTPS access to `https://p.vx.dbt.com`. If the endpoint is unreachable, dbt v2 logs errors on each invocation. For the complete list of outbound endpoints, refer to [dbt v2 networking requirements](https://docs.getdbt.com/docs/local/dbt-networking-requirements.md). To disable anonymous usage statistics, set the following environment variable: ```shell export DBT_ENGINE_SEND_ANONYMOUS_USAGE_STATS=false ``` You can also set `DO_NOT_TRACK=1`. --- ### Apache Spark configurations If you're using Databricks, use `dbt-databricks` If you're using Databricks, the `dbt-databricks` adapter is recommended over `dbt-spark`. If you're still using dbt-spark with Databricks consider [migrating from the dbt-spark adapter to the dbt-databricks adapter](https://docs.getdbt.com/guides/migrate-from-spark-to-databricks.md). For the Databricks version of this page, refer to [Databricks setup](#databricks-setup). note See [Databricks configuration](#databricks-configs) for the Databricks version of this page. #### Configuring tables When materializing a model as `table`, you may include several optional configs that are specific to the dbt-spark plugin, in addition to the standard [model configs](https://docs.getdbt.com/reference/model-configs.md). | Option | Description | Required? | Example | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- | -------------------------------------------------------------------------------------------------- | | file\_format | The file format to use when creating tables (`parquet`, `delta`, `iceberg`, `hudi`, `csv`, `json`, `text`, `jdbc`, `orc`, `hive` or `libsvm`). | Optional | `parquet` | | location\_root [1](#user-content-fn-1) | The created table uses the specified directory to store its data. The table alias is appended to it. | Optional | `/mnt/root` | | partition\_by | Partition the created table by the specified columns. A directory is created for each partition. | Optional | `date_day` | | clustered\_by | Each partition in the created table will be split into a fixed number of buckets by the specified columns. | Optional | `country_code` | | buckets | The number of buckets to create while clustering | Required if `clustered_by` is specified | `8` | | tblproperties | The table properties configure table behavior. Properties differ depending on the file format, see reference docs ([Iceberg](https://iceberg.apache.org/docs/latest/configuration/#table-properties), [Parquet](https://spark.apache.org/docs/3.5.4/sql-data-sources-parquet.html#data-source-option), [Delta](https://docs.databricks.com/aws/en/delta/table-properties#delta-table-properties), [Hudi](https://hudi.apache.org/docs/sql_ddl/#table-properties)). | Optional | `# Iceberg Example tblproperties:   read.split.target-size: 268435456   commit.retry.num-retries: 10` | #### Incremental models dbt seeks to offer useful, intuitive modeling abstractions by means of its built-in configurations and materializations. Because there is so much variance between Apache Spark clusters out in the world—not to mention the powerful features offered to Databricks users by the Delta file format and custom runtime—making sense of all the available options is an undertaking in its own right. Alternatively, you can use Apache Iceberg or Apache Hudi file format with Apache Spark runtime for building incremental models. For that reason, the dbt-spark plugin leans heavily on the [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-strategy.md). This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of three values: * **`append`** (default): Insert new records without updating or overwriting any existing data. * **`insert_overwrite`**: If `partition_by` is specified, overwrite partitions in the table with new data. If no `partition_by` is specified, overwrite the entire table with new data. * **`merge`** (Delta, Iceberg and Hudi file format only): Match records based on a `unique_key`; update old records, insert new ones. (If no `unique_key` is specified, all new data is inserted, similar to `append`.) * `microbatch` Implements the [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) using `event_time` to define time-based ranges for filtering data. Each of these strategies has its pros and cons, which we'll discuss below. As with any model config, `incremental_strategy` may be specified in `dbt_project.yml` or within a model file's `config()` block. ##### The `append` strategy Following the `append` strategy, dbt will perform an `insert into` statement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Apache Spark versions. However, this strategy *cannot* update, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources. Specifying `append` as the incremental strategy is optional, since it's the default strategy used when none is specified. ##### Source code spark\_incremental.sql ```sql {{ config( materialized='incremental', incremental_strategy='append', ) }} -- All rows returned by this query will be appended to the existing table select * from {{ ref('events') }} {% if is_incremental() %} where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ``` ##### Run code spark\_incremental.sql ```sql create temporary view spark_incremental__dbt_tmp as select * from analytics.events where event_ts >= (select max(event_ts) from {{ this }}) ; insert into table analytics.spark_incremental select `date_day`, `users` from spark_incremental__dbt_tmp ``` ##### The `insert_overwrite` strategy This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://downloads.apache.org/spark/docs/3.0.0/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select *all* of the relevant data for a partition when using this incremental strategy. If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead. **Usage notes:** * This strategy is not supported for tables with `file_format: delta`. * This strategy is not available when connecting via Databricks SQL endpoints (`method: odbc` + `endpoint`). * If connecting via a Databricks cluster + ODBC driver (`method: odbc` + `cluster`), you **must** include `set spark.sql.sources.partitionOverwriteMode DYNAMIC` in the [cluster Spark Config](https://docs.databricks.com/clusters/configure.html#spark-config) in order for dynamic partition replacement to work (`incremental_strategy: insert_overwrite` + `partition_by`). [![Databricks cluster: Spark Config](/img/reference/databricks-cluster-sparkconfig-partition-overwrite.png?v=2 "Databricks cluster: Spark Config")](#)Databricks cluster: Spark Config ##### Source code spark\_incremental.sql ```sql {{ config( materialized='incremental', partition_by=['date_day'], file_format='parquet', incremental_strategy='insert_overwrite' ) }} /* Every partition returned by this query will be overwritten when this model runs */ with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select date_day, count(*) as users from events group by 1 ``` ##### Run code spark\_incremental.sql ```sql create temporary view spark_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select date_day, count(*) as users from events group by 1 ; insert overwrite table analytics.spark_incremental partition (date_day) select `date_day`, `users` from spark_incremental__dbt_tmp ``` ##### The `merge` strategy **Usage notes:** The `merge` incremental strategy requires: * `file_format: delta, iceberg or hudi` * Databricks Runtime 5.1 and above for delta file format * Apache Spark for Iceberg or Hudi file format dbt will run an [atomic `merge` statement](https://docs.databricks.com/spark/latest/spark-sql/language-manual/merge-into.html) which looks nearly identical to the default merge behavior on Snowflake and BigQuery. If a `unique_key` is specified (recommended), dbt will update old records with values from new records that match on the key column. If a `unique_key` is not specified, dbt will forgo match criteria and simply insert all new records (similar to `append` strategy). ##### Source code merge\_incremental.sql ```sql {{ config( materialized='incremental', file_format='delta', # or 'iceberg' or 'hudi' unique_key='user_id', incremental_strategy='merge' ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from events group by 1 ``` ##### Run code target/run/merge\_incremental.sql ```sql create temporary view merge_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select user_id, max(date_day) as last_seen from events group by 1 ; merge into analytics.merge_incremental as DBT_INTERNAL_DEST using merge_incremental__dbt_tmp as DBT_INTERNAL_SOURCE on DBT_INTERNAL_SOURCE.user_id = DBT_INTERNAL_DEST.user_id when matched then update set * when not matched then insert * ``` #### Persisting model descriptions Relation-level docs persistence is supported in dbt. For more information on configuring docs persistence, see [the docs](https://docs.getdbt.com/reference/resource-configs/persist_docs.md). When the `persist_docs` option is configured appropriately, you'll be able to see model descriptions in the `Comment` field of `describe [table] extended` or `show table extended in [database] like '*'`. #### Always `schema`, never `database` Apache Spark uses the terms "schema" and "database" interchangeably. dbt understands `database` to exist at a higher level than `schema`. As such, you should *never* use or set `database` as a node config or in the target profile when running dbt-spark. If you want to control the schema/database in which dbt will materialize models, use the `schema` config and `generate_schema_name` macro *only*. #### Default file format configurations To access advanced incremental strategies features, such as [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) and the `merge` incremental strategy, you will want to use the Delta, Iceberg or Hudi file format as the default file format when materializing models as tables. It's quite convenient to do this by setting a top-level configuration in your project file: dbt\_project.yml ```yml models: +file_format: delta # or iceberg or hudi seeds: +file_format: delta # or iceberg or hudi snapshots: +file_format: delta # or iceberg or hudi ``` (Applies to dbt v1.11 and later) #### Retry handling for PyHive connections When using HTTP or Thrift connection methods, you can configure how dbt handles polling, timeouts, and connection retries for long-running queries. These settings help prevent queries from hanging indefinitely and automatically recover from connection interruptions during query execution. There are three profile configurations available: | Parameter | Type | Default | Description | | --------------- | ------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `poll_interval` | Integer | 5 | How frequently (in seconds) the adapter polls the Thrift server to check if an async query has completed. | | `query_timeout` | Integer | None | Maximum duration (in seconds) for query execution. If a query exceeds this duration during polling, dbt raises a `DbtRuntimeError`. No timeout by default. | | `query_retries` | Integer | 1 | How many times the adapter retries when connection loss occurs during query execution. | The adapter catches specific connection exceptions (such as `ConnectionResetError`, `BrokenPipeError`, and `TTransportException`) and retries with a fresh cursor when connection loss occurs. After exhausting all retries, dbt raises a `DbtRuntimeError` and suggests increasing `query_retries` in your profile. #### Footnotes 1. If you configure `location_root`, dbt specifies a location path in the `create table` statement. This changes the table from "managed" to "external" in Spark/Databricks. [↩](#user-content-fnref-1) --- ### arguments (for functions) Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> arguments: - name: <arg name> data_type: <string> # warehouse-specific description: <markdown_string> default_value: <string | boolean | integer> # optional, available in Snowflake and Postgres ``` #### Definition The `arguments` property is used to define the parameters that a resource can accept. Each argument can have a `name`, a `data_type` field, and optional properties such as `description` and `default_value`. For **functions**, you can add `arguments` to a [function property](https://docs.getdbt.com/reference/function-properties.md), which defines the parameters for user-defined functions (UDFs) in your warehouse. The `data_type` for function arguments is warehouse-specific (for example, `STRING`, `VARCHAR`, `INTEGER`) and should match the data types supported by your data platform. #### Properties ##### name The name of the argument. This is a required field if `arguments` is specified. ##### data\_type The data type that the warehouse expects for this parameter. This is a required field if `arguments` is specified and must match the data types supported by your specific data platform. Warehouse-specific data types The `data_type` values are warehouse-specific. Use the data type syntax that your warehouse requires: * **Snowflake**: `STRING`, `NUMBER`, `BOOLEAN`, `TIMESTAMP_NTZ`, etc. * **BigQuery**: `STRING`, `INT64`, `BOOL`, `TIMESTAMP`, `ARRAY<STRING>`, etc. * **Redshift**: `VARCHAR`, `INTEGER`, `BOOLEAN`, `TIMESTAMP`, etc. * **Postgres**: `TEXT`, `INTEGER`, `BOOLEAN`, `TIMESTAMP`, etc. Refer to your warehouse documentation for the complete list of supported data types. ##### description An optional markdown string describing the argument. This is helpful for documentation purposes. ##### default\_value Use the `default_value` property to make a function argument optional. * When an argument isn't defined with a `default_value`, it becomes a required argument, and you must pass a value for them when you use the function. If a required argument isn’t passed, the function call fails. * Arguments with a `default_value` are optional — if you don't pass a value for the argument, the warehouse uses the value you set in `default_value`. This property is supported in [Snowflake](https://docs.snowflake.com/en/developer-guide/udf-stored-procedure-arguments#designating-an-argument-as-optional) and [Postgres](https://www.postgresql.org/docs/current/sql-createfunction.html). When you use `default_value`, the order of your arguments matter. Any required arguments (those without default values) have to come before optional ones. Here's an example with the correct order: functions/schema.yml ```yml functions: - name: sum_2_values description: Add two values together arguments: - name: val1 # this argument comes first because it has no default value data_type: integer description: The first value - name: val2 data_type: integer description: The second value default_value: 0 returns: data_type: integer ``` In this example: * `val1` has no `default_value`, so it’s required. * `val2` has a `default_value` of `0`, so it’s optional. If you don’t provide a value for `val2`, the function uses `0` instead. See the following examples of calling the `sum_2_values` function: ```text sum_2_values(5) # val1 = 5, val2 = 0 (default value used since user did not specify val2) sum_2_values(5, 10) # val1 = 5, val2 = 10 sum_2_values() # ❌ error: val1 is required and must be passed ``` #### Examples ##### Simple function arguments functions/schema.yml ```yml functions: - name: is_positive_int arguments: - name: a_string data_type: string description: "The string that I want to check if it's representing a positive integer (like '10')" returns: data_type: boolean ``` ##### Complex data types functions/schema.yml ```yml functions: - name: calculate_discount arguments: - name: original_price data_type: DECIMAL(10,2) description: "The original price before discount" - name: discount_percent data_type: INTEGER description: "The discount percentage to apply" returns: data_type: DECIMAL(10,2) description: "The discounted price" ``` ##### Array data types (BigQuery example) functions/schema.yml ```yml functions: - name: get_tags arguments: - name: tag_string data_type: STRING description: "Comma-separated string of tags" returns: data_type: ARRAY<STRING> description: "An array of individual tag strings" ``` #### Related documentation * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [Arguments (for macros)](https://docs.getdbt.com/reference/resource-properties/arguments.md) * [Returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### arguments (for macros) macros/\<filename>.yml ```yml macros: - name: <macro name> arguments: - name: <arg name> type: <string> description: <markdown_string> ``` #### Definition The `arguments` property is used to define the parameters that a resource can accept. Each argument can have a `name`, a `type` field, and an optional `description`. For **macros**, you can add `arguments` to a [macro property](https://docs.getdbt.com/reference/macro-properties.md), which helps in documenting the macro and understanding what inputs it requires. #### type (Applies to dbt v1.10 and later) The data type of your argument. Setting [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md) to `true` ensures that documented macro argument names match those in the macro definition and validates their types against the [supported types](#supported-types). When set to `false`, `type` is only used for documentation purposes and there are no restrictions on the values you can specify. tip From dbt v1.10, you can opt into validating the arguments you define in macro documentation using the `validate_macro_args` behavior change flag. When enabled, dbt will: * Infer arguments from the macro and includes them in the [manifest.json](https://docs.getdbt.com/reference/artifacts/manifest-json.md) file if no arguments are documented. * Raise a warning if documented argument names don't match the macro definition. * Raise a warning if `type` fields don't follow [supported formats](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). Learn more about [macro argument validation](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). macros/\<filename>.yml ```yml macros: - name: <macro name> arguments: - name: <arg name> type: <string> ``` ##### Supported types (Applies to dbt v2.0 and later) When you use the [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md) flag, dbt supports the following types for macro arguments: * `string` or `str` * `boolean` or `bool` * `integer` or `int` * `float` * `any` * `list[<Type>]`, for example, `list[string]` * `dict[<Type>, <Type>]`, for example, `dict[str, list[int]]` * `optional[<Type>]`, for example, `optional[integer]` * [`relation`](https://docs.getdbt.com/reference/dbt-classes.md#relation) * [`column`](https://docs.getdbt.com/reference/dbt-classes.md#column) Note that the types follow a Python-like style but are used for documentation and validation only. They are not Python types. #### Examples macros/cents\_to\_dollars.sql ```sql {% macro cents_to_dollars(column_name, scale=2) %} ({{ column_name }} / 100)::numeric(16, {{ scale }}) {% endmacro %} ``` macros/cents\_to\_dollars.yml ```yml macros: - name: cents_to_dollars arguments: - name: column_name type: column description: "The name of a column" - name: scale type: integer description: "The number of decimal places to round to. Default is 2." ``` #### Related documentation * [Macro properties](https://docs.getdbt.com/reference/macro-properties.md) * [Arguments (for functions)](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) --- ### asset-paths dbt\_project.yml ```yml asset-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories to copy to the `target` directory as part of the `docs generate` command. This is useful for rendering images in your repository in your project documentation. #### Default By default, dbt will not copy any additional files as part of docs generate. For example, `asset-paths: []`. Paths specified in `asset-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/assets`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml asset-paths: ["assets"] ``` * ❌ **Don't** * Avoid absolute paths: ```yml asset-paths: ["/Users/username/project/assets"] ``` #### Examples ##### Compile files in the `assets` subdirectory as part of `docs generate` dbt\_project.yml ```yml asset-paths: ["assets"] ``` Any files included in this directory will be copied to the `target/` directory as part of `dbt docs generate`, making them accessible as images in your project documentation. Check out the full writeup on including images in your descriptions [here](https://docs.getdbt.com/reference/resource-properties/description.md#include-an-image-from-your-repo-in-your-descriptions). --- ### AWS Glue configurations #### Configuring tables When materializing a model as `table`, you may include several optional configs that are specific to the dbt-glue plugin, in addition to the [Apache Spark model configuration](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#configuring-tables). | Option | Description | Required? | Example | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ---------------------------------- | | custom\_location | By default, the adapter will store your data in the following path: `location path`/`database`/`table`. If you don't want to follow that default behaviour, you can use this parameter to set your own custom location on S3 | No | `s3://mycustombucket/mycustompath` | #### Incremental models dbt seeks to offer useful, intuitive modeling abstractions by means of its built-in configurations and materializations. For that reason, the dbt-glue plugin leans heavily on the [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-strategy.md). This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of three values: * **`append`** (default): Insert new records without updating or overwriting any existing data. * **`insert_overwrite`**: If `partition_by` is specified, overwrite partitions in the table with new data. If no `partition_by` is specified, overwrite the entire table with new data. * **`merge`** (Apache Hudi only): Match records based on a `unique_key`; update old records, insert new ones. (If no `unique_key` is specified, all new data is inserted, similar to `append`.) Each of these strategies has its pros and cons, which we'll discuss below. As with any model config, `incremental_strategy` may be specified in `dbt_project.yml` or within a model file's `config()` block. **Notes:** The default strategy is **`insert_overwrite`** ##### The `append` strategy Following the `append` strategy, dbt will perform an `insert into` statement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Apache Spark versions. However, this strategy *cannot* update, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources. ##### Source code glue\_incremental.sql ```sql {{ config( materialized='incremental', incremental_strategy='append', ) }} -- All rows returned by this query will be appended to the existing table select * from {{ ref('events') }} {% if is_incremental() %} where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ``` ##### Run code glue\_incremental.sql ```sql create view spark_incremental__dbt_tmp as select * from analytics.events where event_ts >= (select max(event_ts) from {{ this }}) ; insert into table analytics.spark_incremental select `date_day`, `users` from spark_incremental__dbt_tmp ``` ; drop view spark\_incremental\_\_dbt\_tmp ##### The `insert_overwrite` strategy This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://spark.apache.org/docs/3.1.2/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select *all* of the relevant data for a partition when using this incremental strategy. If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead. ##### Source code spark\_incremental.sql ```sql {{ config( materialized='incremental', partition_by=['date_day'], file_format='parquet' ) }} /* Every partition returned by this query will be overwritten when this model runs */ with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select date_day, count(*) as users from events group by 1 ``` ##### Run code spark\_incremental.sql ```sql create view spark_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select date_day, count(*) as users from events group by 1 ; insert overwrite table analytics.spark_incremental partition (date_day) select `date_day`, `users` from spark_incremental__dbt_tmp ; drop view spark_incremental__dbt_tmp ``` Specifying `insert_overwrite` as the incremental strategy is optional, since it's the default strategy used when none is specified. ##### The `merge` strategy **Usage notes:** The `merge` incremental strategy requires: * `file_format: hudi` * AWS Glue runtime 2 with hudi libraries as extra jars You can add hudi libraries as extra jars in the classpath using extra\_jars options in your profiles.yml. Here is an example: ```yml extra_jars: "s3://dbt-glue-hudi/Dependencies/hudi-spark.jar,s3://dbt-glue-hudi/Dependencies/spark-avro_2.11-2.4.4.jar" ``` dbt will run an [atomic `merge` statement](https://hudi.apache.org/docs/writing_data#spark-datasource-writer) which looks nearly identical to the default merge behavior on Snowflake and BigQuery. If a `unique_key` is specified (recommended), dbt will update old records with values from new records that match on the key column. If a `unique_key` is not specified, dbt will forgo match criteria and simply insert all new records (similar to `append` strategy). ##### Source code hudi\_incremental.sql ```sql {{ config( materialized='incremental', incremental_strategy='merge', unique_key='user_id', file_format='hudi' ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from events group by 1 ``` #### Persisting model descriptions Relation-level docs persistence is inherited from dbt-spark, for more details, check [Apache Spark model configuration](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#persisting-model-descriptions). #### Always `schema`, never `database` This section is also inherited from dbt-spark, for more details, check [Apache Spark model configuration](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#always-schema-never-database). --- ### Batch tests Available in v2 The `--batch-tests` flag combines compatible data tests that target the same model into a single query per test type, rather than running one query per test. This reduces the number of queries dbt runs for tests, which can improve performance in projects with many data tests. #### Batching tests Batching applies to `dbt build` and `dbt test`. dbt batches a group of tests when all of these are true: * The tests are `unique` or `not_null` * They're attached to the same model * They use their default configuration * At least two of them are the same type dbt excludes tests from batching when any of the following is set: * `severity` * `error_if` * `warn_if` * `fail_calc` * `limit` * `where` * `store_failures` * `store_failures_as` Excluded tests run on their own, as they do without batching. #### Setting the flag dbt resolves `batch_tests` from three places, in order of precedence: 1. The `--batch-tests` command-line flag 2. The `DBT_ENGINE_BATCH_TESTS` environment variable 3. The `flags` block in your root project's `dbt_project.yml` For example, on the command line: CLI ```bash dbt build --batch-tests ``` With an environment variable: Env var ```bash export DBT_ENGINE_BATCH_TESTS=true dbt build ``` In your `dbt_project.yml`: dbt\_project.yml ```yaml flags: batch_tests: true ``` --- ### batch_size Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### Definition The `batch_size` config determines how large batches are when running a [microbatch incremental model](https://docs.getdbt.com/docs/build/incremental-microbatch.md). Accepted values are `hour`, `day`, `month`, or `year`. You can configure `batch_size` for a [model](https://docs.getdbt.com/docs/build/models.md) in your project YAML file (`dbt_project.yml`), properties YAML file, or config block. #### Examples The following examples set `day` as the `batch_size` for the `user_sessions` model. Example of the `batch_size` config in the `dbt_project.yml` file: dbt\_project.yml ```yml models: my_project: user_sessions: +batch_size: day ``` Example in a property file: models/properties.yml ```yml models: - name: user_sessions config: batch_size: day ``` Example in a config block for a model: models/user\_sessions.sql ```sql {{ config( materialized='incremental', batch_size='day' ) }} ``` --- ### begin Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### Definition Set the `begin` config to the timestamp value at which your [microbatch incremental model](https://docs.getdbt.com/docs/build/incremental-microbatch.md) data should begin — at the point the data becomes relevant for the microbatch model. You can configure `begin` for a [model](https://docs.getdbt.com/docs/build/models.md) in your project YAML file (`dbt_project.yml`), properties YAML file, or SQL file config. The value for `begin` must be a string representing an ISO-formatted date, *or* date and time, *or* [relative dates](#set-begin-to-use-relative-dates). Check out the [examples](#examples) in the next section for more details. #### Examples The following examples set `2024-01-01 00:00:00` as the `begin` config for the `user_sessions` model. ###### Example in the `dbt_project.yml` file dbt\_project.yml ```yml models: my_project: user_sessions: +begin: "2024-01-01 00:00:00" ``` ###### Example in a property YAML file models/properties.yml ```yml models: - name: user_sessions config: begin: "2024-01-01 00:00:00" ``` ###### Example in a SQL config block for a model models/user\_sessions.sql ```sql {{ config( begin='2024-01-01 00:00:00' ) }} ``` ###### Set `begin` to use relative dates To configure `begin` to use relative dates, you can use modules variables [`modules.datetime`](https://docs.getdbt.com/reference/dbt-jinja-functions/modules.md#datetime) and [`modules.pytz`](https://docs.getdbt.com/reference/dbt-jinja-functions/modules.md#pytz) to dynamically specify relative timestamps, such as yesterday's date or the start of the current week. For example, to set `begin` to yesterday's date: ```sql {{ config( materialized = 'incremental', incremental_strategy='microbatch', unique_key = 'run_id', begin=(modules.datetime.datetime.now() - modules.datetime.timedelta(1)).isoformat(), event_time='created_at', batch_size='day', ) }} ``` --- ### BigQuery adapter behavior changes #### The `bigquery_use_batch_source_freshness` flag The `bigquery_use_batch_source_freshness` flag is `false` by default. Setting it to `true` in your `dbt_project.yml` file enables dbt to compute `source freshness` results with a single batched query to BigQuery's [`INFORMATION_SCHEMA.TABLE_STORAGE`](https://cloud.google.com/bigquery/docs/information-schema-table-storage) view as opposed to sending a metadata request for each source. Setting this flag to `true` improves the performance of the `source freshness` command significantly, especially when a project contains a large (1000+) number of sources. Using `loaded_at_field` with batch freshness With `bigquery_use_batch_source_freshness` enabled, dbt determines freshness from BigQuery metadata tables. When you configure a `loaded_at_field` on a source, dbt runs a column-based freshness check for that source instead of metadata-based freshness. However, if `loaded_at_field` is set on *all* sources, freshness fails with a compilation error (`list object has no element 0` in `get_relation_last_modified`). This occurs because the `bigquery_use_batch_source_freshness` flag assumes at least one source uses metadata-based freshness; configuring `loaded_at_field` on all sources breaks this assumption. To avoid this, remove `loaded_at_field` from any sources you want checked using batch freshness. (Applies to dbt v1.12 and later) #### The `bigquery_reject_wildcard_metadata_source_freshness` flag When a BigQuery source uses a wildcard table identifier (for example, `events_*`), metadata-based source freshness checks return incorrect results. BigQuery's `client.get_table()` method creates a temporary union table for wildcard identifiers whose modified timestamp reflects the current time — not the actual modification time of the underlying tables. This makes freshness checks report an age of approximately 0 seconds, masking stale data without any warning. The `bigquery_reject_wildcard_metadata_source_freshness` flag controls how dbt handles metadata-based source freshness checks for BigQuery sources that use wildcard table identifiers. By default, this flag is set to `false`. Tables with wildcard identifiers continue to run metadata-based freshness checks, but dbt emits a deprecation warning advising users to opt in to the new behavior. For example: ```text WARNING: Raise an error when metadata-based source freshness is used with a wildcard table identifier (e.g. 'events_*'). BigQuery returns the current time as the modified timestamp for wildcard tables, causing freshness checks to always report ~0 seconds. You may opt into the new behavior sooner by setting `flags.bigquery_reject_wildcard_metadata_source_freshness` to `true` in `dbt_project.yml`. ``` When you set this flag to `true`, dbt raises a `DbtRuntimeError` when you run metadata-based source freshness checks with wildcard table identifiers. For example: ```text Runtime Error in source my_* (models/schema.yml) Metadata-based source freshness is not supported for wildcard table '`dbt-test-env`.`dbt_username`.`events_*`'. Please set 'loaded_at_field' on this source to use a query-based freshness check instead. ``` To calculate freshness for wildcard tables, configure [`loaded_at_field`](https://docs.getdbt.com/reference/resource-configs/freshness.md) on the source to use query-based freshness checks instead. Example configuration: models/schema.yml ```yaml sources: - name: events database: dbt-test-env schema: dbt_username tables: - name: events identifier: "events_*" # wildcard table identifier in BigQuery config: freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at ``` #### The `bigquery_use_standard_sql_for_partitions` flag BigQuery has [deprecated legacy SQL](https://docs.cloud.google.com/bigquery/docs/release-notes#February_25_2026). The `get_partitions_metadata()` macro currently uses legacy SQL with `$__PARTITIONS_SUMMARY__`, and won't work on projects that don't have [legacy SQL support](https://docs.cloud.google.com/bigquery/docs/legacy-sql-feature-availability) The `bigquery_use_standard_sql_for_partitions` flag controls whether dbt uses standard SQL (`INFORMATION_SCHEMA.PARTITIONS`) instead of legacy SQL (`$__PARTITIONS_SUMMARY__`) when calling `get_partitions_metadata()`. By default, this flag is set to `true`, enabling standard SQL. To revert to legacy SQL, set the flag to `false` in your `dbt_project.yml`. When switching from legacy SQL, note the following: * **Cost:** `$__PARTITIONS_SUMMARY__` is free to query. `INFORMATION_SCHEMA.PARTITIONS` is billed per query at BigQuery's flat-rate pricing, not per-byte. * **Column differences:** Only `partition_id` is compatible between the two sources. The legacy meta-table also returned `project_id`, `dataset_id`, `table_id`, `creation_time`, and `last_modified_time`. If you have custom macros that access those columns from `get_partitions_metadata()` results, you must update them. * **IAM permissions:** `INFORMATION_SCHEMA.PARTITIONS` requires `bigquery.tables.get` and `bigquery.tables.list` instead of `bigquery.tables.getData`. Custom IAM roles may need updating. Standard predefined roles (`roles/bigquery.dataViewer`, `roles/bigquery.dataEditor`, `roles/bigquery.admin`) already include all required permissions and are not affected. --- ### BigQuery configurations #### Use `project` and `dataset` in configurations * `schema` is interchangeable with the BigQuery concept `dataset` * `database` is interchangeable with the BigQuery concept of `project` For our reference documentation, you can declare `project` in place of `database.` This will allow you to read and write from multiple BigQuery projects. Same for `dataset`. #### Using table partitioning and clustering ##### Partition clause BigQuery supports the use of a [partition by](https://cloud.google.com/bigquery/docs/data-definition-language#specifying_table_partitioning_options) clause to easily partition a table by a column or expression. This option can help decrease latency and cost when querying large tables. Note that partition pruning [only works](https://cloud.google.com/bigquery/docs/querying-partitioned-tables#use_a_constant_filter_expression) when partitions are filtered using literal values (so selecting partitions using a subquery won't improve performance). The `partition_by` config can be supplied as a dictionary with the following format: ```python { "field": "<field name>", "data_type": "<timestamp | date | datetime | int64>", "granularity": "<hour | day | month | year>" # Only required if data_type is "int64" "range": { "start": <int>, "end": <int>, "interval": <int> } } ``` ###### Partitioning by a date or timestamp When using a `datetime` or `timestamp` column to partition data, you can create partitions with a granularity of hour, day, month, or year. A `date` column supports granularity of day, month and year. Daily partitioning is the default for all column types. If the `data_type` is specified as a `date` and the granularity is day, dbt will supply the field as-is when configuring table partitioning. ##### Source code bigquery\_table.sql ```sql {{ config( materialized='table', partition_by={ "field": "created_at", "data_type": "timestamp", "granularity": "day" } )}} select user_id, event_name, created_at from {{ ref('events') }} ``` ##### Compiled code bigquery\_table.sql ```sql create table `projectname`.`analytics`.`bigquery_table` partition by timestamp_trunc(created_at, day) as ( select user_id, event_name, created_at from `analytics`.`events` ) ``` ###### Partitioning by an "ingestion" date or timestamp BigQuery supports an [older mechanism of partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#ingestion_time) based on the time when each row was ingested. While we recommend using the newer and more ergonomic approach to partitioning whenever possible, for very large datasets, there can be some performance improvements to using this older, more mechanistic approach. [Read more about the `insert_overwrite` incremental strategy below](#copying-ingestion-time-partitions). dbt will always instruct BigQuery to partition your table by the values of the column specified in `partition_by.field`. By configuring your model with `partition_by.time_ingestion_partitioning` set to `True`, dbt will use that column as the input to a `_PARTITIONTIME` pseudocolumn. Unlike with newer column-based partitioning, you must ensure that the values of your partitioning column match exactly the time-based granularity of your partitions. ##### Source code bigquery\_table.sql ```sql {{ config( materialized="incremental", partition_by={ "field": "created_date", "data_type": "timestamp", "granularity": "day", "time_ingestion_partitioning": true } ) }} select user_id, event_name, created_at, -- values of this column must match the data type + granularity defined above timestamp_trunc(created_at, day) as created_date from {{ ref('events') }} ``` ##### Compiled code bigquery\_table.sql ```sql create table `projectname`.`analytics`.`bigquery_table` (`user_id` INT64, `event_name` STRING, `created_at` TIMESTAMP) partition by timestamp_trunc(_PARTITIONTIME, day); insert into `projectname`.`analytics`.`bigquery_table` (_partitiontime, `user_id`, `event_name`, `created_at`) select created_date as _partitiontime, * EXCEPT(created_date) from ( select user_id, event_name, created_at, -- values of this column must match granularity defined above timestamp_trunc(created_at, day) as created_date from `projectname`.`analytics`.`events` ); ``` ###### Partitioning with integer buckets If the `data_type` is specified as `int64`, then a `range` key must also be provided in the `partition_by` dict. dbt will use the values provided in the `range` dict to generate the partitioning clause for the table. ##### Source code bigquery\_table.sql ```sql {{ config( materialized='table', partition_by={ "field": "user_id", "data_type": "int64", "range": { "start": 0, "end": 100, "interval": 10 } } )}} select user_id, event_name, created_at from {{ ref('events') }} ``` ##### Compiled code bigquery\_table.sql ```sql create table analytics.bigquery_table partition by range_bucket( user_id, generate_array(0, 100, 10) ) as ( select user_id, event_name, created_at from analytics.events ) ``` ###### Additional partition configs If your model has `partition_by` configured, you may optionally specify two additional configurations: * `require_partition_filter` (boolean): If set to `true`, anyone querying this model *must* specify a partition filter, otherwise their query will fail. This is recommended for very large tables with obvious partitioning schemes, such as event streams grouped by day. Note that this will affect other dbt models or tests that try to select from this model, too. * `partition_expiration_days` (integer): If set for date- or timestamp-type partitions, the partition will expire that many days after the date it represents. E.g. A partition representing `2021-01-01`, set to expire after 7 days, will no longer be queryable as of `2021-01-08`, its storage costs zeroed out, and its contents will eventually be deleted. Note that [table expiration](#controlling-table-expiration) will take precedence if specified. bigquery\_table.sql ```sql {{ config( materialized = 'table', partition_by = { "field": "created_at", "data_type": "timestamp", "granularity": "day" }, require_partition_filter = true, partition_expiration_days = 7 )}} ``` ##### Clustering clause BigQuery tables can be [clustered](https://cloud.google.com/bigquery/docs/clustered-tables) to colocate related data. Clustering on a single column: bigquery\_table.sql ```sql {{ config( materialized = "table", cluster_by = "order_id", ) }} select * from ... ``` Clustering on multiple columns: bigquery\_table.sql ```sql {{ config( materialized = "table", cluster_by = ["customer_id", "order_id"], ) }} select * from ... ``` #### Using Reservations The `reservation` config routes dbt-submitted BigQuery jobs to a specific [reservation](https://docs.cloud.google.com/bigquery/docs/reservations-workload-management#flexible). You can set `reservation` at three levels, from lowest to highest precedence: 1. **Target level** (`profiles.yml`) — applies to all jobs for the target. See [Connect BigQuery](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md?version=1.12#reservation). 2. **Project level** (`dbt_project.yml`) — applies to all matching models. dbt\_project.yml ```yaml models: my_project: +reservation: 'projects/abc-123/locations/US/reservations/my-reservation' ``` 3. **Model level** (`{{ config(...) }}`) — overrides project and target settings for a single model. models/my\_model.sql ```sql {{ config( reservation='projects/abc-123/locations/US/reservations/my-reservation' ) }} select ... ``` #### Managing KMS encryption [Customer managed encryption keys](https://cloud.google.com/bigquery/docs/customer-managed-encryption) can be configured for BigQuery tables using the `kms_key_name` model configuration. ##### Using KMS encryption To specify the KMS key name for a model (or a group of models), use the `kms_key_name` model configuration. The following example sets the `kms_key_name` for all of the models in the `encrypted/` directory of your dbt project. dbt\_project.yml ```yaml name: my_project version: 1.0.0 ... models: my_project: encrypted: +kms_key_name: 'projects/PROJECT_ID/locations/global/keyRings/test/cryptoKeys/quickstart' ``` #### Labels and tags ##### Specifying labels dbt supports the specification of BigQuery labels for the tables and views that it creates. These labels can be specified using the `labels` model config. The `labels` config can be provided in a model config, or in the `dbt_project.yml` file, as shown below. A model with no `labels` config inherits the labels from `dbt_project.yml`. Labels set in a model config take precedence over labels set in the `dbt_project.yml`. dbt doesn't combine the two sets. BigQuery key-value pair entries for labels larger than 63 characters are truncated. **Configuring labels in a model file** model.sql ```sql {{ config( materialized = "table", labels = {'contains_pii': 'yes', 'contains_pie': 'no'} ) }} select * from {{ ref('another_model') }} ``` **Configuring labels in dbt\_project.yml** dbt\_project.yml ```yaml models: my_project: snowplow: +labels: domain: clickstream finance: +labels: domain: finance ``` caution In `dbt_project.yml`, quote label values that YAML might interpret as booleans or numbers, such as `yes`, `no`, `on`, `off`, `true`, `false`, or `123`. Label values must be strings, so write `test: "yes"`. In a model `config()` block, quote string values, for example `labels={'contains_pii': 'yes'}`. [![Viewing labels in the BigQuery console](/img/docs/building-a-dbt-project/building-models/73eaa8a-Screen_Shot_2020-01-20_at_12.12.54_PM.png?v=2 "Viewing labels in the BigQuery console")](#)Viewing labels in the BigQuery console ##### Applying labels to jobs While the `labels` configuration applies labels to the tables and views created by dbt, you can also apply labels to the BigQuery *jobs* that dbt runs. Job labels are useful for tracking query costs, monitoring job performance, and organizing your BigQuery job history by dbt metadata. By default, labels are not applied to jobs directly. However, you can enable job labeling through query comments by following these steps: ###### Step 1 Define the `query_comment` macro to add labels to your queries via the query comment: ```sql -- macros/query_comment.sql {% macro query_comment(node) %} {%- set comment_dict = {} -%} {%- do comment_dict.update( app='dbt', dbt_version=dbt_version, profile_name=target.get('profile_name'), target_name=target.get('target_name'), ) -%} {%- if node is not none -%} {%- do comment_dict.update(node.config.get("labels", {})) -%} {% else %} {%- do comment_dict.update(node_id='internal') -%} {%- endif -%} {% do return(tojson(comment_dict)) %} {% endmacro %} ``` This macro creates a JSON comment containing dbt metadata (app, version, profile, target) and merges in any model-specific labels you've configured. ###### Step 2 Enable job labeling in your `dbt_project.yml` by setting `comment: "{{ query_comment(node) }}"` and `job-label: true` in the `query-comment` configuration: ```yaml # dbt_project.yml name: analytics profile: bq version: "1.0.0" models: analytics: +materialized: table query-comment: comment: "{{ query_comment(node) }}" job-label: true ``` When enabled, BigQuery will parse the JSON comment and apply the key-value pairs as labels to each job. You can then filter and analyze jobs in the BigQuery console or via the INFORMATION\_SCHEMA.JOBS view using these labels. ##### Specifying tags BigQuery table and view *tags* can be created by supplying an empty string for the label value. model.sql ```sql {{ config( materialized = "table", labels = {'contains_pii': ''} ) }} select * from {{ ref('another_model') }} ``` You can create a new label with no value or remove a value from an existing label key. A label with a key that has an empty value can also be referred to as a [tag](https://cloud.google.com/bigquery/docs/adding-labels#adding_a_label_without_a_value) in BigQuery. However, this is different from a [BigQuery tag](https://cloud.google.com/bigquery/docs/tags), which conditionally applies IAM policies to BigQuery tables and datasets. For more information, see the [Tags documentation](https://cloud.google.com/resource-manager/docs/tags/tags-overview). ##### Resource tags [BigQuery tags](https://cloud.google.com/bigquery/docs/tags) enable conditional IAM access control for BigQuery tables and views. You can apply these BigQuery tags using the `resource_tags` configuration. This section contains guidelines for using the `resource_tags` configuration parameter. Resource tags are key-value pairs that must follow BigQuery's tag format: `{google_cloud_project_id}/{key_name}: value`. Unlike labels, BigQuery tags are primarily designed for IAM access control using conditional policies, allowing organizations to: * **Implement conditional access control**: Apply IAM policies conditionally based on BigQuery tags (for example, granting access only to tables tagged with `environment:production`). * **Enforce data governance**: Use BigQuery tags with IAM policies to protect sensitive data. * **Control access at scale**: Manage access patterns consistently across different projects and environments. ###### Prerequisites * [Create tag keys and values](https://cloud.google.com/bigquery/docs/tags#create_tag_keys_and_values) in advance before using them in dbt. * Grant the [required IAM permissions](https://cloud.google.com/bigquery/docs/tags#required_permissions) to apply tags to resources. ###### Configuring tags in a model file To configure tags in a model file, refer to the following example: model.sql ```sql {{ config( materialized = "table", resource_tags = { "my-project-id/environment": "production", "my-project-id/data_classification": "sensitive", "my-project-id/access_level": "restricted" } ) }} select * from {{ ref('another_model') }} ``` ###### Configuring tags in `dbt_project.yml` To configure tags in a `dbt_project.yml` file, refer to the following example: dbt\_project.yml ```yaml models: my_project: production: +resource_tags: my-project-id/environment: production my-project-id/data_classification: sensitive staging: +resource_tags: my-project-id/environment: staging my-project-id/data_classification: internal ``` ###### Using both dbt tags and BigQuery tags You can use dbt's existing `tags` configuration alongside BigQuery's `resource_tags`: model.sql ```sql {{ config( materialized = "materialized_view", tags = ["reporting", "daily"], # dbt tags for internal organization resource_tags = { # BigQuery tags for IAM access control "my-project-id/environment": "production", "my-project-id/data_classification": "sensitive" } ) }} select * from {{ ref('my_table') }} ``` For more information on setting up IAM conditional policies with BigQuery tags, see BigQuery's documentation on [tags](https://cloud.google.com/bigquery/docs/tags). ##### Policy tags BigQuery enables [column-level security](https://cloud.google.com/bigquery/docs/column-level-security-intro) by setting [policy tags](https://cloud.google.com/bigquery/docs/best-practices-policy-tags) on specific columns. dbt enables this feature as a column resource property, `policy_tags` (*not* a node config). models/\<filename>.yml ```yaml models: - name: policy_tag_table columns: - name: field policy_tags: - 'projects/<gcp-project>/locations/<location>/taxonomies/<taxonomy>/policyTags/<tag>' ``` Please note that in order for policy tags to take effect, [column-level `persist_docs`](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) must be enabled for the model, seed, or snapshot. Consider using [variables](https://docs.getdbt.com/docs/build/project-variables.md) to manage taxonomies and make sure to add the required security [roles](https://cloud.google.com/bigquery/docs/column-level-security-intro#roles) to your BigQuery service account key. #### Merge behavior (incremental models) The [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-strategy.md) controls how dbt builds incremental models. dbt uses a [merge statement](https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax) on BigQuery to refresh incremental tables. The `incremental_strategy` config can be set to one of the following values: * `merge` (default) * `insert_overwrite` * [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md) ##### Change history The `enable_change_history` parameter enables [BigQuery's change history feature](https://cloud.google.com/bigquery/docs/change-history) which tracks changes made to a BigQuery table. When enabled, you can use the change history to audit and debug the behavior of your incremental models. `enable_change_history` is set to `<boolean>` values. ##### Performance and cost The operations performed by dbt while building a BigQuery incremental model can be made cheaper and faster by using a [clustering clause](#clustering-clause) in your model configuration. See [this guide](https://discourse.getdbt.com/t/benchmarking-incremental-strategies-on-bigquery/981) for more information on performance tuning for BigQuery incremental models. **Note:** These performance and cost benefits are applicable to incremental models built with either the `merge` or the `insert_overwrite` incremental strategy. ##### The `merge` strategy The `merge` incremental strategy will generate a `merge` statement that looks something like: ```merge merge into {{ destination_table }} DEST using ({{ model_sql }}) SRC on SRC.{{ unique_key }} = DEST.{{ unique_key }} when matched then update ... when not matched then insert ... ``` The 'merge' approach automatically updates new data in the destination incremental table but requires scanning all source tables referenced in the model SQL, as well as destination tables. This can be slow and expensive for large data volumes. [Partitioning and clustering](#using-table-partitioning-and-clustering) techniques mentioned earlier can help mitigate these issues. **Note:** The `unique_key` configuration is required when the `merge` incremental strategy is selected. ##### The `insert_overwrite` strategy The `insert_overwrite` strategy generates a merge statement that replaces entire partitions in the destination table. **Note:** this configuration requires that the model is configured with a [Partition clause](#partition-clause). The `merge` statement that dbt generates when the `insert_overwrite` strategy is selected looks something like: ```sql /* Create a temporary table from the model SQL */ create temporary table {{ model_name }}__dbt_tmp as ( {{ model_sql }} ); /* If applicable, determine the partitions to overwrite by querying the temp table. */ declare dbt_partitions_for_replacement array<date>; set (dbt_partitions_for_replacement) = ( select as struct array_agg(distinct date(max_tstamp)) from `my_project`.`my_dataset`.{{ model_name }}__dbt_tmp ); /* Overwrite partitions in the destination table which match the partitions in the temporary table */ merge into {{ destination_table }} DEST using {{ model_name }}__dbt_tmp SRC on FALSE when not matched by source and {{ partition_column }} in unnest(dbt_partitions_for_replacement) then delete when not matched then insert ... ``` For a complete writeup on the mechanics of this approach, see [this explainer post](https://discourse.getdbt.com/t/bigquery-dbt-incremental-changes/982). ###### Determining partitions to overwrite dbt is able to determine the partitions to overwrite dynamically from the values present in the temporary table, or statically using a user-supplied configuration. The "dynamic" approach is simplest (and the default), but the "static" approach will reduce costs by eliminating multiple queries in the model build script. ###### Static partitions To supply a static list of partitions to overwrite, use the `partitions` configuration. models/session.sql ```sql {% set partitions_to_replace = [ 'timestamp(current_date)', 'timestamp(date_sub(current_date, interval 1 day))' ] %} {{ config( materialized = 'incremental', incremental_strategy = 'insert_overwrite', partition_by = {'field': 'session_start', 'data_type': 'timestamp'}, partitions = partitions_to_replace ) }} with events as ( select * from {{ref('events')}} {% if is_incremental() %} -- recalculate yesterday + today where timestamp_trunc(event_timestamp, day) in ({{ partitions_to_replace | join(',') }}) {% endif %} ), ... rest of model ... ``` This example model serves to replace the data in the destination table for both *today* and *yesterday* every day that it is run. It is the fastest and cheapest way to incrementally update a table using dbt. If we wanted this to run more dynamically— let’s say, always for the past 3 days—we could leverage dbt’s baked-in [datetime macros](https://github.com/dbt-labs/dbt/blob/dev/octavius-catto/core/dbt/include/global_project/macros/etc/datetime.sql) and write a few of our own. Think of this as "full control" mode. You must ensure that expressions or literal values in the `partitions` config have proper quoting when templated, and that they match the `partition_by.data_type` (`timestamp`, `datetime`, `date`, or `int64`). Otherwise, the filter in the incremental `merge` statement will raise an error. ###### Dynamic partitions If no `partitions` configuration is provided, dbt will instead: 1. Create a temporary table for your model SQL 2. Query the temporary table to find the distinct partitions to be overwritten 3. Query the destination table to find the *max* partition in the database When building your model SQL, you can take advantage of the introspection performed by dbt to filter for only *new* data. The maximum value in the partitioned field in the destination table will be available using the `_dbt_max_partition` BigQuery scripting variable. **Note:** this is a BigQuery SQL variable, not a dbt Jinja variable, so no Jinja brackets are required to access this variable. **Example model SQL:** ```sql {{ config( materialized = 'incremental', partition_by = {'field': 'session_start', 'data_type': 'timestamp'}, incremental_strategy = 'insert_overwrite' ) }} with events as ( select * from {{ref('events')}} {% if is_incremental() %} -- recalculate latest day's data + previous -- NOTE: The _dbt_max_partition variable is used to introspect the destination table where date(event_timestamp) >= date_sub(date(_dbt_max_partition), interval 1 day) {% endif %} ), ... rest of model ... ``` ###### Copying partitions If you are replacing entire partitions in your incremental runs, you can opt to do so with the [copy table API](https://cloud.google.com/bigquery/docs/managing-tables#copy-table) and partition decorators rather than a `merge` statement. While this mechanism doesn't offer the same visibility and ease of debugging as the SQL `merge` statement, it can yield significant savings in time and cost for large datasets because the copy table API does not incur any costs for inserting the data - it's equivalent to the `bq cp` gcloud command line interface (CLI) command. You can enable this by switching on `copy_partitions: True` in the `partition_by` configuration. This approach works only in combination with "dynamic" partition replacement. bigquery\_table.sql ```sql {{ config( materialized="incremental", incremental_strategy="insert_overwrite", partition_by={ "field": "created_date", "data_type": "timestamp", "granularity": "day", "time_ingestion_partitioning": true, "copy_partitions": true } ) }} select user_id, event_name, created_at, -- values of this column must match the data type + granularity defined above timestamp_trunc(created_at, day) as created_date from {{ ref('events') }} ``` logs/dbt.log ```text ... [0m16:03:13.017641 [debug] [Thread-3 (]: BigQuery adapter: Copying table(s) "/projects/projectname/datasets/analytics/tables/bigquery_table__dbt_tmp$20230112" to "/projects/projectname/datasets/analytics/tables/bigquery_table$20230112" with disposition: "WRITE_TRUNCATE" ... ``` #### Controlling table expiration By default, dbt-created tables never expire. You can configure certain model(s) to expire after a set number of hours by setting `hours_to_expiration`. dbt\_project.yml ```yml models: <resource-path>: +hours_to_expiration: 6 ``` models/\<modelname>.sql ```sql {{ config( hours_to_expiration = 6 ) }} select ... ``` The `hours_to_expiration` config only applies when the underlying table is first created. It doesn't reset on incremental runs.To work around this, call a macro in your +post-hook that resets the expiration timestamp manually. Update the hours argument to match your hours\_to\_expiration value. **Example macro SQL:** ```sql {% macro reset_expiration_for_incremental(hours) %} -- Check if the current model is being run incrementally {% if is_incremental() %} -- Set expiration timestamp because it's only applied on creation, not on merge ALTER TABLE {{ this }} SET OPTIONS (expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL {{ hours }} HOUR)) {% endif %} {% endmacro %} ``` dbt\_project.yml ```yml models: my_project: +post-hook: - "{{ reset_expiration_for_incremental(hours=6) }}" ``` #### Authorized views If the `grant_access_to` config is specified for a model materialized as a view, dbt will grant the view model access to select from the list of datasets provided. See [BQ docs on authorized views](https://cloud.google.com/bigquery/docs/share-access-views) for more details. Note The `grants` config and the `grant_access_to` config are distinct. * **`grant_access_to`:** Enables you to set up authorized views. When configured, dbt provides an authorized view access to show partial information from other datasets, without providing end users with full access to those underlying datasets. For more information, see ["BigQuery configurations: Authorized views"](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#authorized-views) * **`grants`:** Provides specific permissions to users, groups, or service accounts for managing access to datasets you're producing with dbt. For more information, see ["Resource configs: grants"](https://docs.getdbt.com/reference/resource-configs/grants.md) You can use the two features together: "authorize" a view model with the `grants_access_to` configuration, and then add `grants` to that view model to share its query results (and *only* its query results) with other users, groups, or service accounts. dbt\_project.yml ```yml models: <resource-path>: +grant_access_to: - project: project_1 dataset: dataset_1 - project: project_2 dataset: dataset_2 ``` models/\<modelname>.sql ```sql {{ config( grant_access_to=[ {'project': 'project_1', 'dataset': 'dataset_1'}, {'project': 'project_2', 'dataset': 'dataset_2'} ] ) }} ``` Views with this configuration will be able to select from objects in `project_1.dataset_1` and `project_2.dataset_2`, even when they are located elsewhere and queried by users who do not otherwise have access to `project_1.dataset_1` and `project_2.dataset_2`. #### Materialized views The BigQuery adapter supports [materialized views](https://cloud.google.com/bigquery/docs/materialized-views-intro) with the following configuration parameters: | Parameter | Type | Required | Default | Change Monitoring Support | | ---------------------------------------------------------------------------------------------------------- | ---------------------- | -------- | ------- | ------------------------- | | [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) | `<string>` | no | `apply` | n/a | | [`cluster_by`](#clustering-clause) | `[<string>]` | no | `none` | drop/create | | [`partition_by`](#partition-clause) | `{<dictionary>}` | no | `none` | drop/create | | [`enable_refresh`](#auto-refresh) | `<boolean>` | no | `true` | alter | | [`refresh_interval_minutes`](#auto-refresh) | `<float>` | no | `30` | alter | | [`max_staleness`](#auto-refresh) (in Preview) | `<interval>` | no | `none` | alter | | [`description`](https://docs.getdbt.com/reference/resource-properties/description.md) | `<string>` | no | `none` | alter | | [`labels`](#specifying-labels) | `{<string>: <string>}` | no | `none` | alter | | [`resource_tags`](#resource-tags) | `{<string>: <string>}` | no | `none` | alter | | [`hours_to_expiration`](#controlling-table-expiration) | `<integer>` | no | `none` | alter | | [`kms_key_name`](#using-kms-encryption) | `<string>` | no | `none` | alter | ##### Project file dbt\_project.yml ```yaml models: <resource-path>: +materialized: materialized_view +on_configuration_change: apply | continue | fail +cluster_by: <field-name> | [<field-name>] +partition_by: - field: <field-name> - data_type: timestamp | date | datetime | int64 # only if `data_type` is not 'int64' - granularity: hour | day | month | year # only if `data_type` is 'int64' - range: - start: <integer> - end: <integer> - interval: <integer> +enable_refresh: true | false +refresh_interval_minutes: <float> +max_staleness: <interval> +description: <string> +labels: {<label-name>: <label-value>} +resource_tags: {<tag-key>: <tag-value>} +hours_to_expiration: <integer> +kms_key_name: <path-to-key> ``` ##### Property file models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: materialized_view on_configuration_change: apply | continue | fail cluster_by: <field-name> | [<field-name>] partition_by: - field: <field-name> - data_type: timestamp | date | datetime | int64 # only if `data_type` is not 'int64' - granularity: hour | day | month | year # only if `data_type` is 'int64' - range: - start: <integer> - end: <integer> - interval: <integer> enable_refresh: true | false refresh_interval_minutes: <float> max_staleness: <interval> description: <string> labels: {<label-name>: <label-value>} resource_tags: {<tag-key>: <tag-value>} hours_to_expiration: <integer> kms_key_name: <path-to-key> ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized='materialized_view', on_configuration_change="apply" | "continue" | "fail", cluster_by="<field-name>" | ["<field-name>"], partition_by={ "field": "<field-name>", "data_type": "timestamp" | "date" | "datetime" | "int64", # only if `data_type` is not 'int64' "granularity": "hour" | "day" | "month" | "year, # only if `data_type` is 'int64' "range": { "start": <integer>, "end": <integer>, "interval": <integer>, } }, # auto-refresh options enable_refresh= true | false, refresh_interval_minutes=<float>, max_staleness="<interval>", # additional options description="<description>", labels={ "<label-name>": "<label-value>", }, resource_tags={ "<tag-key>": "<tag-value>", }, hours_to_expiration=<integer>, kms_key_name="<path_to_key>", ) }} ``` Many of these parameters correspond to their table counterparts and have been linked above. The set of parameters unique to materialized views covers [auto-refresh functionality](#auto-refresh). Learn more about these parameters in BigQuery's docs: * [CREATE MATERIALIZED VIEW statement](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_materialized_view_statement) * [materialized\_view\_option\_list](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#materialized_view_option_list) ##### Auto-refresh | Parameter | Type | Required | Default | Change Monitoring Support | | ---------------------------- | ------------ | -------- | ------- | ------------------------- | | `enable_refresh` | `<boolean>` | no | `true` | alter | | `refresh_interval_minutes` | `<float>` | no | `30` | alter | | `max_staleness` (in Preview) | `<interval>` | no | `none` | alter | BigQuery supports [automatic refresh](https://cloud.google.com/bigquery/docs/materialized-views-manage#automatic_refresh) configuration for materialized views. By default, a materialized view will automatically refresh within 5 minutes of changes in the base table, but not more frequently than once every 30 minutes. BigQuery only officially supports the configuration of the frequency (the "once every 30 minutes" frequency); however, there is a feature in preview that allows for the configuration of the staleness (the "5 minutes" refresh). dbt will monitor these parameters for changes and apply them using an `ALTER` statement. Learn more about these parameters in BigQuery's docs: * [materialized\_view\_option\_list](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#materialized_view_option_list) * [max\_staleness](https://cloud.google.com/bigquery/docs/materialized-views-create#max_staleness) ##### Limitations As with most data platforms, there are limitations associated with materialized views. Some worth noting include: * Materialized view SQL has a [limited feature set](https://cloud.google.com/bigquery/docs/materialized-views-create#supported-mvs). * Materialized view SQL cannot be updated; the materialized view must go through a `--full-refresh` (DROP/CREATE). * The `partition_by` clause on a materialized view must match that of the underlying base table. * While materialized views can have descriptions, materialized view *columns* cannot. * Recreating/dropping the base table requires recreating/dropping the materialized view. Find more information about materialized view limitations in Google's BigQuery [docs](https://cloud.google.com/bigquery/docs/materialized-views-intro#limitations). #### Python model configuration **Submission methods:** BigQuery supports a few different mechanisms to submit Python code, each with relative advantages. The `dbt-bigquery` adapter uses BigQuery DataFrames (BigFrames) or Dataproc. This process reads data from BigQuery, computes it either natively with BigQuery DataFrames or Dataproc, and writes the results back to BigQuery. ##### BigQuery DataFrames BigQuery DataFrames can execute pandas and scikit-learn. There's no need to manage infrastructure and leverages BigQuery-distributed query engines. It's great for analysts, data scientists, and machine learning engineers who want to manipulate big data using a pandas-like syntax. **Note:** BigQuery DataFrames run on Google Colab's default runtime. If no `default` runtime template is available, the adapter will automatically create one for you and mark it `default` for next time usage (assuming it has the right permissions). **BigQuery DataFrames setup:** ```bash # IAM permission if using service account #Create Service Account gcloud iam service-accounts create dbt-bigframes-sa #Grant BigQuery User Role gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/bigquery.user #Grant BigQuery Data Editor role. This can be restricted at dataset level gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/bigquery.dataEditor #Grant Service Account user gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/iam.serviceAccountUser #Grant Colab Enterprise User gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} --member=serviceAccount:dbt-bigframes-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role=roles/aiplatform.colabEnterpriseUser ``` dbt\_project.yml ```yaml models: my_dbt_project: submission_method: bigframes ``` profiles.yml ```yaml my_dbt_project_sa: outputs: dev: compute_region: us-central1 dataset: <BIGQUERY_DATASET> gcs_bucket: <GCS BUCKET USED FOR BIGFRAME LOGS> job_execution_timeout_seconds: 300 job_retries: 1 keyfile: <SERVICE ACCOUNT KEY FILE> location: US method: service-account priority: interactive project: <BIGQUERY_PROJECT> threads: 1 type: bigquery target: dev ``` ##### Dataproc Dataproc (`serverless` or pre-configured `cluster`) can execute Python models as PySpark jobs, reading from and writing to BigQuery. `serverless` is simpler but slower with limited configuration and pre-installed packages (`pandas`, `numpy`, `scikit-learn`), while `cluster` offers full control and faster runtimes. Good for complex, long-running batch pipelines and legacy Hadoop/Spark workflows but often slower for ad-hoc or interactive workloads. **Dataproc setup:** * Create or use an existing [Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets). * Enable Dataproc APIs for your project and region. * If using the `cluster` submission method: Create or use an existing [Dataproc cluster](https://cloud.google.com/dataproc/docs/guides/create-cluster) with the [Spark BigQuery connector initialization action](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/connectors#bigquery-connectors). (Google recommends copying the action into your own Cloud Storage bucket, rather than using the example version shown in the screenshot.) [![Add the Spark BigQuery connector as an initialization action](/img/docs/building-a-dbt-project/building-models/python-models/dataproc-connector-initialization.png?v=2 "Add the Spark BigQuery connector as an initialization action")](#)Add the Spark BigQuery connector as an initialization action The following configurations are needed to run Python models on Dataproc. You can add these to your [BigQuery profile](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#running-python-models-on-dataproc) or configure them on specific Python models: * `gcs_bucket`: Storage bucket to which dbt will upload your model's compiled PySpark code. * `dataproc_region`: GCP region in which you have enabled Dataproc (for example `us-central1`). * `dataproc_cluster_name`: Name of Dataproc cluster to use for running Python model (executing PySpark job). Only required if `submission_method: cluster`. ```python def model(dbt, session): dbt.config( submission_method="cluster", dataproc_cluster_name="my-favorite-cluster" ) ... ``` ```yml models: - name: my_python_model config: submission_method: serverless ``` Python models running on Dataproc Serverless can be further configured in your [BigQuery profile](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#running-python-models-on-dataproc). Any user or service account that runs dbt Python models will need the following permissions, in addition to the required BigQuery permissions: ```text dataproc.batches.create dataproc.clusters.use dataproc.jobs.create dataproc.jobs.get dataproc.operations.get dataproc.operations.list storage.buckets.get storage.objects.create storage.objects.delete ``` For more information, refer to [Dataproc IAM roles and permissions](https://cloud.google.com/dataproc/docs/concepts/iam/iam). **Installing packages:** Installation of third-party packages on Dataproc varies depending on whether it's a [cluster](https://cloud.google.com/dataproc/docs/guides/create-cluster) or [serverless](https://cloud.google.com/dataproc-serverless/docs). * **Dataproc Cluster** — Google recommends installing Python packages while creating the cluster via initialization actions: * [How initialization actions are used](https://github.com/GoogleCloudDataproc/initialization-actions/blob/master/README.md#how-initialization-actions-are-used) * [Actions for installing via `pip` or `conda`](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/python) You can also install packages at cluster creation time by [defining cluster properties](https://cloud.google.com/dataproc/docs/tutorials/python-configuration#image_version_20): `dataproc:pip.packages` or `dataproc:conda.packages`. * **Dataproc Serverless** — Google recommends using a [custom docker image](https://cloud.google.com/dataproc-serverless/docs/guides/custom-containers) to install third-party packages. The image needs to be hosted in [Google Artifact Registry](https://cloud.google.com/artifact-registry/docs). It can then be used by providing the image path in dbt profiles: profiles.yml ```yml my-profile: target: dev outputs: dev: type: bigquery method: oauth project: abc-123 dataset: my_dataset # for dbt Python models to be run on Dataproc Serverless gcs_bucket: dbt-python dataproc_region: us-central1 submission_method: serverless dataproc_batch: runtime_config: container_image: {HOSTNAME}/{PROJECT_ID}/{IMAGE}:{TAG} ``` [![Adding packages to install via pip at cluster startup](/img/docs/building-a-dbt-project/building-models/python-models/dataproc-pip-packages.png?v=2 "Adding packages to install via pip at cluster startup")](#)Adding packages to install via pip at cluster startup ##### Additional parameters The BigQuery Python models also have the following additional configuration parameters: | Parameter | Type | Required | Default | Valid values | | ----------------------- | ----------- | -------- | --------- | --------------------------------------------------------- | | `enable_list_inference` | `<boolean>` | no | `True` | `True`, `False` | | `intermediate_format` | `<string>` | no | `parquet` | `parquet`, `orc` | | `submission_method` | `<string>` | no | \`\` | `serverless`, `bigframes`, `cluster` | | `notebook_template_id` | `<integer>` | no | \`\` | `<NOTEBOOK RUNTIME TEMPLATE_ID>` | | `compute_region` | `<string>` | no | \`\` | `<COMPUTE_REGION>` | | `gcs_bucket` | `<string>` | no | \`\` | `<GCS_BUCKET>` | | `packages` | `<string>` | no | \`\` | `['numpy<=1.1.1', 'pandas', 'mlflow']` (`bigframes` only) | | `timeout` | `<integer>` | no | \`\` | `<timeout_in_seconds>` | * The `enable_list_inference` parameter * The `enable_list_inference` parameter enables a PySpark data frame to read multiple records in the same operation. By default, this is set to `True` to support the default `intermediate_format` of `parquet`. * The `intermediate_format` parameter * The `intermediate_format` parameter specifies which file format to use when writing records to a table. The default is `parquet`. * The `submission_method` parameter * The `submission_method` parameter specifies whether the job will run on BigQuery DataFrames or Serverless Spark. `submission_method` is not required when `dataproc_cluster_name` is declared. * The `notebook_template_id` parameter * The `notebook_template_id` parameter specifies runtime template in Colab Enterprise. * The `compute_region` parameter * The `compute_region` parameter specifies the region of the job. * The `gcs_bucket` parameter * The `gcs_bucket` parameter specifies the GCS bucket used for storing artifacts for the job. * The `packages` parameter * The `packages` parameter lists Python packages to `pip install` before your model runs. It only works with `submission_method: bigframes`. Dataproc (`serverless` and `cluster`) ignores it without an error or warning. To install packages on Dataproc, use a custom container image (serverless) or initialization actions (cluster) instead. Refer to **Installing packages** under the Dataproc tab in [Python model configuration](#python-model-configuration). * The `timeout` parameter * The `timeout` parameter specifies the maximum execution time in seconds for the Python model. This is particularly useful for BigFrames models that may require longer execution times for complex data processing or machine learning workloads. If not specified, the model will use the default timeout configured for the execution environment. **Related docs:** * [Dataproc overview](https://cloud.google.com/dataproc/docs/concepts/overview) * [Create a Dataproc cluster](https://cloud.google.com/dataproc/docs/guides/create-cluster) * [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) * [PySpark DataFrame syntax](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.html) #### Unit test limitations You must specify all fields in a BigQuery `STRUCT` for [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md). You cannot use only a subset of fields in a `STRUCT`. --- ### Cache ##### Cache population At the start of runs, dbt caches metadata about all the objects in all the schemas where it might materialize resources (such as models). By default, dbt populates the relational cache with information on all schemas related to the project. There are two ways to optionally modify this behavior: * `POPULATE_CACHE` (default: `True`): Whether to populate the cache at all. To skip cache population entirely, use the `--no-populate-cache` flag or (Applies to dbt v1.11 and later) `DBT_ENGINE_POPULATE_CACHE: False`. Note that this does not *disable* the cache; missed cache lookups will run queries, and update the cache afterward. * `CACHE_SELECTED_ONLY` (default `False`): Whether to limit cache population to just the resources selected in the current run. This can offer significant speed improvements when running a small subset of a large project, while still providing the benefit of caching upfront. For example, to quickly compile a model that requires no database metadata or introspective queries: ```text dbt compile --no-populate-cache --select my_model_name ``` Or, to improve speed and performance while focused on developing Salesforce models, which are materialized into their own dedicated schema, you could select those models and pass the `cache-selected-only` flag: ```text dbt run --cache-selected-only --select salesforce ``` ##### Logging relational cache events The `LOG_CACHE_EVENTS` config allows detailed logging for relational cache, which are disabled by default. ```text dbt compile --log-cache-events ``` --- ### Catalog JSON file **Current schema**: [`v1`](https://schemas.getdbt.com/dbt/catalog/v1.json) **Produced by:** (Applies to dbt v2.0 and later) [`--write-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md#--write-catalog-flag) flag This file contains information from your data warehouse about the tables and views produced and defined by the resources in your project. (Applies to dbt v2.0 and later) [dbt Docs v2](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-v2) reads the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) rather than this file. `catalog.json` remains available as a separate artifact for [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and external tools. ##### Top-level keys * [`metadata`](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md#common-metadata) * `nodes`: Dictionary containing information about database objects corresponding to dbt models, seeds, and snapshots. * `sources`: Dictionary containing information about database objects corresponding to dbt sources. * `errors`: Errors received while running metadata queries during (Applies to dbt v2.0 and later) catalog generation (via `--write-catalog` flag). ##### Resource details Within `sources` and `nodes`, each dictionary key is a resource `unique_id`. Each nested resource contains: * `unique_id`: `<resource_type>.<package>.<resource_name>`, same as dictionary key, maps to `nodes` and `sources` in the [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) * `metadata` * `type`: table, view, etc. * `database` * `schema` * `name` * `comment` * `owner` * `columns` (array) * `name` * `type`: data type * `comment` * `index`: ordinal * `stats`: differs by database and relation type --- ### Caveats to state comparison The [`state:` selection method](https://docs.getdbt.com/reference/node-selection/methods.md#state) is a powerful feature, with a lot of underlying complexity. Below are a handful of considerations when setting up automated jobs that leverage state comparison. ##### Seeds dbt stores a file hash of seed files that are <1 MiB in size. If the contents of these seeds is modified, the seed will be included in `state:modified`. If a seed file is >1 MiB in size, dbt cannot compare its contents and will raise a warning as such. Instead, dbt will use only the seed's file path to detect changes. If the file path has changed, the seed will be included in `state:modified`; if it hasn't, it won't. ##### `tags` and `meta` Changes to `tags` and `meta`, whether set at the resource level or on individual columns in a YAML file, don't count as modifications and will not trigger `state:modified`. dbt treats these fields as metadata only, since they don't affect how a resource is materialized. This is intentional behavior. This is different from `description`, which *is* treated as a modification, but only when [`persist_docs`](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) is enabled. Any other config change counts as a modification, because that config could affect materialization. ##### Macros dbt will mark modified any resource that depends on a changed macro, or on a macro that depends on a changed macro. ##### Vars If a model uses a `var` or `env_var` in its definition, dbt is unable to identify that lineage in such a way that it can include the model in `state:modified` because the `var` or `env_var` value has changed. It's likely that the model will be marked modified if the change in variable results in a different configuration. ##### Tests The command `dbt test -s state:modified` will include both: * tests that select from a new/modified resource * tests that are themselves new or modified As long as you're adding or changing tests at the same time that you're adding or changing the resources (models, seeds, snapshots) they select from, all should work the way you expect with "simple" state selection: ```shell dbt run -s "state:modified" dbt test -s "state:modified" ``` This can get complicated, however. If you add a new test without modifying its underlying model, or add a test that selects from a new model and an old unmodified one, you may need to test a model without having first run it. You can defer upstream references when testing. For example, if a test selects from a model that doesn't exist as a database object in your current environment, dbt will look to the other environment instead—the one defined in your state manifest. This enables you to use "simple" state selection without risk of query failure, but it may have some surprising consequences for tests with multiple parents. For instance, if you have a `relationships` test that depends on one modified model and one unmodified model, the test query will select from data "across" two different environments. If you limit or sample your data in development and CI, it may not make much sense to test for referential integrity, knowing there's a good chance of mismatch. If you're a frequent user of `relationships` tests or data tests, or frequently find yourself adding tests without modifying their underlying models, consider tweaking the selection criteria of your CI job. For instance: ```shell dbt run -s "state:modified" dbt test -s "state:modified" --exclude "test_name:relationships" ``` ##### Overwrites the `manifest.json` dbt overwrites the `manifest.json` file during parsing, which means when you reference `--state` from the `target/ directory`, you may encounter a warning indicating that the saved manifest wasn't found. [![Saved manifest not found error](/img/docs/reference/saved-manifest-not-found.png?v=2 "Saved manifest not found error")](#)Saved manifest not found error During the next job run, dbt follows a sequence of steps that lead to the issue. First, it overwrites `target/manifest.json` before it can be used for change detection. Then, when dbt tries to read `target/manifest.json` again to detect changes, it finds none because the previous state has already been overwritten/erased. Avoid setting `--state` and `--target-path` to the same path with state-dependent features like `--defer` and `state:modified` as it can lead to non-idempotent behavior and won't work as expected. ###### Recommendation To prevent the `manifest.json` from being overwritten before dbt reads it for change detection, update your workflow using one of these methods: * Move the `manifest.json` to a dedicated folder (for example `state/`) after dbt generates it in the `target/ folder`. This makes sure dbt references the correct saved state instead of comparing the current state with the just-overwritten version. It also avoids issues caused by setting `--state` and `--target-path` to the same location, which can lead to non-idempotent behavior. * Write the manifest to a different `--target-path` in the build stage (where dbt would generate the `target/manifest.json`) or before it gets overwritten during job execution to avoid issues with change detection. This allows dbt to detect changes instead of comparing the current state with the just-overwritten version. * Pass the `--no-write-json` flag: `dbt ls --no-write-json --select state:modified --state target`: during the reproduction stage. ##### False positives (Applies to dbt v1.9 and later) To reduce false positives during `state:modified` selection due to env-aware logic, you can set the `state_modified_compare_more_unrendered_values` [behavior flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md) to `true`. You need to build the state directory using dbt v1.9 or higher, or [the dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), and you need to set `state_modified_compare_more_unrendered_values` to `true` within your dbt\_project.yml. If the state directory was built with an older dbt version or if the `state_modified_compare_more_unrendered_values` behavior change flag was either not set or set to `false`, you need to rebuild the state directory to avoid false positives during state comparison with `state:modified`. ##### Scheduled jobs [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state) and `state:modified+` only detect changes to your project's code, configuration, or manifest-relevant metadata — not changes to the data itself, like new rows landing in a source table. When dbt detects no code or configuration changes since the deferred manifest was last produced, the job succeeds without building any models. This is expected behavior. If your job needs to build models on every scheduled run regardless of code changes, remove the `state:modified` selector from that job. Reserve it for [CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md) or [merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md), where the intent is to build only what changed in a given pull request or merge. ##### Final note State comparison is complex. We hope to reach eventual consistency between all configuration options, as well as providing users with the control they need to reliably return all modified resources, and only the ones they expect. If you're interested in learning more, read [open issues tagged "state"](https://github.com/dbt-labs/dbt/issues?q=is%3Aopen+is%3Aissue+label%3Astate) in the dbt repository. #### Related docs * [About state in dbt](https://docs.getdbt.com/reference/node-selection/state-selection.md) * [Configure state selection](https://docs.getdbt.com/reference/node-selection/configure-state.md) --- ### Check configurations Available in v2 #### Related documentation * [Checks](https://docs.getdbt.com/docs/build/checks.md) * [Check properties](https://docs.getdbt.com/reference/check-properties.md) * [`dbt check` command](https://docs.getdbt.com/reference/commands/check.md) #### Available configurations ##### Project YAML file dbt\_project.yml ```yaml checks: <resource-path>: +severity: error | warn +enabled: true | false +selection_filter_on: column_name | [column_names] | none +tags: string | [string] +meta: {dictionary} ``` ##### Properties YAML file checks/\_checks.yml ```yaml version: 2 checks: - name: check-name config: severity: error | warn enabled: true | false selection_filter_on: column_name | [column_names] | none tags: string | [string] meta: {dictionary} ``` ##### SQL config checks/\<check\_name>.sql ```sql {{ config( severity = "error" | "warn", enabled = true | false, selection_filter_on = "column_name" | ["column_names"] | "none", tags = ["string"], meta = {"key": "value"} ) }} select ... from {{ info_schema('models') }} where ... ``` #### Examples The following examples show common ways to configure checks. ##### Warn on check failure You can use `severity: warn` when rolling out a new rule gradually. Issues are logged but the build does not fail. checks/\_public\_models\_have\_owners.yml ```yaml checks: - name: public_models_have_owners config: severity: warn # default is error ``` ##### Filter by a specific column The `edges` table has no `unique_id` column, so checks that query it won't return one. When you use `--select`, dbt looks for a `unique_id` column to scope results and finds none, so the check runs against the whole project regardless of the selector. Set `selection_filter_on` to the columns that contain resource IDs so `--select` scopes rows by those columns. For example, the `multiple_sources_joined` check aggregates by `child_unique_id`, so only that column needs to be set: checks/\_multiple\_sources\_joined.yml ```yaml checks: - name: multiple_sources_joined config: selection_filter_on: child_unique_id ``` --- ### Check properties Available in v2 You can declare check properties in `.yml` files in your `checks/` directory (as defined by the [`check-paths` config](https://docs.getdbt.com/reference/project-configs/check-paths.md)). checks/\_checks.yml ```yaml checks: - name: <string> description: <markdown_string> config: <check_config>: <config_value> - name: ... # declare properties of additional checks ``` #### Example checks/\_checks.yml ```yaml version: 2 checks: - name: all_models_have_descriptions description: "Fails if any model is missing a description." config: severity: error tags: ["governance"] meta: owner: "data-platform-team" - name: public_models_have_owners description: "Warns if any public model is missing an owner." config: severity: warn ``` --- ### check_cols (Applies to dbt v1.9 and later) snapshots/\<filename>.yml ```yml snapshots: - name: snapshot_name relation: source('my_source', 'my_table') config: schema: string unique_key: column_name_or_expression strategy: check check_cols: - column_name ``` dbt\_project.yml ```yml snapshots: <resource-path>: +strategy: check +check_cols: [column_name] | all ``` #### Description A list of columns within the results of your snapshot query to check for changes. Alternatively, use all columns using the `all` value (however this may be less performant). This parameter is **required if using the `check` [strategy](https://docs.getdbt.com/reference/resource-configs/strategy.md)**. #### Default No default is provided. #### Examples ##### Check a list of columns for changes (Applies to dbt v1.9 and later) snapshots/orders\_snapshot\_check.yml ```yaml snapshots: - name: orders_snapshot_check relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: check check_cols: - status - is_cancelled ``` To select from this snapshot in a downstream model: `select * from {{ ref('orders_snapshot_check') }}` ##### Check all columns for changes (Applies to dbt v1.9 and later) orders\_snapshot\_check.yml ```yaml snapshots: - name: orders_snapshot_check relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: check check_cols: all ``` To select from this snapshot in a downstream model: `select * from {{ ref('orders_snapshot_check') }}` --- ### check-paths Available in v2 dbt\_project.yml ```yml check-paths: [directorypath] ``` #### Definition Specify custom directories where dbt looks for [checks](https://docs.getdbt.com/docs/build/checks.md). #### Default By default, dbt looks for checks in the `checks` directory. dbt\_project.yml ```yml check-paths: ["checks"] ``` #### Examples Use a subdirectory named `project_rules` instead of `checks`: dbt\_project.yml ```yml check-paths: ["project_rules"] ``` Use multiple directories: dbt\_project.yml ```yml check-paths: ["checks", "shared_checks"] ``` --- ### Checking version compatibility For the first several years of dbt v1's development, breaking changes were more common. For this reason, we encouraged setting [dbt version requirements](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) — especially if they use features that are newer or which may break in future versions of dbt v1. By default, if you run a project with an incompatible dbt version, dbt will raise an error. You can use the `VERSION_CHECK` config to disable this check and suppress the error message: ```text $ dbt run --no-version-check Running with dbt=1.0.0 Found 13 models, 2 tests, 1 archives, 0 analyses, 204 macros, 2 operations.... ``` dbt release tracks When you select a [release track in dbt](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing dbt version upgrades, dbt will ignore the `require-dbt-version` config. dbt Labs is committed to zero breaking changes for code in dbt projects, with ongoing releases to dbt. We also recommend these best practices:  Installing dbt packages If you install dbt packages for use in your project, whether the package is maintained by your colleagues or a member of the open source dbt community, we recommend pinning the package to a specific revision or `version` boundary. dbt manages this out-of-the-box by *locking* the version/revision of packages in development in order to guarantee predictable builds in production. To learn more, refer to [Predictable package installs](https://docs.getdbt.com/reference/commands/deps.md#predictable-package-installs).  Maintaining dbt packages If you maintain dbt packages, whether on behalf of your colleagues or members of the open source community, we recommend writing defensive code that checks to verify that other required packages and global macros are available. For example, if your package depends on the availability of a `date_spine` macro in the global `dbt` namespace, you can write: models/some\_days.sql ```sql {% macro a_few_days_in_september() %} {% if not dbt.get('date_spine') %} {{ exceptions.raise_compiler_error("Expected to find the dbt.date_spine macro, but it could not be found") }} {% endif %} {{ date_spine("day", "cast('2020-01-01' as date)", "cast('2030-12-31' as date)") }} {% endmacro %} ``` --- ### clean-targets dbt\_project.yml ```yml clean-targets: [directorypath] ``` #### Definition Optionally specify a custom list of directories to be removed by the `dbt clean` [command](https://docs.getdbt.com/reference/commands/clean.md). As such, you should only include directories containing artifacts (e.g. compiled files, logs, installed packages) in this list. #### Default If this configuration is not included in your `dbt_project.yml` file, the `clean` command will remove files in your [target-path](https://docs.getdbt.com/reference/global-configs/json-artifacts.md). #### Examples ##### Remove packages and compiled files as part of `dbt clean` (preferred) To remove packages as well as compiled files, include the value of your [packages-install-path](https://docs.getdbt.com/reference/project-configs/packages-install-path.md) configuration in your `clean-targets` configuration. dbt\_project.yml ```yml clean-targets: - target - dbt_packages ``` Now, run `dbt clean`. Both the `target` and `dbt_packages` directory will be removed. Note: this is the configuration in the dbt [starter project](https://github.com/dbt-labs/dbt-starter-project/blob/HEAD/dbt_project.yml), which is generated by the `init` command. ##### Remove `logs` when running `dbt clean` dbt\_project.yml ```yml clean-targets: [target, dbt_packages, logs] ``` --- ### ClickHouse configurations #### ClickHouse configurations ##### View materialization A dbt model can be created as a [ClickHouse view](https://clickhouse.com/docs/en/sql-reference/table-functions/view/) and configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: view ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config(materialized = "view") }} ``` ##### Table materialization A dbt model can be created as a [ClickHouse table](https://clickhouse.com/docs/en/operations/system-tables/tables/) and configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: table +order_by: [ <column-name>, ... ] +engine: <engine-type> +partition_by: [ <column-name>, ... ] ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized = "table", engine = "<engine-type>", order_by = [ "<column-name>", ... ], partition_by = [ "<column-name>", ... ], ... ] ) }} ``` ###### Table configuration | Option | Description | Required? | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | | `materialized` | How the model will be materialized into ClickHouse. Must be `table` to create a table model. | Required | | `engine` | The table engine to use when creating tables. See list of supported engines below. | Optional (default: `MergeTree()`) | | `order_by` | A tuple of column names or arbitrary expressions. This allows you to create a small sparse index that helps find data faster. | Optional (default: `tuple()`) | | `partition_by` | A partition is a logical combination of records in a table by a specified criterion. The partition key can be any expression from the table columns. | Optional | For the complete list of configuration options, see the [ClickHouse documentation](https://clickhouse.com/docs/integrations/dbt). ##### Incremental materialization Table model will be reconstructed for each dbt execution. This may be infeasible and extremely costly for larger result sets or complex transformations. To address this challenge and reduce the build time, a dbt model can be created as an incremental ClickHouse table and is configured using the following syntax: ##### Project file dbt\_project.yml ```yaml models: <resource-path>: +materialized: incremental +order_by: [ <column-name>, ... ] +engine: <engine-type> +partition_by: [ <column-name>, ... ] +unique_key: [ <column-name>, ... ] +inserts_only: [ True|False ] ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized = "incremental", engine = "<engine-type>", order_by = [ "<column-name>", ... ], partition_by = [ "<column-name>", ... ], unique_key = [ "<column-name>", ... ], inserts_only = [ True|False ], ... ] ) }} ``` ###### Incremental table configuration | Option | Description | Required? | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `materialized` | How the model will be materialized into ClickHouse. Must be `table` to create a table model. | Required | | `unique_key` | A tuple of column names that uniquely identify rows. For more details on uniqueness constraints, see [here](https://docs.getdbt.com/docs/build/incremental-models.md#defining-a-unique-key-optional). | Required. If not provided altered rows will be added twice to the incremental table. | | `engine` | The table engine to use when creating tables. See list of supported engines below. | Optional (default: `MergeTree()`) | | `order_by` | A tuple of column names or arbitrary expressions. This allows you to create a small sparse index that helps find data faster. | Optional (default: `tuple()`) | | `partition_by` | A partition is a logical combination of records in a table by a specified criterion. The partition key can be any expression from the table columns. | Optional | | `inserts_only` | (Deprecated, see the `append` materialization strategy). If True, incremental updates will be inserted directly to the target incremental table without creating an intermediate table. | Optional (default: `False`) | | `incremental_strategy` | The strategy to use for incremental materialization. `delete+insert`, `append` and `insert_overwrite` (experimental) are supported. For additional details on strategies, see [here](https://github.com/ClickHouse/dbt-clickhouse#incremental-model-strategies) | Optional (default: 'default') | | `incremental_predicates` | Incremental predicate clause to be applied to `delete+insert` materializations | Optional | For the complete list of configuration options, see the [ClickHouse documentation](https://clickhouse.com/docs/integrations/dbt). #### Snapshot dbt snapshots allow a record to be made of changes to a mutable model over time. This in turn allows point-in-time queries on models, where analysts can “look back in time” at the previous state of a model. This functionality is supported by the ClickHouse connector and is configured using the following syntax: (Applies to dbt v1.9 and later) snapshots/\<model\_name>.sql ```jinja {{ config( schema = "<schema-name>", unique_key = "<column-name>", strategy = "<strategy>", updated_at = "<updated-at-column-name>", ) }} ``` For more information on configuration, check out the [snapshot configs](https://docs.getdbt.com/reference/snapshot-configs.md) reference page. #### Learn more The `dbt-clickhouse` adapter supports most dbt-native features like tests, snapshots, helper macros, and more. For a complete overview of supported features and best practices, see the [ClickHouse documentation](https://clickhouse.com/docs/integrations/dbt). --- ### Cloudera Hive configurations #### Configuring tables When materializing a model as `table`, you may include several optional configs that are specific to the dbt-hive plugin, in addition to the standard [model configs](https://docs.getdbt.com/reference/model-configs.md). | Option | Description | Required? | Example | | --------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------------------ | | partition\_by | partition by a column, typically a directory per partition is created | No | partition\_by=\['name'] | | clustered\_by | second level division of a partitioned column | No | clustered\_by=\['age'] | | file\_format | underlying storage format of the table, see <https://cwiki.apache.org/confluence/display/Hive/FileFormats> for supported formats | No | file\_format='PARQUET' | | location | storage location, typically an hdfs path | No | LOCATION='/user/etl/destination' | | comment | comment for the table | No | comment='this is the cleanest model' | | external | is this an external table - true / false | No | external=true | | tbl\_properties | any metadata can be stored as key/value pair with the table | No | tbl\_properties="('dbt\_test'='1')" | | table\_type | indicates the type of the table | No | table\_type="iceberg" | #### Incremental models Supported modes for incremental model: * **`append`** (default): Insert new records without updating or overwriting any existing data. * **`insert_overwrite`**: For new records, insert data. When used along with partition clause, update data for changed record and insert data for new records. #### Example: Using partition\_by config option hive\_partition\_by.sql ```sql {{ config( materialized='table', unique_key='id', partition_by=['city'], ) }} with source_data as ( select 1 as id, "Name 1" as name, "City 1" as city, union all select 2 as id, "Name 2" as name, "City 2" as city, union all select 3 as id, "Name 3" as name, "City 2" as city, union all select 4 as id, "Name 4" as name, "City 1" as city, ) select * from source_data ``` In the above example, a sample table is created with partition\_by and other config options. One thing to note when using partition\_by option is that the select query should always have the column name used in partition\_by option as the last one, as can be seen for the `city` column name used in the above query. If the partition\_by clause is not the same as the last column in select statement, Hive will flag an error when trying to create the model. --- ### Cloudera Impala configurations #### Configuring tables When materializing a model as `table`, you may include several optional configs that are specific to the dbt-impala plugin, in addition to the standard [model configs](https://docs.getdbt.com/reference/model-configs.md). | Option | Description | Required? | Example | | ------------------- | --------------------------------------------------------------------- | --------- | -------------------------------------------------------- | | partition\_by | partition by a column, typically a directory per partition is created | No | partition\_by=\['name'] | | sort\_by | sort by a column | No | sort\_by=\['age'] | | row\_format | format to be used when storing individual rows | No | row\_format='delimited' | | stored\_as | underlying storage format of the table | No | stored\_as='PARQUET' | | location | storage location, typically an hdfs path | No | LOCATION='/user/etl/destination' | | comment | comment for the table | No | comment='this is the cleanest model' | | serde\_properties | SerDes (\[de-]serialization) properties of table | No | serde\_properties="('quoteChar'=''', 'escapeChar'='\\')" | | tbl\_properties | any metadata can be stored as key/value pair with the table | No | tbl\_properties="('dbt\_test'='1')" | | is\_cached | true or false - if this table is cached | No | is\_cached=false (default) | | cache\_pool | cache pool name to use if is\_cached is set to true | No | | | replication\_factor | cache replication factor to use if is\_cached is set to true | No | | | external | is this an external table - true / false | No | external=true | | table\_type | indicates the type of the table - iceberg / kudu | No | table\_type="iceberg" | For Cloudera specific options for above parameters see documentation of CREATE TABLE (<https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_create_table.html>) #### Incremental models Supported modes for incremental model: * **`append`** (default): Insert new records without updating or overwriting any existing data. * **`insert_overwrite`**: For new records, insert data. When used along with partition clause, update data for changed record and insert data for new records. Unsupported modes: * **`unique_key`** This is not supported option for incremental models in dbt-impala * **`merge`**: Merge is not supported by the underlying warehouse, and hence not supported by dbt-impala #### Example: Using partition\_by config option impala\_partition\_by.sql ```sql {{ config( materialized='table', unique_key='id', partition_by=['city'], ) }} with source_data as ( select 1 as id, "Name 1" as name, "City 1" as city, union all select 2 as id, "Name 2" as name, "City 2" as city, union all select 3 as id, "Name 3" as name, "City 2" as city, union all select 4 as id, "Name 4" as name, "City 1" as city, ) select * from source_data ``` In the above example, a sample table is created with partition\_by and other config options. One thing to note when using partition\_by option is that the select query should always have the column name used in partition\_by option as the last one, as can be seen for the `city` column name used in the above query. If the partition\_by clause is not the same as the last column in select statement, Impala will flag an error when trying to create the model. --- ### column_types #### Description Optionally specify the database type of columns in a [seed](https://docs.getdbt.com/docs/build/seeds.md), by providing a dictionary where the keys are the column names, and the values are a valid datatype (this varies across databases). Without specifying this, dbt will infer the datatype based on the column values in your seed file. #### Usage Specify column types in your `dbt_project.yml` file: dbt\_project.yml ```yml seeds: jaffle_shop: country_codes: +column_types: country_code: varchar(2) country_name: varchar(32) ``` Or: seeds/properties.yml ```yml seeds: - name: country_codes config: column_types: country_code: varchar(2) country_name: varchar(32) ``` If you have previously run `dbt seed`, you'll need to run `dbt seed --full-refresh` for the changes to take effect. Note that you will need to use the fully directory path of a seed when configuring `column_types`. For example, for a seed file at `seeds/marketing/utm_mappings.csv`, you will need to configure it like so: dbt\_project.yml ```yml seeds: jaffle_shop: marketing: utm_mappings: +column_types: ... ``` #### Examples ##### Use a varchar column type to preserve leading zeros in a zipcode dbt\_project.yml ```yml seeds: jaffle_shop: # you must include the project name warehouse_locations: +column_types: zipcode: varchar(5) ``` #### Recommendation Use this configuration only when required, i.e. when the type inference is not working as expected. Otherwise you can omit this configuration. #### Troubleshooting Note: The `column_types` configuration is case-sensitive, regardless of quoting configuration. If you specify a column as `Country_Name` in your Seed, you should reference it as `Country_Name`, and not `country_name`. --- ### columns ##### Models models/\<filename>.yml ```yml models: - name: <model_name> columns: - name: <column_name> data_type: <string> description: <markdown_string> quote: true | false data_tests: ... config: tags: ... meta: ... - name: <another_column> ... ``` ##### Sources models/\<filename>.yml ```yml sources: - name: <source_name> tables: - name: <table_name> columns: - name: <column_name> description: <markdown_string> data_type: <string> quote: true | false data_tests: ... config: tags: ... meta: ... - name: <another_column> ... ``` ##### Seeds seeds/\<filename>.yml ```yml seeds: - name: <seed_name> columns: - name: <column_name> description: <markdown_string> data_type: <string> quote: true | false data_tests: ... config: tags: ... meta: ... - name: <another_column> ... ``` ##### Snapshots snapshots/\<filename>.yml ```yml snapshots: - name: <snapshot_name> columns: - name: <column_name> description: <markdown_string> data_type: <string> quote: true | false data_tests: ... config: tags: ... meta: ... - name: <another_column> ``` ##### Analyses analyses/\<filename>.yml ```yml analyses: - name: <analysis_name> columns: - name: <column_name> description: <markdown_string> data_type: <string> - name: <another_column> ``` Columns are not resources in and of themselves. Instead, they are child properties of another resource type. They can define sub-properties that are similar to properties defined at the resource level: * `tags` * `meta` * `data_tests` * `description` Because columns are not resources, their `tags` and `meta` properties are not true configurations even when nested under a `config` block. They do not inherit the `tags` or `meta` values of their parent resources. However, you can select a generic test, defined on a column, using tags applied to its column or top-level resource; see [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md#run-tests-on-tagged-columns). Columns may optionally define a `data_type`, which is necessary for: * Enforcing a model [contract](https://docs.getdbt.com/reference/resource-configs/contract.md) * Use in other packages or plugins, such as the [`external`](https://docs.getdbt.com/reference/resource-properties/external.md) property of sources and [`dbt-external-tables`](https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/) ##### `quote` The `quote` field can be used to enable or disable quoting for column names. ##### Models models/schema.yml ```yml models: - name: model_name columns: - name: column_name quote: true | false ``` ##### Sources models/schema.yml ```yml sources: - name: source_name tables: - name: table_name columns: - name: column_name quote: true | false ``` ##### Seeds seeds/schema.yml ```yml seeds: - name: seed_name columns: - name: column_name quote: true | false ``` ##### Snapshots snapshots/schema.yml ```yml snapshots: - name: snapshot_name columns: - name: column_name quote: true | false ``` ##### Analyses analysis/schema.yml ```yml analyses: - name: analysis_name columns: - name: column_name quote: true | false ``` ##### Default The default quoting value is `false` ##### Explanation This is particularly relevant to those using Snowflake, where quoting can be particularly fickle. This property is useful when: * A source table has a column that needs to be quoted to be selected, for example, to preserve column casing * A seed was created with `quote_columns: true` ([docs](https://docs.getdbt.com/reference/resource-configs/quote_columns.md)) on Snowflake * A model uses quotes in the SQL, potentially to work around the use of reserved words ```sql select user_group as "group" ``` Without setting `quote: true`: * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) applied to this column may fail due to invalid SQL * Documentation may not render correctly, e.g. `group` and `"group"` may not be matched as the same column name. ##### Example ###### Add data tests to a quoted column in a source table This is especially relevant if using Snowflake: ```yml sources: - name: stripe tables: - name: payment columns: - name: orderID quote: true data_tests: - not_null ``` Without `quote: true`, the following error will occur: ```text $ dbt test -s source:stripe.* Running with dbt=0.16.1 Found 7 models, 22 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 0 seed files, 4 sources 13:33:37 | Concurrency: 4 threads (target='learn') 13:33:37 | 13:33:37 | 1 of 1 START test source_not_null_stripe_payment_order_id............ [RUN] 13:33:39 | 1 of 1 ERROR source_not_null_stripe_payment_order_id................. [ERROR in 1.89s] 13:33:39 | 13:33:39 | Finished running 1 tests in 6.43s. Completed with 1 error and 0 warnings: Database Error in test source_not_null_stripe_payment_order_id (models/staging/stripe/src_stripe.yml) 000904 (42000): SQL compilation error: error line 3 at position 6 invalid identifier 'ORDERID' compiled SQL at target/compiled/jaffle_shop/schema_test/source_not_null_stripe_payment_orderID.sql ``` This is because dbt is trying to run: ```sql select count(*) from raw.stripe.payment where orderID is null ``` Instead of: ```sql select count(*) from raw.stripe.payment where "orderID" is null ``` --- ### Command line options For consistency, command-line interface (CLI) flags should come right after the `dbt` prefix and its subcommands. This includes "global" flags (supported for all commands). For the full list of global CLI options for dbt v1 or dbt v2 (depending on your [selected docs version](https://docs.getdbt.com/docs/dbt-versions.md)), refer to [Available flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). When set, CLI flags override [environment variables](https://docs.getdbt.com/reference/global-configs/environment-variable-configs.md) and [project flags](https://docs.getdbt.com/reference/global-configs/project-flags.md). For example, instead of using: ```bash dbt --no-populate-cache run ``` You should use: ```bash dbt run --no-populate-cache ``` Historically, passing flags (such as "global flags") *before* the subcommand is a legacy functionality that dbt Labs can remove at any time. We do not support using the same flag before and after the subcommand. #### Using boolean and non-boolean flags You can construct your commands with boolean flags to enable or disable or with non-boolean flags that use specific values, such as strings. ##### Non-boolean config Use this non-boolean config structure: * Replacing `<SUBCOMMAND>` with the command this config applies to. * `<THIS-CONFIG>` with the config you are enabling or disabling, and * `<SETTING>` with the new setting for the config. CLI flags ```text <SUBCOMMAND> --<THIS-CONFIG>=<SETTING> ``` ##### Example CLI flags ```text dbt run --printer-width=80 dbt test --indirect-selection=eager ``` ##### Boolean config To enable or disable boolean configs: * Use `<SUBCOMMAND>` this config applies to. * Followed by `--<THIS-CONFIG>` to turn it on, or `--no-<THIS-CONFIG>` to turn it off. * Replace `<THIS-CONFIG>` with the config you are enabling or disabling CLI flags ```text dbt <SUBCOMMAND> --<THIS-CONFIG> dbt <SUBCOMMAND> --no-<THIS-CONFIG> ``` ##### Example CLI flags ```text dbt run --version-check dbt run --no-version-check ``` #### Config precedence There are multiple ways of setting flags, which depend on the use case: * **[CLI options](https://docs.getdbt.com/reference/global-configs/command-line-options.md):** Define behavior specific to *this invocation*. Supported for all dbt commands. * **[Environment variables](https://docs.getdbt.com/reference/global-configs/environment-variable-configs.md):** Define different behavior in different runtime environments (development vs. production vs. [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md)), or different behavior for different users in development (based on personal preferences). * **[Project-level `flags` in `dbt_project.yml`](https://docs.getdbt.com/reference/global-configs/project-flags.md):** Define version-controlled defaults for everyone running this project. Also, opt in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) to manage your migration off legacy functionality. * **[User settings (`~/.dbt/user_settings.yml`)](https://docs.getdbt.com/reference/global-configs/user-settings.md):** Define personal preferences that apply across all projects on your machine. Written automatically by `dbt login`. The most specific setting "wins." CLI options take the highest precedence, followed by environment variables, then `dbt_project.yml`, and finally `user_settings.yml`. If you set the flag in none of those places, it will use the default value defined within dbt. Most flags can be set in all three places: ```yaml # dbt_project.yml flags: # set default for running this project -- anywhere, anytime, by anyone fail_fast: true ``` (Applies to dbt v1.11 and later) ```bash # set this environment variable to 'True' (bash syntax) export DBT_ENGINE_FAIL_FAST=1 dbt run ``` ```bash dbt run --fail-fast # set to True for this specific invocation dbt run --no-fail-fast # set to False ``` There are two categories of exceptions: 1. **Flags setting file paths:** Flags for file paths that are relevant to runtime execution (for example, `--log-path` or `--state`) cannot be set in `dbt_project.yml`. To override defaults, pass CLI options or set environment variables ((Applies to dbt v1.11 and later) `DBT_ENGINE_LOG_PATH` and `DBT_ENGINE_STATE`). Flags that tell dbt where to find project resources (for example, `model-paths`) are set in `dbt_project.yml`, but as a top-level key, outside the `flags` dictionary; these configs are expected to be fully static and never vary based on the command or execution environment. 2. **Opt-in flags:** Flags opting in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) can *only* be defined in `dbt_project.yml`. These are intended to be set in version control and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. --- ### compare_unrendered_code ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +state: compare_unrendered_code: true | false ``` ##### Properties YAML file models/\<filename>.yml ```yaml models: - name: my_model config: state: compare_unrendered_code: true | false ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( state={ "compare_unrendered_code": true | false } ) }} ``` #### Definition | Value | Behavior | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `false` (default) | dbt State compares only rendered SQL. Non-deterministic macros or env vars that change the rendered SQL on each run may trigger unnecessary rebuilds. | | `true` | dbt State avoids unnecessary rebuilds by checking both the Jinja template and rendered SQL. A rebuild only occurs when both have changed. | By default, dbt State compares only rendered SQL and rebuilds when the output differs from the stored hash. How frequently that happens depends on how often the values change. For example: * `{{ invocation_id() }}` resolves to a new value on every run, so it always triggers a rebuild. * `{{ dbt_utils.get_column_values() }}` only changes if the underlying column values change or return in a different order, so it triggers rebuilds less predictably. * `{{ modules.datetime.datetime.now().month }}` only resolves to a new value once a month, so it triggers a rebuild on the next run after the month changes. Set `compare_unrendered_code: true` to avoid these unnecessary rebuilds. When enabled, dbt State also checks the unrendered Jinja template, and only rebuilds when both the unrendered code *and* the rendered SQL have changed. This is useful for nodes with non-deterministic macros or environment variables (such as a macro that returns a UUID, or `{{ env_var('AIRFLOW_RUN_ID') }}`) that produce different rendered SQL on every run even when the template itself is unchanged. #### Interaction with `evaluate_volatile_sql` When `compare_unrendered_code` is enabled, dbt State checks whether the unrendered Jinja template has changed. If the template is unchanged, dbt State skips SQL parsing entirely — volatile SQL functions are not evaluated and [`evaluate_volatile_sql`](https://docs.getdbt.com/reference/resource-configs/evaluate-volatile-sql.md) has no effect. If the unrendered template has changed, dbt State proceeds to parse and compare the rendered SQL, and `evaluate_volatile_sql` applies. #### Examples ##### Model with a non-deterministic environment variable A model with a non-deterministic environment variable is reused as long as neither the model's own Jinja template nor the source of any macros it calls has changed, even if the environment variable (for example, AIRFLOW\_RUN\_ID) resolves to a different value at runtime: models/fct\_events.sql ```sql {{ config(state={"compare_unrendered_code": true}) }} select event_id, user_id, event_type, occurred_at, '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id from {{ ref('stg_events') }} ``` ##### Model with dbt run metadata columns A model that records which dbt run produced each row using [`invocation_id()`](https://docs.getdbt.com/reference/dbt-jinja-functions/invocation_id.md) or [`run_started_at`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_started_at.md) is reused as long as the template hasn't changed, even though those values resolve differently on every run: models/fct\_orders.sql ```sql {{ config(state={"compare_unrendered_code": true}) }} select *, '{{ invocation_id() }}' as dbt_invocation_id, '{{ run_started_at }}' as dbt_run_started_at from {{ ref('stg_orders') }} ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) --- ### compute Available in v2 Compute controls whether a unit test runs against your data platform or locally with DuckDB. Available in v2 with opt in The `compute` config is available in v2 only. It isn't available in v1 and will be ignored. To upgrade to v2, refer to [Upgrade to v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) upgrade guide. Setting `compute: local` on a unit test is in an experimental feature. Set the `DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS` environment variable to `true` before you use it: ```bash export DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS=true ``` Without it, dbt fails with an invalid configuration error that names the variable. By default, each unit test sends a query to your data platform and waits for the result. This can slow down testing and use warehouse compute. Use the `compute: local` config to run [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) locally with DuckDB for faster feedback while you develop while managing warehouse compute cost. models/filename.yml ```yml unit_tests: - name: unit_test_name model: model_name config: compute: local | remote ``` dbt\_project.yml ```yml unit_tests: resource-path: +compute: local | remote ``` #### Definition You can configure `compute` for [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md), either on an individual test in a properties YAML file or for a group of tests in your project YAML file (`dbt_project.yml`). The following values are available for `compute`: * `local`: Run the unit test with DuckDB, wherever dbt is running. Nothing is sent to your data platform, so the test returns quickly and uses no warehouse compute. * `remote` (default): Send the unit test to your data platform to run, using warehouse compute like any other query. Because this is the default, you only need to set it explicitly to opt a test out of a project-level `+compute: local`. Local execution is available for Snowflake and BigQuery. You might also see `sidecar` in error messages, which means the same thing as `local`. A config on an individual unit test overrides the project-level config. So if you set `+compute: local` in your `dbt_project.yml` and `compute: remote` in one test, that single test runs against your data platform while the rest run locally. In your project file: dbt\_project.yml ```yml unit_tests: my_project: +compute: local ``` Then on a single test: models/filename.yml ```yml unit_tests: - name: test_revenue_calculation model: fct_revenue config: compute: remote ``` ###### Things to know about local execution * `compute: local` is refused unless `DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS=true` is set in the environment where dbt runs. This applies whether you set it on an individual test or as `+compute: local` in `dbt_project.yml`. * Your SQL has to be translatable to DuckDB. Platform-specific functions with no DuckDB equivalent fail, and `local` doesn't fall back to your data platform — a translation failure is a test failure. * To translate your SQL, dbt fetches the schemas of your model's direct upstream models from your data platform the first time you run the test, then caches them for later runs. Those upstream models must already exist in your data platform. * Setting `compute: local` also promotes [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) to `strict` for that test, because local execution needs strict analysis to translate your SQL. If you set `static_analysis: off` on the test, it can't run locally and produces an error. #### Related docs * [Run unit tests locally](https://docs.getdbt.com/docs/build/unit-tests.md#run-unit-tests-locally) * [Unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) * [About unit tests property](https://docs.getdbt.com/reference/resource-properties/unit-tests.md) * [static\_analysis](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) --- ### concurrent_batches Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). ##### Project YAML file dbt\_project.yml ```yaml models: +concurrent_batches: true ``` ##### SQL file config models/my\_model.sql ```sql {{ config( materialized='incremental', concurrent_batches=true, incremental_strategy='microbatch' ... ) }} select ... ``` #### Definition `concurrent_batches` is an override which allows you to decide whether or not you want to run batches in parallel or sequentially (one at a time). For more information, refer to [how batch execution works](https://docs.getdbt.com/docs/build/parallel-batch-execution.md#how-parallel-batch-execution-works). #### Example By default, dbt auto-detects whether batches can run in parallel for microbatch models. However, you can override dbt's detection by setting the `concurrent_batches` config to `false` in your `dbt_project.yml` or model `.sql` file to specify parallel or sequential execution, given you meet these conditions: * You've configured a [microbatch incremental strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md). * You're working with cumulative metrics or any logic that depends on batch order. Set `concurrent_batches` config to `false` to ensure batches are processed sequentially. For example: dbt\_project.yml ```yaml models: my_project: cumulative_metrics_model: +concurrent_batches: false ``` models/my\_model.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch' concurrent_batches=false ) }} select ... ``` --- ### config-version The `config-version:` tag is optional. dbt\_project.yml ```yml config-version: 2 ``` #### Definition Specify your `dbt_project.yml` as using the v2 structure. #### Default Without this configuration, dbt will assume your `dbt_project.yml` uses the version 2 syntax. Version 1 has been deprecated. --- ### Configurations and properties, what are they? Understand the difference between properties and configurations in dbt: properties describe resources, while configurations control how dbt builds them in the warehouse. Resources in your project—models, snapshots, seeds, tests, and the rest—can have a number of declared *properties*. Resources can also define *configurations* (configs), which are a special kind of property that bring extra abilities. What's the distinction? * Properties are declared for resources one-by-one in `properties.yml` files. Configs can be defined there, nested under a `config` property. They can also be set one-by-one via a `config()` macro (right within `.sql` files), and for many resources at once in `dbt_project.yml`. * Because configs can be set in multiple places, they are also applied hierarchically. An individual resource might *inherit* or *override* configs set elsewhere. * You can select resources based on their config values using the `config:` selection method, but not the values of non-config properties. * There are slightly different naming conventions for properties and configs depending on the file type. Refer to [naming convention](https://docs.getdbt.com/reference/dbt_project.yml.md#naming-convention) for more details. A rule of thumb: properties declare things *about* your project resources; configs go the extra step of telling dbt *how* to build those resources in your warehouse. This is generally true, but not always, so it's always good to check! For example, you can use resource **properties** to: * Describe models, snapshots, seed files, and their columns * Assert "truths" about a model, in the form of [data tests](https://docs.getdbt.com/docs/build/data-tests.md), e.g. "this `id` column is unique" * Define official downstream uses of your data models, in the form of [exposures](https://docs.getdbt.com/docs/build/exposures.md), and assert an exposure's "type" Whereas you can use **configurations** to: * Change how a model will be materialized (table, view, incremental, etc) * Declare where a seed will be created in the database (`<database>.<schema>.<alias>`) * Declare whether a resource should persist its descriptions as comments in the database * Apply tags and meta to a resource --- ### Configure state selection State and [defer](https://docs.getdbt.com/reference/node-selection/defer.md) can be set by environment variables as well as CLI flags: * `--state` or (Applies to dbt v1.11 and later) `DBT_ENGINE_STATE`: file path * `--defer` or (Applies to dbt v1.11 and later) `DBT_ENGINE_DEFER`: boolean * `--defer-state` or (Applies to dbt v1.11 and later) `DBT_ENGINE_DEFER_STATE`: file path to use for deferral only (optional) If `--defer-state` is not specified, deferral will use the artifacts supplied by `--state`. This enables more granular control in cases where you want to compare against logical state from one environment or past point in time, and defer to applied state from a different environment or point in time. If both the flag and env var are provided, the flag takes precedence. ###### Notes * The `--state` artifacts must be of schema versions that are compatible with the currently running dbt version. * These are powerful, complex features. Read about [known caveats and limitations](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md) to state comparison. Syntax deprecated In [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#behavior-changes>), we deprecated the original syntax for state (`DBT_ARTIFACT_STATE_PATH`) and defer (`DBT_DEFER_TO_STATE`). Although dbt supports backward compatibility with the old syntax, we will remove it in a future release that we have not yet determined. ##### The "result" status Another element of job state is the `result` of a prior dbt invocation. After executing a `dbt run`, for example, dbt creates the `run_results.json` artifact which contains execution times and success / error status for dbt models. You can read more about `run_results.json` on the ['run results'](https://docs.getdbt.com/reference/artifacts/run-results-json.md) page. The following dbt commands produce `run_results.json` artifacts whose results can be referenced in subsequent dbt invocations: * `dbt run` * `dbt test` * `dbt build` * `dbt seed` After issuing one of the above commands, you can reference the results by adding a selector to a subsequent command as follows: (Applies to dbt v1.11 and later) ```bash # You can also set the DBT_ENGINE_STATE environment variable instead of the --state flag. dbt run --select "result:<status>" --defer --state path/to/prod/artifacts ``` The available options depend on the resource (node) type: | `result:\<status>` | model | seed | snapshot | test | | ------------------ | ----- | ---- | -------- | ---- | | `result:error` | ✅ | ✅ | ✅ | ✅ | | `result:success` | ✅ | ✅ | ✅ | | | `result:skipped` | ✅ | | ✅ | ✅ | | `result:fail` | | | | ✅ | | `result:warn` | | | | ✅ | | `result:pass` | | | | ✅ | ##### Combining `state` and `result` selectors The state and result selectors can also be combined in a single invocation of dbt to capture errors from a previous run OR any new or modified models. ```bash dbt run --select "result:<status>+" state:modified+ --defer --state ./<dbt-artifact-path> ``` ##### The "source\_status" status Another element of job state is the `source_status` of a prior dbt invocation. After executing `dbt source freshness`, for example, dbt creates the `sources.json` artifact which contains execution times and `max_loaded_at` dates for dbt sources. You can read more about `sources.json` on the ['sources'](https://docs.getdbt.com/reference/artifacts/sources-json.md) page. The `dbt source freshness` command produces a `sources.json` artifact whose results can be referenced in subsequent dbt invocations. When a job is selected, dbt will surface the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of fresh sources. In your job commands, you can signal dbt to run and test only on the fresher sources and their children by including the `source_status:fresher+` argument. This requires both the previous and current states to have the `sources.json` artifact available. Or plainly said, both job states need to run `dbt source freshness`. After issuing the `dbt source freshness` command, you can reference the source freshness results by adding a selector to a subsequent command: (Applies to dbt v1.11 and later) ```bash # You can also set the DBT_ENGINE_STATE environment variable instead of the --state flag. dbt source freshness # must be run again to compare current to previous state dbt build --select "source_status:fresher+" --state path/to/prod/artifacts ``` For more example commands, refer to [Pro-tips for workflows](https://docs.getdbt.com/best-practices/best-practice-workflows.md#pro-tips-for-workflows). #### Related docs * [About state in dbt](https://docs.getdbt.com/reference/node-selection/state-selection.md) * [State comparison caveats](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md) --- ### Configuring quoting in projects dbt\_project.yml ```yml quoting: database: true | false schema: true | false identifier: true | false snowflake_ignore_case: true | false # v2-only config. Aligns with Snowflake's session parameter QUOTED_IDENTIFIERS_IGNORE_CASE behavior. # Ignored by dbt v1 and other adapters. ``` #### Definition You can optionally enable quoting in a dbt project to control whether dbt wraps database, schema, or identifier names in quotes when generating SQL. dbt uses this configuration when: * Creating relations (For example, tables or views) * Resolving a `ref()` function to a direct relation reference BigQuery terminology Note that for BigQuery quoting configuration, `database` and `schema` should be used here, though these configs will apply to `project` and `dataset` names respectively #### Default The default values vary by database. ##### Default For most adapters, quoting is set to `true` by default. Why? It's equally easy to select from relations with quoted or unquoted identifiers. Quoting allows you to use reserved words and special characters in those identifiers, though we recommend avoiding reserved words and special characters in identifiers whenever possible. dbt\_project.yml ```yml quoting: database: true schema: true identifier: true ``` ##### Snowflake For Snowflake, quoting is set to `false` by default. Creating relations with quoted identifiers also makes those identifiers case sensitive. It's much more difficult to select from them. You can re-enable quoting for relations identifiers that are case sensitive, reserved words, or contain special characters, but we recommend you avoid this as much as possible. dbt\_project.yml ```yml quoting: database: false schema: false identifier: false snowflake_ignore_case: false # v2-only config. Aligns with Snowflake's session parameter QUOTED_IDENTIFIERS_IGNORE_CASE behavior. # Ignored by dbt v1 and other adapters. ``` #### Examples Set quoting to `false` for a project: dbt\_project.yml ```yml quoting: database: false schema: false identifier: false snowflake_ignore_case: false # v2-only config. Aligns with Snowflake's session parameter QUOTED_IDENTIFIERS_IGNORE_CASE behavior. # Ignored by dbt v1 and other adapters. ``` dbt will then create relations without quotes: ```sql create table analytics.dbt_alice.dim_customers ``` #### Recommendations ##### Snowflake If you're using Snowflake, we recommend: * Setting all quoting configs to `False` in your [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md) to avoid quoting model and column names unnecessarily and to help prevent case sensitivity issues. * Setting all quoting configs to `False` also means you cannot use reserved words as identifiers, such as model or table names. We recommend you avoid using these reserved words anyway. * If you're using dbt v2 and your Snowflake environment sets the session parameter `QUOTED_IDENTIFIERS_IGNORE_CASE = true` (for example, in an orchestrator or pre-hook), you should also enable quoting and `snowflake_ignore_case` in your `dbt_project.yml` to preserve the exact case of database, schema, and identifier: ```yml quoting: database: true schema: true identifier: true snowflake_ignore_case: true # v2-only config. Aligns with Snowflake's session parameter QUOTED_IDENTIFIERS_IGNORE_CASE behavior. # Ignored by dbt v1 and other adapters. ``` Setting `snowflake_ignore_case: true` ensures that dbt compiles column and identifier names match Snowflake’s behavior at runtime, preserving parity between compile-time and runtime logic. Without this, you may encounter "column not found" errors. Quoting a source If a Snowflake source table uses a quoted database, schema, or table identifier, you can configure this in the source.yml file. Refer to [configuring quoting](https://docs.getdbt.com/reference/resource-properties/quoting.md) for more information. ###### Explanation dbt skips quoting on Snowflake so lowercase model names work seamlessly in downstream queries and BI tools without worrying about case or quotes. Unlike most databases (which lowercase unquoted identifiers), Snowflake uppercases them. When you quote identifiers, Snowflake will preserve their case and make them case-sensitive. This means when you create a table with quoted, lowercase identifiers, the table should always be referenced with quotes and use the exact same case, which can easily break downstream queries in BI tools or ad-hoc SQL. Because dbt conventions use lowercase model and file names, quoting them in Snowflake risks breaking downstream queries in BI tools or ad-hoc SQL. If dbt instead used uppercase names by convention, the safe defaults for other databases would be at risk of breaking downstream queries. snowflake\_casing.sql ```sql /* Run these queries to understand how Snowflake handles casing and quoting. */ -- This is the output of an example `orders.sql` model with quoting enabled create table "analytics"."orders" as ( select 1 as id ); /* These queries WILL NOT work! Since the table above was created with quotes, Snowflake created the orders table with a lowercase schema and identifier. Since unquoted identifiers are automatically uppercased, both of the following queries are equivalent, and neither will work correctly. */ select * from analytics.orders; select * from ANALYTICS.ORDERS; /* To query this table, you'll need to quote the schema and table. This query should indeed complete without error. */ select * from "analytics"."orders"; /* To avoid this quoting madness, you can disable quoting for schemas and identifiers in your dbt_project.yml file. This means that you won't be able to use reserved words as model names, but you should avoid that anyway! Assuming schema and identifier quoting is disabled, the following query would indeed work: */ select * from analytics.orders; ``` ##### Redshift Leave the default quoting values. You can do this in `dbt_project.yml` by either omitting the `quoting` config or setting `database`, `schema`, and `identifier` to `true`. The `quoting` config only controls whether dbt wraps names in double quotes, so it can't work around Redshift lowercasing mixed-case names. If you need mixed-case names, turn on the Redshift `enable_case_sensitive_identifier` setting and keep `quoting` on. If you set `quoting` to `false`, dbt sends unquoted names, and Redshift lowercases them even when `enable_case_sensitive_identifier` is `true`. By default, Redshift lowercases database, schema, table, and column names, even when they're quoted. The `enable_case_sensitive_identifier` setting controls this: * `false` (default): Redshift lowercases all names, whether or not they're quoted. * `true`: Redshift keeps the case of quoted names. Unquoted names are still lowercased. You can set this parameter to `true` for a cluster or workgroup. A `SET` command applies only to the current session. Refer to [Turn on case sensitivity](https://docs.aws.amazon.com/redshift/latest/mgmt/zero-etl-setting-up.case-sensitivity.html) and the [parameter reference](https://docs.aws.amazon.com/redshift/latest/dg/r_enable_case_sensitive_identifier.html) in the Amazon Redshift documentation. The following example shows the difference between setting the parameter to `false` or `true` when run in a Redshift session: redshift\_casing.sql ```sql -- Default: quoted names are still lowercased SET enable_case_sensitive_identifier TO false; CREATE TABLE "MixedCasedTable" ("MixedCasedColumn" int); INSERT INTO "MixedCasedTable" VALUES (1); -- These work. Redshift stored the names in lowercase. SELECT mixedcasedcolumn FROM mixedcasedtable; SELECT "MixedCasedColumn" FROM "MixedCasedTable"; DROP TABLE mixedcasedtable; -- With the setting on, quoted names keep their case. SET enable_case_sensitive_identifier TO true; CREATE TABLE "MixedCasedTable" ("MixedCasedColumn" int); INSERT INTO "MixedCasedTable" ("MixedCasedColumn") VALUES (1); -- This works. You must quote the names and match the case. SELECT "MixedCasedColumn" FROM "MixedCasedTable"; -- This fails: relation "mixedcasedtable" does not exist SELECT mixedcasedcolumn FROM mixedcasedtable; DROP TABLE "MixedCasedTable"; ``` ##### Other warehouses Leave the default values for your warehouse. --- ### Configuring quoting in sources models/\<filename>.yml ```yml sources: - name: jaffle_shop quoting: database: true | false schema: true | false identifier: true | false tables: - name: orders quoting: database: true | false schema: true | false identifier: true | false ``` #### Definition Optionally configure whether dbt should quote databases, schemas, and identifiers when resolving a `{{ source() }}` function to a direct relation reference. This config can be specified for all tables in a source, or for a specific source table. Quoting configs defined for a specific source table override the quoting configs specified for the top-level source. BigQuery Terminology Note that for BigQuery quoting configuration, `database` and `schema` should be used here, though these configs will apply to `project` and `dataset` names respectively #### Default The default values vary by database. For most adapters, quoting is set to *true* by default. Why? It's equally easy to select from relations with quoted or unquoted identifiers. Quoting allows you to use reserved words and special characters in those identifiers, though we recommend avoiding this whenever possible. On Snowflake, quoting is set to *false* by default. Creating relations with quoted identifiers also makes those identifiers case sensitive. It's much more difficult to select from them. You can re-enable quoting for relations identifiers that are case sensitive, reserved words, or contain special characters, but we recommend you avoid this as much as possible. #### Example models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw quoting: database: true schema: true identifier: true tables: - name: orders - name: customers # This overrides the `jaffle_shop` quoting config quoting: identifier: false ``` In a downstream model: models/\<filename>.yml ```sql select ... -- this should be quoted from {{ source('jaffle_shop', 'orders') }} -- here, the identifier should be unquoted left join {{ source('jaffle_shop', 'customers') }} using (order_id) ``` This will get compiled to: ```sql select ... -- this should be quoted from "raw"."jaffle_shop"."orders" -- here, the identifier should be unquoted left join "raw"."jaffle_shop".customers using (order_id) ``` --- ### Confluent Cloud configurations The `dbt-confluent` adapter supports the following materializations and configurations for building models on [Confluent Cloud for Apache Flink](https://docs.confluent.io/cloud/current/flink/overview.html). For more detailed documentation of how to get started, visit the [dbt documentation on Confluent Cloud](https://docs.confluent.io/cloud/current/flink/operate-and-deploy/deploy-flink-dbt.html). #### Materializations | Materialization | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `view` | Drop-and-recreate Flink SQL view. | | `streaming_table` | Creates a Flink SQL table and a separate long-running `INSERT INTO` statement that continuously writes query results to the table. If the table already exists, the adapter checks for schema drift and skips creation; use `--full-refresh` to drop and recreate. | | `streaming_source` | Creates a Flink SQL table backed by a connector (for example, `faker` for mock data generation). Requires the `connector` config. The model SQL defines the column definitions. If the table already exists, the adapter checks for schema drift and skips creation; use `--full-refresh` to drop and recreate. | | `ephemeral` | Standard dbt CTE-based query fragment, not materialized in Flink. | ##### Unsupported materializations * **`table`**: Not officially supported. Coming soon. * **`materialized_view`**: Not supported. Use `streaming_table` instead. * **`incremental`**: Not supported. dbt's batch-incremental semantics do not map to Flink's continuous processing model. Use `streaming_table` instead. * **`snapshot`**: Not supported. Flink SQL lacks the batch operations (`MERGE`, `UPDATE`) required by dbt snapshots. #### Materialization-specific configurations ##### `streaming_table` The `streaming_table` materialization creates a Kafka-topic-backed table and a continuously running `INSERT INTO` statement. models/my\_streaming\_model.sql ```sql {{ config( materialized='streaming_table', with={ 'changelog.mode': 'upsert', 'kafka.retention.time': '7 d' } ) }} SELECT order_id, customer_id, total_amount FROM {{ ref('raw_orders') }} WHERE status = 'completed' ``` ###### `with` config | Config | Type | Description | | ------ | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `with` | `dict` | A dictionary of Confluent Cloud Flink SQL table options passed to the `WITH` clause of the `CREATE TABLE` statement. Common options include `changelog.mode` (`append`, `upsert`, `retract`), `kafka.retention.time`, `key.format`, `value.format`, `scan.startup.mode`, and others. See the [CREATE TABLE WITH options](https://docs.confluent.io/cloud/current/flink/reference/statements/create-table.html#with-options) reference for the full list. | ##### `streaming_source` The `streaming_source` materialization creates a table backed by a connector. The `connector` config is **required**. The model SQL defines the column definitions (rather than a `SELECT` query). In Confluent Cloud, valid connector values include `faker` (mock data generation) and external table connectors for AI search. See the [Confluent connector catalog](https://docs.confluent.io/cloud/current/connectors/index.html) and [Flink CREATE TABLE documentation](https://docs.confluent.io/cloud/current/flink/reference/statements/create-table.html) for available connectors and options. models/my\_fake\_orders.sql ```sql {{ config( materialized='streaming_source', connector='faker', with={ 'rows-per-second': '1', 'number-of-rows': '100', 'changelog.mode': 'append', } ) }} order_id BIGINT, price DECIMAL(10, 2), order_time TIMESTAMP(3), WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND, PRIMARY KEY(order_id) NOT ENFORCED ``` ###### `connector` config | Config | Type | Required | Description | | ----------- | -------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `connector` | `string` | Yes | The connector type for the source table. Valid values in Confluent Cloud include `faker` (mock data) and external AI search connectors. | | `with` | `dict` | No | Additional table options passed to the `WITH` clause, alongside the connector. Valid options depend on the connector type. | #### Stateful behavior and `--full-refresh` Confluent Cloud Flink SQL tables are stateful, long-running resources. The `streaming_table` and `streaming_source` materializations behave differently from traditional batch-oriented dbt materializations: * **First run**: The table is created and (for `streaming_table`) a continuously running `INSERT INTO` statement begins populating it. * **Subsequent runs without `--full-refresh`**: If the table already exists, the adapter compares the existing column names, data types, and `WITH` options against the model. If nothing has drifted, the run skips the model to avoid dropping a table that has accumulated state or has downstream consumers. If drift is detected, the run fails with a compilation error. Drift detection can be disabled per model with `config(on_schema_drift='ignore')`. * **Runs with `--full-refresh`**: The existing table is dropped and recreated from scratch, reprocessing all data. Use `--full-refresh` when you need to change a table's schema, modify `WITH` options, or reprocess data from the beginning: ```bash dbt run --full-refresh --select my_streaming_model ``` #### Known limitations * **No schema management**: The adapter cannot create or drop schemas (Kafka clusters) or databases (environments). These must be managed in Confluent Cloud. * **No table renames**: `ALTER TABLE RENAME` is not supported in Flink SQL. * **Non-transactional**: Confluent Cloud Flink SQL does not support transactions. `BEGIN` and `COMMIT` are no-ops. * **Seeds require `full_refresh`**: The adapter sets `full_refresh: true` for seeds by default. --- ### constraints Constraints are a feature of many data platforms. When specified, the platform will perform additional validation on data as it is being populated in a new table or inserted into a preexisting table. If the validation fails, the table creation or update fails, the operation is rolled back, and you will see a clear error message. When enforced, a constraint guarantees that you will never see invalid data in the table materialized by your model. Enforcement varies significantly by data platform. #### Prerequisites Before using constraints, ensure the following requirements are met: * **You use supported materializations** — Constraints only work on `table` and `incremental` models. Constraints are never applied on `ephemeral` models or those materialized as `view`. * **You enforce a contract** — To use constraints, your model must declare and enforce a [contract](https://docs.getdbt.com/reference/resource-configs/contract.md). This means you need to explicitly define the `data_type` for every column in your model's schema configuration. ##### Platform constraint support Select the adapter-specific tab for more information on [constraint](https://docs.getdbt.com/reference/resource-properties/constraints.md) support across platforms. Constraints fall into three categories based on definability and platform enforcement: * **Definable and enforced** — The model won't build if it violates the constraint. * **Definable and not enforced** — The platform supports specifying the type of constraint, but a model can still build even if building the model violates the constraint. This constraint exists for metadata purposes only. This approach is more typical in cloud data warehouses than in transactional databases, where strict rule enforcement is more common. * **Not definable and not enforced** — You can't specify the type of constraint for the platform. ##### Redshift | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### Snowflake | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### BigQuery | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ❌ | ❌ | | check | ❌ | ❌ | ##### Postgres | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ✅ | | foreign\_key | ✅ | ✅ | | unique | ✅ | ✅ | | check | ✅ | ✅ | ##### Spark Currently, `not_null` and `check` constraints are enforced only after a model is built. Because of this platform limitation, dbt considers these constraints definable but not enforced, which means they're not part of the *model contract* since they can't be enforced at build time. This table will change as the features evolve. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ❌ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ✅ | ❌ | ##### Databricks Currently, `not_null` and `check` constraints are enforced only after a model is built. Because of this platform limitation, dbt considers these constraints definable but not enforced, which means they're not part of the *model contract* since they can't be enforced at build time. This table will change as the features evolve. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ❌ | ❌ | | check | ✅ | ✅ | ##### Fabric Microsoft Fabric creates the table first, then adds some constraints in separate steps. Only `not_null` is checked when the model builds. Primary key, foreign key, and unique constraints are recorded on the table for reference, but Fabric does not check them when new data is added. Define `primary_key`, `foreign_key`, and `unique` at the model level. The adapter ignores those constraint types at the column level. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### Athena | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ❌ | ❌ | | primary\_key | ❌ | ❌ | | foreign\_key | ❌ | ❌ | | unique | ❌ | ❌ | | check | ❌ | ❌ | #### Defining constraints Constraints may be defined for a single column, or at the model level for one or more columns. As a general rule, we recommend defining single-column constraints directly on those columns. If you define multiple `primary_key` constraints for a single model, those *must* be defined at the model level. Defining multiple `primary_key` constraints at the column level is not supported. The structure of a constraint is: * `type` (required): one of `not_null`, `unique`, `primary_key`, `foreign_key`, `check`, `custom` * `expression`: Free text input to qualify the constraint. Required for certain constraint types, and optional for others. * `name` (optional): Human-friendly name for this constraint. Supported by some data platforms. * `columns` (model-level only): List of column names to apply the constraint over. (Applies to dbt v1.9 and later) Foreign key constraints accept two additional inputs: * `to`: A relation input, likely [`ref()`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md)] and [`source()`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md), indicating the referenced table. * `to_columns`: A list of column(s) in that table containing the corresponding primary or unique key. This syntax for defining foreign keys uses `ref`, meaning it will capture dependencies and works across different environments. It's available in [dbt **v1 Latest**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and [dbt v1.9+](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md). Since constraints support and enforcement [varies by platform](https://docs.getdbt.com/reference/resource-properties/constraints.md#platform-specific-support), dbt offers two optional fields you can specify on any filter: * `warn_unenforced`: Set to `False` to skip warnings for constraints that are supported by your platform but not enforced (like `primary_key` in Snowflake). * `warn_unsupported`: Set to `False` to skip warnings for constraints that your platform doesn't support at all (like `check` in Redshift). models/schema.yml ```yml models: - name: <model_name> # required config: materialized: table contract: {enforced: true} # model-level constraints constraints: - type: primary_key columns: [first_column, second_column, ...] warn_unsupported: True # show a warning if unsupported - type: foreign_key # multi_column columns: [first_column, second_column, ...] to: ref('my_model_to') | source('source', 'source_table') to_columns: [other_model_first_column, other_model_second_columns, ...] - type: check columns: [first_column, second_column, ...] expression: "first_column != second_column" name: human_friendly_name - type: ... columns: - name: first_column data_type: string # column-level constraints constraints: - type: not_null - type: unique - type: foreign_key to: ref('my_model_to') | source('source', 'source_table') to_columns: [other_model_column] warn_unenforced: False # skips warning if supported but not enforced - type: ... ``` Supported dbt-adapters use these fields when populated, to render out the foreign key constraint instead of `expression`. For more information on the adapters which support foreign key constraints, have a look at our guide on [Platform constraint support](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md#platform-constraint-support). #### Platform-specific support In transactional databases, it is possible to define "constraints" on the allowed values of certain columns, stricter than just the data type of those values. For example, Postgres supports and enforces all the constraints in the ANSI SQL standard (`not null`, `unique`, `primary key`, `foreign key`), plus a flexible row-level `check` constraint that evaluates to a boolean expression. Most analytical data platforms support and enforce a `not null` constraint, but they either do not support or do not enforce the rest. It is sometimes still desirable to add an "informational" constraint, knowing it is *not* enforced, for the purpose of integrating with legacy data catalog or entity-relation diagram tools ([dbt-labs/dbt#3295](https://github.com/dbt-labs/dbt/issues/3295)). Some data platforms can optionally use primary or foreign key constraints for query optimization if you specify an additional keyword. To that end, there are two optional fields you can specify on any filter: * `warn_unenforced: False` to skip warning on constraints that are supported, but not enforced, by this data platform. The constraint will be included in templated DDL. * `warn_unsupported: False` to skip warning on constraints that aren't supported by this data platform, and therefore won't be included in templated DDL. ##### Postgres * PostgreSQL constraints documentation: [here](https://www.postgresql.org/docs/current/ddl-constraints.html#id-1.5.4.6.6) models/constraints\_example.sql ```sql {{ config( materialized = "table" ) }} select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ``` models/schema.yml ```yml models: - name: constraints_example config: contract: enforced: true columns: - name: id data_type: int constraints: - type: not_null - type: primary_key - type: check expression: "id > 0" - name: customer_name data_type: text - name: first_transaction_date data_type: date ``` Expected DDL to enforce constraints: target/run/.../constraints\_example.sql ```sql create table "database_name"."schema_name"."constraints_example__dbt_tmp" ( id integer not null primary key check (id > 0), customer_name text, first_transaction_date date ) ; insert into "database_name"."schema_name"."constraints_example__dbt_tmp" ( id, customer_name, first_transaction_date ) ( select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ); ``` ##### Redshift Redshift currently only enforces `not null` constraints; all other constraints are metadata only. Additionally, Redshift does not allow column checks at the time of table creation. See more in the Redshift documentation [here](https://docs.aws.amazon.com/redshift/latest/dg/t_Defining_constraints.html). models/constraints\_example.sql ```sql {{ config( materialized = "table" ) }} select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ``` models/schema.yml ```yml models: - name: constraints_example config: contract: enforced: true columns: - name: id data_type: integer constraints: - type: not_null - type: primary_key # not enforced -- will warn & include - type: check # not supported -- will warn & skip expression: "id > 0" data_tests: - unique # primary_key constraint is not enforced - name: customer_name data_type: varchar - name: first_transaction_date data_type: date ``` Note that Redshift limits the maximum length of the `varchar` values to 256 characters by default (or when specified without a length). This means that any string data exceeding 256 characters might get truncated *or* return a "value too long for character type" error. To allow the maximum length, use `varchar(max)`. For example, `data_type: varchar(max)`. Expected DDL to enforce constraints: target/run/.../constraints\_example.sql ```sql create table "database_name"."schema_name"."constraints_example__dbt_tmp" ( id integer not null, customer_name varchar, first_transaction_date date, primary key(id) ) ; insert into "database_name"."schema_name"."constraints_example__dbt_tmp" ( select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ); ``` ##### Snowflake * Snowflake constraints documentation: [here](https://docs.snowflake.com/en/sql-reference/constraints-overview.html) * Snowflake data types: [here](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types.html) Snowflake supports four types of constraints: `unique`, `not null`, `primary key`, and `foreign key`. It is important to note that only the `not null` (and the `not null` property of `primary key`) are actually checked at present. The rest of the constraints are purely metadata, not verified when inserting data. Although Snowflake does not validate `unique`, `primary`, or `foreign_key` constraints, you may optionally instruct Snowflake to use them for query optimization by specifying [`rely`](https://docs.snowflake.com/en/user-guide/join-elimination) in the constraint `expression` field. Currently, Snowflake doesn't support the `check` syntax and dbt will skip the `check` config and raise a warning message if it is set on some models in the dbt project. models/constraints\_example.sql ```sql {{ config( materialized = "table" ) }} select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ``` models/schema.yml ```yml models: - name: constraints_example config: contract: enforced: true columns: - name: id data_type: integer description: hello constraints: - type: not_null - type: primary_key # not enforced -- will warn & include - type: check # not supported -- will warn & skip expression: "id > 0" data_tests: - unique # need this test because primary_key constraint is not enforced - name: customer_name data_type: text - name: first_transaction_date data_type: date ``` Expected DDL to enforce constraints: target/run/.../constraints\_example.sql ```sql create or replace transient table <database>.<schema>.constraints_example ( id integer not null primary key, customer_name text, first_transaction_date date ) as ( select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ); ``` ##### BigQuery BigQuery allows defining and enforcing `not null` constraints, and defining (but *not* enforcing) `primary key` and `foreign key` constraints (which can be used for query optimization). BigQuery does not support defining or enforcing other constraints. For more information, refer to [Platform constraint support](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md#platform-constraint-support) Documentation: <https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language> Data types: <https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types> models/constraints\_example.sql ```sql {{ config( materialized = "table" ) }} select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ``` models/schema.yml ```yml models: - name: constraints_example config: contract: enforced: true columns: - name: id data_type: int constraints: - type: not_null - type: primary_key # not enforced -- will warn & include - type: check # not supported -- will warn & skip expression: "id > 0" data_tests: - unique # primary_key constraint is not enforced - name: customer_name data_type: string - name: first_transaction_date data_type: date ``` ##### Expected DDL to enforce constraints: target/run/.../constraints\_example.sql ```sql create or replace table `<project>`.`<dataset>`.`constraints_example` ( id integer not null, customer_name string, first_transaction_date date ) as ( select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ); ``` ##### Column-level constraint on nested column: models/nested\_column\_constraints\_example.sql ```sql {{ config( materialized = "table" ) }} select 'string' as a, struct( 1 as id, 'name' as name, struct(2 as id, struct('test' as again, 2 as even_more) as another) as double_nested ) as b ``` models/nested\_fields.yml ```yml models: - name: nested_column_constraints_example config: contract: enforced: true columns: - name: a data_type: string - name: b.id data_type: integer constraints: - type: not_null - name: b.name description: test description data_type: string - name: b.double_nested.id data_type: integer - name: b.double_nested.another.again data_type: string - name: b.double_nested.another.even_more data_type: integer constraints: - type: not_null ``` ##### Databricks Databricks allows you to define: * a `not null` constraint * and/or additional `check` constraints, with conditional expressions including one or more columns As Databricks does not support transactions nor allows using `create or replace table` with a column schema, the table is first created without a schema, and `alter` statements are then executed to add the different constraints. This means that: * The names and order of columns is checked but not their type * If the `constraints` and/or `constraint_check` fails, the table with the failing data will still exist in the Warehouse See [this page](https://docs.databricks.com/tables/constraints.html) with more details about the support of constraints on Databricks. models/constraints\_example.sql ```sql {{ config( materialized = "table" ) }} select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ``` models/schema.yml ```yml models: - name: constraints_example config: contract: enforced: true columns: - name: id data_type: int constraints: - type: not_null - type: primary_key # not enforced -- will warn & include - type: check # not supported -- will warn & skip expression: "id > 0" data_tests: - unique # primary_key constraint is not enforced - name: customer_name data_type: text - name: first_transaction_date data_type: date ``` Expected DDL to enforce constraints: target/run/.../constraints\_example.sql ```sql create or replace table schema_name.constraints_example using delta as select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ``` Followed by the statements ```sql alter table schema_name.constraints_example change column id set not null; alter table schema_name.constraints_example add constraint 472394792387497234 check (id > 0); ``` ##### Fabric Microsoft Fabric allows you to define: * A `not_null` constraint, which is checked when the model builds * The `primary_key`, `foreign_key`, and `unique` constraints, which are recorded on the table for reference but not checked when new data is added Be aware of the following limitations and requirements: * `check` constraints are not supported on Microsoft Fabric. Use [data tests](https://docs.getdbt.com/docs/build/data-tests.md) such as `dbt_utils.expression_is_true` to validate custom rules instead. * Microsoft Fabric does not include all constraints in the initial `create table` statement. dbt creates the table first, then runs separate statements to add model-level constraints. * You must provide a `name` for each model-level constraint. * Columns in [snapshot](https://docs.getdbt.com/docs/build/snapshots.md) source tables cannot have constraints. If a source column has a `not_null` constraint or similar rule, the snapshot may fail. For more on Microsoft Fabric setup, refer to [Microsoft Fabric Data Warehouse configurations](https://docs.getdbt.com/reference/resource-configs/fabric-configs.md). models/constraints\_example.sql ```sql {{ config( materialized = "table" ) }} select 1 as id, 'My Favorite Customer' as customer_name, cast('2019-01-01' as date) as first_transaction_date ``` models/schema.yml ```yml models: - name: constraints_example config: contract: enforced: true constraints: - type: primary_key columns: [id] name: pk_constraints_example columns: - name: id data_type: int constraints: - type: not_null - type: check # not supported -- will warn & skip expression: "id > 0" data_tests: - unique # primary_key constraint is not enforced - name: customer_name data_type: varchar(100) - name: first_transaction_date data_type: date ``` Expected SQL to enforce constraints: target/run/.../constraints\_example.sql ```sql create table schema_name.constraints_example ( id int not null, customer_name varchar(100), first_transaction_date date ); alter table schema_name.constraints_example add constraint pk_constraints_example primary key nonclustered(id) not enforced; ``` #### Custom constraints In dbt, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. Custom constraints allow you to add configuration to specific columns. For example: * Set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). * Other data warehouses (such as [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) have their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways:  Custom constraints with tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: models/constraints\_example.yml ```yaml models: - name: constraints_example config: contract: enforced: true materialized: table columns: - name: id data_type: int constraints: - type: custom expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. ``` Using this syntax requires configuring all the columns and their types as it’s the only way to send a create or replace `<cols_info_with_masking> mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source).  Custom constraints without tags Alternatively, you can add a masking policy without tags: models/constraints\_example.yml ```yaml models: - name: constraints_example config: contract: enforced: true materialized: table columns: - name: id data_type: int constraints: - type: custom expression: "masking policy my_policy" ``` --- ### contract When the `contract` configuration is enforced, dbt will ensure that your model's returned dataset exactly matches the attributes you have defined in YAML: * `name` and `data_type` for every column * Additional [`constraints`](https://docs.getdbt.com/reference/resource-properties/constraints.md), as supported for this materialization and data platform This is to ensure that the people querying your model downstream—both inside and outside dbt—have a predictable and consistent set of columns to use in their analyses. Even a subtle change in data type, such as from `boolean` (`true`/`false`) to `integer` (`0`/`1`), could cause queries to fail in surprising ways. Contracts give you control over how schemas are enforced, whether that’s on a single model or consistently across many models in a project. #### Prerequisites **These places support model contracts:** * `dbt_project.yml` file * `properties.yml` file * SQL models * Models materialized as one of the following: * `table` * `view` — views offer support for column names and data types, but not `constraints` * `incremental` — with `on_schema_change: append_new_columns` or `on_schema_change: fail` * Certain data platforms, but the supported and [enforced `constraints`](https://docs.getdbt.com/reference/resource-properties/constraints.md) vary by platform **These places do *NOT* support model contracts:** * Python models * `materialized view` or `ephemeral` — materialized SQL models * Custom materializations (unless added by the author) * Models with recursive CTE's in BigQuery * Other resource types, such as `sources`, `seeds`, `snapshots`, and so on Refer to the [Examples](https://docs.getdbt.com/reference/resource-configs/contract.md#examples) to see how to apply contracts in your project. #### Data type aliasing dbt uses built-in type aliasing for the `data_type` defined in your YAML. For example, you can specify `string` in your contract, and on Postgres/Redshift, dbt will convert it to `text`. If dbt doesn't recognize the `data_type` name among its known aliases, it will pass it through as-is. This is enabled by default, but you can opt-out by setting `alias_types` to `false`. Example for disabling: FOLDER\_NAME/FILE\_NAME.yml ```yml models: - name: my_model config: contract: enforced: true alias_types: false # true by default ``` #### Size, precision, and scale When dbt compares data types, it will not compare granular details such as size, precision, or scale. We don't think you should sweat the difference between `varchar(256)` and `varchar(257)`, because it doesn't really affect the experience of downstream queriers. You can accomplish a more-precise assertion by [writing or using a custom test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md). Note that you need to specify a varchar size or numeric scale, otherwise dbt relies on default values. For example, if a `numeric` type defaults to a precision of 38 and a scale of 0, then the numeric column stores 0 digits to the right of the decimal (it only stores whole numbers), which might cause it to fail contract enforcement. To avoid this implicit coercion, specify your `data_type` with a nonzero scale, like `numeric(38, 6)`. dbt 1.7 and higher provides a warning if you don't specify precision and scale when providing a numeric data type. ##### Examples models/dim\_customers.yml ```yml models: - name: dim_customers config: materialized: table contract: enforced: true columns: - name: customer_id data_type: int constraints: - type: not_null - name: customer_name data_type: string - name: non_integer data_type: numeric(38,3) ``` Let's say your model is defined as: models/dim\_customers.sql ```sql select 'abc123' as customer_id, 'My Best Customer' as customer_name ``` When you `dbt run` your model, *before* dbt has materialized it as a table in the database, you will see this error: ```txt 20:53:45 Compilation Error in model dim_customers (models/dim_customers.sql) 20:53:45 This model has an enforced contract that failed. 20:53:45 Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition. 20:53:45 20:53:45 | column_name | definition_type | contract_type | mismatch_reason | 20:53:45 | ----------- | --------------- | ------------- | ------------------ | 20:53:45 | customer_id | TEXT | INT | data type mismatch | 20:53:45 20:53:45 20:53:45 > in macro assert_columns_equivalent (macros/materializations/models/table/columns_spec_ddl.sql) ``` ##### Project YAML Use a contract enforcement in your `dbt_project.yml` to enforce contracts consistently across multiple models: ```yml models: property_management: # replace with your dbt project name +contract: enforced: true ``` ##### Properties YAML Define a model’s contract in a `properties.yml` by specifying the expected columns and data types: ```yml models: - name: stg_rental_applications # replace with your model name config: contract: enforced: true columns: - name: column_1_id # example id column. Replace with your column data_type: int # replace with your column's data type - name: column_2_created_at # example column tracking when something was created data_type: timestamp - name: column_3_status # example status column, which typically store text values ("active", "pending", "completed", etc.) data_type: string ``` ##### SQL file config Enforce a contract in a model SQL file when you want to apply it to a single model and maintain fine-grained control: ```sql {{ config( contract = { "enforced": true } -- Enables contract enforcement for this model ) }} select column_1_id, -- replace with your column column_2_created_at, -- replace with your column column_3_status -- replace with your column from {{ source('property_management', 'rental_applications') }} -- replace with your source name and table ``` Refer to [General configurations](https://docs.getdbt.com/reference/model-configs.md#general-configurations) for more information on the supported configs available for model SQL files, `dbt_project.yml` and `properties.yml`. ##### Incremental models and `on_schema_change` Why require that incremental models also set [`on_schema_change`](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change), and why to `append_new_columns` or `fail`? Imagine: * You add a new column to both the SQL and the YAML spec * You don't set `on_schema_change`, or you set `on_schema_change: 'ignore'` * dbt doesn't actually add that new column to the existing table — and the upsert/merge still succeeds, because it does that upsert/merge on the basis of the already-existing "destination" columns only (this is long-established behavior) * The result is a delta between the YAML-defined contract, and the actual table in the database - which means the contract is now incorrect! Why `append_new_columns` (or `fail`) rather than `sync_all_columns`? Because removing existing columns is a breaking change for contracted models! `sync_all_columns` works like `append_new_columns` but also removes deleted columns, which you're not supposed to do with contracted models unless you upgrade the version. #### Troubleshooting  I’m getting a contract mismatch error **What went wrong:** When a model has an enforced contract, dbt ensures the model’s returned dataset exactly matches the YAML-defined `name` and `data_type` for every column; if they don’t match, dbt errors. **Solution:** Ensure the `name`, `data_type`, and number of columns in the contract match the columns in the model’s definition. For details, refer to the [contract docs](https://docs.getdbt.com/reference/resource-configs/contract.md).  My new column isn’t showing up in my incremental model **What went wrong:** Contracts on incremental models are supported when `on_schema_change` is set to `append_new_columns` or `fail`. If you add a new column but don’t set `on_schema_change` (or set it to `ignore`), dbt doesn’t add that column to the existing table, which can create a mismatch between the YAML-defined contract and the actual table schema. **Solution:** Set `on_schema_change: append_new_columns` (or `fail`) for contracted incremental models. For details, refer to [Incremental models and `on_schema_change`](https://docs.getdbt.com/reference/resource-configs/contract.md#incremental-models-and-on_schema_change).  A column disappeared from my incremental model table **What went wrong:** `sync_all_columns` adds any new columns to the existing table, and removes any columns that are now missing, so a column can be removed from the target table if it’s missing from the new model query. **Solution:** For contracted incremental models, use `append_new_columns` (or `fail`) rather than `sync_all_columns`. Removing existing columns is a breaking change for contracted models. For details, refer to [Incremental models and `on_schema_change`](https://docs.getdbt.com/reference/resource-configs/contract.md#incremental-models-and-on_schema_change).  I’m getting a data type mismatch I didn’t expect **What went wrong:** dbt applies built-in type aliasing for YAML `data_type` values, and relying on default precision/scale (especially for `numeric`) can lead to implicit coercion that may cause contract enforcement to fail. **Solution:** If you want to avoid aliasing, set `alias_types: false`. To avoid implicit numeric coercion, specify a `data_type` with a nonzero scale (for example, `numeric(38, 6)`). For details, refer to the [contract docs](https://docs.getdbt.com/reference/resource-configs/contract.md). #### Related documentation * [What is a model contract?](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) * [Defining `columns`](https://docs.getdbt.com/reference/resource-properties/columns.md) * [Defining `constraints`](https://docs.getdbt.com/reference/resource-properties/constraints.md) --- ### Cross-database macros These macros benefit three different user groups: * If you maintain a package, your package is more likely to work on other adapters by using these macros (rather than a specific database's SQL syntax) * If you maintain an adapter, your adapter is more likely to support more packages by implementing (and testing) these macros. * If you're an end user, more packages and adapters are likely to "just work" for you (without you having to do anything). Note Please make sure to take a look at the [SQL expressions section](#sql-expressions) to understand quoting syntax for string values and date literals. #### All functions (alphabetical) * [Cross-database macros](#cross-database-macros) * [All functions (alphabetical)](#all-functions-alphabetical) * [Data type functions](#data-type-functions) * [type\_bigint](#type_bigint) * [type\_boolean](#type_boolean) * [type\_float](#type_float) * [type\_int](#type_int) * [type\_numeric](#type_numeric) * [type\_string](#type_string) * [type\_timestamp](#type_timestamp) * [current\_timestamp](#current_timestamp) * [Set functions](#set-functions) * [except](#except) * [intersect](#intersect) * [Array functions](#array-functions) * [array\_append](#array_append) * [array\_concat](#array_concat) * [array\_construct](#array_construct) * [String functions](#string-functions) * [concat](#concat) * [hash](#hash) * [length](#length) * [position](#position) * [replace](#replace) * [right](#right) * [split\_part](#split_part) * [String literal functions](#string-literal-functions) * [escape\_single\_quotes](#escape_single_quotes) * [string\_literal](#string_literal) * [Aggregate and window functions](#aggregate-and-window-functions) * [any\_value](#any_value) * [bool\_or](#bool_or) * [listagg](#listagg) * [Cast functions](#cast-functions) * [cast](#cast) * [cast\_bool\_to\_text](#cast_bool_to_text) * [safe\_cast](#safe_cast) * [Comparison functions](#comparison-functions) * [equals](#equals) * [Date and time functions](#date-and-time-functions) * [date](#date) * [dateadd](#dateadd) * [datediff](#datediff) * [date\_trunc](#date_trunc) * [last\_day](#last_day) * [Date and time parts](#date-and-time-parts) * [SQL expressions](#sql-expressions) [**Data type functions**](#data-type-functions) * [type\_bigint](#type_bigint) * [type\_boolean](#type_boolean) * [type\_float](#type_float) * [type\_int](#type_int) * [type\_numeric](#type_numeric) * [type\_string](#type_string) * [type\_timestamp](#type_timestamp) [**Set functions**](#set-functions) * [except](#except) * [intersect](#intersect) [**Array functions**](#array-functions) * [array\_append](#array_append) * [array\_concat](#array_concat) * [array\_construct](#array_construct) [**String functions**](#string-functions) * [concat](#concat) * [hash](#hash) * [length](#length) * [position](#position) * [replace](#replace) * [right](#right) * [split\_part](#split_part) [**String literal functions**](#string-literal-functions) * [escape\_single\_quotes](#escape_single_quotes) * [string\_literal](#string_literal) [**Aggregate and window functions**](#aggregate-and-window-functions) * [any\_value](#any_value) * [bool\_or](#bool_or) * [listagg](#listagg) [**Cast functions**](#cast-functions) * [cast](#cast) * [cast\_bool\_to\_text](#cast_bool_to_text) * [safe\_cast](#safe_cast) [**Comparison functions**](#comparison-functions) * [equals](#equals) [**Date and time functions**](#date-and-time-functions) * [date](#date) * [dateadd](#dateadd) * [datediff](#datediff) * [date\_trunc](#date_trunc) * [last\_day](#last_day) #### Data type functions ##### type\_bigint **Args**: * None This macro yields the database-specific data type for a `BIGINT`. **Usage**: ```sql {{ dbt.type_bigint() }} ``` **Sample Output (PostgreSQL)**: ```sql bigint ``` ##### type\_boolean **Args**: * None This macro yields the database-specific data type for a `BOOLEAN`. **Usage**: ```sql {{ dbt.type_boolean() }} ``` **Sample Output (PostgreSQL)**: ```sql BOOLEAN ``` ##### type\_float **Args**: * None This macro yields the database-specific data type for a `FLOAT`. **Usage**: ```sql {{ dbt.type_float() }} ``` **Sample Output (PostgreSQL)**: ```sql FLOAT ``` ##### type\_int **Args**: * None This macro yields the database-specific data type for an `INT`. **Usage**: ```sql {{ dbt.type_int() }} ``` **Sample Output (PostgreSQL)**: ```sql INT ``` ##### type\_numeric **Args**: * None This macro yields the database-specific data type for a `NUMERIC`. **Usage**: ```sql {{ dbt.type_numeric() }} ``` **Sample Output (PostgreSQL)**: ```sql numeric(28,6) ``` ##### type\_string **Args**: * None This macro yields the database-specific data type for `TEXT`. **Usage**: ```sql {{ dbt.type_string() }} ``` **Sample Output (PostgreSQL)**: ```sql TEXT ``` ##### type\_timestamp **Args**: * None This macro yields the database-specific data type for a `TIMESTAMP` (which may or may not match the behavior of `TIMESTAMP WITHOUT TIMEZONE` from ANSI SQL-92). **Usage**: ```sql {{ dbt.type_timestamp() }} ``` **Sample Output (PostgreSQL)**: ```sql TIMESTAMP ``` ##### current\_timestamp This macro returns the current date and time for the system. Depending on the adapter: * The result may be an aware or naive timestamp. * The result may correspond to the start of the statement or the start of the transaction. **Args** * None **Usage** * You can use the `current_timestamp()` macro within your dbt SQL files like this: ```sql {{ dbt.current_timestamp() }} ``` **Sample output (PostgreSQL)** ```sql now() ``` #### Set functions ##### except **Args**: * None `except` is one of the set operators specified ANSI SQL-92 (along with `union` and `intersect`) and is akin to [set difference](https://en.wikipedia.org/wiki/Complement_\(set_theory\)#Relative_complement). **Usage**: ```sql {{ dbt.except() }} ``` **Sample Output (PostgreSQL)**: ```sql except ``` ##### intersect **Args**: * None `intersect` is one of the set operators specified ANSI SQL-92 (along with `union` and `except`) and is akin to [set intersection](https://en.wikipedia.org/wiki/Intersection_\(set_theory\)). **Usage**: ```sql {{ dbt.intersect() }} ``` **Sample Output (PostgreSQL)**: ```sql intersect ``` #### Array functions ##### array\_append **Args**: * `array` (required): The array to append to. * `new_element` (required): The element to be appended. This element must *match the data type of the existing elements* in the array in order to match PostgreSQL functionality and *not null* to match BigQuery functionality. This macro appends an element to the end of an array and returns the appended array. **Usage**: ```sql {{ dbt.array_append("array_column", "element_column") }} {{ dbt.array_append("array_column", "5") }} {{ dbt.array_append("array_column", "'blue'") }} ``` **Sample Output (PostgreSQL)**: ```sql array_append(array_column, element_column) array_append(array_column, 5) array_append(array_column, 'blue') ``` ##### array\_concat **Args**: * `array_1` (required): The array to append to. * `array_2` (required): The array to be appended to `array_1`. This array must match the data type of `array_1` in order to match PostgreSQL functionality. This macro returns the concatenation of two arrays. **Usage**: ```sql {{ dbt.array_concat("array_column_1", "array_column_2") }} ``` **Sample Output (PostgreSQL)**: ```sql array_cat(array_column_1, array_column_2) ``` ##### array\_construct **Args**: * `inputs` (optional): The list of array contents. If not provided, this macro will create an empty array. All inputs must be the *same data type* in order to match PostgreSQL functionality and *not null* to match BigQuery functionality. * `data_type` (optional): Specifies the data type of the constructed array. This is only relevant when creating an empty array (will otherwise use the data type of the inputs). If `inputs` are `data_type` are both not provided, this macro will create an empty array of type integer. This macro returns an array constructed from a set of inputs. **Usage**: ```sql {{ dbt.array_construct(["column_1", "column_2", "column_3"]) }} {{ dbt.array_construct([], "integer") }} {{ dbt.array_construct([1, 2, 3, 4]) }} {{ dbt.array_construct(["'blue'", "'green'"]) }} ``` **Sample Output (PostgreSQL)**: ```sql array[ column_1 , column_2 , column_3 ] array[]::integer[] array[ 1 , 2 , 3 , 4 ] array[ 'blue' , 'green' ] ``` #### String functions ##### concat **Args**: * `fields`: Jinja array of [attribute names or expressions](#sql-expressions). This macro combines a list of strings together. **Usage**: ```sql {{ dbt.concat(["column_1", "column_2"]) }} {{ dbt.concat(["year_column", "'-'" , "month_column", "'-'" , "day_column"]) }} {{ dbt.concat(["first_part_column", "'.'" , "second_part_column"]) }} {{ dbt.concat(["first_part_column", "','" , "second_part_column"]) }} ``` **Sample Output (PostgreSQL)**: ```sql column_1 || column_2 year_column || '-' || month_column || '-' || day_column first_part_column || '.' || second_part_column first_part_column || ',' || second_part_column ``` ##### hash **Args**: * `field`: [attribute name or expression](#sql-expressions). This macro provides a hash (such as [MD5](https://en.wikipedia.org/wiki/MD5)) of an [expression](#sql-expressions) cast as a string. **Usage**: ```sql {{ dbt.hash("column") }} {{ dbt.hash("'Pennsylvania'") }} ``` **Sample Output (PostgreSQL)**: ```sql md5(cast(column as varchar )) md5(cast('Pennsylvania' as varchar )) ``` ##### length **Args**: * `expression`: string [expression](#sql-expressions). This macro calculates the number of characters in a string. **Usage**: ```sql {{ dbt.length("column") }} ``` **Sample Output (PostgreSQL)**: ```sql length( column ) ``` ##### position **Args**: * `substring_text`: [attribute name or expression](#sql-expressions). * `string_text`: [attribute name or expression](#sql-expressions). This macro searches for the first occurrence of `substring_text` within `string_text` and returns the 1-based position if found. **Usage**: ```sql {{ dbt.position("substring_column", "text_column") }} {{ dbt.position("'-'", "text_column") }} ``` **Sample Output (PostgreSQL)**: ```sql position( substring_column in text_column ) position( '-' in text_column ) ``` ##### replace **Args**: * `field`: [attribute name or expression](#sql-expressions). * `old_chars`: [attribute name or expression](#sql-expressions). * `new_chars`: [attribute name or expression](#sql-expressions). This macro updates a string and replaces all occurrences of one substring with another. The precise behavior may vary slightly from one adapter to another. **Usage**: ```sql {{ dbt.replace("string_text_column", "old_chars_column", "new_chars_column") }} {{ dbt.replace("string_text_column", "'-'", "'_'") }} ``` **Sample Output (PostgreSQL)**: ```sql replace( string_text_column, old_chars_column, new_chars_column ) replace( string_text_column, '-', '_' ) ``` ##### right **Args**: * `string_text`: [attribute name or expression](#sql-expressions). * `length_expression`: numeric [expression](#sql-expressions). This macro returns the N rightmost characters from a string. **Usage**: ```sql {{ dbt.right("string_text_column", "length_column") }} {{ dbt.right("string_text_column", "3") }} ``` **Sample Output (PostgreSQL)**: ```sql right( string_text_column, length_column ) right( string_text_column, 3 ) ``` ##### split\_part **Args**: * `string_text` (required): Text to be split into parts. * `delimiter_text` (required): Text representing the delimiter to split by. * `part_number` (required): Requested part of the split (1-based). If the value is negative, the parts are counted backward from the end of the string. This macro splits a string of text using the supplied delimiter and returns the supplied part number (1-indexed). **Usage**: When referencing a column, use one pair of quotes. When referencing a string, use single quotes enclosed in double quotes. ```sql {{ dbt.split_part(string_text='column_to_split', delimiter_text='delimiter_column', part_number=1) }} {{ dbt.split_part(string_text="'1|2|3'", delimiter_text="'|'", part_number=1) }} ``` **Sample Output (PostgreSQL)**: ```sql split_part( column_to_split, delimiter_column, 1 ) split_part( '1|2|3', '|', 1 ) ``` #### String literal functions ##### escape\_single\_quotes **Args**: * `value`: Jinja string literal value This macro adds escape characters for any single quotes within the provided string literal. Note: if given a column, it will only operate on the column *name*, not the values within the column. To escape quotes for column values, consider a macro like [replace](#replace) or a regular expression replace. **Usage**: ```sql {{ dbt.escape_single_quotes("they're") }} {{ dbt.escape_single_quotes("ain't ain't a word") }} ``` **Sample Output (PostgreSQL)**: ```sql they''re ain''t ain''t a word ``` ##### string\_literal **Args**: * `value`: Jinja string value This macro converts a Jinja string into a SQL string literal. To cast column values to a string, consider a macro like [safe\_cast](#safe_cast) or an ordinary cast. **Usage**: ```sql select {{ dbt.string_literal("Pennsylvania") }} ``` **Sample Output (PostgreSQL)**: ```sql select 'Pennsylvania' ``` #### Aggregate and window functions ##### any\_value **Args**: * `expression`: an [expression](#sql-expressions). This macro returns some value of the expression from the group. The selected value is non-deterministic (rather than random). **Usage**: ```sql {{ dbt.any_value("column_name") }} ``` **Sample Output (PostgreSQL)**: ```sql any(column_name) ``` ##### bool\_or **Args**: * `expression`: [attribute name or expression](#sql-expressions). This macro returns the logical `OR` of all non-`NULL` expressions -- `true` if at least one record in the group evaluates to `true`. **Usage**: ```sql {{ dbt.bool_or("boolean_column") }} {{ dbt.bool_or("integer_column = 3") }} {{ dbt.bool_or("string_column = 'Pennsylvania'") }} {{ dbt.bool_or("column1 = column2") }} ``` **Sample Output (PostgreSQL)**: ```sql bool_or(boolean_column) bool_or(integer_column = 3) bool_or(string_column = 'Pennsylvania') bool_or(column1 = column2) ``` ##### listagg **Args**: * `measure` (required): The [attribute name or expression](#sql-expressions) that determines the values to be concatenated. To only include distinct values add keyword `DISTINCT` to beginning of expression (example: 'DISTINCT column\_to\_agg'). * `delimiter_text` (required): Text representing the delimiter to separate concatenated values by. * `order_by_clause` (optional): An expression (typically one or more column names separated by commas) that determines the order of the concatenated values. * `limit_num` (optional): Specifies the maximum number of values to be concatenated. This macro returns the concatenated input values from a group of rows separated by a specified delimiter. **Usage**: Note: If there are instances of `delimiter_text` within your `measure`, you cannot include a `limit_num`. ```sql {{ dbt.listagg(measure="column_to_agg", delimiter_text="','", order_by_clause="order by order_by_column", limit_num=10) }} ``` **Sample Output (PostgreSQL)**: ```sql array_to_string( (array_agg( column_to_agg order by order_by_column ))[1:10], ',' ) ``` #### Cast functions ##### cast **Availability**: dbt v1.8 or higher. For more information, select the version from the documentation navigation menu. **Args**: * `field`: [attribute name or expression](#sql-expressions). * `type`: data type to convert to This macro casts a value to the specified data type. Unlike [safe\_cast](#safe_cast), this macro will raise an error when the cast fails. **Usage**: ```sql {{ dbt.cast("column_1", api.Column.translate_type("string")) }} {{ dbt.cast("column_2", api.Column.translate_type("integer")) }} {{ dbt.cast("'2016-03-09'", api.Column.translate_type("date")) }} ``` **Sample Output (PostgreSQL)**: ```sql cast(column_1 as TEXT) cast(column_2 as INT) cast('2016-03-09' as date) ``` ##### cast\_bool\_to\_text **Args**: * `field`: boolean [attribute name or expression](#sql-expressions). This macro casts a boolean value to a string. **Usage**: ```sql {{ dbt.cast_bool_to_text("boolean_column_name") }} {{ dbt.cast_bool_to_text("false") }} {{ dbt.cast_bool_to_text("true") }} {{ dbt.cast_bool_to_text("0 = 1") }} {{ dbt.cast_bool_to_text("1 = 1") }} {{ dbt.cast_bool_to_text("null") }} ``` **Sample Output (PostgreSQL)**: ```sql cast(boolean_column_name as varchar ) cast(false as varchar ) cast(true as varchar ) cast(0 = 1 as varchar ) cast(1 = 1 as varchar ) cast(null as varchar ) ``` ##### safe\_cast **Args**: * `field`: [attribute name or expression](#sql-expressions). * `type`: data type to convert to For databases that support it, this macro will return `NULL` when the cast fails (instead of raising an error). **Usage**: ```sql {{ dbt.safe_cast("column_1", api.Column.translate_type("string")) }} {{ dbt.safe_cast("column_2", api.Column.translate_type("integer")) }} {{ dbt.safe_cast("'2016-03-09'", api.Column.translate_type("date")) }} ``` **Sample Output (PostgreSQL)**: ```sql cast(column_1 as TEXT) cast(column_2 as INT) cast('2016-03-09' as date) ``` #### Comparison functions Comparison functions are macros that compare two SQL expressions and return a boolean SQL expression (for example, `TRUE`, `FALSE`, or `UNKNOWN`). ##### equals **Args**: * `a`: [attribute name or expression](#sql-expressions). * `b`: [attribute name or expression](#sql-expressions). This macro compares two expressions for equality. By default, the `equals()` macro follows SQL's [three-valued logic (3VL)](https://modern-sql.com/concept/three-valued-logic), so `NULL = NULL` evaluates to `UNKNOWN` rather than `TRUE`. When the [`enable_truthy_nulls_equals_macro`](https://docs.getdbt.com/reference/global-configs/behavior-flags/enable_truthy_nulls_equals_macro.md) flag is enabled, `equals()` behaves like the [`IS NOT DISTINCT FROM`](https://modern-sql.com/feature/is-distinct-from) SQL operator and treats two `NULL` values as the same. **Usage**: ```sql {{ dbt.equals("column_a", "column_b") }} {{ dbt.equals("id", "previous_id") }} ``` **Sample output (PostgreSQL with [`enable_truthy_nulls_equals_macro`](https://docs.getdbt.com/reference/global-configs/behavior-flags/enable_truthy_nulls_equals_macro.md) enabled)**: ```sql (column_a IS NOT DISTINCT FROM column_b) (id IS NOT DISTINCT FROM previous_id) ``` #### Date and time functions ##### date **Availability**: dbt v1.8 or later. For more information, select the version from the documentation navigation menu. **Args**: * `year`: an integer * `month`: an integer * `day`: an integer This macro converts the `year`, `month`, and `day` into an SQL `DATE` type. **Usage**: ```sql {{ dbt.date(2023, 10, 4) }} ``` **Sample output (PostgreSQL)**: ```sql to_date('2023-10-04', 'YYYY-MM-DD') ``` ##### dateadd **Args**: * `datepart`: [date or time part](#date-and-time-parts). * `interval`: integer count of the `datepart` to add (can be positive or negative) * `from_date_or_timestamp`: date/time [expression](#sql-expressions). This macro adds a time/day interval to the supplied date/timestamp. Note: The `datepart` argument is database-specific. **Usage**: ```sql {{ dbt.dateadd(datepart="day", interval=1, from_date_or_timestamp="'2016-03-09'") }} {{ dbt.dateadd(datepart="month", interval=-2, from_date_or_timestamp="'2016-03-09'") }} ``` **Sample Output (PostgreSQL)**: ```sql '2016-03-09' + ((interval '10 day') * (1)) '2016-03-09' + ((interval '10 month') * (-2)) ``` ##### datediff **Args**: * `first_date`: date/time [expression](#sql-expressions). * `second_date`: date/time [expression](#sql-expressions). * `datepart`: [date or time part](#date-and-time-parts). This macro calculates the difference between two dates. **Usage**: ```sql {{ dbt.datediff("column_1", "column_2", "day") }} {{ dbt.datediff("column", "'2016-03-09'", "month") }} {{ dbt.datediff("'2016-03-09'", "column", "year") }} ``` **Sample Output (PostgreSQL)**: ```sql ((column_2)::date - (column_1)::date) ((date_part('year', ('2016-03-09')::date) - date_part('year', (column)::date)) * 12 + date_part('month', ('2016-03-09')::date) - date_part('month', (column)::date)) (date_part('year', (column)::date) - date_part('year', ('2016-03-09')::date)) ``` ##### date\_trunc **Args**: * `datepart`: [date or time part](#date-and-time-parts). * `date`: date/time [expression](#sql-expressions). This macro truncates / rounds a timestamp to the first instant for the given [date or time part](#date-and-time-parts). **Usage**: ```sql {{ dbt.date_trunc("day", "updated_at") }} {{ dbt.date_trunc("month", "updated_at") }} {{ dbt.date_trunc("year", "'2016-03-09'") }} ``` **Sample Output (PostgreSQL)**: ```sql date_trunc('day', updated_at) date_trunc('month', updated_at) date_trunc('year', '2016-03-09') ``` ##### last\_day **Args**: * `date`: date/time [expression](#sql-expressions). * `datepart`: [date or time part](#date-and-time-parts). This macro gets the last day for a given date and datepart. **Usage**: * The `datepart` argument is database-specific. * This macro currently only supports dateparts of `month` and `quarter`. ```sql {{ dbt.last_day("created_at", "month") }} {{ dbt.last_day("'2016-03-09'", "year") }} ``` **Sample Output (PostgreSQL)**: ```sql cast( date_trunc('month', created_at) + ((interval '10 month') * (1)) + ((interval '10 day') * (-1)) as date) cast( date_trunc('year', '2016-03-09') + ((interval '10 year') * (1)) + ((interval '10 day') * (-1)) as date) ``` #### Date and time parts Often supported date and time parts (case insensitive): * `year` * `quarter` * `month` * `week` * `day` * `hour` * `minute` * `second` * `millisecond` * `microsecond` * `nanosecond` This listing is not meant to be exhaustive, and some of these date and time parts may not be supported for particular adapters. Some macros may not support all date and time parts. Some adapters may support more or less precision. #### SQL expressions A SQL expression may take forms like the following: * function * column name * date literal * string literal * \<other data type> literal (number, etc) * `NULL` Example: Suppose there is an `orders` table with a column named `order_date`. The following shows 3 different types of expressions: ```sql select date_trunc(month, order_date) as expression_function, order_date as expression_column_name, '2016-03-09' as expression_date_literal, 'Pennsylvania' as expression_string_literal, 3 as expression_number_literal, NULL as expression_null, from orders ``` Note that the string literal example includes single quotes. (Note: the string literal character may vary per database. For this example, we suppose a single quote.) To refer to a SQL string literal in Jinja, surrounding double quotes are required. So within Jinja, the string values would be: * `"date_trunc(month, order_date)"` * `"order_date"` * `"'2016-03-09'"` * `"'Pennsylvania'"` * `"NULL"` --- ### Cumulative metrics | require\_nested\_cumulative\_type\_params | dbt **v1 Latest** | dbt v1 | | ----------------------------------------- | ----------------- | ------ | | Introduced | 2024.11 | 1.9.0 | | Matured (default → `true`) | 2026.09 | 1.12.0 | | Removed | — | — | <br /> [Cumulative-type metrics](https://docs.getdbt.com/docs/build/cumulative.md#parameters) are nested under the `cumulative_type_params` field in [the dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), dbt v1.9 and newer. Starting in dbt v1.12, this flag defaults to `true`, which causes dbt to raise an error instead of a warning when cumulative metrics use the un-nested syntax. Use the following metric configured with the syntax before v1.9 as an example: ```yaml type: cumulative type_params: measure: order_count window: 7 days ``` If you run `dbt parse` with that syntax on dbt v1.9 or [the dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), you will receive a warning like: ```bash 15:36:22 [WARNING]: Cumulative fields `type_params.window` and `type_params.grain_to_date` has been moved and will soon be deprecated. Please nest those values under `type_params.cumulative_type_params.window` and `type_params.cumulative_type_params.grain_to_date`. See documentation on behavior changes: https://docs.getdbt.com/reference/global-configs/behavior-changes ``` Because `require_nested_cumulative_type_params` defaults to `true`, running `dbt parse` produces an error like: ```bash 21:39:18 Cumulative fields `type_params.window` and `type_params.grain_to_date` should be nested under `type_params.cumulative_type_params.window` and `type_params.cumulative_type_params.grain_to_date`. Invalid metrics: orders_last_7_days. See documentation on behavior changes: https://docs.getdbt.com/reference/global-configs/behavior-changes. ``` Once the metric is updated, it will work as expected: ```yaml type: cumulative type_params: measure: name: order_count cumulative_type_params: window: 7 days ``` #### Impact Any project with a cumulative metric still using the un-nested syntax stops parsing entirely on the first command. Because parsing fails, the error affects every dbt command: `run`, `build`, `test`, `compile`, `docs generate`, the Semantic Layer, and more.  Recommended actions If `dbt parse` fails with `Semantic Manifest validation failed` referencing one or more cumulative metrics, migrate the metrics by moving `window` and `grain_to_date` from `type_params` into `type_params.cumulative_type_params`. Alternatively, you can migrate to the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md#type_params), which removes `type_params` entirely. ```yaml # Legacy metrics: - name: weekly_active_users type: cumulative type_params: measure: distinct_users window: 7 days # Nested metrics: - name: weekly_active_users type: cumulative type_params: measure: distinct_users cumulative_type_params: window: 7 days # Latest spec metrics: - name: weekly_active_users type: cumulative input_metric: distinct_users window: 7 days ``` Re-run `dbt parse` to confirm the manifest validates. To opt out of this behavior, set the flag to `false`: dbt\_project.yml ```yaml flags: require_nested_cumulative_type_params: false ``` --- ### Custom microbatch strategy Removed in v2 This flag was removed in v2. The new behavior is always enabled. If you're upgrading, remove this flag from your `dbt_project.yml`. | require\_batched\_execution\_for\_custom\_microbatch\_strategy | dbt **v1 Latest** | dbt v1 | | -------------------------------------------------------------- | ----------------- | ------ | | Introduced | 2024.11 | 1.9.0 | | Matured (default → `true`) | 2026.09 | 1.12.0 | | Removed | — | v2.0 | <br /> The `require_batched_execution_for_custom_microbatch_strategy` flag is only relevant if you already have a custom `get_incremental_microbatch_sql` macro in your project. If you don't have a custom microbatch macro, you don't need to set this flag — dbt handles microbatching automatically for any model using the [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md#how-microbatch-compares-to-other-incremental-strategies). Starting in dbt v1.12, this flag defaults to `true`, meaning dbt executes custom microbatch strategies in batches. If you have a custom microbatch macro and set the flag to `false`, dbt issues a deprecation warning. Previously, users needed to set the `DBT_EXPERIMENTAL_MICROBATCH` environment variable to `true` to prevent unintended interactions with existing custom incremental strategies. Setting `DBT_EXPERIMENTAL_MICROBATCH` no longer has any effect on runtime functionality. #### Impact This flag is only relevant if your project has a custom `get_incremental_microbatch_sql` macro. Projects without a custom microbatch macro are unaffected — the built-in macro already runs in batches. If you have overridden `get_incremental_microbatch_sql` — typically to work around an adapter limitation or implement a custom partition strategy — your macro is invoked under a batched contract for which it was never written. Possible outcomes: * The macro ignores the smaller `event_time_start` / `event_time_end` window and re-processes the full range every batch, leading to wasted compute, duplicate rows, or `MERGE`/`INSERT` conflicts. * The macro errors because it expects a single invocation per run. * Row counts in the destination table differ between flag values.  Recommended actions If a microbatch model errors mid-run, produces duplicate rows or `MERGE`/`INSERT` conflicts, or reprocesses the full date range for every batch, check whether the `get_incremental_microbatch_sql` macro filters by the `event_time_start` / `event_time_end` window. If it uses a hard-coded date filter (for example, `where event_ts >= current_date - interval '1 day'`) or reads only `_dbt_max_partition`, the macro ignores the per-batch window and reprocesses the full range on every iteration. To fix, rewrite the macro to use the per-batch window: ```sql {% macro get_incremental_microbatch_sql(arg_dict) %} {%- set target = arg_dict["target_relation"] -%} {%- set source = arg_dict["temp_relation"] -%} {%- set event_time = arg_dict["incremental_predicates"] -%} -- Use the per-batch window dbt passes via event_time_start/event_time_end -- (exposed through arg_dict["incremental_predicates"] or model.config.event_time_*), -- NOT a hard-coded date filter. insert into {{ target }} select * from {{ source }} {% endmacro %} ``` To verify, run `dbt run --event-time-start <date> --event-time-end <date> -s <model>` for a single batch and confirm the row count matches that single day. To opt out of this behavior, set the flag to `false`: dbt\_project.yml ```yaml flags: require_batched_execution_for_custom_microbatch_strategy: false ``` --- ### Data test configurations #### Related documentation * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) Data tests can be configured in a few different ways: 1. Properties within `.yml` definition (generic tests only, see [test properties](https://docs.getdbt.com/reference/resource-properties/data-tests.md) for full syntax) 2. A `config()` block within the test's SQL definition 3. In `dbt_project.yml` Data test configs are applied hierarchically, in the order of specificity outlined above. In the case of a singular test, the `config()` block within the SQL definition takes precedence over configs in the project YAML file. In the case of a specific instance of a generic test, the test's `.yml` properties would take precedence over any values set in its generic SQL definition's `config()`, which in turn would take precedence over values set in the project YAML file (`dbt_project.yml`). #### Available configurations Click the link on each configuration option to read more about what it can do. ##### Data test-specific configurations Resource-specific configurations are applicable to only one dbt resource type rather than multiple resource types. You can define these settings in the project file (`dbt_project.yml`), a property file (`models/properties.yml` for models, similarly for other resources), or within the resource’s file using the `{{ config() }}` macro.<br /> The following resource-specific configurations are only available to Data tests: ##### Project file dbt\_project.yml ```yaml data_tests: <resource-path>: +fail_calc: <string> +limit: <integer> +severity: error | warn +error_if: <string> +warn_if: <string> +store_failures: true | false +where: <string> ``` ##### SQL file config ```jinja {{ config( fail_calc = "<string>", limit = <integer>, severity = "error | warn", error_if = "<string>", warn_if = "<string>", store_failures = true | false, where = "<string>" ) }} ``` ##### Property file ```yaml <resource_type>: - name: <resource_name> data_tests: - <test_name>: # # Actual name of the test. For example, dbt_utils.equality name: # Human friendly name for the test. For example, equality_fct_test_coverage description: "markdown formatting" arguments: # Available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: fail_calc: <string> limit: <integer> severity: error | warn error_if: <string> warn_if: <string> store_failures: true | false where: <string> # Available in v1.12 and higher. Requires enabling the `require_sql_header_in_test_configs` flag. sql_header: <string> columns: - name: <column_name> data_tests: - <test_name>: name: description: "markdown formatting" arguments: # Available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: fail_calc: <string> limit: <integer> severity: error | warn error_if: <string> warn_if: <string> store_failures: true | false where: <string> # Available in v1.12 and higher. Requires enabling the `require_sql_header_in_test_configs` flag. sql_header: <string> ``` This configuration mechanism is supported for specific instances of generic tests only. To configure a specific singular test, you should use the `config()` macro in its SQL definition. Starting in dbt v1.12, you can set [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) in the `config` of a generic data test at the model or column level of your `properties.yml`. Enable the [`require_sql_header_in_test_configs`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_sql_header_in_test_configs.md) flag to use `config.sql_header` in your data tests. ##### General configurations General configurations provide broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project file, property files, or within resource-specific files. ##### Project file dbt\_project.yml ```yaml data_tests: <resource-path>: +enabled: true | false +tags: <string> | [<string>] +meta: {dictionary} # relevant for store_failures only +database: <string> +schema: <string> +alias: <string> ``` ##### SQL file config ```jinja {{ config( enabled=true | false, tags="<string>" | ["<string>"] meta={dictionary}, database="<string>", schema="<string>", alias="<string>", ) }} ``` ##### Property file ```yaml <resource_type>: - name: <resource_name> data_tests: - <test_name>: # Actual name of the test. For example, dbt_utils.equality name: # Human friendly name for the test. For example, equality_fct_test_coverage description: "markdown formatting" arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: enabled: true | false tags: <string> | [<string>] meta: {dictionary} # relevant for store_failures only database: <string> schema: <string> alias: <string> columns: - name: <column_name> data_tests: - <test_name>: name: description: "markdown formatting" arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. <argument_name>: <argument_value> config: enabled: true | false tags: <string> | [<string>] meta: {dictionary} # relevant for store_failures only database: <string> schema: <string> alias: <string> ``` This configuration mechanism is supported for specific instances of generic data tests only. To configure a specific singular test, you should use the `config()` macro in its SQL definition. ##### Examples ###### Add a tag to one test If a specific instance of a generic data test: models/\<filename>.yml ```yml models: - name: my_model columns: - name: id data_tests: - unique: config: tags: ['my_tag'] # changed to config in v1.10 ``` If a singular data test: tests/\<filename>.sql ```sql {{ config(tags = ['my_tag']) }} select ... ``` ###### Set the default severity for all instances of a generic data test macros/\<filename>.sql ```sql {% test my_test() %} {{ config(severity = 'warn') }} select ... {% endtest %} ``` ###### Disable all data tests from a package dbt\_project.yml ```yml data_tests: package_name: +enabled: false ``` ###### Specify custom configurations for generic data tests Beginning in dbt v1.9, you can use any custom config key to specify custom configurations for data tests. For example, the following specifies the `snowflake_warehouse` custom config that dbt should use when executing the `accepted_values` data test: ```yml models: - name: my_model columns: - name: color data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['blue', 'red'] config: severity: warn snowflake_warehouse: my_warehouse ``` Given the config, the data test runs on a different Snowflake virtual warehouse than the one in your default connection to enable better price-performance with a different warehouse size or more granular cost allocation and visibility. ###### Add a description to generic and singular tests Starting from dbt v1.9 (also available to dbt [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md)), you can add [descriptions](https://docs.getdbt.com/reference/resource-properties/data-tests.md#description) to both generic and singular tests. For a generic test, add the description in line with the existing YAML: models/staging/\<filename>.yml ```yml models: - name: my_model columns: - name: delivery_status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['delivered', 'pending', 'failed'] description: "This test checks whether there are unexpected delivery statuses. If it fails, check with logistics team" ``` You can also add descriptions to the Jinja macro that provides the core logic of a generic data test. Refer to the [Add description to generic data test logic](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md#add-description-to-generic-data-test-logic) for more information. For a singular test, define it in the test's directory: tests/my\_custom\_test.yml ```yml data_tests: - name: my_custom_test description: "This test checks whether the rolling average of returns is inside of expected bounds. If it isn't, flag to customer success team" ``` For more information refer to [Add a description to a data test](https://docs.getdbt.com/reference/resource-properties/description.md#add-a-description-to-a-data-test). (Applies to dbt v1.12 and later) ###### Set `sql_header` in a generic data test When the [`require_sql_header_in_test_configs`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_sql_header_in_test_configs.md) flag is enabled, you can set [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) in the `config` of a generic data test so that the specified SQL runs before the test executes (for example, to set session parameters or add a comment): models/properties.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - not_null: name: not_null_orders_order_id config: sql_header: "-- SQL_HEADER_TEST_MARKER" ``` --- ### database ##### Model Specify a custom database for a model in your `dbt_project.yml` file. For example, if you have a model that you want to load into a database other than the target database, you can configure it like this: dbt\_project.yml ```yml models: your_project: sales_metrics: +database: reporting ``` This would result in the generated relation being located in the `reporting` database, so the full relation name would be `reporting.finance.sales_metrics` instead of the default target database. ##### Seeds Configure a database in your `dbt_project.yml` file. For example, to load a seed into a database called `staging` instead of the target database, you can configure it like this: dbt\_project.yml ```yml seeds: your_project: product_categories: +database: staging ``` This would result in the generated relation being located in the `staging` database, so the full relation name would be `staging.finance.product_categories`. ##### Snapshots (Applies to dbt v1.9 and later) Specify a custom database for a snapshot in your `dbt_project.yml`, snapshot.yml file, or config file. For example, if you have a snapshot that you want to load into a database other than the target database, you can configure it like this: dbt\_project.yml ```yml snapshots: your_project: your_snapshot: +database: snapshots ``` Or in a `snapshot_name.yml` file: snapshots/snapshot\_name.yml ```yaml snapshots: - name: snapshot_name config: database: snapshots ``` This results in the generated relation being located in the `snapshots` database so the full relation name would be `snapshots.finance.your_snapshot` instead of the default target database. ##### Tests Customize the database for storing test results in your `dbt_project.yml` file. For example, to save test results in a specific database, you can configure it like this: dbt\_project.yml ```yml data_tests: +store_failures: true +database: test_results ``` This would result in the test results being stored in the `test_results` database. #### Definition Optionally specify a custom database for a [model](https://docs.getdbt.com/docs/build/sql-models.md), [seed](https://docs.getdbt.com/docs/build/seeds.md), [snapshot](https://docs.getdbt.com/docs/build/snapshots.md), or [data test](https://docs.getdbt.com/docs/build/data-tests.md). When dbt creates a relation (table/view) in a database, it creates it as: `{{ database }}.{{ schema }}.{{ identifier }}`, e.g. `analytics.finance.payments` The standard behavior of dbt is: * If a custom database is *not* specified, the database of the relation is the target database (`{{ target.database }}`). * If a custom database is specified, the database of the relation is the `{{ database }}` value. To learn more about changing the way that dbt generates a relation's `database`, read [Using Custom Databases](https://docs.getdbt.com/docs/build/custom-databases.md) #### Warehouse specific information * BigQuery: `project` and `database` are interchangeable * Databricks: `catalog` and `database` are interchangeable --- ### Database permissions Database permissions are access rights and privileges granted to users or roles within a database or data platform. They help you specify what actions users or roles can perform on various database objects, like tables, views, schemas, or even the entire database. ##### Why are they useful * Database permissions are essential for security and data access control. * They ensure that only authorized users can perform specific actions. * They help maintain data integrity, prevent unauthorized changes, and limit exposure to sensitive data. * Permissions also support compliance with data privacy regulations and auditing. ##### How to use them * Users and administrators can grant and manage permissions at various levels (such as table, schema, and so on) using SQL statements or through the database system's interface. * Assign permissions to individual users or roles (groups of users) based on their responsibilities. * Typical permissions include "SELECT" (read), "INSERT" (add data), "UPDATE" (modify data), "DELETE" (remove data), and administrative rights like "CREATE" and "DROP." * Users should be assigned permissions that ensure they have the necessary access to perform their tasks without overextending privileges. Something to note is that each data platform provider might have different approaches and names for privileges. Refer to their documentation for more details. ##### Examples Refer to the following database permission pages for more info on examples and how to set up database permissions: * [Databricks](https://docs.getdbt.com/reference/database-permissions/databricks-permissions.md) * [Postgres](https://docs.getdbt.com/reference/database-permissions/postgres-permissions.md) * [Redshift](https://docs.getdbt.com/reference/database-permissions/redshift-permissions.md) * [Snowflake](https://docs.getdbt.com/reference/database-permissions/snowflake-permissions.md) --- ### Databricks adapter behavior changes The following are the current [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) that are specific to `dbt-databricks`: | Flag | `dbt-databricks`: Intro | `dbt-databricks`: Maturity | Status | | ------------------------------------------------------------------------------------------- | ----------------------- | -------------------------- | --------------------------- | | [`use_info_schema_for_columns`](#use-information-schema-for-columns) | 1.9.0 | N/A | **Removed in 1.11.0** | | [`use_user_folder_for_python`](#use-users-folder-for-python-model-notebooks) | 1.9.0 | 1.11.0 | Default changed to `true` | | [`use_materialization_v2`](#use-restructured-materializations) | 1.10.0 | TBD | Active | | [`use_managed_iceberg`](#use-managed-iceberg) | 1.11.0 | 1.12.0 | Active | | [`use_replace_on_for_insert_overwrite`](#use-replace-on-for-insert_overwrite-strategy) | 1.11.0 | 1.12.0 | Active, defaults to `true` | | [`use_describe_as_json_for_relation_metadata`](#use-describe-as-json-for-relation-metadata) | 1.12.0 | TBD | Active, defaults to `false` | #### Use information schema for columns Removed in v1.11.0 The `use_info_schema_for_columns` flag has been **removed** as of dbt-databricks v1.11.0. The adapter now uses [`DESCRIBE EXTENDED ... AS JSON`](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-describe-table) (available in DBR 16.2+) to efficiently retrieve complex type information, eliminating the need for this flag. If you're still using this flag in your project configuration, you can safely remove it. The new approach provides better performance and doesn't require the `REPAIR TABLE` operations that were needed with `information_schema`. ##### Legacy documentation *This applies to dbt-databricks versions v1.11 and older* The `use_info_schema_for_columns` flag was `false` by default in versions 1.9 and 1.10. Setting this flag to `true` would use `information_schema` rather than `describe extended` to get column metadata for Unity Catalog tables. This setting helped avoid issues where `describe extended` truncates information when the type is a complex struct. For complex types If your complex type comes from processing JSON using `from_json`, you have an alternative: use [`parse_json` to create the column as the `variant` type](https://docs.databricks.com/aws/en/sql/language-manual/functions/parse_json). The `variant` type might be a reasonable alternative in terms of performance, while avoiding type truncation issues. #### Use user's folder for Python model notebooks Default changed in v1.11.0 As of dbt-databricks v1.11.0, the `use_user_folder_for_python` flag defaults to **`true`**. The `use_user_folder_for_python` flag controls where uploaded Python model notebooks are stored in Databricks: * **`true` (default in v1.11+)**: Notebooks are written to `/Users/{{current user}}/{{catalog}}/{{schema}}/`. * **`false` (default in v1.9-v1.10)**: Notebooks are written to `/Shared/dbt_python_models/{{schema}}/`. Databricks deprecated writing to the `Shared` folder as it doesn't align with governance best practices. Using user-specific folders provides better isolation, access control, and aligns with Unity Catalog security models. To preserve the legacy behavior for backward compatibility, you can explicitly set this flag to `false` in your `dbt_project.yml`: ```yaml flags: use_user_folder_for_python: false ``` #### Use restructured materializations The `use_materialization_v2` flag is `false` by default and guards significant rewrites of the core materializations in `dbt-databricks` while they are still in an experimental stage. When set to `true`, `dbt-databricks` uses the updated logic for all model types (views, tables, incremental, seeds). It also enables additional, optional config options for more fine-tuned control: * `view_update_via_alter` — When enabled, this config attempts to update the view in place using alter view, instead of using create or replace to replace it. * `use_safer_relation_operation` — When enabled (and if `view_update_via_alter` isn't set), this config makes dbt model updates more safe by staging relations and using rename operations to ensure the live version of the table or view is not disrupted by failures. These configs aren't required to receive the core benefits of this flag — like better performance and column/constraint functionality — but they are gated behind the flag because they introduce more significant changes to how materializations behave. In v1.11.0, this flag will stay set to `false` by default. Based on feedback about the new materialization’s lack of atomicity (all-or-nothing updates), we won’t enable it automatically. We’ll explore other ways to achieve the same benefits without losing atomicity. Given feedback about lack of atomicity of the new materialization approach, we will not be flipping this flag to `true`. Instead, we will be investigating new ways to provide the same benefits while maintaining atomicity. ##### Changes to the Seed materialization The seeds materialization should have the smallest difference between the old and new materialization, as the primary difference is just removing calls to methods that are not supported by Databricks, such as transaction operations. ##### Changes to the View materialization With the `use_materialization_v2` flag set to `true`, there are two model configuration options that can customize how we handle the view materialization when we detect an existing relation at the target location. * `view_update_via_alter` — Updates the view in place using alter view, instead of using create or replace to replace it. This allows continuity of history for the view, keeps the metadata, and helps with Unity Catalog compatibility. Here's an example of how to configure this: schema.yml ```yaml models: - name: market_summary config: materialized: view view_update_via_alter: true columns: - name: country data_tests: - unique - not_null ... ``` There is currently no support for altering the comment on a view via Databricks SQL. As such, we must replace the view whenever you change its description * `use_safer_relation_operations` — When enabled (and if `view_update_via_alter` isn't set), this config makes dbt model updates more safe by creating a new relation in a staging location, swapping it with the existing relation, and deleting the old relation afterward. The following example shows how to configure this: schema.yml ```yaml models: - name: market_summary config: materialized: view use_safer_relation_operations: true columns: - name: country data_tests: - unique - not_null ... ``` This configuration option may increase costs and disrupt Unity Catalog history. While this approach is equivalent to the default dbt view materialization, it will create additional UC objects, as compared to alternatives. Since this config does not use atomic 'create or replace...' for any materialization, the history of the object in Unity Catalog may not behave as you expect. Consider carefully before using this model config broadly. ##### Changes to the Table materialization This flag may increase storage costs for tables. As with views, these materialization changes could increase costs. More temporary objects are used, consistent with other dbt adapters' materializations. We consider these changes experimental in part because we do not have enough data quantifying the price impact of this change. The benefits though are improvements in performance, safety, and unblocking features that cannot be delivered with the existing materialization. When `use_materialization_v2` is set to `true`, all materialization paths are updated. The key change is that table creation is separated from inserting rows into the table. This separation greatly improves performance for setting table comments, since adding comments at create time is faster than using separate `alter table` statements. It also resolves compatibility issues in Databricks, where creating and inserting in one step prevents setting comments. Additionally, this change makes it possible to support other column features — like column-level masks — that aren’t compatible with inserting data during creation. While these features aren’t included in version 1.10.0, they can now be added in future releases. ###### Constraints For several feature releases now, dbt-databricks supported both dbt's [constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md) implementation and our own alternative, earlier version called `persist_constraints`. With the `use_materialization_v2` flag, we're beginning to deprecate `persist_constraints` and shifting fully to dbt's native constraint support. One new enhancement is support for the `expression` field on primary and foreign keys, which lets you pass additional Databricks options — like using [`RELY` to tell the Databricks optimizer that it may exploit the constraint to rewrite queries](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-table-constraint). Separating `create` and `insert` also changes how constraints behave. Previously, we would create a table with data and then apply constraints. If the new data violated a constraint, the run would fail — but by then, it had already replaced the valid table from the previous run. As with views, you can select between performance and safety with the [`use_safer_relation_operations` flag](#use_safer_relation_operations), but regardless of setting, the new materialization approach ensures constraint violations don't make it into the target table. ###### `use_safer_relation_operations` When using this model configuration with tables, we first create a staging table. After successfully inserting data into the table, we rename it to replace the target materialization. Since Databricks doesn’t support rollbacks, this is a safer approach — if something fails before the rename, the original table stays intact. That gives you time to troubleshoot without worrying that exposures or work streams relying on that table are broken in the mean time. If this config is set to `false` (the default), the target table will still never contain constraint-violating data, but it might end up empty if the insert fails due to the constraint. The key difference is whether we replace the target directly or use a staging-and-rename approach. This configuration option may increase costs and disrupt Unity Catalog history. As with views, there is a cost to using additional temporary objects, in the form of creating more UC objects with their own history. Consider carefully whether you need this behavior. ##### Changes to the Incremental materialization All the changes made to the [Table materialization section](#changes-to-the-table-materialization) also apply to Incremental materializations. We’ve also added a new config: `incremental_apply_config_changes`. This config lets you control whether dbt should apply changes to things like `tags`, `tblproperties`, and comments during incremental runs. Many users wanted the capability to configure table metadata in Databricks — like AI-generated comments — without dbt overwriting them. Previously, dbt-databricks always applied detected changes during incremental runs. With the V2 materialization, you can now set `incremental_apply_config_changes` to `false` to stop that behavior. (It defaults to `true` to match the previous behavior.) The following example shows how to configure this: schema.yml ```yaml models: - name: incremental_market_updates config: materialized: incremental incremental_apply_config_changes: false ... ``` #### Use managed Iceberg When you set `table_format` to `iceberg`, the `use_managed_iceberg` flag controls how the table is created. By default, this flag is set to `false` and dbt creates a [UniForm](https://www.databricks.com/blog/delta-uniform-universal-format-lakehouse-interoperability) table. When set to `true`, dbt creates a [managed Iceberg](https://docs.databricks.com/aws/en/tables/managed) table. #### Use `replace on` for `insert_overwrite` strategy The `use_replace_on_for_insert_overwrite` flag controls which SQL syntax dbt generates for incremental models using the `insert_overwrite` strategy. This flag defaults to `true` by default and results in using the [`insert into ... replace on`](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-dml-insert-into#replace-on) syntax to perform dynamic partition/cluster overwrites, which is the same behavior as in cluster computes. When the flag is set to `false`, `insert_overwrite` will truncate the entire table when used with SQL warehouses. The flag is not relevant for cluster computes because the `insert_overwrite`'s behavior has always been dynamic partition/cluster overwrites in cluster computes. | Flag value | SQL generated | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `true` (default) | [`INSERT INTO ... REPLACE ON`](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-dml-insert-into#replace-on) | Uses the latest, recommended Databricks syntax to replace matching partitions. | | `false` | `INSERT OVERWRITE` | Uses the older Spark syntax to overwrite partitions. Depends on Spark session settings. | If you previously relied on this behavior to get full table replacement without dropping existing metadata, that behavior continues to exist with the flag set to `true`, provided you do not use any partitions or liquid clustering clusters. These data layout optimizations only tend to have a significant effect for tables that are approximately 1 TB large or greater, at which point regular replacement of all of the data is probably not the best approach. #### Use `DESCRIBE AS JSON` for relation metadata The `use_describe_as_json_for_relation_metadata` flag controls how dbt fetches relation-level metadata such as constraints (primary key, foreign key, and non-null), column masks, row filters, and view descriptions for Databricks tables and views. It accepts two values: * **`false`** (default): dbt issues multiple `information_schema` queries per relation to fetch this metadata. * **`true`**: dbt fetches all of this metadata in a single [`DESCRIBE TABLE EXTENDED ... AS JSON`](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-aux-describe-table) call per relation. Enabling this flag is useful for projects that have many models with constraints, column masks, or row filters, because it reduces the number of metadata queries dbt has to issue. ###### Requirements dbt uses the `DESCRIBE AS JSON` path for a given relation only when *all* of the following are true: * The relation is in Unity Catalog (not the Hive metastore). * The relation is not a foreign table. * The compute is a SQL warehouse, or runs on DBR 17.3 or later. If any condition isn't met, dbt falls back to `information_schema` queries for that relation. The fallback happens per relation, so you can safely enable the flag on projects with mixed compute types. --- ### Databricks configurations #### Configuring tables When materializing a model as `table`, you may include several optional configs that are specific to the dbt-databricks plugin, in addition to the standard [model configs](https://docs.getdbt.com/reference/model-configs.md). dbt-databricks v1.9 adds support for the `table_format: iceberg` config. Try it now on the [dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). All other table configurations were also supported in 1.8. | Option | Description | Required? | Model support | Example | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- | ---------------- | --------------------------- | | table\_format | Whether or not to provision [Iceberg](https://docs.databricks.com/en/delta/uniform.html) compatibility for the materialization | Optional | SQL, Python | `iceberg` | | use\_uniform 1 | Controls whether dbt creates a Unity Catalog managed Iceberg table (`false`) or a UniForm Delta table with Iceberg reads enabled (`true`) when `table_format` is `iceberg`. Refer to [Databricks Iceberg support](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md#choose-between-managed-iceberg-and-uniform). | Optional | SQL, Python | `true` | | file\_format 2 | The file format to use when creating tables (`parquet`, `delta`, `hudi`, `csv`, `json`, `text`, `jdbc`, `orc`, `hive` or `libsvm`). | Optional | SQL, Python | `delta` | | location\_root | The created table uses the specified directory to store its data. The table alias is appended to it. | Optional | SQL, Python | `/mnt/root` | | include\_full\_name\_in\_path | Whether to use the full table path to qualify the location root. If this is set, the database, schema, and table alias are all appended to the location root. | Optional | SQL, Python | `true` | | partition\_by | Partition the created table by the specified columns. A directory is created for each partition. | Optional | SQL, Python | `date_day` | | liquid\_clustered\_by3 | Cluster the created table by the specified columns. Clustering method is based on [Delta's Liquid Clustering feature](https://docs.databricks.com/en/delta/clustering.html). Available since dbt-databricks 1.6.2. | Optional | SQL, Python | `date_day` | | auto\_liquid\_cluster4 | The created table is [automatically clustered by Databricks](https://docs.databricks.com/aws/en/delta/clustering#automatic-liquid-clustering). Available since dbt-databricks 1.10.0 | Optional | SQL, Python | `auto_liquid_cluster: true` | | clustered\_by | Each partition in the created table will be split into a fixed number of buckets by the specified columns. | Optional | SQL, Python | `country_code` | | buckets | The number of buckets to create while clustering | Required if `clustered_by` is specified | SQL, Python | `8` | | tblproperties | [Tblproperties](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-tblproperties.html) to be set on the created table | Optional | SQL, Python5 | `{'this.is.my.key': 12}` | | databricks\_tags | [Tags](https://docs.databricks.com/en/data-governance/unity-catalog/tags.html) to be set on the created table | Optional | SQL 6 , Python 6 | `{'my_tag': 'my_value'}` | | compression | Set the compression algorithm. | Optional | SQL, Python | `zstd` | | skip\_optimize7 | Skip the post-materialization `OPTIMIZE` operation for this model while keeping `zorder` / `liquid_clustered_by` / `auto_liquid_cluster` in the table definition. Available since dbt-databricks 1.12.2. | Optional | SQL, Python | `skip_optimize: true` | 1 `use_uniform` applies to dbt v2 only. `dbt-databricks` doesn't support this config yet — the adapter logs a warning and ignores the value. In `dbt-databricks`, use the [`use_managed_iceberg`](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-managed-iceberg) behavior flag instead. 2 When `table_format` is `iceberg`, `file_format` must be `delta`. This requirement applies to `dbt-databricks` only. In dbt v2, managed Iceberg tables use `parquet`. 3 When `liquid_clustered_by` is enabled, dbt-databricks issues an `OPTIMIZE` (Liquid Clustering) operation after each run. To disable this behavior, set the variable `DATABRICKS_SKIP_OPTIMIZE=true`, which can be passed into the dbt run command (`dbt run --vars "{'databricks_skip_optimize': true}"`) or set as an environment variable. See [issue #802](https://github.com/databricks/dbt-databricks/issues/802). 4 Do not use `liquid_clustered_by` and `auto_liquid_cluster` on the same model. 5 We do not yet have a PySpark API to set tblproperties at table creation, so this feature is primarily to allow users to annotate their python-derived tables with tblproperties. 6 `databricks_tags` are applied via `ALTER` statements. Tags cannot be removed via dbt-databricks once applied. To remove tags, use Databricks directly or a post-hook. Starting in `dbt-databricks` v1.12, `databricks_tags` set at multiple config hierarchy levels [merge additively](#databricks_tags) instead of the lower (more specific) level fully replacing the higher one. 7 `skip_optimize` gives you per-model control over the post-materialization `OPTIMIZE` call. Because it's a standard dbt model config, you can also set it at the folder or project level through config inheritance. The run-wide `DATABRICKS_SKIP_OPTIMIZE` variable takes precedence over `skip_optimize`; if you set `DATABRICKS_SKIP_OPTIMIZE=true` (or `databricks_skip_optimize: true`), the variable skips `OPTIMIZE` for every model, and you can't re-enable it for an individual model with `skip_optimize: false`. Use `skip_optimize` when you want to keep `OPTIMIZE` on for most models but opt specific ones out — for example, if you delegate `OPTIMIZE` to [Predictive Optimization](https://docs.databricks.com/en/optimizations/predictive-optimization.html) or schedule it out of band. Refer to [issue #703](https://github.com/databricks/dbt-databricks/issues/703). In dbt-databricks v1.10, there are several new model configurations options gated behind the `use_materialization_v2` flag. For details, see the [documentation of Databricks behavior flags](https://docs.getdbt.com/reference/global-configs/databricks-changes.md). ##### Python submission methods *Available in versions 1.9 or higher* In dbt-databricks v1.9 (try it now in [the dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md)), you can use these four options for `submission_method`: * `all_purpose_cluster`: Executes the python model either directly using the [command api](https://docs.databricks.com/api/workspace/commandexecution) or by uploading a notebook and creating a one-off job run * `job_cluster`: Creates a new job cluster to execute an uploaded notebook as a one-off job run * `serverless_cluster`: Uses a [serverless cluster](https://docs.databricks.com/en/jobs/run-serverless-jobs.html) to execute an uploaded notebook as a one-off job run * `workflow_job`: Creates/updates a reusable workflow and uploaded notebook, for execution on all-purpose, job, or serverless clusters. caution This approach gives you maximum flexibility, but will create persistent artifacts in Databricks (the workflow) that users could run outside of dbt. We are currently in a transitionary period where there is a disconnect between old submission methods (which were grouped by compute), and the logically distinct submission methods (command, job run, workflow). As such, the supported config matrix is somewhat complicated: | Config | Use | Default | `all_purpose_cluster`\* | `job_cluster` | `serverless_cluster` | `workflow_job` | | --------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------- | ------------- | -------------------- | -------------- | | `create_notebook` | if false, use Command API, otherwise upload notebook and use job run | `false` | ✅ | ❌ | ❌ | ❌ | | `timeout` | maximum time to wait for command/job to run | `0` (No timeout) | ✅ | ✅ | ✅ | ✅ | | `job_cluster_config` | configures a [new cluster](https://docs.databricks.com/api/workspace/jobs/submit#tasks-new_cluster) for running the model | `{}` | ❌ | ✅ | ❌ | ✅ | | `access_control_list` | directly configures [access control](https://docs.databricks.com/api/workspace/jobs/submit#access_control_list) for the job | `{}` | ✅ | ✅ | ✅ | ✅ | | `packages` | list of packages to install on the executing cluster | `[]` | ✅ | ✅ | ✅ | ✅ | | `index_url` | url to install `packages` from | `None` (uses pypi) | ✅ | ✅ | ✅ | ✅ | | `additional_libs` | directly configures [libraries](https://docs.databricks.com/api/workspace/jobs/submit#tasks-libraries) | `[]` | ✅ | ✅ | ✅ | ✅ | | `python_job_config` | additional configuration for jobs/workflows (see table below) | `{}` | ✅ | ✅ | ✅ | ✅ | | `cluster_id` | id of existing all purpose cluster to execute against | `None` | ✅ | ❌ | ❌ | ✅ | | `http_path` | path to existing all purpose cluster to execute against | `None` | ✅ | ❌ | ❌ | ❌ | \* Only `timeout` and `cluster_id`/`http_path` are supported when `create_notebook` is false With the introduction of the `workflow_job` submission method, we chose to segregate further configuration of the python model submission under a top level configuration named `python_job_config`. This keeps configuration options for jobs and workflows namespaced in such a way that they do not interfere with other model config, allowing us to be much more flexible with what is supported for job execution. The support matrix for this feature is divided into `workflow_job` and all others (assuming `all_purpose_cluster` with `create_notebook`==true). Each config option listed must be nested under `python_job_config`: | Config | Use | Default | `workflow_job` | All others | | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------- | -------------- | ---------- | | `name` | The name to give (or used to look up) the created workflow | `None` | ✅ | ❌ | | `grants` | A simplified way to specify access control for the workflow | `{}` | ✅ | ✅ | | `existing_job_id` | Id to use to look up the created workflow (in place of `name`) | `None` | ✅ | ❌ | | `post_hook_tasks` | [Tasks](https://docs.databricks.com/api/workspace/jobs/create#tasks) to include after the model notebook execution | `[]` | ✅ | ❌ | | `additional_task_settings` | Additional [task config](https://docs.databricks.com/api/workspace/jobs/create#tasks) to include in the model task | `{}` | ✅ | ❌ | | [Other job run settings](https://docs.databricks.com/api/workspace/jobs/submit) | Config will be copied into the request, outside of the model task | `None` | ❌ | ✅ | | [Other workflow settings](https://docs.databricks.com/api/workspace/jobs/create) | Config will be copied into the request, outside of the model task | `None` | ✅ | ❌ | This example uses the new configuration options in the previous table: schema.yml ```yaml models: - name: my_model config: submission_method: workflow_job # Define a job cluster to create for running this workflow # Alternately, could specify cluster_id to use an existing cluster, or provide neither to use a serverless cluster job_cluster_config: spark_version: "15.3.x-scala2.12" node_type_id: "rd-fleet.2xlarge" runtime_engine: "{{ var('job_cluster_defaults.runtime_engine') }}" data_security_mode: "{{ var('job_cluster_defaults.data_security_mode') }}" autoscale: { "min_workers": 1, "max_workers": 4 } python_job_config: # These settings are passed in, as is, to the request email_notifications: { on_failure: ["me@example.com"] } max_retries: 2 name: my_workflow_name # Override settings for your model's dbt task. For instance, you can # change the task key additional_task_settings: { "task_key": "my_dbt_task" } # Define tasks to run before/after the model # This example assumes you have already uploaded a notebook to /my_notebook_path to perform optimize and vacuum post_hook_tasks: [ { "depends_on": [{ "task_key": "my_dbt_task" }], "task_key": "OPTIMIZE_AND_VACUUM", "notebook_task": { "notebook_path": "/my_notebook_path", "source": "WORKSPACE" }, }, ] # Simplified structure, rather than having to specify permission separately for each user grants: view: [{ "group_name": "marketing-team" }] run: [{ "user_name": "other_user@example.com" }] manage: [] ``` #### Configuring columns *Available in versions 1.10 or higher* When materializing models of various types, you may include several optional column-level configs that are specific to the dbt-databricks plugin, in addition to the standard [column configs](https://docs.getdbt.com/reference/resource-properties/columns.md). Support for column tags and column masks were added in dbt-databricks v1.10.4. | Option | Description | Required? | Model support | Materialization support | Example | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------- | ------------------------------------------------------ | ------------------------------------------------- | | databricks\_tags | [Tags](https://docs.databricks.com/en/data-governance/unity-catalog/tags.html) to be set on individual columns | Optional | SQL†, Python† | Table, Incremental, Materialized View, Streaming Table | `{'data_classification': 'pii'}` | | column\_mask | [Column mask](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-column-mask) configuration for dynamic data masking. Accepts `function` and optional `using_columns` properties\* | Optional | SQL, Python | Table, Incremental, Streaming Table | `{'function': 'my_catalog.my_schema.mask_email'}` | \* `using_columns` supports all parameter types listed in [Databricks column mask parameters](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-column-mask#parameters). † `databricks_tags` are applied via `ALTER` statements. Tags cannot be removed via dbt-databricks once applied. To remove tags, use Databricks directly or a post-hook. Starting in `dbt-databricks` v1.12, `databricks_tags` set at multiple config hierarchy levels [merge additively](#databricks_tags) instead of the lower (more specific) level fully replacing the higher one. This example uses the column-level configurations in the previous table: schema.yml ```yaml models: - name: customers columns: - name: customer_id databricks_tags: data_classification: "public" - name: email databricks_tags: data_classification: "pii" column_mask: function: my_catalog.my_schema.mask_email using_columns: "customer_id, 'literal string'" ``` #### Setting row filters *Available in versions 1.12 or higher* You can set `row_filter` to apply a [Unity Catalog row filter](https://docs.databricks.com/aws/en/tables/row-and-column-filters) to a model, restricting which rows a query returns based on a SQL UDF. dbt applies the filter with a `WITH ROW FILTER` clause when it creates the relation, and emits `ALTER ... SET ROW FILTER` / `ALTER ... DROP ROW FILTER` to add, update, or remove the filter on subsequent runs. `row_filter` is an optional model-level config. When you set it, both of the following properties are required: | Property | Description | Required? | Example | | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | --------------- | | function | The row-filter UDF to apply. Provide either an unqualified name (dbt qualifies it with the model's catalog and schema) or a fully qualified `catalog.schema.function`. dbt rejects a two-part schema.function name as ambiguous. | Yes | `region_filter` | | columns | The columns passed as arguments to the filter function. Can be a single string or a list. Required when `function` is set. | Yes | `[region]` | Row filters are supported on the `table`, `incremental`, `materialized_view`, and `streaming_table` materializations. They are *not* supported on regular views or on Hive Metastore relations. Configuring `row_filter` on either raises a compiler error. This example applies a row filter to a model: schema.yml ```yaml models: - name: orders config: row_filter: function: my_catalog.my_schema.region_filter columns: [region] ``` #### Incremental models *Available in versions 1.9 or higher* Breaking change in v1.11.0 dbt-databricks v1.11.0 requires Databricks Runtime 12.2 LTS or higher for incremental models This version introduces a fix for column order mismatches in incremental models by using Databricks' `INSERT BY NAME` syntax (available since DBR 12.2). This prevents data corruption that could occur when column order changed in models using `on_schema_change: sync_all_columns`. If you're using an older runtime: * Pin your `dbt-databricks` version to `1.10.x` * Or upgrade to DBR 12.2 LTS or higher This breaking change affects all incremental strategies: `append`, `insert_overwrite`, `replace_where`, `delete+insert`, and `merge` (via intermediate table creation). For more details on v1.11.0 changes, see the [dbt-databricks v1.11.0 changelog](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md). dbt-databricks plugin leans heavily on the [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-strategy.md). This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of six values: * `append`: Insert new records without updating or overwriting any existing data. * `insert_overwrite`: If `partition_by` is specified, overwrite partitions in the table with new data. If no `partition_by` is specified, overwrite the entire table with new data. * `merge`(default; Delta and Hudi file format only): Match records based on a `unique_key`, updating old records, and inserting new ones. (If no `unique_key` is specified, all new data is inserted, similar to `append`.) * `replace_where` (Delta file format only): Match records based on `incremental_predicates`, replacing all records that match the predicates from the existing table with records matching the predicates from the new data. (If no `incremental_predicates` are specified, all new data is inserted, similar to `append`.) * `delete+insert` (Delta file format only, available in v1.11+): Match records based on a required `unique_key`, delete matching records, and insert new records. Optionally filter using `incremental_predicates`. * `microbatch` (Delta file format only): Implements the [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) using `replace_where` with predicates generated based `event_time`. Each of these strategies has its pros and cons, which we'll discuss below. As with any model config, `incremental_strategy` may be specified in `dbt_project.yml` or within a model file's `config()` block. ##### The `append` strategy Following the `append` strategy, dbt will perform an `insert into` statement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Apache Spark versions. However, this strategy *cannot* update, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources. ##### Source code databricks\_incremental.sql ```sql {{ config( materialized='incremental', incremental_strategy='append', ) }} -- All rows returned by this query will be appended to the existing table select * from {{ ref('events') }} {% if is_incremental() %} where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ``` ##### Run code databricks\_incremental.sql ```sql create temporary view databricks_incremental__dbt_tmp as select * from analytics.events where event_ts >= (select max(event_ts) from {{ this }}) ; insert into table analytics.databricks_incremental select `date_day`, `users` from databricks_incremental__dbt_tmp ``` ##### The `insert_overwrite` strategy The `insert_overwrite` strategy updates data in a table by replacing existing records instead of just adding new ones. This strategy is most effective when specified alongside a `partition_by` or `liquid_clustered_by` clause in your model config, which helps identify the specific partitions or clusters affected by your query. dbt will run an [atomic `insert into ... replace on` statement](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-dml-insert-into#replace-on) that dynamically replaces all partitions/clusters included in your query, instead of rebuilding the entire table. **Important!** Be sure to re-select *all* of the relevant data for a partition or cluster when using this incremental strategy. When using `liquid_clustered_by`, the `replace on` keys used will be the same as the `liquid_clustered_by` keys (same as `partition_by` behavior). When you set [`use_replace_on_for_insert_overwrite`](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-replace-on-for-insert_overwrite-strategy) to `true` (in SQL warehouses or when using cluster computes) dbt dynamically overwrites partitions and only replaces the partitions or clusters returned by your model query. dbt runs a [partitionOverwriteMode='dynamic' `insert overwrite` statement](https://docs.databricks.com/aws/en/delta/selective-overwrite#dynamic-partition-overwrites-with-partitionoverwritemode-legacyl), which helps reduce unnecessary overwrites and improves performance. When you set [`use_replace_on_for_insert_overwrite`](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#use-replace-on-for-insert_overwrite-strategy) to `false` in SQL warehouses, dbt truncates (empties) the entire table before inserting new data. This replaces all rows in the table each time the model runs, which can increase run time and cost for large datasets If you don't specify `partition_by` or `liquid_clustered_by`, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` and `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead. ##### Source code databricks\_incremental.sql ```sql {{ config( materialized='incremental', partition_by=['date_day'], file_format='parquet' ) }} /* Every partition returned by this query will be overwritten when this model runs */ with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select date_day, count(*) as users from new_events group by 1 ``` ##### Run code databricks\_incremental.sql ```sql create temporary view databricks_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select date_day, count(*) as users from events group by 1 ; insert overwrite table analytics.databricks_incremental partition (date_day) select `date_day`, `users` from databricks_incremental__dbt_tmp ``` ##### The `merge` strategy The `merge` incremental strategy requires: * `file_format: delta or hudi` * Databricks Runtime 5.1 and above for delta file format * Apache Spark for hudi file format The Databricks adapter will run an [atomic `merge` statement](https://docs.databricks.com/spark/latest/spark-sql/language-manual/merge-into.html) similar to the default merge behavior on Snowflake and BigQuery. If a `unique_key` is specified (recommended), dbt will update old records with values from new records that match on the key column. If a `unique_key` is not specified, dbt will forgo match criteria and simply insert all new records (similar to `append` strategy). Specifying `merge` as the incremental strategy is optional since it's the default strategy used when none is specified. ##### Source code merge\_incremental.sql ```sql {{ config( materialized='incremental', file_format='delta', # or 'hudi' unique_key='user_id', incremental_strategy='merge' ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from events group by 1 ``` ##### Run code target/run/merge\_incremental.sql ```sql create temporary view merge_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select user_id, max(date_day) as last_seen from events group by 1 ; merge into analytics.merge_incremental as DBT_INTERNAL_DEST using merge_incremental__dbt_tmp as DBT_INTERNAL_SOURCE on DBT_INTERNAL_SOURCE.user_id = DBT_INTERNAL_DEST.user_id when matched then update set * when not matched then insert * ``` Beginning with 1.9, `merge` behavior can be modified with the following additional configuration options: * `target_alias`, `source_alias`: Aliases for the target and source to allow you to describe your merge conditions more naturally. These default to `DBT_INTERNAL_DEST` and `DBT_INTERNAL_SOURCE`, respectively. * `skip_matched_step`: If set to `true`, the 'matched' clause of the merge statement will not be included. * `skip_not_matched_step`: If set to `true`, the 'not matched' clause will not be included. * `matched_condition`: Condition to apply to the `WHEN MATCHED` clause. You should use the `target_alias` and `source_alias` to write a conditional expression, such as `DBT_INTERNAL_DEST.col1 = hash(DBT_INTERNAL_SOURCE.col2, DBT_INTERNAL_SOURCE.col3)`. This condition further restricts the matched set of rows. * `not_matched_condition`: Condition to apply to the `WHEN NOT MATCHED [BY TARGET]` clause. This condition further restricts the set of rows in the target that do not match the source that will be inserted into the merged table. * `not_matched_by_source_condition`: Condition to apply to the further filter `WHEN NOT MATCHED BY SOURCE` clause. Only used in conjunction with `not_matched_by_source_action`. * `not_matched_by_source_action`: The action to apply when the condition is met. Configure as an expression. For example: `not_matched_by_source_action: "update set t.attr1 = 'deleted', t.tech_change_ts = current_timestamp()"`. * `merge_with_schema_evolution`: If set to `true`, the merge statement includes the `WITH SCHEMA EVOLUTION` clause. For more details on the meaning of each merge clause, please see [the Databricks documentation](https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html). The following is an example demonstrating the use of these new options: ##### Source code merge\_incremental\_options.sql ```sql {{ config( materialized = 'incremental', unique_key = 'id', incremental_strategy='merge', target_alias='t', source_alias='s', matched_condition='t.tech_change_ts < s.tech_change_ts', not_matched_condition='s.attr1 IS NOT NULL', not_matched_by_source_condition='t.tech_change_ts < current_timestamp()', not_matched_by_source_action='delete', merge_with_schema_evolution=true ) }} select id, attr1, attr2, tech_change_ts from {{ ref('source_table') }} as s ``` ##### Run code target/run/merge\_incremental\_options.sql ```sql create temporary view merge_incremental__dbt_tmp as select id, attr1, attr2, tech_change_ts from upstream.source_table ; merge with schema evolution into target_table as t using ( select id, attr1, attr2, tech_change_ts from source_table as s ) on t.id <=> s.id when matched and t.tech_change_ts < s.tech_change_ts then update set id = s.id, attr1 = s.attr1, attr2 = s.attr2, tech_change_ts = s.tech_change_ts when not matched and s.attr1 IS NOT NULL then insert ( id, attr1, attr2, tech_change_ts ) values ( s.id, s.attr1, s.attr2, s.tech_change_ts ) when not matched by source and t.tech_change_ts < current_timestamp() then delete ``` ##### The `replace_where` strategy The `replace_where` incremental strategy requires: * `file_format: delta` * Databricks Runtime 12.0 and above dbt will run an [atomic `replace where` statement](https://docs.databricks.com/en/delta/selective-overwrite.html#arbitrary-selective-overwrite-with-replacewhere) which selectively overwrites data matching one or more `incremental_predicates` specified as a string or array. Only rows matching the predicates will be inserted. If no `incremental_predicates` are specified, dbt will perform an atomic insert, as with `append`. caution `replace_where` inserts data into columns in the order provided, rather than by column name. If you reorder columns and the data is compatible with the existing schema, you may silently insert values into an unexpected column. If the incoming data is incompatible with the existing schema, you will instead receive an error. ##### Source code replace\_where\_incremental.sql ```sql {{ config( materialized='incremental', file_format='delta', incremental_strategy = 'replace_where' incremental_predicates = 'user_id >= 10000' # Never replace users with ids < 10000 ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from events group by 1 ``` ##### Run code target/run/replace\_where\_incremental.sql ```sql create temporary view replace_where__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select user_id, max(date_day) as last_seen from events group by 1 ; insert into analytics.replace_where_incremental replace where user_id >= 10000 table `replace_where__dbt_tmp` ``` ##### The `delete+insert` strategy *Available in versions 1.11 or higher* The `delete+insert` incremental strategy requires: * `file_format: delta` * A required `unique_key` configuration * Databricks Runtime 12.2 LTS or higher The `delete+insert` strategy is a simpler alternative to the `merge` strategy for cases where you want to replace matching records without the complexity of updating specific columns. This strategy works in two steps: 1. **Delete**: Remove all rows from the target table where the `unique_key` matches rows in the new data. 2. **Insert**: Insert all new rows from the staging data. This strategy is particularly useful when: * You want to replace entire records rather than update specific columns * Your business logic requires a clean "remove and replace" approach * You need a simpler incremental strategy than `merge` for full record replacement When using Databricks Runtime 17.1 or higher, dbt uses the efficient [`INSERT INTO ... REPLACE ON` syntax](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-dml-insert-into#replace-on) to perform this operation atomically. For older runtime versions, dbt executes separate `DELETE` and `INSERT` statements. You can optionally use `incremental_predicates` to further filter which records are processed, providing more control over which rows are deleted and inserted. ##### Source code delete\_insert\_incremental.sql ```sql {{ config( materialized='incremental', file_format='delta', incremental_strategy='delete+insert', unique_key='user_id', incremental_predicates='user_id >= 10000' # Never delete/insert users with ids < 10000 ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from new_events group by 1 ``` ##### Run code (DBR 17.1+) target/run/delete\_insert\_incremental.sql ```sql create temporary view delete_insert_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select user_id, max(date_day) as last_seen from new_events group by 1 ; insert into table analytics.delete_insert_incremental as target replace on (target.user_id <=> temp.user_id) (select `user_id`, `last_seen` from delete_insert_incremental__dbt_tmp where user_id >= 10000) as temp ``` ##### Run code (DBR < 17.1) target/run/delete\_insert\_incremental.sql ```sql create temporary view delete_insert_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select user_id, max(date_day) as last_seen from new_events group by 1 ; -- Step 1: Delete matching rows delete from analytics.delete_insert_incremental where analytics.delete_insert_incremental.user_id IN (SELECT user_id FROM delete_insert_incremental__dbt_tmp) and user_id >= 10000; -- Step 2: Insert new rows insert into analytics.delete_insert_incremental by name select `user_id`, `last_seen` from delete_insert_incremental__dbt_tmp where user_id >= 10000 ``` ##### The `microbatch` strategy *Available in versions 1.9 or higher* The Databricks adapter implements the `microbatch` strategy using `replace_where`. Note the requirements and caution statements for `replace_where` above. For more information about this strategy, see the [microbatch reference page](https://docs.getdbt.com/docs/build/incremental-microbatch.md). In the following example, the upstream table `events` have been annotated with an `event_time` column called `ts` in its schema file. ##### Source code microbatch\_incremental.sql ```sql {{ config( materialized='incremental', file_format='delta', incremental_strategy = 'microbatch' event_time='date' # Use 'date' as the grain for this microbatch table ) }} with new_events as ( select * from {{ ref('events') }} ) select user_id, date, count(*) as visits from events group by 1, 2 ``` ##### Run code target/run/replace\_where\_incremental.sql ```sql create temporary view replace_where__dbt_tmp as with new_events as ( select * from (select * from analytics.events where ts >= '2024-10-01' and ts < '2024-10-02') ) select user_id, date, count(*) as visits from events group by 1, 2 ; insert into analytics.replace_where_incremental replace where CAST(date as TIMESTAMP) >= '2024-10-01' and CAST(date as TIMESTAMP) < '2024-10-02' table `replace_where__dbt_tmp` ``` #### Python model configuration The Databricks adapter supports Python models. Databricks uses PySpark as the processing framework for these models. **Submission methods:** Databricks supports a few different mechanisms to submit PySpark code, each with relative advantages. Some are better for supporting iterative development, while others are better for supporting lower-cost production deployments. The options are: * `all_purpose_cluster` (default): dbt will run your Python model using the cluster ID configured as `cluster` in your connection profile or for this specific model. These clusters are more expensive but also much more responsive. We recommend using an interactive all-purpose cluster for quicker iteration in development. * `create_notebook: True`: dbt will upload your model's compiled PySpark code to a notebook in the namespace `/Shared/dbt_python_model/{schema}`, where `{schema}` is the configured schema for the model, and execute that notebook to run using the all-purpose cluster. The appeal of this approach is that you can easily open the notebook in the Databricks UI for debugging or fine-tuning right after running your model. Remember to copy any changes into your dbt `.py` model code before re-running. * `create_notebook: False` (default): dbt will use the [Command API](https://docs.databricks.com/dev-tools/api/1.2/index.html#run-a-command), which is slightly faster. * `job_cluster`: dbt will upload your model's compiled PySpark code to a notebook in the namespace `/Shared/dbt_python_model/{schema}`, where `{schema}` is the configured schema for the model, and execute that notebook to run using a short-lived jobs cluster. For each Python model, Databricks will need to spin up the cluster, execute the model's PySpark transformation, and then spin down the cluster. As such, job clusters take longer before and after model execution, but they're also less expensive, so we recommend these for longer-running Python models in production. To use the `job_cluster` submission method, your model must be configured with `job_cluster_config`, which defines key-value properties for `new_cluster`, as defined in the [JobRunsSubmit API](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsSubmit). You can configure each model's `submission_method` in all the standard ways you supply configuration: ```python def model(dbt, session): dbt.config( submission_method="all_purpose_cluster", create_notebook=True, cluster_id="abcd-1234-wxyz" ) ... ``` ```yml models: - name: my_python_model config: submission_method: job_cluster job_cluster_config: spark_version: ... node_type_id: ... ``` ```yml # dbt_project.yml models: project_name: subfolder: # set defaults for all .py models defined in this subfolder +submission_method: all_purpose_cluster +create_notebook: False +cluster_id: abcd-1234-wxyz ``` If not configured, `dbt-spark` will use the built-in defaults: the all-purpose cluster (based on `cluster` in your connection profile) without creating a notebook. The `dbt-databricks` adapter will default to the cluster configured in `http_path`. We encourage explicitly configuring the clusters for Python models in Databricks projects. **Installing packages:** When using all-purpose clusters, we recommend installing packages which you will be using to run your Python models. **Related docs:** * [PySpark DataFrame syntax](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.html) * [Databricks: Introduction to DataFrames - Python](https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html) #### Selecting compute per model Beginning in version 1.7.2, you can assign which compute resource to use on a per-model basis. For SQL models, you can select a SQL Warehouse (serverless or provisioned) or an all purpose cluster. For details on how this feature interacts with python models, see [Specifying compute for Python models](#specifying-compute-for-python-models). note This is an optional setting. If you do not configure this as shown below, we will default to the compute specified by http\_path in the top level of the output section in your profile. This is also the compute that will be used for tasks not associated with a particular model, such as gathering metadata for all tables in a schema. To take advantage of this capability, you will need to add compute blocks to your profile: profiles.yml ```yaml profile-name: target: target-name # this is the default target outputs: target-name: type: databricks catalog: optional catalog name if you are using Unity Catalog schema: schema name # Required host: yourorg.databrickshost.com # Required ### This path is used as the default compute http_path: /sql/your/http/path # Required ### New compute section compute: ### Name that you will use to refer to an alternate compute Compute1: http_path: '/sql/your/http/path' # Required of each alternate compute ### A third named compute, use whatever name you like Compute2: http_path: '/some/other/path' # Required of each alternate compute ... target-name: # additional targets ... ### For each target, you need to define the same compute, ### but you can specify different paths compute: ### Name that you will use to refer to an alternate compute Compute1: http_path: '/sql/your/http/path' # Required of each alternate compute ### A third named compute, use whatever name you like Compute2: http_path: '/some/other/path' # Required of each alternate compute ... ``` The new compute section is a map of user chosen names to objects with an http\_path property. Each compute is keyed by a name which is used in the model definition/configuration to indicate which compute you wish to use for that model/selection of models. We recommend choosing a name that is easily recognized as the compute resources you're using, such as the name of the compute resource inside the Databricks UI. note You need to use the same set of names for compute across your outputs, though you may supply different http\_paths, allowing you to use different computes in different deployment scenarios. To configure this inside of dbt, use the [extended attributes feature](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the desired environments: ```yaml compute: Compute1: http_path: /SOME/OTHER/PATH Compute2: http_path: /SOME/OTHER/PATH ``` ##### Specifying the compute for models As with many other configuration options, you can specify the compute for a model in multiple ways, using `databricks_compute`. In your `dbt_project.yml`, the selected compute can be specified for all the models in a given directory: dbt\_project.yml ```yaml ... models: +databricks_compute: "Compute1" # use the `Compute1` warehouse/cluster for all models in the project... my_project: clickstream: +databricks_compute: "Compute2" # ...except for the models in the `clickstream` folder, which will use `Compute2`. snapshots: +databricks_compute: "Compute1" # all Snapshot models are configured to use `Compute1`. ``` For an individual model the compute can be specified in the model config in your schema file. schema.yml ```yaml models: - name: table_model config: databricks_compute: Compute1 columns: - name: id data_type: int ``` Alternatively the warehouse can be specified in the config of a model's SQL file. model.sql ```sql {{ config( materialized='table', databricks_compute='Compute1' ) }} select * from {{ ref('seed') }} ``` To validate that the specified compute is being used, look for lines in your dbt.log like: ```text Databricks adapter ... using default compute resource. ``` or ```text Databricks adapter ... using compute resource <name of compute>. ``` ##### Specifying compute for Python models Materializing a python model requires execution of SQL as well as python. Specifically, if your python model is incremental, the current execution pattern involves executing python to create a staging table that is then merged into your target table using SQL. The python code needs to run on an all purpose cluster (or serverless cluster, see [Python Submission Methods](#python-submission-methods)), while the SQL code can run on an all purpose cluster or a SQL Warehouse. When you specify your `databricks_compute` for a python model, you are currently only specifying which compute to use when running the model-specific SQL. If you wish to use a different compute for executing the python itself, you must specify an alternate compute in the config for the model. For example: model.py ```python def model(dbt, session): dbt.config( http_path="sql/protocolv1/..." ) ``` If your default compute is a SQL Warehouse, you will need to specify an all purpose cluster `http_path` in this way. #### Persisting model descriptions Relation-level docs persistence is supported. For more information on configuring docs persistence, see [the docs](https://docs.getdbt.com/reference/resource-configs/persist_docs.md). When the `persist_docs` option is configured appropriately, you'll be able to see model descriptions in the `Comment` field of `describe [table] extended` or `show table extended in [database] like '*'`. #### Query tags *Available in versions 1.11 or higher* [Query tags](https://docs.databricks.com/aws/en/sql/user/queries/query-tags) are a Databricks feature that allows you to attach custom key-value metadata to SQL queries. This metadata appears in system tables and query history, making it useful for tracking query costs, debugging, and auditing. Feature availability Query tags may not yet be available in all Databricks workspaces. Check the [Databricks documentation](https://docs.databricks.com/aws/en/sql/user/queries/query-tags) for the latest information on feature availability. dbt-databricks supports setting query tags at both the connection level (in your profile) and the model level (in model configs). When you run dbt, it automatically includes default tags containing dbt metadata, such as the model name and dbt version. ##### Default query tags dbt-databricks automatically adds the following tags to every query: | Tag key | Description | | -------------------------- | --------------------------------------------------------------- | | `@@dbt_model_name` | The name of the model being executed | | `@@dbt_core_version` | The version of dbt being used | | `@@dbt_databricks_version` | The version of dbt-databricks being used | | `@@dbt_materialized` | The materialization type (table, view, incremental, and so on.) | These reserved keys cannot be overridden by user-defined tags. ##### Configuring query tags You can set query tags at the connection level in your profile or at the model level in your model config. Model-level tags take precedence over connection-level tags. ###### Connection-level query tags To set query tags for all queries in a connection, add the `query_tags` parameter to your `profiles.yml` file as a JSON string: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: databricks catalog: my_catalog schema: my_schema host: yourorg.databrickshost.com http_path: /sql/your/http/path token: dapiXXXXXXXXXXXXXXXXXXXXXXX query_tags: '{"team": "analytics", "project": "customer_360"}' ``` ###### Model-level query tags To set query tags for a specific model, use the `query_tags` config: models/my\_model.sql ```sql {{ config( query_tags = {'cost_center': 'marketing', 'priority': 'high'} ) }} select * from {{ ref('upstream_model') }} ``` You can also configure query tags in your `dbt_project.yml` for groups of models: dbt\_project.yml ```yaml models: my_project: marketing: +query_tags: {'department': 'marketing'} finance: +query_tags: {'department': 'finance'} ``` ##### Tag precedence and merging When query tags are defined at multiple levels, they are merged with the following precedence (highest to lowest): 1. Model-level tags (from `config()` or schema.yml) 2. Connection-level tags (from `profiles.yml`) 3. Default dbt tags (automatically added) If the same key appears at multiple levels, the higher-precedence value wins. Why connection-level tags? Due to how dbt merges configs, specifying `query_tags` at the model level in `config()` or `schema.yml` will **replace** any `query_tags` you defined in `dbt_project.yml` rather than merging them. This is standard dbt behavior for dictionary configs. To work around this limitation, dbt-databricks accepts `query_tags` in your connection profile (`profiles.yml`). Connection-level tags are always merged with model-level tags, allowing you to define common tags once in your profile and selectively add or override specific keys at the model level. **Recommended pattern:** * Define shared tags (team, project, environment) in your profile's `query_tags` * Use model-level `query_tags` when you need to add model-specific tags ##### Limitations * **Maximum 20 tags**: The total number of query tags (including default tags) cannot exceed 20. * **Value length**: Tag values must be at most 128 characters. Default tag values that exceed this limit are automatically truncated. * **Special characters**: Backslash (`\`), comma (`,`), and colon (`:`) characters in tag values are automatically escaped. A warning is logged when escaping occurs. * **Reserved keys**: The keys `@@dbt_model_name`, `@@dbt_core_version`, `@@dbt_databricks_version`, and `@@dbt_materialized` are reserved and cannot be used in user-defined tags. ##### Viewing query tags Query tags appear in Databricks system tables and query history. For information on how to query and analyze query tags, see the [Databricks query tags documentation](https://docs.databricks.com/aws/en/sql/user/queries/query-tags). #### Default file format configurations To access advanced incremental strategies features, such as [snapshots](https://docs.getdbt.com/reference/commands/snapshot.md) and the `merge` incremental strategy, you will want to use the Delta or Hudi file format as the default file format when materializing models as tables. It's quite convenient to do this by setting a top-level configuration in your project file: dbt\_project.yml ```yml models: +file_format: delta # or hudi seeds: +file_format: delta # or hudi snapshots: +file_format: delta # or hudi ``` #### Materialized views and streaming tables [Materialized views](https://docs.databricks.com/en/sql/user/materialized-views.html) and [streaming tables](https://docs.databricks.com/en/sql/load-data-streaming-table.html) are alternatives to incremental tables that are powered by [Delta Live Tables](https://docs.databricks.com/en/delta-live-tables/index.html). Refer to [What are Delta Live Tables?](https://docs.databricks.com/en/delta-live-tables/index.html#what-are-delta-live-tables-datasets) for more information and use cases. In order to adopt these materialization strategies, you will need a workspace that is enabled for Unity Catalog and serverless SQL Warehouses. materialized\_view.sql ```sql {{ config( materialized = 'materialized_view' ) }} ``` or streaming\_table.sql ```sql {{ config( materialized = 'streaming_table' ) }} ``` We support [on\_configuration\_change](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) for most available properties of these materializations. The following table summarizes our configuration support. Refer to [Configuration details](#configuration-details) for more details on each config: | Databricks Concept | Config Name | MV/ST support | Version | | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------- | ------- | | [PARTITIONED BY](https://docs.databricks.com/en/sql/language-manual/sql-ref-partition.html#partitioned-by) | `partition_by` | MV/ST | All | | [CLUSTER BY](https://docs.databricks.com/en/delta/clustering.html) | `liquid_clustered_by` | MV/ST | v1.11+ | | COMMENT | [`description`](https://docs.getdbt.com/reference/resource-properties/description.md) | MV/ST | All | | [TBLPROPERTIES](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-tblproperties.html#tblproperties) | `tblproperties` | MV/ST | All | | [TAGS](https://docs.databricks.com/en/data-governance/unity-catalog/tags.html) | `databricks_tags` | MV/ST | v1.11+ | | [SCHEDULE CRON](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view.html#parameters) | `schedule: { 'cron': '\<cron schedule\>', 'time_zone_value': '\<time zone value\>' }` | MV/ST | All | | [SCHEDULE EVERY](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view#parameters) | `schedule: { 'every': '\<n\> \<unit\>' }` | MV/ST | v1.12+ | | [TRIGGER ON UPDATE](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view#parameters) | `schedule: { 'on_update': true, 'at_most_every': '\<n\> \<unit\>' }` | MV/ST | v1.12+ | | [WITH ROW FILTER](https://docs.databricks.com/aws/en/tables/row-and-column-filters) | `row_filter` | MV/ST | v1.12+ | | query | defined by your model SQL | on\_configuration\_change for MV only | All | mv\_example.sql ```sql {{ config( materialized='materialized_view', partition_by='id', schedule = { 'cron': '0 0 * * * ? *', 'time_zone_value': 'Etc/UTC' }, tblproperties={ 'key': 'value' }, ) }} select * from {{ ref('my_seed') }} ``` ##### Configuration details ###### partition\_by `partition_by` works the same as for views and tables, i.e. can be a single column, or an array of columns to partition by. ###### liquid\_clustered\_by *Available in versions 1.11 or higher* `liquid_clustered_by` enables [liquid clustering](https://docs.databricks.com/en/delta/clustering.html) for materialized views and streaming tables. Liquid clustering optimizes query performance by co-locating similar data within the same files, particularly beneficial for queries with selective filters on the clustered columns. **Note:** You cannot use both `partition_by` and `liquid_clustered_by` on the same materialization, as Databricks doesn't allow combining these features. ###### databricks\_tags *Available in versions 1.11 or higher* `databricks_tags` allows you to apply [Unity Catalog tags](https://docs.databricks.com/en/data-governance/unity-catalog/tags.html) to your materialized views and streaming tables for data governance and organization. Tags are key-value pairs that can be used for data classification, access control policies, and metadata management. ```sql {{ config( materialized='streaming_table', databricks_tags={'pii': 'contains_email', 'team': 'analytics'} ) }} ``` `dbt-databricks` v1.12+ adds support for key-only tags. To set a tag that has a key but no value, set the tag's value to an empty string `''` or to `None`: ```sql {{ config( materialized='streaming_table', databricks_tags={'sensitive': '', 'reviewed': None} ) }} ``` This applies to both table-level and column-level `databricks_tags`. Non-string values, such as numbers or booleans, are converted to strings. Tags are applied via `ALTER` statements after the materialization is created. Once applied, tags cannot be removed through dbt-databricks configuration changes. To remove tags, you must use Databricks directly or a post-hook. Behavior change in v1.12 Starting in `dbt-databricks` v1.12.0, `databricks_tags` configurations are merged additively across config hierarchy levels (for example, project-level and model-level), rather than having lower-level configs completely replace higher-level ones. When the same tag key is defined at multiple levels, the lower-level value takes precedence. Tag keys defined only at higher levels are retained. This behavior applies anywhere `databricks_tags` can be configured, including tables, columns, materialized views, and streaming tables. For example, with the following project-level and model-level configs: dbt\_project.yml ```yaml models: my_project: +databricks_tags: a: "b" c: "project_value" ``` models/my\_model.sql ```sql {{ config( databricks_tags={'c': 'model_value', 'k': 'v'} ) }} ``` The resulting tags are: * `a: b` — retained from the project level * `c: model_value` — the model-level value overrides the project-level `c` * `k: v` — added at the model level ###### description As with views and tables, adding a `description` to your configuration will lead to a table-level comment getting added to your materialization. ###### tblproperties `tblproperties` works the same as for views and tables with an important exception: the adapter maintains a list of keys that are set by Databricks when making an materialized view or streaming table which are ignored for the purpose of determining configuration changes. ###### schedule Set the refresh schedule for the model using one of three mutually exclusive modes: | Mode | Config | Format | Version | | ----------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `cron` | `schedule: { 'cron': '...', 'time_zone_value': '...' }` | Cron string ([Databricks format](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view#parameters)). `time_zone_value` is optional. | All | | `every` | `schedule: { 'every': '<n> <unit>' }` | `'<n> <unit>'` where unit is `HOURS`, `DAYS`, or `WEEKS` — for example, `'2 HOURS'` | v1.12+ | | `on_update` | `schedule: { 'on_update': true, 'at_most_every': '<n> <unit>' }` | Set to `true` to refresh when upstream data changes. `at_most_every` is optional and rate-limits refreshes (minimum 60 seconds). For example, `'15 MINUTES'` | v1.12+ | **Refresh behavior by mode:** * `cron`: dbt requests a manual refresh on every run. * `every` and `on_update`: Databricks auto-manages the refresh; dbt does not trigger a manual refresh on a no-op re-run. If a schedule exists in Databricks but your dbt project doesn't specify one, the schedule resets to manual on your next run (when `on_configuration_change` is set to `apply`). ###### query For materialized views, if the compiled query differs from what's in the database, dbt takes the configured `on_configuration_change` action. Query changes aren't currently detectable for streaming tables. Refer to [on\_configuration\_change](#on_configuration_change) for details. ###### row\_filter *Available in versions 1.12 or higher* `row_filter` applies a [Unity Catalog row filter](https://docs.databricks.com/aws/en/tables/row-and-column-filters) to a model. It is supported on `table`, `incremental`, `materialized_view`, and `streaming_table` materializations. Refer to [Setting row filters](#setting-row-filters) for the full config reference and examples. ##### on\_configuration\_change | Materialization | Drop and recreate required? | Notes | | ------------------ | -------------------------------------------- | ---------------------------------------------------------------------------------------- | | Materialized views | Yes, for all changes except schedule updates | Databricks SQL API limitation | | Streaming tables | Only when `partition_by` changes | All other supported changes use `CREATE OR REFRESH` plus an `ALTER` for schedule changes | Note on streaming table query changes: there's currently no way for the adapter to detect if a streaming table query has changed. Regardless of `on_configuration_change` behavior, dbt uses `CREATE OR REFRESH`, which applies the updated query to future rows only — previously processed rows aren't reprocessed. To reprocess available source data with an updated query, run with `--full-refresh`. (Applies to dbt v1.12 and later) #### Metric views Set `materialized='metric_view'` to manage a [Unity Catalog metric view](https://docs.databricks.com/aws/en/metric-views/) with dbt. Instead of SQL, the body of the model is the metric view's YAML definition: a `version`, a `source`, `dimensions`, `measures`, and an optional `filter`. dbt creates the metric view with `CREATE OR REPLACE VIEW ... WITH METRICS LANGUAGE YAML`. order\_metrics.sql ```sql {{ config(materialized='metric_view') }} version: 1.1 source: "{{ ref('source_orders') }}" filter: status = 'completed' dimensions: - name: order_date expr: order_date - name: status expr: status synonyms: [state, order_state] measures: - name: total_orders expr: count(1) - name: total_revenue expr: sum(revenue) synonyms: [revenue, sales] ``` Reference the source relation in `source` with `ref()` so dbt resolves dependencies. Query the resulting metric view with the `MEASURE()` function. dbt passes the YAML body through to Databricks unchanged, so a metric view supports the **entire** [Unity Catalog metric view YAML specification](https://docs.databricks.com/aws/en/business-semantics/metric-views/yaml-reference), not only the keys shown above. Any field Databricks accepts server-side works through dbt, including [`synonyms`](https://docs.databricks.com/aws/en/metric-views/semantic-metadata) and `display_name` on dimensions and measures, and `format` and `window` on measures. You can also set `databricks_tags` and [`grants`](https://docs.getdbt.com/reference/resource-configs/grants.md) on a metric view. `tblproperties` are applied only when the view is updated in place (with `view_update_via_alter`) or replaced, not on first creation. ##### Updating a metric view By default, dbt rebuilds the metric view with `CREATE OR REPLACE VIEW` on every run. When you set [`view_update_via_alter`](https://docs.getdbt.com/reference/global-configs/databricks-changes.md#changes-to-the-view-materialization) to `true`, dbt applies incremental changes in place instead of replacing the view: * Changes to the YAML definition are applied with `ALTER VIEW ... AS`. * Changes to `databricks_tags` or `tblproperties` are applied with `ALTER VIEW ... SET`. If neither the definition nor the tags or properties have changed, dbt skips the update. #### Setting table properties [Table properties](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-tblproperties.html) can be set with your configuration for tables or views using `tblproperties`: with\_table\_properties.sql ```sql {{ config( tblproperties={ 'delta.autoOptimize.optimizeWrite' : 'true', 'delta.autoOptimize.autoCompact' : 'true' } ) }} ``` caution These properties are sent directly to Databricks without validation in dbt. You'll need to do a full refresh of incremental materializations if you change their `tblproperties`. One use case is making `delta` tables compatible with `iceberg` readers using the [Universal Format](https://docs.databricks.com/en/delta/uniform.html): ```sql {{ config( tblproperties={ 'delta.enableIcebergCompatV2' = 'true' 'delta.universalFormat.enabledFormats' = 'iceberg' } ) }} ``` `tblproperties` can be specified for Python models, but they're applied via an `ALTER` statement after table creation due to a PySpark limitation. --- ### Databricks permissions In Databricks, permissions are used to control who can perform certain actions on different database objects. Use SQL statements to manage permissions in a Databricks database. #### Example Databricks permissions The following example provides you with the SQL statements you can use to manage permissions. **Note** that you can grant permissions on `securable_objects` to `principals` (This can be user, service principal, or group). For example, `grant privilege_type` on `securable_object` to `principal`. ```text grant all privileges on schema schema_name to principal; grant create table on schema schema_name to principal; grant create view on schema schema_name to principal; ``` Check out the [official documentation](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/privileges.html#privilege-types-by-securable-object-in-unity-catalog) for more information. --- ### dbt Classes dbt has a number of classes it uses to represent objects in a data warehouse, parts of a dbt project, and the results of a command. These classes are often useful when building advanced dbt models and macros. #### Relation The `Relation` object is used to interpolate schema and table names into SQL code with appropriate quoting. This object should *always* be used instead of interpolating values with `{{ schema }}.{{ table }}` directly. Quoting of the Relation object can be configured using the [`quoting` config](https://docs.getdbt.com/reference/project-configs/quoting.md). ##### Creating relations A `Relation` can be created by calling the `create` class method on the `Relation` class. Relation.create ```python class Relation: def create(database=None, schema=None, identifier=None, type=None): """ database (optional): The name of the database for this relation schema (optional): The name of the schema (or dataset, if on BigQuery) for this relation identifier (optional): The name of the identifier for this relation type (optional): Metadata about this relation, eg: "table", "view", "cte" """ ``` ##### Using relations In addition to `api.Relation.create`, dbt returns a Relation when you use [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) or [`this`](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md). relation\_usage.sql ```jinja2 {% set relation = api.Relation.create(schema='snowplow', identifier='events') %} -- Return the `database` for this relation {{ relation.database }} -- Return the `schema` (or dataset) for this relation {{ relation.schema }} -- Return the `identifier` for this relation {{ relation.identifier }} -- Return relation name without the database {{ relation.include(database=false) }} -- Return true if the relation is a table {{ relation.is_table }} -- Return true if the relation is a view {{ relation.is_view }} -- Return true if the relation is a cte {{ relation.is_cte }} ``` #### Column The `Column` object is used to encode information about a column in a relation. column.py ```python class Column(object): def __init__(self, column, dtype, char_size=None, numeric_size=None): """ column: The name of the column represented by this object dtype: The data type of the column (database-specific) char_size: If dtype is a variable width character type, the size of the column, or else None numeric_size: If dtype is a fixed precision numeric type, the size of the column, or else None """ # Example usage: col = Column('name', 'varchar', 255) col.is_string() # True col.is_numeric() # False col.is_number() # False col.is_integer() # False col.is_float() # False col.string_type() # character varying(255) col.numeric_type('numeric', 12, 4) # numeric(12,4) ``` ##### Column API ##### Properties * **char\_size**: Returns the maximum size for character varying columns * **column**: Returns the name of the column * **data\_type**: Returns the data type of the column (with size/precision/scale included) * **dtype**: Returns the data type of the column (without any size/precision/scale included) * **name**: Returns the name of the column (identical to `column`, provided as an alias). * **numeric\_precision**: Returns the maximum precision for fixed decimal columns * **numeric\_scale**: Returns the maximum scale for fixed decimal columns * **quoted**: Returns the name of the column wrapped in quotes ##### Instance methods * **is\_string()**: Returns True if the column is a String type (eg. text, varchar), else False * **is\_numeric()**: Returns True if the column is a fixed-precision Numeric type (eg. `numeric`), else False * **is\_number()**: Returns True if the column is a number-y type (eg. `numeric`, `int`, `float`, or similar), else False * **is\_integer()**: Returns True if the column is an integer (eg. `int`, `bigint`, `serial` or similar), else False * **is\_float()**: Returns True if the column is a float type (eg. `float`, `float64`, or similar), else False * **string\_size()**: Returns the width of the column if it is a string type, else, an exception is raised ##### Static methods * **string\_type(size)**: Returns a database-useable representation of the string type (eg. `character varying(255)`) * **numeric\_type(dtype, precision, scale)**: Returns a database-useable representation of the numeric type (eg. `numeric(12, 4)`) ##### Using columns column\_usage.sql ```jinja2 -- String column {%- set string_column = api.Column('name', 'varchar', char_size=255) %} -- Return true if the column is a string {{ string_column.is_string() }} -- Return true if the column is a numeric {{ string_column.is_numeric() }} -- Return true if the column is a number {{ string_column.is_number() }} -- Return true if the column is an integer {{ string_column.is_integer() }} -- Return true if the column is a float {{ string_column.is_float() }} -- Numeric column {%- set numeric_column = api.Column('distance_traveled', 'numeric', numeric_precision=12, numeric_scale=4) %} -- Return true if the column is a string {{ numeric_column.is_string() }} -- Return true if the column is a numeric {{ numeric_column.is_numeric() }} -- Return true if the column is a number {{ numeric_column.is_number() }} -- Return true if the column is an integer {{ numeric_column.is_integer() }} -- Return true if the column is a float {{ numeric_column.is_float() }} -- Static methods -- Return the string data type for this database adapter with a given size {{ api.Column.string_type(255) }} -- Return the numeric data type for this database adapter with a given precision and scale {{ api.Column.numeric_type('numeric', 12, 4) }} ``` #### BigQuery columns The `Column` type is overridden as a `BigQueryColumn` in BigQuery dbt projects. This object works the same as the `Column` type described above, with the exception of extra properties and methods: ##### Properties * **fields**: Returns the list of subfields contained within a field (if the column is a STRUCT) * **mode**: Returns the "mode" of the column, eg. `REPEATED` ##### Instance methods **flatten()**: Return a flattened list of `BigQueryColumns` in which subfields are expanded into their own columns. For example, this nested field: ```text [{"hits": {"pageviews": 1, "bounces": 0}}] ``` will be expanded to: ```text [{"hits.pageviews": 1, "hits.bounces": 0}] ``` #### Result objects The execution of a resource in dbt generates a `Result` object. This object contains information about the executed node, timing, status, and metadata returned by the adapter. At the end of an invocation, dbt records these objects in [`run_results.json`](https://docs.getdbt.com/reference/artifacts/run-results-json.md). * `node`: Full object representation of the dbt resource (model, seed, snapshot, test) executed, including its `unique_id` * `status`: dbt's interpretation of runtime success, failure, or error * `thread_id`: Which thread executed this node? E.g. `Thread-1` * `execution_time`: Total time spent executing this node, measured in seconds. * `timing`: Array that breaks down execution time into steps (often `compile` + `execute`) * `message`: How dbt will report this result on the CLI, based on information returned from the database - `adapter_response`: Dictionary of metadata returned from the database, which varies by adapter. For example, success `code`, number of `rows_affected`, total `bytes_processed`, and so on. Not applicable for [data tests](https://docs.getdbt.com/docs/build/data-tests.md). * `rows_affected` returns the number of rows modified by the last statement executed. In cases where the query's row count can't be determined or isn't applicable (such as when creating a view), a [standard value](https://peps.python.org/pep-0249/#rowcount) of `-1` is returned for `rowcount`. --- ### dbt Command reference You can run dbt using the following tools: * In your browser with the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) * On the command line interface using the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), or self-hosted dbt — either [dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md?version=2.0) or [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md?version=1.0). A key distinction is that dbt platform CLI and Studio IDE are designed to support safe parallel execution of dbt commands, leveraging dbt platform's infrastructure and its comprehensive [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md). In contrast, dbt v1 *doesn't support* safe parallel execution for multiple invocations in the same process. Learn more in the [parallel execution](#parallel-execution) section. #### Parallel execution dbt platform allows for concurrent execution of commands, enhancing efficiency without compromising data integrity. This enables you to run multiple commands at the same time. However, it's important to understand which commands can be run in parallel and which can't. In contrast, [`dbt` *doesn't* support](https://docs.getdbt.com/reference/programmatic-invocations.md#parallel-execution-not-supported) safe parallel execution for multiple invocations in the same process, and requires users to manage concurrency manually to ensure data integrity and system stability. To ensure your dbt workflows are both efficient and safe, you can run different types of dbt commands in parallel — for example, `dbt build` (write operation) can safely run alongside `dbt parse` (read operation). However, you can't run `dbt build` and `dbt run` (both write operations) in parallel. dbt commands can be `read` or `write` commands: | Command type | Description | Example | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | | **Write** | These commands perform actions that change data or metadata in your data platform.<br /><br />Limited to one invocation at any given time, which prevents any potential conflicts, such as overwriting the same table in your data platform at the same time. | `dbt build`<br />`dbt run` | | **Read** | These commands involve operations that fetch or read data without making any changes to your data platform.<br /><br />Can have multiple invocations in parallel and aren't limited to one invocation at any given time. This means read commands can run in parallel with other read commands and a single write command. | `dbt parse`<br />`dbt compile` | #### Available commands The following sections outline the commands supported by dbt and their relevant flags. They are available in all tools and all [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) unless noted otherwise. You can run these commands in your specific tool by prefixing them with `dbt` — for example, to run the `test` command, type `dbt test`. You can also call these commands and flags programmatically using `dbtRunner.invoke`. For details, refer to [Programmatic invocations](https://docs.getdbt.com/reference/programmatic-invocations.md). For information about selecting models on the command line, refer to [Model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). Commands marked ❌ indicate write commands, commands marked ✅ indicate read commands, and commands marked N/A indicate that parallel execution isn't relevant for that command. | Command | Description | Parallel execution | Caveats | | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | | [build](https://docs.getdbt.com/reference/commands/build.md) | Builds and tests all selected resources. | ❌ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | cancel | Cancels the most recent invocation. | N/A | dbt platform CLI<br />Requires [dbt v1.6 or higher](https://docs.getdbt.com/docs/dbt-versions.md) | | [check](https://docs.getdbt.com/reference/commands/check.md) | Runs checks against the [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) | ✅ | dbt v2 and later | | [clean](https://docs.getdbt.com/reference/commands/clean.md) | Deletes artifacts present in the dbt project | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [clone](https://docs.getdbt.com/reference/commands/clone.md) | Clones selected models from the specified state | ❌ | All tools<br />Requires [dbt v1.6 or higher](https://docs.getdbt.com/docs/dbt-versions.md) | | [compile](https://docs.getdbt.com/reference/commands/compile.md) | Compiles (but does not run) the models in a project | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [debug](https://docs.getdbt.com/reference/commands/debug.md) | Debugs dbt connections and projects | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [deps](https://docs.getdbt.com/reference/commands/deps.md) | Downloads dependencies for a project | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [docs](https://docs.getdbt.com/reference/commands/cmd-docs.md) | Generates documentation for a project | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [environment](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) | Enables you to interact with your dbt platform environment. | N/A | dbt platform CLI<br />Requires [dbt v1.5 or higher](https://docs.getdbt.com/docs/dbt-versions.md) | | [freshness](https://docs.getdbt.com/reference/commands/freshness.md) | Checks whether sources and models with freshness configured meet your staleness thresholds | ✅ | All tools<br />Requires [dbt v2.0 or higher](https://docs.getdbt.com/docs/dbt-versions.md) | | help | Displays help information for any command | N/A | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [init](https://docs.getdbt.com/reference/commands/init.md) | Initializes a new dbt project | ✅ | dbt v2<br />dbt v1<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [invocation](https://docs.getdbt.com/reference/commands/invocation.md?version=2.0) | Enables users to debug long-running sessions by interacting with active invocations. | N/A | dbt platform CLI<br />Requires [dbt v1.5 or higher](https://docs.getdbt.com/docs/dbt-versions.md) | | [lint](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) | Lints SQL files in a project for style, correctness, and convention violations | ✅ | All tools<br />dbt v2 only | | [list](https://docs.getdbt.com/reference/commands/list.md) | Lists resources defined in a dbt project | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [login](https://docs.getdbt.com/reference/commands/login.md?version=2.0) | Logs in to your dbt platform account | N/A | v1.13 and later<br />dbt platform CLI | | [parse](https://docs.getdbt.com/reference/commands/parse.md) | Parses a project and writes detailed timing info | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | reattach | Reattaches to the most recent invocation to retrieve logs and artifacts. | N/A | dbt platform CLI<br />Requires [dbt v1.6 or higher](https://docs.getdbt.com/docs/dbt-versions.md) | | [retry](https://docs.getdbt.com/reference/commands/retry.md) | Retry the last run `dbt` command from the point of failure | ✅ | All tools<br />Requires [dbt v1.6 or higher](https://docs.getdbt.com/docs/dbt-versions.md) | | [run](https://docs.getdbt.com/reference/commands/run.md) | Runs the models in a project | ❌ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [run-operation](https://docs.getdbt.com/reference/commands/run-operation.md) | Invokes a macro, including running arbitrary maintenance SQL against the database | ❌ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [seed](https://docs.getdbt.com/reference/commands/seed.md) | Loads CSV files into the database | ❌ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [show](https://docs.getdbt.com/reference/commands/show.md) | Previews table rows post-transformation | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [snapshot](https://docs.getdbt.com/reference/commands/snapshot.md) | Executes "snapshot" jobs defined in a project | ❌ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [source](https://docs.getdbt.com/reference/commands/source.md) | Provides tools for working with source data (including validating that sources are "fresh") | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [system](https://docs.getdbt.com/reference/commands/system.md?version=2.0) | Manages the CLI installation: update to a new version, uninstall, or pre-install ADBC adapter drivers | N/A | dbt v2 only | | [test](https://docs.getdbt.com/reference/commands/test.md) | Executes tests defined in a project | ✅ | All tools<br />All [supported versions](https://docs.getdbt.com/docs/dbt-versions.md)<br />dbt v2 flag `--warn-error` not yet supported | | [wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) | Starts an agentic dbt development session with dbt Wizard from the command line | N/A | [All supported versions](https://docs.getdbt.com/docs/dbt-versions.md) | Note, use the [`--version`](https://docs.getdbt.com/reference/commands/version.md) flag to display the installed dbt v1 or dbt platform CLI version. (Not applicable for the Studio IDE). Available on all [supported versions](https://docs.getdbt.com/docs/dbt-versions.md). --- ### dbt freshness Available in v2 The `dbt freshness` command evaluates whether sources and models with [freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md) configured meet your `warn_after` and `error_after` thresholds, reporting warnings and errors accordingly. #### Usage ```bash dbt freshness [--select SELECTOR] [--resource-type RESOURCE_TYPE] [--exclude-resource-type RESOURCE_TYPE] ``` ##### Run freshness for all sources and models ```bash dbt freshness ``` ##### Include or exclude resource types Use the flag to specify which resource type you want to include or exclude when running the command: ```bash # Include sources only dbt freshness --resource-type source # Include models only dbt freshness --resource-type model # Exclude sources only dbt freshness --exclude-resource-type source ``` ##### Select specific model or source Use `--select` to specify which model or source you want to include when running the command: ```bash # Select a specific model dbt freshness --select stg_orders # Select all sources in a namespace dbt freshness --select "source:jaffle_shop" # Select a specific source table dbt freshness --select "source:jaffle_shop.orders" ``` #### How freshness is evaluated `dbt freshness` evaluates sources or models that have `warn_after` or `error_after` set in their [`freshness` config](https://docs.getdbt.com/reference/resource-configs/freshness.md?version=2). dbt retrieves the latest timestamp using one of the following methods, then compares it with the current timestamp to determine the age of the data. | Method | When used | How dbt retrieves the timestamp | | ----------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `loaded_at_query` | When configured on the resource | Runs the custom SQL expression to retrieve the latest timestamp. | | `loaded_at_field` | When configured on the resource | Queries `MAX(<loaded_at_field>)` against the materialized relation. | | Adapter metadata | When neither `loaded_at_query` nor `loaded_at_field` is configured | Retrieves the last-modified time from adapter relation metadata. Available for sources, and for models materialized as `table`, `incremental`, `materialized_view`, or `dynamic_table`, where supported by the adapter. Models materialized as `view` or `external` must use either `loaded_at_field` or `loaded_at_query`. | You can't configure both `loaded_at_query` and `loaded_at_field` on the same resource. Setting both raises a parse error. #### Command output ##### freshness.json After `dbt freshness` completes, dbt writes results for the evaluated sources and models to `target/freshness.json`. Each entry includes a `resource_type` field that identifies whether the resource is a source or a model. For the full schema, refer to: [`freshness.json`](https://docs.getdbt.com/reference/artifacts/freshness-json.md). ```json { "metadata": { "generated_at": "2026-08-28T00:00:00.000000Z" }, "results": [ { "unique_id": "model.jaffle_shop.stg_orders", "resource_type": "model", "max_loaded_at": "2026-08-27T22:00:00+00:00", "snapshotted_at": "2026-08-28T00:00:00+00:00", "max_loaded_at_time_ago_in_s": 7200, "status": "Pass", "criteria": { "warn_after": {"count": 24, "period": "hour"}, "error_after": {"count": 48, "period": "hour"} } }, { "unique_id": "source.jaffle_shop.jaffle_shop.orders", "resource_type": "source", "max_loaded_at": "2026-08-27T23:30:00+00:00", "snapshotted_at": "2026-08-28T00:00:00+00:00", "max_loaded_at_time_ago_in_s": 1800, "status": "Pass", "criteria": { "warn_after": {"count": 12, "period": "hour"}, "error_after": {"count": 24, "period": "hour"} } } ] } ``` ##### sources.json (legacy) For backward compatibility, whenever sources are included in a `dbt freshness` run, dbt also writes `target/sources.json`. It contains sources only, with no `resource_type` field. If the run includes only models, dbt does not overwrite `sources.json`. For the full schema, refer to [`sources.json`](https://docs.getdbt.com/reference/artifacts/sources-json.md). The legacy [`dbt source freshness`](https://docs.getdbt.com/reference/commands/source.md) command still works for backward compatibility and produces only `sources.json`. #### Related docs * [Freshness config](https://docs.getdbt.com/reference/resource-configs/freshness.md) * [Source data freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) * [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) --- ### dbt Information Schema tables Available in v2 The [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) is a contracted interface into the metadata for all of the resources in your dbt project. Information Schema tables are produced by [`dbt build`](https://docs.getdbt.com/reference/commands/build.md), [`dbt run`](https://docs.getdbt.com/reference/commands/run.md), [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md), or [`dbt parse`](https://docs.getdbt.com/reference/commands/parse.md) with the `--generate-info-schema` flag. #### Tables The dbt Information Schema v1 generates tables across these namespaces: * [`dbt`](#dbt-namespace): Contains metadata about your project's structure, resources, and configuration. Answers the question: *What is your project?* * [`dbt_rt`](#dbt_rt-namespace): Contains runtime execution data. Answers the question: *What happened when you ran your project?* ##### `dbt` namespace The `dbt` namespace tables contain metadata about your project's structure, resources, and configuration. | Table | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt.dag_nodes` | All DAG participants with `unique_id` and `resource_type` | | `dbt.project` | Project-level metadata | | `dbt.packages` | Installed packages, one row per package | | `dbt.project_vars` | Project variables, one row per (project scope, variable name) | | `dbt.project_env_vars` | Environment variables used in the project | | `dbt.models` | All models | | `dbt.seeds` | All seeds | | `dbt.sources` | All sources | | `dbt.snapshots` | All snapshots | | `dbt.functions` | All user-defined functions | | `dbt.analyses` | All analyses | | `dbt.hooks` | All `on-run-start` and `on-run-end` hook operations | | `dbt.checks` | All checks | | `dbt.data_tests` | All data tests (generic and singular) | | `dbt.unit_tests` | All unit tests | | `dbt.macros` | All macros | | `dbt.groups` | All groups | | `dbt.exposures` | All exposures | | `dbt.metrics` | All metrics | | `dbt.docs_blocks` | All docs blocks | | `dbt.saved_queries` | All saved queries | | `dbt.semantic_models` | All semantic models | | `dbt.semantic_entities` | Semantic model entities | | `dbt.semantic_measures` | Semantic model measures | | `dbt.semantic_dimensions` | Semantic model dimensions | | `dbt.semantic_relationships` | Relationships between semantic models | | `dbt.time_spines` | Time spine definitions | | `dbt.classifiers` | Resource classifiers | | `dbt.edges` | DAG edges (parent → child) | | `dbt.node_columns` | Column names, types, and descriptions for nodes; for sources, only columns declared in YAML are included (use [`--write-catalog`](https://docs.getdbt.com/docs/build/dbt-information-schema.md#generating-the-information-schema) to include all columns from sources used by the selected nodes) | | `dbt.column_lineage` | Column-level lineage (populated with `--static-analysis strict`) | ##### `dbt_rt` namespace The `dbt_rt` namespace tables contain runtime execution data. These tables are not available for parse-time checks. | Table | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `dbt_rt.invocations` | One row per dbt invocation | | `dbt_rt.run_results` | Results for each node in each invocation | | `dbt_rt.run_results_latest` | View of the most recent run result per node | | `dbt_rt.relations` | Warehouse catalog data for each materialized relation (populated with `--write-catalog`) | | `dbt_rt.freshness` | Freshness check results for sources and models; `resource_type` indicates whether a row is a `source` or `model` | | `dbt_rt.diagnostics` | Diagnostic data from invocations | | `dbt_rt.adapter_queries` | Adapter queries issued during invocations | #### Columns available for checks When you use `{{ info_schema() }}` macro in a check, you can access only columns resolved at parse time. dbt exposes these columns through a view with the same name as the Information Schema table. Pass that name without the `dbt.` prefix to the macro. The full table in `target/info_schema/v1/` may have more columns. You can access all columns when you use the macro with `dbt show --inline`. | View | Columns | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `models` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `ingested_at` | | `seeds` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `ingested_at` | | `snapshots` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `raw_code`, `ingested_at` | | `functions` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `raw_code`, `ingested_at` | | `analyses` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `raw_code`, `ingested_at` | | `hooks` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `raw_code`, `ingested_at` | | `checks` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `raw_code`, `ingested_at` | | `sources` | `unique_id`, `name`, `resource_type`, `package_name`, `original_file_path`, `fqn`, `alias`, `description`, `node_language`, `database_name`, `schema_name`, `relation_name`, `identifier`, `enabled`, `materialized`, `config`, `access`, `group`, `contract_enforced`, `version`, `latest_version`, `deprecation_date`, `primary_key`, `properties_yml_file_path`, `tags`, `meta`, `source_name`, `source_description`, `loader`, `loaded_at_field`, `ingested_at` | | `data_tests` | `unique_id`, `name`, `package_name`, `original_file_path`, `fqn`, `description`, `database_name`, `schema_name`, `relation_name`, `enabled`, `materialized`, `config`, `tags`, `meta`, `group`, `properties_yml_file_path`, `test_name`, `test_definition_package`, `arguments`, `column_name`, `node_unique_id`, `severity`, `warn_if`, `error_if`, `fail_calc`, `store_failures`, `store_failures_as`, `ingested_at` | | `unit_tests` | `unique_id`, `name`, `model`, `description`, `package_name`, `original_file_path`, `fqn`, `given`, `expect`, `overrides`, `versions`, `ingested_at` | | `node_columns` | `node_unique_id`, `column_name`, `data_type_declared`, `description`, `tags`, `ingested_at` | | `dag_nodes` | `unique_id`, `resource_type`, `ingested_at` — one row per enabled resource that participates in the DAG, including exposures, metrics, and unit tests | | `edges` | `parent_unique_id`, `child_unique_id`, `ingested_at` | | `macros` | `unique_id`, `name`, `package_name`, `original_file_path`, `macro_sql`, `description`, `depends_on_macros`, `arguments`, `docs_show`, `properties_yml_file_path`, `meta`, `created_at`, `ingested_at` | | `docs_blocks` | `unique_id`, `name`, `package_name`, `original_file_path`, `content`, `ingested_at` | | `groups` | `unique_id`, `name`, `description`, `owner_name`, `owner_email`, `ingested_at` | | `exposures` | `unique_id`, `name`, `exposure_type`, `label`, `owner_name`, `owner_email`, `url`, `maturity`, `description`, `package_name`, `original_file_path`, `fqn`, `depends_on`, `tags`, `created_at`, `ingested_at` | | `metrics` | `unique_id`, `name`, `label`, `metric_type`, `description`, `package_name`, `original_file_path`, `fqn`, `type_params`, `metric_filter`, `time_granularity`, `input_metric_names`, `group`, `tags`, `meta`, `config`, `created_at`, `ingested_at` | | `saved_queries` | `unique_id`, `name`, `label`, `description`, `package_name`, `original_file_path`, `fqn`, `query_params`, `exports`, `depends_on`, `group`, `tags`, `created_at`, `ingested_at` | | `semantic_models` | `unique_id`, `name`, `model`, `label`, `description`, `fqn`, `node_relation`, `primary_entity`, `defaults`, `group`, `created_at`, `ingested_at` | | `semantic_entities` | `unique_id`, `name`, `entity_type`, `description`, `label`, `entity_role`, `expr`, `ingested_at` | | `semantic_measures` | `unique_id`, `name`, `agg`, `description`, `label`, `expr`, `create_metric`, `agg_time_dimension`, `agg_params`, `non_additive_dimension`, `ingested_at` | | `semantic_dimensions` | `unique_id`, `name`, `dimension_type`, `description`, `label`, `expr`, `is_partition`, `time_granularity`, `validity_params`, `ingested_at` | | `time_spines` | `unique_id`, `primary_column`, `primary_granularity`, `custom_granularities`, `node_relation`, `ingested_at` | | `project` | `project_name`, `dbt_version`, `adapter_type`, `git_sha`, `git_branch`, `git_uncommitted_changes`, `ingested_at` | | `packages` | `package_name`, `ingested_at` | | `project_vars` | `project_name`, `var_name`, `var_value`, `ingested_at` — a package-scoped var appears on that package's `project_name` | | `project_env_vars` | `env_var_name`, `ingested_at` | #### Related docs * [dbt Information Schema](https://docs.getdbt.com/docs/build/dbt-information-schema.md) * [Checks](https://docs.getdbt.com/docs/build/checks.md) --- ### dbt Jinja functions and context variables #### [adapter](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md) [Wrap the internal database adapter with the Jinja object \`adapter\`.](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md) --- ### dbt State configurations When you run a dbt command with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) enabled, dbt compares a node's logic and data against previous builds and takes the most efficient path: * **Reuse** — if the object exists in the target schema, its logic hasn't changed, and it is not yet due for rebuilding under its `lag_tolerance` (its last build is still within the tolerance window, or its upstream data hasn't changed), the node is reused. * **Clone** — if reuse isn't possible but the object exists in the deferred environment with the same logic and sufficiently fresh data, dbt State clones it. * **Normal build** — if neither reuse nor clone is possible, the node builds as normal, using deferral for any unselected upstream nodes. Use the following configs to control how dbt State makes these decisions. ##### Project YAML file dbt\_project.yml ```yaml models: +state: lag_tolerance: <duration> compare_unrendered_code: true | false require_fresh_data_from: any | all evaluate_volatile_sql: true | false pre_clone: never | if_missing | always execute_hooks_on_any_reuse: true | false ``` ##### Properties YAML file models/schema.yml ```yaml models: - name: <model_name> config: state: lag_tolerance: <duration> compare_unrendered_code: true | false require_fresh_data_from: any | all evaluate_volatile_sql: true | false pre_clone: never | if_missing | always execute_hooks_on_any_reuse: true | false ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( state={ "lag_tolerance": "<duration>", "compare_unrendered_code": true | false, "require_fresh_data_from": "any" | "all", "evaluate_volatile_sql": true | false, "pre_clone": "never" | "if_missing" | "always", "execute_hooks_on_any_reuse": true | false } ) }} ``` Profile-level settings are configured in `profiles.yml` and apply to a specific target environment: profiles.yml ```yaml my_project: outputs: dev: type: snowflake # ... dev connection settings defer_to_target: prod # self-managed only prod: type: snowflake # ... prod connection settings allow_clones: true | false metadata_warehouse: <warehouse_name> # Snowflake only target: dev ``` | Config | Default | Scope | Description | | ---------------------------------------------------------------------------------------------------------------- | ------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) | `45m` | Node, folder, or project-level via model config | Sets how long dbt State waits before rebuilding a node after its upstream data changes. A node rebuilds only when its last build is older than this window and its upstream data has changed. Acts as a compute-saving buffer that helps align builds with freshness SLAs. Applies to data freshness only; SQL changes always trigger a rebuild regardless of this setting. | | [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md) | `false` | Node, folder, or project-level via model config | Controls whether dbt State checks both the Jinja template (unrendered code) and rendered SQL when detecting code changes. Useful for models with non-deterministic macros or environment variables that produce different rendered SQL on every run. | | [`require_fresh_data_from`](https://docs.getdbt.com/reference/resource-configs/require-fresh-data-from.md) | `any` | Node, folder, or project-level via model config | Whether `any` or `all` direct parents need fresh data before a node is eligible for a rebuild. | | [`pre_clone`](https://docs.getdbt.com/reference/resource-configs/pre-clone.md) | `if_missing` | Node, folder, or project-level via model config | Whether dbt State pre-populates incremental models and snapshots by cloning production before a run. | | [`execute_hooks_on_any_reuse`](https://docs.getdbt.com/reference/resource-configs/execute-hooks-on-any-reuse.md) | `false` | Node, folder, or project-level via model config | Whether pre- and post-hooks run when a node is reused without rebuilding. | | [`evaluate_volatile_sql`](https://docs.getdbt.com/reference/resource-configs/evaluate-volatile-sql.md) | `false` | Node, folder, or project-level via model config | Whether dbt State stores and compares the runtime output of volatile SQL functions when deciding whether to rebuild. | | [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) | `prod` | Profile | (Self-managed only) Which profile target dbt State defers to. | | [`allow_clones`](https://docs.getdbt.com/reference/resource-configs/allow-clones.md) | `true` | Profile | Whether dbt State can clone tables into a target. For example, set to `false` on `prod` to prevent dbt State from cloning dev tables into production. | | [`metadata_warehouse`](https://docs.getdbt.com/reference/resource-configs/metadata-warehouse.md) | Profile `warehouse` | Profile | (Snowflake only) A separate warehouse for dbt State metadata lookups. When set, dbt issues multiple, individual queries (one per schema) instead of a single, consolidated query. | #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) --- ### dbt state explain (Applies to dbt v2.0 and later) `dbt state explain` is a CLI command available when you enable [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). After a job finishes, run this command to see why dbt State made each decision for every node and whether it executed, skipped, or cloned the node. Use it to audit State behavior, debug unexpected skips, or confirm that freshness and query checks work as expected. (Applies to dbt v2.0 and later) ```bash dbt state explain ``` The output shows all nodes from the last run, with a summary of the State decision for each: ```shell SKIP_EXECUTION model.jaffle_shop.customers - model was a no-op because its query is up to date and its upstream data is within freshness tolerance READY_TO_EXECUTE model.jaffle_shop.orders - model was executed because the view definition is newer than the cached execution READY_TO_EXECUTE test.jaffle_shop.not_null_customers_customer_id - data test was executed because it has no prior execution or its query changed UNKNOWN unit_test.jaffle_shop.orders.test_order_items_compute_to_bools_correctly - dbt State explain details unavailable ``` If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. #### Specifying a log file By default, (Applies to dbt v2.0 and later) `dbt state explain` reads from the most recent execution. To analyze a previous run, you can use `--log-file` (or `-l`) to specify a state file from the `logs/state/` directory: (Applies to dbt v2.0 and later) ```bash dbt state explain --log-file 'logs/state/responses_2026_08_25_11_00_15_667.jsonl' ``` #### Using verbose mode Use the `--verbose` flag to see the full step-by-step analysis for each node and add a run configuration summary. Use `-s` to filter the output to a specific node. (Applies to dbt v2.0 and later) ```bash dbt state explain --verbose -s my_node_name ``` In dbt v2, `--verbose` adds a run configuration summary at the top and shows the full step-by-step analysis for each node. ```shell Run configuration: - started at: 2026-08-17T12:24:15.822733+00:00 - profile: my_profile - target: dev - defer to target: prod - freshness tolerance: 2700 seconds - tolerate nondeterminism: true - clone incremental in dev: IF_TABLE_MISSING - metadata cache TTL: 0 seconds - select: fqn:my_node_name SKIP_EXECUTION model.jaffle_shop.customers - model was a no-op because its query is up to date and its upstream data is within freshness tolerance - table analysis ("ANALYTICS"."MY_SCHEMA"."CUSTOMERS") - the model table exists already [SUCCESS, TARGET_TABLE_EXISTS] - query analysis - the model query has not changed [SUCCESS, NODE_QUERY_UNCHANGED] - upstream model queries have not changed [SUCCESS] - data freshness analysis - upstream dependencies - "ANALYTICS"."MY_SCHEMA_RAW"."RAW_CUSTOMERS" [FRESH] - no updates since "ANALYTICS"."MY_SCHEMA"."CUSTOMERS" last executed - last updated: 5 days ago - upstream data is up to date [SUCCESS] ``` The `--verbose` flag produces a decision breakdown that may include the following analyses, depending on the node type: * **Table analysis**: whether the target table already exists in the schema * **Query analysis**: whether the model query or its upstream queries have changed * **Data freshness analysis**: whether upstream data is fresh or within the configured [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### dbt v2 telemetry and observability dbt v2 provides a comprehensive observability system that replaces [dbt v1's structured logging](https://docs.getdbt.com/reference/events-logging.md#structured-logging). Built on [OpenTelemetry](https://opentelemetry.io/) conventions and backed by a stable protobuf schema, it enables deep integration with orchestrators, observability platforms, and custom tooling. For shared CLI logging configs such as `--log-format` and `--log-level`, refer to [Logs](https://docs.getdbt.com/reference/global-configs/logs.md). This system is separate from the anonymous usage statistics that dbt sends to dbt Labs. To configure anonymous usage statistics, refer to [Anonymous usage stats](https://docs.getdbt.com/reference/global-configs/usage-stats.md). This uses the same integration that dbt platform relies on for orchestration and monitoring, providing proven and production-ready features that work at scale. #### Available output formats dbt v2 telemetry supports three output formats, which you can enable independently: | Format | Use case | Availability | | ----------- | --------------------------------------------------------------------- | --------------------------- | | **JSONL** | Real-time monitoring, streaming to downstream systems. | Written as events occur. | | **Parquet** | Post-run analysis, querying, and long-term storage. | Written when runs complete. | | **OTLP** | Integration with observability platforms (Datadog, Jaeger, and more). | Streamed in real-time. | ##### Enabling telemetry output The following are some examples of options for enabling telemetry output (You can combine multiple outputs in a single run): Write JSONL to a file (saves to the `logs/` directory): ```bash dbtf build --otel-file-name telemetry.jsonl ``` Stream JSONL to stdout: ```bash dbtf build --log-format otel ``` Write a Parquet file (saves to `target/metadata/` directory): ```bash dbtf build --otel-parquet-file-name telemetry.parquet ``` Write a Parquet file with full trace-level logs (recommended for debugging): ```bash dbtf build --log-level-file trace --otel-parquet-file-name telemetry.parquet ``` Export to an OpenTelemetry collector: ```bash OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" dbtf build --export-to-otlp ``` ##### Download telemetry from platform job runs On the dbt platform, dbt v2 job runs store OTel telemetry as Parquet artifacts for dbt command steps. From a completed run, open the **Run summary** tab, select a step, and click **Download** > **Download OTel log**. The option appears only for dbt v2 runs where the step produced an OTel file. For step-by-step instructions, refer to [Downloading logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs). ###### Retrieve telemetry using the API You can also retrieve the OTel Parquet artifact for a run step through the [dbt Administrative API v2](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retrieve%20Run%20Artifact), which lets you download artifacts after a job completes. Use this to automate ingestion of node outcomes and test outcomes into a downstream system, such as a data quality framework in your warehouse. Each dbt v2 command step that produces telemetry writes a `telemetry-STEP_NUMBER-otel.parquet` artifact. Some steps like `dbt deps` don't produce a Parquet artifact. You can use the Retrieve Run Artifact endpoint to fetch this artifact: ```bash GET https://YOUR_ACCESS_URL/api/v2/accounts/ACCOUNT_ID/runs/RUN_ID/artifacts/metadata/telemetry-STEP_NUMBER-otel.parquet?step=STEP_NUMBER ``` Replace `YOUR_ACCESS_URL` with the [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan, and `ACCOUNT_ID`, `RUN_ID`, `STEP_NUMBER` with your values. Authenticate with a [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). For example, you can do this with `curl`: ```bash curl --request GET \ --url 'https://YOUR_ACCESS_URL/api/v2/accounts/12345/runs/67890/artifacts/metadata/telemetry-4-otel.parquet?step=4' \ --header 'Authorization: Token YOUR_TOKEN' \ --output telemetry-4-otel.parquet ``` To find which step produced the telemetry artifact you want, list the run's steps by including `run_steps` in the run details request: ```bash GET https://YOUR_ACCESS_URL/api/v2/accounts/ACCOUNT_ID/runs/RUN_ID/?include_related=["run_steps"] ``` You can only retrieve this artifact for dbt v2 steps that emitted an OTel log. #### Telemetry data dbt v2 telemetry contains two types of records: * **Spans** — Operations with a start and end time (like compiling a model or running a test). * **Log records** — Point-in-time events within a span. ##### Telemetry hierarchy Every dbt command creates a hierarchy of spans: ```text Invocation (dbtf build) ├── Phase (Parse) ├── Phase (Compile) │ ├── Node (model.project.customers) │ └── Node (model.project.orders) └── Phase (Run) ├── Node (model.project.customers) └── Node (model.project.orders) ``` The `trace_id` (also known as `invocation_id`) remains consistent across all telemetry records for a single dbt command, making it easy to correlate events. #### Node outcome Every node produces a result for each phase it participates in. Some phases, such as `parse`, don't involve node-level execution, so they don't produce node spans or node outcomes. The `node_outcome` field indicates whether or not dbt v2 executed the node's operation. | Outcome | Description | | ---------- | ----------------------------------------------------------------- | | `success` | The node operation was completed with no errors. | | `error` | The node operation failed to execute (for example, syntax error). | | `skipped` | The node was not evaluated (see [skip reasons](#skip-reasons)). | | `canceled` | The node was interrupted (for example, user pressed Ctrl+C). | ##### Skip reasons When dbt v2 skips a node, the telemetry includes a reason: | Skip reason | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `upstream` | A dependency failed. | | `cached` | dbt v2 reused results from cache (no changes detected via [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)). | | `phase_disabled` | The phase was disabled (for example, `--static-analysis off`). | | `noop` | Node doesn't perform work in this phase (for example, ephemeral models). | ##### Test outcomes When a test executes successfully (`node_outcome: success`), it reports the test result: | Test outcome | Description | | ------------ | ---------------------------------------------- | | `passed` | No failures detected. | | `warned` | Failures detected, but configured as warnings. | | `failed` | Failures detected (data quality issue). | Test outcomes A test with `node_outcome: success` and `test_outcome: failed` means dbt v2 successfully ran the test, and the test reported data quality issues. This differs from `node_outcome: error`, which means the test itself couldn't run (for example, invalid SQL). #### Querying telemetry data Query the telemetry data to gain deeper insights into your dbt runs. ##### JSONL examples The following are some examples of querying the JSONL telemetry data. Watch for errors in real-time: ```bash tail -f telemetry.jsonl | jq 'select(.severity_text == "ERROR")' ``` List skipped nodes, reasons, and upstream details: ```bash cat telemetry.jsonl | jq 'select(.attributes.node_outcome == "NODE_OUTCOME_SKIPPED") | {node: .attributes.unique_id, reason: .attributes.node_skip_reason, upstream: .attributes.node_skip_upstream_detail.upstream_unique_id }' ``` ##### Downloading telemetry from the dbt platform If you ran a job in the dbt platform, you can download the OpenTelemetry (OTEL) Parquet artifact from the run page using the **Download** dropdown. The download includes the `telemetry-<step>-otel.parquet` file for each step in the run. ##### Parquet analysis with DuckDB Leverage DuckDB to better understand your telemetry data stored in Parquet files. Find slowest nodes by **processing cost** (not wall-clock lifetime at the connection gate): ```python import duckdb duckdb.sql(""" SELECT attributes.unique_id, attributes.duration_ms, attributes.idle_time_ms FROM 'telemetry.parquet' WHERE event_type LIKE '%NodeProcessed%' AND attributes.duration_ms IS NOT NULL ORDER BY attributes.duration_ms DESC LIMIT 10 """).show() ``` Choose the right timing metric Telemetry provides several ways to measure node performance: * **Processing time (`attributes.duration_ms`)** measures the time v2 spent actively processing the node, including nested `NodeEvaluated` work. It excludes time spent waiting for upstream nodes or internal backpressure. Use this metric to identify the nodes that take the longest to process. * **Node lifetime (`end_time_unix_nano - start_time_unix_nano`)** measures the full time from the start to the end of the span, including time spent waiting at the connection-limit gate. In builds with saturated threads, this metric might surface nodes with the longest queue time rather than the most processing work. * **Idle time (`attributes.idle_time_ms`)** measures how long the node spent waiting instead of being actively processed, such as while waiting for an upstream node or available processing capacity. Use it to identify where resource constraints are causing delays. * **Warehouse execution time** is the sum of `QueryExecuted` span durations for each `unique_id`. It excludes v2-side work such as compilation and static analysis. Use this metric to compare telemetry with your warehouse query history. Find nodes with the highest warehouse time (optional): ```python duckdb.sql(""" SELECT attributes.unique_id, SUM((end_time_unix_nano - start_time_unix_nano) / 1e6) AS warehouse_ms FROM 'telemetry.parquet' WHERE event_type LIKE '%QueryExecuted%' GROUP BY attributes.unique_id ORDER BY warehouse_ms DESC LIMIT 10 """).show() ``` Count outcomes by type: ```python duckdb.sql(""" SELECT attributes.node_outcome, COUNT(*) as count FROM 'telemetry.parquet' WHERE attributes.node_outcome IS NOT NULL GROUP BY attributes.node_outcome """).show() ``` ##### Web-based Parquet viewers For ad hoc exploration without a local install, web-based Parquet viewers (such as [PondPilot](https://app.pondpilot.io/)) let you upload a Parquet file and run SQL queries in the browser. Some viewers support LLM-assisted query generation to help you explore an unfamiliar schema. #### OpenTelemetry integration dbt v2's native OTLP support lets you send telemetry directly to any OpenTelemetry-compatible receiver, including Datadog, Jaeger, Google Cloud Trace, Grafana Tempo, and Honeycomb. This enables: * Integration with existing observability — No custom integrations needed. * Custom alerts that trigger notifications on failures or slow builds. * Correlate across systems that links dbt traces with downstream services. * Centralized monitoring to view dbt alongside your other infrastructure. ##### Setting up OTLP export The following example configures the OTLP export: ```bash export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" dbtf build --export-to-otlp ``` #### Mapping to dbt v1 concepts If you're familiar with dbt v1's structured logging, here's how dbt v2 telemetry maps: | dbt v1 | dbt v2 telemetry | | ---------------------------------- | ------------------------------------------------ | | `invocation_id` | `trace_id` (same value, different format) | | `run_results.json` status | `node_outcome` + `skip_reason` or `test_outcome` | | Event `code` (for example, `Q001`) | `event_type` | | `--log-format json` | `--log-format otel` or `--otel-file-name` | ##### Node status mapping | dbt v1 status | dbt v2 outcome | | -------------- | ------------------------------------------------ | | `success` | `node_outcome: success` | | `error` | `node_outcome: error` | | `skipped` | `node_outcome: skipped`, `skip_reason: upstream` | | `pass` (tests) | `node_outcome: success`, `test_outcome: passed` | | `warn` (tests) | `node_outcome: success`, `test_outcome: warned` | | `fail` (tests) | `node_outcome: success`, `test_outcome: failed` | Note that dbt v1's `fail` status maps to dbt v2's `node_outcome: success` because dbt v2 distinguishes between "the test ran successfully and found data issues" versus "the test couldn't run." This separation enables more precise alerting and retry logic. dbt v2 adds `skip_reason: cached` for nodes reused via [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), which has no dbt v1 equivalent. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). #### Record structure Each telemetry record contains envelope fields plus event-specific `attributes`: ```json { "record_type": "SpanEnd", "trace_id": "f9a0a9e64c924b878133363ba3515e50", "span_id": "0000000000000036", "span_name": "Node(model.project.customers)", "parent_span_id": "0000000000000017", "start_time_unix_nano": "1756139116981079652", "end_time_unix_nano": "1756139117234567890", "severity_text": "INFO", "event_type": "v1.public.events.fusion.node.NodeEvaluated", "attributes": { "unique_id": "model.project.customers", "phase": "Run", "node_outcome": "success" } } ``` | Field | Description | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `record_type` | `SpanStart`, `SpanEnd`, or `LogRecord` | | `trace_id` | Unique identifier for the invocation (same data as `invocation_id` but in OTEL format). | | `span_id` / `parent_span_id` | For reconstructing the span hierarchy. | | `event_type` | Type identifier for filtering and parsing. | | `attributes` | Event-specific data (schema varies by event type, but unlike OTEL conventions, it's strictly backed by a stable protobuf schema). | #### Schema stability Unlike dbt v1's structured logging, dbt v2 telemetry is backed by a public protobuf schema with strict compatibility guarantees: * **Additive only** — New fields and event types may be added, but existing fields are never removed or changed. * **Forward compatible** — Your integrations will continue to work as the schema evolves. This makes dbt v2 telemetry a reliable foundation for production integrations, orchestrators, and long-term analytics pipelines. #### Official client library dbt Labs provides an official open-source client library. Built in Rust for performance, it is available as: * A standalone Rust crate and CLI (`dbt-telemetry-export-cli`). * A fully typed Python package wrapping the Rust core, installable via `uv tool install`. The library provides type-safe, forward-compatible access to telemetry data. Stream JSONL in real time, query Parquet files, and build custom integrations with confidence that schema changes won't break your code. We'll be announcing the public release in the near future. --- ### dbt_cloud.yml file dbt platform The `dbt_cloud.yml` file stores the credentials that dbt tools — like the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md), and more — use to authenticate with dbt platform. You can download it from dbt platform and save it locally to your `.dbt` directory. This page covers: * [How to download it and set up the `.dbt` directory](#download-dbt_cloudyml) for the dbt platform CLI or the VS Code extension * [Update or switch projects](#update-or-switch-projects) * [The file structure](#file-structure) and field reference * [The companion `dbt-cloud` block](#the-dbt-cloud-block-in-dbt_projectyml) in `dbt_project.yml` Keep this file safe The `dbt_cloud.yml` file contains API keys. Store it securely and make sure you *do not* commit it to version control. #### Download dbt\_cloud.yml How you download the file depends on whether you're configuring the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md). The downloaded `dbt_cloud.yml` includes your [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). 1. In dbt platform, select the project you want to work on. The project must already have a [development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment) set up. 2. Go to **Account settings** → **Your profile**, then follow the steps for your tool: * dbt platform CLI: Navigate to **CLI** → **Configure Cloud authentication** and click **Download CLI configuration file**. * dbt VS Code extension: Navigate to **VS Code Extension** → **Set up your credentials** and click **Download credentials**. 3. Move the file to your `.dbt` directory. If you don't have one yet, try one of these quick setup (one command) options: ##### Mac / Linux ```bash mkdir -p ~/.dbt && mv ~/Downloads/dbt_cloud.yml ~/.dbt/dbt_cloud.yml ``` ##### Windows ```powershell mkdir %USERPROFILE%\.dbt 2>nul & move %USERPROFILE%\Downloads\dbt_cloud.yml %USERPROFILE%\.dbt\dbt_cloud.yml ```  Manually create and move your .dbt directory If you've never had a `.dbt` directory, you should perform the following recommended steps to create one. If you already have a `.dbt` directory, move the `dbt_cloud.yml` file into it. Some information about the `.dbt` directory: * A `.dbt` directory is a hidden folder in the root of your filesystem. It's used to store your dbt configuration files. The `.` prefix is used to create a hidden folder, which means it's not visible in Finder or File Explorer by default. * To view hidden files and folders, press Command + Shift + G on macOS or Ctrl + Shift + G on Windows. This opens the "Go to Folder" dialog where you can search for the `.dbt` directory. ##### Create a .dbt directory 1. Clone your dbt project repository locally. 2. Use the `mkdir` command followed by the name of the folder you want to create. * If using macOS, add the `~` prefix to create a `.dbt` folder in the root of your filesystem: * macOS: `mkdir ~/.dbt` * Windows: `mkdir %USERPROFILE%\.dbt` ##### Move the dbt\_cloud.yml file You can move the `dbt_cloud.yml` file into the `.dbt` directory using the `mv` command or by dragging and dropping the file into the `.dbt` directory by opening the Downloads folder using the "Go to Folder" dialog and then using drag-and-drop in the UI. To move the file using the terminal, use the `mv/move` command. This command moves the `dbt_cloud.yml` from the `Downloads` folder to the `.dbt` folder. If your `dbt_cloud.yml` file is located elsewhere, adjust the path accordingly. ###### Mac or Linux In your command line, use the `mv` command to move your `dbt_cloud.yml` file into the `.dbt` directory. If you've just downloaded the `dbt_cloud.yml` file and it's in your Downloads folder, the command might look something like this: ```bash mv ~/Downloads/dbt_cloud.yml ~/.dbt/dbt_cloud.yml ``` ###### Windows In your command line, use the move command. Assuming your file is in the Downloads folder, the command might look like this: ```bash move %USERPROFILE%\Downloads\dbt_cloud.yml %USERPROFILE%\.dbt\dbt_cloud.yml ``` If your downloaded file has a numerical suffix (for example, `dbt_cloud(2).yml`), rename it to `dbt_cloud.yml` before moving it. The dbt platform CLI and extension only look for the exact filename. 4. Confirm that the `project-id` in your [`dbt_project.yml` `dbt-cloud` block](#the-dbt-cloud-block-in-dbt_projectyml) matches the project you're working on. This registers and connects your tool to dbt platform and enables platform features such as Mesh and deferral. #### Update or switch projects The `dbt_cloud.yml` file is local to your machine and doesn't update automatically. Make sure to re-download it when: * You get access to a new project and want to work on it locally * A project is removed from your account or your project access changes * Your token changes or is rotated * Your account host changes, such as when your account moves regions The file can include multiple projects from the same dbt platform account. To switch projects, update `context.active-project` to the `project-id` for the project you want to use. The active project must match one of the projects listed under `projects`. If you work in multiple dbt platform accounts, keep a separate `dbt_cloud.yml` file for each account and move the file you want to use into your `.dbt` directory. #### File structure Your `dbt_cloud.yml` file has the following structure: dbt\_cloud.yml ```yaml version: "1" context: active-host: your_active_host # for example, "abc123.us1.dbt.com" active-project: your_project_id # for example, "123456" projects: - project-name: your_project_name project-id: your_project_id account-name: your_account_name account-id: your_account_id account-host: your_account_host # for example, "abc123.us1.dbt.com" token-name: your_token_name # for example, "cloud-cli-1234" token-value: your_token_value - project-name: your_project_name project-id: your_project_id account-name: your_account_name account-id: your_account_id account-host: your_account_host token-name: your_token_name token-value: your_token_value ``` For example, if you have a Jaffle and wizard shop account, here's what your `dbt_cloud.yml` file would look like: dbt\_cloud.yml ```yaml version: "1" context: active-host: "abc123.us1.dbt.com" active-project: "123456" projects: - project-name: "Project 1" project-id: "123" account-name: "Jaffle and wizard shop" account-id: "1" account-host: "abc123.us1.dbt.com" token-name: "cloud-cli-1091" token-value: "dbtu_token_value" # this would be a longer token value - project-name: "Project 2" project-id: "456" account-name: "Jaffle and wizard shop" account-id: "1" account-host: "abc123.us1.dbt.com" token-name: "cloud-cli-1091" token-value: "dbtu_token_value" - project-name: "Project 3" project-id: "789" account-name: "Jaffle and wizard shop" account-id: "1" account-host: "abc123.us1.dbt.com" token-name: "cloud-cli-1091" token-value: "dbtu_token_value" ``` ##### Field reference | Field | Required | Description | | ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `version` | Yes | The schema version of the file. Currently `"1"`. | | `context.active-host` | Yes | The `account-host` to use by default. You can find it in the **Account settings** page. | | `context.active-project` | Yes | The `project-id` of the project to use by default when running commands. Must match a `project-id` listed under `projects`. | | `context.defer-env-id` | No | The environment ID to defer to for build artifacts. Optional override of the project's default deferral environment. dbt platform CLI only. Refer to [Configure deferral](https://docs.getdbt.com/docs/platform/about-defer.md#configure-deferral-environment-id). | | `projects.project-name` | Yes | A human-readable name for the project. | | `projects.project-id` | Yes | The dbt platform project ID. Find it in the URL when viewing your project (for example, `…/projects/123456`). | | `projects.account-name` | Yes | A human-readable name for the account. | | `projects.account-id` | Yes | The dbt platform account ID. | | `projects.account-host` | Yes | The host for your account, for example `cloud.getdbt.com`, `emea.dbt.com`, or your single-tenant access URL. | | `projects.token-name` | Yes | A name for the [Personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). | | `projects.token-value` | Yes | The PAT value. Treat this as a secret. | #### The dbt-cloud block in dbt\_project.yml The `dbt-cloud` block is a companion config that lives in your project's `dbt_project.yml` file (not in `dbt_cloud.yml`). It tells the dbt platform CLI, the Studio IDE, and dbt v2 which dbt platform project your local project corresponds to. dbt\_project.yml ```yaml name: version: # Your project configs... dbt-cloud: project-id: your_project_id defer-env-id: '123456' # optional ``` | Field | Required | Description | | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `project-id` | Yes | The dbt platform project ID this local project maps to. Find it in the URL when viewing your project (for example, `https://YOUR_ACCESS_URL/develop/26228/projects/123456` → `123456`). | | `defer-env-id` | No | The environment ID to defer to for build artifacts. Used for dbt v2 [auto-deferral](https://docs.getdbt.com/docs/platform/about-defer.md) and dbt platform CLI deferral overrides. | | `account_id` | No | dbt v2 only. The dbt platform account ID this local project belongs to. Note the underscore — unlike the other fields in this block, this one isn't hyphenated. | #### Related docs * [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) * [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md) --- ### dbt_project.yml The dbt\_project.yml file is a required file for all dbt projects. It contains important information that tells dbt how to operate your project. Every [dbt project](https://docs.getdbt.com/docs/build/projects.md) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project. It works as follows: * dbt uses [YAML](https://yaml.org/) in a few different places. If you're new to YAML, it would be worth learning how arrays, dictionaries, and strings are represented. * By default, dbt looks for the `dbt_project.yml` in your current working directory and its parents, but you can set a different directory using the `--project-dir` flag or the (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` environment variable. * Specify your dbt project ID in the `dbt_project.yml` file using `project-id` under the [`dbt-cloud` config](https://docs.getdbt.com/reference/dbt_cloud.yml.md#the-dbt-cloud-block-in-dbt_projectyml). Find your project ID in your dbt project URL: For example, in `https://YOUR_ACCESS_URL/develop/projects/123456`, the project ID is `123456`. * Note, you can't set up a "property" in the `dbt_project.yml` file if it's not a config (an example is [macros](https://docs.getdbt.com/reference/macro-properties.md)). This applies to all types of resources. Refer to [Configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for more detail. #### Example The following example is a list of all available configurations in the `dbt_project.yml` file: dbt\_project.yml ```yml name: string config-version: 2 version: version profile: profilename model-paths: [directorypath] seed-paths: [directorypath] test-paths: [directorypath] analysis-paths: [directorypath] macro-paths: [directorypath] snapshot-paths: [directorypath] docs-paths: [directorypath] asset-paths: [directorypath] function-paths: [directorypath] osi-paths: [directorypath] check-paths: [directorypath] skill-paths: [directorypath] packages-install-path: directorypath clean-targets: [directorypath] query-comment: string require-dbt-version: version-range | [version-range] flags: <global-configs> dbt-cloud: project-id: project_id # Required defer-env-id: environment_id # Optional account_id: account_id # Optional, v2 only; note the underscore, unlike the other dbt-cloud fields account-host: account-host # Defaults to 'cloud.getdbt.com'; Required if use a different Access URL analyses: # Requires the require_corrected_analysis_fqns flag; available starting v1.12 <analysis-configs> exposures: +enabled: true | false quoting: database: true | false schema: true | false identifier: true | false snowflake_ignore_case: true | false # v2-only config. Aligns with Snowflake's session parameter QUOTED_IDENTIFIERS_IGNORE_CASE behavior. # Ignored by dbt v1 and other adapters. metrics: <metric-configs> models: <model-configs> seeds: <seed-configs> semantic-models: <semantic-model-configs> saved-queries: <saved-queries-configs> skills: <skill-configs> snapshots: <snapshot-configs> sources: <source-configs> checks: <check-configs> data_tests: <test-configs> info_schema: version: 1 # Pins which version of the dbt Information Schema the {{ info_schema() }} macro resolves to vars: <variables> on-run-start: sql-statement | [sql-statement] on-run-end: sql-statement | [sql-statement] dispatch: - macro_namespace: packagename search_order: [packagename] restrict-access: true | false functions: <function-configs> ``` #### The `+` prefix dbt demarcates between a folder name and a configuration by using a `+` prefix before the configuration name. The `+` prefix is used for configs *only* and applies to `dbt_project.yml` under the corresponding resource key. It doesn't apply to: * `config()` Jinja macro within a resource file * config property in a `.yml` file. For more information, refer to [Using the `+` prefix](https://docs.getdbt.com/reference/resource-configs/plus-prefix.md). #### Naming convention It's important to follow the correct YAML naming conventions for the configs in your `dbt_project.yml` file to ensure dbt can process them properly. This is especially true for resource types with more than one word. * For the multi-word resource types `saved-queries` and `semantic-models`, use dashes (`-`) in `dbt_project.yml`. Here's an example for [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md#configure-saved-query): dbt\_project.yml ```yml saved-queries: # Use dashes for saved-queries and semantic-models in dbt_project.yml. my_saved_query: +cache: enabled: true ``` * For [data tests](https://docs.getdbt.com/docs/build/data-tests.md) and [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md), use underscores (`_`) everywhere, including in `dbt_project.yml`: dbt\_project.yml ```yml data_tests: # Use underscores for data_tests and unit_tests, even in dbt_project.yml. +store_failures: true unit_tests: +enabled: true ``` * For YAML files other than `dbt_project.yml`, use underscores (`_`) for multi-word resource types. For example, the same saved queries resource in a properties file: models/saved\_queries.yml ```yml saved_queries: # Use underscores outside of dbt_project.yml. - name: saved_query_name ... # Rest of the saved queries configuration. config: cache: enabled: true ``` --- ### dbt_valid_to_current Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). snapshots/schema.yml ```yaml snapshots: - name: my_snapshot config: dbt_valid_to_current: "string" ``` snapshots/\<filename>.sql ```sql {{ config( unique_key='id', strategy='timestamp', updated_at='updated_at', dbt_valid_to_current='string' ) }} ``` dbt\_project.yml ```yml snapshots: <resource-path>: +dbt_valid_to_current: "string" ``` #### Description Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This approach makes it easier to assign a custom date, work in a join, or perform range-based filtering that requires an end date. warning To avoid any unintentional data modification, dbt will *not* automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. Any new records inserted *after* applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (like '9999-12-31'), instead of the default `NULL` value. ##### Considerations * **Date expressions** — Provide a hardcoded date expression compatible with your data platform, such as `to_date('9999-12-31')`. Note that syntax may vary by warehouse (for example, `to_date('YYYY-MM-DD'`) or `date(YYYY, MM, DD)`). * **Jinja limitation** — `dbt_valid_to_current` only accepts static SQL expressions. Jinja expressions (like `{{ var('my_future_date') }}`) are not supported. * **Deferral and `state:modified`** — Changes to `dbt_valid_to_current` are compatible with deferral and `--select state:modified`. When this configuration changes, it'll appear in `state:modified` selections, raising a warning to manually make the necessary snapshot updates. #### Default By default, `dbt_valid_to` is set to `NULL` for current (most recent) records in your snapshot table. This means that these records are still valid and have no defined end date. If you prefer to use a specific value instead of `NULL` for `dbt_valid_to` in current and future records, you can use the `dbt_valid_to_current` configuration option. For example, setting a date in the far future, `9999-12-31`. The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. Use expressions that work within the data platform. #### Impact on snapshot records When you set `dbt_valid_to_current`, it affects how dbt manages the `dbt_valid_to` column in your snapshot table: * **For existing records** — To avoid any unintentional data modification, dbt will *not* automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. * **For new records** — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can manually update existing records in your snapshot table (where `dbt_valid_to` is `NULL`) to match your `dbt_valid_to_current` value. #### Example snapshots/schema.yml ```yaml snapshots: - name: my_snapshot config: strategy: timestamp updated_at: updated_at dbt_valid_to_current: "to_date('9999-12-31')" columns: - name: dbt_valid_from description: The timestamp when the record became valid. - name: dbt_valid_to description: > The timestamp when the record ceased to be valid. For current records, this is either `NULL` or the value specified in `dbt_valid_to_current` (like `'9999-12-31'`). ``` The resulting snapshot table contains the configured dbt\_valid\_to column value: | id | dbt\_scd\_id | dbt\_updated\_at | dbt\_valid\_from | dbt\_valid\_to | | -- | -------------------- | ---------------- | ---------------- | -------------- | | 1 | 60a1f1dbdf899a4dd... | 2024-10-02 ... | 2024-10-02 ... | 9999-12-31 ... | | 2 | b1885d098f8bcff51... | 2024-10-02 ... | 2024-10-02 ... | 9999-12-31 ... | --- ### Defer Defer is a powerful feature that makes it possible to run a subset of (Applies to dbt v1.11 and later) models, tests, or functions in a [sandbox environment](https://docs.getdbt.com/docs/environments-in-dbt.md) without having to first build their upstream parents. This can save time and computational resources when you want to test a small number of models in a large project. [![Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream.](/img/docs/reference/defer-diagram.png?v=2 "Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream.")](#)Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream. Defer requires a manifest from a previous dbt invocation. Provide the path using the `--state flag` or by setting the (Applies to dbt v1.11 and later) `DBT_ENGINE_STATE` environment variable. Together with the `state:` selection method, these features enable "Slim CI". Read more about [state](https://docs.getdbt.com/reference/node-selection/state-selection.md). For some use cases, you can use `dbt clone` to achieve similar functionality. For more details, refer to [clone](https://docs.getdbt.com/reference/commands/clone.md#when-to-use-dbt-clone-instead-of-deferral). It is possible to use separate state for `state:modified` and `--defer`, by passing paths to different manifests to each of the `--state`/(Applies to dbt v1.11 and later) `DBT_ENGINE_STATE` and `--defer-state`/(Applies to dbt v1.11 and later) `DBT_ENGINE_DEFER_STATE`. This enables more granular control in cases where you want to: * Compare against logical state from one environment or past point in time * Defer to applied state from a different environment or point in time If `--defer-state` is not specified, deferral will use the manifest supplied to `--state`. In most cases, you will want to use the same state for both; compare logical changes against production, and also "fail over" to the production environment for unbuilt upstream resources. ##### Usage ```shell dbt run --select [...] --defer --state path/to/artifacts dbt test --select [...] --defer --state path/to/artifacts ``` By default, dbt uses the [`target`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) namespace to resolve `ref` calls. When `--defer` is enabled, dbt resolves `ref`(Applies to dbt v1.11 and later) and `function` calls using the state manifest instead, but only if: 1. The node isn’t among the selected nodes, *and* 2. It doesn’t exist in the database (or `--favor-state` is used). Ephemeral models are never deferred, since they serve as "passthroughs" for other `ref` calls. (Applies to dbt v1.11 and later) [User-defined functions (UDFs)](https://docs.getdbt.com/docs/build/udfs.md) referenced using `{{ function('...') }}` are deferred under the same conditions. When deferred, `function()` resolves to the function definition in the state manifest if the UDF is not selected or not built in the current target. info When using defer, you may be selecting from production datasets, development datasets, or a mix of both. Note that this can yield unexpected results: * If you apply environment-specific limits in development but not in production, you may select more data than expected. * Tests that depend on multiple parents (for example, `relationships`), may run across environments. Deferral requires both `--defer` and `--state` to be set, either by passing flags explicitly or by setting environment variables ((Applies to dbt v1.11 and later) `DBT_ENGINE_DEFER` and `DBT_ENGINE_STATE`). Refer to [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) for more information. ###### Favor state When `--favor-state` is passed, dbt prioritizes node definitions from the `--state` directory. However, this doesn’t apply if the node is also part of the selected nodes. ##### Example In my local development environment, I create all models in my target schema, `dev_alice`. In production, the same models are created in a schema named `prod`. I access the dbt-generated [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) (namely `manifest.json`) from a production run, and copy them into a local directory called `prod-run-artifacts`. ##### run I've been working on `model_b`: models/model\_b.sql ```sql select id, count(*) from {{ ref('model_a') }} group by 1 ``` I want to test my changes. Nothing exists in my development schema, `dev_alice`. ##### Standard run ```shell dbt run --select "model_b" ``` target/run/my\_project/model\_b.sql ```sql create or replace view dev_me.model_b as ( select id, count(*) from dev_alice.model_a group by 1 ) ``` Unless I had previously run `model_a` into this development environment, `dev_alice.model_a` will not exist, thereby causing a database error. ##### Deferred run ```shell dbt run --select "model_b" --defer --state prod-run-artifacts ``` target/run/my\_project/model\_b.sql ```sql create or replace view dev_me.model_b as ( select id, count(*) from prod.model_a group by 1 ) ``` Because `model_a` is unselected, dbt will check to see if `dev_alice.model_a` exists. If it doesn't exist, dbt will resolve all instances of `{{ ref('model_a') }}` to `prod.model_a` instead. ##### test I also have a `relationships` test that establishes referential integrity between `model_a` and `model_b`: models/resources.yml ```yml models: - name: model_b columns: - name: id data_tests: - relationships: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. to: ref('model_a') field: id ``` (This is a simplified example, since all the data in `model_b` already comes from `model_a`) ##### Without defer ```shell dbt test --select "model_b" ``` target/compiled/.../relationships\_model\_b\_id\_\_id\_\_ref\_model\_a\_.sql ```sql select count(*) as validation_errors from ( select id as id from dev_alice.model_b ) as child left join ( select id as id from dev_alice.model_a ) as parent on parent.id = child.id where child.id is not null and parent.id is null ``` The `relationships` test requires both `model_a` and `model_b`. Because I did not build `model_a` in my previous `dbt run`, `dev_alice.model_a` does not exist and this test query fails. ##### With defer ```shell dbt test --select "model_b" --defer --state prod-run-artifacts ``` target/compiled/.../relationships\_model\_b\_id\_\_id\_\_ref\_model\_a\_.sql ```sql select count(*) as validation_errors from ( select id as id from dev_alice.model_b ) as child left join ( select id as id from prod.model_a ) as parent on parent.id = child.id where child.id is not null and parent.id is null ``` dbt will check to see if `dev_alice.model_a` exists. If it doesn't exist, dbt will resolve all instances of `{{ ref('model_a') }}`, including those in schema tests, to use `prod.model_a` instead. The query succeeds. Whether I really want to test for referential integrity across environments is a different question. #### Related docs * [Using defer in dbt](https://docs.getdbt.com/docs/platform/about-defer.md) * [on\_configuration\_change](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) --- ### defer_to_target Self-managed deployments only This configuration applies to self-managed dbt deployments. If you're using the dbt platform, deferral is configured through your environment settings in the UI. `defer_to_target` is part of [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md), which manages model caching and deferral across environments. profiles.yml ```yaml my_project: outputs: dev: type: snowflake # ... dev connection settings staging: type: snowflake # ... staging connection settings uat: type: snowflake # ... uat connection settings defer_to_target: staging prod: type: snowflake # ... prod connection settings target: prod ``` #### Definition `defer_to_target` specifies which target environment dbt State should defer to when resolving unselected upstream nodes and evaluating whether models need to be rebuilt. In self-managed deployments with multiple environments, you can configure each target to defer to a different upstream environment. For example, a `uat` environment can defer to `staging`, while `prod` manages its own state independently. #### Default `prod`. When omitted, all targets defer to the target named `prod`. Use this config if you want to defer to a different target (for example, `staging`), or if your production target is named something other than `prod` (for example, `production`). #### Example ##### Multi-environment deferral chain Configure a pipeline where `uat` defers to `staging`: profiles.yml ```yaml my_project: outputs: dev: type: snowflake account: abc12345 database: ANALYTICS schema: DBT_DEV staging: type: snowflake account: abc12345 database: ANALYTICS schema: DBT_STAGING uat: type: snowflake account: abc12345 database: ANALYTICS schema: DBT_UAT defer_to_target: staging prod: type: snowflake account: abc12345 database: ANALYTICS schema: DBT_PROD target: prod ``` #### Caveats to dbt State without a manifest When dbt State tries to guess where the production version of a relation was built, it re-renders the database and schema names using the configured target from `defer_to_target`. If either of the generated names depend on dynamic data that has changed since the production run, or the node's alias depends on any target-specific logic, then: * The guessed location may not exist, forcing an unnecessary rebuild from scratch. * The guessed location may point to the wrong object, resulting in invalid data. The following examples show scenarios that will cause issues with an `orders` node. Assume the dev profile sets `schema: dbt_developer` and the prod profile sets `schema: analytics`. ##### Environment variables and CLI vars If your schema name depends on an environment variable or a CLI var, dbt State may guess the wrong location because those values differ between production and local runs. ```sql {% macro generate_schema_name(custom_schema_name, node) %} {{ target.schema }}_{{ env_var('DBT_BRANCH', 'main') | replace('/', '_') }} {% endmacro %} ``` * Production runs with `DBT_BRANCH=main`, so the relation is built at `analytics_main.orders`. * When a developer runs with `DBT_BRANCH=feature/x`, dbt State swaps `target.schema` to the prod value (`analytics`) but reads `DBT_BRANCH` from the developer's shell, producing the guess `analytics_feature_x.orders`. * That relation does not exist in production, so no clone is performed and the node rebuilds from scratch. The same issue occurs with CLI vars: ```sql {% macro generate_schema_name(custom_schema_name, node) %} {{ target.schema }}_{{ var('environment', 'dev') }} {% endmacro %} ``` * Production runs `dbt run --vars '{environment: prod}'`, so the relation is built at `analytics_prod.orders`. * When a developer runs `dbt run` with no `--vars`, `var('environment')` falls back to its default (`dev`), so dbt State guesses `analytics_dev.orders`. * That relation does not exist in production. Vars passed at the original prod invocation are not stored anywhere dbt State can recover them. ##### Path-derived names when files move A common pattern is to use a node's directory as its schema: ```sql {% macro generate_schema_name(custom_schema_name, node) %} {{ target.schema }}_{{ node.fqn[-2] }} {% endmacro %} ``` * Production builds `models/finance/orders.sql` into `analytics_finance.orders`. * If a developer reorganizes the project and renames the directory to `models/accounting/`, then runs `dbt run -s orders`, their local manifest records the node's parent directory as `accounting`. dbt State then guesses `analytics_accounting.orders`. * Production is still at `analytics_finance.orders`, the clone misses, and the node rebuilds from scratch. ##### Target-specific `generate_alias_name` logic Unlike database and schema, dbt State currently does not re-render aliases. An override on `generate_alias_name` that varies by target or environment breaks the guess. ```sql {% macro generate_alias_name(custom_alias_name=none, node=none) %} {%- if custom_alias_name -%} {{ custom_alias_name | trim }}_{{ target.name }} {%- else -%} {{ node.name }}_{{ target.name }} {%- endif -%} {% endmacro %} ``` * When `target.name == 'prod'`, production builds `analytics.orders_prod`. * When a developer runs with `target.name == 'dev'`, dbt State swaps the schema component to `analytics` but leaves the alias as-is, producing the guess `analytics.orders_dev`. caution This scenario is the most likely to cause data corruption. If you have custom `generate_alias_name` logic, provide a `manifest.json` to guarantee accurate results. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) --- ### Define configs Learn how to define configurations for your resources in a dbt project Depending on the resource type, you can define configurations in a dbt project and also in an installed package by: (Applies to dbt v1.9 and later) 1. Using a [`config` property](https://docs.getdbt.com/reference/resource-properties/config.md) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses`, `tests/`, and more. 2. From the [`dbt_project.yml` file](https://docs.getdbt.com/reference/dbt_project.yml.md), under the corresponding resource key (`models:`, `snapshots:`, `data_tests:`, and so on) #### Config inheritance The most specific config always takes precedence. This generally follows the order above: an in-file `config()` block --> properties defined in a `.yml` file --> config defined in the project file. Note - Generic data tests work a little differently when it comes to specificity. See [test configs](https://docs.getdbt.com/reference/data-test-configs.md). Within the project file, configurations are also applied hierarchically. The most specific config always takes precedence. In the project file, for example, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model or directory of models, define the [resource path](https://docs.getdbt.com/reference/resource-configs/resource-path.md) as nested dictionary keys. Configurations in your root dbt project have *higher* precedence than configurations in installed packages. This enables you to override the configurations of installed packages, providing more control over your dbt runs. #### Combining configs Most configurations are "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Note that a few configs have different merge behavior: * [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md) are additive. If a model has some tags configured in `dbt_project.yml`, and more tags are applied in its `.sql` file, the final set of tags will include all of them. * When using the [`freshness`](https://docs.getdbt.com/reference/resource-configs/freshness.md) config, a more specific key-value pair replaces a less specific value with the same key. * [`pre-hook` and `post-hook`](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) are also additive. * [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) dictionaries are **shallow-merged**, meaning dbt merges only the top-level keys and does not look inside nested dictionaries. When the same top-level key appears at more than one level, the more specific value replaces the less specific one entirely — even if that value is itself a dictionary. Top-level keys that appear at only one level are kept. For example, if `dbt_project.yml` sets: ```yaml +meta: {owner: "alice", dagster: {automation_condition: "eager"}} ``` and a model sets: ```yaml meta: {dagster: {asset_key: "my_key"}} ``` the result is: ```yaml {owner: "alice", dagster: {asset_key: "my_key"}} ``` `owner` is kept because it appears at only one level. `dagster` appears at both levels, so the more specific value replaces it as a whole — and because the merge never looks inside `dagster`, the nested `automation_condition` is lost. A deep (recursive) merge would instead combine the nested keys, producing `dagster: {automation_condition: "eager", asset_key: "my_key"}`. `meta` doesn't do this — nested dictionaries are replaced, not merged. ##### Which config wins across levels For clobbering and merging configurations that are inherited from multiple levels, the general rules are: * Node-level configs (more specific) clobber project-level configs (less specific). * For sources, table-level configs (more specific) clobber source-level configs (less specific). * The root project's configuration in `dbt_project.yml` clobbers configuration within package files. This is so that users can control the behavior of packages they are installing using `dbt deps` without needing to edit the code in those package files directly. #### The `+` prefix dbt demarcates between a folder name and a configuration by using a `+` prefix before the configuration name. The `+` prefix is used for configs *only* and applies to `dbt_project.yml` under the corresponding resource key. It doesn't apply to: * `config()` Jinja macro within a resource file * config property in a `.yml` file. For more information, refer to [Using the `+` prefix](https://docs.getdbt.com/reference/resource-configs/plus-prefix.md). #### Example Here's an example that defines both `sources` and `models` for a project: models/jaffle\_shop.yml ```yml version: 2 sources: - name: raw_jaffle_shop description: A replica of the postgres database used to power the jaffle_shop app. tables: - name: customers columns: - name: id description: Primary key of the table data_tests: - unique - not_null - name: orders columns: - name: id description: Primary key of the table data_tests: - unique - not_null - name: user_id description: Foreign key to customers - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] models: - name: stg_jaffle_shop__customers # Must match the filename of a model -- including case sensitivity. config: tags: ['pii'] columns: - name: customer_id data_tests: - unique - not_null - name: stg_jaffle_shop__orders config: materialized: view columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] config: severity: warn ``` #### Related documentation You can find an exhaustive list of each supported property and config, broken down by resource type: * Model [properties](https://docs.getdbt.com/reference/model-properties.md) and [configs](https://docs.getdbt.com/reference/model-configs.md) * Source [properties](https://docs.getdbt.com/reference/source-properties.md) and [configs](https://docs.getdbt.com/reference/source-configs.md) * Seed [properties](https://docs.getdbt.com/reference/seed-properties.md) and [configs](https://docs.getdbt.com/reference/seed-configs.md) * Snapshot [properties](https://docs.getdbt.com/reference/snapshot-properties.md) * Analysis [properties](https://docs.getdbt.com/reference/analysis-properties.md) * Macro [properties](https://docs.getdbt.com/reference/macro-properties.md) * Exposure [properties](https://docs.getdbt.com/reference/exposure-properties.md) #### FAQs Does my \`.yml\` file containing tests and descriptions need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. If I can name these files whatever I'd like, what should I name them? It's up to you! Here's a few options: * Default to the existing terminology: `schema.yml` (though this does make it hard to find the right file over time) * Use the same name as your directory (assuming you're using sensible names for your directories) * If you test and document one model (or seed, snapshot, macro etc.) per file, you can give it the same name as the model (or seed, snapshot, macro etc.) Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). Should I use separate files to declare resource properties, or one large file? It's up to you: * Some folks find it useful to have one file per model (or source / snapshot / seed etc) * Some find it useful to have one per directory, documenting and testing multiple models in one file Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). Can I add tests and descriptions in a SQL config block? dbt has the ability to define node configs in YAML files, in addition to `config()` blocks and `dbt_project.yml`. But the reverse isn't always true: there are some things in `.yml` files that can *only* be defined there. Certain properties are special, because: * They have a unique Jinja rendering context * They create new project resources * They don't make sense as hierarchical configuration * They're older properties that haven't yet been redefined as configs These properties are: * [`description`](https://docs.getdbt.com/reference/resource-properties/description.md) * [`tests`](https://docs.getdbt.com/reference/resource-properties/data-tests.md) * [`docs`](https://docs.getdbt.com/reference/resource-configs/docs.md) * `columns` * [`quote`](https://docs.getdbt.com/reference/resource-properties/columns.md#quote) * [`source` properties](https://docs.getdbt.com/reference/source-properties.md) (e.g. `loaded_at_field`, `freshness`) * [`exposure` properties](https://docs.getdbt.com/reference/exposure-properties.md) (e.g. `type`, `maturity`) * [`macro` properties](https://docs.getdbt.com/reference/resource-properties/arguments.md) (e.g. `arguments`) Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future Can I use a YAML file extension? No. At present, dbt will only search for files with a `.yml` file extension. In a future release of dbt, dbt will also search for files with a `.yaml` file extension. #### Troubleshooting common errors  Invalid test config given in \[model name] This error occurs when your `.yml` file does not conform to the structure expected by dbt. A full error message might look like: ```text * Invalid test config given in models/schema.yml near {'namee': 'event', ...} Invalid arguments passed to "UnparsedNodeUpdate" instance: 'name' is a required property, Additional properties are not allowed ('namee' was unexpected) ``` While verbose, an error like this should help you track down the issue. Here, the `name` field was provided as `namee` by accident. To fix this error, ensure that your `.yml` conforms to the expected structure described in this guide.  Invalid syntax in your schema.yml file If your `.yml` file is not valid yaml, then dbt will show you an error like this: ```text Runtime Error Syntax error near line 6 ------------------------------ 5 | - name: events 6 | description; "A table containing clickstream events from the marketing website" 7 | Raw Error: ------------------------------ while scanning a simple key in "<unicode string>", line 6, column 5: description; "A table containing clickstream events from the marketing website" ^ ``` This error occurred because a semicolon (`;`) was accidentally used instead of a colon (`:`) after the `description` field. To resolve issues like this, find the `.yml` file referenced in the error message and fix any syntax errors present in the file. There are online YAML validators that can be helpful here, but please be mindful of submitting sensitive information to third-party applications! --- ### Define properties Learn how to define properties for your resources in a properties.yml file In dbt, you can use `properties.yml` files to define properties for resources. You can declare properties in `.yml` files, in the same directory as your resources. You can name these files `whatever_you_want.yml` and nest them arbitrarily in sub-folders within each directory. We highly recommend that you define properties in dedicated paths alongside the resources they're describing. info ###### schema.yml files Previous versions of the docs referred to these as `schema.yml` files — we've moved away from that terminology since the word `schema` is used to mean other things when talking about databases, and people often thought that you *had* to name these files `schema.yml`. Instead, we now refer to these files as `properties.yml` files. (Of course, you're still free to name your files `schema.yml`) ##### Which properties are *not* also configs? In dbt, you can define node configs in `properties.yml` files, in addition to `config()` blocks and `dbt_project.yml`. However, some special properties can only be defined in the `.yml` file and you cannot configure them using `config()` blocks or the `dbt_project.yml` file: Certain properties are special, because: * They have a unique Jinja rendering context * They create new project resources * They don't make sense as hierarchical configuration * They're older properties that haven't yet been redefined as configs These properties are: * [`columns`](https://docs.getdbt.com/reference/resource-properties/columns.md) * [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md) * [`description`](https://docs.getdbt.com/reference/resource-properties/description.md) * [`quote`](https://docs.getdbt.com/reference/resource-properties/columns.md#quote) * [`source` properties](https://docs.getdbt.com/reference/source-properties.md) (for example, `loaded_at_field`) * [`exposure` properties](https://docs.getdbt.com/reference/exposure-properties.md) (for example, `type`, `maturity`) * Note that while most exposure properties must be configured directly in `properties.yml` files, you can set the [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) config at the [project level](https://docs.getdbt.com/reference/exposure-properties.md#project-level-configs) in the`dbt_project.yml` file. * [`macro` properties](https://docs.getdbt.com/reference/macro-properties.md) (for example, `arguments`) * [`tests`](https://docs.getdbt.com/reference/resource-properties/data-tests.md) * [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md) #### Example Here's an example that defines both `sources` and `models` for a project: models/jaffle\_shop.yml ```yml version: 2 sources: - name: raw_jaffle_shop description: A replica of the postgres database used to power the jaffle_shop app. tables: - name: customers columns: - name: id description: Primary key of the table data_tests: - unique - not_null - name: orders columns: - name: id description: Primary key of the table data_tests: - unique - not_null - name: user_id description: Foreign key to customers - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] models: - name: stg_jaffle_shop__customers # Must match the filename of a model -- including case sensitivity. config: tags: ['pii'] columns: - name: customer_id data_tests: - unique - not_null - name: stg_jaffle_shop__orders config: materialized: view columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] config: severity: warn ``` #### Related documentation You can find an exhaustive list of each supported property and config, broken down by resource type: * Model [properties](https://docs.getdbt.com/reference/model-properties.md) and [configs](https://docs.getdbt.com/reference/model-configs.md) * Source [properties](https://docs.getdbt.com/reference/source-properties.md) and [configs](https://docs.getdbt.com/reference/source-configs.md) * Seed [properties](https://docs.getdbt.com/reference/seed-properties.md) and [configs](https://docs.getdbt.com/reference/seed-configs.md) * Snapshot [properties](https://docs.getdbt.com/reference/snapshot-properties.md) * Analysis [properties](https://docs.getdbt.com/reference/analysis-properties.md) * Macro [properties](https://docs.getdbt.com/reference/macro-properties.md) * Exposure [properties](https://docs.getdbt.com/reference/exposure-properties.md) #### FAQs Does my \`.yml\` file containing tests and descriptions need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. If I can name these files whatever I'd like, what should I name them? It's up to you! Here's a few options: * Default to the existing terminology: `schema.yml` (though this does make it hard to find the right file over time) * Use the same name as your directory (assuming you're using sensible names for your directories) * If you test and document one model (or seed, snapshot, macro etc.) per file, you can give it the same name as the model (or seed, snapshot, macro etc.) Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). Should I use separate files to declare resource properties, or one large file? It's up to you: * Some folks find it useful to have one file per model (or source / snapshot / seed etc) * Some find it useful to have one per directory, documenting and testing multiple models in one file Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). Can I add tests and descriptions in a SQL config block? dbt has the ability to define node configs in YAML files, in addition to `config()` blocks and `dbt_project.yml`. But the reverse isn't always true: there are some things in `.yml` files that can *only* be defined there. Certain properties are special, because: * They have a unique Jinja rendering context * They create new project resources * They don't make sense as hierarchical configuration * They're older properties that haven't yet been redefined as configs These properties are: * [`description`](https://docs.getdbt.com/reference/resource-properties/description.md) * [`tests`](https://docs.getdbt.com/reference/resource-properties/data-tests.md) * [`docs`](https://docs.getdbt.com/reference/resource-configs/docs.md) * `columns` * [`quote`](https://docs.getdbt.com/reference/resource-properties/columns.md#quote) * [`source` properties](https://docs.getdbt.com/reference/source-properties.md) (e.g. `loaded_at_field`, `freshness`) * [`exposure` properties](https://docs.getdbt.com/reference/exposure-properties.md) (e.g. `type`, `maturity`) * [`macro` properties](https://docs.getdbt.com/reference/resource-properties/arguments.md) (e.g. `arguments`) Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future Can I use a YAML file extension? No. At present, dbt will only search for files with a `.yml` file extension. In a future release of dbt, dbt will also search for files with a `.yaml` file extension. #### Troubleshooting common errors  Invalid test config given in \[model name] This error occurs when your `.yml` file does not conform to the structure expected by dbt. A full error message might look like: ```text * Invalid test config given in models/schema.yml near {'namee': 'event', ...} Invalid arguments passed to "UnparsedNodeUpdate" instance: 'name' is a required property, Additional properties are not allowed ('namee' was unexpected) ``` While verbose, an error like this should help you track down the issue. Here, the `name` field was provided as `namee` by accident. To fix this error, ensure that your `.yml` conforms to the expected structure described in this guide.  Invalid syntax in your schema.yml file If your `.yml` file is not valid yaml, then dbt will show you an error like this: ```text Runtime Error Syntax error near line 6 ------------------------------ 5 | - name: events 6 | description; "A table containing clickstream events from the marketing website" 7 | Raw Error: ------------------------------ while scanning a simple key in "<unicode string>", line 6, column 5: description; "A table containing clickstream events from the marketing website" ^ ``` This error occurred because a semicolon (`;`) was accidentally used instead of a colon (`:`) after the `description` field. To resolve issues like this, find the `.yml` file referenced in the error message and fix any syntax errors present in the file. There are online YAML validators that can be helpful here, but please be mindful of submitting sensitive information to third-party applications! --- ### Defining a database source property models/\<filename>.yml ```yml sources: - name: <source_name> database: <database_name> tables: - name: <table_name> - ... ``` #### Definition The database that your source is stored in. Note that to use this parameter, your warehouse must allow cross-database queries. ###### BigQuery terminology If you're using BigQuery, use the *project* name as the `database:` property. #### Default By default, dbt will search in your target database (i.e. the database that you are creating tables and views). #### Examples ##### Define a source that is stored in the `raw` database models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw tables: - name: orders - name: customers ``` --- ### Defining a schema source property models/\<filename>.yml ```yml sources: - name: <source_name> database: <database_name> schema: <schema_name> tables: - name: <table_name> - ... ``` #### Definition The schema name as stored in the database. This parameter is useful if you want to use a [source](https://docs.getdbt.com/reference/source-properties.md) name that differs from the schema name. BigQuery terminology If you're using BigQuery, use the *dataset* name as the `schema` property. #### Default By default, dbt will use the source's `name` parameter as the schema name. #### Examples ##### Use a simpler name for a source schema than the one in your database models/\<filename>.yml ```yml sources: - name: jaffle_shop schema: postgres_backend_public_schema tables: - name: orders ``` In a downstream model: ```sql select * from {{ source('jaffle_shop', 'orders') }} ``` Will get compiled to: ```sql select * from postgres_backend_public_schema.orders ``` --- ### delimiter Did you know... Available from dbt v1.7 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### Definition You can use this optional seed configuration to customize how you separate values in a [seed](https://docs.getdbt.com/docs/build/seeds.md) by providing the one-character string. * The delimiter defaults to a comma when not specified. * Explicitly set the `delimiter` configuration value if you want seed files to use a different delimiter, such as "|" or ";". #### Usage Specify a delimiter in your `dbt_project.yml` file to customize the global separator for all seed values: dbt\_project.yml ```yml seeds: <project_name>: +delimiter: "|" # default project delimiter for seeds will be "|" <seed_subdirectory>: +delimiter: "," # delimiter for seeds in seed_subdirectory will be "," ``` Or use a custom delimiter to override the values for a specific seed: seeds/properties.yml ```yml seeds: - name: <seed_name> config: delimiter: "|" ``` #### Examples For a project with: * `name: jaffle_shop` in the `dbt_project.yml` file * `seed-paths: ["seeds"]` in the `dbt_project.yml` file ##### Use a custom delimiter to override global values You can set a default behavior for all seeds with an exception for one seed, `seed_a`, which uses a comma: dbt\_project.yml ```yml seeds: jaffle_shop: +delimiter: "|" # default delimiter for seeds in jaffle_shop project will be "|" seed_a: +delimiter: "," # delimiter for seed_a will be "," ``` Your corresponding seed files would be formatted like this: seeds/my\_seed.csv ```text col_a|col_b|col_c 1|2|3 4|5|6 ... ``` seeds/seed\_a.csv ```text name,id luna,1 doug,2 ... ``` Or you can configure custom behavior for one seed. The `country_codes` uses the ";" delimiter: seeds/properties.yml ```yml seeds: - name: country_codes config: delimiter: ";" ``` The `country_codes` seed file would be formatted like this: seeds/country\_codes.csv ```text country_code;country_name US;United States CA;Canada GB;United Kingdom ... ``` --- ### DeltaStream resource configurations #### Supported materializations DeltaStream supports several unique materialization types that align with its streaming processing capabilities: ##### Standard materializations | Materialization | Description | | ------------------- | --------------------------------------------------------------------------------- | | `ephemeral` | This materialization uses common table expressions in DeltaStream under the hood. | | `table` | Traditional batch table materialization | | `materialized_view` | Continuously updated view that automatically refreshes as underlying data changes | ##### Streaming materializations | Materialization | Description | | --------------- | -------------------------------------------------------------- | | `stream` | Pure streaming transformation that processes data in real-time | | `changelog` | Change data capture (CDC) stream that tracks changes in data | ##### Infrastructure materializations | Materialization | Description | | ------------------- | ---------------------------------------------------- | | `store` | External system connection (Kafka, PostgreSQL, etc.) | | `entity` | Entity definition within a store | | `database` | Database definition | | `compute_pool` | Compute pool definition for resource management | | `function` | User-defined functions (UDFs) in Java | | `function_source` | JAR file sources for UDFs | | `descriptor_source` | Protocol buffer schema sources | | `schema_registry` | Schema registry connections (Confluent, and so on.) | #### SQL model configurations ##### Table materialization Creates a traditional batch table for aggregated data: **Project YAML file configuration:** ```yaml models: <resource-path>: +materialized: table ``` **SQL configuration:** ```sql {{ config(materialized = "table") }} SELECT date, SUM(amount) as daily_total FROM {{ ref('transactions') }} GROUP BY date ``` ##### Stream materialization Creates a continuous streaming transformation: **Project YAML file configuration:** ```yaml models: <resource-path>: +materialized: stream +parameters: topic: 'stream_topic' value.format: 'json' key.format: 'primitive' key.type: 'VARCHAR' timestamp: 'event_time' ``` **SQL configuration:** ```sql {{ config( materialized='stream', parameters={ 'topic': 'purchase_events', 'value.format': 'json', 'key.format': 'primitive', 'key.type': 'VARCHAR', 'timestamp': 'event_time' } ) }} SELECT event_time, user_id, action FROM {{ ref('source_stream') }} WHERE action = 'purchase' ``` ###### Stream configuration options | Option | Description | Required? | | -------------- | --------------------------------------------------------------------------------- | --------- | | `materialized` | How the model will be materialized. Must be `stream` to create a streaming model. | Required | | `topic` | The topic name for the stream output. | Required | | `value.format` | Format for the stream values (like 'json', 'avro'). | Required | | `key.format` | Format for the stream keys (like 'primitive', 'json'). | Optional | | `key.type` | Data type for the stream keys (like 'VARCHAR', 'BIGINT'). | Optional | | `timestamp` | Column name to use as the event timestamp. | Optional | ##### Changelog materialization Captures changes in the data stream: **Project YAML file configuration:** ```yaml models: <resource-path>: +materialized: changelog +parameters: topic: 'changelog_topic' value.format: 'json' +primary_key: [column_name] ``` **SQL configuration:** ```sql {{ config( materialized='changelog', parameters={ 'topic': 'order_updates', 'value.format': 'json' }, primary_key=['order_id'] ) }} SELECT order_id, status, updated_at FROM {{ ref('orders_stream') }} ``` ###### Changelog configuration options | Option | Description | Required? | | -------------- | ------------------------------------------------------------------------------------ | --------- | | `materialized` | How the model will be materialized. Must be `changelog` to create a changelog model. | Required | | `topic` | The topic name for the changelog output. | Required | | `value.format` | Format for the changelog values (like 'json', 'avro'). | Required | | `primary_key` | List of column names that uniquely identify rows for change tracking. | Required | ##### Materialized view Creates a continuously updated view: **SQL configuration:** ```sql {{ config(materialized='materialized_view') }} SELECT product_id, COUNT(*) as purchase_count FROM {{ ref('purchase_events') }} GROUP BY product_id ``` #### YAML-only resource configurations DeltaStream supports two types of model definitions for infrastructure components: 1. **Managed Resources (Models)** - Automatically included in the dbt DAG 2. **Unmanaged Resources (Sources)** - Created on-demand using specific macros ##### Should you use managed or unmanaged resources? * Use managed resources if you plan to recreate all the infrastructure in different environments and/or use graph operators to execute only the creation of specific resources and downstream transformations. * Otherwise, it might be simpler to use unmanaged resources to avoid placeholder files. ##### Managed resources (models) Managed resources are automatically included in the dbt DAG and defined as models: ```yaml version: 2 models: - name: my_kafka_store config: materialized: store parameters: type: KAFKA access_region: "AWS us-east-1" uris: "kafka.broker1.url:9092,kafka.broker2.url:9092" tls.ca_cert_file: "@/certs/us-east-1/self-signed-kafka-ca.crt" - name: ps_store config: materialized: store parameters: type: POSTGRESQL access_region: "AWS us-east-1" uris: "postgresql://mystore.com:5432/demo" postgres.username: "user" postgres.password: "password" - name: user_events_stream config: materialized: stream columns: event_time: type: TIMESTAMP not_null: true user_id: type: VARCHAR action: type: VARCHAR parameters: topic: 'user_events' value.format: 'json' key.format: 'primitive' key.type: 'VARCHAR' timestamp: 'event_time' - name: order_changes config: materialized: changelog columns: order_id: type: VARCHAR not_null: true status: type: VARCHAR updated_at: type: TIMESTAMP primary_key: - order_id parameters: topic: 'order_updates' value.format: 'json' - name: pv_kinesis config: materialized: entity store: kinesis_store parameters: 'kinesis.shards': 3 - name: my_compute_pool config: materialized: compute_pool parameters: 'compute_pool.size': 'small' 'compute_pool.timeout_min': 5 - name: my_function_source config: materialized: function_source parameters: file: '@/path/to/my-functions.jar' description: 'Custom utility functions' - name: my_descriptor_source config: materialized: descriptor_source parameters: file: '@/path/to/schemas.desc' description: 'Protocol buffer schemas for data structures' - name: my_custom_function config: materialized: function parameters: args: - name: input_text type: VARCHAR returns: VARCHAR language: JAVA source.name: 'my_function_source' class.name: 'com.example.TextProcessor' - name: my_schema_registry config: materialized: schema_registry parameters: type: "CONFLUENT" access_region: "AWS us-east-1" uris: "https://url.to.schema.registry.listener:8081" 'confluent.username': 'fake_username' 'confluent.password': 'fake_password' 'tls.client.cert_file': '@/path/to/tls/client_cert_file' 'tls.client.key_file': '@/path/to/tls_key' ``` **Note:** Due to current dbt limitations, managed YAML-only resources require a placeholder .sql file that doesn't contain a SELECT statement. For example, create `my_kafka_store.sql` with: ```sql -- Placeholder ``` ##### Unmanaged resources (sources) Unmanaged resources are defined as sources and created on-demand using specific macros: ```yaml version: 2 sources: - name: infrastructure tables: - name: my_kafka_store config: materialized: store parameters: type: KAFKA access_region: "AWS us-east-1" uris: "kafka.broker1.url:9092,kafka.broker2.url:9092" tls.ca_cert_file: "@/certs/us-east-1/self-signed-kafka-ca.crt" - name: ps_store config: materialized: store parameters: type: POSTGRESQL access_region: "AWS us-east-1" uris: "postgresql://mystore.com:5432/demo" postgres.username: "user" postgres.password: "password" - name: user_events_stream config: materialized: stream columns: event_time: type: TIMESTAMP not_null: true user_id: type: VARCHAR action: type: VARCHAR parameters: topic: 'user_events' value.format: 'json' key.format: 'primitive' key.type: 'VARCHAR' timestamp: 'event_time' - name: order_changes config: materialized: changelog columns: order_id: type: VARCHAR not_null: true status: type: VARCHAR updated_at: type: TIMESTAMP primary_key: - order_id parameters: topic: 'order_updates' value.format: 'json' - name: pv_kinesis config: materialized: entity store: kinesis_store parameters: 'kinesis.shards': 3 - name: compute_pool_small config: materialized: compute_pool parameters: 'compute_pool.size': 'small' 'compute_pool.timeout_min': 5 - name: my_function_source config: materialized: function_source parameters: file: '@/path/to/my-functions.jar' description: 'Custom utility functions' - name: my_descriptor_source config: materialized: descriptor_source parameters: file: '@/path/to/schemas.desc' description: 'Protocol buffer schemas for data structures' - name: my_custom_function config: materialized: function parameters: args: - name: input_text type: VARCHAR returns: VARCHAR language: JAVA source.name: 'my_function_source' class.name: 'com.example.TextProcessor' - name: my_schema_registry config: materialized: schema_registry parameters: type: "CONFLUENT" access_region: "AWS us-east-1" uris: "https://url.to.schema.registry.listener:8081" 'confluent.username': 'fake_username' 'confluent.password': 'fake_password' 'tls.client.cert_file': '@/path/to/tls/client_cert_file' 'tls.client.key_file': '@/path/to/tls_key' ``` To create unmanaged resources: ```bash # Create all sources dbt run-operation create_sources # Create a specific source dbt run-operation create_source_by_name --args '{source_name: infrastructure}' ``` #### Store configurations ##### Kafka store ```yaml - name: my_kafka_store config: materialized: store parameters: type: KAFKA access_region: "AWS us-east-1" uris: "kafka.broker1.url:9092,kafka.broker2.url:9092" tls.ca_cert_file: "@/certs/us-east-1/self-signed-kafka-ca.crt" ``` ##### PostgreSQL store ```yaml - name: postgres_store config: materialized: store parameters: type: POSTGRESQL access_region: "AWS us-east-1" uris: "postgresql://mystore.com:5432/demo" postgres.username: "user" postgres.password: "password" ``` #### Entity configuration ```yaml - name: kinesis_entity config: materialized: entity store: kinesis_store parameters: 'kinesis.shards': 3 ``` #### Compute pool configuration ```yaml - name: processing_pool config: materialized: compute_pool parameters: 'compute_pool.size': 'small' 'compute_pool.timeout_min': 5 ``` #### Referencing resources ##### Managed resources Use the standard `ref()` function: ```sql select * from {{ ref('my_kafka_stream') }} ``` ##### Unmanaged resources Use the `source()` function: ```sql SELECT * FROM {{ source('infrastructure', 'user_events_stream') }} ``` #### Seeds Load CSV data into existing DeltaStream entities using the `seed` materialization. Unlike traditional dbt seeds that create new tables, DeltaStream seeds insert data into pre-existing entities. ##### Configuration Seeds must be configured in YAML with the following properties: **Required:** * `entity`: The name of the target entity to insert data into **Optional:** * `store`: The name of the store containing the entity (omit if entity is not in a store) * `with_params`: A dictionary of parameters for the WITH clause * `quote_columns`: Control which columns get quoted. Default: `false` (no columns quoted). Can be: * `true`: Quote all columns * `false`: Quote no columns (default) * `string`: If set to `'*'`, quote all columns * `list`: List of column names to quote ##### Example configuration **With Store (quoting enabled):** ```yaml # seeds.yml version: 2 seeds: - name: user_data_with_store_quoted config: entity: 'user_events' store: 'kafka_store' with_params: kafka.topic.retention.ms: '86400000' partitioned: true quote_columns: true # Quote all columns ``` ##### Usage 1. Place CSV files in your `seeds/` directory 2. Configure seeds in YAML with the required `entity` parameter 3. Optionally specify `store` if the entity is in a store 4. Run `dbt seed` to load the data Important The target entity must already exist in DeltaStream before running seeds. Seeds only insert data, they do not create entities. #### Function and source materializations DeltaStream supports user-defined functions (UDFs) and their dependencies through specialized materializations. ##### File attachment support The adapter provides seamless file attachment for function sources and descriptor sources: * **Standardized Interface**: Common file handling logic for both function sources and descriptor sources * **Path Resolution**: Supports both absolute paths and relative paths (including `@` syntax for project-relative paths) * **Automatic Validation**: Files are validated for existence and accessibility before attachment ##### Function source Creates a function source from a JAR file containing Java functions: **SQL configuration:** ```sql {{ config( materialized='function_source', parameters={ 'file': '@/path/to/my-functions.jar', 'description': 'Custom utility functions' } ) }} SELECT 1 as placeholder ``` ##### Descriptor source Creates a descriptor source from compiled protocol buffer descriptor files: **SQL configuration:** ```sql {{ config( materialized='descriptor_source', parameters={ 'file': '@/path/to/schemas.desc', 'description': 'Protocol buffer schemas for data structures' } ) }} SELECT 1 as placeholder ``` Note Descriptor sources require compiled `.desc` files, not raw `.proto` files. Compile your protobuf schemas using: ```bash protoc --descriptor_set_out=schemas/my_schemas.desc schemas/my_schemas.proto ``` ##### Function Creates a user-defined function that references a function source: **SQL configuration:** ```sql {{ config( materialized='function', parameters={ 'args': [ {'name': 'input_text', 'type': 'VARCHAR'} ], 'returns': 'VARCHAR', 'language': 'JAVA', 'source.name': 'my_function_source', 'class.name': 'com.example.TextProcessor' } ) }} SELECT 1 as placeholder ``` ##### Schema registry Creates a schema registry connection: **SQL configuration:** ```sql {{ config( materialized='schema_registry', parameters={ 'type': 'CONFLUENT', 'access_region': 'AWS us-east-1', 'uris': 'https://url.to.schema.registry.listener:8081', 'confluent.username': 'fake_username', 'confluent.password': 'fake_password', 'tls.client.cert_file': '@/path/to/tls/client_cert_file', 'tls.client.key_file': '@/path/to/tls_key' } ) }} SELECT 1 as placeholder ``` #### Query management macros DeltaStream dbt adapter provides macros to help you manage and terminate running queries directly from dbt. ##### List all queries The `list_all_queries` macro displays all queries currently known to DeltaStream, including their state, owner, and SQL: ```bash dbt run-operation list_all_queries ``` ##### Describe query Use the `describe_query` macro to check the logs and details of a specific query: ```bash dbt run-operation describe_query --args '{query_id: "<QUERY_ID>"}' ``` ##### Terminate a specific query Use the `terminate_query` macro to terminate a query by its ID: ```bash dbt run-operation terminate_query --args '{query_id: "<QUERY_ID>"}' ``` ##### Terminate all running queries Use the `terminate_all_queries` macro to terminate all currently running queries: ```bash dbt run-operation terminate_all_queries ``` ##### Restart a query Use the `restart_query` macro to restart a failed query by its ID: ```bash dbt run-operation restart_query --args '{query_id: "<QUERY_ID>"}' ``` #### Application macro ##### Execute multiple statements as a unit The `application` macro allows you to execute multiple DeltaStream SQL statements as a single unit of work with all-or-nothing semantics: ```bash dbt run-operation application --args '{ application_name: "my_data_pipeline", statements: [ "USE DATABASE my_db", "CREATE STREAM user_events WITH (topic='"'"'events'"'"', value.format='"'"'json'"'"')", "CREATE MATERIALIZED VIEW user_counts AS SELECT user_id, COUNT(*) FROM user_events GROUP BY user_id" ] }' ``` #### Troubleshooting ##### Function source readiness If you encounter "function source is not ready" errors when creating functions: 1. **Automatic Retry**: The adapter automatically retries function creation with exponential backoff 2. **Timeout Configuration**: The default 30-second timeout can be extended if needed for large JAR files 3. **Dependency Order**: Ensure function sources are created before dependent functions 4. **Manual Retry**: If automatic retry fails, wait a few minutes and retry the operation ##### File attachment issues For problems with file attachments in function sources and descriptor sources: 1. **File Paths**: Use `@/path/to/file` syntax for project-relative paths 2. **File Types**: * Function sources require `.jar` files * Descriptor sources require compiled `.desc` files (not `.proto`) 3. **File Validation**: The adapter validates file existence before attempting attachment 4. **Compilation**: For descriptor sources, ensure protobuf files are compiled: ```bash protoc --descriptor_set_out=output.desc input.proto ``` --- ### Deprecation & warnings overview When using dbt, you may see warnings or other changes that need your attention. These changes help us move forward with the latest version of dbt and improve the experience for all users. Use this page to understand the different types of changes, what to do, and where to find more information. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/reference/deprecations.md) ###### [Deprecations](https://docs.getdbt.com/reference/deprecations.md) [Features in your project code (models, YAML, macros) that still work but will be removed.](https://docs.getdbt.com/reference/deprecations.md) <br /> <br /> [**Impact:** Currently warnings; will cause errors in future versions.](https://docs.getdbt.com/reference/deprecations.md) <br /> <br /> [**Action:** Update your project code to use the new syntax.](https://docs.getdbt.com/reference/deprecations.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) ###### [Behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) [Settings in your dbt\_project.yml file that let you opt in or out of new behaviors during migration periods.](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) <br /> <br /> [**Impact:** Controls whether dbt uses old or new behavior; defaults change over time.](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) <br /> <br /> [**Action:** Set flags to control timing of adoption.](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) [![](/img/icons/square-terminal.svg)](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) ###### [Deprecated CLI flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) [Command-line flags passed to dbt commands that are being removed in dbt v2.](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) <br /> <br /> [**Impact:** Some ignored (with warnings); **--models** flag will error in dbt v2.](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) <br /> <br /> [**Action:** Update job definitions and scripts to remove or replace these flags.](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) #### Preparing for dbt v2 If you're upgrading to dbt v2, you should: * \[ ] Resolve all [deprecations](https://docs.getdbt.com/reference/deprecations.md) to avoid causing errors in dbt v2. * \[ ] Review [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) and the **Removed** column to see which flags are always-on in dbt v2 versus which remain configurable. * \[ ] Update [deprecated CLI flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) to avoid errors in dbt v2. #### Related docs * [Full deprecations list](https://docs.getdbt.com/reference/deprecations.md) * [Behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) * [Upgrading to dbt v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) * [Events and logging](https://docs.getdbt.com/reference/events-logging.md) --- ### deprecation_date models/\<schema>.yml ```yml models: - name: my_model description: deprecated deprecation_date: 1999-01-01 00:00:00.00+00:00 ``` models/\<schema>.yml ```yml models: - name: my_model description: deprecating in the future deprecation_date: 2999-01-01 00:00:00.00+00:00 ``` #### Definition The deprecation date of the model is formatted as a date, optionally with a timezone offset. Supported RFC 3339 formats include: * `YYYY-MM-DD hh:mm:ss.sss±hh:mm` * `YYYY-MM-DD hh:mm:ss.sss` * `YYYY-MM-DD` When `deprecation_date` does not include an offset from UTC, then it is interpreted as being in the system time zone of the dbt execution environment. #### Explanation ##### Purpose Declaring a `deprecation_date` for a dbt model provides a mechanism to communicate plans and timelines for long-term support and maintenance and to facilitate change management. Setting a `deprecation_date` works well in conjunction with other [model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) features like [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md), but can also be used independently from them. info If your model has an [enforced contract](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md), you cannot delete the model until after the `deprecation_date` has passed. dbt doesn't allow deleting models with enforced contracts before their `deprecation_date` to protect downstream consumers. If you try to delete a versioned model before its `deprecation_date`, dbt will raise an error during development runs and cause jobs to fail. ##### Warning messages When a project references a model that's slated for deprecation or the deprecation date has passed, a warning is generated. If it's a versioned model, with a newer version available, then the warning says so. This added bit of cross-team communication, from producers to consumers, is an advantage of using dbt's built-in functionality around model versions to facilitate migrations. Additionally, [`WARN_ERROR_OPTIONS`](https://docs.getdbt.com/reference/global-configs/warnings.md) gives a mechanism whereby users can promote these warnings to actual runtime errors: | Warning | Scenario | Affected projects | | ------------------------------ | -------------------------------------------------- | ---------------------- | | `DeprecatedModel` | Parsing a project that defines a deprecated model | Producer | | `DeprecatedReference` | Referencing a model with a past deprecation date | Producer and consumers | | `UpcomingReferenceDeprecation` | Referencing a model with a future deprecation date | Producer and consumers | **Example** Example output for an `UpcomingReferenceDeprecation` warning: ```text $ dbt parse 15:48:14 Running with dbt=1.6.0 15:48:14 Registered adapter: postgres=1.6.0 15:48:14 [WARNING]: While compiling 'my_model_ref': Found a reference to my_model, which is slated for deprecation on '2038-01-19T03:14:07-00:00'. ``` ##### Selection syntax There is not specific [node selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) for `deprecation_date`. [Programmatic invocations](https://docs.getdbt.com/reference/programmatic-invocations.md) is one way to identify deprecated models (potentially in conjunction with [dbt list](https://docs.getdbt.com/reference/commands/list.md)). e.g., `dbt ls -q --output json --output-keys database schema alias deprecation_date`. ##### Deprecation process Additional steps are necessary to save on build-related compute and storage costs for a deprecated model. Deprecated models can continue to be built by producers and be selected by consumers until they are [disabled](https://docs.getdbt.com/reference/resource-configs/enabled.md) or removed. Just like it does not automatically [drop relations when models are deleted](https://docs.getdbt.com/faqs/Models/removing-deleted-models.md), dbt does not drop relations for deprecated models. Strategies similar to [here](https://discourse.getdbt.com/t/faq-cleaning-up-removed-models-from-your-production-schema/113) or [here](https://discourse.getdbt.com/t/clean-your-warehouse-of-old-and-deprecated-models/1547) can be used to drop relations that have been deprecated and are no longer in use. ##### Table expiration on BigQuery dbt-bigquery can set an [`hours_to_expiration`](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#controlling-table-expiration) that translates to `expiration_timestamp` within BigQuery. dbt does not automatically synchronize `deprecation_date` and `hours_to_expiration`, but users may want to coordinate them in some fashion (such as setting a model to expire 48 hours after its `deprecation_date`). Expired tables in BigQuery will be deleted and their storage reclaimed. --- ### Deprecations Deprecations are about features in your project code (models, configs, syntax) that will stop working in future versions. note Deprecated functionality still works in the v1.10 release but is no longer supported and will be removed in a future version. Deprecations currently show as warnings but don't prevent runs and other commands (unless you've configured [warnings as errors](https://docs.getdbt.com/reference/global-configs/warnings.md)), but will cause errors after upgrading if not addressed. Not the same as [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) (which are opt-in/out flags in your `dbt_project.yml` file) or [deprecated CLI flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) (which are command-line flags being removed in dbt v2). See the [Changes overview](https://docs.getdbt.com/reference/changes-overview.md) for a quick comparison. Upgrading to [dbt v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md)? You must resolve all deprecations listed on this page before upgrading. As dbt runs, it generates different categories of [events](https://docs.getdbt.com/reference/events-logging.md), one of which is *deprecations*. Deprecations are a special type of warning that lets you know that there are problems in parts of your project that will result in breaking changes in a future version of dbt. Although it's just a warning for now, it is important to resolve any deprecation warnings in your project to enable you to work with more safety, feedback, and confidence going forward. #### Identify deprecation warnings Finding deprecations that impact your code can be a daunting task when looking at the standard logs. Identifying them is the first step towards remediation. There are several methods for quickly locating deprecations and automatically remediating some of them. ##### dbt CLI To view deprecations from your CLI, run: ```bash dbt parse --no-partial-parse --show-all-deprecations ``` The `--no-partial-parse` flag ensures that even deprecations only picked up during parsing are included. The `--show-all-deprecations` flag ensures that each occurrence of the deprecations is listed instead of just the first. (Applies to dbt v2.0 and later) dbt v2 and `dbt parse` When you use dbt v2, omit `--no-partial-parse` from the command above. That flag is deprecated in dbt v2 and may log deprecation warning `dbt1700`. Run `dbt parse --show-all-deprecations` without `--no-partial-parse`. For more information, refer to [Deprecated flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) in the guide to upgrading to dbt v2. ```bash 19:15:13 [WARNING]: Deprecated functionality Summary of encountered deprecations: - MFTimespineWithoutYamlConfigurationDeprecation: 1 occurrence ``` ##### The dbt platform If you're using dbt, you can view deprecation warnings from the **Dashboard** area of your account. [![The deprecation warnings listed on the dbt dashboard.](/img/docs/dbt-platform/deprecation-warnings.png?v=2 "The deprecation warnings listed on the dbt dashboard.")](#)The deprecation warnings listed on the dbt dashboard. Click into a job to view more details and locate the deprecation warnings in the logs (or run the `parse` command with flags from the Studio IDE or dbt platform CLI). [![Deprecation warnings listed in the logs.](/img/docs/dbt-platform/deprecation-list.png?v=2 "Deprecation warnings listed in the logs.")](#)Deprecation warnings listed in the logs. ##### Automatic remediation Some deprecations can be automatically fixed with a script. Read more about it in [this dbt blog post](https://www.getdbt.com/blog/how-to-get-ready-for-the-new-dbt-engine#:~:text=2.%20Resolve%20deprecation%20warnings). [Download the script](https://github.com/dbt-labs/dbt-autofix) and follow the installation instructions to get started. You can run this script directly from [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md) using the **Autofix warnings** button to resolve deprecation warnings in dbt platform. [![Use the Autofix warnings button in the Studio IDE to address deprecation warnings in dbt platform.](/img/docs/dbt-platform/platform-ide/autofix-button.png?v=2 "Use the Autofix warnings button in the Studio IDE to address deprecation warnings in dbt platform.")](#)Use the Autofix warnings button in the Studio IDE to address deprecation warnings in dbt platform. ##### Silence deprecation warnings dbt Labs recommends fixing deprecation warnings rather than silencing them. If you need to temporarily reduce noise in logs, use `--warn-error-options` with the `silence` parameter. This works with any dbt command, including `dbt test` and `dbt parse`. For full configuration options (CLI flag, environment variable, and `dbt_project.yml`), refer to [Warnings](https://docs.getdbt.com/reference/global-configs/warnings.md). To silence all deprecation warnings in dbt v1: ```bash dbt test --warn-error-options '{"silence": ["Deprecations"]}' ``` (Applies to dbt v2.0 and later) In dbt v2, behavior differs from dbt v1: * The `Deprecations` group may not silence all deprecation-style warnings. Prefer the specific event name from your logs (for example, `SemanticModelDeprecated`). * Some deprecated configs are hard errors in dbt v2 and cannot be silenced. You must fix them. For more information about how dbt v2 handles `warn_error_options`, supported names, and hard-error cases, refer to [dbt v2 behavior and warning codes](https://docs.getdbt.com/reference/global-configs/warnings.md#dbt-v2-behavior-and-warning-codes). #### List of Deprecation Warnings The following are deprecation warnings in dbt today and the associated version number in which they first appear. ##### ArgumentsPropertyInGenericTestDeprecation dbt has deprecated the ability to specify a custom top-level property called `arguments` on generic tests. This deprecation warning is only raised when the behavior flag `require_generic_test_arguments_property` is set to `false`. ###### ArgumentsPropertyInGenericTestDeprecation warning resolution For example, you may have previously had a property called `arguments` on custom generic tests: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - my_custom_generic_test: arguments: [1,2,3] expression: "order_items_subtotal = subtotal" ``` You should set the `require_generic_test_arguments_property` flag to `true` and nest any keyword arguments to your test under the new `arguments` property: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - my_custom_generic_test: arguments: arguments: [1,2,3] expression: "order_items_subtotal = subtotal" ``` Alternatively, the original `arguments` keyword could be renamed to something else that does not collide with the new `arguments` key in the dbt framework. This renaming would also need to occur in the test macro definition, and the renamed key would still need to be specified within the `arguments` property to be valid syntactically. For example: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - my_custom_generic_test: arguments: renamed_arguments: [1,2,3] expression: "order_items_subtotal = subtotal" ``` ##### ConfigDataPathDeprecation In [dbt v1.0](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md>) `data-paths` has been renamed to [seed-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md). If you receive this deprecation warning, it means that `data-paths` is still being used in your project's `dbt_project.yml`. Example warning: CLI ```bash 23:14:58 [WARNING]: Deprecated functionality The `data-paths` config has been renamed to `seed-paths`. Please update your `dbt_project.yml` configuration to reflect this change. ``` ###### ConfigDataPathDeprecation warning resolution Change `data-paths` to `seed-paths` in your `dbt_project.yml`. ##### ConfigLogPathDeprecation [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) specifying `log-path` in `dbt_project.yml` was deprecated. Receiving this deprecation warning means that `log-path` is still specified in your `dbt_project.yml` and it's not set to the default value `logs`. Example: CLI (Applies to dbt v1.11 and later) ```bash 23:39:18 [WARNING]: Deprecated functionality The `log-path` config in `dbt_project.yml` has been deprecated and will no longer be supported in a future version of dbt-core. If you wish to write dbt logs to a custom directory, please use the --log-path CLI flag or DBT_ENGINE_LOG_PATH env var instead. ``` ###### ConfigLogPathDeprecation warning resolution Remove `log-path` from your `dbt_project.yml` and specify it via either the CLI flag `--log-path` or environment variable (Applies to dbt v1.11 and later) `DBT_ENGINE_LOG_PATH` [as documented here](https://docs.getdbt.com/reference/global-configs/logs.md#log-and-target-paths). ##### ConfigSourcePathDeprecation In [dbt v1.0](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md>) `source-paths` has been renamed to [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md). Receiving this deprecation warning means that `source-paths` is still being used in your project's `dbt_project.yml`. Example: CLI ```bash 23:03:47 [WARNING]: Deprecated functionality The `source-paths` config has been renamed to `model-paths`. Please update your `dbt_project.yml` configuration to reflect this change. 23:03:47 Registered adapter: postgres=1.9.0 ``` ###### ConfigSourcePathDeprecation warning resolution Change `source-paths` to `model-paths` in your `dbt_project.yml`. ##### ConfigTargetPathDeprecation In [dbt 1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) specifying `target-path` in `dbt_project.yml` was deprecated. Receiving this deprecation warning means that `target-path` is still specified in your `dbt_project.yml` and it's not set to the default value, `target`. Example: CLI (Applies to dbt v1.11 and later) ```bash 23:22:01 [WARNING]: Deprecated functionality The `target-path` config in `dbt_project.yml` has been deprecated and will no longer be supported in a future version of dbt-core. If you wish to write dbt artifacts to a custom directory, please use the --target-path CLI flag or DBT_ENGINE_TARGET_PATH env var instead. ``` ###### ConfigTargetPathDeprecation warning resolution Remove `target-path` from your `dbt_project.yml` and specify it via either the CLI flag `--target-path` or environment variable (Applies to dbt v1.11 and later) [`DBT_ENGINE_TARGET_PATH`](https://docs.getdbt.com/reference/global-configs/logs.md#log-and-target-paths). ##### CustomKeyInConfigDeprecation This warning is raised when you use custom config keys that dbt does not recognize as part of the official config spec. This applies to configuration blocks in both SQL and YAML files. Example that results in the warning: ```yaml models: - name: my_model config: custom_config_key: value ``` ###### CustomKeyInConfigDeprecation warning resolution Nest custom keys under `meta` and ensure `meta` is nested under `config` (similar to [`PropertyMovedToConfigDeprecation`](#propertymovedtoconfigdeprecation)). For example: ```yaml models: - name: my_model config: meta: custom_config_key: value ``` ##### CustomKeyInObjectDeprecation This warning is displayed when you specify a config that dbt does not recognize as part of the official config spec. This could be custom configs or defining `meta` as top-level keys in the `columns` list. Previously, when you could define any additional fields directly under `config`, it could lead to collisions between pre-existing user-defined configurations and official configurations of the dbt framework. As of dbt v1.10 and in dbt v2, top-level config keys will be reserved for official configurations of the dbt framework. This deprecation warning is only raised for the following adapters: * Snowflake * Databricks * BigQuery * Redshift ###### CustomKeyInObjectDeprecation warning resolution Nest custom configs under `meta` and ensure `meta` is nested under `config` (similar to [`PropertyMovedToConfigDeprecation`](#propertymovedtoconfigdeprecation)). The same resolution applies whether the custom key is in a model config or a generic test definition. Select the relevant tab for an example: ##### Model config Example that results in the warning: ```yaml models: - name: my_model config: custom_config_key: value columns: - name: my_column meta: some_key: some_value ``` Example of the resolution: ```yaml models: - name: my_model config: meta: custom_config_key: value columns: - name: my_column config: meta: some_key: some_value ``` ##### Generic test definition If you define a custom key directly under a test in a `tests:` block, nest it under `config.meta`. Example that results in the warning: ```yaml tests: - name: custom_generic_test description: My custom generic test arguments: - name: active_timestamp type: timestamp description: The active timestamp for the model ``` Example of the resolution: ```yaml tests: - name: custom_generic_test description: My custom generic test config: meta: arguments: - name: active_timestamp type: timestamp description: The active timestamp for the model ``` ###### Accessing nested configurations To access custom configurations nested under attributes of `meta`, use `config.get('meta')` and then index the meta dictionary by the name of your custom attribute. Users will need to adjust their code that accesses the custom config keys directly as top-level keys. Example before custom configurations were nested under meta: ```jinja {% set my_custom_config = config.get('custom_config_key') %} ``` After configs are nested: ```jinja {% set my_custom_config = config.get('meta').custom_config_key %} ``` ##### CustomOutputPathInSourceFreshnessDeprecation dbt has deprecated the `--output` (or `-o`) flag for overriding the location of source freshness results from the `sources.json` file destination. ###### CustomOutputPathInSourceFreshnessDeprecation warning resolution Remove the `--output` or `-o` flag and associated path configuration from any jobs running dbt source freshness commands. There is no alternative for changing the location of only the source freshness results. However, you can still use `--target-path` to write *all* artifacts from the step to a custom location. ##### CustomTopLevelKeyDeprecation This warning informs users when they use custom top-level keys in their YAML files that are not supported by dbt. This deprecation warning is only raised for the following adapters: * Snowflake * Databricks * BigQuery * Redshift ###### CustomTopLevelKeyDeprecation warning resolution Move custom top-level keys in your YAML files under `config.meta`. For example, when you use a custom top-level key such as `custom_metdata`: dbt\_project.yml ```yaml models: my_project: staging: +materialized: view marts: +materialized: table custom_metadata: owner: "data_team" description: "This project contains models for our analytics platform" last_updated: "2025-07-01" ``` You should move the key under `config.meta`: dbt\_project.yml ```yaml models: my_project: staging: +materialized: view marts: +materialized: table config: meta: custom_metadata: owner: "data_team" description: "This project contains models for our analytics platform" last_updated: "2025-07-01" ``` ##### DuplicateNameDistinctNodeTypesDeprecation dbt raises this warning when two unversioned resources in the same package share the same name (for example, a model and a seed both named `sales`) and the `require_unique_project_resource_names` flag is set to `false`. Previously, dbt did not always detect these name conflicts, which meant duplicate names could sometimes point to the wrong resource. When the `require_unique_project_resource_names` flag is set to `true`, dbt raises a `DuplicateResourceNameError`. For more information, see [Unique project resource names](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_unique_project_resource_names.md). ###### DuplicateNameDistinctNodeTypesDeprecation warning resolution Rename one of the conflicting resources to ensure all names are unique. ##### DuplicateYAMLKeysDeprecation This warning is raised when two identical keys exist in the `profiles.yml`. Previously, if identical keys existed in the [`profiles.yml` file](https://docs.getdbt.com/docs/local/profiles.yml.md), dbt would use the last configuration listed in the file. profiles.yml ```yml my_profile: target: outputs: ... my_profile: # dbt would use this profile key target: outputs: ... ``` Note that in a future version, dbt will stop supporting duplicate keys with silent overwrite. ###### DuplicateYAMLKeysDeprecation warning resolution Remove duplicate keys from your `profiles.yml` file. ##### EnvironmentVariableNamespaceDeprecation This warning is raised when you're using environment variables that conflict with dbt's reserved namespace `DBT_ENGINE`. Previously, both dbt internal variables and custom variables used the `DBT_` prefix⁠. If the environment variable defined in dbt collides with a custom environment variable, the project may break. All new dbt environment variables are now prefixed with `DBT_ENGINE` to prevent naming collisions and minimize disruption for users. ###### EnvironmentVariableNamespaceDeprecation Review your custom environment variables and ensure they don't conflict with dbt's reserved namespace `DBT_ENGINE`. ##### ExposureNameDeprecation In [dbt 1.3](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.3.md#new-and-changed-documentation>), dbt began allowing only letters, numbers, and underscores in the `name` property of [exposures](https://docs.getdbt.com/reference/exposure-properties.md). Example: CLI ```bash 23:55:00 [WARNING]: Deprecated functionality Starting in v1.3, the 'name' of an exposure should contain only letters, numbers, and underscores. Exposures support a new property, 'label', which may contain spaces, capital letters, and special characters. stg_&customers does not follow this pattern. Please update the 'name', and use the 'label' property for a human-friendly title. This will raise an error in a future version of dbt-core. ``` ###### ExposureNameDeprecation warning resolution Ensure your exposure names only contain letters, numbers, and underscores. A more human-readable name can be put in the [`label`](https://docs.getdbt.com/reference/exposure-properties.md#overview) property of exposures. ##### GenerateSchemaNameNullValueDeprecation dbt raises this deprecation warning when a custom `generate_schema_name` macro returns a `null` value. Returning `null` schema names can lead to invalid or unpredictable behavior. This deprecation warning is raised when the [`require_valid_schema_from_generate_schema_name` flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_valid_schema_from_generate_schema_name.md) is set to `false`. When the flag is set to `true`, dbt raises an error during parsing. ###### GenerateSchemaNameNullValueDeprecation warning resolution If your project defines a `generate_schema_name` macro, update the macro to return a valid schema name. For example: macros/get\_custom\_schema.sql ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- if custom_schema_name is none -%} {{ return(target.schema) }} {%- else -%} {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` ##### GenericJSONSchemaValidationDeprecation This deprecation type is a catch-all/fallback. dbt attempts to handle all JSON schema validation errors with specific deprecation event types, but it is possible that we missed something. Missing something means that either dbt failed to handle a specific case with a deprecation event *or* the JSON schema is incorrect in a particular area. ###### GenericJSONSchemaValidationDeprecation warning resolution If you are seeing this warning, unfortunately, there isn't much you can do at this time, but we are continuing to work on reducing instances of this deprecation. If you would like guidance on a specific instance you are seeing, please [contact support](mailto:support@getdbt.com) (available for cloud-based dbt platform customers) or the [community Slack](https://www.getdbt.com/community) (for dbt v1 users). ##### MFCumulativeTypeParamsDeprecation In dbt [v1.9](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md) implementing `window` and `time_to_grain` directly on the `type_params` of a [metric](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md) was deprecated. Example: CLI ```bash 15:36:22 [WARNING]: Cumulative fields `type_params.window` and `type_params.grain_to_date` has been moved and will soon be deprecated. Please nest those values under `type_params.cumulative_type_params.window` and `type_params.cumulative_type_params.grain_to_date`. See documentation on behavior changes: https://docs.getdbt.com/reference/global-configs/behavior-changes. ``` ###### MFCumulativeTypeParamsDeprecation warning resolution Nest your `window` and `time_to_grain` under the `cumulative_type_params` property within the `type_params` of the relevant metric. ##### MFTimespineWithoutYamlConfigurationDeprecation Before dbt v1.9, the MetricFlow time spine configuration was stored in a `metricflow_time_spine.sql` file. In [v1.9](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md) dbt introduced the [YAML timespine definition](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#configuring-time-spine-in-yaml) for MetricFlow. It was then decided that it would be the standard going forward. If you see this deprecation warning, you don't have a YAML timespine definition for Metricflow. Example: CLI ```bash 19:56:41 [WARNING]: Time spines without YAML configuration are in the process of deprecation. Please add YAML configuration for your 'metricflow_time_spine' model. See documentation on MetricFlow time spines: https://docs.getdbt.com/docs/build/metricflow-time-spine and behavior change documentation: https://docs.getdbt.com/reference/global-configs/behavior-changes ``` ###### MFTimespineWithoutYamlConfigurationDeprecation warning resolution Define your MetricFlow timespine in [YAML](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#creating-a-time-spine-table). ##### MissingArgumentsPropertyInGenericTestDeprecation dbt has deprecated specifying keyword arguments as properties on custom generic data tests or data tests that use the [alternative `test_name` format](https://docs.getdbt.com/reference/resource-properties/data-tests.md#alternative-format-for-defining-tests). Instead, arguments to tests should be specified under the new `arguments` property. This deprecation warning is only raised when the behavior flag `require_generic_test_arguments_property` is set to `true`. ###### MissingArgumentsPropertyInGenericTestDeprecation warning resolution If you previously set arguments as top-level properties on custom generic tests: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - dbt_utils.expression_is_true: expression: "order_items_subtotal = subtotal" ``` Or using the alternative `test_name` format: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - name: arbitrary_name test_name: dbt_utils.expression_is_true expression: "order_items_subtotal = subtotal" where: "1=1" ``` You should now nest arguments under `arguments` and framework configurations under `config`: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - dbt_utils.expression_is_true: arguments: expression: "order_items_subtotal = subtotal" ``` Or with framework configurations: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - name: arbitrary_name test_name: dbt_utils.expression_is_true arguments: expression: "order_items_subtotal = subtotal" config: where: "1=1" ``` ##### MissingPlusPrefixDeprecation dbt has deprecated specifying configurations without [the `+` prefix](https://docs.getdbt.com/reference/dbt_project.yml#the--prefix) in `dbt_project.yml`. Only folder and file names can be specified without the `+` prefix within resource configurations in `dbt_project.yml`. Example: CLI ```bash 18:16:06 [WARNING][MissingPlusPrefixDeprecation]: Deprecated functionality Missing '+' prefix on `tags` found at `my_path.sub_path.another_path.tags` in file `dbt_project.yml`. Hierarchical config values without a '+' prefix are deprecated in dbt_project.yml. ``` This deprecation warning is only raised for the following adapters: * Snowflake * Databricks * BigQuery * Redshift ###### MissingPlusPrefixDeprecation warning resolution If you previously set one of the impacted configurations without a `+`, such as `materialized`: dbt\_project.yml ```yaml models: marts: materialized: table ``` You should now set it with the `+` prefix to disambiguate between paths: dbt\_project.yml ```yaml models: marts: +materialized: table ``` ##### ModelParamUsageDeprecation The `--models` / `--model` / `-m` flag was renamed to `--select` / `--s` way back in dbt v0.21 (Oct 2021). Silently skipping this flag means ignoring your command's selection criteria, which could mean building your entire DAG when you only meant to select a small subset. For this reason, the `--models` / `--model` / `-m` flag will raise a warning in dbt v1.10, and an error in dbt v2. Please update your job definitions accordingly. ###### ModelParamUsageDeprecation warning resolution Update your job definitions and remove the `--models` / `--model` / `-m` flag and replace it with `--select` / `--s`. ##### ModulesItertoolsUsageDeprecation dbt has deprecated the use of `modules.itertools` in Jinja. Example: CLI ```bash 15:49:33 [WARNING]: Deprecated functionality Usage of itertools modules is deprecated. Please use the built-in functions instead. ``` ###### ModulesItertoolsUsageDeprecation warning resolution If you are currently using functions from the `itertools` module within Jinja SQL templates, use the available built-in [dbt functions](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) and [Jinja methods](https://docs.getdbt.com/docs/build/jinja-macros.md) instead. For example, the following SQL file: models/itertools\_usage.sql ```sql {%- set A = [1, 2] -%} {%- set B = ['x', 'y', 'z'] -%} {%- set AB_cartesian = modules.itertools.product(A, B) -%} {%- for item in AB_cartesian %} {{ item }} {%- endfor -%} ``` Should be converted to use alternative built-in dbt Jinja methods. For example: macros/cartesian\_product.sql ```sql {%- macro cartesian_product(list1, list2) -%} {%- set result = [] -%} {%- for item1 in list1 -%} {%- for item2 in list2 -%} {%- set _ = result.append((item1, item2)) -%} {%- endfor -%} {%- endfor -%} {{ return(result) }} {%- endmacro -%} ``` models/itertools\_usage.sql ```sql {%- set A = [1, 2] -%} {%- set B = ['x', 'y', 'z'] -%} {%- set AB_cartesian = cartesian_product(A, B) -%} {%- for item in AB_cartesian %} {{ item }} {%- endfor -%} ``` ##### PackageInstallPathDeprecation The default location where packages are installed when running `dbt deps` has been updated from `dbt_modules` to `dbt_packages`. During a `dbt clean` dbt detected that `dbt_modules` is defined in the [clean-targets](https://docs.getdbt.com/reference/project-configs/clean-targets.md) property in `dbt_project.yml` even though `dbt_modules` is not the [`packages-install-path`](https://docs.getdbt.com/reference/project-configs/packages-install-path.md). Example: CLI ```bash 22:48:01 [WARNING]: Deprecated functionality The default package install path has changed from `dbt_modules` to `dbt_packages`. Please update `clean-targets` in `dbt_project.yml` and check `.gitignore`. Or, set `packages-install-path: dbt_modules` If you'd like to keep the current value. ``` ###### PackageInstallPathDeprecation warning resolution The following are recommended approaches: 1. Replace `dbt_modules` with `dbt_packages` in your `clean-targets` spec (and `.gitignore`). 2. Set `packages-install-path: dbt_modules` if you want to keep having packages installed in `dbt_modules`. ##### PackageMaterializationOverrideDeprecation The behavior where installed packages could override built-in materializations without your explicit opt-in is deprecated. Setting the [`require_explicit_package_overrides_for_builtin_materializations` flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_explicit_package_overrides_for_builtin_materializations.md) to `false` in your `dbt_project.yml` allowed packages that matched the name of a built-in materialization to continue to be included in the search and resolution order. ###### PackageMaterializationOverrideDeprecation warning resolution Explicitly override built-in materializations, in favor of a materialization defined in a package, by reimplementing the built-in materialization in your root project and wrapping the package implementation. For example: ```jinja {% materialization table, snowflake %} {{ return (package_name.materialization_table_snowflake()) }} {% endmaterialization %} ``` Once you've added the override for your package, remove the `require_explicit_package_overrides_for_builtin_materializations: false` flag from your `dbt_project.yml` to resolve the warning. ##### PackageRedirectDeprecation This deprecation warning means a package currently used in your project, defined in `packages.yml`, has been renamed. This generally happens when the ownership of a package has changed or the scope of the package has changed. It is likely that the package currently referenced in your `packages.yml` has stopped being actively maintained (as development has been moved to the new package name), and at some point, the named package will cease working with dbt. CLI ```bash 22:31:38 [WARNING]: Deprecated functionality The `fishtown-analytics/dbt_utils` package is deprecated in favor of `dbt-labs/dbt_utils`. Please update your `packages.yml` configuration to use `dbt-labs/dbt_utils` instead. ``` ###### PackageRedirectDeprecation warning resolution Begin referencing the new package in your `packages.yml` instead of the old package. ##### ProjectFlagsMovedDeprecation The `config` property that had been configurable in `profiles.yml` was deprecated in favor of `flags` in the `dbt_project.yaml`. If you see this deprecation warning, dbt detected the `config` property in your `profiles.yml`. Example: CLI ```bash 00:08:12 [WARNING]: Deprecated functionality User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml. ``` ###### ProjectFlagsMovedDeprecation warning resolution Remove `config` from `profiles.yml`. Add any previous [`config`](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) in `profiles.yml` to `flags` in `dbt_project.yml`. ##### PropertyMovedToConfigDeprecation Some historical properties are moving entirely to configs. This will include: `freshness`, `meta`, `tags`, `docs`, `group`, and `access` Changing certain properties to configs is beneficial because you can set them for many resources at once in `dbt_project.yml` (project-level/folder-level defaults). More info on the difference between properties and configs [here](https://docs.getdbt.com/reference/configs-and-properties.md). ###### PropertyMovedToConfigDeprecation warning resolution If you previously set one of the impacted properties, such as `freshness`: ```yaml sources: - name: ecom schema: raw description: E-commerce data for the Jaffle Shop freshness: warn_after: count: 24 period: hour ``` You should now set it under `config`: ```yaml sources: - name: ecom schema: raw description: E-commerce data for the Jaffle Shop config: freshness: warn_after: count: 24 period: hour ``` ##### ResourceNamesWithSpacesDeprecation In [dbt 1.8](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.8.md#managing-changes-to-legacy-behaviors), allowing resource names to have spaces in them was deprecated. If you get this deprecation warning, dbt detected a resource name with a space in it. Example: CLI ```bash 16:37:58 [WARNING]: Found spaces in the name of `model.jaffle_shop.stg supplies` ``` ###### ResourceNamesWithSpacesDeprecation warning resolution Rename the resource in violation so it no longer contains a space in its name. ##### SourceFreshnessProjectHooksNotRun If you are seeing this, it means that the behavior flag `source_freshness_run_project_hooks` is set to `false` and either `on-run-start` or `on-run-end` is defined ([docs](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md)). Previously, project hooks wouldn't be run on sources when `dbt source freshness` was run. Example: CLI ```bash 19:51:56 [WARNING]: In a future version of dbt, the `source freshness` command will start running `on-run-start` and `on-run-end` hooks by default. For more information: https://docs.getdbt.com/reference/global-configs/legacy-behaviors ``` ###### SourceFreshnessProjectHooksNotRun warning resolution Set `source_freshness_run_project_hooks` to `true`. For instructions on skipping project hooks during a `dbt source freshness` invocation, check out the [behavior change documentation](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md). ##### SourceOverrideDeprecation The `overrides` property for sources is deprecated. This deprecation warning is only raised for the following adapters: * Snowflake * Databricks * BigQuery * Redshift ###### SourceOverrideDeprecation warning resolution Remove the `overrides` property and [enable or disable a source](https://docs.getdbt.com/reference/source-configs.md#configuring-sources) from a package instead. ##### UnexpectedJinjaBlockDeprecation If you have an unexpected Jinja block - an orphaned Jinja block or a Jinja block outside of a macro context - you will receive a warning, and in a future version, dbt will stop supporting unexpected Jinja blocks. Previously, these unexpected Jinja blocks were silently ignored. macros/my\_macro.sql ```sql {% endmacro %} # orphaned endmacro jinja block {% macro hello() %} hello! {% endmacro %} ``` ###### UnexpectedJinjaBlockDeprecation warning resolution Delete the unexpected Jinja blocks. ##### WEOIncludeExcludeDeprecation The `include` and `exclude` options for `warn_error_options` have been deprecated and replaced with `error` and `warn`, respectively. ###### WEOIncludeExcludeDeprecation warning resolution Anywhere `warn_error_options` is configured, replace: * `include` with `error` * `exclude` with `warn` For example: ```yaml ... flags: warn_error_options: include: - NoNodesForSelectionCriteria ``` Should now be configured as: ```yaml ... flags: warn_error_options: error: - NoNodesForSelectionCriteria ``` --- ### description ##### Models models/schema.yml ```yml models: - name: model_name description: markdown_string columns: - name: column_name description: markdown_string ``` ##### Sources models/schema.yml ```yml sources: - name: source_name description: markdown_string tables: - name: table_name description: markdown_string columns: - name: column_name description: markdown_string ``` ##### Seeds seeds/schema.yml ```yml seeds: - name: seed_name description: markdown_string columns: - name: column_name description: markdown_string ``` ##### Snapshots snapshots/schema.yml ```yml snapshots: - name: snapshot_name description: markdown_string columns: - name: column_name description: markdown_string ``` ##### Analyses analysis/schema.yml ```yml analyses: - name: analysis_name description: markdown_string columns: - name: column_name description: markdown_string ``` ##### Macros macros/schema.yml ```yml macros: - name: macro_name description: markdown_string arguments: - name: argument_name description: markdown_string ``` ##### Data tests (Applies to dbt v1.9 and later) You can add a description to a [singular data test](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests) or a [generic data test](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). tests/schema.yml ```yml # Singular data test example version: 2 data_tests: - name: data_test_name description: markdown_string ``` tests/schema.yml ```yml # Generic data test example version: 2 models: - name: model_name columns: - name: column_name data_tests: - unique: description: markdown_string ``` ##### Unit tests models/schema.yml ```yml unit_tests: - name: unit_test_name description: "markdown_string" model: model_name given: ts - input: ref_or_source_call rows: - {column_name: column_value} - {column_name: column_value} - {column_name: column_value} - {column_name: column_value} - input: ref_or_source_call format: csv rows: dictionary | string expect: format: dict | csv | sql fixture: fixture_name ``` ##### Groups models/schema.yml ```yml groups: - name: group_name description: markdown_string # Supported in v1.10 and later owner: email: owner@example.com ``` #### Definition A user-defined description used to document: * a model, and model columns * sources, source tables, and source columns * seeds, and seed columns * snapshots, and snapshot columns * analyses, and analysis columns * macros, and macro arguments * data tests, and data test columns * unit tests for models * groups (dbt v1.10+) These descriptions are used in the documentation website rendered by dbt (refer to [the documentation guide](https://docs.getdbt.com/docs/build/documentation.md) or [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md)). Descriptions can include markdown, as well as the [`doc` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md). You may need to quote your YAML Be mindful of YAML semantics when providing a description. If your description contains special YAML characters like curly brackets, colons, or square brackets, you may need to quote your description. An example of a quoted description is shown [below](#use-some-markdown-in-a-description). #### Examples This section contains examples of how to add descriptions to various resources: * [Add a simple description to a model and column](#add-a-simple-description-to-a-model-and-column) <br /> * [Add a multiline description to a model](#add-a-multiline-description-to-a-model) <br /> * [Use some markdown in a description](#use-some-markdown-in-a-description) <br /> * [Use a docs block in a description](#use-a-docs-block-in-a-description) <br /> * [Link to another model in a description](#link-to-another-model-in-a-description) * [Include an image from your repo in your descriptions](#include-an-image-from-your-repo-in-your-descriptions) <br /> * [Include an image from the web in your descriptions](#include-an-image-from-the-web-in-your-descriptions) <br /> * [Add a description to a data test](#add-a-description-to-a-data-test) <br /> * [Add a description to a unit test](#add-a-description-to-a-unit-test) <br /> ##### Add a simple description to a model and column models/schema.yml ```yml version: 2 models: - name: dim_customers description: One record per customer columns: - name: customer_id description: Primary key ``` ##### Add a multiline description to a model You can use YAML [block notation](https://yaml-multiline.info/) to split a longer description over multiple lines: models/schema.yml ```yml version: 2 models: - name: dim_customers description: > One record per customer. Note that a customer must have made a purchase to be included in this table — customer accounts that were created but never used have been filtered out. columns: - name: customer_id description: Primary key. ``` ##### Use some markdown in a description You can use markdown in your descriptions, but you may need to quote your description to ensure the YAML parser doesn't get confused by special characters! models/schema.yml ```yml version: 2 models: - name: dim_customers description: "**[Read more](https://www.google.com/)**" columns: - name: customer_id description: Primary key. ``` ##### Use a docs block in a description If you have a long description, especially if it contains markdown, it may make more sense to leverage a [`docs` block](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md). A benefit of this approach is that code editors will correctly highlight markdown, making it easier to debug as you write. models/schema.yml ```yml version: 2 models: - name: fct_orders description: This table has basic information about orders, as well as some derived facts based on payments columns: - name: status description: '{{ doc("orders_status") }}' ``` models/docs.md ```text {% docs orders_status %} Orders can be one of the following statuses: | status | description | |----------------|---------------------------------------------------------------------------| | placed | The order has been placed but has not yet left the warehouse | | shipped | The order has been shipped to the customer and is currently in transit | | completed | The order has been received by the customer | | returned | The order has been returned by the customer and received at the warehouse | {% enddocs %} ``` ##### Link to another model in a description You can use relative links to link to another model. It's a little hacky — but to do this: 1. Serve your docs site. 2. Navigate to the model you want to link to, e.g. `http://127.0.0.1:8080/#!/model/model.jaffle_shop.stg_stripe__payments` 3. Copy the url\_path, i.e. everything after `http://127.0.0.1:8080/`, so in this case `#!/model/model.jaffle_shop.stg_stripe__payments` 4. Paste it as the link models/schema.yml ```yml version: 2 models: - name: customers description: "Filtering done based on [stg_stripe__payments](#!/model/model.jaffle_shop.stg_stripe__payments)" columns: - name: customer_id description: Primary key ``` ##### Include an image from your repo in your descriptions This section applies to dbt v1 users only. Including an image from your repository ensures your images are version-controlled. Both dbt and dbt v1 users can [include an image from the web](#include-an-image-from-the-web-in-your-descriptions), which offers dynamic content, reduced repository size, accessibility, and ease of collaboration. To include an image in your model's `description` field: 1. Add the file in a subdirectory, e.g. `assets/dbt-logo.svg` 2. Set the [`asset-paths` config](https://docs.getdbt.com/reference/project-configs/asset-paths.md) in your `dbt_project.yml` file so that this directory gets copied to the `target/` directory as part of `dbt docs generate` dbt\_project.yml ```yml asset-paths: ["assets"] ``` 2. Use a Markdown link to the image in your `description:` models/schema.yml ```yml version: 2 models: - name: customers description: "![dbt Logo](assets/dbt-logo.svg)" columns: - name: customer_id description: Primary key ``` 3. Run `dbt docs generate` — the `assets` directory will be copied to the `target` directory 4. Run `dbt docs serve` — the image will be rendered as part of your project documentation: If mixing images and text, also consider using a docs block. ##### Include an image from the web in your descriptions This section applies to dbt and dbt v1 users. Including an image from the web offers dynamic content, reduced repository size, accessibility, and ease of collaboration. To include images from the web, specify the image URL in your model's `description` field: models/schema.yml ```yml version: 2 models: - name: customers description: "![dbt Logo](https://raw.githubusercontent.com/dbt-labs/dbt/refs/heads/1.latest/docs/images/dbt-core.svg)" columns: - name: customer_id description: Primary key ``` If mixing images and text, also consider using a docs block. ##### Add a description to a data test You can add a `description` property to a generic or singular data test. ###### Generic data test This example shows a generic data test that checks for unique values in a column for the `orders` model. models/\<filename>.yml ```yaml version: 2 models: - name: orders columns: - name: order_id data_tests: - unique: description: "The order_id is unique for every row in the orders model" ``` You can also add descriptions to the Jinja macro that provides the core logic of a generic data test. Refer to the [Add description to generic data test logic](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md#add-description-to-generic-data-test-logic) for more information. ###### Singular data test This example shows a singular data test that checks to ensure all values in the `payments` model are not negative (≥ 0). tests/\<filename>.yml ```yaml data_tests: - name: assert_total_payment_amount_is_positive description: > Refunds have a negative amount, so the total amount should always be >= 0. Therefore return records where total amount < 0 to make the test fail. ``` Note that in order for the test to run, the `tests/assert_total_payment_amount_is_positive.sql` SQL file has to exist in the `tests` directory. ##### Add a description to a unit test This example shows a unit test that checks to ensure the `opened_at` timestamp is properly truncated to a date for the `stg_locations` model. models/\<filename>.yml ```yaml unit_tests: - name: test_does_location_opened_at_trunc_to_date description: "Check that opened_at timestamp is properly truncated to a date." model: stg_locations given: - input: source('ecom', 'raw_stores') rows: - {id: 1, name: "Rego Park", tax_rate: 0.2, opened_at: "2016-09-01T00:00:00"} - {id: 2, name: "Jamaica", tax_rate: 0.1, opened_at: "2079-10-27T23:59:59.9999"} expect: rows: - {location_id: 1, location_name: "Rego Park", tax_rate: 0.2, opened_date: "2016-09-01"} - {location_id: 2, location_name: "Jamaica", tax_rate: 0.1, opened_date: "2079-10-27"} ``` --- ### Dimension properties Dimensions are non-aggregatable expressions that define how metrics can be grouped or sliced. They are always defined within a [semantic model](https://docs.getdbt.com/reference/semantic-model-properties.md). See [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) for concepts and examples. Availability The latest YAML spec is supported in the following environments: * **dbt platform (v1 Latest release track)** * **dbt v2** * **dbt v1.12** For more information, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). (Applies to dbt v1.12 and later) #### Latest spec (model YAML) Dimensions are defined at the column level. ##### Column-level placement (latest spec) Add these keys to a **column** under the semantic model’s `columns:` list — they are not nested inside the `dimension:` object. | Location | Type | Required | Description | | -------------- | ------------------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dimension:` | block or shorthand | Yes (to define a dimension) | Attaches a dimension to the column. Use a mapping (`dimension:` with `type`, etc.) or the shorthand `dimension: categorical` / `dimension: time`. | | `granularity:` | string | Yes for **time** dimensions | Time grain of the underlying column data (for example, `day`, `week`, `month`). Place `granularity:` on the column next to `dimension:`, not inside the `dimension:` block. For **categorical** dimensions, `granularity` is not meaningful; if it appears in YAML, validation should surface an error. | ##### Properties inside the `dimension:` block (latest spec) | Property | Type | Required | Description | | ----------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | | `type` | string | Yes | `time` or `categorical`. | | `name` | string | No | Unique within the semantic model; defaults to column name. | | `description` | string | No | Documentation for the dimension. | | `label` | string | No | Display value in downstream tools. | | `is_partition` | boolean | No | Whether this dimension is a partition dimension for the model (supported for time and categorical dimensions in YAML). | | `config` | object | No | Metadata and config. | | `validity_params` | object | No | For **time** dimensions: SCD-style validity (for example, `is_start`, `is_end`). | **Derived dimensions:** To define dimensions with an `expr` that is not tied to a single column, use the semantic model’s optional `derived_semantics.dimensions` list. That structure is part of the [semantic model](https://docs.getdbt.com/reference/semantic-model-properties.md) configuration (alongside `columns:`), not a property nested under a column’s `dimension:` block. See [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) and [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) for examples. * **Column-level:** Under the model's `columns:` list, each column can have a `dimension:` block with *time* or *categorical* type, and optional `name`, `description`, `label`, `is_partition`, `config`. * **Time dimensions:** The column must also have a top-level `granularity:` (for example, `day`). * **Validity (SCD):** Time dimensions can specify `validity_params` (for example, `is_start`, `is_end`). For concepts and usage patterns, refer to [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md). For the latest spec, refer to [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md). --- ### dispatch (config) dbt\_project.yml ```yml dispatch: - macro_namespace: packagename search_order: [packagename] - macro_namespace: packagename search_order: [packagename] ``` #### Definition Optionally override the [dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) search locations for macros in certain namespaces. If not specified, `dispatch` will look in your root project *first*, by default, and then look for implementations in the package named by `macro_namespace`. #### Examples I want to "shim" the `dbt_utils` package with the `spark_utils` compatibility package. dbt\_project.yml ```yml dispatch: - macro_namespace: dbt_utils search_order: ['spark_utils', 'dbt_utils'] ``` I've reimplemented certain macros from the `dbt_utils` package in my root project (`'my_root_project'`), and I want my versions to take precedence. Otherwise, fall back to the versions in `dbt_utils`. *Note: This is the default behavior. You may optionally choose to express that search order explicitly as:* dbt\_project.yml ```yml dispatch: - macro_namespace: dbt_utils search_order: ['my_root_project', 'dbt_utils'] ``` --- ### docs ##### Models You can configure `docs` behavior for many resources at once by setting in `dbt_project.yml`. You can also use the `docs` config in `properties.yaml` files, to set or override documentation behaviors for specific resources: dbt\_project.yml ```yml models: <resource-path>: +docs: show: true | false node_color: color_id # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") ``` models/schema.yml ```yml models: - name: model_name config: docs: # changed to config in v1.10 show: true | false node_color: color_id # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") ``` ##### Sources The `docs` config isn’t supported for sources. ##### Seeds You can use the `docs` config in YAML files, including the `dbt_project.yml`: dbt\_project.yml ```yml seeds: <resource-path>: +docs: show: true | false node_color: color_id # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") ``` seeds/schema.yml ```yml seeds: - name: seed_name config: docs: # changed to config in v1.10 show: true | false node_color: color_id # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") ``` ##### Snapshots You can use the `docs` config in YAML files, including the `dbt_project.yml`: dbt\_project.yml ```yml snapshots: <resource-path>: +docs: show: true | false node_color: color_id # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") ``` snapshots/schema.yml ```yml snapshots: - name: snapshot_name config: docs: # changed to config in v1.10 show: true | false node_color: color_id # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") ``` ##### Analyses You can configure `docs` for analyses in the `config` block under `analyses:` in your YAML file. Refer to [Analysis properties](https://docs.getdbt.com/reference/analysis-properties.md) for more information. analysis/schema.yml ```yml analyses: - name: analysis_name config: docs: # changed to config in v1.10 show: true | false node_color: color_id # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") ``` ##### Macros You can configure `docs` for macros in the `config` block under `macros:` in your YAML file. Refer to [Macro properties](https://docs.getdbt.com/reference/macro-properties.md) for more information. macros/schema.yml ```yml macros: - name: macro_name config: docs: # changed to config in v1.11 show: true | false ``` Note that for backwards compatibility, `docs` is supported as a top-level key, but without the capabilities of config inheritance. #### Definition You can use the `docs` config to provide documentation-specific configuration for resources. It supports the following attributes: * `show`: Controls whether nodes appears in the auto-generated documentation website. * `node_color`: Controls the colors of nodes displayed in the DAG. This is supported for models, seeds, snapshots, and analyses. Other node types are not supported. **Note:** Hidden models will still appear in the dbt DAG visualization but will be identified as "hidden.” #### Default The default value for `show` is `true`. #### Examples ##### Mark a model as hidden ```yml models: - name: sessions__tmp docs: show: false ``` ##### Mark a subfolder of models as hidden **Note:** This can also hide dbt packages. dbt\_project.yml ```yml models: # hiding models within the staging subfolder tpch: staging: +materialized: view +docs: show: false # hiding a dbt package dbt_artifacts: +docs: show: false ``` #### Custom node colors The `docs` attribute supports `node_color` to customize the display color of some node types in the DAG within [dbt Docs](https://docs.getdbt.com/docs/build/view-documentation.md). You can define node colors in the following files and apply overrides where needed. * `node_color` hierarchy: * `<example-sql-file.sql>` overrides `schema.yml` overrides `dbt_project.yml` Note, you need to run or re-run the `dbt docs generate` command to apply and view the customized colors. Custom node colors not applicable in Catalog The custom `node_color` attribute isn't applicable in Catalog. Instead, Explorer provides [lenses](https://docs.getdbt.com/docs/explore/explore-projects.md#lenses), which are map layers for your DAG. Lenses help you better understand your project's contextual metadata at scale and distinguish specific models or subsets of models. #### Examples Add custom `node_colors` to models that support it within subdirectories based on hex codes or a plain color name. [![Example](/img/node_color_example.png?v=2 "Example")](#)Example `marts/core/fct_orders.sql` with `node_color: red` overrides `dbt_project.yml` with `node_color: gold` `marts/core/schema.yml` with `node_color: #000000` overrides `dbt_project.yml` with `node_color: gold` dbt\_project.yml ```yml models: tpch: staging: +materialized: view +docs: node_color: "#cd7f32" marts: core: materialized: table +docs: node_color: "gold" ``` marts/core/schema.yml ```yml models: - name: dim_customers description: Customer dimensions table docs: node_color: '#000000' ``` marts/core/fct\_orders.sql ```sql {{ config( materialized = 'view', tags=['finance'], docs={'node_color': 'red'} ) }} with orders as ( select * from {{ ref('stg_tpch_orders') }} ), order_item as ( select * from {{ ref('order_items') }} ), order_item_summary as ( select order_key, sum(gross_item_sales_amount) as gross_item_sales_amount, sum(item_discount_amount) as item_discount_amount, sum(item_tax_amount) as item_tax_amount, sum(net_item_sales_amount) as net_item_sales_amount from order_item group by 1 ), final as ( select orders.order_key, orders.order_date, orders.customer_key, orders.status_code, orders.priority_code, orders.clerk_name, orders.ship_priority, 1 as order_count, order_item_summary.gross_item_sales_amount, order_item_summary.item_discount_amount, order_item_summary.item_tax_amount, order_item_summary.net_item_sales_amount from orders inner join order_item_summary on orders.order_key = order_item_summary.order_key ) select * from final order by order_date ``` If a `node_color` is incompatible with dbt docs, you will see a compile error, as in the example below. ```shell Invalid color name for docs.node_color: aweioohafio23f. It is neither a valid HTML color name nor a valid HEX code. ``` dbt\_project.yml ```yml models: tpch: marts: core: materialized: table +docs: node_color: "aweioohafio23f" ``` --- ### docs-paths dbt\_project.yml ```yml docs-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [docs blocks](https://docs.getdbt.com/docs/build/documentation.md#docs-blocks) are located. #### Default (Applies to dbt v1.9 and later) By default, dbt will search in all resource paths for docs blocks (for example, the combined list of [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md), [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md), [analysis-paths](https://docs.getdbt.com/reference/project-configs/analysis-paths.md), [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md), [macro-paths](https://docs.getdbt.com/reference/project-configs/macro-paths.md), and [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md)). If this option is configured, dbt will *only* look in the specified directory for docs blocks. Paths specified in `docs-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/docs`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml docs-paths: ["docs"] ``` * ❌ **Don't** * Avoid absolute paths: ```yml docs-paths: ["/Users/username/project/docs"] ``` #### Example Use a subdirectory named `docs` for docs blocks: dbt\_project.yml ```yml docs-paths: ["docs"] ``` **Note:** We typically omit this configuration as we prefer dbt's default behavior. --- ### Doris/SelectDB configurations #### Models | Type | Supported? | Details | | --------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | view materialization | YES | Creates a [view](https://doris.apache.org/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-VIEW/). | | table materialization | YES | Creates a [table](https://doris.apache.org/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE/). | | incremental materialization | YES | Creates a table if it doesn't exist, and then item table model must be '[unique](https://doris.apache.org/docs/data-table/data-model#uniq-model/)'. | ##### View Materialization A dbt model can be created as a Doris view and configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: view ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config(materialized = "view") }} ``` ##### Table Materialization A dbt model can be created as a [Doris table](https://doris.apache.org/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE/) and configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: table +duplicate_key: [ <column-name>, ... ], +partition_by: [ <column-name>, ... ], +partition_type: <engine-type>, +partition_by_init: [<pertition-init>, ... ] +distributed_by: [ <column-name>, ... ], +buckets: int, +properties: {<key>:<value>,...} ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized = "table", duplicate_key = [ "<column-name>", ... ], partition_by = [ "<column-name>", ... ], partition_type = "<engine-type>", partition_by_init = ["<pertition-init>", ... ] distributed_by = [ "<column-name>", ... ], buckets = "int", properties = {"<key>":"<value>",...} ... ] ) }} ``` ###### Table Configuration | Option | Description | Required? | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | | `materialized` | How the model will be materialized into Doris. Must be `table` to create a table model. | Required | | `duplicate_key` | The key list of Doris table model :'[duplicate](https://doris.apache.org/docs/data-table/data-model#duplicate-model)'. | Required | | `partition_by` | The partition key list of Doris. ([Doris partition](https://doris.apache.org/docs/data-table/data-partition)) | Optional | | `partition_type` | The partition type of Doris. | Optional (default: `RANGE`) | | `partition_by_init` | The partition rule or some real partitions item. | Optional | | `distributed_by` | The bucket key list of Doris. ([Doris distribute](https://doris.apache.org/docs/data-table/data-partition#partitioning-and-bucket)) | Required | | `buckets` | The bucket number in one Doris partition. | Required | | `properties` | The other configuration of Doris. ([Doris properties](https://doris.apache.org/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE/?&_highlight=properties)) | Required | ##### Incremental Materialization An incremental Doris table, item table model must be 'unique' and is configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: incremental +unique_key: [ <column-name>, ... ], +partition_by: [ <column-name>, ... ], +partition_type: <engine-type>, +partition_by_init: [<pertition-init>, ... ] +distributed_by: [ <column-name>, ... ], +buckets: int, +properties: {<key>:<value>,...} ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized = "incremental", unique_key = [ "<column-name>", ... ], partition_by = [ "<column-name>", ... ], partition_type = "<engine-type>", partition_by_init = ["<pertition-init>", ... ] distributed_by = [ "<column-name>", ... ], buckets = "int", properties = {"<key>":"<value>",...} ... ] ) }} ``` ###### Incremental Table Configuration | Option | Description | Required? | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | | `materialized` | How the model will be materialized into Doris. Must be `table` to create a table model. | Required | | `unique_key` | The key list of Doris table model :'[Doris unique](https://doris.apache.org/docs/data-table/data-model#uniq-model)'. | Required | | `partition_by` | The partition key list of Doris. ([Doris partition](https://doris.apache.org/docs/data-table/data-partition)) | Optional | | `partition_type` | The partition type of Doris. | Optional (default: `RANGE`) | | `partition_by_init` | The partition rule or some real partitions item. | Optional | | `distributed_by` | The bucket key list of Doris. ([Doris distribute](https://doris.apache.org/docs/data-table/data-partition#partitioning-and-bucket)) | Required | | `buckets` | The bucket number in one Doris partition. | Required | | `properties` | The other configuration of Doris. ([Doris properties](https://doris.apache.org/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE/?&_highlight=properties)) | Required | --- ### DuckDB configurations These configurations are specific to `dbt-duckdb`. For profile setup and connection options, refer to [Connect DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md). For general dbt concepts, refer to [Materializations](https://docs.getdbt.com/docs/build/materializations.md) and [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md). Some features require a minimum version of `dbt-duckdb`. Version requirements are noted inline throughout this page. #### Secrets manager Use the [DuckDB Secrets Manager](https://duckdb.org/docs/configuration/secrets_manager.html) to manage credentials for cloud storage. Configure the `secrets` field in your profile: ```yml default: outputs: dev: type: duckdb path: /tmp/dbt.duckdb extensions: - httpfs - parquet secrets: - type: s3 region: my-aws-region key_id: "{{ env_var('S3_ACCESS_KEY_ID') }}" secret: "{{ env_var('S3_SECRET_ACCESS_KEY') }}" target: dev ``` ##### Fetch credentials from context Instead of specifying credentials directly, you can use the `credential_chain` secret provider to use any supported AWS mechanism (for example, web identity tokens). Refer to the [DuckDB secret providers documentation](https://duckdb.org/docs/configuration/secrets_manager.html#secret-providers) for details. ```yml secrets: - type: s3 provider: credential_chain ``` ##### Scoped credentials by storage prefix Secrets can be scoped so that different storage paths use different credentials: ```yml secrets: - type: s3 provider: credential_chain scope: [ "s3://bucket-in-eu-region", "s3://bucket-2-in-eu-region" ] region: "eu-central-1" - type: s3 region: us-west-2 scope: "s3://bucket-in-us-region" ``` When fetching a secret for a path, the secret scopes are compared to the path. In the case of multiple matching secrets, the longest prefix is chosen. #### Cloud storage with fsspec In `dbt-duckdb 1.4.1` and later, you can experimentally use DuckDB filesystems implemented via [fsspec](https://duckdb.org/docs/guides/python/filesystems.html). The `fsspec` library supports [a variety of cloud data storage systems](https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations), including S3, GCS, and Azure Blob Storage. To use an `fsspec` implementation, install the relevant Python modules and configure `filesystems` in your profile: ```yml default: outputs: dev: type: duckdb path: /tmp/dbt.duckdb filesystems: - fs: s3 anon: false key: "{{ env_var('S3_ACCESS_KEY_ID') }}" secret: "{{ env_var('S3_SECRET_ACCESS_KEY') }}" client_kwargs: endpoint_url: "http://localhost:4566" target: dev ``` Each entry must include an `fs` property that identifies the `fsspec` protocol to load (`s3`, `gcs`, `abfs`, etc.) and can include additional key-value pairs to configure that implementation. #### Arbitrary ATTACH options For the basic `attach` profile syntax, refer to [Connecting to DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md#attaching-additional-databases). Use the `options` dictionary when you need to pass additional key-value pairs to DuckDB's `ATTACH` statement: ```yml attach: - path: /tmp/db1.sqlite type: sqlite read_only: true - path: /tmp/special.duckdb options: cache_size: 1GB threads: 4 enable_fsst: true ``` If you specify the same option in both a direct field (`type`, `secret`, `read_only`) and in the `options` dict, `dbt-duckdb` raises an error to prevent conflicts. #### DuckLake [DuckLake](https://ducklake.select) is a table format that provides ACID transactions and time travel for DuckDB. You can use DuckLake with both local databases and MotherDuck. ##### DuckLake on MotherDuck In `dbt-duckdb 1.9.6` and later, you can connect to [hosted DuckLake on MotherDuck](https://motherduck.com/blog/ducklake-motherduck/) by creating a DuckLake database and setting `is_ducklake: true`. To set up DuckLake on MotherDuck: 1. Create your DuckLake database in MotherDuck: ```sql CREATE DATABASE my_ducklake (TYPE ducklake, DATA_PATH 's3://...') ``` 2. Configure your profile: ```yml default: outputs: dev: type: duckdb path: "md:my_db?motherduck_token={{ env_var('MOTHERDUCK_TOKEN') }}" attach: - path: "md:my_ducklake" is_ducklake: true target: dev ``` You must identify DuckLake must with `is_ducklake: true` so that dbt applies safe DDL operations. For local DuckLake, use `ducklake:` in the path: ```yml attach: - path: "ducklake:my_ducklake.ddb" ``` ##### DuckLake table partitioning For DuckLake-backed tables (including MotherDuck-managed DuckLake), you can configure physical partitioning for `table` or `incremental` models using `partitioned_by`: ```sql {{ config(materialized='table', partitioned_by=['year', 'month']) }} select *, year(event_time) as year, month(event_time) as month from {{ ref('upstream_model') }} ``` `partition_by` is accepted as an alias for `partitioned_by`. This setting is only applied for DuckLake relations; on non-DuckLake targets, it is ignored with a warning. DuckLake applies partitioning using `ALTER TABLE ... SET PARTITIONED BY (...)`, and partitioning only affects new data. For first builds or full refreshes, `dbt-duckdb` creates an empty table, sets partitioning, then inserts data so the initial load is partitioned. Refer to the [DuckLake partitioning documentation](https://ducklake.select/docs/stable/duckdb/advanced_features/partitioning) for more details. #### Incremental strategies `dbt-duckdb` supports the following strategies for incremental table models: * [`append`](#append-strategy) * [`delete+insert`](#deleteinsert-strategy) * [`merge`](#merge-strategy) * [`microbatch`](#microbatch-strategy) ##### Append strategy | Configuration | Type | Default | Description | | ------------------------ | -------- | ------- | ---------------------------------------------------- | | `incremental_predicates` | `<list>` | null | SQL conditions to filter which records get appended. | ##### Delete+insert strategy | Configuration | Type | Default | Description | | ------------------------ | ------------------- | ------- | ---------------------------------------------------------- | | `unique_key` | `<string>`/`<list>` | — | Required. Columns used to identify records for deletion. | | `incremental_predicates` | `<list>` | null | SQL conditions to filter the delete and insert operations. | ##### Merge strategy The `merge` strategy requires DuckDB 1.4.0 or later and provides access to DuckDB's native `MERGE` statement. **Basic configuration** When you specify only `unique_key`, `dbt-duckdb` uses DuckDB's `UPDATE BY NAME` and `INSERT BY NAME` operations, which automatically match columns by name: ```yml models: - name: my_incremental_model config: materialized: incremental incremental_strategy: merge unique_key: id ``` **Enhanced configuration** Additional options for finer control: | Configuration | Type | Default | Description | | ------------------------------ | --------------- | ------- | ------------------------------------------------------------- | | `unique_key` | `<string/list>` | — | Required. Columns used for the MERGE join condition. | | `incremental_predicates` | `<list>` | null | Additional SQL conditions to filter the MERGE operation. | | `merge_update_condition` | `<string>` | null | SQL condition to control when matched records are updated. | | `merge_insert_condition` | `<string>` | null | SQL condition to control when unmatched records are inserted. | | `merge_update_columns` | `<list>` | null | Specific columns to update. | | `merge_exclude_columns` | `<list>` | null | Columns to exclude from updates. | | `merge_update_set_expressions` | `<dict>` | null | Custom expressions for column updates. | For maximum flexibility, use `merge_clauses` to define custom `when_matched` and `when_not_matched` behaviors. When using DuckLake, MERGE statements are limited to a single UPDATE or DELETE action in `when_matched` clauses due to DuckLake's current MERGE implementation constraints. In conditions and expressions, use `DBT_INTERNAL_SOURCE` to reference the incoming data and `DBT_INTERNAL_DEST` to reference the existing target table. ##### Microbatch strategy The `microbatch` strategy requires dbt 1.9 or later and runs incremental builds in time-based batches using a configured `event_time` column. | Configuration | Type | Default | Description | | ------------------------ | ---------- | ------- | --------------------------------------------------------------------- | | `event_time` | `<string>` | — | Required. Name of the timestamp column used for microbatch windowing. | | `begin` | `<string>` | — | Required. Start time for batching (for example, `2025-01-01`). | | `batch_size` | `<string>` | — | Required. Batch grain (for example, `day`, `hour`). | | `incremental_predicates` | `<list>` | null | Optional additional predicates applied within each batch. | tip Microbatching might not always be the best option from a performance perspective. DuckDB operates on row groups, not physical partitions (unless you have explicitly partitioned data in a DuckLake). Be sure to test different amounts of threads to match your use case. #### More information * For connection modes and profile setup, refer to [Connect DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md). * For adapter source code and plugins, refer to the [`dbt-duckdb` repository](https://github.com/duckdb/dbt-duckdb). For adapter release notes, refer to the [`dbt-duckdb` releases page](https://github.com/duckdb/dbt-duckdb/releases). * For dbt concepts used on this page, refer to [Materializations](https://docs.getdbt.com/docs/build/materializations.md), [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md), and [Python models](https://docs.getdbt.com/docs/build/python-models.md). --- ### enabled ##### Models dbt\_project.yml ```yml models: <resource-path>: +enabled: true | false ``` models/\<modelname>.sql ```sql {{ config( enabled=true | false ) }} select ... ``` ##### Seeds dbt\_project.yml ```yml seeds: <resource-path>: +enabled: true | false ``` ##### Snapshots dbt\_project.yml ```yml snapshots: <resource-path>: +enabled: true | false ``` (Applies to dbt v1.9 and later) snapshots/snapshot\_name.yml ```yaml snapshots: - name: snapshot_name config: enabled: true | false ``` snapshots/\<filename>.sql ```sql # Configuring in a SQL file is a legacy method and not recommended. Use the property file instead. {% snapshot snapshot_name %} {{ config( enabled=true | false ) }} select ... {% endsnapshot %} ``` ##### Tests dbt\_project.yml ```yml data_tests: <resource-path>: +enabled: true | false ``` tests/\<filename>.sql ```sql {% test <testname>() %} {{ config( enabled=true | false ) }} select ... {% endtest %} ``` tests/\<filename>.sql ```sql {{ config( enabled=true | false ) }} ``` ##### Unit tests Did you know... Available from dbt v1.8 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). dbt\_project.yml ```yml unit_tests: <resource-path>: +enabled: true | false ``` models/\<filename>.yml ```yaml unit_tests: - name: [<test-name>] config: enabled: true | false ``` ##### Sources dbt\_project.yml ```yaml sources: <resource-path>: +enabled: true | false ``` models/properties.yml ```yaml sources: - name: [<source-name>] config: enabled: true | false tables: - name: [<source-table-name>] config: enabled: true | false ``` ##### Analyses analyses/\<filename>.yml ```yaml analyses: - name: <analysis_name> config: enabled: true | false ``` To configure analyses at the project level, set the [`require_corrected_analysis_fqns`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md) flag to `true` in your `dbt_project.yml`. dbt\_project.yml ```yaml flags: require_corrected_analysis_fqns: true analyses: +enabled: true | false ``` ##### Metrics dbt\_project.yml ```yaml metrics: <resource-path>: +enabled: true | false ``` models/metrics.yml ```yaml metrics: - name: [<metric-name>] config: enabled: true | false ``` ##### Exposures dbt\_project.yml ```yaml exposures: <resource-path>: +enabled: true | false ``` models/exposures.yml ```yaml exposures: - name: [<exposure-name>] config: enabled: true | false ``` ##### Semantic models dbt\_project.yml ```yaml semantic-models: <resource-path>: +enabled: true | false ``` (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: model_name semantic_model: enabled: true | false # Required under 'semantic_model' ``` ##### Saved queries dbt\_project.yml ```yaml saved-queries: <resource-path>: +enabled: true | false ``` models/semantic\_models.yml ```yaml saved_queries: - name: [<saved_query_name>] config: enabled: true | false ``` ##### Skills (Applies to dbt v2.0 and later) dbt\_project.yml ```yaml skills: <resource-path>: +enabled: true | false ``` For [agent skills](https://docs.getdbt.com/docs/dbt-ai/package-skills.md), the resource path is the package that ships the skill, followed by the skill's name: dbt\_project.yml ```yaml skills: demo_skills: naming-conventions: +enabled: false ``` A package's own `skills` config sets the defaults for the skills it ships, and your root project's `skills` config overrides it. Disabling a skill is how you resolve two packages that ship skills with the same name. #### Definition An optional configuration for enabling or disabling a resource. * Default: true When a resource is disabled, dbt will not consider it as part of your project. Note that this can cause compilation errors. If you instead want to exclude a model from a particular run, consider using the `--exclude` parameter as part of the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) If you are disabling models because they are no longer being used, but you want to version control their SQL, consider making them an [analysis](https://docs.getdbt.com/docs/build/analyses.md) instead. #### Examples ##### Disable a model in a package in order to use your own version of the model. This could be useful if you want to change the logic of a model in a package. For example, if you need to change the logic in the `segment_web_page_views` from the `segment` package ([original model](https://github.com/dbt-labs/segment/blob/a8ff2f892b009a69ec36c3061a87e437f0b0ea93/models/base/segment_web_page_views.sql)): 1. Add a model named `segment_web_page_views` (the same name) to your own project. 2. To avoid a compilation error due to duplicate models, disable the segment package's version of the model like so: dbt\_project.yml ```yml models: segment: base: segment_web_page_views: +enabled: false ``` (Applies to dbt v2.0 and later) ##### Disable Semantic Layer resources from a package Some packages may define Semantic Layer resources (semantic models, metrics, saved queries) using an older specification that isn’t compatible with dbt v2. To use these packages with dbt v2 while keeping your own semantic layer definitions, disable the package’s semantic layer resources in the relevant YAML file. dbt\_project.yml ```yml # Disable the package's time spine model (if it conflicts with yours) models: ad_reporting: semantic_models: metricflow_time_spine: +enabled: false # Disable all semantic layer resources from the package semantic-models: ad_reporting: +enabled: false metrics: ad_reporting: +enabled: false saved-queries: ad_reporting: +enabled: false ``` --- ### entry_point Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> config: entry_point: <string> # required for Snowflake and BigQuery; optional and ignored on Databricks ``` #### Definition When creating Python UDFs, specify the Python function to be called in `entry_point`. Python UDFs are currently supported in Snowflake, BigQuery, and Databricks. Each warehouse uses a different name for the entry point function. The following table shows what they’re called: | Warehouse | How `entry_point` is used | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Snowflake | Becomes the `HANDLER` name in `LANGUAGE PYTHON UDF` | | BigQuery | Becomes the `entry_point` in `OPTIONS(...)` | | Databricks | Accepted for compatibility but has no effect. The function body is used directly, so dbt displays a warning if you set it. Available in `dbt-databricks` v1.12+. | #### Example For example, if you have a Python UDF in `functions/my_function.py` with the following code which uses the function `main` as the entry point: functions/my\_function.py ```python import re def _digits_only(s: str) -> bool: return bool(re.search(r'^[0-9]+$', s or '')) def _to_flag(is_match: bool) -> int: return 1 if is_match else 0 def main(a_string: str) -> int: """ This is used as the entry point for the UDF. Returns 1 if a_string represents a positive integer (e.g., "10"), else 0. """ return _to_flag(_digits_only(a_string)) ``` After defining the UDF, you can specify `main` as the `entry_point` in the YAML file. `entry_point: main` points to the `main` function as the entry point for the UDF, while `_digits_only` and `_to_flag` are helper functions. functions/schema.yml ```yaml functions: - name: is_positive_int description: Returns 1 if a_string matches ^[0-9]+$, else 0 config: runtime_version: "3.11" # required entry_point: main # required: points to the function above arguments: - name: a_string data_type: string returns: data_type: integer ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [type](https://docs.getdbt.com/reference/resource-configs/type.md) * [volatility](https://docs.getdbt.com/reference/resource-configs/volatility.md) * [runtime\_version](https://docs.getdbt.com/reference/resource-configs/runtime-version.md) * [packages](https://docs.getdbt.com/reference/resource-configs/packages.md) * [arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) * [returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### Environment variable configs (Applies to dbt v1.11 and later) Environment variables use the `DBT_ENGINE_` prefix. For a list of all dbt environment variables you can set, refer to [Available flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). Env var ```text $ export DBT_ENGINE_<THIS-CONFIG>=True dbt run ``` For more detailed information, read our [environment variables page](https://docs.getdbt.com/docs/build/environment-variables.md). #### Config precedence There are multiple ways of setting flags, which depend on the use case: * **[CLI options](https://docs.getdbt.com/reference/global-configs/command-line-options.md):** Define behavior specific to *this invocation*. Supported for all dbt commands. * **[Environment variables](https://docs.getdbt.com/reference/global-configs/environment-variable-configs.md):** Define different behavior in different runtime environments (development vs. production vs. [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md)), or different behavior for different users in development (based on personal preferences). * **[Project-level `flags` in `dbt_project.yml`](https://docs.getdbt.com/reference/global-configs/project-flags.md):** Define version-controlled defaults for everyone running this project. Also, opt in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) to manage your migration off legacy functionality. * **[User settings (`~/.dbt/user_settings.yml`)](https://docs.getdbt.com/reference/global-configs/user-settings.md):** Define personal preferences that apply across all projects on your machine. Written automatically by `dbt login`. The most specific setting "wins." CLI options take the highest precedence, followed by environment variables, then `dbt_project.yml`, and finally `user_settings.yml`. If you set the flag in none of those places, it will use the default value defined within dbt. Most flags can be set in all three places: ```yaml # dbt_project.yml flags: # set default for running this project -- anywhere, anytime, by anyone fail_fast: true ``` (Applies to dbt v1.11 and later) ```bash # set this environment variable to 'True' (bash syntax) export DBT_ENGINE_FAIL_FAST=1 dbt run ``` ```bash dbt run --fail-fast # set to True for this specific invocation dbt run --no-fail-fast # set to False ``` There are two categories of exceptions: 1. **Flags setting file paths:** Flags for file paths that are relevant to runtime execution (for example, `--log-path` or `--state`) cannot be set in `dbt_project.yml`. To override defaults, pass CLI options or set environment variables ((Applies to dbt v1.11 and later) `DBT_ENGINE_LOG_PATH` and `DBT_ENGINE_STATE`). Flags that tell dbt where to find project resources (for example, `model-paths`) are set in `dbt_project.yml`, but as a top-level key, outside the `flags` dictionary; these configs are expected to be fully static and never vary based on the command or execution environment. 2. **Opt-in flags:** Flags opting in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) can *only* be defined in `dbt_project.yml`. These are intended to be set in version control and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. --- ### evaluate_volatile_sql ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +state: evaluate_volatile_sql: true | false ``` ##### Properties YAML file models/\<filename>.yml ```yaml models: - name: my_model config: state: evaluate_volatile_sql: true | false ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( state={ "evaluate_volatile_sql": true | false } ) }} ``` #### Definition dbt State determines whether a node's data has changed based on the freshness of its parents. However, volatile SQL functions like `CURRENT_TIMESTAMP()`, `RANDOM()`, or `UUID_STRING()` can change a node's output from one run to the next even when no upstream data has changed. By default, dbt State treats volatile functions as static parts of the node's compiled code, not as a source of new data. This is the right choice in most cases — otherwise, no node containing `CURRENT_TIMESTAMP()` would ever be reusable. When `evaluate_volatile_sql` is set to `true`, dbt State stores the *result* of each volatile function call and uses those stored values for future comparisons. This means the node is rebuilt when the function's output changes — for example, when `current_date()` returns a new value after midnight. | Value | Behavior | | ----------------- | ------------------------------------------------------------------------------------------------------------------------- | | `false` (default) | Volatile functions are treated as static code. The node can be reused even if function outputs would differ between runs. | | `true` | dbt State stores and compares the runtime result of volatile functions. The node rebuilds when those results change. | #### Examples ##### Rebuild when the date changes Use `evaluate_volatile_sql: true` for a node whose output is meaningfully affected by the current date: models/\<filename>.sql ```sql {{ config(state={"evaluate_volatile_sql": true}) }} select *, account_created_at >= current_date() as signed_up_today from {{ ref('dim_customers') }} ``` With this config enabled, dbt State stores the result of `current_date()` after each build. On the next run after midnight, the stored value no longer matches and the node is rebuilt, even if `dim_customers` hasn't changed. ##### Default behavior For nodes where volatile function output doesn't affect business logic, no config is required: models/\<filename>.sql ```sql select *, getdate() as _dbt_built_at from {{ ref('dim_customers') }} ``` dbt State treats `getdate()` as static code. The node remains reusable as long as its parents haven't changed. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) --- ### event_time Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). ##### Models dbt\_project.yml ```yml models: resource-path: +event_time: my_time_field ``` models/properties.yml ```yml models: - name: model_name config: event_time: my_time_field ``` models/modelname.sql ```sql {{ config( event_time='my_time_field' ) }} ``` ##### Seeds dbt\_project.yml ```yml seeds: resource-path: +event_time: my_time_field ``` seeds/properties.yml ```yml seeds: - name: seed_name config: event_time: my_time_field ``` ##### Snapshots dbt\_project.yml ```yml snapshots: resource-path: +event_time: my_time_field ``` (Applies to dbt v1.9 and later) snapshots/properties.yml ```yml snapshots: - name: snapshot_name config: event_time: my_time_field ``` ##### Sources dbt\_project.yml ```yml sources: resource-path: +event_time: my_time_field ``` models/properties.yml ```yml sources: - name: source_name config: event_time: my_time_field ``` #### Definition dbt uses `event_time` to understand when an event occurred. Configure it in your project YAML file (`dbt_project.yml`), properties YAML file (`models/properties.yml`), or SQL file config for [models](https://docs.getdbt.com/docs/build/models.md), [seeds](https://docs.getdbt.com/docs/build/seeds.md), or [sources](https://docs.getdbt.com/docs/build/sources.md). Required For incremental microbatch models, if your upstream models don't have `event_time` configured, dbt *cannot* automatically filter them during batch processing and will perform full table scans on every batch run. To avoid this, configure `event_time` on every upstream model that should be filtered. Learn how to exclude a model from auto-filtering by [opting out of auto-filtering](https://docs.getdbt.com/docs/build/incremental-microbatch.md#opting-out-of-auto-filtering). ##### Usage `event_time` is required for the [incremental microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) strategy(Applies to dbt v1.10 and later) , the [`--sample` flag](https://docs.getdbt.com/docs/build/sample-flag.md), and highly recommended for [Advanced CI's compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#optimizing-comparisons) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments. ##### Best practices Set the `event_time` to the name of the field that represents the actual timestamp of the event (like `account_created_at`). The timestamp of the event should represent "at what time did the row occur" rather than an event ingestion date. Marking a column as the `event_time` when it isn't, diverges from the semantic meaning of the column which may result in user confusion when other tools make use of the metadata. However, if an ingestion date (like `loaded_at`, `ingested_at`, or `last_updated_at`) are the only timestamps you use, you can set `event_time` to these fields. Here are some considerations to keep in mind if you do this: * Using `last_updated_at` or `loaded_at` — May result in duplicate entries in the resulting table in the data warehouse over multiple runs. Setting an appropriate [lookback](https://docs.getdbt.com/reference/resource-configs/lookback.md) value can reduce duplicates but it can't fully eliminate them since some updates outside the lookback window won't be processed. * Using `ingested_at` — Since this column is created by your ingestion/EL tool instead of coming from the original source, it will change if/when you need to resync your connector for some reason. This means that data will be reprocessed and loaded into your warehouse for a second time against a second date. As long as this never happens (or you run a full refresh when it does), microbatches will be processed correctly when using `ingested_at`. Here are some examples of recommended and not recommended `event_time` columns: | Status | Column name | Description | | ------------------ | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ Recommended | `account_created_at` | Represents the specific time when an account was created, making it a fixed event in time. | | ✅ Recommended | `session_began_at` | Captures the exact timestamp when a user session started, which won’t change and directly ties to the event. | | ❌ Not recommended | `_fivetran_synced` | This represents the time the event was ingested, not when it happened. | | ❌ Not recommended | `last_updated_at` | Changes over time and isn't tied to the event itself. If used, note the considerations mentioned earlier in [best practices](#best-practices). | #### Examples ##### Models Here's an example in the `dbt_project.yml` file: dbt\_project.yml ```yml models: my_project: user_sessions: +event_time: session_start_time ``` Example in a property file: models/properties.yml ```yml models: - name: user_sessions config: event_time: session_start_time ``` Example in a config block for a model: models/user\_sessions.sql ```sql {{ config( event_time='session_start_time' ) }} ``` This setup sets `session_start_time` as the `event_time` for the `user_sessions` model. ##### Seeds Here's an example in the `dbt_project.yml` file: dbt\_project.yml ```yml seeds: my_project: my_seed: +event_time: record_timestamp ``` Example in a seed properties YAML: seeds/properties.yml ```yml seeds: - name: my_seed config: event_time: record_timestamp ``` This setup sets `record_timestamp` as the `event_time` for `my_seed`. ##### Snapshots Here's an example in the `dbt_project.yml` file: dbt\_project.yml ```yml snapshots: my_project: my_snapshot: +event_time: record_timestamp ``` Example in a snapshot properties YAML: my\_project/properties.yml ```yml snapshots: - name: my_snapshot config: event_time: record_timestamp ``` This setup sets `record_timestamp` as the `event_time` for `my_snapshot`. ##### Sources Here's an example of a source property file: models/properties.yml ```yml sources: - name: source_name tables: - name: table_name config: event_time: event_timestamp ``` This setup sets `event_timestamp` as the `event_time` for the specified source table. --- ### Events and logs As dbt runs, it generates events. The most common way to see those events is as log messages, written in real time to two places: * The command line terminal (`stdout`), to provide interactive feedback while running dbt. * The debug log file (`logs/dbt.log`), to enable detailed [debugging of errors](https://docs.getdbt.com/guides/debug-errors.md) when they occur. The text-formatted log messages in this file include all `DEBUG`-level events, as well as contextual information, such as log level and thread name. The location of this file can be configured via [the `log-path` flag](https://docs.getdbt.com/reference/global-configs/logs.md). CLI ```bash 21:35:48 6 of 7 OK created view model dbt_testing.name_list......................... [CREATE VIEW in 0.17s] ``` logs/dbt.log ```text ============================== 21:21:15.272780 | 48cef052-3819-4550-a83a-4a648aef5a31 ============================== 21:21:15.272780 [info ] [MainThread]: Running with dbt=1.5.0-b5 21:21:15.273802 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'fail_fast': 'False', 'debug': 'False', 'log_path': '/Users/jerco/dev/scratch/testy/logs', 'profiles_dir': '/Users/jerco/.dbt', 'version_check': 'False', 'use_colors': 'False', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'static_parser': 'True', 'introspect': 'True', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'target_path': 'None', 'send_anonymous_usage_stats': 'True'} 21:21:16.190990 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed. 21:21:16.191404 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing 21:21:16.207330 [info ] [MainThread]: Found 2 models, 0 tests, 0 snapshots, 1 analysis, 535 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics, 0 groups ``` #### Structured logging (Applies to dbt v2.0 and later) dbt v2 emits structured run telemetry using [OpenTelemetry](https://opentelemetry.io/) conventions instead of dbt v1's JSON event logs. To view it locally, use `--log-format otel` or the options in [dbt v2 telemetry and observability](https://docs.getdbt.com/reference/telemetry-observability.md#available-output-formats). In dbt platform, you can [download OTel logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs) from dbt v2 job runs. For `--log-format`, `--log-level`, and related CLI configs, refer to [Logs](https://docs.getdbt.com/reference/global-configs/logs.md). #### Python interface (Applies to dbt v2.0 and later) dbt v2 doesn't share dbt v1's Python event interface. We are currently [developing](https://github.com/dbt-labs/dbt/issues/13102) a Python API with Rust bindings for programmatic invocations. --- ### Exasol configurations #### Incremental materialization strategies In dbt-exasol, the following incremental materialization strategies are supported: * `append` (default when `unique_key` is not defined) * `merge` * `delete+insert` (default when `unique_key` is defined) * [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md) All of these strategies are inherited from dbt. For more information on incremental strategies, refer to the [incremental strategy documentation](https://docs.getdbt.com/docs/build/incremental-strategy.md). #### Performance optimizations ##### Table distribution and partitioning Starting from dbt-exasol 1.8.1, you can configure table distribution and partitioning strategies to optimize query performance in Exasol. These configurations are available for models materialized as `table` or `incremental`. Exasol supports the following performance optimization configurations: | Parameter | Type | Required | Description | | ---------------------- | -------------------------- | -------- | -------------------------------------------------------------------------- | | `partition_by_config` | `<string>` or `[<string>]` | no | Partitions the table by specified column(s) for improved query performance | | `distribute_by_config` | `<string>` | no | Distributes data across cluster nodes by specified column | | `primary_key_config` | `[<string>]` | no | Defines primary key constraint(s) | ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: table +partition_by_config: <column-name> +distribute_by_config: <column-name> +primary_key_config: [<column-name>] ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: table partition_by_config: <column-name> distribute_by_config: <column-name> primary_key_config: [<column-name>] ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized="table", partition_by_config="<column-name>", distribute_by_config="<column-name>", primary_key_config=["<column-name>"] ) }} ``` ###### Single column example The following example creates a table partitioned by `order_date`, distributed by `customer_id`, with a primary key on `customer_id`: models/orders.sql ```sql {{ config( materialized='table', primary_key_config=['customer_id'], partition_by_config='order_date', distribute_by_config='customer_id' ) }} select customer_id, order_date, order_total, order_status from {{ source('sales', 'orders') }} ``` ###### Multiple columns example When configuring multiple columns for `primary_key_config`, provide them as a list: models/order\_items.sql ```sql {{ config( materialized='incremental', primary_key_config=['order_id', 'item_id'], partition_by_config='order_date', distribute_by_config='order_id', unique_key=['order_id', 'item_id'] ) }} select order_id, item_id, order_date, quantity, price from {{ source('sales', 'order_items') }} ``` info When configuring multiple columns for `primary_key_config`, always provide them as a list: `['column1', 'column2']` For more information about Exasol's table distribution and partitioning, refer to the [Exasol documentation](https://docs.exasol.com/db/latest/sql/create_table.htm). #### Model contracts Exasol supports [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) with the following database constraints: | Constraint Type | Support Status | Description | | --------------- | ---------------- | ------------------------------------------- | | `not_null` | ✅ Enforced | Prevents NULL values in the column | | `primary_key` | ✅ Enforced | Enforces uniqueness and NOT NULL | | `foreign_key` | ✅ Enforced | References another table's primary key | | `check` | ❌ Not supported | Custom validation expressions not supported | | `unique` | ❌ Not supported | Unique constraints not supported | ##### Example with enforced constraints models/customers.yml ```yaml models: - name: customers config: contract: enforced: true columns: - name: customer_id data_type: integer constraints: - type: not_null - type: primary_key - name: email data_type: varchar(255) constraints: - type: not_null - name: country_id data_type: integer constraints: - type: foreign_key expression: countries (country_id) ``` For more information on model contracts, refer to the [model contracts documentation](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md). #### Timestamp format Starting from dbt-exasol 1.2.2, the default timestamp format is `YYYY-MM-DDTHH:MI:SS.FF6`. You can customize the timestamp format in your [profile configuration](https://docs.getdbt.com/docs/local/connect-data-platform/exasol-setup.md): profiles.yml ```yaml outputs: dev: type: exasol timestamp_format: 'YYYY-MM-DD HH24:MI:SS.FF3' # ... other settings ``` ##### Microbatch strategy considerations When using the [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md) incremental strategy, Exasol requires timestamps without timezone suffix in model definitions: ```sql -- ✅ Correct (Exasol compatible) TIMESTAMP '2024-01-01 10:00:00' -- ❌ Incorrect (will cause parse errors) TIMESTAMP '2024-01-01 10:00:00-0' ``` The dbt-exasol adapter automatically handles timestamp formatting for microbatch boundaries. For more information about the microbatch strategy, refer to the [microbatch documentation](https://docs.getdbt.com/docs/build/incremental-microbatch.md). #### Materialized views Exasol does not support materialized views. If you attempt to use `materialized='materialized_view'`, the operation will fail with an error. ##### Workarounds * Use `materialized='table'` with appropriate refresh logic * Use `materialized='incremental'` with suitable [incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) #### Clone operations Exasol does not support table cloning operations. This affects dbt features that rely on `CLONE` functionality. #### Unit test limitations Exasol has specific limitations with [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md): ##### Empty string handling In Exasol, empty strings are treated as `NULL`. This affects test fixtures that use empty string literals to simulate empty values. When writing unit tests with seed data, be aware that: ```yaml # This seed data id,name,value 1,test,"" # Empty string # Will be interpreted as id,name,value 1,test,NULL # NULL value in Exasol ``` ##### Cross-database testing Unit tests that rely on sources in a database different from the models are not supported. All test fixtures and models must exist in the same database. ##### Aggregate functions in CTEs Exasol does not support certain aggregate functions (`LISTAGG`, `MEDIAN`, `PERCENTILE_CONT`) when used within common table expressions (CTEs) created from dbt's unit test fixtures. These functions require user-created tables. **Workaround:** Create actual tables for test fixtures rather than using inline CTEs when testing models with these functions. If you are interested in supporting materialized test fixtures, we encourage you to participate in this issue in GitHub: [dbt-labs/dbt#8499](https://github.com/dbt-labs/dbt/issues/8499) #### Connection configuration For information about connection parameters such as encryption, SSL/TLS validation, OpenID authentication, and other profile settings, refer to the [Exasol setup documentation](https://docs.getdbt.com/docs/local/connect-data-platform/exasol-setup.md). --- ### Exclude models from your run ##### Excluding models dbt provides an `--exclude` flag with the same semantics as `--select`. Models specified with the `--exclude` flag will be removed from the set of models selected with `--select`. ```bash dbt run --select "my_package".*+ --exclude "my_package.a_big_model+" # select all models in my_package and their children except a_big_model and its children ``` Exclude a specific resource by its name or lineage: ```bash # test dbt test --exclude "not_null_orders_order_id" # test all models except the not_null_orders_order_id test dbt test --exclude "orders" # test all models except tests associated with the orders model # seed dbt seed --exclude "account_parent_mappings" # load all seeds except account_parent_mappings # snapshot dbt snapshot --exclude "snap_order_statuses" # execute all snapshots except snap_order_statuses ``` --- ### execute_hooks_on_any_reuse ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +state: execute_hooks_on_any_reuse: true | false ``` ##### Properties YAML file models/\<filename>.yml ```yaml models: - name: my_model config: state: execute_hooks_on_any_reuse: true | false ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( state={ "execute_hooks_on_any_reuse": true | false } ) }} ``` #### Definition When dbt State skips a node because it's still fresh, that node's pre- and post-hooks are not executed by default. This matches dbt's standard behavior: if the node wasn't executed, its hooks don't run. Set `execute_hooks_on_any_reuse: true` if you have audit hooks or other hooks that must run on every job invocation, regardless of whether the node was rebuilt or reused. | Value | Behavior | | ----------------- | ----------------------------------------------------------- | | `false` (default) | Hooks are skipped when a node is reused without rebuilding. | | `true` | Hooks run even when a node is reused. | Hooks always execute when dbt State *clones* a node from another environment, because the clone step creates a new object in the warehouse. #### Example ##### Always run audit hooks Use `execute_hooks_on_any_reuse: true` for nodes with audit or lineage hooks that must run on every job invocation: models/fct\_orders.yml ```yaml models: - name: fct_orders config: state: execute_hooks_on_any_reuse: true post-hook: - "INSERT INTO audit_log VALUES ('{{ model.name }}', CURRENT_TIMESTAMP())" ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) --- ### Exit codes When dbt exits, it will return an exit code of either 0, 1, or 2. | Exit Code | Condition | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | The dbt invocation completed without error. | | 1 | The dbt invocation completed with at least one handled error (eg. model syntax error, bad permissions, etc). The run was completed, but some models may have been skipped. | | 2 | The dbt invocation completed with an unhandled error (eg. ctrl-c, network interruption, etc). | While these exit codes may change in the future, a zero exit code will always imply success whereas a nonzero exit code will always imply failure. --- ### Exposure properties #### Related documentation * [Using exposures](https://docs.getdbt.com/docs/build/exposures.md) * [Declaring resource properties](https://docs.getdbt.com/reference/configs-and-properties.md) #### Overview Exposures are defined in `properties.yml` files nested under an `exposures:` key. You may define `exposures` in YAML files that also define `sources` or `models`. Exposure properties are "special properties" in that you can't configure them in the `dbt_project.yml` file or using `config()` blocks. Refer to [Configs and properties](https://docs.getdbt.com/reference/define-properties#which-properties-are-not-also-configs) for more info.<br /> Note that while most exposure properties must be configured directly in these YAML files, you can set the [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) config at the [project level](#project-level-configs) in the`dbt_project.yml` file. You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `models/` directory. Exposure names must contain only letters, numbers, and underscores (no spaces or special characters). For a short human-friendly name with title casing, spaces, and special characters, use the `label` property. models/\<filename>.yml ```yml exposures: - name: <string_with_underscores> description: <markdown_string> type: {dashboard, notebook, analysis, ml, application} url: <string> maturity: {high, medium, low} # Indicates level of confidence or stability in the exposure enabled: true | false config: # 'tags' and 'meta' changed to config in v1.10 tags: [<string>] meta: {<dictionary>} enabled: true | false owner: # supports 'name' and 'email' only name: <string> email: <string> depends_on: - ref('model') - ref('seed') - source('name', 'table') - metric('metric_name') label: "Human-Friendly Name for this Exposure!" - name: ... # declare properties of additional exposures ``` #### Example models/jaffle/exposures.yml ```yaml exposures: - name: weekly_jaffle_metrics label: Jaffles by the Week # optional type: dashboard # required maturity: high # optional url: https://bi.tool/dashboards/1 # optional description: > # optional Did someone say "exponential growth"? depends_on: # expected - ref('fct_orders') - ref('dim_customers') - source('gsheets', 'goals') - metric('count_orders') owner: name: Callum McData email: data@jaffleshop.com - name: jaffle_recommender maturity: medium type: ml url: https://jupyter.org/mycoolalg description: > Deep learning to power personalized "Discover Sandwiches Weekly" depends_on: - ref('fct_orders') owner: name: Data Science Drew email: data@jaffleshop.com - name: jaffle_wrapped type: application description: Tell users about their favorite jaffles of the year depends_on: [ ref('fct_orders') ] owner: { email: summer-intern@jaffleshop.com } ``` ###### Project-level configs You can define project-level configs for exposures in the `dbt_project.yml` file under the `exposures:` key using the `+` prefix. Currently, only the [`enabled` config](https://docs.getdbt.com/reference/resource-configs/enabled.md) is supported: dbt\_project.yml ```yml name: 'project_name' # rest of dbt_project.yml exposures: +enabled: true ``` --- ### external models/\<filename>.yml ```yml sources: - name: <source_name> tables: - name: <table_name> external: location: <string> file_format: <string> row_format: <string> tbl_properties: <string> partitions: - name: <column_name> data_type: <string> description: <string> meta: {dictionary} - ... <additional_property>: <additional_value> ``` #### Definition Use `external` on a source to describe a table that reads files in cloud storage, such as S3 or GCS. dbt stores these properties in the manifest, and the [`dbt-external-tables`](https://github.com/dbt-labs/dbt-external-tables) package uses them to create the table in your warehouse. There are optional built-in properties, with simple type validation, that roughly correspond to the Hive external table spec. dbt doesn't validate most of these properties, so you can add your own too. You may wish to define the `external` property in order to: * Power macros that introspect [`graph.sources`](https://docs.getdbt.com/reference/dbt-jinja-functions/graph.md) * Define metadata that you can later extract from the [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) For an example of how this property can be used to power custom workflows, see the [`dbt-external-tables`](https://github.com/dbt-labs/dbt-external-tables) package. #### Partitions A partition is a column whose value comes from the file's folder path, for example `dt` in `events/dt=2025-01-01/`. Declaring one lets your data platform skip folders you don't query. Which properties you need depends on your data platform. For example: | Data platform | Properties | | ------------- | ----------------------------------------- | | Snowflake | `name`, `data_type`, `expression` | | BigQuery | `name`, `data_type` | | Redshift | `name`, `data_type`, `vals`, `path_macro` | | Spark | `name` | For the full list, see the [`dbt-external-tables` sample sources](https://github.com/dbt-labs/dbt-external-tables/tree/main/sample_sources). info Keep `meta` directly on the partition. dbt doesn't move it under `config` for `external`, and `dbt-external-tables` doesn't read `config.meta`. #### Examples The following examples show an external table over event files in cloud storage, partitioned by date (notice the different partition properties depending on your data platform). ##### Snowflake ```yml sources: - name: snowplow tables: - name: event_ext_tbl external: location: "@raw.snowplow.snowplow" # an existing external stage file_format: "( type = json )" auto_refresh: true partitions: - name: collector_hour data_type: timestamp expression: to_timestamp(substr(metadata$filename, 8, 13), 'YYYY/MM/DD/HH24') ``` `metadata$filename` is the file path. The `expression` pulls the hour out of it. ##### BigQuery ```yml sources: - name: snowplow tables: - name: event external: location: 'gs://bucket/path/*' options: format: csv skip_leading_rows: 1 hive_partition_uri_prefix: 'gs://bucket/path/' partitions: - name: collector_date data_type: date ``` BigQuery reads the partition from Hive-style folders, such as `gs://bucket/path/collector_date=2020-01-01/`. You only name the column and give its type. ##### Redshift ```yml sources: - name: snowplow tables: - name: event external: location: "s3://bucket/path" row_format: > serde 'org.openx.data.jsonserde.JsonSerDe' partitions: - name: appId data_type: varchar(255) vals: - dev - prod path_macro: dbt_external_tables.key_value ``` Redshift needs the values (`vals`) and a macro (`path_macro`) that turns each value into a folder path. --- ### fail_calc Test queries are written to return a set of failing records, ones not matching the expectation or assertion declared by that test: duplicate records, null values, etc. Most often, this is the count of rows returned by the test query: the default value of `fail_calc` is `count(*)`. But it can also be a custom calculation, whether an aggregate calculation or simply the name of a column to be selected from the test query. Most tests do not use the `fail_calc` config, preferring to return a count of failing rows. For the tests that do, the most common place to set the `fail_calc` config is right within a generic test block, alongside its query definition. All the same, `fail_calc` can be set in all the same places as other configs. For instance, you can configure a `unique` test to return `sum(n_records)` instead of `count(*)` as the failure calculation: that is, the number of rows in the model containing a duplicated column value, rather than the number of distinct column values that are duplicated. Tip Beware using functions like `sum()` for `fail_calc` in any test that has the potential to return no rows at all. If no rows are returned, the test won't pass or fail but will return the following error: ```text None is not of type 'integer' Failed validating 'type' in schema['properties']['failures']: {'type': 'integer'} On instance['failures']: None ``` To avoid this issue, use a case statement to ensure that `0` is returned when no rows exist: ```yaml fail_calc: "case when count(*) > 0 then sum(n_records) else 0 end" ``` ##### Specific test Configure a specific instance of a generic (schema) test: models/\<filename>.yml ```yaml models: - name: my_model columns: - name: my_columns data_tests: - unique: config: fail_calc: "case when count(*) > 0 then sum(n_records) else 0 end" ``` ##### One-off test Configure a one-off (data) test: tests/\<filename>.sql ```sql {{ config(fail_calc = "sum(total_revenue) - sum(revenue_accounted_for)") }} select ... ``` ##### Generic test block Set the default for all instances of a generic (schema) test, by setting the config inside its test block (definition): macros/\<filename>.sql ```sql {% test <testname>(model, column_name) %} {{ config(fail_calc = "missing_in_a + missing_in_b") }} select ... {% endtest %} ``` ##### Project level Set the default for all tests in a package or project: dbt\_project.yml ```yaml data_tests: +fail_calc: count(*) # all tests <package_name>: +fail_calc: count(distinct id) # tests in <package_name> ``` --- ### Failing fast Supply the `-x` or `--fail-fast` flag to `dbt run`, `dbt build`, or `dbt test` to make dbt exit immediately on the first failure. `fail_fast` is a global flag, so it stops on both run errors and test errors —— not just run errors. If other models are in-progress when the first failure happens, then dbt will terminate the connections for these still-running models. For the full precedence rules and environment-variable equivalent (`DBT_FAIL_FAST`), refer to [`fail_fast` row in About global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). For example, you can select four models to run, but if a failure occurs in the first model, the failure will prevent other models from running: ```text $ dbt run -x --threads 1 Running with dbt=1.0.0 Found 4 models, 1 test, 1 snapshot, 2 analyses, 143 macros, 0 operations, 1 seed file, 0 sources 14:47:39 | Concurrency: 1 threads (target='dev') 14:47:39 | 14:47:39 | 1 of 4 START table model test_schema.model_1........... [RUN] 14:47:40 | 1 of 4 ERROR creating table model test_schema.model_1.. [ERROR in 0.06s] 14:47:40 | 2 of 4 START view model test_schema.model_2............ [RUN] 14:47:40 | CANCEL query model.debug.model_2....................... [CANCEL] 14:47:40 | 2 of 4 ERROR creating view model test_schema.model_2... [ERROR in 0.05s] Database Error in model model_1 (models/model_1.sql) division by zero compiled SQL at target/run/debug/models/model_1.sql Encountered an error: FailFast Error in model model_1 (models/model_1.sql) Failing early due to test failure or runtime error ``` --- ### Failures in on-run-start hooks Removed in v2 This flag was removed in v2. The new behavior is always enabled. If you're upgrading, remove this flag from your `dbt_project.yml`. | skip\_nodes\_if\_on\_run\_start\_fails | dbt **v1 Latest** | dbt v1 | | -------------------------------------- | ----------------- | ------ | | Introduced | 2024.10 | 1.9.0 | | Matured (default → `true`) | 2026.09 | 1.12.0 | | Removed | — | v2.0 | <br /> Starting in dbt v1.12, `skip_nodes_if_on_run_start_fails` defaults to `true`, skipping all selected resources if there is a failure on an `on-run-start` hook. For more information, refer to [`on-run-start` / `on-run-end`](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md). #### Impact If your project uses `on-run-start` hooks for non-critical work (for example, telemetry, notifications, audit inserts, attaching session settings), your build stops producing output whenever a hook fails. Tables and views that previously refreshed daily will stop updating the next time the hook fails.  Recommended actions If every model is skipped after an `on-run-start` hook fails, find which `on-run-start` hook failed in the run log — it appears as an `ERROR` immediately before all the nodes marked `SKIP`. From there, either remove the hook if it's not required, or fix the root cause of the failure by reviewing the hook's SQL statements. To opt out of this behavior, set the flag to `false`: dbt\_project.yml ```yaml flags: skip_nodes_if_on_run_start_fails: false ``` --- ### Fast test edges Experimental This flag is experimental and has not been validated to produce the same results across all project configurations. The `--use-fast-test-edges` flag reduces the number of test edges dbt adds to the execution graph during `dbt build`. When you run `dbt build`, dbt adds edges so tests on upstream nodes tend to finish before downstream models run. In large projects, that can add a large number of edges, which slows the run and increases memory use. Using the `--use-fast-test-edges` flag with `dbt build` produces the same execution order with significantly fewer edges. If you experience slow run times or high memory usage during `dbt build`, use this flag. You can set this flag using the CLI or an environment variable: CLI ```bash dbt build --use-fast-test-edges ``` Env var ```bash export DBT_ENGINE_USE_FAST_TEST_EDGES=true dbt build ``` --- ### Firebolt configurations #### Setting `quote_columns` To prevent a warning, make sure to explicitly set a value for `quote_columns` in your `dbt_project.yml`. See the [doc on quote\_columns](https://docs.getdbt.com/reference/resource-configs/quote_columns.md) for more information. ```yaml seeds: +quote_columns: false #or `true` if you have CSV column headers with spaces ``` #### Model configuration for fact tables A dbt model can be created as a Firebolt fact table and configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: table +table_type: fact +primary_index: [ <column-name>, ... ] +indexes: - index_type: aggregating key_columns: [ <column-name>, ... ] aggregation: [ <agg-sql>, ... ] ... ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: <model-name> config: materialized: table table_type: fact primary_index: [ <column-name>, ... ] indexes: - index_type: aggregating key_columns: [ <column-name>, ... ] aggregation: [ <agg-sql>, ... ] ... ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized = "table" table_type = "fact" primary_index = [ "<column-name>", ... ], indexes = [ { "index_type": "aggregating" "key_columns": [ "<column-name>", ... ], "aggregation": [ "<agg-sql>", ... ], }, ... ] ) }} ``` ###### Fact table configurations | Configuration | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a fact table. | | `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. | | `primary_index` | Sets the primary index for the fact table using the inputted list of column names from the model. Required for fact tables. | | `indexes` | A list of aggregating indexes to create on the fact table. | | `index_type` | Specifies that the index is an [aggregating index](https://docs.firebolt.io/godocs/Guides/working-with-indexes/using-aggregating-indexes.html). Should be set to `aggregating`. | | `key_columns` | Sets the grouping of the aggregating index using the inputted list of column names from the model. | | `aggregation` | Sets the aggregations on the aggregating index using the inputted list of SQL agg expressions. | ###### Example of a fact table with an aggregating index ```text {{ config( materialized = "table", table_type = "fact", primary_index = "id", indexes = [ { "index_type": "aggregating", "key_columns": "order_id", "aggregation": ["COUNT(DISTINCT status)", "AVG(customer_id)"] } ] ) }} ``` #### Model configuration for dimension tables A dbt model can be materialized as a Firebolt dimension table and configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: table +table_type: dimension ... ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: <model-name> config: materialized: table table_type: dimension ... ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized = "table", table_type = "dimension", ... ) }} ``` Dimension tables do not support aggregation indexes. ###### Dimension table configurations | Configuration | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a dimension table. | | `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. | #### How aggregating indexes are named In dbt-firebolt, you do not provide names for aggregating indexes; they are named programmatically. dbt will generate index names using the following convention: ```text <table-name>__<key-column>__<index-type>_<unix-timestamp-at-execution> ``` For example, a join index could be named `my_users__id__join_1633504263` and an aggregating index could be named `my_orders__order_date__aggregating_1633504263`. #### Managing ingestion via external tables `dbt-firebolt` supports dbt's [external tables feature](https://docs.getdbt.com/reference/resource-properties/external.md), which allows dbt to manage the table ingestion process from S3 into Firebolt. This is an optional feature but can be highly convenient depending on your use case. More information on using external tables including properly configuring IAM can be found in the Firebolt [documentation](https://docs.firebolt.io/godocs/Guides/loading-data/working-with-external-tables.html). ###### Installation of external tables package To install and use `dbt-external-tables` with Firebolt, you must: 1. Add this package to your packages.yml: ```yaml packages: - package: dbt-labs/dbt_external_tables version: <version> ``` 2. Add these fields to your `dbt_project.yml`: ```yaml dispatch: - macro_namespace: dbt_external_tables search_order: ['dbt', 'dbt_external_tables'] ``` 3. Pull in the `packages.yml` dependencies by calling `dbt deps`. ###### Using external tables To use external tables, you must define a table as `external` in your `dbt_project.yml` file. Every external table must contain the fields `url`, `type`, and `object_pattern`. Note that the Firebolt external table specification requires fewer fields than what is specified in the dbt documentation. In addition to specifying the columns, an external table may specify partitions. Partitions are not columns and they cannot have the same name as columns. To avoid YAML parsing errors, remember to encase string literals (such as the `url` and `object_pattern` values) in single quotation marks. ###### dbt\_project.yml syntax for an external table ```yml sources: - name: firebolt_external schema: "{{ target.schema }}" loader: S3 tables: - name: <table-name> external: url: 's3://<bucket_name>/' object_pattern: '<regex>' type: '<type>' credentials: aws_key_id: <key-id> aws_secret_key: <key-secret> object_pattern: '<regex>' compression: '<compression-type>' partitions: - name: <partition-name> data_type: <partition-type> regex: '<partition-definition-regex>' columns: - name: <column-name> data_type: <type> ``` `aws_key_id` and `aws_secret_key` are the credentials that allow Firebolt access to your S3 bucket. Learn how to set them up by following this [guide](https://docs.firebolt.io/godocs/Guides/loading-data/creating-access-keys-aws.html). If your bucket is public these parameters are not necessary. ###### Running external tables The `stage_external_sources` macro is inherited from the [dbt-external-tables package](https://github.com/dbt-labs/dbt-external-tables#syntax) and is the primary point of entry when using this package. It has two operational modes: standard and "full refresh." ```bash # iterate through all source nodes, create if missing, refresh metadata $ dbt run-operation stage_external_sources # iterate through all source nodes, create or replace (no refresh command is required as data is fetched live from remote) $ dbt run-operation stage_external_sources --vars "ext_full_refresh: true" ``` #### Incremental models The [`incremental_strategy` configuration](https://docs.getdbt.com/docs/build/incremental-strategy.md) controls how dbt builds incremental models. Firebolt currently supports `append`, `insert_overwrite` and `delete+insert` configuration. You can specify `incremental_strategy` in `dbt_project.yml` or within a model file's `config()` block. The `append` configuration is the default. Specifying this configuration is optional. The `append` strategy performs an `INSERT INTO` statement with all the new data based on the model definition. This strategy doesn't update or delete existing rows, so if you do not filter the data to the most recent records only, it is likely that duplicate records will be inserted. Example source code: ```text {{ config( materialized = 'incremental', incremental_strategy='append' ) }} /* All rows returned by this query will be appended to the existing model */ select * from {{ ref('raw_orders') }} {% if is_incremental() %} where order_date > (select max(order_date) from {{ this }}) {% endif %} ``` Example run code: ```sql CREATE DIMENSION TABLE IF NOT EXISTS orders__dbt_tmp AS SELECT * FROM raw_orders WHERE order_date > (SELECT MAX(order_date) FROM orders); INSERT INTO orders VALUES ([columns]) SELECT ([columns]) FROM orders__dbt_tmp; ``` #### Seeds behavior When running the `dbt seed` command we perform a `DROP CASCADE` operation instead of `TRUNCATE`. #### Practice You can look at our modified version of the jaffle\_shop, [jaffle\_shop\_firebolt](https://github.com/firebolt-db/jaffle_shop_firebolt), to see how indexes, as well as external tables, can be set or clone and execute the commands listed in the README.md --- ### freshness Use the `freshness` config to declare how fresh your [source](#source-freshness) or [model](#model-freshness) data should be. (Applies to dbt v2.0 and later) Run [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) to check every source and model with freshness configured. note `dbt source freshness` is a legacy command that checks freshness for sources *only*. It's still supported for backward compatibility and continues to produce `sources.json`, but we recommend using `dbt freshness` going forward. #### Configuration Use the following fields to configure freshness for sources and models unless otherwise noted: | Field | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `warn_after` | How old the most recent data can be before a freshness check reports a warning. Requires both `count` and `period`. | | `error_after` | How old the most recent data can be before a freshness check reports an error. Same format as `warn_after`. | | `loaded_at_field` | Column dbt queries to determine the most recent loaded timestamp. Required when adapter metadata is unavailable. | | `loaded_at_query` | A SQL expression that returns the most recent loaded timestamp. Alternative to `loaded_at_field`. Setting both `loaded_at_query` and `loaded_at_field` on the same resource is a parse error. Available in dbt v1.10 and later. | | `filter` | Adds a `WHERE` clause to the freshness query to limit data scanned. Useful for BigQuery partitioned tables or large tables on Snowflake, Databricks, or Spark. Does not affect other uses of the source or model. Does not apply to `loaded_at_query`. | One or both of `warn_after` and `error_after` can be provided. If neither is set, dbt will not check freshness for that resource. Each of `warn_after` and `error_after` requires both `count` and `period`; setting only one issues a parse warning but causes an error when `dbt freshness` runs. #### Source freshness ##### Project file dbt\_project.yml ```yaml sources: <resource-path>: +freshness: warn_after: count: <positive_integer> period: minute | hour | day ``` ##### Property file models/\<filename>.yml ```yaml sources: - name: <source_name> config: freshness: # changed to config in v1.9 warn_after: count: <positive_integer> period: minute | hour | day error_after: count: <positive_integer> period: minute | hour | day filter: <boolean_sql_expression> # changed to config in v1.10 loaded_at_field: <column_name_or_expression> # or use loaded_at_query in v1.10 or higher loaded_at_query: <sql_expression> tables: - name: <table_name> config: # source.table.config.freshness overrides source.config.freshness freshness: warn_after: count: <positive_integer> period: minute | hour | day error_after: count: <positive_integer> period: minute | hour | day filter: <boolean_sql_expression> loaded_at_field: <column_name_or_expression> loaded_at_query: <sql_expression> ``` Freshness blocks are applied hierarchically: * A `freshness` and `loaded_at_field` set on a source apply to all tables in that source. * A `freshness` and `loaded_at_field` set on a source *table* override the source-level values. To exclude a source from freshness calculations, explicitly set `freshness: null`. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: * If `loaded_at_field` is provided, dbt calculates freshness via a select query. * If `loaded_at_field` is *not* provided, dbt calculates freshness via warehouse metadata tables when possible. (Applies to dbt v1.12 and later) Wildcard table identifiers On BigQuery, metadata-based freshness checks are not reliable for sources defined with wildcard table identifiers (for example, `events_*`). To prevent incorrect freshness results, enable the [`bigquery_reject_wildcard_metadata_source_freshness`](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_reject_wildcard_metadata_source_freshness-flag) flag in your `dbt_project.yml`. When enabled, dbt raises an error if metadata-based freshness is used with a wildcard table identifier. To calculate freshness for wildcard tables, configure `loaded_at_field` to use query-based freshness checks instead. ##### Examples ###### Using `loaded_at_field` ```yml sources: - name: jaffle_shop config: # Cast a date field to timestamp loaded_at_field: "completed_date::timestamp" tables: - name: orders config: # Cast a non-UTC timestamp to UTC loaded_at_field: "convert_timezone('Australia/Sydney', 'UTC', created_at_local)" freshness: warn_after: {count: 12, period: hour} ``` ###### Using `loaded_at_query` (Applies to dbt v1.10 and later) ```yaml sources: - name: jaffle_shop tables: - name: orders loaded_at_query: | select max(_sdc_batched_at) from ( select * from {{ this }} where _sdc_batched_at > dateadd(day, -7, current_date) qualify count(*) over (partition by _sdc_batched_at::date) > 2000 ) config: freshness: warn_after: {count: 12, period: hour} ``` ###### Complete example models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw config: freshness: # default freshness for all tables warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at tables: - name: customers # uses the freshness defined above - name: orders config: freshness: # more strict for orders warn_after: {count: 6, period: hour} error_after: {count: 12, period: hour} filter: datediff('day', _etl_loaded_at, current_timestamp) < 2 - name: product_skus config: freshness: null # do not check freshness for this table ``` (Applies to dbt v2.0 and later) When running [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md), the following query will be run against the `orders` table: ##### Compiled SQL ```sql select max(_etl_loaded_at) as max_loaded_at, convert_timezone('UTC', current_timestamp()) as snapshotted_at from raw.jaffle_shop.orders where datediff('day', _etl_loaded_at, current_timestamp) < 2 ``` ##### Jinja SQL ```sql select max({{ loaded_at_field }}) as max_loaded_at, {{ current_timestamp() }} as snapshotted_at from {{ source }} {% if filter %} where {{ filter }} {% endif %} ``` *[Source code](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/src/dbt/include/global_project/macros/adapters/freshness.sql#L5-L16)* (Applies to dbt v2.0 and later) #### Model freshness Use the `freshness` config on a model to: * **Set a freshness threshold**: Set `warn_after` and `error_after` thresholds to declare how stale a model's data can get, then run `dbt freshness` to check each model against its thresholds and report a warning or an error. * **Schedule builds (`build_after`)**: Control how often a model rebuilds when new upstream data is available. Available on dbt platform Enterprise tiers only. `build_after` is part of state-aware orchestration, which has been deprecated and is now [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). ##### Project file dbt\_project.yml ```yml models: <resource-path>: +loaded_at_field: <column_name> # or loaded_at_query +loaded_at_query: <sql_expression> # alternative to loaded_at_field +freshness: warn_after: {count: <positive_integer>, period: minute | hour | day} error_after: {count: <positive_integer>, period: minute | hour | day} ``` ##### Property file models/\<filename>.yml ```yml models: - name: stg_orders config: loaded_at_field: updated_at # or loaded_at_query freshness: warn_after: {count: 24, period: hour} error_after: {count: 48, period: hour} ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( loaded_at_field="updated_at", freshness={ "warn_after": {"count": 24, "period": "hour"}, "error_after": {"count": 48, "period": "hour"} } ) }} ``` Not all materializations support freshness checks the same way. dbt validates your config at parse time and raises an error for invalid combinations. | Materialization | `loaded_at_field` / `loaded_at_query` | Behavior | | ------------------------------------------------------------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `table`, `incremental`, `materialized_view`, `dynamic_table` | Optional | If unset, dbt falls back to adapter metadata (for example, the table's last modified time). | | `view`, `external` | Required | Views don't expose row-level metadata. Set `loaded_at_field` or `loaded_at_query` to measure freshness. An empty string (`loaded_at_field: ""`) is treated the same as unset and raises a parse error. | | `ephemeral` | Not supported | Nothing is materialized to measure. Raises a parse error. | An incomplete freshness rule (for example, `warn_after` with `count` but no `period`) issues a warning at parse time for all materializations; `dbt run` and `dbt build` warn but still succeed. Only `dbt freshness` treats it as an error. Separately, `view` and `external` models require l`oaded_at_field` or `loaded_at_query` — omitting both is a parse error that fails `dbt run`, `dbt build`, and `dbt freshness`, regardless of whether the freshness rule is complete. ##### Cross-project freshness For public models in a [dbt Mesh](https://docs.getdbt.com/docs/mesh/about-mesh.md), dbt stores the freshness config so downstream projects can check upstream model freshness without running the upstream project. For example, `project_a` owns a public model `orders` with freshness configured: ```yaml # project_a: models/orders.yml models: - name: orders access: public config: loaded_at_field: updated_at freshness: warn_after: {count: 24, period: hour} error_after: {count: 48, period: hour} ``` `project_b` depends on `orders`. To check whether `orders` data is fresh, run `dbt freshness` from `project_b` — no need to re-run `project_a`: ```bash # run from project_b dbt freshness --select project_a.orders ``` ##### Examples ###### Using `warn_after` only ```yaml models: - name: stg_orders config: materialized: table freshness: warn_after: {count: 24, period: hour} ``` ###### Using `loaded_at_query` ```yaml models: - name: stg_events config: materialized: table freshness: warn_after: {count: 6, period: hour} error_after: {count: 12, period: hour} loaded_at_query: "select max(_loaded_at) from {{ this }} where _batch_complete = true" ``` ##### Scheduling builds State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). ##### Project file dbt\_project.yml ```yaml models: <resource-path>: +freshness: build_after: # Available only on dbt platform Enterprise tiers count: <positive_integer> period: minute | hour | day updates_on: any | all # optional, default is `any` ``` ##### Property file models/\<filename>.yml ```yml models: - name: stg_orders config: freshness: build_after: # Available only on dbt platform Enterprise tiers count: <positive_integer> period: minute | hour | day updates_on: any | all # optional, default is `any` ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( freshness={ "build_after": { "count": <positive_integer>, "period": "minute" | "hour" | "day", "updates_on": "any" | "all" } } ) }} ``` The `build_after` config applies to state-aware orchestration (SAO), which is now deprecated. `build_after` rebuilds models *only when new source or upstream data is available*. This is useful for models that depend on other models but only need to be updated periodically. `freshness` works alongside dbt job orchestration by helping you determine when models should be rebuilt in a scheduled job. When a job runs, dbt makes sure models run only when needed, which helps avoid overbuilding models unnecessarily. dbt does this by: * Checking if there's new data available for the model * Ensuring enough time has passed since the last build, based on `count` and `period` For sources and upstream models (for mesh), dbt considers data "new" based on custom freshness calculations (if configured). If a source's freshness goes past its warning/error threshold, dbt raises a warning/error during the build. The configuration consists of the following parts: | Configuration | Description | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `build_after` | Available on dbt platform Enterprise tiers only. Config nested under `freshness`. Used to determine whether a model should be rebuilt when new data is present, based on whether the specified count and period have passed since the model was last built. Although dbt checks for new data every time the job runs, `build_after` ensures the model is only rebuilt if enough time has passed and new data is available. | | `count` and `period` | Specify how often dbt should check for new data. For example, `count: 4, period: hour` means dbt will check every 4 hours. Both `count` and `period` are required when configuring `build_after`. | | `updates_on` | Optional. Default is `any`. Determines when upstream data changes should trigger a job build. Use the following values:<br />- `any` (default): The model will build once *any* direct upstream node has new data since the last build. Faster and may increase spend.<br />- `all`: The model will only build when *all* direct upstream nodes have new data since the last build. Less spend and more requirements. | If you're using [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), the `build_after` configs have moved out of the `freshness` block and into the `state` block: | State-aware orchestration | dbt State | | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `freshness.build_after.count` + `freshness.build_after.period` | [`state.lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) | | `freshness.build_after.updates_on` | [`state.require_fresh_data_from`](https://docs.getdbt.com/reference/resource-configs/require-fresh-data-from.md) | For more information, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). ###### Default Default for the `build_after` key is: ```yaml build_after: count: 0 period: minute updates_on: any ``` The default for `updates_on` is `any`. This means that by default, the model will be built every time a scheduled job runs for any amount of new data. ###### Examples The following examples show how to configure models to run less frequently, more frequently, or on a custom frequency. ###### Less frequent You can build a model that runs less frequently (which reduces spend) by configuring the model to only build no more often than every X amount of time, as long as it has new data. ```yaml models: - name: stg_wizards config: freshness: build_after: count: 4 period: hour updates_on: all - name: stg_worlds config: freshness: build_after: count: 4 period: hour updates_on: all ``` When the state-aware orchestration job triggers, dbt checks for two things: * Whether new source data is available on all upstream models * Whether the models `stg_wizards` and `stg_worlds` were built more than 4 hours ago When *both* conditions are met, dbt builds the model. In this case, the `updates_on: all` config is set. If the `raw.wizards` source has new data, but `stg_wizards` and `stg_worlds` were last built 3 hours ago, then nothing would be built. ###### More frequent If you want to build a model that runs more frequently, you can configure the model to build as soon as *any* dependency has new data instead of waiting for all dependencies. ```yaml models: - name: stg_wizards config: freshness: build_after: count: 1 period: hour updates_on: any - name: stg_worlds config: freshness: build_after: count: 1 period: hour updates_on: any ``` When the state-aware orchestration job runs, dbt checks two things: * If new source data is available on at least one upstream model. * If `stg_wizards` or `stg_worlds` wasn't built in the last hour. If *both* conditions are met, dbt rebuilds the model. If neither model has new data, nothing will be built. In this example, because `updates_on: any` is set, even if only the `raw.wizards` source has new data and only `stg_wizards` was built in the last hour (while `stg_worlds` hasn't been updated), dbt will still build the model because it only needs one source update and one eligible (stale) model. ###### Custom frequency You can also use custom logic with `build_after` to set different frequencies for different days, or to skip builds during a specific period (for example, on a weekend). ##### Project file dbt\_project.yml ```yaml +freshness: build_after: # wait at least 48 hours before building again, if Saturday or Sunday # otherwise, wait at least 1 hour before building again count: "{{ 48 if modules.datetime.datetime.today().weekday() in (5, 6) else 1 }}" period: hour updates_on: any ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( freshness={ "build_after": { "count": 48 if modules.datetime.datetime.today().weekday() in (5, 6) else 1, "period": "hour", "updates_on": "any" } } ) }} ``` --- ### Freshness JSON file Available in v2 * **Current schema**: [v0](https://schemas.getdbt.com/dbt/freshness/v0/index.html) * **Produced by:** [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) This file contains freshness results for all sources and models with [freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md) configured. dbt writes it to `target/freshness.json` after every `dbt freshness` invocation. Each entry includes a `resource_type` field that identifies whether a resource is a source or a model. ##### Top-level keys * `metadata`: Invocation metadata with the following fields: * `dbt_schema_version`: Schema URL for this artifact (always `https://schemas.getdbt.com/dbt/freshness/v0.json`). * `dbt_version`: dbt version that produced the file. * `generated_at`: Timestamp when the file was written. * `invocation_id`: Unique identifier for this invocation. * `invocation_started_at`: Timestamp when the invocation started. Omitted if not available. * `env`: Map of `DBT_ENV_CUSTOM_ENV_*` environment variables, if set. * `results`: Array of freshness results. Each entry in `results` is a dictionary with the following keys: * `unique_id`: Unique node identifier, mapping results to `nodes` or `sources` in the [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md). * `resource_type`: Either `model` or `source`. * `max_loaded_at`: Max value of the loaded-at timestamp when queried. * `snapshotted_at`: Timestamp when the check ran. * `max_loaded_at_time_ago_in_s`: Interval between `max_loaded_at` and `snapshotted_at` in seconds. * `status`: Freshness status: `Pass`, `Warn`, or `Error`. If the freshness query fails, dbt logs an error but omits the node from the file rather than writing it with a failed status. * `criteria`: The freshness configuration for this node, including `warn_after`, `error_after`, `filter`, `loaded_at_field`, and `loaded_at_query`. * `thread_id`: Identifier for the thread that ran this check. --- ### full_refresh The `full_refresh` config allows you to control whether a resource will always or never perform a full-refresh. This config overrides the `--full-refresh` command-line flag. ##### Models dbt\_project.yml ```yml models: <resource-path>: +full_refresh: false | true ``` models/\<modelname>.sql ```sql {{ config( full_refresh = false | true ) }} select ... ``` ##### Seeds dbt\_project.yml ```yml seeds: <resource-path>: +full_refresh: false | true ``` #### Description The `full_refresh` config allows you to optionally configure whether a resource will always or never perform a full-refresh. This config is an override for the `--full-refresh` command line flag used when running dbt commands. You can set the `full_refresh` config in the `dbt_project.yml` file or in a resource config. | `full_refresh` value | Behavior | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | If set to `true` | The resource *always* performs a full refresh, regardless of whether you pass the `--full-refresh` flag in the dbt command. | | If set to `false` | The resource *never* performs a full refresh, regardless of whether you pass the `--full-refresh` flag in the dbt command. | | If set to `none` or omitted | The resource follows the behavior of the `--full-refresh` flag. If the flag is used, the resource will perform a full refresh; otherwise, it will not. | Snapshots ignore full refresh Snapshots ignore both the `full_refresh` config and the `--full-refresh` flag. A command such as `dbt build --full-refresh` or `dbt snapshot --full-refresh` that includes a snapshot node runs the snapshot as normal — it won't drop or recreate the snapshot table, so existing snapshot history is preserved. ###### Note * The `--full-refresh` flag also supports a short name, `-f`. * The [`should_full_refresh()`](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/configs.sql) macro has logic encoded. #### Usage ##### Incremental models * [How do I rebuild an incremental model?](https://docs.getdbt.com/docs/build/incremental-models.md#how-do-i-rebuild-an-incremental-model) * [What if the columns of my incremental model change?](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change) ##### Seeds The columns of my seed changed, and now I get an error when running the \`seed\` command, what should I do? If you changed the columns of your seed, you may get a `Database Error`: ##### Snowflake ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 1 seed file, 0 sources 12:12:27 | Concurrency: 8 threads (target='dev_snowflake') 12:12:27 | 12:12:27 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:12:30 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 2.78s] 12:12:31 | 12:12:31 | Finished running 1 seed in 10.05s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) 000904 (42000): SQL compilation error: error line 1 at position 62 invalid identifier 'COUNTRY_NAME' Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` ##### Redshift ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 149 macros, 0 operations, 1 seed file, 0 sources 12:14:46 | Concurrency: 1 threads (target='dev_redshift') 12:14:46 | 12:14:46 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:14:46 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 0.23s] 12:14:46 | 12:14:46 | Finished running 1 seed in 1.75s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) column "country_name" of relation "country_codes" does not exist Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` In this case, you should rerun the command with a `--full-refresh` flag, like so: ```text dbt seed --full-refresh ``` **Why is this the case?** When you typically run dbt seed, dbt truncates the existing table and reinserts the data. This pattern avoids a `drop cascade` command, which may cause downstream objects (that your BI users might be querying!) to get dropped. However, when column names are changed, or new columns are added, these statements will fail as the table structure has changed. The `--full-refresh` flag will force dbt to `drop cascade` the existing table before rebuilding it. #### Recommendation * Set `full_refresh: false` for models of especially large datasets, which you would *never* want dbt to fully drop and recreate. * You cannot override an existing `full_refresh` config. To change its behavior in certain circumstances, remove the config logic or update it using variables so the behavior can be overridden when needed. For example, if you have an incremental model with the following config: ```sql {{ config( materialized = 'incremental', full_refresh = var("force_full_refresh", false) ) }} ``` Then override the `full_refresh` config to `true` using the [`--vars` flag](https://docs.getdbt.com/docs/build/project-variables.md#defining-variables-on-the-command-line): `dbt run --vars '{"force_full_refresh": true}'`. #### Reference docs * [on\_configuration\_change](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) --- ### Function configurations Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### Available configurations ##### Function-specific configurations Resource-specific configurations are applicable to only one dbt resource type rather than multiple resource types. You can define these settings in the project file (`dbt_project.yml`), a property file (`models/properties.yml` for models, similarly for other resources), or within the resource’s file using the `{{ config() }}` macro.<br /> The following resource-specific configurations are only available to Functions: ##### Project YAML file dbt\_project.yml ```yml functions: <resource-path>: # Function-specific configs are defined in the properties YAML file # See functions/schema.yml examples below ``` ##### Properties YAML file functions/schema.yml ```yaml functions: - name: [<function-name>] config: type: scalar # optional, defaults to scalar. Eventually will include aggregate | table volatility: deterministic | stable | non-deterministic # optional runtime_version: <string> # required for Python UDFs entry_point: <string> # required for Python UDFs packages: [<string>] # optional, Python UDFs only snowflake: quote_args: true | false # optional, JavaScript UDFs on Snowflake only, available in 1.12+ # standard configs that apply to functions database: <string> schema: <string> alias: <string> tags: <string> | [<string>] meta: {<dictionary>} ``` ##### General configurations General configurations provide broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project file, property files, or within resource-specific files. Database, schema, and alias configuration Functions support `database`, `schema`, and `alias` configurations just like models. These determine where the function is created in your warehouse. The function will use the standard dbt configuration precedence (specific config > project config > target profile defaults). ##### Project YAML file dbt\_project.yml ```yaml functions: <resource-path>: +enabled: true | false +tags: <string> | [<string>] +database: <string> +schema: <string> +alias: <string> +meta: {<dictionary>} ``` ##### Properties YAML file functions/schema.yml ```yaml functions: - name: [<function-name>] config: enabled: true | false tags: <string> | [<string>] database: <string> schema: <string> alias: <string> meta: {<dictionary>} ``` #### Configuring functions Functions are configured in YAML files, either in `dbt_project.yml` or within an individual function's YAML properties file. The function body is defined in a SQL file in the `functions/` directory. Function configurations, like model configurations, are applied hierarchically. For more info, refer to [config inheritance](https://docs.getdbt.com/reference/define-configs.md#config-inheritance). Functions respect the same name-generation macros as models: [`generate_database_name`](https://docs.getdbt.com/docs/build/custom-databases.md), [`generate_schema_name`](https://docs.getdbt.com/docs/build/custom-schemas.md#how-does-dbt-generate-a-models-schema-name), and [`generate_alias_name`](https://docs.getdbt.com/docs/build/custom-aliases.md). ##### Examples ###### Apply the `schema` configuration to all functions To apply a configuration to all functions, including those in any installed [packages](https://docs.getdbt.com/docs/build/packages.md), nest the configuration directly under the `functions` key: dbt\_project.yml ```yml functions: +schema: udf_schema ``` ###### Apply the `schema` configuration to all functions in your project To apply a configuration to all functions in your project only (i.e. *excluding* any functions in installed packages), provide your [project name](https://docs.getdbt.com/reference/project-configs/name.md) as part of the resource path. For a project named `jaffle_shop`: dbt\_project.yml ```yml functions: jaffle_shop: +schema: udf_schema ``` Similarly, you can use the name of an installed package to configure functions in that package. ###### Apply the `schema` configuration to one function only To apply a configuration to one function only in a properties file, specify the configuration in the function's `config` block: functions/schema.yml ```yml functions: - name: is_positive_int config: schema: udf_schema ``` To apply a configuration to one function only in `dbt_project.yml`, provide the full resource path (including the project name and subdirectories). For a project named `jaffle_shop`, with a function file at `functions/is_positive_int.sql`: dbt\_project.yml ```yml functions: jaffle_shop: is_positive_int: +schema: udf_schema ``` #### Example function configuration The following example shows how to configure functions in a project named `jaffle_shop` that has two function files: * `functions/is_positive_int.sql` * `functions/marketing/clean_url.sql` dbt\_project.yml ```yml name: jaffle_shop ... functions: jaffle_shop: +enabled: true +schema: udf_schema # This configures functions/is_positive_int.sql is_positive_int: +tags: ['validation'] marketing: +schema: marketing_udfs # this will take precedence ``` functions/schema.yml ```yml functions: - name: is_positive_int description: Determines if a string represents a positive integer config: type: scalar volatility: deterministic database: analytics schema: udf_schema arguments: - name: a_string data_type: string description: The string to check returns: data_type: boolean description: Returns true if the string represents a positive integer ``` --- ### Function properties Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Function properties can be declared in `.yml` files under a `functions` key. We recommend that you put them in the `functions/` directory. You can name these files `schema.yml` or `whatever_you_want.yml`, and nest them in subfolders within that directory. functions/\<filename>.yml ```yml functions: - name: <string> # required description: <markdown_string> # optional config: # optional <function_config>: <config_value> type: scalar | aggregate # optional, defaults to scalar. volatility: deterministic | stable | non-deterministic # optional runtime_version: <string> # required for Python UDFs entry_point: <string> # required for Python UDFs packages: [<string>] # optional, Python UDFs only docs: show: true | false node_color: <color_id> # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") arguments: # optional - name: <string> # required if arguments is specified data_type: <string> # required if arguments is specified, warehouse-specific description: <markdown_string> # optional default_value: <string | boolean | integer> # optional, available in Snowflake and Postgres - name: ... # declare additional arguments returns: # required data_type: <string> # required, warehouse-specific description: <markdown_string> # optional overloads: # optional, SQL UDFs (Snowflake and Postgres) and Python UDFs (Snowflake), available in v1.12+ - defined_in: <string> # required, name of the SQL or Python file containing this overload's body arguments: # optional - name: <string> # required if arguments is specified data_type: <string> # required if arguments is specified, warehouse-specific description: <markdown_string> # optional default_value: <string | boolean | integer> # optional, available in Snowflake and Postgres - name: ... # declare additional arguments returns: # optional, inherits from root function if omitted data_type: <string> # required if returns is specified, warehouse-specific description: <markdown_string> # optional - defined_in: ... # declare additional overloads - name: ... # declare properties of additional functions ``` #### Example functions/schema.yml ```yml functions: - name: is_positive_int description: Determines if a string represents a positive (+) integer config: type: scalar volatility: deterministic database: analytics schema: udf_schema arguments: - name: a_string data_type: string description: The string that I want to check if it's representing a positive integer (like "10") returns: data_type: boolean description: Returns true if the input string represents a positive integer, false otherwise overloads: - defined_in: is_positive_int_numeric arguments: - name: a_num data_type: numeric description: The number that I want to check if it's a positive integer returns: data_type: boolean ``` --- ### function-paths dbt\_project.yml ```yml function-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [user-defined functions (UDFs)](https://docs.getdbt.com/docs/build/udfs.md) are located. #### Default By default, dbt will search for functions in the `functions` directory, for example, `function-paths: ["functions"]` #### Examples Use a subdirectory named `udfs` instead of `functions`: dbt\_project.yml ```yml function-paths: ["udfs"] ``` Use multiple directories to organize your functions: dbt\_project.yml ```yml function-paths: ["functions", "custom_udfs"] ``` --- ### Generic test arguments property This flag has reached maturity The default value is now `true`. To preserve the previous behavior, set this flag to `false` in your `dbt_project.yml`. | require\_generic\_test\_arguments\_property | dbt **v1 Latest** | dbt v1 | | ------------------------------------------- | ----------------- | ------ | | Introduced | 2025.07 | 1.10.5 | | Matured (default → `true`) | 2025.08 | 1.10.8 | | Removed | — | — | dbt supports parsing key-value arguments that are inputs to generic tests when specified under the `arguments` property. In the past, dbt didn't support a way to clearly disambiguate between properties that were inputs to generic tests and framework configurations, and only accepted arguments as top-level properties. If you are on an older dbt v1 version where the flag is `false`, dbt will recognize the `arguments` property but raise the `ArgumentsPropertyInGenericTestDeprecation` warning. Here's an example using the new `arguments` property: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - dbt_utils.expression_is_true: arguments: expression: "order_items_subtotal = subtotal" ``` Here's an example using the alternative `test_name` format: model.yml ```yaml models: - name: my_model_with_generic_test data_tests: - name: arbitrary_name test_name: dbt_utils.expression_is_true arguments: expression: "order_items_subtotal = subtotal" config: where: "1=1" ``` With this flag enabled, dbt will: * Parse any key-value pairs under `arguments` in generic tests as inputs to the generic test macro. * Raise a `MissingArgumentsPropertyInGenericTestDeprecation` warning if additional non-config arguments are specified outside of the `arguments` property. To preserve the previous behavior: dbt\_project.yml ```yaml flags: require_generic_test_arguments_property: false ``` --- ### grants You can manage access to the datasets you're producing with dbt by using grants. To implement these permissions, define grants as resource configs on each model, seed, or snapshot. Define the default grants that apply to the entire project in your `dbt_project.yml`, and define model-specific grants within each model's SQL or YAML property file. The grant resource configs enable you to apply permissions at build time to a specific set of recipients and model, seed, or snapshot. When your model, seed, or snapshot finishes building, dbt ensures that the grants on its view or table match exactly the grants you have configured. dbt aims to use the most efficient approach when updating grants, which varies based on the adapter you're using, and whether dbt is replacing or updating an object that already exists. You can always check the debug logs for the full set of grant and revoke statements that dbt runs. You should define grants as resource configs whenever possible, but you might occasionally need to write grants statements manually and run them using [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md). For example, hooks may be appropriate if you want to: * Apply grants on other database objects besides views and tables. * Create more granular row- and column-level access, use masking policies, or apply future grants. * Take advantage of more advanced permission capabilities offered by your data platform, for which dbt does not offer out-of-the-box support using resource configuration. * Apply grants in a more complex or custom manner, beyond what the built-in grants capability can provide. For more information on hooks, see [Hooks & operations](https://docs.getdbt.com/docs/build/hooks-operations.md). #### Definition You can use the `grants` field to set permissions or grants for a resource. When you `run` a model, `seed` data, or `snapshot` a dataset, dbt will run `grant` and/or `revoke` statements to ensure that the permissions on the database object match the `grants` you have configured on the resource. Like all configurations, `grants` will be included in dbt project metadata, including [the manifest artifact](https://docs.getdbt.com/reference/artifacts/manifest-json.md). ##### Common syntax Grants have two key components: * **Privilege:** A right to perform a specific action or set of actions on an object in the database, such as selecting data from a table. * **Grantees:** One or more recipients of granted privileges. Some platforms also call these "principals." For example, a grantee could be a user, a group of users, a role held by one or more users (Snowflake), or a service account (BigQuery/GCP). #### Configuring grants You can configure `grants` in `dbt_project.yml` to apply grants to many resources at once—all models in your project, a package, or a subfolder—and you can also configure `grants` one-by-one for specific resources, in YAML `config:` blocks or right within their `.sql` files. ##### Models models/schema.yml ```yml models: - name: specific_model config: grants: select: ['reporter', 'bi'] ``` The `grants` config can also be defined: * under the `models` config in the project file (`dbt_project.yml`) * in a `config()` Jinja macro within a model's SQL file See [configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for details. ##### Seeds seeds/schema.yml ```yml seeds: - name: seed_name config: grants: select: ['reporter', 'bi'] ``` The `grants` config can also be defined under the `seeds` config in the project file (`dbt_project.yml`). See [configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for details. ##### Snapshots snapshots/schema.yml ```yml snapshots: - name: snapshot_name config: grants: select: ['reporter', 'bi'] ``` The `grants` config can be defined: * Under the `snapshots` config in the property file (`snapshots/schema.yml`) * Under the `snapshots` config in the project file (`dbt_project.yml`) * In a snapshot's SQL file `config()` Jinja macro See [configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for details. ##### Grant config inheritance When you set `grants` for the same model in multiple places, such as in `dbt_project.yml` and in a more-specific `.sql` or `.yml` file, dbt's default behavior replaces the less-specific set of grantees with the more-specific set of grantees. This "merge and clobber" behavior updates each privilege when dbt parses your project. For example: dbt\_project.yml ```yml models: +grants: # In this case the + is not optional, you must include it for your project to parse. select: ['user_a', 'user_b'] ``` models/specific\_model.sql ```sql {{ config(grants = {'select': ['user_c']}) }} ``` As a result of this configuration, `specific_model` will be configured to grant the `select` privilege to `user_c` *only*. After you run `specific_model`, that is the only granted privilege you would see in the database, and the only `grant` statement you would find in dbt's logs. Let's say we wanted to *add* `user_c` to the existing list of grantees receiving the `select` privilege on `specific_model`, rather than *replacing* that list entirely. To accomplish that, we can use the `+` ("addition") symbol, prefixing the name of the privilege: models/specific\_model.sql ```sql {{ config(grants = {'+select': ['user_c']}) }} ``` Now, the model will grant select to `user_a`, `user_b`, AND `user_c`! **Notes:** * This will only take effect for privileges which include the `+` prefix. Each privilege controls that behavior separately. If we were granting other privileges, in addition to `select`, and those privilege names lacked the `+` prefix, they would continue to "clobber" rather than "add" new grantees. * This use of `+`, controlling clobber vs. add merge behavior, is distinct from the use of `+` in `dbt_project.yml` (shown in the example above) for defining configs with dictionary values. For more information, see [the plus prefix](https://docs.getdbt.com/reference/resource-configs/plus-prefix.md). * `grants` is the first config to support a `+` prefix for controlling config merge behavior. Currently, it's the only one. If it proves useful, we may extend this capability to new and existing configs in the future. ##### Conditional grants Like any other config, you can use Jinja to vary the grants in different contexts. For example, you might grant different permissions in prod than dev: dbt\_project.yml ```yml models: +grants: select: "{{ ['user_a', 'user_b'] if target.name == 'prod' else ['user_c'] }}" ``` #### Revoking grants dbt only modifies grants on a node (including revocation) when a `grants` configuration is attached to that node. For example, imagine you had originally specified the following grants in `dbt_project.yml`: dbt\_project.yml ```yml models: +grants: select: ['user_a', 'user_b'] ``` If you delete the entire `+grants` section, dbt assumes you no longer want it to manage grants and doesn't change anything. To have dbt revoke all existing grants from a node, provide an empty list of grantees. ##### Revoke from one user dbt\_project.yml ```yml models: +grants: select: ['user_b'] ``` ##### Revoke from all users dbt\_project.yml ```yml models: +grants: select: [] ``` ##### Stop dbt from managing grants dbt\_project.yml ```yml models: # this section intentionally left blank ``` #### General examples You can grant each permission to a single grantee, or a set of multiple grantees. In this example, we're granting `select` on this model to just `bi_user`, so that it can be queried in our Business Intelligence (BI) tool. models/table\_model.sql ```sql {{ config(materialized = 'table', grants = { 'select': 'bi_user' }) }} ``` When dbt runs this model for the first time, it will create the table, and then run code like: ```sql grant select on schema_name.table_model to bi_user; ``` In this case, we're creating an incremental model, and granting the `select` privilege to two recipients: `bi_user` and `reporter`. models/incremental\_model.sql ```sql {{ config(materialized = 'incremental', grants = { 'select': ['bi_user', 'reporter'] }) }} ``` When dbt runs this model for the first time, it will create the table, and then run code like: ```sql grant select on schema_name.incremental_model to bi_user, reporter; ``` In subsequent runs, dbt will use database-specific SQL to show the grants already on `incremental_model`, and then determine if any `revoke` or `grant` statements are needed. #### Database-specific requirements and notes While we try to standardize the terms we use to describe different features, you will always find nuances in different databases. This section outlines some of those database-specific requirements and notes. In our examples above and below, you will find us referring to a privilege named `select`, and a grantee named `another_user`. Many databases use these or similar terms. Be aware that your database may require different syntax for privileges and grantees; you must configure `grants` in dbt with the appropriate names for both. ##### BigQuery On BigQuery, "privileges" are called "roles," and they take the form `roles/service.roleName`. For instance, instead of granting `select` on a model, you would grant `roles/bigquery.dataViewer`. Grantees can be users, groups, service accounts, domains—and each needs to be clearly demarcated as such with a prefix. For instance, to grant access on a model to `someone@yourcompany.com`, you need to specify them as `user:someone@yourcompany.com`. We encourage you to read Google's documentation for more context: * [Understanding GCP roles](https://cloud.google.com/iam/docs/understanding-roles) * [How to format grantees](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-control-language#user_list) Note The `grants` config and the `grant_access_to` config are distinct. * **`grant_access_to`:** Enables you to set up authorized views. When configured, dbt provides an authorized view access to show partial information from other datasets, without providing end users with full access to those underlying datasets. For more information, see ["BigQuery configurations: Authorized views"](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#authorized-views) * **`grants`:** Provides specific permissions to users, groups, or service accounts for managing access to datasets you're producing with dbt. For more information, see ["Resource configs: grants"](https://docs.getdbt.com/reference/resource-configs/grants.md) You can use the two features together: "authorize" a view model with the `grants_access_to` configuration, and then add `grants` to that view model to share its query results (and *only* its query results) with other users, groups, or service accounts. ##### BigQuery examples Granting permission using SQL and BigQuery: ```sql {{ config(grants = {'roles/bigquery.dataViewer': ['user:someone@yourcompany.com']}) }} ``` Granting permission in a model schema using BigQuery: models/schema.yml ```yml models: - name: specific_model config: grants: roles/bigquery.dataViewer: ['user:someone@yourcompany.com'] ``` ##### Databricks * OSS Apache Spark / Delta Lake do not support `grants`. * Databricks automatically enables `grants` on SQL endpoints. For interactive clusters, admins should enable grant functionality using these two setup steps in the Databricks documentation: * [Enable table access control for your workspace](https://docs.databricks.com/administration-guide/access-control/table-acl.html) * [Enable table access control for a cluster](https://docs.databricks.com/security/access-control/table-acls/table-acl.html) * In order to grant `READ_METADATA` or `USAGE`, use [post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) ##### Redshift * Redshift supports granting to users, [groups](https://docs.aws.amazon.com/redshift/latest/dg/r_Groups.html), and [roles](https://docs.aws.amazon.com/redshift/latest/dg/r_roles-managing.html). Use the `group:` or `role:` prefix in grantee names to grant to groups or roles. Unprefixed names are treated as users. ```yaml models: +grants: select: ["user1", "user:user2", "group:analysts", "role:reporter"] ``` ##### Snowflake * dbt accounts for the [`copy_grants` configuration](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#copying-grants) when calculating which grants need to be added or removed. * Granting to / revoking from is only fully supported for Snowflake roles (not [database roles](https://docs.snowflake.com/user-guide/security-access-control-overview#types-of-roles)). --- ### Graph operators ##### The "plus" operator The `+` operator expands your selection to include ancestors (upstream dependencies) or descendants (downstream dependencies) of a resource. This operator works for individual models, tags, and other resources. * Placed after a model/resource — Includes the resource itself and all its descendants (downstream dependencies). * Placed before a model/resource — Includes the resource itself and all its ancestors (upstream dependencies). * Placed on both sides of a model/resource — Includes the resource itself, all its ancestors, and all its descendants. ```bash dbt run --select "my_model+" # select my_model and all descendants dbt run --select "+my_model" # select my_model and all ancestors dbt run --select "+my_model+" # select my_model, and all of its ancestors and descendants ``` You can use it with selectors for a more specific scope in your commands. You can also combine it with [`--exclude`](https://docs.getdbt.com/reference/node-selection/exclude.md) flag for even more finer control over what gets included in your command. ##### The "n-plus" operator You can adjust the behavior of the `+` operator by quantifying the number of edges to step through. ```bash dbt run --select "my_model+1" # select my_model and its first-degree descendants dbt run --select "2+my_model" # select my_model, its first-degree ancestors ("parents"), and its second-degree ancestors ("grandparents") dbt run --select "3+my_model+4" # select my_model, its ancestors up to the 3rd degree, and its descendants down to the 4th degree ``` ##### The "at" operator The `@` operator is similar to `+`, but will also include *all ancestors of all descendants of the selected model*. This is useful in continuous integration environments where you want to build a model and all of its descendants, but the *ancestors* of those descendants might not exist in the schema yet. The `@` operator (which can only be placed at the front of the model name) will select as many degrees of ancestors ("parents," "grandparents," and so on) as is needed to successfully build all descendants of the specified model. The selector `@snowplow_web_page_context` will build all three models shown in the diagram below. [![@snowplow\_web\_page\_context will select all of the models shown here](/img/docs/running-a-dbt-project/command-line-interface/1643e30-Screen_Shot_2019-03-11_at_7.18.20_PM.png?v=2 "@snowplow_web_page_context will select all of the models shown here")](#)@snowplow\_web\_page\_context will select all of the models shown here ```bash dbt run --select "@my_model" # select my_model, its descendants, and the ancestors of its descendants ``` --- ### Greenplum configurations #### Performance Optimizations Tables in Greenplum have powerful optimization configurations to improve query performance: * distribution * column orientation * compression * `appendonly` toggle * partitions Supplying these values as model-level configurations apply the corresponding settings in the generated `CREATE TABLE`(except partitions). Note that these settings will have no effect for models set to `view`. ##### Distribution In Greenplum, you can choose a [distribution key](https://gpdb.docs.pivotal.io/6-4/admin_guide/distribution.html), that will be used to sort data by segments. Joining on the partition will become more performant after specifying distribution. By default dbt-greenplum distributes data `RANDOMLY`. To implement a distribution key you need to specify the `distributed_by` parameter in model's config: ```sql {{ config( ... distributed_by='<field_name>' ... ) }} select ... ``` Also you can choose `DISTRIBUTED REPLICATED` option: ```sql {{ config( ... distributed_replicated=true ... ) }} select ... ``` ##### Column orientation Greenplum supports two type of [orientation](https://gpdb.docs.pivotal.io/6-6/admin_guide/ddl/ddl-storage.html#topic39) row and column: ```sql {{ config( ... orientation='column' ... ) }} select ... ``` ##### Compression Compression allows reducing read-write time. Greenplum suggest several [algorithms](https://gpdb.docs.pivotal.io/6-6/admin_guide/ddl/ddl-storage.html#topic40) algorithms to compress append-optimized tables: * RLE\_TYPE(only for column oriented table) * ZLIB * ZSTD * QUICKLZ ```sql {{ config( ... appendonly='true', compresstype='ZLIB', compresslevel=3, blocksize=32768 ... ) }} select ... ``` As you can see, you can also specify `compresslevel` and `blocksize`. ##### Partition Greenplum does not support partitions with `create table as` [construction](https://gpdb.docs.pivotal.io/6-9/ref_guide/sql_commands/CREATE_TABLE_AS.html), so you need to build model in two steps 1. create table schema 2. insert data To implement partitions into your dbt-model you need to specify the following config parameters: * `fields_string` - definition of columns name, type and constraints * `raw_partition` - partition specification ```sql {% set fields_string %} some_filed int4 null, date_field timestamp NULL {% endset %} {% set raw_partition %} PARTITION BY RANGE (date_field) ( START ('2021-01-01'::timestamp) INCLUSIVE END ('2023-01-01'::timestamp) EXCLUSIVE EVERY (INTERVAL '1 day'), DEFAULT PARTITION default_part ); {% endset %} {{ config( ... fields_string=fields_string, raw_partition=raw_partition, ... ) }} select * ``` --- ### group ##### Models dbt\_project.yml ```yml models: <resource-path>: +group: GROUP_NAME ``` models/schema.yml ```yml models: - name: MODEL_NAME config: group: GROUP # changed to config in v1.10 ``` models/\<modelname>.sql ```sql {{ config( group='GROUP_NAME' ) }} select ... ``` ##### Seeds dbt\_project.yml ```yml models: <resource-path>: +group: GROUP_NAME ``` seeds/properties.yml ```yml seeds: - name: [SEED_NAME] config: group: GROUP_NAME # changed to config in v1.10 ``` ##### Snapshots dbt\_project.yml ```yml snapshots: <resource-path>: +group: GROUP_NAME ``` (Applies to dbt v1.9 and later) snapshots/properties.yml ```yaml snapshots: - name: snapshot_name config: group: GROUP_NAME ``` snapshots/\<filename>.sql ```sql {% snapshot snapshot_name %} {{ config( group='GROUP_NAME' ) }} select ... {% endsnapshot %} ``` ##### Tests dbt\_project.yml ```yml data_tests: <resource-path>: +group: GROUP_NAME ``` tests/properties.yml ```yml <resource_type>: - name: <resource_name> data_tests: - <test_name>: config: group: GROUP_NAME ``` tests/\<filename>.sql ```sql {% test <testname>() %} {{ config( group='GROUP_NAME' ) }} select ... {% endtest %} ``` tests/\<filename>.sql ```sql {{ config( group='GROUP_NAME' ) }} ``` ##### Analyses analyses/\<filename>.yml ```yml analyses: - name: ANALYSIS_NAME config: group: GROUP_NAME # changed to config in v1.10 ``` ##### Metrics dbt\_project.yml ```yaml metrics: <resource-path>: +group: GROUP_NAME ``` models/metrics.yml ```yaml metrics: - name: [METRIC_NAME] config: group: GROUP_NAME ``` ##### Semantic models dbt\_project.yml ```yaml semantic-models: <resource-path>: +group: GROUP_NAME ``` (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: model_name semantic_model: enabled: true group: group_name ``` ##### Saved queries dbt\_project.yml ```yaml saved-queries: <resource-path>: +group: GROUP_NAME ``` models/semantic\_models.yml ```yaml saved_queries: - name: SAVED_QUERY_NAME config: group: GROUP_NAME ``` Note that for backwards compatibility, `group` is supported as a top-level key, but without the capabilities of config inheritance. #### Definition An optional configuration for assigning a group to a resource. When a resource is grouped, dbt will allow it to reference private models within the same group. For more details on reference access between resources in groups, check out [model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups). #### Examples ##### Prevent a 'marketing' group model from referencing a private 'finance' group model This is useful if you want to prevent other groups from building on top of models that are rapidly changing, experimental, or otherwise internal to a group or team. models/schema.yml ```yml models: - name: finance_model config: group: finance # changed to config in v1.10 access: private # changed to config in v1.10 - name: marketing_model config: group: marketing # changed to config in v1.10 ``` models/marketing\_model.sql ```sql select * from {{ ref('finance_model') }} ``` ```shell $ dbt run -s marketing_model ... dbt.exceptions.DbtReferenceError: Parsing Error Node model.jaffle_shop.marketing_model attempted to reference node model.jaffle_shop.finance_model, which is not allowed because the referenced node is private to the finance group. ``` #### Related docs * [Model Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) * [Defining groups](https://docs.getdbt.com/docs/build/groups.md) --- ### hard_deletes Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). snapshots/schema.yml ```yaml snapshots: - name: <snapshot_name> config: hard_deletes: 'ignore' | 'invalidate' | 'new_record' ``` dbt\_project.yml ```yml snapshots: <resource-path>: +hard_deletes: "ignore" | "invalidate" | "new_record" ``` snapshots/\<filename>.sql ```sql {{ config( unique_key='id', strategy='timestamp', updated_at='updated_at', hard_deletes='ignore' | 'invalidate' | 'new_record' ) }} ``` #### Description The `hard_deletes` config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table. You can use `hard_deletes` with dbt-postgres, dbt-bigquery, dbt-snowflake, and dbt-redshift adapters.  When to use the hard\_deletes and invalidate\_hard\_deletes config? **Use `invalidate_hard_deletes` (v1.8 and earlier) if:** * Gaps in the snapshot history (missing records for deleted rows) are acceptable. * You want to invalidate deleted rows by setting their `dbt_valid_to` timestamp to the current time (implicit delete). * You are working with smaller datasets where tracking deletions as a separate state is unnecessary. **Use `hard_deletes: new_record` (v1.9 and higher) if:** * You want to maintain continuous snapshot history without gaps. * You want to explicitly track deletions by adding new rows with a `dbt_is_deleted` column (explicit delete). * You are working with larger datasets where explicitly tracking deleted records improves data lineage clarity. warning If you're updating an existing snapshot to use the `hard_deletes` config, dbt *will not* handle migrations automatically. We recommend either only using these settings for net-new snapshots, or [arranging an update](https://docs.getdbt.com/reference/snapshot-configs.md#snapshot-configuration-migration) of pre-existing tables before enabling this setting. #### Default By default, if you don’t specify `hard_deletes`, it'll automatically default to `ignore`. Deleted rows will not be tracked and their `dbt_valid_to` column remains `NULL`. The `hard_deletes` config has three methods: | Methods | Description | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ignore` (default) | No action for deleted records. | | `invalidate` | Behaves the same as the existing `invalidate_hard_deletes=true`, where deleted records are invalidated by setting `dbt_valid_to` to current time. This method replaces the `invalidate_hard_deletes` config to give you more control on how to handle deleted rows from the source. | | `new_record` | Tracks deleted records as new rows using the `dbt_is_deleted` meta field when records are deleted. | #### Considerations * **Backward compatibility**: The `invalidate_hard_deletes` config is still supported for existing snapshots but can't be used alongside `hard_deletes`. * **New snapshots**: For new snapshots, we recommend using `hard_deletes` instead of `invalidate_hard_deletes`. * **Migration**: If you switch an existing snapshot to use `hard_deletes` without migrating your data, you may encounter inconsistent or incorrect results, such as a mix of old and new data formats. #### Example snapshots/schema.yml ```yaml snapshots: - name: my_snapshot config: hard_deletes: new_record # options are: 'ignore', 'invalidate', or 'new_record' strategy: timestamp updated_at: updated_at columns: - name: dbt_valid_from description: Timestamp when the record became valid. - name: dbt_valid_to description: Timestamp when the record stopped being valid. - name: dbt_is_deleted description: Indicates whether the record was deleted. ``` The resulting snapshot table contains the `hard_deletes: new_record` configuration. If a record is deleted and later restored, the resulting snapshot table might look like this: | id | dbt\_scd\_id | Status | dbt\_updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_is\_deleted | | -- | -------------------- | ------- | ---------------- | ---------------- | -------------- | ---------------- | | 1 | 60a1f1dbdf899a4dd... | pending | 2024-10-02 ... | 2024-05-19... | 2024-05-20 ... | False | | 1 | b1885d098f8bcff51... | pending | 2024-10-02 ... | 2024-05-20 ... | 2024-06-03 ... | True | | 1 | b1885d098f8bcff53... | shipped | 2024-10-02 ... | 2024-06-03 ... | | False | | 2 | b1885d098f8bcff55... | active | 2024-10-02 ... | 2024-05-19 ... | | False | In this example, the `dbt_is_deleted` column is set to `True` when the record is deleted. When the record is restored, the `dbt_is_deleted` column is set to `False`. --- ### IBM Db2 configurations #### Instance requirements To use IBM Db2 with the `ibm-dbt-db2` adapter, ensure the instance has an attached database that supports creating, renaming, altering, and dropping objects such as tables and views. The user connecting to the instance via the `ibm-dbt-db2` adapter must have the necessary permissions for the target database and schema. For more details, please visit the official [IBM Db2 documentation](https://www.ibm.com/docs/en/db2) #### Supported Db2 platforms The `ibm-dbt-db2` adapter supports the following IBM Db2 platforms: * **Db2 for Linux, Unix, and Windows (LUW)** - Version 9.7 and higher * **Db2 for z/OS** - Version 11 and higher * **Db2 for iSeries** - Compatible versions #### Seeds and data types The `ibm-dbt-db2` adapter offers comprehensive support for all Db2 data types in seed files. To leverage this functionality, you must explicitly define the data types for each column in your seed configuration. You can configure column data types either in the `dbt_project.yml` file or in property files, as supported by dbt. For more details on seed configuration and best practices, refer to the [dbt seed configuration documentation](https://docs.getdbt.com/reference/seed-configs.md). ##### Example seed configuration In your `dbt_project.yml`: ```yaml seeds: my_project: raw_customers: +column_types: id: INTEGER name: VARCHAR(100) email: VARCHAR(255) created_at: TIMESTAMP is_active: BOOLEAN ``` ##### Supported Db2 data types | Db2 Data Type | Description | Example Usage | | ------------- | ------------------------------ | ------------------------------- | | INTEGER | 32-bit integer | `id: INTEGER` | | BIGINT | 64-bit integer | `large_id: BIGINT` | | SMALLINT | 16-bit integer | `status_code: SMALLINT` | | DECIMAL(p,s) | Fixed-point decimal | `price: DECIMAL(10,2)` | | FLOAT | Floating-point number | `rate: FLOAT` | | DOUBLE | Double precision float | `measurement: DOUBLE` | | VARCHAR(n) | Variable-length string | `name: VARCHAR(100)` | | CHAR(n) | Fixed-length string | `code: CHAR(10)` | | CLOB | Character large object | `description: CLOB` | | BLOB | Binary large object | `image_data: BLOB` | | DATE | Date value | `birth_date: DATE` | | TIME | Time value | `start_time: TIME` | | TIMESTAMP | Date and time | `created_at: TIMESTAMP` | | BOOLEAN | Boolean value | `is_active: BOOLEAN` | | XML | XML document | `config: XML` | | GRAPHIC(n) | Fixed-length graphic string | `unicode_text: GRAPHIC(50)` | | VARGRAPHIC(n) | Variable-length graphic string | `unicode_name: VARGRAPHIC(100)` | | DBCLOB | Double-byte CLOB | `large_unicode: DBCLOB` | #### Materialization strategies The `ibm-dbt-db2` adapter supports all standard dbt materializations: ##### Table materialization Creates a physical table in the database. This is the default materialization if not specified. ```sql {{ config(materialized='table') }} SELECT * FROM source_table ``` ##### View materialization Creates a database view. Views are virtual tables that don't store data physically. ```sql {{ config(materialized='view') }} SELECT * FROM source_table ``` ##### Incremental materialization Builds tables incrementally, processing only new or changed records. The `ibm-dbt-db2` adapter supports two incremental strategies: ###### Merge strategy (default) Uses Db2's MERGE statement for efficient upserts: ```sql {{ config( materialized='incremental', unique_key='id', incremental_strategy='merge' ) }} SELECT * FROM source_table {% if is_incremental() %} WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }}) {% endif %} ``` ###### Delete+insert strategy Deletes matching records then inserts new ones: ```sql {{ config( materialized='incremental', unique_key='id', incremental_strategy='delete+insert' ) }} SELECT * FROM source_table {% if is_incremental() %} WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }}) {% endif %} ``` ##### Ephemeral materialization Creates a Common Table Expression (CTE) that exists only for the duration of a single dbt run: ```sql {{ config(materialized='ephemeral') }} SELECT * FROM source_table ``` #### Snapshots The `ibm-dbt-db2` adapter supports dbt snapshots for tracking slowly changing dimensions (SCD Type 2). Snapshots capture the state of your data at different points in time. Example snapshot: ```sql {% snapshot customers_snapshot %} {{ config( target_schema='snapshots', unique_key='id', strategy='timestamp', updated_at='updated_at' ) }} SELECT * FROM {{ source('raw', 'customers') }} {% endsnapshot %} ``` #### Constraints The `ibm-dbt-db2` adapter supports defining constraints in your models, but enforcement varies: | Constraint Type | Support Level | Notes | | --------------- | --------------- | --------------------------------------- | | NOT NULL | ✅ Enforced | Fully supported and enforced by Db2 | | CHECK | ⚠️ Not Enforced | Defined but not enforced in dbt context | | UNIQUE | ⚠️ Not Enforced | Defined but not enforced in dbt context | | PRIMARY KEY | ⚠️ Not Enforced | Defined but not enforced in dbt context | | FOREIGN KEY | ⚠️ Not Enforced | Defined but not enforced in dbt context | Example with constraints: ```sql {{ config( materialized='table' ) }} SELECT id, name, email, created_at FROM source_table ``` Define constraints in your schema YAML: ```yaml models: - name: customers columns: - name: id constraints: - type: not_null - type: primary_key - name: email constraints: - type: not_null ``` #### Performance optimization ##### Indexing While dbt doesn't directly manage indexes, you can create them using post-hooks: ```sql {{ config( materialized='table', post_hook=[ "CREATE INDEX idx_customer_email ON {{ this }} (email)", "CREATE INDEX idx_customer_created ON {{ this }} (created_at)" ] ) }} SELECT * FROM source_table ``` ##### Table organization Db2 supports different table organization types. You can specify these using post-hooks: ```sql {{ config( materialized='table', post_hook=[ "ALTER TABLE {{ this }} ORGANIZE BY ROW" ] ) }} SELECT * FROM source_table ``` #### Grants management The `ibm-dbt-db2` adapter supports managing grants on database objects: ```sql {{ config( materialized='table', grants={ 'select': ['reporting_role', 'analyst_user'], 'insert': ['etl_role'], 'update': ['etl_role'] } ) }} SELECT * FROM source_table ``` #### Case sensitivity Db2 uppercases unquoted identifiers by default. The adapter handles this automatically, but keep in mind: * Unquoted table and column names will be uppercased * Use double quotes to preserve case: `"MyTable"` vs `MYTABLE` * The adapter automatically handles case conversion for dbt operations #### Recommendations * **Check SQL Documentation**: Review IBM Db2 [SQL Reference](https://www.ibm.com/docs/en/db2) to understand platform-specific features and limitations. * **Use Incremental Models**: For large datasets, use incremental materializations with the `merge` strategy for optimal performance. * **Explicit Data Types**: Always define data types explicitly in seed configurations to ensure correct data loading. * **Connection Pooling**: Use appropriate thread counts in your profile to balance parallelism and connection overhead. * **SSL/TLS in Production**: Enable SSL/TLS security for production databases to protect data in transit. * **Monitor Performance**: Use Db2's built-in monitoring tools to track query performance and optimize your models. * **Test Incrementally**: When developing incremental models, test with small data samples before running on full datasets. --- ### IBM Netezza configurations #### Instance requirements To use IBM Netezza with `dbt-ibm-netezza` adapter, ensure the instance has an attached catalog that supports creating, renaming, altering, and dropping objects such as tables and views. The user connecting to the instance via the `dbt-ibm-netezza` adapter must have the necessary permissions for the target database. For more details, please visit the official [IBM documentation](https://cloud.ibm.com/docs/netezza?topic=netezza-getstarted) ##### IBM Netezza SQL Extension Toolkit Ensure that you have the SQL Extension Toolkit installed on your IBM Netezza system. This is a pre-reqsuisite to run all the function which require string data manipulation and view options. Check [docs](https://www.ibm.com/docs/en/netezza?topic=toolkit-sql-extensions-installation-setup) for more details. #### Seeds and prepared statements The `dbt-ibm-netezza` adapter offers comprehensive support for all [datatypes](https://www.ibm.com/docs/en/netezza?topic=nrl-data-types) in seed files. To leverage this functionality, you must explicitly define the data types for each column. You can configure column data types either in the dbt\_project.yml file or in property files, as supported by dbt. For more details on seed configuration and best practices, refer to the [dbt seed configuration documentation](https://docs.getdbt.com/reference/seed-configs.md). ##### Recommendations * **Check SQL Documentation:** Review IBM Netezza [SQL command reference ](https://www.ibm.com/docs/en/netezza?topic=dud-netezza-performance-server-sql-command-reference)to create your dbt project. --- ### IBM watsonx.data Presto configurations #### Instance requirements To use IBM watsonx.data Presto(java) with `dbt-watsonx-presto` adapter, ensure the instance has an attached catalog that supports creating, renaming, altering, and dropping objects such as tables and views. The user connecting to the instance via the `dbt-watsonx-presto` adapter must have the necessary permissions for the target catalog. For detailed setup instructions, including setting up watsonx.data, adding the Presto(Java) engine, configuring storages, registering data sources, and managing permissions, refer to the official IBM documentation: * watsonx.data Software Documentation: [IBM watsonx.data Software Guide](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x) * watsonx.data SaaS Documentation: [IBM watsonx.data SaaS Guide](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started) #### Session properties With IBM watsonx.data SaaS/Software instance, you can [set session properties](https://prestodb.io/docs/current/sql/set-session.html) to modify the current configuration for your user session. To temporarily adjust session properties for a specific dbt model or a group of models, use a [dbt hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). For example: ```sql {{ config( pre_hook="set session query_max_run_time='10m'" ) }} ``` #### Connector properties IBM watsonx.data SaaS/Software support various connector properties to manage how your data is represented. These properties are particularly useful for file-based connectors like Hive. For information on what is supported for each data source, refer to the following resources: * [watsonx.data SaaS Catalog](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-reg_database) * [watsonx.data Software Catalog](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=components-adding-data-source) #### File format configuration File-based connectors, such as Hive and Iceberg, allow customization of table materialization, data formats, and partitioning strategies in dbt models. The following examples demonstrate how to configure these options for each connector. ##### Hive Configuration Hive supports specifying file formats and partitioning strategies using the properties parameter in dbt models. The example below demonstrates how to create a partitioned table stored in Parquet format: ```sql {{ config( materialized='table', properties={ "format": "'PARQUET'", -- Specifies the file format "partitioned_by": "ARRAY['id']", -- Defines the partitioning column(s) } ) }} ``` For more details about Hive table creation and supported properties, refer to the [Hive connector documentation](https://prestodb.io/docs/current/connector/hive.html#create-a-managed-table). ##### Iceberg Configuration Iceberg supports defining file formats and advanced partitioning strategies to optimize query performance. The following example demonstrates how to create a ORC table partitioned using a bucketing strategy: ```sql {{ config( materialized='table', properties={ "format": "'ORC'", -- Specifies the file format "partitioning": "ARRAY['bucket(id, 2)']", -- Defines the partitioning strategy } ) }} ``` For more information about Iceberg table creation and supported configurations, refer to the [Iceberg connector documentation](https://prestodb.io/docs/current/connector/iceberg.html#create-table). #### Seeds The `dbt-watsonx-presto` adapter offers comprehensive support for all [watsonx.data Presto datatypes](https://www.ibm.com/support/pages/node/7157339) in seed files. To leverage this functionality, you must explicitly define the data types for each column. You can configure column data types either in the dbt\_project.yml file or in property files, as supported by dbt. For more details on seed configuration and best practices, refer to the [dbt seed configuration documentation](https://docs.getdbt.com/reference/seed-configs.md). #### Materializations The `dbt-watsonx-presto` adapter supports both table and view materializations, allowing you to manage how your data is stored and queried in watsonx.data Presto(java). For further information on configuring materializations, refer to the [dbt materializations documentation](https://docs.getdbt.com/reference/resource-configs/materialized.md). ##### Table The `dbt-watsonx-presto` adapter enables you to create and update tables through table materialization, making it easier to work with data in watsonx.data Presto. ###### Recommendations * **Check Permissions:** Ensure that the necessary permissions for table creation are enabled in the catalog or schema. * **Check Connector Documentation:** Review watsonx.data Presto [sql statement support](https://www.ibm.com/support/pages/node/7157339) to ensure it supports table creation and modification. ###### Limitations with some connectors Certain watsonx.data Presto connectors, particularly read-only ones or those with restricted permissions, do not allow creating or modifying tables. If you attempt to use table materialization with these connectors, you may encounter an error like: ```sh PrestoUserError(type=USER_ERROR, name=NOT_SUPPORTED, message="This connector does not support creating tables with data", query_id=20241206_071536_00026_am48r) ``` ##### View The `dbt-watsonx-presto` adapter automatically creates views by default, as views are the standard materialization in dbt. If no materialization is explicitly specified, dbt will create a view in watsonx.data Presto. To confirm whether your connector supports view creation, refer to the watsonx.data [sql statement support](https://www.ibm.com/support/pages/node/7157339). #### Unsupported features The following features are not supported by the `dbt-watsonx-presto` adapter * Incremental Materialization * Materialized Views * Snapshots --- ### IBM watsonx.data Spark configurations #### Instance requirements To use IBM watsonx.data Spark with `dbt-watsonx-spark` adapter, ensure the instance has an attached catalog that supports creating, renaming, altering, and dropping objects such as tables and views. The user connecting to the instance via the `dbt-watsonx-spark` adapter must have the necessary permissions for the target catalog. For detailed setup instructions, including setting up watsonx.data, adding the Spark engine, configuring storages, registering data sources, and managing permissions, refer to the official IBM documentation: * watsonx.data Software Documentation: [IBM watsonx.data Software Guide](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x) * watsonx.data SaaS Documentation: [IBM watsonx.data SaaS Guide](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started) #### Session properties With IBM watsonx.data SaaS/Software instance, you can [set session properties](https://sparkdb.io/docs/current/sql/set-session.html) to modify the current configuration for your user session. To temporarily adjust session properties for a specific dbt model or a group of models, use a [dbt hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). For example: ```sql {{ config( pre_hook="set session query_max_run_time='10m'" ) }} ``` #### Connector properties IBM watsonx.data SaaS/Software supports various Spark-specific connector properties to control data representation, execution performance, and storage format. For more details on supported configurations for each data source, refer to: * [watsonx.data SaaS Catalog](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-reg_database) * [watsonx.data Software Catalog](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=components-adding-data-source) ##### Additional configuration The `dbt-watsonx-spark` adapter allows additional configurations to be set in the catalog profile: * `Catalog:` Specifies the catalog to use for the Spark connection. The plugin can automatically detect the file format type `(Iceberg, Hive, or Delta)` based on the catalog type. * `use_ssl:` Enables SSL encryption for secure connections. Example configuration: ```yaml project_name: target: "dev" outputs: dev: type: watsonx_spark method: http schema: [schema name] host: [hostname] uri: [uri] catalog: [catalog name] use_ssl: false auth: instance: [Watsonx.data Instance ID] user: [username] apikey: [apikey] ``` *** ##### File format configuration The supported file formats depend on the catalog type: * **Iceberg Catalog:** Supports **Iceberg** tables. * **Hive Catalog:** Supports **Hive** tables. * **Delta Lake Catalog:** Supports **Delta** tables. * **Hudi Catalog:** Supports **Hudi** tables. The plugin **automatically** detects the file format type based on the catalog specified in the configuration. By specifying file format dbt models. For example: ```sql {{ config( materialized='table', file_format='iceberg' or 'hive' or 'delta' or 'hudi' ) }} ``` **For more details**, refer to the [documentation.](https://spark.apache.org/docs/3.5.3/sql-ref-syntax.html#sql-syntax) #### Seeds and prepared statements You can configure column data types either in the dbt\_project.yml file or in property files, as supported by dbt. For more details on seed configuration and best practices, refer to the [dbt seed configuration documentation](https://docs.getdbt.com/reference/seed-configs.md). #### Materializations The `dbt-watsonx-spark` adapter supports table materializations, allowing you to manage how your data is stored and queried in watsonx.data Spark. For further information on configuring materializations, refer to the [dbt materializations documentation](https://docs.getdbt.com/reference/resource-configs/materialized.md). ##### Table The `dbt-watsonx-spark` adapter enables you to create and update tables through table materialization, making it easier to work with data in watsonx.data Spark. ##### View The adapter automatically creates views by default if no materialization is explicitly specified. ##### Incremental Incremental materialization is supported but requires additional configuration for partitioning and performance tuning. ###### Recommendations * **Check Permissions:** Ensure that the necessary permissions for table creation are enabled in the catalog or schema. * **Check Connector Documentation:** Review watsonx.data Spark [data ingestion in watsonx.data](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=data-overview-ingestion) to ensure it supports table creation and modification. #### Unsupported features Despite its extensive capabilities, the `dbt-watsonx-spark` adapter has some limitations: * **Incremental Materialization**: Supported but requires additional configuration for partitioning and performance tuning. * **Materialized Views**: Not natively supported in Spark SQL within Watsonx.data. * **Snapshots**: Not supported due to Spark’s lack of built-in snapshot functionality. * **Performance Considerations**: * Large datasets may require tuning of Spark configurations such as shuffle partitions and memory allocation. * Some transformations may be expensive due to Spark’s in-memory processing model. By understanding these capabilities and constraints, users can maximize the effectiveness of dbt with Watsonx.data Spark for scalable data transformations and analytics. --- ### identifier models/\<filename>.yml ```yml sources: - name: <source_name> database: <database_name> tables: - name: <table_name> identifier: <table_identifier> ``` #### Definition The table name as stored in the database. This parameter is useful if you want to use a source table name that differs from the table name in the database. #### Default By default, dbt will use the table's `name` parameter as the identifier. #### Examples ##### Use a simpler name for a source table than the one in your database models/\<filename>.yml ```yml sources: - name: jaffle_shop tables: - name: orders identifier: api_orders ``` In a downstream model: ```sql select * from {{ source('jaffle_shop', 'orders') }} ``` Will get compiled to: ```sql select * from jaffle_shop.api_orders ``` ##### Reference sharded tables as a source in BigQuery models/\<filename>.yml ```yml sources: - name: ga tables: - name: events identifier: "events_*" ``` In a downstream model: ```sql select * from {{ source('ga', 'events') }} -- filter on shards by suffix where _table_suffix > '20200101' ``` Will get compiled to: ```sql select * from `my_project`.`ga`.`events_*` -- filter on shards by suffix where _table_suffix > '20200101' ``` --- ### Indirect selection Indirect selection determines which tests to run when you select models or other resources. It applies to tests that are related to your selected resources through relationships in your DAG — for example, tests on upstream or downstream models, or tests that reference multiple models. Use the `--indirect-selection` flag with `dbt test` or `dbt build` to configure this behavior. You can set this as a CLI flag or an environment variable. In dbt v1, you can also configure user configurations in [YAML selectors](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) or in the `flags:` block of `dbt_project.yml`, which sets project-level flags. Indirect selection happens by default Even without explicitly using the [`--indirect-selection` flag](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md?indirect-selection-mode=eager#indirect-selection), dbt uses indirect selection when you run commands like `dbt test --select "stg_model_a+"`. The default mode is `eager`, which runs all tests that reference your selected models. For example, `dbt test --select model_b` will run tests defined on `model_b`, as well as tests defined on upstream models if those tests reference `model_b`. When all flags are set, the order of precedence is as follows. Refer to [About global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) for more details: 1. CLI configurations 2. Environment variables 3. User configurations You can set the flag to: `empty`, `buildable`, `cautious`, or `eager` (default). Learn more about these options in [Indirect selection in Test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md?indirect-selection-mode=eager#indirect-selection). Indirect selection modes control which tests run based on the models you select and their relationships in your DAG. These modes determine how dbt handles tests that reference your selected models, either directly or through upstream/downstream relationships. You can use the following modes (with `eager` as the default). Test exclusion is always greedy: if ANY parent is explicitly excluded, the test will be excluded as well. Building subsets of a DAG The `buildable` and `cautious` modes can be useful when you're only building a subset of your DAG, and you want to avoid test failures in `eager` mode caused by unbuilt resources. You can also achieve this with [deferral](https://docs.getdbt.com/reference/node-selection/defer.md). ###### Eager mode (default) Most inclusive and runs tests if *any* of the parent nodes are selected, regardless of whether all dependencies are met. This includes *any* tests that reference the selected nodes, even if they also reference other unselected nodes. For example, if you run `dbt test --select model_b`, eager mode will run: * Tests directly on `model_b` * Tests in upstream models (like `model_a`) that reference `model_b` * Tests in downstream models that reference `model_b` dbt builds models that depend on the selected model. In this mode, any tests depending on unbuilt resources will raise an error. ###### Buildable mode Buildable mode is a middle ground between `cautious` and `eager`, running only tests that reference selected nodes (or their ancestors). This mode is slightly more inclusive than `cautious` by including tests whose references are each within the selected nodes (or their ancestors). This mode is useful when a test depends on a model *and* a direct ancestor of that model, like confirming an aggregation has the same totals as its input. ###### Cautious mode Cautious is the most exclusive mode and ensures that tests are executed and models are built only when all necessary dependencies of the selected models are met. Restricts tests to only those that exclusively reference selected nodes. Tests will only be executed if all the nodes they depend on are selected, which prevents tests from running if one or more of its parent nodes are unselected and, consequently, unbuilt. ###### Empty mode Empty mode runs no tests and restricts the build to the selected node, ignoring all indirect dependencies. It doesn't execute any tests, whether they are directly attached to the selected node or not. The empty mode is automatically used for [interactive compilation](https://docs.getdbt.com/reference/commands/compile.md#interactive-compile). The following is a visualization of the impact `--indirect-selection` and the various flags have using three models, three tests, and `dbt build` as an example: [![dbt build](/img/docs/reference/indirect-selection-dbt-build.png?v=2 "dbt build")](#)dbt build [![Eager (default)](/img/docs/reference/indirect-selection-eager.png?v=2 "Eager (default)")](#)Eager (default) [![Buildable](/img/docs/reference/indirect-selection-buildable.png?v=2 "Buildable")](#)Buildable [![Cautious](/img/docs/reference/indirect-selection-cautious.png?v=2 "Cautious")](#)Cautious [![Empty](/img/docs/reference/indirect-selection-empty.png?v=2 "Empty")](#)Empty For example, you can run tests that only refer to selected nodes using a CLI configuration: Usage ```shell dbt test --indirect-selection cautious ``` Or you can run tests that only refer to selected nodes using an environment variable: Env var (Applies to dbt v1.11 and later) ```text $ export DBT_ENGINE_INDIRECT_SELECTION=cautious dbt run ``` You can also run tests that only refer to selected nodes using `dbt_project.yml` project-level flags: dbt\_project.yml ```yaml flags: indirect_selection: cautious ``` --- ### Infer configurations #### Authentication To connect to Infer from your dbt instance you need to set up a correct profile in your `profiles.yml`. The format of this should look like this: \~/.dbt/profiles.yml ```yaml <profile-name>: target: <target-name> outputs: <target-name>: type: infer url: "<infer-api-endpoint>" username: "<infer-api-username>" apikey: "<infer-apikey>" data_config: [configuration for your underlying data warehouse] ``` ##### Description of Infer Profile Fields | Field | Required | Description | | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Yes | Must be set to `infer`. This must be included either in `profiles.yml` or in the `dbt_project.yml` file. | | `url` | Yes | The host name of the Infer server to connect to. Typically this is `https://app.getinfer.io`. | | `username` | Yes | Your Infer username - the one you use to login. | | `apikey` | Yes | Your Infer api key. | | `data_config` | Yes | The configuration for your underlying data warehouse. The format of this follows the format of the configuration for your data warehouse adapter. | --- ### Input for unit tests Use inputs in your unit tests to reference a specific model or source for the test: * For `input:`, use a string that represents a `ref` or `source` call: * `ref('my_model')` or `ref('my_model', v='2')` or `ref('dougs_project', 'users')` * `source('source_schema', 'source_name')` * Optionally use for seeds: * If you don’t supply an input for a seed, we will use the seed *as* the input. * If you do supply an input for a seed, we will use that input instead. * Use “empty” inputs by setting rows to an empty list `rows: []` models/schema.yml ```yml unit_tests: - name: test_is_valid_email_address # this is the unique name of the test model: dim_customers # name of the model I'm unit testing given: # the mock data for your inputs - input: ref('stg_customers') rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') rows: - {tld: example.com} - {tld: gmail.com} ... ``` --- ### invalidate_hard_deletes legacy Legacy opt-in configuration to enable invalidating hard deleted records while snapshotting the query. This is a legacy config — Use the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config instead. In dbt release tracks and dbt 1.9 and higher, the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config replaces the `invalidate_hard_deletes` config for better control over how to handle deleted rows from the source. For new snapshots, set the config to `hard_deletes='invalidate'` instead of `invalidate_hard_deletes=true`. For existing snapshots, [arrange an update](https://docs.getdbt.com/reference/snapshot-configs.md#snapshot-configuration-migration) of pre-existing tables before enabling this setting. (Applies to dbt v1.9 and later) snapshots/\<filename>.yml ```yaml snapshots: - name: snapshot relation: source('my_source', 'my_table') config: strategy: timestamp invalidate_hard_deletes: true | false ``` dbt\_project.yml ```yml snapshots: <resource-path>: +strategy: timestamp +invalidate_hard_deletes: true ``` #### Description Opt-in feature to enable invalidating hard deleted records while snapshotting the query. #### Default By default the feature is disabled. #### Example (Applies to dbt v1.9 and later) snapshots/orders.yml ```yaml snapshots: - name: orders_snapshot relation: source('jaffle_shop', 'orders') config: schema: snapshots database: analytics unique_key: id strategy: timestamp updated_at: updated_at invalidate_hard_deletes: true ``` --- ### Jinja file extensions | allow\_jinja\_file\_extensions | dbt **v1 Latest** | dbt v1 | | ------------------------------ | ----------------- | ------ | | Introduced | 2026.5 | 1.12.0 | | Matured (default → `true`) | — | — | | Removed | — | — | The `allow_jinja_file_extensions` flag is set to `false` by default. When set to `true`, dbt recognizes Jinja-style extension suffixes (for example, `.j2`, `.jinja`, and `.jinja2`) appended to `.sql` and `.md` files. This lets you use Jinja-aware syntax highlighting in IDEs that associate these suffixes with Jinja templating. dbt strips the Jinja suffix when determining node names; resource names remain unchanged regardless of whether the Jinja suffix is present. For example, a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) file named `my_docs.md.j2` is parsed identically to `my_docs.md`, and a model file named `my_model.sql.j2` is parsed as the model `my_model`. When this flag is `false` or unset, dbt ignores files with these suffixes without logging a warning. If you've already added schema properties for that file, you'll see a "Did not find matching node for patch warning on schema.yml" warning. --- ### Jinja support by file type Jinja support varies by file type. Some files support Jinja expressions, built-in dbt functions, and custom macros. Others support only a few built-in functions, and some don't render Jinja at all. Use this page to check what each file type supports. This page covers three kinds of support: * **Jinja expressions** (`{{ }}`, `{% %}`, and `{# #}`): Control flow, string output, and comments * **Built-in dbt functions and variables**: Context methods like [`env_var()`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md), [`ref()`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), and [`target`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) * **Custom macros**: Project-defined macros like `{{ my_macro() }}` Jinja expressions are how you write dynamic logic in a file. Custom macros are one kind of custom Jinja block you define with tags like `{% macro %}`. Other block types include `{% test %}`, `{% materialization %}`, `{% snapshot %}`, and `{% docs %}`. Refer to [Where to define custom Jinja blocks](#where-to-define-custom-jinja-blocks) for where each belongs. #### Summary table | File type | Jinja expressions | dbt functions and variables | Custom macros | Notes | | ------------------------------------------------- | ---------------------- | --------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `.sql` models, analyses, singular tests | ✅ | ✅ | ✅ | Call macros here, including hooks and operations. Refer to [Where to define custom Jinja blocks](#where-to-define-custom-jinja-blocks). | | `.sql` generic tests (`{% test %}`) | ✅ | ✅ | ✅ | | | `.sql` snapshots (legacy `{% snapshot %}`) | ✅ | ✅ | ✅ | Prefer [YAML snapshots](https://docs.getdbt.com/docs/build/snapshots.md). | | `.sql` materializations (`{% materialization %}`) | ✅ | ✅ | ✅ | | | `.md` docs blocks (`{% docs %}`) | ✅ | ❌ | ❌ | Pure Jinja only; use [`doc()`](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md) to reference blocks. | | `dbt_project.yml` | ✅ | Limited | v1: ❌ / v2: Limited | Custom macros in v2 are Limited to hooks. The top-level `vars:` block is literal (not Jinja). Refer to [dbt\_project.yml context](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt-project-yml-context.md). | | `profiles.yml` | ✅ | Limited | ❌ | Supports the `DBT_ENV_SECRET_` prefix for [`env_var()`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md). Refer to [profiles.yml context](https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context.md). | | `packages.yml` | ✅ | Limited | ❌ | Supports the `DBT_ENV_SECRET_` prefix for [`env_var()`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md). Refer to [packages.yml context](<https://docs.getdbt.com/reference/dbt-jinja-functions/packages.yml context.md>). | | `properties.yml` / `schema.yml` | ✅ | Limited | v1: ❌ / v2: ✅ | In v2 you can call custom macros in YAML values (for example `description`). `ref()` and `source()` are not available. Refer to [properties.yml context](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt-properties-yml-context.md). | | `selectors.yml` | ✅ | Limited | v1: ❌ / v2: ✅ | Also supports `env_var()`, `target`, `as_bool()`, and related context. Refer to [YAML selectors](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md). | | `dependencies.yml` | v1: ❌ / v2: ✅ | v1: ❌ / v2: Limited | ❌ | In v2, Jinja works (for example `env_var()` in a git URL). Prefer `packages.yml` if the project must also run on v1. | | `.py` Python models | ❌ | ❌ | ❌ | Use the Python `dbt` class API instead. | | `.csv` seeds | ❌ | ❌ | ❌ | Data files only. | | Unit test SQL fixtures | ❌ | ❌ | ❌ | Refer to [data formats](https://docs.getdbt.com/reference/resource-properties/data-formats.md). | | `.sql.j2` / `.md.j2` (optional) | Same as `.sql` / `.md` | Same as `.sql` / `.md` | Same as `.sql` / `.md` | Requires [`allow_jinja_file_extensions`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) (`false` by default). | <br /> **Table key** * **Limited:** Only a subset of built-in dbt functions and variables is available (typically [`env_var()`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md), [`var()`](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md), and [`target`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md)). The exact list depends on the file type. * For custom macros, Limited means support only in specific fields (for example hooks in `dbt_project.yml` in v2). * **Same as `.sql` / `.md`:** Optional `.j2` extensions match the non-`.j2` file type (for example `.sql.j2` matches `.sql`). #### Where to define custom Jinja blocks Calling a macro (`{{ my_macro() }}`) uses Jinja expressions. Defining one uses a custom Jinja block (`{% macro %}`). Use the table below for where each block type belongs: | Block | File type | Location | Notes | | ----------------------- | --------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `{% macro %}` | `.sql` | `macros/` | Call from models, tests, hooks, and more. | | `{% test %}` | `.sql` | `tests/generic/` or `macros/` | Can call other macros. | | `{% materialization %}` | `.sql` | `macros/` | Define custom materializations here. | | `{% snapshot %}` | `.sql` | `snapshots/` (legacy) | Prefer [YAML snapshots](https://docs.getdbt.com/docs/build/snapshots.md). Refer to [Legacy snapshot configurations](https://docs.getdbt.com/reference/resource-configs/snapshots-jinja-legacy.md) for the older syntax. | | `{% docs %}` | `.md` | Any resource path or `docs/` | Pure Jinja only; no custom macro calls. | #### Common exceptions Some config fields are stricter than the summary table. ##### `vars:` block in `dbt_project.yml` Values in the `vars:` block in `dbt_project.yml` are literal, not Jinja. To set values dynamically, use `--vars` on the command line. Refer to [Project variables](https://docs.getdbt.com/docs/build/project-variables.md). ##### `sql_header` config The [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) config doesn't support Jinja or macros like `ref()`, `source()`, and `{{ this }}`. #### Jinja differences between v1 and v2 v2 supports some Jinja that v1 doesn't: * Custom macros in `properties.yml` and `selectors.yml` values * Custom macros in `dbt_project.yml` hooks (Limited) * Jinja in `dependencies.yml` If your project must also run on v1, don't use these. In v1, a custom macro in a YAML file fails with an undefined macro error. Jinja in `dependencies.yml` stays an unrendered string that dbt passes to git, which typically fails the install. This example calls a custom macro in a property value. It works in v2 but fails in v1: models/schema.yml ```yaml models: - name: customers description: "{{ my_macro() }}" ``` This example uses `env_var()` in a package spec. Put it in `packages.yml` so it works in v1 and v2: packages.yml ```yaml packages: - git: "https://{{ env_var('DBT_ENV_SECRET_GIT_CREDENTIAL') }}@github.com/dbt-labs/awesome_repo.git" ``` For more on package files, refer to [Packages use cases](https://docs.getdbt.com/docs/build/packages.md#use-cases). #### Related docs * [Jinja and macros](https://docs.getdbt.com/docs/build/jinja-macros.md) * [YAML tips](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) * [dbt Jinja functions and context variables](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) * [Packages use cases](https://docs.getdbt.com/docs/build/packages.md#use-cases) --- ### JSON artifacts ##### Write JSON artifacts The `WRITE_JSON` config determines whether dbt writes [JSON artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) (for example, `manifest.json`, `run_results.json`) to the `target/` directory. JSON serialization can be slow, and turning this flag off *might* make invocations of dbt faster. Alternatively, you can disable this config to perform a dbt operation and avoid overwriting artifacts from a previous run step. Usage ```text dbt run --no-write-json ``` ##### Target path By default, dbt will write JSON artifacts and compiled SQL files to a directory named `target/`. This directory is located relative to `dbt_project.yml` of the active project. Just like other global configs, it is possible to override these values for your environment or invocation by using the CLI option (`--target-path`) or environment variables ((Applies to dbt v1.11 and later) `DBT_ENGINE_TARGET_PATH`). --- ### lag_tolerance ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +state: lag_tolerance: <duration_string> ``` ##### Properties YAML file models/\<filename>.yml ```yaml models: - name: my_model config: state: lag_tolerance: <duration_string> ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( state={ "lag_tolerance": "<duration_string>" } ) }} ``` #### Definition Source systems may update more frequently than some downstream models need to rebuild. For example, a model used for daily reporting doesn't need to refresh more than once per day, even if new upstream data is available hourly, while the model powering customer facing metrics that uses some of the same sources may need to update every 30 minutes. `lag_tolerance` sets how long dbt State waits before rebuilding a node once its upstream data changes. A node rebuilds only when *both* are true: its last build is older than the `lag_tolerance` window, and its upstream data has changed since that build. This acts as a compute-saving buffer that helps you stay aligned with data freshness [Service Level Agreements (SLAs)](https://www.getdbt.com/blog/data-slas-best-practices) without unnecessary rebuilds. It supports two key scenarios: * **Aligning builds with SLA requirements**: `lag_tolerance` allows you to align model execution directly with data freshness SLA requirements, decoupling high-frequency upstream changes from downstream models that operate under wider, less demanding freshness requirements. * **Protecting compute during upstream SLA breaches**: `lag_tolerance` protects your compute budget during freshness SLA breaches, preventing costly downstream rebuilds on static data when an upstream dependency fails its freshness SLA. The `lag_tolerance` config accepts two value types: * **Duration strings** in the format `<number><unit>`: | Unit | Accepted values | | ------- | ------------------------ | | Seconds | `s`, `second`, `seconds` | | Minutes | `m`, `minute`, `minutes` | | Hours | `h`, `hour`, `hours` | | Days | `d`, `day`, `days` | | Weeks | `w`, `week`, `weeks` | * **Jinja expressions** - `lag_tolerance` is evaluated as a Jinja template, so you can use any dbt context variables (`target`, `var()`, `env_var()`) to set dynamic tolerances. This is useful for applying different tolerances per environment without duplicating config blocks: ```yaml lag_tolerance: "{{ '4h' if target.name == 'prod' else '7d' }}" ``` ##### When a node rebuilds dbt State rebuilds a node only when *both* conditions are met. If either is false, it reuses the existing node: | Condition | What it means | | ------------------------- | ----------------------------------------------------------------- | | The tolerance has elapsed | The time since the node's last build exceeds its `lag_tolerance`. | | Upstream data has changed | At least one upstream dependency has new data since that build. | `lag_tolerance` sets a minimum time between rebuilds `lag_tolerance` controls how often a node can rebuild, not how fresh its upstream data has to be. ###### Example Let's say you have a job that runs every 30 minutes, your model has a `45m` tolerance, and it last built at `08:00`. New upstream data arrived at `08:20`. | Job run | Time since last build | New upstream data since last build? | Result | | ------- | --------------------- | ----------------------------------- | ----------------------------------------------------------------- | | `08:30` | `30m` | Yes | Reuse: tolerance hasn't elapsed. | | `09:00` | `60m` | Yes | Rebuild: both conditions are met. | | `09:30` | `30m` | No | Reuse: tolerance hasn't elapsed and upstream data hasn't changed. | | `10:00` | `60m` | No | Reuse: upstream data hasn't changed since `09:00`. | The `08:20` data waited until `09:00` to be picked up. This was the first run where the build was older than `45m` *and* upstream data had changed. At `10:00`, the build was old enough, but because no new upstream data had arrived since `09:00`, dbt reused the node. To rebuild a node whenever its upstream data changes, set `lag_tolerance` to `0s`: ```yaml state: lag_tolerance: 0s ``` ##### When does `lag_tolerance` apply `lag_tolerance` only applies to data freshness checks. A downstream model still rebuilds within its tolerance window if an upstream model's compiled SQL has changed since the last run, regardless of the `lag_tolerance` setting. This often happens with incremental models. The first time an incremental model runs, it executes a full load with no `WHERE` clause. On subsequent runs, `is_incremental()` becomes true and a filter is appended, changing the compiled SQL. dbt State detects this as a query change on the upstream model and rebuilds all downstream models, even those whose `lag_tolerance` has not elapsed. For example, `fct_orders` is an incremental model that `agg_orders_daily` depends on: models/fct\_orders.sql ```sql {{ config(materialized='incremental', unique_key='id') }} select id, amount from {{ ref('raw_orders') }} {% if is_incremental() %} where id > (select max(id) from {{ this }}) {% endif %} ``` models/agg\_orders\_daily.sql ```sql {{ config(materialized='table', state={'lag_tolerance': '3h'}) }} select date_trunc('day', created_at) as day, sum(amount) as total from {{ ref('fct_orders') }} group by 1 ``` When `fct_orders` transitions from a full load to an incremental run, its compiled SQL changes. `agg_orders_daily` rebuilds on that run despite its 3-hour `lag_tolerance`. tip To help you tune `lag_tolerance` values, the **dbt State** page on the dbt platform provides [lag tolerance recommendations](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#lag-tolerance-recommendations) based on your models' 30-day build history, so you can see which models would benefit from a higher tolerance. #### Default If you don't set `lag_tolerance`, dbt State uses `45m` (45 minutes). #### Examples ##### Use different tolerances per environment Use a Jinja expression to set a shorter tolerance in production and a longer tolerance elsewhere. This keeps production data fresh while reducing unnecessary rebuilds during development: dbt\_project.yml ```yaml models: +state: lag_tolerance: "{{ '4h' if target.name == 'prod' else '7d' }}" ``` In this example, models in the `prod` target rebuild once their last build is more than 4 hours old and their upstream data has changed. In all other environments, models rebuild once their last build is more than 7 days old and their upstream data has changed. ##### Vary tolerance by day of the week Use a Jinja expression to evaluate the day of the week and apply a tighter tolerance on weekdays than on weekends: dbt\_project.yml ```yaml models: +state: lag_tolerance: "{{ '24h' if modules.datetime.datetime.today().weekday() in (5, 6) else '1h' }}" ``` In this example, models rebuild once their last build is more than 1 hour old (Monday to Friday) or more than 24 hours old (Saturday to Sunday) and their upstream data has changed. ##### Apply different tolerances per folder Set different tolerances for different parts of your project by targeting folders: dbt\_project.yml ```yaml models: <your_project>: marts: +state: lag_tolerance: 1d staging: +state: lag_tolerance: 1h ``` ##### Override for a specific model Override the project-level default for a single model: models/my\_model.yml ```yaml models: - name: my_model config: state: lag_tolerance: 1h ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) --- ### Latest version pointer for versioned models | latest\_version\_pointer\_enabled\_by\_default | dbt **v1 Latest** | dbt v1 | dbt v2 | | ---------------------------------------------- | ----------------- | ------ | -------------- | | Introduced | 2026.5 | 1.12.0 | Early preview | | Matured (default → `true`) | — | — | Already `true` | | Removed | — | — | — | The `latest_version_pointer_enabled_by_default` flag is set to `false` by default in dbt v1. In dbt v2, this flag defaults to `true`, which enables the latest version pointer for all versioned models automatically. When you set it to `true`, dbt automatically creates a [latest version pointer](https://docs.getdbt.com/docs/mesh/govern/model-versions.md#pointing-to-the-latest-version) view for every versioned model in the project, without requiring per-model configuration. The pointer view is named after the model's base name (for example, `dim_customers`) and always points to the relation for the model with `is_latest_version: true` (for example, `dim_customers_v2`). Without this flag, you must opt in per model by setting [`latest_version_pointer.enabled: true`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) in the model config. --- ### latest_version models/\<schema>.yml ```yml models: - name: model_name latest_version: 2 versions: - v: 2 - v: 1 ``` #### Definition The latest version of this model. The "latest" version is relevant for: 1. Resolving `ref()` calls to this model that are "unpinned" (a version is not explicitly specified) 2. Selecting model versions using the [`version:` selection method](https://docs.getdbt.com/reference/node-selection/methods.md#version), based on whether a given model version is `latest`, `prerelease`, or `old` This value can be a string or a numeric (integer or float) value. It must be one of the [version identifiers](https://docs.getdbt.com/reference/resource-properties/versions.md#v) specified in this model's list of `versions`. To run the latest version of a model, you can use the [`--select` flag](https://docs.getdbt.com/reference/node-selection/syntax.md). Refer to [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md#run-a-model-with-multiple-versions) for more information and syntax. #### Default If not specified for a versioned model, `latest_version` defaults to the largest [version identifier](https://docs.getdbt.com/reference/resource-properties/versions.md#v): numerically greatest (if all version identifiers are numeric), otherwise the alphabetically last (if they are strings). For a non-versioned model (no `versions` list), `latest_version` has no value. If `latest_version` is not specified for a versioned model, `latest_version` defaults to the largest. #### Example models/\<schema>.yml ```yml models: - name: model_name versions: - v: 3 - v: 2 - v: 1 ``` If `latest_version` is not specified, the `latest_version` is `3`. Any unpinned references -- `ref('model_name')` -- will resolve to `model_name.v3`. Both `v1` and `v2` are considered "old" versions. models/\<schema>.yml ```yml models: - name: model_name latest_version: 2 versions: - v: 3 - v: 2 - v: 1 ``` In this case, the `latest_version` is explicitly set to `2`. Any unpinned references will resolve to `model_name.v2`. `v3` is considered "prerelease", and `v1` is considered "old". --- ### latest_version_pointer Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). ##### Project file dbt\_project.yml ```yaml models: <resource-path>: +latest_version_pointer: enabled: true | false alias: <string> ``` ##### Property file models/schema.yml ```yaml models: - name: [<model-name>] config: latest_version_pointer: enabled: true | false alias: <string> ``` ##### SQL config models/\<model\_name>.sql ```sql {{ config( latest_version_pointer={ "enabled": true | false, "alias": "<string>" } ) }} ``` #### Definition dbt platform on dbt v2 In dbt v2, `latest_version_pointer` is enabled by default for all versioned models. If you have a versioned model with an explicit `alias` that matches the model's base name, you may see a `dbt1005` collision error. See [Naming collisions](#naming-collisions) below for how to resolve it. The `latest_version_pointer` config creates a view named after a [versioned model's](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) base name (for example, `dim_customers`) that always points to the latest versioned relation (for example, `dim_customers_v2`). The view is created after the model with `is_latest_version = true` materializes successfully and is skipped for all other versions. You can also enable this feature globally for all versioned models by setting the [`latest_version_pointer_enabled_by_default`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) flag to `true` in `dbt_project.yml`: dbt\_project.yml ```yaml flags: latest_version_pointer_enabled_by_default: true ``` `latest_version_pointer` accepts two optional sub-keys for per-model control: | Key | Type | Default | Description | | --------- | ------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | boolean | — | Enables or disables the pointer view for this model. Overrides the `latest_version_pointer_enabled_by_default` project flag when set; defers to the flag when not set. | | `alias` | string | Model base name | Custom name for the pointer view. Overrides the default base name. For more information, see [Alias customization](#alias-customization). | #### Alias customization By default, the pointer view uses the model's base name (for example, `dim_customers`). You can customize this in two ways: * **Per model**: Set `latest_version_pointer.alias` in the model config. * **Globally**: Override the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro in your project. This macro follows the same pattern as [`generate_alias_name`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_alias_name). #### Naming collisions To prevent naming collisions, dbt raises a `dbt1005` error if the latest version's alias is the same as the pointer view name. In dbt v2, where `latest_version_pointer` is enabled by default, this error can surface on models that have an explicit `alias` matching the model's base name, even if you never configured the pointer yourself. For example, the following configuration would raise `dbt1005` because both `dim_customers_v2` and the pointer view would resolve to `dim_customers`: ```text dbt1005 (Cannot create latest version pointer: the latest version of 'dim_customers' is already aliased to 'dim_customers') ``` ```yaml models: - name: dim_customers versions: - v: 1 - v: 2 config: alias: dim_customers # collides with the pointer view name config: latest_version_pointer: enabled: true ``` To fix this, select one of the following options: * [Remove the `alias` (recommended)](#remove-the-alias-recommended) * [Disable the latest version pointer for that model](#disable-the-latest-version-pointer-for-that-model) * [Set a unique `alias`](#set-a-unique-alias) * [Override the `generate_latest_version_pointer_alias` macro](#override-the-generate_latest_version_pointer_alias-macro) ###### Remove the `alias` (recommended) Remove the `alias` from the latest version and let the automatic pointer handle it: ```yaml config: alias: dim_customers ``` ###### Disable the latest version pointer for that model This approach is immediately backward-compatible for pre-existing `alias` configs: ```yaml config: alias: dim_customers config: latest_version_pointer: enabled: false ``` ###### Set a unique `alias` ```yaml config: alias: dim_customers_latest ``` ###### Override the `generate_latest_version_pointer_alias` macro Override the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro to use a different naming convention globally: macros/generate\_latest\_version\_pointer\_alias.sql ```sql {% macro generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {{ node.name ~ "_latest" }} {%- endmacro %} ``` #### Related documentation * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) * [Pointing to the latest version](https://docs.getdbt.com/docs/mesh/govern/model-versions.md#pointing-to-the-latest-version) * [`latest_version_pointer_enabled_by_default` flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) * [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md) * [`latest_version`](https://docs.getdbt.com/reference/resource-properties/latest_version.md) --- ### Legacy snapshot configuration legacy Use legacy SQL-based snapshot configurations with Jinja blocks in any dbt version. dbt v1.9 introduced YAML-based configs for better readability and environment awareness. There are situations where you want to use the legacy syntax for [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) in any dbt version or release track. This page details how you can use the legacy SQL-based configurations if you need to. In dbt v1.9, this syntax was replaced with a [YAML-based configuration](https://docs.getdbt.com/reference/snapshot-configs.md#configuring-snapshots) in [dbt's **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). The benefits of YAML-based configurations are that the snapshots are environment aware, meaning you don't have to specify `schema` or `database`, and the syntax is more concise. For new snapshots, we recommend using these latest YAML-based configs. If you'd like to move to the YAML-based configuration for existing snapshots, you can [migrate over](https://docs.getdbt.com/reference/snapshot-configs.md#snapshot-configuration-migration). When would you want to use the SQL-based syntax and YAML-based syntax? * SQL-based syntax: * Defined in `.sql` files within a snapshot Jinja block, typically located in your `snapshots` directory. Available in all versions. * Useful for existing snapshots already using this syntax. * Suitable for performing very light transformations (but creating a separate ephemeral model for transformations is recommended for better maintainability). * YAML-based syntax: * Defined in `whatever_name.yml` or in the `snapshots` or `models` directory you prefer. Available in dbt's **v1 Latest** release track and dbt v1.9 and later. * Ideal for new snapshots or existing snapshots that need to be [migrated](https://docs.getdbt.com/reference/snapshot-configs.md#snapshot-configuration-migration). * Create transformations separate from the snapshot file by creating an ephemeral model and referencing it in the snapshot using the `relation` field. #### Snapshot configurations Although you can use the more performant YAML-based configuration, you might still want to use the legacy configuration to define your snapshots if it suits your needs. Snapshots can be configured in two main ways: * Using [snapshot-specific configurations](#snapshot-specific-configurations) * Or using [general configurations](#general-configuration) These configurations allow you to control how dbt detects changes in your data and where snapshots are stored. Both types of configurations can coexist in your project in the same `config` block (or from your `dbt_project.yml` file or `properties.yaml` file). One of the most important configs you can decide is [strategies](#snapshot-strategies), which tells dbt how to detect modified rows. ##### Snapshot specific configurations Snapshot-specific configurations are applicable to only one dbt resource type rather than multiple resource types. You can define these settings within the resource’s file using the `{{ config() }}` macro (as well as in the project file (`dbt_project.yml`) or a property file (`models/properties.yml` for models, similarly for other resources)). snapshots/orders\_snapshot.sql ```sql { % snapshot orders_snapshot %} {{ config( target_schema="<string>", target_database="<string>", unique_key="<column_name_or_expression>", strategy="timestamp" | "check", updated_at="<column_name>", check_cols=["<column_name>"] | "all" invalidate_hard_deletes=true | false ) }} select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ``` ##### General configuration Use general configurations for broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project YAML file, properties YAML files, or within resource-specific files using a config block. snapshots/snapshot.sql ```sql {{ config( enabled=true | false, tags="<string>" | ["<string>"], alias="<string>", pre_hook="<sql-statement>" | ["<sql-statement>"], post_hook="<sql-statement>" | ["<sql-statement>"] persist_docs={<dict>} grants={<dict>} ) }} ``` ##### Snapshot strategies Snapshot "strategies" define how dbt knows if a row has changed. There are two strategies built-in to dbt that require the `strategy` parameter: * [Timestamp](https://docs.getdbt.com/reference/resource-configs/snapshots-jinja-legacy.md?strategy=timestamp#snapshot-strategies) — Uses an `updated_at` column to determine if a row has changed. * [Check](https://docs.getdbt.com/reference/resource-configs/snapshots-jinja-legacy.md?strategy=check#snapshot-strategies) — Compares a list of columns between their current and historical values to determine if a row has changed. Uses the `check_cols` parameter. ##### Timestamp The timestamp strategy uses an `updated_at` field to determine if a row has changed. If the configured `updated_at` column for a row is more recent than the last time the snapshot ran, then dbt will invalidate the old record and record the new one. If the timestamps are unchanged, then dbt will not take any action. ###### Example snapshots/timestamp\_example.sql ```sql {% snapshot orders_snapshot_timestamp %} {{ config( target_schema='snapshots', strategy='timestamp', unique_key='id', updated_at='updated_at', ) }} select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ``` ##### Check The check strategy is useful for tables which do not have a reliable `updated_at` column. It requires the `check_cols` parameter, which is a list of columns within the results of your snapshot query to check for changes. Alternatively, use all columns using the all value (however this may be less performant). ###### Example snapshots/check\_example.sql ```sql {% snapshot orders_snapshot_check %} {{ config( strategy='check', unique_key='id', check_cols=['status', 'is_cancelled'], ) }} select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ``` ###### Examples  Check a list of columns for changes snapshots/check\_example.sql ```sql {% snapshot orders_snapshot_check %} {{ config( strategy='check', unique_key='id', check_cols=['status', 'is_cancelled'], ) }} select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ```  Check all columns for changes snapshots/check\_example.sql ```sql {% snapshot orders_snapshot_check %} {{ config( strategy='check', unique_key='id', check_cols='all', ) }} select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ``` #### Configuration reference Configure your snapshot to tell dbt how to detect record changes. Snapshots are `select` statements, defined within a snapshot block in a `.sql` file (typically in your `snapshots` directory or any other directory). The following table outlines the configurations available for snapshots: (Applies to dbt v1.9 and later) | Config | Description | Required? | Example | | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------- | ----------- | | [database](https://docs.getdbt.com/reference/resource-configs/database.md) | Specify a custom database for the snapshot | No | analytics | | [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) | Specify a custom schema for the snapshot | No | snapshots | | [strategy](https://docs.getdbt.com/reference/resource-configs/strategy.md) | The snapshot strategy to use. One of `timestamp` or `check` | Yes | timestamp | | [unique\_key](https://docs.getdbt.com/reference/resource-configs/unique_key.md) | A primary key column or expression for the record | Yes | id | | [check\_cols](https://docs.getdbt.com/reference/resource-configs/check_cols.md) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | \["status"] | | [updated\_at](https://docs.getdbt.com/reference/resource-configs/updated_at.md) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated\_at | | [invalidate\_hard\_deletes](https://docs.getdbt.com/reference/resource-configs/invalidate_hard_deletes.md) | Find hard deleted records in source, and set `dbt_valid_to` current time if no longer exists | No | True | * A number of other configurations are also supported (like, `tags` and `post-hook`), check out the full list [here](https://docs.getdbt.com/reference/snapshot-configs.md). * Snapshots can be configured from both your `dbt_project.yml` file and a `config` block, check out the [configuration docs](https://docs.getdbt.com/reference/snapshot-configs.md) for more information. * Note: BigQuery users can use `target_project` and `target_dataset` as aliases for `target_database` and `target_schema`, respectively. * Before v1.9, `target_schema` (required) and `target_database` (optional) set a fixed schema or database for snapshots, making it hard to separate dev and prod environments. In v1.9, `target_schema` became optional, allowing environment-aware snapshots. By default, snapshots now use `generate_schema_name` or `generate_database_name`, but developers can still specify a custom location using [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) and [database](https://docs.getdbt.com/reference/resource-configs/database.md), consistent with other resource types. #### Add snapshot to a project To add a snapshot to your project: 1. Create a file in your `snapshots` directory with a `.sql` file extension. For example,`snapshots/orders.sql` 2. Use a `snapshot` block to define the start and end of a snapshot: snapshots/orders\_snapshot.sql ```sql {% snapshot orders_snapshot %} {% endsnapshot %} ``` 3. Write a `select` statement within the snapshot block (tips for writing a good snapshot query are below). This select statement defines the results that you want to snapshot over time. You can use `sources` or `refs` here. snapshots/orders\_snapshot.sql ```sql {% snapshot orders_snapshot %} select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ``` 4. Check whether the result set of your query includes a reliable timestamp column that indicates when a record was last updated. For our example, the `updated_at` column reliably indicates record changes, so we can use the `timestamp` strategy. If your query result set does not have a reliable timestamp, you'll need to instead use the `check` strategy — more details on this in the next step. 5. Add configurations to your snapshot using a `config` block. You can also [configure your snapshot from your `dbt_project.yml` file](https://docs.getdbt.com/reference/snapshot-configs.md). (Applies to dbt v1.9 and later) snapshots/orders\_snapshot.sql ```sql {% snapshot orders_snapshot %} {{ config( database='analytics', schema='snapshots', unique_key='id', strategy='timestamp', updated_at='updated_at', ) The following table outlines the configurations available for snapshots: select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ``` 6. Run the `dbt snapshot` [command](https://docs.getdbt.com/reference/commands/snapshot.md). For our example, a new table will be created at `analytics.snapshots.orders_snapshot`. You can change the `target_database` configuration, the `target_schema` configuration and the name of the snapshot (as defined in `{% snapshot .. %}`) will change how dbt names this table. ```dbt Running with dbt=1.8.0 15:07:36 | Concurrency: 8 threads (target='dev') 15:07:36 | 15:07:36 | 1 of 1 START snapshot snapshots.orders_snapshot...... [RUN] 15:07:36 | 1 of 1 OK snapshot snapshots.orders_snapshot..........[SELECT 3 in 1.82s] 15:07:36 | 15:07:36 | Finished running 1 snapshots in 0.68s. Completed successfully Done. PASS=2 ERROR=0 SKIP=0 TOTAL=1 ``` 1. Inspect the results by selecting from the table dbt created. After the first run, you should see the results of your query, plus the [snapshot meta fields](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields) as described earlier. 2. Run the `dbt snapshot` command again, and inspect the results. If any records have been updated, the snapshot should reflect this. 3. Select from the `snapshot` in downstream models using the `ref` function. models/changed\_orders.sql ```sql select * from {{ ref('orders_snapshot') }} ``` 10. Snapshots are only useful if you run them frequently — schedule the `snapshot` command to run regularly. #### Examples This section outlines some examples of how to apply configurations to snapshots using the legacy method.  Apply configurations to one snapshot only Use config blocks if you need to apply a configuration to one snapshot only. snapshots/postgres\_app/orders\_snapshot.sql ```sql {% snapshot orders_snapshot %} {{ config( unique_key='id', strategy='timestamp', updated_at='updated_at' ) }} -- Pro-Tip: Use sources in snapshots! select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ```  Using the updated\_at parameter The `updated_at` parameter is required if using the timestamp strategy. The `updated_at` parameter is a column within the results of your snapshot query that represents when the record row was last updated. snapshots/orders.sql ```sql {{ config( strategy="timestamp", updated_at="column_name" ) }} ``` ###### Examples * ###### Using a column name `updated_at`: (Applies to dbt v1.9 and later) snapshots/orders.sql ```sql {% snapshot orders_snapshot %} {{ config( schema='snapshots', unique_key='id', strategy='timestamp', updated_at='updated_at' ) }} select * from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ``` * ###### Coalescing two columns to create a reliable `updated_at` column: Consider a data source that only has an `updated_at` column filled in when a record is updated (so a `null` value indicates that the record hasn't been updated after it was created). Since the `updated_at` configuration only takes a column name, rather than an expression, you should update your snapshot query to include the coalesced column. (Applies to dbt v1.9 and later) snapshots/orders.sql ```sql {% snapshot orders_snapshot %} {{ config( schema='snapshots', unique_key='id', strategy='timestamp', updated_at='updated_at_for_snapshot' ) }} select *, coalesce(updated_at, created_at) as updated_at_for_snapshot from {{ source('jaffle_shop', 'orders') }} {% endsnapshot %} ```  Using the unique\_key parameter The `unique_key` is a column name or expression that is unique for the inputs of a snapshot. dbt uses [`unique_key`](https://docs.getdbt.com/reference/resource-configs/unique_key.md) to match records between a result set and an existing snapshot, so that changes can be captured correctly. snapshots/orders.sql ```sql {{ config( unique_key="column_name" ) }} ``` ###### Examples * Using an `id` column as a unique key snapshots/orders.sql ```sql {{ config( unique_key="id" ) }} ``` You can also write this in YAML. This might be a good idea if multiple snapshots share the same `unique_key` (though we prefer to apply this configuration in a config block, as above). * ###### Using a combination of two columns as a unique key This configuration accepts a valid column expression. As such, you can concatenate two columns together as a unique key if required. It's a good idea to use a separator (like, '-') to ensure uniqueness. snapshots/transaction\_items\_snapshot.sql ```sql {% snapshot transaction_items_snapshot %} {{ config( unique_key="transaction_id||'-'||line_item_id", ... ) }} select transaction_id||'-'||line_item_id as id, * from {{ source('erp', 'transactions') }} {% endsnapshot %} ``` Though, it's probably a better idea to construct this column in your query and use that as the `unique_key`: snapshots/transaction\_items\_snapshot.sql ```sql {% snapshot transaction_items_snapshot %} {{ config( unique_key="id", ... ) }} select transaction_id || '-' || line_item_id as id, * from {{ source('erp', 'transactions') }} {% endsnapshot %} ``` --- ### limit Limit the number of failures that will be returned by a test query. We recommend using this config when working with large datasets and [storing failures in the database](https://docs.getdbt.com/reference/resource-configs/store_failures.md). ##### Specific test Configure a specific instance of a generic (schema) test: models/\<filename>.yml ```yaml models: - name: large_table columns: - name: very_unreliable_column data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ["a", "b", "c"] config: limit: 1000 # will only include the first 1000 failures ``` ##### One-off test Configure a one-off (data) test: tests/\<filename>.sql ```sql {{ config(limit = 1000) }} select ... ``` ##### Generic test block Set the default for all instances of a generic (schema) test, by setting the config inside its test block (definition): macros/\<filename>.sql ```sql {% test <testname>(model, column_name) %} {{ config(limit = 500) }} select ... {% endtest %} ``` ##### Project level Set the default for all tests in a package or project: dbt\_project.yml ```yaml data_tests: +limit: 1000 # all tests <package_name>: +limit: 50 # tests in <package_name> ``` --- ### loader models/\<filename>.yml ```yml sources: - name: <source_name> database: <database_name> loader: <string> tables: - ... ``` #### Definition Describe the tool that loads this source into your warehouse. Note that this property is for documentation purposes only — dbt does not meaningfully use this. #### Examples ##### Indicate which EL tool loaded data models/\<filename>.yml ```yml sources: - name: jaffle_shop loader: fivetran tables: - name: orders - name: customers - name: stripe loader: stitch tables: - name: payments ``` --- ### log **Args**: * `msg`: The message (string) to log * `info`: If False, write to the log file. If True, write to both the log file and stdout (default=False) Logs a line to either the log file or stdout. Details Code source Refer to [GitHub](https://github.com/dbt-labs/dbt/blob/HEAD/core/dbt/context/base.py#L549-L566) or the following code as a source: <br /> <br /> ```python def log(msg: str, info: bool = False) -> str: """Logs a line to either the log file or stdout. :param msg: The message to log :param info: If `False`, write to the log file. If `True`, write to both the log file and stdout. > macros/my_log_macro.sql {% macro some_macro(arg1, arg2) %} {{ log("Running some_macro: " ~ arg1 ~ ", " ~ arg2) }} {% endmacro %}" """ if info: fire_event(JinjaLogInfo(msg=msg, node_info=get_node_info())) else: fire_event(JinjaLogDebug(msg=msg, node_info=get_node_info())) return "" ``` ```sql {% macro some_macro(arg1, arg2) %} {{ log("Running some_macro: " ~ arg1 ~ ", " ~ arg2) }} {% endmacro %} ``` --- ### Logs ##### Log formatting dbt outputs logs to two different locations: CLI console and the log file. (Applies to dbt v2.0 and later) The `LOG_FORMAT` and `LOG_FORMAT_FILE` configs specify how dbt's logs should be formatted. Shared options are `text`, `json`, and `default`. The `otel` option is available for `LOG_FORMAT` only (console output). Setting `--log-format-file otel` has no effect. The `otel` format streams [OpenTelemetry](https://opentelemetry.io/)-style structured telemetry to the console. It uses a different schema than dbt v1's `json` logs. For JSONL files, Parquet export, OTLP, and how this maps to dbt v1 structured logging, refer to [dbt v2 telemetry and observability](https://docs.getdbt.com/reference/telemetry-observability.md). Usage ```text dbt build --log-format otel ``` (Applies to dbt v2.0 and later) The `text` format is the default for console logs and prints plain text progress messages: ```text dbt 2.0.1 Loading ~/.dbt/profiles.yml ``` The `json` format outputs fully structured logs in the JSON format: (Applies to dbt v2.0 and later) ```json {"data": {"log_version": 3, "version": "=2.0.0-preview.181"}, "info": {"category": "", "code": "A001", "extra": {}, "invocation_id": "019fb7d4-ce89-7712-8d06-5ad013a23be9", "level": "info", "msg": "Running with dbt-fusion=2.0.0-preview.181", "name": "MainReportVersion", "pid": 92554, "thread": "tokio-rt-worker", "ts": "2026-07-31T11:00:04.877509Z"}} {"data": {"msg": "Loading ~/.dbt/profiles.yml"}, "info": {"category": "", "code": "", "extra": {}, "invocation_id": "019fb7d4-ce89-7712-8d06-5ad013a23be9", "level": "info", "msg": "Loading ~/.dbt/profiles.yml", "name": "Generic", "pid": 92554, "thread": "tokio-rt-worker", "ts": "2026-07-31T11:00:04.882928Z"}} ``` (Applies to dbt v2.0 and later) When the `LOG_FORMAT` is set explicitly to `text`, `json`, or `default`, it takes effect in both the console and log files. The `otel` value applies to console output only. Use `LOG_FORMAT_FILE` to set a different format for the log file (`text`, `json`, or `default`). Usage ```text dbt run --log-format-file json ``` (Applies to dbt v2.0 and later) Tip: structured observability Use `--log-format otel` to stream OpenTelemetry-style telemetry to the console, or use `--otel-file-name` and related flags for file and platform integrations. Refer to [dbt v2 telemetry and observability](https://docs.getdbt.com/reference/telemetry-observability.md). For JSON-formatted log lines, use `--log-format json` with the `DEBUG` config: ```text dbt build --debug --log-format json ``` ##### Log Level The `LOG_LEVEL` config sets the minimum severity of events captured in the console and file logs. This is a more flexible alternative to the `--debug` flag. The available options for the log levels are `debug`, `info`, `warn`, `error`, or `none`. * Setting the `--log-level` will configure console and file logs. ```text dbt run --log-level debug ``` * Setting the `LOG_LEVEL` to `none` will disable information from being sent to either the console or file logs. ```text dbt run --log-level none ``` * To set the file log level as a different value than the console, use the `--log-level-file` flag. ```text dbt run --log-level-file error ``` * To only disable writing to the logs file but keep console logs, set `LOG_LEVEL_FILE` config to none. ```text dbt run --log-level-file none ``` ##### Debug-level logging The `DEBUG` config redirects dbt's debug logs to standard output. This has the effect of showing debug-level log information in the terminal in addition to the `logs/dbt.log` file. This output is verbose. The `--debug` flag is also available via shorthand as `-d`. Usage ```text dbt run --debug ``` ##### Log and target paths By default, dbt will write logs to a directory named `logs/`, and all other artifacts to a directory named `target/`. Both of those directories are located relative to `dbt_project.yml` of the active project. Just like other global configs, it is possible to override these values for your environment or invocation by using CLI options (`--target-path`, `--log-path`) or environment variables ((Applies to dbt v1.11 and later) `DBT_ENGINE_TARGET_PATH`, `DBT_ENGINE_LOG_PATH`). ##### Suppress non-error logs in output By default, dbt shows all logs in standard out (stdout). You can use the `QUIET` config to show only error logs in stdout. Logs will still include the output of anything passed to the [`print()`](https://docs.getdbt.com/reference/dbt-jinja-functions/print.md) macro. For example, you might suppress all but error logs to more easily find and debug a Jinja error. profiles.yml ```yaml config: quiet: true ``` Supply the `-q` or `--quiet` flag to `dbt run` to show only error logs and suppress non-error logs. ```text dbt run --quiet ``` ##### dbt list logging In [dbt version 1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#behavior-changes>), we updated the logging behavior of the [dbt list](https://docs.getdbt.com/reference/commands/list.md) command to include `INFO` level logs by default. You can use either of these parameters to ensure clean output that's compatible with downstream processes, such as piping results to [`jq`](https://jqlang.github.io/jq/manual/), a file, or another process: * `dbt list --log-level warn` (recommended; equivalent to previous default) * `dbt list --quiet` (suppresses all logging less than `ERROR` level, except for "printed" messages and list output) ##### Logging relational cache events The `LOG_CACHE_EVENTS` config allows detailed logging for [relational cache](https://docs.getdbt.com/reference/global-configs/cache), which are disabled by default. ```text dbt compile --log-cache-events ``` ##### Color You can set the color preferences for the file logs only within `profiles.yml` or using the `--use-colors-file / --no-use-colors-file` flags. profiles.yml ```yaml config: use_colors_file: False ``` ```text dbt run --use-colors-file dbt run --no-use-colors-file ``` --- ### lookback Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### Definition Configure a `lookback` window to reprocess additional batches during [microbatch incremental model](https://docs.getdbt.com/docs/build/incremental-microbatch.md) runs. It processes X batches up to the latest bookmark (the last successfully processed data point) to capture late-arriving records. Set the `lookback` to an integer greater than or equal to zero. The default value is `1`. You can configure `lookback` for a [microbatch incremental model](https://docs.getdbt.com/docs/build/incremental-microbatch.md) in your project YAML file (`dbt_project.yml`), properties YAML file (`models/properties.yml`), or SQL file config. #### Examples The following examples set `2` as the `lookback` config for the `user_sessions` model. Example in the `dbt_project.yml` file: dbt\_project.yml ```yml models: my_project: user_sessions: +lookback: 2 ``` Example in a property file: models/properties.yml ```yml models: - name: user_sessions config: lookback: 2 ``` Example in SQL config block: models/user\_sessions.sql ```sql {{ config( lookback=2 ) }} ``` --- ### Macro argument validation | validate\_macro\_args | dbt **v1 Latest** | dbt v1 | | -------------------------- | ----------------- | ------ | | Introduced | 2025.03 | 1.10.0 | | Matured (default → `true`) | 2026.09 | 1.12.0 | | Removed | — | — | <br /> dbt validates macro arguments using the `validate_macro_args` flag. Starting in dbt v1.12, this flag defaults to `true`. In the past, dbt didn't enforce a standard vocabulary for the [`type`](https://docs.getdbt.com/reference/resource-properties/arguments.md#type) field on macro arguments in YAML. Because of this, the `type` field was used for documentation only, and dbt didn't check that: * the argument names matched those in your macro * the argument types were valid or consistent with the macro's Jinja definition Here's an example of a documented macro: macros/filename.yml ```yaml macros: - name: <macro name> arguments: - name: <arg name> type: <string> ``` With this flag enabled (the default), dbt: * Validates macro arguments during project parsing. * Checks that all argument names in your YAML match those in the macro definition. * Raises warnings if the names or types don't match. * Validates that the [`type` values follow the supported format](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). * If no arguments are documented in the YAML, infers them from the macro and includes them in the [`manifest.json` file](https://docs.getdbt.com/reference/artifacts/manifest-json.md).  When does validation occur? Macro argument validation runs during project parsing, not during macro execution. Any dbt command that parses the project triggers validation. * In dbt v1: * Validation runs as part of parsing for most commands (`parse`, `build`, `run`, `test`, `seed`, `snapshot`, `compile`). * With a full parse, dbt validates all macros. * With partial parsing (the default), dbt validates only macros affected by changed files. * Use `--no-partial-parse` to force validation of all macros. #### Impact On its own, the flag emits warnings and builds continue. However, these warnings use the force-handled path and respect `--warn-error`, so projects with `--warn-error` set will see build failures at parse time. This affects projects where the `arguments:` listed in a macro's YAML patch no longer match the macro's actual Jinja signature. For those projects, every command fails at parse time until you either update the YAML arguments to match the macro or remove the `arguments:` block entirely.  Recommended actions If `InvalidMacroAnnotation` warnings are appearing at parse time (or causing parse failures with `--warn-error` set), check each log line — it names the macro and the specific mismatch. In the `macros:` YAML block for each affected macro, align the `arguments:` entries with the `{% macro name(args) %}` declaration in the `.sql` file, or remove the `arguments:` block entirely. To silence the warnings without fixing the YAML, use `warn_error_options`: dbt\_project.yml ```yaml flags: warn_error_options: silence: - InvalidMacroAnnotation ``` To opt out of this behavior, set the flag to `false`: dbt\_project.yml ```yaml flags: validate_macro_args: false ``` --- ### Macro properties You can declare macro properties and configs in `.yml` files in your project. Macro properties are "special properties" in that you can't configure them in the `dbt_project.yml` file or using `config()` blocks. Refer to [Configs and properties](https://docs.getdbt.com/reference/define-properties#which-properties-are-not-also-configs) for more info. Macros support a `config` block. You can define `meta` and `docs` within `config`. You can name these files `whatever_you_want.yml` and nest them arbitrarily deep in sub-folders. macros/\<filename>.yml ```yml macros: - name: <macro name> description: <markdown_string> config: docs: show: true | false meta: {<dictionary>} arguments: - name: <arg name> type: <string> description: <markdown_string> - ... # declare properties of additional arguments - name: ... # declare properties of additional macros ``` #### Example macros/schema.yml ```yaml macros: - name: cents_to_dollars description: Converts a numeric column from cents to dollars. config: docs: show: true meta: owner: analytics arguments: - name: column_name type: column description: The column to convert - name: precision type: integer description: Number of decimal places. Defaults to 2. ``` --- ### macro-paths dbt\_project.yml ```yml macro-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) are located. Note that you cannot co-locate models and macros. #### Default By default, dbt will search for macros in a directory named `macros`. For example, `macro-paths: ["macros"]`. Paths specified in `macro-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/macros`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml macro-paths: ["macros"] ``` * ❌ **Don't:** * Avoid absolute paths: ```yml macro-paths: ["/Users/username/project/macros"] ``` #### Examples ##### Use a subdirectory named `custom_macros` instead of `macros` dbt\_project.yml ```yml macro-paths: ["custom_macros"] ``` --- ### Manifest JSON file | dbt version | Manifest version | | ----------- | ------------------------------------------------------------- | | dbt v2.0 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.11 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.10 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.9 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.8 | [v12](https://schemas.getdbt.com/dbt/manifest/v12/index.html) | | dbt v1.7 | [v11](https://schemas.getdbt.com/dbt/manifest/v11/index.html) | | dbt v1.6 | [v10](https://schemas.getdbt.com/dbt/manifest/v10/index.html) | | dbt v1.5 | [v9](https://schemas.getdbt.com/dbt/manifest/v9/index.html) | | dbt v1.4 | [v8](https://schemas.getdbt.com/dbt/manifest/v8/index.html) | | dbt v1.3 | [v7](https://schemas.getdbt.com/dbt/manifest/v7/index.html) | <br /> **Produced by**: Any [dbt command](https://docs.getdbt.com/category/list-of-commands.md) that parses the project. This includes all commands, *except* [`deps`](https://docs.getdbt.com/reference/commands/deps.md), [`clean`](https://docs.getdbt.com/reference/commands/clean.md), [`debug`](https://docs.getdbt.com/reference/commands/debug.md), and [`init`](https://docs.getdbt.com/reference/commands/init.md). After executing a dbt command, the `manifest.json` file can be found in the project's `target/` directory: * If developing locally: Open the `target/` directory in your project folder * In the Studio IDE: Open the `target/` directory in the file tree * In dbt platform jobs: Download the `manifest.json` from the [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) tab for a given job run This file contains a full representation of your dbt project's resources (models, tests, macros, and more), including all node configurations and resource properties. Even if you're only running some models or tests, all resources will appear in the manifest (unless they are disabled) with most of their properties. Some properties, such as `compiled_sql`, are included only for executed nodes. Today, dbt uses this file to populate the [docs site](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md), and to perform [state comparison](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). Members of the community also use it to analyze project health, such as checking for missing descriptions or tests. ##### Top-level keys * [`metadata`](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md#common-metadata) * `nodes`: Dictionary of all analyses, models, seeds, snapshots, and tests. * `sources`: Dictionary of sources * `metrics`: Dictionary of metrics * `exposures`: Dictionary of exposures * `groups`: Dictionary of groups (**Note:** Added in v1.5) * `macros`: Dictionary of macros * `docs`: Dictionary of `docs` blocks * `parent_map`: Dictionary that contains the first-order parents of each resource * `child_map`: Dictionary that contains the first-order children of each resource * `group_map`: Dictionary that maps group names to their resource nodes * `selectors`: Expanded dictionary representation of [YAML `selectors`](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) * `disabled`: Array of resources with `enabled: false` ##### Resource details All resources nested within `nodes`, `sources`, `metrics`, `exposures`, `macros`, and `docs` have the following base properties: * `name`: Resource name * `unique_id`: `<resource_type>.<package>.<resource_name>`, same as dictionary key * `package_name`: Name of package that defines this resource * `root_path`: Absolute file path of this resource's package. (**Note:** This was removed for most node types in dbt v1.4 / manifest v8 to reduce duplicative information across nodes, but it is still present for seeds.) * `path`: Relative file path of this resource's definition within its "resource path" (`model-paths`, `seed-paths`, and more). * `original_file_path`: Relative file path of this resource's definition, including its resource path. Each has several additional properties related to its resource type. ##### dbt JSON schema You can refer to the [dbt JSON schema](https://schemas.getdbt.com/) for information on describing and consuming dbt-generated artifacts. **Note**: The `manifest.json` version number is related to (but not *equal* to) your dbt version, so you *must* use the correct `manifest.json` version for your dbt version. To find the correct `manifest.json` version, select the dbt version on the top navigation (such as `v1.5`). Refer to the table at the beginning of [this page](https://docs.getdbt.com/reference/artifacts/manifest-json.md) to understand how the manifest version matches the dbt version. --- ### Materialize configurations #### Performance optimizations ##### Clusters Enable the configuration of [clusters](https://github.com/MaterializeInc/materialize/blob/main/misc/dbt-materialize/CHANGELOG.md#120---2022-08-31). The default [cluster](https://materialize.com/docs/overview/key-concepts/#clusters) that is used to maintain materialized views or indexes can be configured in your [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) using the `cluster` connection parameter. To override the cluster that is used for specific models (or groups of models), use the `cluster` configuration parameter. my\_view\_cluster.sql ```sql {{ config(materialized='materializedview', cluster='not_default') }} select ... ``` dbt\_project.yml ```yaml models: project_name: +materialized: materializedview +cluster: not_default ``` ##### Incremental models: Materialized Views Materialize, at its core, is a real-time database that delivers incremental view updates without ever compromising on latency or correctness. Use [materialized views](https://materialize.com/docs/overview/key-concepts/#materialized-views) to compute and incrementally update the results of your query. ##### Indexes Enable additional configuration for [indexes](https://github.com/MaterializeInc/materialize/blob/main/misc/dbt-materialize/CHANGELOG.md#120---2022-08-31). Like in any standard relational database, you can use [indexes](https://materialize.com/docs/overview/key-concepts/#indexes) to optimize query performance in Materialize. Improvements can be significant, reducing response times down to single-digit milliseconds. Materialized views (`materializedview`), views (`view`) and sources (`source`) may have a list of `indexes` defined. Each [Materialize index](https://materialize.com/docs/sql/create-index/) can have the following components: * `columns` (list, required): one or more columns on which the index is defined. To create an index that uses *all* columns, use the `default` component instead. * `name` (string, optional): the name for the index. If unspecified, Materialize will use the materialization name and column names provided. * `cluster` (string, optional): the cluster to use to create the index. If unspecified, indexes will be created in the cluster used to create the materialization. * `default` (bool, optional): Default: `False`. If set to `True`, creates a default index that uses all columns. my\_view\_index.sql ```sql {{ config(materialized='view', indexes=[{'columns': ['col_a'], 'cluster': 'cluster_a'}]) }} indexes=[{'columns': ['symbol']}]) }} select ... ``` my\_view\_default\_index.sql ```sql {{ config(materialized='view', indexes=[{'default': True}]) }} select ... ``` ##### Data tests If you set the optional `--store-failures` flag or [`store_failures` config](https://docs.getdbt.com/reference/resource-configs/store_failures.md), dbt will create a materialized view for each configured test that can keep track of failures over time. By default, test views are created in a schema suffixed with `dbt_test__audit`. To specify a custom suffix, use the `schema` config. dbt\_project.yml ```yaml data_tests: project_name: +store_failures: true +schema: test ``` --- ### materialized ##### Project YAML file dbt\_project.yml ```yaml config-version: 2 models: <resource-path>: +materialized: <materialization_name> ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: <model_name> config: materialized: <materialization_name> ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized="<materialization_name>" ) }} select ... ``` #### Definition [Materializations](https://docs.getdbt.com/docs/build/materializations.md#materializations) are strategies for persisting dbt models in a warehouse. These are the materialization types built into dbt: * `ephemeral` — [ephemeral](https://docs.getdbt.com/docs/build/materializations.md#ephemeral) models are not directly built into the database * `table` — a model is rebuilt as a [table](https://docs.getdbt.com/docs/build/materializations.md#table) on each run * `view` — a model is rebuilt as a [view](https://docs.getdbt.com/docs/build/materializations.md#view) on each run * `materialized_view` — allows the creation and maintenance of [materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) in the target database * `incremental` — [incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) models allow dbt to insert or update records into a `table` since the last time that model was run You can also configure [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md?step=1) in dbt. Custom materializations are a powerful way to extend dbt's functionality to meet your specific needs. #### Creation precedence Materializations are implemented following this "drop through" life cycle: 1. If a model does not exist with the provided path, create the new model. 2. If a model exists, but has a different type, drop the existing model and create the new model. 3. If [`--full-refresh`](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) is supplied, replace the existing model regardless of configuration changes and the [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) setting. * BigQuery users may need to run `dbt run --full-refresh` (instead of `dbt run`) after changing a model’s materialization (for example, from `table` to `view`) to ensure dbt fully replaces the existing relation and the change is fully applied. 4. If there are no configuration changes, perform the default action for that type (e.g. apply refresh for a materialized view). 5. Determine whether to apply the configuration changes according to the `on_configuration_change` setting. --- ### meta ##### Models dbt\_project.yml ```yml models: <resource-path>: +meta: {<dictionary>} ``` models/schema.yml ```yml models: - name: model_name config: meta: {<dictionary>} columns: - name: column_name config: meta: {<dictionary>} # changed to config in v1.10 and backported to 1.9 ``` The `meta` config can be defined: * Under the `models` config in the project file (shown in previous 'models/schema.yml' example) * Under the `models` config in the project file (`dbt_project.yml`) * in a `config()` Jinja macro within a model's SQL file See [configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for details. ##### Sources dbt\_project.yml ```yml sources: <resource-path>: +meta: {<dictionary>} ``` models/schema.yml ```yml sources: - name: model_name config: meta: {<dictionary>} tables: - name: table_name config: meta: {<dictionary>} columns: - name: column_name config: meta: {<dictionary>} # changed to config in v1.10 and backported to 1.9 ``` ##### Seeds dbt\_project.yml ```yml seeds: <resource-path>: +meta: {<dictionary>} ``` seeds/schema.yml ```yml seeds: - name: seed_name config: meta: {<dictionary>} columns: - name: column_name config: meta: {<dictionary>} # changed to config in v1.10 and backported to 1.9 ``` The `meta` config can be defined: * Under the `seeds` config in the property file (shown in previous 'seeds/schema.yml' example) * Under the `seeds` config in the project file (`dbt_project.yml`). See [configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for details. ##### Snapshots dbt\_project.yml ```yml snapshots: <resource-path>: +meta: {<dictionary>} ``` snapshots/schema.yml ```yml snapshots: - name: snapshot_name config: meta: {<dictionary>} columns: - name: column_name config: meta: {<dictionary>} # changed to config in v1.10 and backported to 1.9 ``` The `meta` config can be defined: * under the `snapshots` config in the properties file (shown in previous `snapshots/schema.yml` example) * under the `snapshots` config in the project file (`dbt_project.yml`) * in a `config()` Jinja macro within a snapshot's SQL block See [configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for details. ##### Tests Use the `meta` field to add metadata to [generic](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests) or [singular tests](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests). `meta` accepts key-value pairs, is compiled into `manifest.json`, and appears in auto-generated documentation. **Generic data tests** Add `meta` under the `config` block in your `properties.yml` file: models/properties.yml ```yaml models: - name: my_model columns: - name: my_column data_tests: - unique: config: meta: owner: "docs team" ``` Or set defaults in `dbt_project.yml`: dbt\_project.yml ```yaml data_tests: my_project: +meta: owner: "docs team" ``` **Singular data tests** Add `meta` in the SQL test file using `config()`: tests/my\_singular\_test.sql ```sql {{ config(meta={'owner': 'docs team'}) }} select * from {{ ref('my_model') }} where my_column is null ``` Or document in `tests/properties.yml`: tests/properties.yml ```yaml data_tests: - name: my_singular_test config: meta: owner: "analytics_team" ``` ##### Unit tests Did you know... Available from dbt v1.8 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). dbt\_project.yml ```yml unit_tests: <resource-path>: +meta: {<dictionary>} ``` models/\<filename>.yml ```yml unit_tests: - name: <test-name> config: meta: {<dictionary>} ``` ##### Analyses The `meta` config is not currently supported for analyses. ##### Macros dbt\_project.yml ```yml macros: <resource-path>: +meta: {<dictionary>} ``` macros/schema.yml ```yml macros: - name: macro_name config: meta: {<dictionary>} # changed to config in v1.11 arguments: - name: argument_name ``` ##### Exposures dbt\_project.yml ```yml exposures: <resource-path>: +meta: {<dictionary>} ``` models/exposures.yml ```yml exposures: - name: exposure_name config: meta: {<dictionary>} # changed to config in v1.10 ``` ##### Semantic models (Applies to dbt v1.12 and later) Configure `meta` in the [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) embedded within your model YAML file or under the `semantic-models` config block in the `dbt_project.yml` file. dbt\_project.yml ```yml semantic-models: <resource-path>: +meta: {<dictionary>} ``` models/file\_name.yml ```yml models: - name: model_name semantic_model: enabled: true config: meta: {<dictionary>} ``` (Applies to dbt v1.12 and later) [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md), [entities](https://docs.getdbt.com/docs/build/entities.md), and metrics can also have their own `meta` configurations. models/file\_name.yml ```yml models: - name: model_name semantic_model: enabled: true config: meta: {<dictionary>} agg_time_dimension: your_time_dimension_name columns: - name: entity_column_name entity: type: primary name: entity_name config: meta: {<dictionary>} - name: dimension_column_name dimension: type: categorical name: dimension_name config: meta: {<dictionary>} metrics: - name: simple_metric_name description: "Description of the metric" type: simple agg: sum expr: column_name config: meta: {<dictionary>} ``` The `meta` config can be defined: * Under the `semantic-models` config in the properties file (as showing in previous `models/semantic_models.yml` example) * Under the `semantic-models` config in the project file (`dbt_project.yml`). See [configs and properties](https://docs.getdbt.com/reference/configs-and-properties.md) for details. ##### Metrics dbt\_project.yml ```yml metrics: <resource-path>: +meta: {<dictionary>} ``` (Applies to dbt v1.12 and later) models/file\_name.yml ```yml models: - name: model_name semantic_model: enabled: true agg_time_dimension: your_time_dimension columns: - name: column_name dimension: type: time granularity: day metrics: - name: number_of_people type: simple description: Total count of people agg: count expr: people config: meta: my_meta_config: 'config_value' ``` ##### Saved queries dbt\_project.yml ```yml saved-queries: <resource-path>: +meta: {<dictionary>} ``` models/semantic\_models.yml ```yml saved_queries: - name: saved_query_name config: meta: {<dictionary>} ``` #### Definition The `meta` config sets metadata for a resource and accepts any key-value pairs. This metadata is compiled into the `manifest.json` file generated by dbt, and is visible in the auto-generated documentation. Depending on the resource you're configuring, `meta` may be available within the `config` property, and/or as a top-level key. (For backwards compatibility, `meta` is often (but not always) supported as a top-level key, though without the capabilities of config inheritance.) Effect on state comparison Changes to `meta`, including at the column level, don't trigger [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state). dbt treats `meta` (and `tags`) as metadata only, since it doesn't affect how a resource is materialized. Refer to [caveats to state comparison](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md#tags-and-meta) for more detail. #### Examples To demonstrate how to use the `meta` config, here are some examples: * [Designate a model owner](#designate-a-model-owner) * [Designate a source column as containing PII](#designate-a-source-column-as-containing-pii) * [Configure one meta attribute for all seeds](#configure-one-meta-attribute-for-all-seeds) * [Override one meta attribute for a single model](#override-one-meta-attribute-for-a-single-model) * [Assign owner and favorite\_color in the dbt\_project.yml as a config property](#assign-owner-and-favorite_color-in-the-dbt_projectyml-as-a-config-property) * [Assign meta to semantic model](#assign-meta-to-semantic-model) * [Assign meta to dimensions, measures, entities](#assign-meta-to-dimensions-measures-entities) * [Add meta to generic and singular data tests](#add-meta-to-generic-and-singular-data-tests) * [Access meta values in Python models](#access-meta-values-in-python-models) ##### Designate a model owner Additionally, indicate the maturity of a model using a `model_maturity:` key. models/schema.yml ```yml models: - name: users config: meta: owner: "@alice" model_maturity: in dev ``` ##### Designate a source column as containing PII models/schema.yml ```yml sources: - name: salesforce tables: - name: account config: meta: contains_pii: true columns: - name: email config: meta: # changed to config in v1.10 and backported to 1.9 contains_pii: true ``` ##### Configure one meta attribute for all seeds dbt\_project.yml ```yml seeds: +meta: favorite_color: red ``` ##### Override one meta attribute for a single model models/my\_model.sql ```sql {{ config(meta = { 'single_key': 'override' }) }} select 1 as id ``` <br /> ##### Assign owner and favorite\_color in the dbt\_project.yml as a config property dbt\_project.yml ```yml models: jaffle_shop: +meta: owner: "@alice" favorite_color: red ``` ##### Assign meta to semantic model (Applies to dbt v1.12 and later) The following example shows how to assign a `meta` value to a [semantic model](https://docs.getdbt.com/docs/build/semantic-models.md) in the model YAML file and `dbt_project.yml` file: ##### Semantic model ```yaml models: - name: fact_transactions description: "Transaction fact table at the transaction level. This table contains one row per transaction and includes the transaction timestamp." semantic_model: enabled: true name: transaction config: meta: data_owner: "Finance team" used_in_reporting: true agg_time_dimension: transaction_date ``` ##### dbt\_project.yml ```yaml semantic-models: jaffle_shop: +meta: used_in_reporting: true ``` ##### Assign meta to dimensions, measures, entities (Applies to dbt v1.12 and later) ##### Semantic model The following example shows how to assign a `meta` value to a [dimension](https://docs.getdbt.com/docs/build/dimensions.md), [entity](https://docs.getdbt.com/docs/build/entities.md), and [simple metrics](https://docs.getdbt.com/docs/build/simple.md) in a semantic model: model\_name.yml ```yml models: - name: model_name semantic_model: enabled: true name: semantic_model agg_time_dimension: order_date columns: - name: order_date dimension: type: time config: meta: data_owner: "Finance team" used_in_reporting: true - name: customer_id entity: type: primary config: meta: description: "Unique identifier for customers" data_owner: "Sales team" used_in_reporting: false metrics: - name: count_of_users type: simple agg: count_distinct expr: user_id config: meta: used_in_reporting: true ``` ##### dbt\_project.yml This second example shows how to assign a `data_owner` and additional metadata value to a dimension in the `dbt_project.yml` file using the `+meta` syntax. The similar syntax can be used for entities and simple metrics. dbt\_project.yml ```yml semantic-models: jaffle_shop: ... dimensions: - name: order_date config: meta: data_owner: "Finance team" used_in_reporting: true ``` ##### Add meta to generic and singular data tests The following examples show how to add `meta` to [generic data tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests) in a `properties.yml` file, and to [singular data tests](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests) using `config()`. You can also set defaults in `dbt_project.yml` or `tests/properties.yml`. ##### Generic data test models/properties.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - not_null: config: meta: owner: "@data_team" ``` ##### Singular data test tests/assert\_order\_ids.sql ```sql {{ config(meta={'owner': '@data_team'}) }} select * from {{ ref('orders') }} where order_id is null ``` ##### Access meta values in Python models To access custom `meta` values in [Python models](https://docs.getdbt.com/docs/build/python-models.md), use the `dbt.config.meta_get()` method. For example, if you have a model named `my_python_model` and you want to store custom values, you can do the following: models/schema.yml ```yml models: - name: my_python_model config: meta: batch_size: 1000 processing_mode: "incremental" ``` models/my\_python\_model.py ```python def model(dbt, session): # Access custom values stored in meta directly batch_size = dbt.config.meta_get("batch_size") processing_mode = dbt.config.meta_get("processing_mode") # Use the meta values in your model logic df = dbt.ref("upstream_model") if processing_mode == "incremental": df = df.limit(batch_size) return df ``` --- ### metadata_warehouse Snowflake only This configuration is available in Snowflake only. profiles.yml ```yaml my_project: outputs: prod: type: snowflake # ... other connection settings warehouse: TRANSFORMING metadata_warehouse: METADATA_XS target: prod ``` #### Definition dbt State performs metadata introspection queries to determine whether models need to be rebuilt. On Snowflake, these queries run against your configured `warehouse` by default, which can cause queuing when your primary warehouse is under heavy load. `metadata_warehouse` lets you route these queries to a separate, smaller warehouse to keep introspection overhead off your main compute resource. When set, dbt issues multiple, individual queries (one per schema) in parallel on the dedicated warehouse — faster than the single, consolidated query dbt runs by default when `metadata_warehouse` is not configured. If metadata queries on your main warehouse take longer than 15 seconds, dbt emits a warning suggesting you configure a dedicated warehouse. You can also use `metadata_warehouse` on the dbt platform by adding it as an [extended attribute](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) on the [connection profile](https://docs.getdbt.com/docs/platform/about-profiles.md) for a deployment environment. note This configuration currently applies only to dbt State metadata queries. It might be used more broadly in the future. See [dbt-labs/dbt#12122](https://github.com/dbt-labs/dbt/issues/12122) for the feature request. #### Default Falls back to the `warehouse` setting in `profiles.yml`. #### Example ##### Separate metadata and execution warehouses profiles.yml ```yaml my_project: outputs: prod: type: snowflake account: abc12345 database: ANALYTICS schema: DBT_PROD warehouse: TRANSFORMING # used for model execution metadata_warehouse: METADATA_XS # used for dbt State introspection target: prod ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Snowflake configuration](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md) --- ### Metric properties Metrics define measurable quantities that you can query through the Semantic Layer. You define them in different places, depending on your dbt version: * In a model using the *latest* YAML spec. Top-level `metrics:` list on a [model](https://docs.getdbt.com/reference/model-properties.md) that has semantic modeling enabled, alongside `semantic_model:` and `columns:`. Available in the dbt platform **v1 Latest** release track and dbt v2. * In the standalone *legacy* YAML spec. Refer to [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) for more information. Availability The latest YAML spec is supported in the following environments: * **dbt platform (v1 Latest release track)** * **dbt v2** * **dbt v1.12** For more information, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). (Applies to dbt v1.12 and later) #### Latest spec (model YAML) In the latest YAML spec, you can define metrics on a model that has semantic modeling enabled. Add a top-level `metrics` list alongside `semantic_model` and `columns` (metrics are not nested under `semantic_model`). Type-specific settings are top-level keys on each metric. ##### Available metric properties (latest spec) | Property | Type | Required | Description | | ----------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | name | string | Yes | Unique metric name. Use lowercase letters, numbers, and underscores. | | type | string | Yes | One of: `simple`, `cumulative`, `ratio`, `derived`, `conversion`. | | description | string | No | Documentation for the metric. | | label | string | No | Display name in downstream tools. | | filter | string | No | MetricFlow filter expression (dimensions, entities, or other metrics). | | config | object | No | Supports [meta](https://docs.getdbt.com/reference/resource-configs/meta.md), [group](https://docs.getdbt.com/reference/resource-configs/group.md), [tags](https://docs.getdbt.com/reference/resource-configs/tags.md), [enabled](https://docs.getdbt.com/reference/resource-configs/enabled.md). | ##### Properties by metric type (latest spec) | Metric type | Key properties | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | All | name, type, description, label, hidden, filter, config | | [Simple](https://docs.getdbt.com/docs/build/simple.md) | agg, expr, time\_granularity, agg\_time\_dimension, join\_to\_timespine, fill\_nulls\_with; optionally non\_additive\_dimension | | [Derived](https://docs.getdbt.com/docs/build/derived.md) | expr, input\_metrics (each with optional alias, filter, offset\_window) | | [Ratio](https://docs.getdbt.com/docs/build/ratio.md) | numerator, denominator (each a metric name or a dict with name, filter, alias) | | [Conversion](https://docs.getdbt.com/docs/build/conversion.md) | entity, calculation, base\_metric, conversion\_metric, window; optional constant\_properties | | [Cumulative](https://docs.getdbt.com/docs/build/cumulative.md) | input\_metric, window, grain\_to\_date, period\_agg | **Cross-model metrics:** Metrics under a model's `metrics:` list can only reference that semantic model. Metrics that depend on other semantic models (for example, cross-model cumulative, ratio, derived, or conversion) go in a top-level `metrics:` block (outside `models:`). This can live in the same YAML file or a separate file. **Note:** For the legacy spec, all metrics were defined in standalone YAML; there was no model-level `metrics:` list. For the latest spec, refer to [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md). For metric types, `type_params`, and more examples, refer to [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). --- ### MetricFlow time spine YAML Removed in v2 This flag was removed in v2. The new behavior is always enabled. If you're upgrading, remove this flag from your `dbt_project.yml`. | require\_yaml\_configuration\_for\_mf\_time\_spines | dbt **v1 Latest** | dbt v1 | | --------------------------------------------------- | ----------------- | ------ | | Introduced | 2024.10 | 1.9.0 | | Matured (default → `true`) | 2026.09 | 1.12.0 | | Removed | — | v2.0 | <br /> In previous versions (dbt 1.8 and earlier), the MetricFlow time spine configuration was stored in a `metricflow_time_spine.sql` file. Starting in dbt v1.12, this flag defaults to `true`, suppressing the `MFTimespineWithoutYamlConfigurationDeprecation` deprecation warning. The legacy SQL file configuration continues to work in both cases. The MetricFlow properties YAML file should have the `time_spine:` field. Refer to [MetricFlow timespine](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) for more details. #### Impact This flag has no functional impact; the legacy time-spine model continues to work in both cases. The only visible changes are: * The `MFTimespineWithoutYamlConfigurationDeprecation` warning no longer appears in logs. * If you use `--warn-error`, the warning no longer fires and will no longer escalate to an error.  Opting out of this flag No action is required for most projects. The legacy `metricflow_time_spine.sql` model continues to work with or without this flag. If you rely on the `MFTimespineWithoutYamlConfigurationDeprecation` warning firing under `--warn-error` to enforce a YAML migration, you can opt out by setting the flag to `false` in `dbt_project.yml`: dbt\_project.yml ```yaml flags: require_yaml_configuration_for_mf_time_spines: false ``` To remove the deprecation warning permanently, migrate `metricflow_time_spine.sql` to a YAML `time_spine` block under a model entry in `models:`. Refer to [MetricFlow timespine](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) for the current syntax. --- ### Microsoft Azure Synapse DWH configurations All [configuration options for the Microsoft SQL Server adapter](https://docs.getdbt.com/reference/resource-configs/mssql-configs.md) also apply to this adapter. Additionally, the configuration options below are available. ##### Indices and distributions The main index and the distribution type can be set for models that are materialized to tables. ##### Model config models/example.sql ```sql {{ config( index='HEAP', dist='ROUND_ROBIN' ) }} select * from ... ``` ##### Project config dbt\_project.yml ```yaml models: your_project_name: materialized: view staging: materialized: table index: HEAP ``` The following are the supported index types: * `CLUSTERED COLUMNSTORE INDEX` (default) * `HEAP` * `CLUSTERED INDEX (COLUMN_NAME)` * `CLUSTERED COLUMNSTORE INDEX ORDER(COLUMN_NAME)` The following are the supported distribution types: * `ROUND_ROBIN` (default) * `HASH(COLUMN_NAME)` * `REPLICATE` --- ### Microsoft Fabric Data Warehouse configurations This page describes configuration options specific to the `dbt-fabric` adapter for Microsoft Fabric Data Warehouse. It outlines supported materializations, incremental strategies (including [merge](#merge) and [microbatch](#microbatch)), [model contracts](#model-contracts), cross-warehouse references, warehouse snapshots, and profile setup. #### Materializations Ephemeral materialization is not supported due to T-SQL not supporting nested CTEs. It may work in some cases when you're working with very simple ephemeral models. ##### Tables Tables are the default materialization in dbt-fabric. When you configure a model as a table, dbt will create or replace the table in Fabric Data Warehouse on each run. ##### Model config models/example.sql ```sql {{ config( materialized='table' ) }} select * from ... ``` ##### Project config dbt\_project.yml ```yaml models: your_project_name: materialized: view staging: materialized: table ``` > **Limitation:** Nested CTE aren't supported in model materialization. Models using multiple nested CTEs may fail during compilation or execution. #### Table Clone The `table_clone` materialization creates a physical copy of an existing table using Fabric’s cloning capabilities. This is useful for versioning, branching, or snapshot-like workflows. ```sql {{ config(materialized='table_clone', clone_from='staging_table') }} select * from staging_table ``` **Notes:** * The source table must exist in the target warehouse. * Cloning preserves the schema and data state at the time of creation. * Ideal for scenarios requiring fast, zero-copy duplication for testing or rollback. #### Seeds By default, `dbt-fabric` will attempt to insert seed files in batches of 400 rows. If this exceeds Microsoft Fabric Data Warehouse 2100 parameter limit, the adapter will automatically limit to the highest safe value possible. To set a different default seed value, you can set the variable `max_batch_size` in your project configuration. dbt\_project.yml ```yaml vars: max_batch_size: 200 # Any integer less than or equal to 2100 will do. ``` #### Views You can create views using the `view` materialization: ```sql {{ config(materialized='view') }} select * from source_data ``` You can set this globally as well: ```yaml models: my_project: +materialized: view ``` > **Limitation:** Nested CTEs (Common Table Expressions) are not supported in model materialization. Models using multiple nested CTEs may fail during compilation or execution. #### Model contracts The `dbt-fabric` adapter supports [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) and [constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md) on `table` and `incremental` models. | Constraint type | Supported | Checked when the model builds | | --------------- | --------- | ----------------------------- | | `not_null` | ✅ | ✅ | | `primary_key` | ✅ | ❌ | | `foreign_key` | ✅ | ❌ | | `unique` | ✅ | ❌ | | `check` | ❌ | ❌ | <br /> Only `not_null` is checked when the model builds. Other supported constraint types are recorded on the table for reference, but Microsoft Fabric does not check them when new data is added. Use [data tests](https://docs.getdbt.com/docs/build/data-tests.md) if you need to verify those rules. Define `primary_key`, `foreign_key`, and `unique` at the model level. The adapter ignores those constraint types at the column level. Microsoft Fabric does not include all constraints in the initial `create table` statement. dbt creates the table first, then runs separate statements to add model-level constraints. You must provide a `name` for each model-level constraint. models/dim\_customers.yml ```yaml models: - name: dim_customers config: contract: enforced: true constraints: - type: primary_key columns: [id] name: pk_dim_customers columns: - name: id data_type: int constraints: - type: not_null - name: customer_name data_type: varchar(100) - name: first_transaction_date data_type: date ``` For more information, refer to [constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md) and [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md). #### Snapshots Columns in source tables can not have any constraints. If, for example, any column has a `NOT NULL` constraint, an error will be thrown. #### Indexes Indexes are not supported by Microsoft Fabric Data Warehouse. Any Indexes provided as a configuration is ignored by the adapter. #### Grants with auto provisioning Grants with auto provisioning is not supported by Microsoft Fabric Data Warehouse at this time. #### Incremental models Incremental materializations are supported with multiple strategies. In **dbt-fabric**, the **default strategy is `merge`**, introduced in v1.9.7. Other supported strategies include `append`, `delete+insert`, and `microbatch`. ##### Merge (default) The `merge` strategy automatically updates existing records and inserts new ones based on the configured `unique_key`. ```sql {{ config( materialized='incremental', unique_key='id' ) }} select * from source_table {% if is_incremental() %} where updated_at > (select max(updated_at) from {{ this }}) {% endif %} ``` ##### Append Appends new records to the existing dataset. ```sql {{ config( materialized='incremental', incremental_strategy='append' ) }} select * from new_data ``` ##### Delete+Insert Deletes and re-inserts based on `unique_key`. ```sql {{ config( materialized='incremental', incremental_strategy='delete+insert', unique_key='id' ) }} select * from updated_data ``` ##### Microbatch The `microbatch` strategy processes data in bounded time intervals using an event timestamp column. ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', event_time='event_timestamp', batch_size='1 day' ) }} select * from raw_events ``` ###### Notes * [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) must be a valid timestamp column. * dbt processes each batch independently, allowing efficient incremental refresh of large time-series datasets. * If you don't specify a `unique_key`, dbt-fabric defaults to `append`. For more details, see [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md). #### Permissions The Microsoft Entra identity (user or service principal) must be a Fabric Workspace admin to work on the database level at this time. Fine grain access control will be incorporated in the future. #### Cross-warehouse references The dbt-fabric adapter supports cross-warehouse queries using `source()` or `ref()` macros. ```sql select * from {{ source('sales_dw', 'transactions') }} union all select * from {{ ref('customer_dim') }} ``` Ensure that the corresponding model or source definitions specify the correct `database:` parameter to reference another Fabric Warehouse. Example `sources.yml`: ```yaml sources: - name: sales_dw database: saleswarehouse schema: sales tables: - name: transactions ``` > To use cross-warehouse references or warehouse snapshots, ensure the identity configured here has access to all referenced Fabric Warehouses. #### Warehouse snapshots Microsoft Fabric warehouse snapshots are read-only copies of your warehouse at a specific moment, kept for up to 30 days. They allow analysts query a stable dataset, even while ELT processes are updating the warehouse. By moving the snapshot’s timestamp forward, changes are applied all at once (atomically). dbt-fabric supports warehouse snapshots, which helps track changes in Fabric Data Warehouse objects between dbt runs. Fabric automatically creates snapshots *before* and *after* you run the `dbt run`, `dbt build`, or `dbt snapshot` commands. To use them, your `profiles.yml` must include the `workspace_id` and the warehouse snapshot name so dbt can create the snapshot as a child item of your warehouse. Learn more [here](https://learn.microsoft.com/en-us/fabric/data-warehouse/warehouse-snapshot) ```yaml fabric_dw: target: dev outputs: dev: type: fabric server: "<your-fabric-server-name>" database: "<your-warehouse-name>" schema: "<default-schema>" authentication: CLI workspace_id: e4487eff-d67d-4b58-917c-ffbb61a5c05f warehouse_snapshot_name: dbt-dwtests-snpshot ### Behavior - Before a dbt operation (`run`, `build`, `snapshot`), the adapter captures the pre-state of affected tables. - After execution, the warehouse snapshot is created with snapshot timestamp. For additional details: - dbt snapshot documentation - Fabric adapter snapshots reference ## dbt-utils Not supported at this time. However, dbt-fabric offers some dbt-utils macros. Please check out the tsql-utils package. ``` --- ### Microsoft Fabric Spark configurations #### Configuring tables When materializing a model as `table`, you may include several optional configs that are specific to the dbt-spark plugin, in addition to the standard [model configs](https://docs.getdbt.com/reference/model-configs.md). | Option | Description | Required? | Example | | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | file\_format | The file format to use when creating tables (`parquet`, `delta`, `csv`). | Optional | `delta` | | location\_root [1](#user-content-fn-1) | The specified directory used to store table data. The table alias is appended to it. | Optional | `Files/<folder>` or `Tables/<tableName>` | | partition\_by | Partition the table by the specified columns. A directory is created for each partition. | Optional | `date_day` | | clustered\_by | Each partition in the table will be split into a fixed number of buckets by the specified columns. | Optional | `country_code` | | buckets | The number of buckets to create while clustering | Required if `clustered_by` is specified | `8` | | tblproperties | The table properties configure table behavior. Properties differ depending on the file format, see reference docs ([Parquet](https://spark.apache.org/docs/3.5.4/sql-data-sources-parquet.html#data-source-option), [Delta](https://docs.delta.io/latest/table-properties.html)). | Optional | `Provider=delta Location=abfss://.../Files/tables/sales_data TableProperty.created.by=data_engineering_team TableProperty.purpose=sales analytics CreatedBy=Delta Lake CreatedAt=2024-12-01 14:21:00 Format=Parquet PartitionColumns=region MinReaderVersion=1 MinWriterVersion=2` | #### Incremental models dbt seeks to offer useful, intuitive modeling abstractions by means of its built-in configurations and materializations. Because there is so much variance between Spark clusters out in the world—not to mention the powerful features offered to open source users by the Delta file format and custom runtime—making sense of all the available options is an undertaking in its own right. For that reason, the dbt-fabricspark plugin leans heavily on the [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-strategy.md). This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of three values: * **`append`** (default): Insert new records without updating or overwriting any existing data. * **`insert_overwrite`**: If `partition_by` is specified, overwrite partitions in the table with new data. If no `partition_by` is specified, overwrite the entire table with new data. * **`merge`** (Delta file format only): Match records based on a `unique_key`; update old records, insert new ones. (If no `unique_key` is specified, all new data is inserted, similar to `append`.) * `microbatch` Implements the [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) using `event_time` to define time-based ranges for filtering data. Each of these strategies has its pros and cons, which we'll discuss below. As with any model config, `incremental_strategy` may be specified in `dbt_project.yml` or within a model file's `config()` block. ##### The `append` strategy Following the `append` strategy, dbt will perform an `insert into` statement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Fabric Spark versions. However, this strategy *cannot* update, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources. Specifying `append` as the incremental strategy is optional, since it's the default strategy used when none is specified. ##### Source code fabricspark\_incremental.sql ```sql {{ config( materialized='incremental', incremental_strategy='append', ) }} -- All rows returned by this query will be appended to the existing table select * from {{ ref('events') }} {% if is_incremental() %} where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ``` ##### Run code fabricspark\_incremental.sql ```sql create temporary view fabricspark_incremental__dbt_tmp as select * from analytics.events where event_ts >= (select max(event_ts) from {{ this }}) ; insert into table analytics.fabricspark_incremental select `date_day`, `users` from spark_incremental__dbt_tmp ``` ##### The `insert_overwrite` strategy This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select *all* of the relevant data for a partition when using this incremental strategy. If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead. **Usage notes:** * This strategy is not supported for tables with `file_format: delta`. ##### Source code fabricspark\_incremental.sql ```sql {{ config( materialized='incremental', partition_by=['date_day'], file_format='parquet' ) }} /* Every partition returned by this query will be overwritten when this model runs */ with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select date_day, count(*) as users from events group by 1 ``` ##### Run code fabricspark\_incremental.sql ```sql create temporary view fabricspark_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select date_day, count(*) as users from events group by 1 ; insert overwrite table analytics.fabricspark_incremental partition (date_day) select `date_day`, `users` from spark_incremental__dbt_tmp ``` ##### The `merge` strategy **Usage notes:** The `merge` incremental strategy requires: * `file_format: delta` * Fabric Spark Runtime 3.0 and above for delta file format dbt will run an atomic `merge` statement which looks nearly identical to the default merge behavior on Fabric Warehouse or SQL database or Snowflake and BigQuery. If a `unique_key` is specified (recommended), dbt will update old records with values from new records that match on the key column. If a `unique_key` is not specified, dbt will forgo match criteria and simply insert all new records (similar to `append` strategy). ##### Source code merge\_incremental.sql ```sql {{ config( materialized='incremental', file_format='delta', unique_key='user_id', incremental_strategy='merge' ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from events group by 1 ``` ##### Run code target/run/merge\_incremental.sql ```sql create temporary view merge_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select user_id, max(date_day) as last_seen from events group by 1 ; merge into analytics.merge_incremental as DBT_INTERNAL_DEST using merge_incremental__dbt_tmp as DBT_INTERNAL_SOURCE on DBT_INTERNAL_SOURCE.user_id = DBT_INTERNAL_DEST.user_id when matched then update set * when not matched then insert * ``` #### Persisting model descriptions Relation-level docs persistence is supported in dbt. For more information on configuring docs persistence, see [the docs](https://docs.getdbt.com/reference/resource-configs/persist_docs.md). When the `persist_docs` option is configured appropriately, you'll be able to see model descriptions in the `Comment` field of `describe [table] extended` or `show table extended in [database] like '*'`. #### Always `schema`, never `database` Fabric Spark uses the terms "schema" and "database" interchangeably. dbt understands `database` to exist at a higher level than `schema`. As such, you should *never* use or set `database` as a node config or in the target profile when running dbt-fabricspark. Move over, the adapter does not support schemas within Lakehouse. #### Default file format configurations To access advanced incremental strategies features, such as [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) and the `merge` incremental strategy, you will want to use the Delta file format as the default file format when materializing models as tables. It's quite convenient to do this by setting a top-level configuration in your project file: dbt\_project.yml ```yml models: +file_format: delta seeds: +file_format: delta snapshots: +file_format: delta ``` #### Footnotes 1. If you configure `location_root`, dbt specifies a location path in the `create table` statement. This changes the table from "managed" to "external" in Fabric Lakehouse. [↩](#user-content-fnref-1) --- ### Microsoft SQL Server configurations #### Materializations Ephemeral materialization is not supported due to T-SQL not supporting nested CTEs. It may work in some cases when you're working with very simple ephemeral models. ##### Tables Tables will, by default, be materialized as a columnstore tables. This requires SQL Server 2017 or newer for on-premise instances or service tier S2 or higher for Azure. This behaviour can be disabled by setting the `as_columnstore` configuration option to `False`. ##### Model config models/example.sql ```sql {{ config( as_columnstore=false ) }} select * from ... ``` ##### Project config dbt\_project.yml ```yaml models: your_project_name: materialized: view staging: materialized: table as_columnstore: False ``` #### Seeds By default, `dbt-sqlserver` will attempt to insert seed files in batches of 400 rows. If this exceeds SQL Server's 2100 parameter limit, the adapter will automatically limit to the highest safe value possible. To set a different default seed value, you can set the variable `max_batch_size` in your project configuration. dbt\_project.yml ```yaml vars: max_batch_size: 200 # Any integer less than or equal to 2100 will do. ``` #### Snapshots Columns in source tables can not have any constraints. If, for example, any column has a `NOT NULL` constraint, an error will be thrown. #### Indices You can specify indices to be created for your table by specifying post-hooks calling purpose-built macros. The following macros are available: * `create_clustered_index(columns, unique=False)`: columns is a list of columns, unique is an optional boolean (defaults to False). * `create_nonclustered_index(columns, includes=columns)`: columns is a list of columns, includes is an optional list of columns to include in the index. * `drop_all_indexes_on_table()`: drops current indices on a table. Only meaningful if the model is incremental.\` Some examples: models/example.sql ```sql {{ config({ "as_columnstore": false, "materialized": 'table', "post-hook": [ "{{ create_clustered_index(columns = ['row_id', 'row_id_complement'], unique=True) }}", "{{ create_nonclustered_index(columns = ['modified_date']) }}", "{{ create_nonclustered_index(columns = ['row_id'], includes = ['modified_date']) }}", ] }) }} select * from ... ``` #### Grants with auto provisioning dbt 1.2 introduced the capability to grant/revoke access using the `grants` [configuration option](https://docs.getdbt.com/reference/resource-configs/grants.md). In dbt-sqlserver, you can additionally set `auto_provision_aad_principals` to `true` in your model configuration if you are using Microsoft Entra ID authentication with an Azure SQL Database or Azure Synapse Dedicated SQL Pool. This will automatically create the Microsoft Entra ID principal inside your database if it does not exist yet. Note that the principals need to exist in your Microsoft Entra ID, this just makes them available to use in your database. Principals are not removed again when they are removed from the grants configuration. dbt\_project.yml ```yaml models: your_project_name: auto_provision_aad_principals: true ``` #### Permissions The following permissions are required for the user executing dbt: * `CREATE SCHEMA` on the database level (or you can create the schema in advance) * `CREATE TABLE` on the database level (or on the user's own schema if the schema is already created) * `CREATE VIEW` on the database level (or on the user's own schema if the schema is already created * `SELECT` on the tables/views being used as dbt sources The 3 `CREATE` permissions above are required on the database level if you want to make use of tests or snapshots in dbt. You can work around this by creating the schemas used for testing and snapshots in advance and granting the right roles. #### cross-database macros The following macros are currently not supported: * `bool_or` * `array_construct` * `array_concat` * `array_append` #### dbt-utils Many [`dbt-utils`](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) are supported, but require the installation of the [`tsql_utils`](https://hub.getdbt.com/dbt-msft/tsql_utils/latest/) dbt package. --- ### MindsDB configurations #### Authentication To successfully connect dbt to MindsDB, you will need to provide the following configuration from the MindsDB instance. | Key | Required | Description | Self-hosted | MindsDB Cloud | | -------- | -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | type | ✔️ | The specific adapter to use | `mindsdb` | `mindsdb` | | host | ✔️ | The MindsDB (hostname) to connect to | Default to `172.0.0.1` | Default to `cloud.mindsdb.com` | | port | ✔️ | The port to use | Default to `47335` | Default to `3306` | | schema | ✔️ | Specify the schema (database) to build models into | The MindsDB [integration name](https://docs.mindsdb.com/sql/create/databases/) | The MindsDB [integration name](https://docs.mindsdb.com/sql/create/databases/) | | username | ✔️ | The username to use to connect to the server | Default to `mindsdb` | Your mindsdb cloud username | | password | ✔️ | The password to use for authenticating to the server | No password by default | Your mindsdb cloud password | #### Usage Create dbt project, choose mindsdb as the database and set up the connection. Verify your connection works `dbt debug` `dbt init <project_name>` To create a predictor, create a dbt model with a "predictor" materialization. The name of the model will be the name of predictor. ###### Parameters: * `integration` - name of used integration to get data from and save result to. Must be created in mindsdb beforehand using the [`CREATE DATABASE` syntax](https://docs.mindsdb.com/sql/create/databases/). * `predict` - field for prediction * `predict_alias` \[optional] - alias for predicted field * `using` \[optional] - options for configure trained model ```sql -- my_first_model.sql {{ config( materialized='predictor', integration='photorep', predict='name', predict_alias='name', using={ 'encoders.location.module': 'CategoricalAutoEncoder', 'encoders.rental_price.module': 'NumericEncoder' } ) }} select * from stores ``` To apply predictor add dbt model with "table" materialization. It creates or replaces table in selected integration with results of predictor. Name of the model is used as name of the table to store prediction results. If you need to specify schema you can do it with dot separator: schema\_name.table\_name.sql ###### Parameters * `predictor_name` - name of the predictor. It has to be created in mindsdb. * `integration` - name of used integration to get data from and save result to. Must be created in mindsdb beforehand using the [`CREATE DATABASE` syntax](https://docs.mindsdb.com/sql/create/databases/). ```text {{ config(materialized='table', predictor_name='TEST_PREDICTOR_NAME', integration='photorep') }} select a, bc from ddd where name > latest ``` --- ### Model configurations #### Related documentation * [Models](https://docs.getdbt.com/docs/build/models.md) * [`run` command](https://docs.getdbt.com/reference/commands/run.md) #### Available configurations ##### Model-specific configurations Resource-specific configurations are applicable to only one dbt resource type rather than multiple resource types. You can define these settings in the project file (`dbt_project.yml`), a property file (`models/properties.yml` for models, similarly for other resources), or within the resource’s file using the `{{ config() }}` macro.<br /> The following resource-specific configurations are only available to Models: ##### Project file dbt\_project.yml (Applies to dbt v1.12 and later) ```yaml models: <resource-path>: +materialized: <materialization_name> +sql_header: <string> +on_configuration_change: apply | continue | fail # only for materialized views on supported adapters +unique_key: <column_name_or_expression> +freshness: <dict> +on_error: skip_children | continue +latest_version_pointer: <dict> ``` ##### Property file (Applies to dbt v2.0 and later) models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: <materialization_name> sql_header: <string> on_configuration_change: apply | continue | fail # only for materialized views on supported adapters unique_key: <column_name_or_expression> freshness: warn_after: {count: 24, period: hour} error_after: {count: 48, period: hour} build_after: <dict> # build scheduling — Enterprise only loaded_at_field: updated_at # or loaded_at_query on_error: skip_children | continue latest_version_pointer: <dict> ``` ##### SQL file config models/\<model\_name>.sql (Applies to dbt v1.12 and later) ```sql {{ config( materialized="<materialization_name>", sql_header="<string>" on_configuration_change: apply | continue | fail # only for materialized views for supported adapters unique_key='column_name_or_expression' freshness=<dict> on_error="skip_children" | "continue" latest_version_pointer=<dict> ) }} ``` ##### General configurations General configurations provide broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project file, property files, or within resource-specific files. ##### Project file dbt\_project.yml (Applies to dbt v1.9 and later) ```yaml models: <resource-path>: +enabled: true | false +tags: <string> | [<string>] +pre-hook: <sql-statement> | [<sql-statement>] +post-hook: <sql-statement> | [<sql-statement>] +database: <string> +schema: <string> +alias: <string> +persist_docs: <dict> +full_refresh: <boolean> +meta: {<dictionary>} +grants: {<dictionary>} +contract: {<dictionary>} +event_time: my_time_field ``` ##### Property file models/properties.yml (Applies to dbt v1.9 and later) ```yaml models: - name: [<model-name>] config: enabled: true | false tags: <string> | [<string>] pre_hook: <sql-statement> | [<sql-statement>] post_hook: <sql-statement> | [<sql-statement>] database: <string> schema: <string> alias: <string> persist_docs: <dict> full_refresh: <boolean> meta: {<dictionary>} grants: {<dictionary>} contract: {<dictionary>} event_time: my_time_field ``` ##### SQL file config models/\<model\_name>.sql (Applies to dbt v1.9 and later) ```sql {{ config( enabled=true | false, tags="<string>" | ["<string>"], pre_hook="<sql-statement>" | ["<sql-statement>"], post_hook="<sql-statement>" | ["<sql-statement>"], database="<string>", schema="<string>", alias="<string>", persist_docs={<dict>}, meta={<dict>}, grants={<dict>}, contract={<dictionary>}, event_time='my_time_field', ) }} ``` ##### Warehouse-specific configurations * [BigQuery configurations](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md) * [Redshift configurations](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md) * [Snowflake configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md) * [Databricks configurations](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md) * [Spark configurations](https://docs.getdbt.com/reference/resource-configs/spark-configs.md) #### Configuring models Model configurations are applied hierarchically. You can configure models from within an installed package and also from within your dbt project in the following ways, listed in order of precedence: 1. Using a `config()` Jinja macro within a model. 2. Using a `config` [resource property](https://docs.getdbt.com/reference/model-properties.md) in a `.yml` file. 3. From the project YAML file (`dbt_project.yml`), under the `models:` key. In this case, the model that's nested the deepest will have the highest priority. The most specific configuration always takes precedence. In the project YAML file, for example, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model or directory of models, define the [resource path](https://docs.getdbt.com/reference/resource-configs/resource-path.md) as nested dictionary keys. Model configurations in your root dbt project have *higher* precedence than configurations in installed packages. This enables you to override the configurations of installed packages, providing more control over your dbt runs. #### Example ##### Configuring directories of models in `dbt_project.yml` To configure models in your `dbt_project.yml` file, use the `models:` configuration option. Be sure to namespace your configurations to your project (shown below): dbt\_project.yml ```yml name: dbt_labs models: # Be sure to namespace your model configs to your project name dbt_labs: # This configures models found in models/events/ events: +enabled: true +materialized: view # This configures models found in models/events/base # These models will be ephemeral, as the config above is overridden base: +materialized: ephemeral ... ``` ##### Apply configurations to one model only Some types of configurations are specific to a particular model. In these cases, placing configurations in the `dbt_project.yml` file can be unwieldy. Instead, you can specify these configurations at the top of a model `.sql` file, or in its individual YAML properties. models/events/base/base\_events.sql ```sql {{ config( materialized = "table", tags = ["core", "events"] ) }} select * from {{ ref('raw_events') }} ``` models/events/base/properties.yml ```yaml models: - name: base_events description: "Standardized event data from raw sources" columns: - name: user_id description: "Unique identifier for a user" data_tests: - not_null - unique - name: event_type description: "Type of event recorded (click, purchase, etc.)" ``` (Applies to dbt v2.0 and later) ##### Configuring model freshness You can use the `freshness` config in the following ways: * **Freshness SLA** (`warn_after`, `error_after`): To declare how stale this model's data is allowed to be. dbt evaluates these thresholds when you run [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) and reports `Pass`, `Warn`, or `Error`. Support varies by materialization and measurement method. Refer to the [measurement methods table](https://docs.getdbt.com/reference/commands/freshness.md#what-gets-checked) for more information. * **Build scheduling** (`build_after`): To skip rebuilding a model when upstream data hasn't changed since the last build. Available on dbt platform Enterprise tiers only. models/my\_model.yml ```yml models: - name: stg_orders config: freshness: # Freshness SLA — report warn/error when data is stale warn_after: {count: 24, period: hour} error_after: {count: 48, period: hour} # Build scheduling — Enterprise only build_after: count: 6 period: hour updates_on: any # optional loaded_at_field: updated_at # or loaded_at_query ``` --- ### Model properties Models properties can be declared in `.yml` files in your `models/` directory (as defined by the [`model-paths` config](https://docs.getdbt.com/reference/project-configs/model-paths.md)). You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `models/` directory. Availability The latest YAML spec is supported in the following environments: * **dbt platform (v1 Latest release track)** * **dbt v2** * **dbt v1.12** For more information, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). #### Available top-level model properties | Property | Type | Required | Description | | ---------------------------------------------------------------------------------------------- | ------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [name](https://docs.getdbt.com/reference/resource-properties/model_name.md) | string | Yes | The model name (must match the model filename). | | [description](https://docs.getdbt.com/reference/resource-properties/description.md) | string | No | Documentation for the model. | | [columns](https://docs.getdbt.com/reference/resource-properties/columns.md) | array | No | List of column definitions. | | [config](https://docs.getdbt.com/reference/resource-properties/config.md) | object | No | Model configuration (materialization, tags, etc.). | | [constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md) | array | No | Model-level constraints (primary key, foreign key, etc.). | | [data\_tests](https://docs.getdbt.com/reference/resource-properties/data-tests.md) | array | No | Model-level data tests. | | tests | array | No | Legacy alias for data\_tests. | | [versions](https://docs.getdbt.com/reference/resource-properties/versions.md) | array | No | Model version definitions. | | [latest\_version](https://docs.getdbt.com/reference/resource-properties/latest_version.md) | string/float | No | The latest version of the model. | | [deprecation\_date](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md) | string | No | Date when the model is deprecated. | | [access](https://docs.getdbt.com/reference/resource-configs/access.md) | string | No | Access level: private, protected, or public. Supported at the top-level for backwards compatibility only. | | [time\_spine](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) | object | No | Time spine configuration for the Semantic Layer. | | [semantic\_model](https://docs.getdbt.com/reference/semantic-model-properties.md) | object | No | *Latest YAML spec only.* Enable semantic model configuration for the Semantic Layer with `enabled: true`. For other properties, refer to [Semantic model properties](https://docs.getdbt.com/reference/semantic-model-properties.md). | | [metrics](https://docs.getdbt.com/reference/metric-properties.md) | array | No | *Latest YAML spec only.* Metrics derived from this semantic model; list is alongside (not under) `semantic_model` and `columns`. For other properties, refer to [Metric properties](https://docs.getdbt.com/reference/metric-properties.md). | (Applies to dbt v1.12 and later) ##### Example file (latest) The latest YAML spec includes Semantic Layer properties: `semantic_model:`, top-level `agg_time_dimension` and `primary_entity`, and `metrics:`. Columns can include `entity:` and `dimension:` blocks and `granularity` for time dimensions. See [Semantic model properties](https://docs.getdbt.com/reference/semantic-model-properties.md) and [Metric properties](https://docs.getdbt.com/reference/metric-properties.md) for the full structure. models/\<filename>.yml ```yml models: # Model name must match the filename of a model -- including case sensitivity - name: model_name description: <markdown_string> latest_version: <version_identifier> deprecation_date: <YAML_DateTime> config: <model_config>: <config_value> docs: show: true | false node_color: <color_id> # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") access: private | protected | public constraints: - <constraint> data_tests: - <test> - ... # declare additional data tests columns: - name: <column_name> # required description: <markdown_string> quote: true | false constraints: - <constraint> data_tests: - <test> - ... # declare additional data tests config: meta: {<dictionary>} tags: [<string>] # only required in conjunction with time_spine key granularity: <any supported time granularity> # In the latest YAML spec, a column can optionally include an entity or dimension block for Semantic Layer. # # entity: # type: primary | foreign | unique | natural # name: <entity_name> # dimension: # type: time | categorical # name: <dimension_name> - name: ... # declare properties of additional columns time_spine: standard_granularity_column: <column_name> # Latest YAML spec only: semantic model and metrics on the model semantic_model: enabled: true name: <semantic_model_name> agg_time_dimension: <time_dimension_name> # top-level; references dimension name primary_entity: <primary_entity_name> # optional; use when no column has type: primary metrics: - name: <metric_name> type: simple | cumulative | ratio | derived | conversion # ... type-specific properties; see Metric properties versions: - v: <version_identifier> # required defined_in: <definition_file_name> description: <markdown_string> constraints: - <constraint> config: <model_config>: <config_value> docs: show: true | false access: private | protected | public data_tests: - <test> - ... # declare additional data tests columns: # include/exclude columns from the top-level model properties - include: <include_value> exclude: <exclude_list> # specify additional columns - name: <column_name> # required quote: true | false constraints: - <constraint> data_tests: - <test> - ... # declare additional data tests tags: [<string>] - v: ... # declare additional versions ``` --- ### model_name models/\<schema>.yml ```yml models: - name: model_name ``` #### Definition The name of the model you are declaring properties for. Must match the *filename* of a model — including case sensitivity. Any mismatched casing can prevent dbt from applying configurations correctly and may affect metadata in Catalog. #### Default This is a **required property**, no default exists. --- ### model-paths dbt\_project.yml ```yml model-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [models](https://docs.getdbt.com/docs/build/models.md), [sources](https://docs.getdbt.com/docs/build/sources.md), and [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) are located. #### Default By default, dbt will search for models and sources in the `models` directory. For example, `model-paths: ["models"]`. Paths specified in `model-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/models`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml model-paths: ["models"] ``` * ❌ **Don't:** * Avoid absolute paths: ```yml model-paths: ["/Users/username/project/models"] ``` #### Examples ##### Use a subdirectory named `transformations` instead of `models` dbt\_project.yml ```yml model-paths: ["transformations"] ``` --- ### name dbt\_project.yml ```yml name: string ``` #### Definition **Required configuration** The name of a dbt project. Must be letters, digits and underscores only, and cannot start with a digit. #### Recommendation Often an organization has one dbt project, so it is sensible to name a project with your organization's name, in `snake_case`. For example: * `name: acme` * `name: jaffle_shop` * `name: evilcorp` #### Troubleshooting ##### Invalid project name ```text Encountered an error while reading the project: ERROR: Runtime Error at path ['name']: 'jaffle-shop' does not match '^[^\\d\\W]\\w*$' Runtime Error Could not run dbt ``` This project has: dbt\_project.yml ```yml name: jaffle-shop ``` In this case, change your project name to be `snake_case` instead: dbt\_project.yml ```yml name: jaffle_shop ``` --- ### name ###### name The `name` field can be used to specify both model names and column names in schema.yml files. When `name` is used to denote a model (as shown on line 4 above), dbt will look up that model by its filename. The example above will look for a model called `events.sql`, for instance. --- ### No + prefix on resource paths dbt v2-only flag This behavior change flag is specific to dbt v2 and has no equivalent in dbt v1. | require\_resource\_names\_without\_plus\_prefix | dbt v2 | | ----------------------------------------------- | ------------------ | | Introduced | 2.0.0-preview.208 | | Matured (default → `true`) | — | | Removed | — | The `require_resource_names_without_plus_prefix` flag is set to `false` by default. In `dbt_project.yml`, the [`+` prefix](https://docs.getdbt.com/reference/resource-configs/plus-prefix.md) marks a config, while a key without the prefix is treated as part of a [resource path](https://docs.getdbt.com/reference/resource-configs/resource-path.md) (a folder or file name). dbt historically accepted resource paths that start with `+` (for example, a folder named `+my_folder` under `models`) even though the `+` prefix is meant for configs only. This is accidental behavior that produces misleading error messages when a config is missing its `+` prefix, because dbt can't reliably tell whether a `+`-prefixed key is a folder name or a mistyped config. When the `require_resource_names_without_plus_prefix` flag is set to `false`, dbt raises a warning for each resource path that starts with `+`: ```shell [warning] [InvalidConfig (dbt1005)]: Resource path `snapshot_contract.+path_1` in dbt_project.yml starts with `+`. This will be deprecated in future versions of dbt. ``` When the flag is set to `true`, dbt treats a `+`-prefixed key as an unrecognized config key and raises an error instead: ```shell [error] [SerializationError (dbt1013)]: Invalid model definition `snapshot_contract.+path_1`: Unrecognized key `snapshot_contract.+path_1`. Custom keys must go under `+meta`. ``` To resolve the warning or error, remove the + prefix from any folder or file name in dbt\_project.yml. For custom keys, nest them under [`+meta`](https://docs.getdbt.com/reference/resource-configs/meta.md). For example, name the resource path so it doesn't start with `+`: dbt\_project.yml ```yml models: my_project: my_folder: # resource path — no + prefix +enabled: true # config — keep the + prefix ``` --- ### No spaces in resource names Removed in v2 This flag was removed in v2. The new behavior is always enabled. If you're upgrading, remove this flag from your `dbt_project.yml`. | require\_resource\_names\_without\_spaces | dbt **v1 Latest** | dbt v1 | | ----------------------------------------- | ----------------- | ------ | | Introduced | 2024.05 | 1.8.0 | | Matured (default → `true`) | 2025.05 | 1.10.0 | | Removed | — | v2.0 | dbt raises an error if it detects a space in a resource name. Resource names should contain letters, numbers, and underscores only. dbt raises the [`ResourceNamesWithSpacesDeprecation`](https://docs.getdbt.com/reference/deprecations.md#resourcenameswithspacesdeprecation) warning if it detects a space in a resource name. When this flag is set to `true` (now always-on in v2), dbt raises an error instead. models/model name with spaces.sql ```sql -- This model file should be renamed to model_name_with_underscores.sql ``` See also [`require_source_and_semantic_model_names_without_spaces`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_source_and_semantic_model_names_without_spaces.md), which extends this behavior to source and semantic model names. --- ### No spaces in source and semantic model names | require\_source\_and\_semantic\_model\_names\_without\_spaces | dbt **v1 Latest** | dbt v1 | | ------------------------------------------------------------- | ----------------- | ------ | | Introduced | 2026.4 | 1.12.0 | | Matured (default → `true`) | — | — | | Removed | — | — | The `require_source_and_semantic_model_names_without_spaces` flag is set to `false` by default. Source names and semantic model names should contain letters, numbers, and underscores — *not* spaces. dbt raises the [`ResourceNamesWithSpacesDeprecation`](https://docs.getdbt.com/reference/deprecations.md#resourcenameswithspacesdeprecation) warning if it detects a space in a source name or semantic model name. When the `require_source_and_semantic_model_names_without_spaces` flag is set to `true`, dbt raises an error. This flag extends [`require_resource_names_without_spaces`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_spaces.md) to cover source and semantic model names specifically. --- ### No specific configurations for this adapter If you were guided to this page from a data platform setup article, it most likely means: * Setting up the profile is the only action the end-user needs to take on the data platform, or * The subsequent actions the end-user needs to take are not currently documented, or * Relevant information is provided on the documentation pages of the data platform vendor. If you'd like to contribute to data platform-specific configuration information, refer to [Documenting a new adapter](https://docs.getdbt.com/guides/adapter-creation.md) --- ### Node selector methods Selector methods return all resources that share a common property, using the syntax `method:value`. While it is recommended to explicitly denote the method, you can omit it (the default value will be one of `path`, `file` or `fqn`). tip You can combine multiple selector methods in one `--select` command by separating them with commas (`,`) without whitespace (for example, `dbt run --select "marts.finance,tag:nightly"`). This only selects resources that satisfy *all* arguments. In this example, the command runs models that are in the `marts/finance` subdirectory and tagged `nightly`. For more information, see [Set operators](https://docs.getdbt.com/reference/node-selection/set-operators.md). Many of the methods below support Unix-style wildcards: | Wildcard | Description | | -------- | --------------------------------------------------------- | | \* | matches any number of any characters (including none) | | ? | matches any single character | | \[abc] | matches one character given in the bracket | | \[a-z] | matches one character from the range given in the bracket | For example: ```bash dbt list --select "*.folder_name.*" dbt list --select "package:*_source" ``` ##### access The `access` method selects models based on their [access](https://docs.getdbt.com/reference/resource-configs/access.md) property. ```bash dbt list --select "access:public" # list all public models dbt list --select "access:private" # list all private models dbt list --select "access:protected" # list all protected models ``` ##### config The `config` method is used to select models that match a specified [node config](https://docs.getdbt.com/reference/configs-and-properties.md). ```bash dbt run --select "config.materialized:incremental" # run all models that are materialized incrementally dbt run --select "config.schema:audit" # run all models that are created in the `audit` schema dbt run --select "config.cluster_by:geo_country" # run all models clustered by `geo_country` ``` While most config values are strings, you can also use the `config` method to match boolean configs, dictionary keys, and values in lists. For example, given a model with the following configurations: ```bash {{ config( materialized = 'incremental', unique_key = ['column_a', 'column_b'], grants = {'select': ['reporter', 'analysts']}, meta = {"contains_pii": true}, transient = true ) }} select ... ``` You can select using any of the following: ```bash dbt ls -s config.materialized:incremental dbt ls -s config.unique_key:column_a dbt ls -s config.grants.select:reporter dbt ls -s config.meta.contains_pii:true dbt ls -s config.transient:true ``` ##### exposure The `exposure` method is used to select parent resources of a specified [exposure](https://docs.getdbt.com/docs/build/exposures.md). Use in conjunction with the `+` operator. ```bash dbt run --select "+exposure:weekly_kpis" # run all models that feed into the weekly_kpis exposure dbt test --select "+exposure:*" # test all resources upstream of all exposures dbt ls --select "+exposure:*" --resource-type source # list all source tables upstream of all exposures ``` ##### file (Applies to dbt v1.11 and later) The `file` method can be used to select a model or a function by its filename, including the file extension (`.sql`). ```bash # These are equivalent dbt run --select "file:some_model.sql" dbt run --select "some_model.sql" dbt run --select "some_model" # These are equivalent dbt build --select "file:my_function.sql" dbt build --select "my_function.sql" dbt build --select "my_function" # To build all models that use the function dbt build --select "my_function+" ``` ##### fqn The `fqn` method is used to select nodes based off their "fully qualified names" (FQN) within the dbt graph. The default output of [`dbt list`](https://docs.getdbt.com/reference/commands/list.md) is a listing of FQN. The default FQN format is composed of the project name, subdirectories within the path, and the file name (without extension) separated by periods. ```bash dbt run --select "fqn:some_model" dbt run --select "fqn:your_project.some_model" dbt run --select "fqn:some_package.some_other_model" dbt run --select "fqn:some_path.some_model" dbt run --select "fqn:your_project.some_path.some_model" ``` ##### group The `group` method is used to select models defined within a [group](https://docs.getdbt.com/reference/resource-configs/group.md). ```bash dbt run --select "group:finance" # run all models that belong to the finance group. ``` ##### metric The `metric` method is used to select parent resources of a specified [metric](https://docs.getdbt.com/docs/build/build-metrics-intro.md). Use in conjunction with the `+` operator. ```bash dbt build --select "+metric:weekly_active_users" # build all resources upstream of weekly_active_users metric dbt ls --select "+metric:*" --resource-type source # list all source tables upstream of all metrics ``` ##### package The `package` method is used to select models defined within the root project or an installed dbt package. While the `package:` prefix is not explicitly required, it may be used to make selectors unambiguous. ```bash # These three selectors are equivalent dbt run --select "package:snowplow" dbt run --select "snowplow" dbt run --select "snowplow.*" ``` Use the `this` package to select nodes from the current project. From the example, running `dbt run --select "package:this"` from the `snowplow` project runs the exact same set of models as the other three selectors. Since `this` always refers to the current project, using `package:this` ensures that you're only selecting models from the project you're working in. ##### path (Applies to dbt v1.11 and later) The `path` method is used to select models, sources, or functions defined at or under a specific path. Model definitions are in SQL/Python files (not YAML), and source definitions are in YAML files. Functions are defined in SQL files. While the `path` prefix is not explicitly required, it may be used to make selectors unambiguous. ```bash # These two selectors are equivalent dbt run --select "path:models/staging/github" dbt run --select "models/staging/github" # These two selectors are equivalent dbt run --select "path:models/staging/github/stg_issues.sql" dbt run --select "models/staging/github/stg_issues.sql" # These two selectors are equivalent dbt build --select "path:functions/my_function.sql" dbt build --select "functions/my_function.sql" ``` ##### resource\_type (Applies to dbt v1.11 and later) Use the `resource_type` method to select nodes of a particular type (`model`, `test`, `exposure`, `function`, and so on). This is similar to the `--resource-type` flag used by the `dbt build`, `dbt test`, `dbt clone`, and `dbt list` [commands](https://docs.getdbt.com/reference/dbt-commands.md#available-commands). ```bash dbt build --select "resource_type:exposure" # build all resources upstream of exposures dbt build --select "resource_type:function" # build all functions in your project dbt list --select "resource_type:test" # list all tests in your project dbt list --select "resource_type:source" # list all sources in your project ``` ##### result The `result` method is related to the [`state` method](https://docs.getdbt.com/reference/node-selection/methods.md#state) and can be used to select resources based on their result status from a prior run. Note that one of the dbt commands \[`run`, `test`, `build`, `seed`] must have been performed in order to create the result on which a result selector operates. You can use `result` selectors in conjunction with the `+` operator. ```bash # run all models that generated errors on the prior invocation of dbt run dbt run --select "result:error" --state path/to/artifacts # run all tests that failed on the prior invocation of dbt test dbt test --select "result:fail" --state path/to/artifacts # run all the models associated with failed tests from the prior invocation of dbt build dbt build --select "1+result:fail" --state path/to/artifacts # run all seeds that generated errors on the prior invocation of dbt seed dbt seed --select "result:error" --state path/to/artifacts ``` * Only use `result:fail` when you want to re-run tests that failed during the last invocation. This selector is specific to test nodes. Tests don't have downstream nodes in the DAG, so using the `result:fail+` selector will only return the failed test itself and not the model or anything built on top of it. * On the other hand, `result:error` selects any resource (models, tests, snapshots, and more) that returned an error. * As an example, to re-run upstream and downstream resources associated with failed tests, you can use one of the following selectors: ```bash # reruns all the models associated with failed tests from the prior invocation of dbt build dbt build --select "1+result:fail" --state path/to/artifacts # reruns the models associated with failed tests and all downstream dependencies - especially useful in deferred state workflows dbt build --select "1+result:fail+" --state path/to/artifacts ``` ##### saved\_query The `saved_query` method selects [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md). ```bash dbt list --select "saved_query:*" # list all saved queries dbt list --select "+saved_query:orders_saved_query" # list your saved query named "orders_saved_query" and all upstream resources ``` (Applies to dbt v1.12 and later) ##### selector The `selector` method selects the nodes defined by a named [YAML selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) in `selectors.yml`. Use it in `--select` or `--exclude` strings so you can compose a named selector with other [methods](https://docs.getdbt.com/reference/node-selection/methods.md), [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md), and [set operators](https://docs.getdbt.com/reference/node-selection/set-operators.md). ```bash dbt run --select "selector:my_selector" # same node set as `dbt run --selector my_selector` dbt build --select "selector:staging selector:nightly" # union (space-separated) dbt build --select "selector:staging,selector:nightly" # intersection (comma-separated) dbt build --select "1+selector:staging" # graph operators dbt build --select "selector:staging tag:nightly" # combine with other methods dbt run --select "selector:staging" --exclude "selector:exclude_tests" ``` When you use the legacy `--selector` flag together with `--select` or `--exclude`, dbt only uses `--selector` for node selection and ignores `--select` and `--exclude`. Starting in dbt v1.12, dbt raises `SelectExcludeIgnoredWithSelectorWarning` when `--selector` is combined with `--select` or `--exclude`. If you want to combine a selector with these flags, use the `selector:` method instead. When you run an "unqualified" command (without `--select` or `--exclude`), dbt applies the [default selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md#default) if you have defined one in `selectors.yml`. When you use `--select` or `--exclude`, dbt ignores the default selector. To include a selector in a `--select` or `--exclude` string, you must explicitly reference it using the `selector:` method. If selector definitions reference each other in a cycle, dbt raises the `DbtRecursionError` at runtime. For more information, refer to [Selector inheritance](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md#selector-inheritance). ##### semantic\_model The `semantic_model` method selects [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md). ```bash dbt list --select "semantic_model:*" # list all semantic models dbt list --select "+semantic_model:orders" # list your semantic model named "orders" and all upstream resources ``` ##### source The `source` method is used to select models that select from a specified [source](https://docs.getdbt.com/docs/build/sources.md#using-sources). Use in conjunction with the `+` operator. ```bash dbt run --select "source:snowplow+" # run all models that select from Snowplow sources dbt run --select "source:snowplow.events+" # run all models downstream of the events table in the Snowplow source ``` Refer to [source FAQs](https://docs.getdbt.com/docs/build/sources.md#faqs) for more info. ##### source\_status Another element of job state is the `source_status` of a prior dbt invocation. After executing `dbt source freshness`, for example, dbt creates the `sources.json` artifact which contains execution times and `max_loaded_at` dates for dbt sources. You can read more about `sources.json` on the ['sources'](https://docs.getdbt.com/reference/artifacts/sources-json.md) page. The following dbt commands produce `sources.json` artifacts whose results can be referenced in subsequent dbt invocations: * `dbt source freshness` After issuing one of the above commands, you can reference the source freshness results by adding a selector to a subsequent command as follows: (Applies to dbt v1.11 and later) ```bash # You can also set the DBT_ENGINE_STATE environment variable instead of the --state flag. dbt source freshness # must be run again to compare current to previous state dbt build --select "source_status:fresher+" --state path/to/prod/artifacts ``` ##### state **N.B.** [State-based selection](https://docs.getdbt.com/reference/node-selection/state-selection.md) is a powerful, complex feature. Read about [known caveats and limitations](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md) to state comparison. The `state` method is used to select nodes by comparing them against a previous version of the same project, which is represented by a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md). The file path of the comparison manifest *must* be specified via the `--state` flag or (Applies to dbt v1.11 and later) `DBT_ENGINE_STATE` environment variable. `state:new`: There is no node with the same `unique_id` in the comparison manifest `state:modified`: All new nodes, plus any changes to existing nodes. ```bash dbt test --select "state:new" --state path/to/artifacts # run all tests on new models + and new tests on old models dbt run --select "state:modified" --state path/to/artifacts # run all models that have been modified dbt ls --select "state:modified" --state path/to/artifacts # list all modified nodes (not just models) ``` Because state comparison is complex, and everyone's project is different, dbt supports subselectors that include a subset of the full `modified` criteria: * `state:modified.body`: Changes to node body (e.g. model SQL, seed values) * `state:modified.configs`: Changes to any node configs, excluding `database`/`schema`/`alias`/`tags`/`meta` * `state:modified.relation`: Changes to `database`/`schema`/`alias` (the database representation of this node), irrespective of `target` values or `generate_x_name` macros * `state:modified.persisted_descriptions`: Changes to relation- or column-level `description`, *if and only if* `persist_docs` is enabled at each level * `state:modified.macros`: Changes to upstream macros (whether called directly or indirectly by another macro) * `state:modified.contract`: Changes to a model's [contract](https://docs.getdbt.com/reference/resource-configs/contract.md), which currently include the `name` and `data_type` of `columns`. Removing or changing the type of an existing column is considered a breaking change, and will raise an error. Remember that `state:modified` includes *all* of the criteria above, as well as some extra resource-specific criteria, such as modifying a source's `freshness` or `quoting` rules or an exposure's `maturity` property. (View the source code for the full set of checks used when comparing [sources](https://github.com/dbt-labs/dbt/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L660-L681), [exposures](https://github.com/dbt-labs/dbt/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L768-L783), and [executable nodes](https://github.com/dbt-labs/dbt/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L319-L330).) dbt treats `tags` and `meta` as metadata-only fields, not modifications, at both the resource and column level. Changing a tags or meta value (including on individual columns in a YAML file) doesn't trigger `state:modified`, because these fields don't affect how dbt materializes the resource. This is different from `description` (for example), which *does* count as a modification when `persist_docs` is enabled. dbt treats any other config change as a modification, since that config could affect materialization. There are two additional `state` selectors that complement `state:new` and `state:modified` by representing the inverse of those functions: * `state:old` — A node with the same `unique_id` exists in the comparison manifest * `state:unmodified` — All existing nodes with no changes These selectors can help you shorten run times by excluding unchanged nodes. Currently, no subselectors are available at this time, but that might change as use cases evolve. ###### `state:modified` node and reference impacts `state:modified` identifies any new nodes added, changes to existing nodes, and any changes made to: * [access](https://docs.getdbt.com/reference/resource-configs/access.md) permissions * [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md) * [`latest_version`](https://docs.getdbt.com/reference/resource-properties/latest_version.md) If a node changes its group, downstream references may break, potentially causing build failures. As `group` is a config, and configs are generally included in `state:modified` detection, modifying the group name everywhere it's referenced will flag those nodes as "modified". Depending on whether partial parsing is enabled, you will catch the breakage as part of CI workflows. * If you change a group name everywhere it's referenced, and partial parsing is enabled, dbt may only re-parse the changed model. * If you update a group name in all its references without partial parsing enabled, dbt will re-parse all models and identify any invalid downstream references. An error along the lines of "there's nothing to do" can occur when you change the group name *and* something is picked up to be run via `dbt build --select state:modified`. This error will be caught at runtime so long as the CI job is selecting `state:modified+` (including downstreams). Certain factors can affect how references are used or resolved later on, including: * Modifying access: if permissions or access rules change, some references might stop working. * Modifying `deprecation_date`: if a reference or model version is marked deprecated, new warnings might appear that affect how references are processed. * Modifying `latest_version`: if there's no tie to a specific version, the reference or model will point to the latest version. * If a newer version is released, the reference will automatically resolve to the new version, potentially changing the behavior or output of the system that relies on it. dbt handles state comparison for seed files differently depending on their size: * **Seed files smaller than 1 MiB** — Included in the `state:modified` selector only when the contents change. * **Seed files 1 MiB or larger** — Included in the `state:modified` selector only when the seed file path changes. ###### Overwrites the `manifest.json` dbt overwrites the `manifest.json` file during parsing, which means when you reference `--state` from the `target/ directory`, you may encounter a warning indicating that the saved manifest wasn't found. [![Saved manifest not found error](/img/docs/reference/saved-manifest-not-found.png?v=2 "Saved manifest not found error")](#)Saved manifest not found error During the next job run, dbt follows a sequence of steps that lead to the issue. First, it overwrites `target/manifest.json` before it can be used for change detection. Then, when dbt tries to read `target/manifest.json` again to detect changes, it finds none because the previous state has already been overwritten/erased. Avoid setting `--state` and `--target-path` to the same path with state-dependent features like `--defer` and `state:modified` as it can lead to non-idempotent behavior and won't work as expected. ###### Recommendation To prevent the `manifest.json` from being overwritten before dbt reads it for change detection, update your workflow using one of these methods: * Move the `manifest.json` to a dedicated folder (for example `state/`) after dbt generates it in the `target/ folder`. This makes sure dbt references the correct saved state instead of comparing the current state with the just-overwritten version. It also avoids issues caused by setting `--state` and `--target-path` to the same location, which can lead to non-idempotent behavior. * Write the manifest to a different `--target-path` in the build stage (where dbt would generate the `target/manifest.json`) or before it gets overwritten during job execution to avoid issues with change detection. This allows dbt to detect changes instead of comparing the current state with the just-overwritten version. * Pass the `--no-write-json` flag: `dbt ls --no-write-json --select state:modified --state target`: during the reproduction stage. ##### tag The `tag:` method is used to select models that match a specified [tag](https://docs.getdbt.com/reference/resource-configs/tags.md). ```bash dbt run --select "tag:nightly" # run all models with the `nightly` tag ``` ##### test\_name The `test_name` method is used to select tests based on the name of the generic test that defines it. For more information about how generic tests are defined, read about [data tests](https://docs.getdbt.com/docs/build/data-tests.md). ```bash dbt test --select "test_name:unique" # run all instances of the `unique` test dbt test --select "test_name:equality" # run all instances of the `dbt_utils.equality` test dbt test --select "test_name:range_min_max" # run all instances of a custom schema test defined in the local project, `range_min_max` ``` ##### The test\_type The `test_type` method is used to select tests based on their type: * [Unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md): * [Singular](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests) * [Generic](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests) ```bash dbt test --select "test_type:unit" # run all unit tests dbt test --select "test_type:data" # run all data tests dbt test --select "test_type:generic" # run all generic data tests dbt test --select "test_type:singular" # run all singular data tests ``` ##### unit\_test The `unit_test` method selects [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md). ```bash dbt list --select "unit_test:*" # list all unit tests dbt list --select "+unit_test:orders_with_zero_items" # list your unit test named "orders_with_zero_items" and all upstream resources ``` ##### version The `version` method selects [versioned models](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) based on their [version identifier](https://docs.getdbt.com/reference/resource-properties/versions.md) and [latest version](https://docs.getdbt.com/reference/resource-properties/latest_version.md). ```bash dbt list --select "version:latest" # only 'latest' versions dbt list --select "version:prerelease" # versions newer than the 'latest' version dbt list --select "version:old" # versions older than the 'latest' version dbt list --select "version:none" # models that are *not* versioned ``` --- ### Null-safe equality (equals macro) | enable\_truthy\_nulls\_equals\_macro | dbt **v1 Latest** | dbt v1 | | ------------------------------------ | ----------------- | ------ | | Introduced | 2025.02 | 1.9.0 | | Matured (default → `true`) | — | — | | Removed | — | — | The `enable_truthy_nulls_equals_macro` flag is `false` by default. Setting it to `true` in your `dbt_project.yml` enables null-safe equality in the dbt [equals](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md#equals) macro, which is used in incremental and snapshot materializations. By default, the `equals()` macro follows SQL's [three-valued logic (3VL)](https://modern-sql.com/concept/three-valued-logic), so `NULL = NULL` evaluates to `UNKNOWN` rather than `TRUE`. When the `enable_truthy_nulls_equals_macro` flag is enabled, the `equals()` macro uses the semantics of the [`IS NOT DISTINCT FROM`](https://modern-sql.com/feature/is-distinct-from) operator with two `NULL` values treated as equal. To enable the flag, add it under `flags` in `dbt_project.yml`: dbt\_project.yml ```yml flags: enable_truthy_nulls_equals_macro: true ``` --- ### on_configuration_change info This functionality is currently only supported for [materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) on a subset of adapters. The `on_configuration_change` config has three settings: * `apply` (default) — Attempt to update the existing database object if possible, avoiding a complete rebuild. * *Note:* If any individual configuration change requires a full refresh, a full refresh is performed in lieu of individual alter statements. * `continue` — Allow runs to continue while also providing a warning that the object was left untouched. * *Note:* This could result in downstream failures as those models may depend on these unimplemented changes. * `fail` — Force the entire run to fail if a change is detected. ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: <materialization_name> +on_configuration_change: apply | continue | fail ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: <materialization_name> on_configuration_change: apply | continue | fail ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized="<materialization_name>", on_configuration_change="apply" | "continue" | "fail" ) }} ``` #### Creation precedence Materializations are implemented following this "drop through" life cycle: 1. If a model does not exist with the provided path, create the new model. 2. If a model exists, but has a different type, drop the existing model and create the new model. 3. If [`--full-refresh`](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) is supplied, replace the existing model regardless of configuration changes and the [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) setting. * BigQuery users may need to run `dbt run --full-refresh` (instead of `dbt run`) after changing a model’s materialization (for example, from `table` to `view`) to ensure dbt fully replaces the existing relation and the change is fully applied. 4. If there are no configuration changes, perform the default action for that type (e.g. apply refresh for a materialized view). 5. Determine whether to apply the configuration changes according to the `on_configuration_change` setting. --- ### on_error Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). ##### Project file dbt\_project.yml ```yaml models: <resource-path>: +on_error: skip_children | continue ``` ##### Property file models/properties.yml ```yaml models: - name: [<model-name>] config: on_error: skip_children | continue ``` ##### SQL config models/\<model\_name>.sql ```sql {{ config( on_error="skip_children" | "continue" ) }} ``` #### Definition The `on_error` config controls what happens to downstream (child) models when a model fails during a run. This config only applies to models; it has no effect on tests. To control whether downstream models run after a test failure, use the [`severity`](https://docs.getdbt.com/reference/resource-configs/severity.md) config on your tests instead. `on_error` accepts two values: * `skip_children` (default): All downstream models are skipped when the model fails. * `continue`: Downstream models continue running when the model fails, instead of being skipped. #### Example Set `on_error: continue` when downstream models can still run meaningfully even if an upstream model fails (for example, when they have fallback logic or independent data sources). models/my\_model.sql ```sql {{ config( materialized='table', on_error='continue' ) }} select 1 as id ``` When `on_error` is set to `continue` on a model that fails, dbt runs its downstream models rather than skipping them. The failed model still appears as an error in the run results, and the overall run still fails even if all downstream models succeed. The [`--fail-fast`](https://docs.getdbt.com/reference/global-configs/failing-fast.md) flag takes precedence over `on_error: continue`. When `--fail-fast` is set, dbt stops at the first failure and skips all remaining models, regardless of their `on_error` configuration. #### Behavior with multiple upstream models When a model has multiple upstream models, `skip_children` takes precedence over `continue`. If any failed upstream model uses `skip_children`, the downstream model is skipped — even if other failed upstream models use `continue`. For example, consider a DAG where `model_c` depends on both `model_a` and `model_b`: * `model_a` uses `on_error: skip_children` * `model_b` uses `on_error: continue` The following table shows how `model_c` behaves based on the outcome of its upstream models: | `model_a` result (`skip_children`) | `model_b` result (`continue`) | `model_c` behavior | | ---------------------------------- | ----------------------------- | ------------------ | | Success | Success | Runs | | Error | Success | Skipped | | Success | Error | Runs | | Error | Error | Skipped | <br /> When `model_a` errors, `model_c` is always skipped because `model_a` uses `skip_children`. When only `model_b` errors, `model_c` still runs because `model_b` uses `continue`, which does not block downstream models. --- ### on-run-start & on-run-end dbt\_project.yml ```yml on-run-start: sql-statement | [sql-statement] on-run-end: sql-statement | [sql-statement] ``` #### Definition A SQL statement (or list of SQL statements) to be run at the start or end of the following commands: `dbt build`, `dbt compile`, `dbt docs generate`, `dbt run`, `dbt seed`, `dbt snapshot`, or `dbt test`. `on-run-start` and `on-run-end` hooks can also [call macros](#call-a-macro-to-grant-privileges) that return SQL statements. #### Usage notes * The `on-run-end` hook has additional Jinja variables available in the context — check out the [docs](https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context.md). #### Examples ##### Grant privileges on all schemas that dbt uses at the end of a run This leverages the [schemas](https://docs.getdbt.com/reference/dbt-jinja-functions/schemas.md) variable that is only available in an `on-run-end` hook. dbt\_project.yml ```yml on-run-end: - "{% for schema in schemas %}grant usage on schema {{ schema }} to group reporter; {% endfor %}" ``` ##### Call a macro to grant privileges dbt\_project.yml ```yml on-run-end: "{{ grant_select(schemas) }}" ``` ##### Additional examples We've compiled some more in-depth examples [here](https://docs.getdbt.com/docs/build/hooks-operations.md#additional-examples). --- ### Oracle configurations (Applies to dbt v1.3.2 and later) #### Use `parallel` hint Table materialization supports specifying the number of parallel executions as shown below ```sql -- Create a dbt model using 4 parallel executions {{config(materialized='table', parallel=4}} SELECT c.cust_id, c.cust_first_name, c.cust_last_name from {{ source('sh_database', 'customers') }} c ``` #### Use `table_compression_clause` Table materialization supports different compression clauses as shown below ##### Advanced Row Compression With Advanced compression enabled, Oracle Database maintains compression during all types of data manipulation operations, including conventional DML such as INSERT and UPDATE. `ROW STORE COMPRESS ADVANCED` is recommended in OLTP systems. ```sql -- Advanced Row compression {{config(materialized='table', table_compression_clause='ROW STORE COMPRESS ADVANCED')}} SELECT c.cust_id, c.cust_first_name, c.cust_last_name from {{ source('sh_database', 'customers') }} c ``` ##### Hybrid Columnar Compression ###### Querying `COLUMN STORE COMPRESS FOR QUERY` is useful in data warehouse environments. Valid values are `HIGH` or `LOW`, with `HIGH` providing a higher compression ratio. The default is `HIGH` ```sql {{config(materialized='table', table_compression_clause='COLUMN STORE COMPRESS FOR QUERY LOW')}} SELECT c.cust_id, c.cust_first_name, c.cust_last_name from {{ source('sh_database', 'customers') }} c ``` or ```sql {{config(materialized='table', table_compression_clause='COLUMN STORE COMPRESS FOR QUERY HIGH')}} SELECT c.cust_id, c.cust_first_name, c.cust_last_name from {{ source('sh_database', 'customers') }} c ``` ###### Archival `COLUMN STORE COMPRESS FOR ARCHIVE` supports a higher compression ratio than `COLUMN STORE COMPRESS FOR QUERY` and is useful for archival. Valid values are `HIGH` or `LOW` with `HIGH` providing the highest compression ratio. The default is `LOW` ```sql {{config(materialized='table', table_compression_clause='COLUMN STORE COMPRESS FOR ARCHIVE LOW')}} SELECT c.cust_id, c.cust_first_name, c.cust_last_name from {{ source('sh_database', 'customers') }} c ``` or ```sql {{config(materialized='table', table_compression_clause='COLUMN STORE COMPRESS FOR ARCHIVE HIGH')}} SELECT c.cust_id, c.cust_first_name, c.cust_last_name from {{ source('sh_database', 'customers') }} c ``` #### Partitioning Table and Incremental materialization configuration supports adding a partitioning clause: ```sql { config( materialized='incremental', unique_key='group_id', parallel=4, partition_config={"clause": "PARTITION BY HASH(PROD_NAME) PARTITIONS 4"}, table_compression_clause='COLUMN STORE COMPRESS FOR QUERY LOW') }} SELECT * FROM {{ source('sh_database', 'sales') }} ``` #### Session info in `v$session` Custom session information can be supplied under `session_info` in `profiles.yml` ```yaml dbt_test: target: dev outputs: dev: type: oracle user: "{{ env_var('DBT_ORACLE_USER') }}" pass: "{{ env_var('DBT_ORACLE_PASSWORD') }}" database: "{{ env_var('DBT_ORACLE_DATABASE') }}" tns_name: "{{ env_var('DBT_ORACLE_TNS_NAME') }}" schema: "{{ env_var('DBT_ORACLE_SCHEMA') }}" threads: 4 session_info: action: "dbt run" client_identifier: "dbt-unique-client-uuid" client_info: "dbt Python3.9 thin driver" module: "dbt-oracle-1.8.x" ``` This helps to track dbt sessions in the Database view [V$SESSION](https://docs.oracle.com/en/database/oracle/oracle-database/19/refrn/V-SESSION.html) --- ### osi-paths Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). dbt\_project.yml ```yml osi-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [Apache Ossie semantic layer documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md) are located. #### Default By default, dbt will search for Ossie documents in the `OSI` directory, for example, `osi-paths: ["OSI"]`. Paths specified in `osi-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/OSI`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml osi-paths: ["OSI"] ``` * ❌ **Don't:** * Avoid absolute paths: ```yml osi-paths: ["/Users/username/project/OSI"] ``` #### Examples Use a subdirectory named `semantic_interchange` instead of `OSI`: dbt\_project.yml ```yml osi-paths: ["semantic_interchange"] ``` Use multiple directories to organize your Ossie documents: dbt\_project.yml ```yml osi-paths: ["OSI", "external_osi"] ``` --- ### Other artifact files ##### index.html **Produced by:** [`docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) This file is the skeleton of the [auto-generated dbt documentation website](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md). The contents of the site are populated by the [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) and [catalog](https://docs.getdbt.com/reference/artifacts/catalog-json.md). Note: the source code for `index.json` comes from the [dbt-docs repo](https://github.com/dbt-labs/dbt-docs). Head over there if you want to make a bug report, suggestion, or contribution relating to the documentation site. ##### partial\_parse.msgpack **Produced by:** [manifest commands](https://docs.getdbt.com/reference/artifacts/manifest-json.md) + [`parse`](https://docs.getdbt.com/reference/commands/parse.md) This file is used to store a compressed representation of files dbt has parsed. If you have [partial parsing](https://docs.getdbt.com/reference/parsing.md#partial-parsing) enabled, dbt will use this file to identify the files that have changed and avoid re-parsing the rest. ##### graph.gpickle **Produced by:** commands supporting [node selection](https://docs.getdbt.com/reference/node-selection/syntax.md) Stores the network representation of the dbt resource DAG. ##### graph\_summary.json **Produced by:** [manifest commands](https://docs.getdbt.com/reference/artifacts/manifest-json.md) This file is useful for investigating performance issues in dbt v1's graph algorithms. It is more anonymized and compact than [`manifest.json`](https://docs.getdbt.com/reference/artifacts/manifest-json.md) and [`graph.gpickle`](#graph.gpickle). It includes that information at two separate points in time: 1. `linked` — immediately after the graph is linked together, and 2. `with_test_edges` — after test edges have been added. Each of those points in time contains the `name` and `type` of each node and `succ` contains the keys of its child nodes. --- ### overloads Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> arguments: - name: <arg name> data_type: <string> returns: data_type: <string> overloads: - defined_in: <string> # required, name of the SQL, Python, or JavaScript file arguments: # optional - name: <arg name> # required if arguments is specified data_type: <string> # required if arguments is specified, warehouse-specific description: <markdown_string> default_value: <string | boolean | integer> # optional, Snowflake and Postgres only returns: # optional, inherits from root function if omitted data_type: <string> # required if returns is specified, warehouse-specific description: <markdown_string> - defined_in: ... # declare additional overloads ``` #### Definition The `overloads` property lets you define multiple argument signatures for the same [user-defined function UDF](https://docs.getdbt.com/docs/build/udfs.md). This lets you call the same function name with different input types, without creating separate UDFs for each variant. The warehouse calls the right version based on the argument types. `overloads` is supported for SQL UDFs in Snowflake and Postgres, and Python and JavaScript UDFs in Snowflake. Each overload references a separate file that contains its function body, with optional `arguments` and `returns`. All overloads are grouped into one DAG node (the root function), so they're built and selected together. On retry, dbt skips overloads that succeeded and reruns only those that failed. #### Behavior dbt runs all overloads regardless of individual failures, so you get a complete picture of which overloads succeeded and which failed. The following behaviors apply: * If any overload fails, dbt marks the function node as `PARTIAL_SUCCESS` and skips downstream nodes. * [`dbt retry`](https://docs.getdbt.com/reference/commands/retry.md) skips overloads that already succeeded and only re-runs the previously failed ones. * [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#the-state-method) detects changes to any overload's function body, arguments, or return type and marks the root function node as modified. #### Properties Each entry in the `overloads` list supports the following properties. ##### defined\_in The name of the file (without extension) that contains the overload's function body. The file must exist in the `functions/` directory. For example, `defined_in: null_if_empty_numeric` references `functions/null_if_empty_numeric.sql` for SQL UDFs, `functions/null_if_empty_numeric.py` for Python UDFs, or `functions/null_if_empty_numeric.js` for JavaScript UDFs. Each overload must reference a unique file. The root function's file and all `defined_in` values must be distinct. dbt raises a parsing error if: * A `defined_in` value matches the root function's own file. * Two overloads reference the same file. * The referenced file doesn't exist in the `functions/` directory. ##### arguments The argument list for the overload. Follows the same structure as [function arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md). ##### returns The return type for the overload. Follows the same structure as [returns](https://docs.getdbt.com/reference/resource-properties/returns.md). If omitted, the overload inherits the return type of the root function. #### Example ##### SQL functions/null\_if\_empty.yml ```yml functions: - name: null_if_empty arguments: - name: val data_type: varchar returns: data_type: varchar overloads: - defined_in: null_if_empty_numeric arguments: - name: val data_type: numeric returns: data_type: numeric ``` Create a separate SQL file for each overload body. In this example, the base function handles empty strings, and the overload handles numeric values: functions/null\_if\_empty.sql ```sql # syntax for Snowflake CASE WHEN val = '' THEN NULL ELSE val END # syntax for Postgres SELECT CASE WHEN val = '' THEN NULL ELSE val END ``` functions/null\_if\_empty\_numeric.sql ```sql # syntax for Snowflake CASE WHEN val = 0 THEN NULL ELSE val END # syntax for Postgres SELECT CASE WHEN val = 0 THEN NULL ELSE val END ``` ##### Python functions/null\_if\_empty.yml ```yml functions: - name: null_if_empty config: runtime_version: "3.11" entry_point: main arguments: - name: val data_type: varchar returns: data_type: varchar overloads: - defined_in: null_if_empty_numeric arguments: - name: val data_type: numeric returns: data_type: numeric ``` Create a separate Python file for each overload body. In this example, the base function handles empty strings, and the overload handles numeric values: functions/null\_if\_empty.py ```python def main(val): return None if val == '' else val ``` functions/null\_if\_empty\_numeric.py ```python def main(val): return None if val == 0 else val ``` ##### JavaScript functions/null\_if\_empty.yml ```yml functions: - name: null_if_empty arguments: - name: val data_type: varchar returns: data_type: varchar overloads: - defined_in: null_if_empty_numeric arguments: - name: val data_type: numeric returns: data_type: numeric ``` Create a separate JavaScript file for each overload body. In this example, the base function handles empty strings, and the overload handles numeric values: functions/null\_if\_empty.js ```javascript if (val === '') return null; return val; ``` functions/null\_if\_empty\_numeric.js ```javascript if (val === 0) return null; return val; ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) * [returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### overrides Deprecation The `overrides` property is [deprecated in v1.10](https://docs.getdbt.com/reference/deprecations.md#sourceoverridedeprecation). You can [enable or disable a source](https://docs.getdbt.com/reference/source-configs.md#configuring-sources) from a package instead. models/\<filename>.yml ```yml sources: - name: <source_name> overrides: <package name> # deprecated in v1.10 database: ... schema: ... ``` #### Definition Override a source defined in an included package. The properties defined in the overriding source will be applied on top of the base properties of the overridden source. The following source properties can be overridden: * [description](https://docs.getdbt.com/reference/resource-properties/description.md) * [meta](https://docs.getdbt.com/reference/resource-configs/meta.md) * [database](https://docs.getdbt.com/reference/resource-properties/database.md) * [schema](https://docs.getdbt.com/reference/resource-properties/schema.md) * [loader](https://docs.getdbt.com/reference/resource-properties/loader.md) * [quoting](https://docs.getdbt.com/reference/resource-properties/quoting.md) * [freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md) * [loaded\_at\_field](https://docs.getdbt.com/reference/resource-configs/freshness.md) * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) #### Examples ##### Supply your database and schema name for a source defined in a package This example is based on the [Fivetran GitHub Source package](https://github.com/fivetran/dbt_github_source/blob/830ba43ac2948e4853a3c167ab7ee88b8b425fa0/models/src_github.yml#L3-L29). Here, the database and schema are overridden in the parent dbt project which includes the `github_source` package. models/src\_github.yml ```yml sources: - name: github overrides: github_source # deprecated in v1.10 database: RAW schema: github_data ``` ##### Configure your own source freshness for a source table in a package You can override configurations at both the source and the table level models/src\_github.yml ```yml sources: - name: github overrides: github_source # deprecated in v1.10 config: freshness: # changed to config in v1.9 warn_after: count: 1 period: day error_after: count: 2 period: day tables: - name: issue_assignee config: freshness: warn_after: count: 2 period: day error_after: count: 4 period: day ``` --- ### Package overrides for built-in materializations Removed in v2 This flag was removed in v2. The new behavior is always enabled. If you're upgrading, remove this flag from your `dbt_project.yml`. | require\_explicit\_package\_overrides\_for\_builtin\_materializations | dbt **v1 Latest** | dbt v1 | | --------------------------------------------------------------------- | ----------------- | -------------- | | Introduced | 2024.04 | 1.6.14, 1.7.14 | | Matured (default → `true`) | 2024.06 | 1.8.0 | | Removed | — | v2.0 | Installed packages can no longer override built-in materializations without your explicit opt-in. A materialization defined in a package that matches the name of a built-in materialization is no longer included in the search and resolution order. Unlike macros, materializations don't use the `search_order` defined in the project `dispatch` config. The built-in materializations are `'view'`, `'table'`, `'incremental'`, `'materialized_view'` for models, as well as `'test'`, `'unit'`, `'snapshot'`, `'seed'`, and `'clone'`. You can still explicitly override built-in materializations by reimplementing the built-in materialization in your root project and wrapping the package implementation: macros/materialization\_view.sql ```sql {% materialization view, snowflake %} {{ return(my_installed_package_name.materialization_view_snowflake()) }} {% endmaterialization %} ``` In the future, we may extend the project-level [`dispatch` configuration](https://docs.getdbt.com/reference/project-configs/dispatch-config.md) to support a list of authorized packages for overriding built-in materializations. --- ### Package ref search order | require\_ref\_searches\_node\_package\_before\_root | dbt **v1 Latest** | dbt v1 | | --------------------------------------------------- | ----------------- | ------ | | Introduced | 2025.12 | 1.11.0 | | Matured (default → `true`) | — | — | | Removed | — | — | The `require_ref_searches_node_package_before_root` flag controls the search order when dbt resolves `ref()` calls defined within a package. The flag is set to `false` by default in **v1 Latest** and dbt v1.11. When dbt resolves a `ref()` in a package model, it searches for the referenced model in the root project *first*, then in the package where the model is defined. For example, the following model in the package `my_package` is imported by the project `my_project`: my\_package/model\_downstream.sql ```sql select * from {{ ref('model_upstream') }} ``` By default, dbt searches for `model_upstream` in this order: 1. First in `my_project` (root project) 2. Then in `my_package` (where the model is defined) When you set the `require_ref_searches_node_package_before_root` flag to `true`, dbt searches the package where the model is defined *before* searching the root project. Using the same example, dbt searches for `model_upstream` in this order: 1. First in `my_package` (where the model is defined) 2. Then in `my_project` (root project) The current default behavior is considered a [bug in dbt](https://github.com/dbt-labs/dbt/issues/11351) because it can potentially lead to unexpected dependency cycles. However, because this is long-standing behavior, changing the default requires setting `require_ref_searches_node_package_before_root` to `true` to avoid breaking existing projects. --- ### packages Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> config: packages: [<string>] # optional, Python UDFs only ``` #### Definition When creating Python UDFs, you can use the optional `packages` config to specify public third-party PyPI packages. You can list package names (for example, `numpy` and `pandas`) and pin versions (for example, `pandas==1.5.0`). The warehouse installs these packages when it creates the UDF, so your UDF can use functionality from external Python libraries. On Snowflake, some packages are installed from the Anaconda repository, and you may need to [accept Anaconda's Terms of Service](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages#using-third-party-packages-from-anaconda) before you can use them. Python UDFs are currently supported in Snowflake and BigQuery. Each warehouse uses a different mechanism to specify packages. For example: ##### Snowflake functions/schema.yml ```yaml functions: - name: is_positive_int description: Returns 1 if a_string matches ^[0-9]+$, else 0 config: runtime_version: "3.11" entry_point: main packages: - numpy - pandas==1.5.0 arguments: - name: a_string data_type: string returns: data_type: integer ``` Compiled SQL: ```sql CREATE OR REPLACE FUNCTION my_schema.is_positive_int(a_string STRING) RETURNS INTEGER LANGUAGE PYTHON RUNTIME_VERSION = '3.11' HANDLER = 'main' PACKAGES = ('numpy','pandas==1.5.0') AS $$ def main(a_string): ... $$ ``` ##### BigQuery functions/schema.yml ```yaml functions: - name: is_positive_int description: Returns 1 if a_string matches ^[0-9]+$, else 0 config: runtime_version: "3.11" entry_point: main packages: - numpy - pandas==1.5.0 arguments: - name: a_string data_type: string returns: data_type: integer ``` Compiled SQL: ```sql CREATE OR REPLACE FUNCTION my_dataset.is_positive_int(a_string STRING) RETURNS INT64 LANGUAGE python OPTIONS( runtime_version = "python-3.11", entry_point = "main", packages = ['numpy', 'pandas==1.5.0'] ) AS r''' def main(a_string): ... ''' ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [runtime\_version](https://docs.getdbt.com/reference/resource-configs/runtime-version.md) * [entry\_point](https://docs.getdbt.com/reference/resource-configs/entry-point.md) * [type](https://docs.getdbt.com/reference/resource-configs/type.md) * [volatility](https://docs.getdbt.com/reference/resource-configs/volatility.md) * [arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) * [returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### packages-install-path dbt\_project.yml ```yml packages-install-path: directorypath ``` #### Definition Optionally specify a custom directory where [packages](https://docs.getdbt.com/docs/build/packages.md) are installed when you run the `dbt deps` [command](https://docs.getdbt.com/reference/commands/deps.md). Note that this directory is usually git-ignored. #### Default By default, dbt will install packages in the `dbt_packages` directory, i.e. `packages-install-path: dbt_packages` #### Examples ##### Install packages in a subdirectory named `packages` instead of `dbt_packages` dbt\_project.yml ```yml packages-install-path: packages ``` --- ### Parsing ##### Partial Parsing (Applies to dbt v2.0 and later) dbt v2 and partial parsing dbt v2 job runs no longer support the `--partial-parse` and `--no-partial-parse` CLI flags. If you pass them (for example, from a dbt v1 command or script), dbt logs deprecation warning `dbt1700`. Remove these flags from your dbt v2 job commands. For more information, refer to [Deprecated flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) in the guide to upgrading to dbt v2. The `PARTIAL_PARSE` flag can turn partial parsing on or off in your project. See [the docs on parsing](https://docs.getdbt.com/reference/parsing.md#partial-parsing) for more details. dbt\_project.yml ```yaml flags: partial_parse: true ``` Usage ```text dbt run --no-partial-parse ``` ##### Static parser The `STATIC_PARSER` config can enable or disable the use of the static parser. See [the docs on parsing](https://docs.getdbt.com/reference/parsing.md#static-parser) for more details. profiles.yml ```yaml config: static_parser: true ``` ##### Opt-in v2 parser (Applies to dbt v1.12 and later) dbt v1 flag The v2 parser flag only applies to dbt v1.12 and higher 1.x versions. If you're already on v2, the flag has no impact. The `use_v2_parser` flag delegates parsing to the v2 parser. This is an opt-in flag. The v2 parser is the Rust-based parser from dbt v2. It's significantly faster than the v1 Python parser, especially on larger projects, where it can be 5–10× quicker. Enabling it can speed up your development workflow and cut down on job startup times. Because it delegates to the parser used in v2.0, it's also a low-risk way to test compatibility with v2 from within dbt v1.12. You can enable the v2 parser in three ways: * CLI flag: `--use-v2-parser` * Environment variable: `DBT_ENGINE_USE_V2_PARSER=true` * `dbt_project.yml` under `flags:`: dbt\_project.yml ```yaml flags: use_v2_parser: true ``` Note: Partial parsing is disabled when `--use-v2-parser` is set. Any stale `partial_parse.msgpack` from a prior run is automatically removed. Because the flag only affects project parsing, the fastest way to check v2 parse compatibility is with `dbt parse`. You can also use `--use-v2-parser` with any other command. Usage ```bash # Test v2 parser compatibility without running models (recommended) dbt parse --use-v2-parser # Or use it with any command dbt run --use-v2-parser ``` Plugin authors `get_nodes` plugin hooks are not supported when `--use-v2-parser` is enabled. --- ### persist_docs ##### Models dbt\_project.yml ```yml models: <resource-path>: +persist_docs: relation: true columns: true ``` models/\<modelname>.sql ```sql {{ config( persist_docs={"relation": true, "columns": true} ) }} select ... ``` ##### Sources This config is not implemented for sources. ##### Seeds dbt\_project.yml ```yml seeds: <resource-path>: +persist_docs: relation: true columns: true ``` ##### Snapshots dbt\_project.yml ```yml snapshots: <resource-path>: +persist_docs: relation: true columns: true ``` (Applies to dbt v1.9 and later) snapshots/snapshot\_name.yml ```yaml snapshots: - name: snapshot_name config: persist_docs: relation: true columns: true ``` snapshots/\<filename>.sql ```sql {% snapshot snapshot_name %} {{ config( persist_docs={"relation": true, "columns": true} ) }} select ... {% endsnapshot %} ``` #### Definition Optionally persist [resource descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) as column and relation comments in the database. By default, documentation persistence is disabled, but it can be enabled for specific resources or groups of resources as needed. #### Support The `persist_docs` config is supported on the most widely used dbt adapters: * Postgres * Redshift * Snowflake * BigQuery * Databricks * Apache Spark * Starburst Galaxy (`dbt-trino`) However, some databases limit where and how descriptions can be added to database objects. Those database adapters might not support `persist_docs`, or might offer only partial support. Some known issues and limitations: ##### Databricks * Column-level comments require `file_format: delta` (or another "v2 file format"). ##### Snowflake * If a column name in a SQL model is in a mixed-case format (for example, `ca_net_ht_N`), the docs for that column will not be persisted. For the docs to persist, there are two options: * Define the column name in the corresponding YML file using lowercase or uppercase letters only. * Use the [`quote`](https://docs.getdbt.com/reference/resource-properties/columns.md#quoter) configuration in the corresponding YML file. See the following sample steps on how to use the `quote` field for columns in a mixed-case format. 1. Create the following SQL and YML files: \<modelname>.sql ```sql {{ config(materialized='table') }} select 1 as "ca_net_ht_N" # note the use of double quotes for the column name ``` \<modelname>.yml ```yml models: - name: <modelname> description: This is the table description columns: - name: "ca_net_ht_N" description: This should be the description of the column quote: true ``` 2. Run `dbt build -s models/<modelname>.sql --full-refresh`. 3. Open the logs at `logs/dbt.log` and check the column description: ```log alter table analytics.<schema>.<modelname> alter "ca_net_ht_N" COMMENT $$This should be the description of the column$$; ``` ##### Starburst Galaxy (dbt-trino) * Requires configuring an [API Auth token](https://docs.starburst.io/starburst-galaxy/developer-tools/api/api-auth-token.html) and setting `starburst_url`, `starburst_client_id`, and `starburst_secret_key` in the [connection profile](https://docs.getdbt.com/docs/local/connect-data-platform/trino-setup.md#additional-parameters). #### Usage ##### Documenting columns and relations Supply a [description](https://docs.getdbt.com/reference/resource-properties/description.md) for a model: models/schema.yml ```yml models: - name: dim_customers description: One record per customer columns: - name: customer_id description: Primary key ``` Enable `persist_docs` for columns and relations in your project: dbt\_project.yml ```yml models: +persist_docs: relation: true columns: true ``` Run dbt and observe that the created relation and columns are annotated with your descriptions: [![Relation descriptions in BigQuery](/img/reference/persist_docs_relation.png?v=2 "Relation descriptions in BigQuery")](#)Relation descriptions in BigQuery [![Column descriptions in BigQuery](/img/reference/persist_docs_columns.png?v=2 "Column descriptions in BigQuery")](#)Column descriptions in BigQuery --- ### Platform-specific configs #### [Microsoft Azure Synapse DWH configurations](https://docs.getdbt.com/reference/resource-configs/azuresynapse-configs.md) [Configure index and distribution settings for Microsoft Azure Synapse data warehouse tables in dbt.](https://docs.getdbt.com/reference/resource-configs/azuresynapse-configs.md) --- ### Platform-specific data types Unit tests are designed to test for the expected *values*, not for the data types themselves. dbt takes the value you provide and attempts to cast it to the data type as inferred from the input and output models. How you specify input and expected values in your unit test YAML definitions are largely consistent across data warehouses, with some variation for more complex data types. The following are platform-specific data types: ##### Snowflake ```yml unit_tests: - name: test_my_data_types model: fct_data_types given: - input: ref('stg_data_types') rows: - int_field: 1 float_field: 2.0 str_field: my_string str_escaped_field: "my,cool'string" date_field: 2020-01-02 timestamp_field: 2013-11-03 00:00:00-0 timestamptz_field: 2013-11-03 00:00:00-0 number_field: 3 variant_field: 3 geometry_field: POINT(1820.12 890.56) geography_field: POINT(-122.35 37.55) object_field: {'Alberta':'Edmonton','Manitoba':'Winnipeg'} str_array_field: ['a','b','c'] int_array_field: [1, 2, 3] binary_field: 19E1FFDCCB6CDEE788BF631C1C4905D1 ``` ##### BigQuery ```yml unit_tests: - name: test_my_data_types model: fct_data_types given: - input: ref('stg_data_types') rows: - int_field: 1 float_field: 2.0 str_field: my_string str_escaped_field: "my,cool'string" date_field: 2020-01-02 timestamp_field: 2013-11-03 00:00:00-0 timestamptz_field: 2013-11-03 00:00:00-0 bigint_field: 1 geography_field: 'st_geogpoint(75, 45)' json_field: {"name": "Cooper", "forname": "Alice"} str_array_field: ['a','b','c'] int_array_field: [1, 2, 3] date_array_field: ['2020-01-01'] struct_field: 'struct("Isha" as name, 22 as age)' struct_of_struct_field: 'struct(struct(1 as id, "blue" as color) as my_struct)' struct_array_field: ['struct(st_geogpoint(75, 45) as my_point)', 'struct(st_geogpoint(75, 35) as my_point)'] # Make sure to include all the fields in a BigQuery `struct` within the unit test. # It's not currently possible to use only a subset of columns in a 'struct' ``` ##### Redshift ```yml unit_tests: - name: test_my_data_types model: fct_data_types given: - input: ref('stg_data_types') rows: - int_field: 1 float_field: 2.0 str_field: my_string str_escaped_field: "my,cool'string" date_field: 2020-01-02 timestamp_field: 2013-11-03 00:00:00-0 timestamptz_field: 2013-11-03 00:00:00-0 json_field: '{"bar": "baz", "balance": 7.77, "active": false}' ``` Currently, the `array` is not supported. ##### Spark ```yml unit_tests: - name: test_my_data_types model: fct_data_types given: - input: ref('stg_data_types') rows: - int_field: 1 float_field: 2.0 str_field: my_string str_escaped_field: "my,cool'string" bool_field: true date_field: 2020-01-02 timestamp_field: 2013-11-03 00:00:00-0 timestamptz_field: 2013-11-03 00:00:00-0 int_array_field: 'array(1, 2, 3)' map_field: 'map("10", "t", "15", "f", "20", NULL)' named_struct_field: 'named_struct("a", 1, "b", 2, "c", 3)' ``` ##### Postgres ```yml unit_tests: - name: test_my_data_types model: fct_data_types given: - input: ref('stg_data_types') rows: - int_field: 1 float_field: 2.0 numeric_field: 1 str_field: my_string str_escaped_field: "my,cool'string" bool_field: true date_field: 2020-01-02 timestamp_field: 2013-11-03 00:00:00-0 timestamptz_field: 2013-11-03 00:00:00-0 json_field: '{"bar": "baz", "balance": 7.77, "active": false}' ``` Currently, the `array` is not supported. --- ### Postgres configurations #### Incremental materialization strategies In dbt-postgres, the following incremental materialization strategies are supported: * `append` (default when `unique_key` is not defined) * `merge` * `delete+insert` (default when `unique_key` is defined) * [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md) #### Performance optimizations ##### Unlogged "Unlogged" tables can be considerably faster than ordinary tables, as they are not written to the write-ahead log nor replicated to read replicas. They are also considerably less safe than ordinary tables. See [Postgres docs](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-UNLOGGED) for details. my\_table.sql ```sql {{ config(materialized='table', unlogged=True) }} select ... ``` dbt\_project.yml ```yaml models: +unlogged: true ``` ##### Indexes While Postgres works reasonably well for datasets smaller than about 10m rows, database tuning is sometimes required. It's important to create indexes for columns that are commonly used in joins or where clauses. Table models, incremental models, seeds, snapshots, and materialized views may have a list of `indexes` defined. Each Postgres index can have three components: * `columns` (list, required): one or more columns on which the index is defined * `unique` (boolean, optional): whether the index should be [declared unique](https://www.postgresql.org/docs/9.4/indexes-unique.html) * `type` (string, optional): a supported [index type](https://www.postgresql.org/docs/current/indexes-types.html) (B-tree, Hash, GIN, etc) my\_table.sql ```sql {{ config( materialized = 'table', indexes=[ {'columns': ['column_a'], 'type': 'hash'}, {'columns': ['column_a', 'column_b'], 'unique': True}, ] )}} select ... ``` If one or more indexes are configured on a resource, dbt will run `create index` DDL statement(s) as part of that resource's materialization, within the same transaction as its main `create` statement. For the index's name, dbt uses a hash of its properties and the current timestamp, in order to guarantee uniqueness and avoid namespace conflict with other indexes. ```sql create index if not exists "3695050e025a7173586579da5b27d275" on "my_target_database"."my_target_schema"."indexed_model" using hash (column_a); create unique index if not exists "1bf5f4a6b48d2fd1a9b0470f754c1b0d" on "my_target_database"."my_target_schema"."indexed_model" (column_a, column_b); ``` You can also configure indexes for a number of resources at once: dbt\_project.yml ```yaml models: project_name: subdirectory: +indexes: - columns: ['column_a'] type: hash ``` #### Materialized views The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) with the following configuration parameters: | Parameter | Type | Required | Default | Change Monitoring Support | | ---------------------------------------------------------------------------------------------------------- | ------------------ | -------- | ------- | ------------------------- | | [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) | `<string>` | no | `apply` | n/a | | [`indexes`](#indexes) | `[{<dictionary>}]` | no | `none` | alter | ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: materialized_view +on_configuration_change: apply | continue | fail +indexes: - columns: [<column-name>] unique: true | false type: hash | btree ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: materialized_view on_configuration_change: apply | continue | fail indexes: - columns: [<column-name>] unique: true | false type: hash | btree ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized="materialized_view", on_configuration_change="apply" | "continue" | "fail", indexes=[ { "columns": ["<column-name>"], "unique": true | false, "type": "hash" | "btree", } ] ) }} ``` The [`indexes`](#indexes) parameter corresponds to that of a table, as explained above. It's worth noting that, unlike tables, dbt monitors this parameter for changes and applies the changes without dropping the materialized view. This happens via a `DROP/CREATE` of the indexes, which can be thought of as an `ALTER` of the materialized view. Learn more about these parameters in Postgres's [docs](https://www.postgresql.org/docs/current/sql-creatematerializedview.html). --- ### Postgres Permissions In Postgres, permissions are used to control who can perform certain actions on different database objects. Use SQL statements to manage permissions in a Postgres database. #### Example Postgres permissions The following example provides you with the SQL statements you can use to manage permissions. These examples allow you to run dbt smoothly without encountering permission issues, such as creating schemas, reading existing data, and accessing the information schema. **Note** that `database_name`, `source_schema`, `destination_schema`, and `user_name` are placeholders and you can replace them as needed for your organization's naming convention. ```sql grant connect on database database_name to user_name; -- Grant read permissions on the source schema grant usage on schema source_schema to user_name; grant select on all tables in schema source_schema to user_name; alter default privileges in schema source_schema grant select on tables to user_name; -- Create destination schema and make user_name the owner create schema if not exists destination_schema; alter schema destination_schema owner to user_name; -- Grant write permissions on the destination schema grant usage on schema destination_schema to user_name; grant create on schema destination_schema to user_name; grant insert, update, delete, truncate on all tables in schema destination_schema to user_name; alter default privileges in schema destination_schema grant insert, update, delete, truncate on tables to user_name; ``` Check out the [official documentation](https://www.postgresql.org/docs/current/sql-grant.html) for more information. --- ### pre_clone ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +state: pre_clone: never | if_missing | always snapshots: <resource-path>: +state: pre_clone: never | if_missing | always ``` ##### Properties YAML file models/\<filename>.yml ```yaml models: - name: my_model config: state: pre_clone: never | if_missing | always ``` snapshots/\<filename>.yml ```yaml snapshots: - name: my_snapshot relation: source('my_source', 'my_table') config: state: pre_clone: never | if_missing | always ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( state={ "pre_clone": "never" | "if_missing" | "always" } ) }} ``` #### Definition `pre_clone` controls whether and when dbt State pre-populates incremental models and snapshots in your development environment by cloning their production counterparts before a run. | Value | Behavior | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `never` | Never pre-clone. Incremental models and snapshots start from whatever exists in the development environment. If it doesn't already exist, it's built from scratch. | | `if_missing` (default) | Clone only if the target table does not yet exist in development. After the first clone, subsequent runs build incrementally on top of the cloned data. | | `always` | Clone before every run, regardless of whether the development table already exists. Guarantees development incremental state matches production on every run. | note Non-incremental materializations are never pre-cloned. This setting has no effect on them. #### Examples ##### Always sync development with production Use `always` when you want development to reflect the current production state before every run: models/fct\_orders.yml ```yaml models: - name: fct_orders config: state: pre_clone: always ``` ##### Isolate dev state from production Use `never` when you want development to be fully independent from production: dbt\_project.yml ```yaml models: +state: pre_clone: never ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) --- ### pre-hook & post-hook ##### Models In these examples, we use the `|` symbol to separate two different formatting options for SQL statements in pre-hooks and post-hooks. The first option (without brackets) accepts a single SQL statement as a string, while the second (with brackets) accepts multiple SQL statements as an array of strings. Replace `SQL-STATEMENT` with your SQL. dbt\_project.yml ```yml models: <resource-path>: +pre-hook: SQL-statement | [SQL-statement] +post-hook: SQL-statement | [SQL-statement] ``` models/\<model\_name>.sql ```sql {{ config( pre_hook="SQL-statement" | ["SQL-statement"], post_hook="SQL-statement" | ["SQL-statement"], ) }} select ... ``` models/properties.yml ```yml models: - name: [<model_name>] config: pre_hook: <sql-statement> | [<sql-statement>] post_hook: <sql-statement> | [<sql-statement>] ``` ##### Seeds In these examples, we use the `|` symbol to separate two different formatting options for SQL statements in pre-hooks and post-hooks. The first option (without brackets) accepts a single SQL statement as a string, while the second (with brackets) accepts multiple SQL statements as an array of strings. Replace `SQL-STATEMENT` with your SQL. dbt\_project.yml ```yml seeds: <resource-path>: +pre-hook: SQL-statement | [SQL-statement] +post-hook: SQL-statement | [SQL-statement] ``` seeds/properties.yml ```yml seeds: - name: [<seed_name>] config: pre_hook: <sql-statement> | [<sql-statement>] post_hook: <sql-statement> | [<sql-statement>] ``` ##### Snapshots In these examples, we use the `|` symbol to separate two different formatting options for SQL statements in pre-hooks and post-hooks. The first option (without brackets) accepts a single SQL statement as a string, while the second (with brackets) accepts multiple SQL statements as an array of strings. Replace `SQL-STATEMENT` with your SQL. dbt\_project.yml ```yml snapshots: <resource-path>: +pre-hook: SQL-statement | [SQL-statement] +post-hook: SQL-statement | [SQL-statement] ``` snapshots/snapshot.yml ```yml snapshots: - name: [<snapshot_name>] config: pre_hook: <sql-statement> | [<sql-statement>] post_hook: <sql-statement> | [<sql-statement>] ``` #### Definition A SQL statement (or list of SQL statements) to be run before or after a model, seed, or snapshot is built. Pre- and post-hooks can also call macros that return SQL statements. If your macro depends on values available only at execution time, such as using model configurations or `ref()` calls to other resources as inputs, you will need to [wrap your macro call in an extra set of curly braces](https://docs.getdbt.com/best-practices/dont-nest-your-curlies.md#an-exception). ##### Why would I use hooks? dbt aims to provide all the boilerplate SQL you need (DDL, DML, and DCL) via out-of-the-box functionality, which you can configure quickly and concisely. In some cases, there may be SQL that you want or need to run, specific to functionality in your data platform, which dbt does not (yet) offer as a built-in feature. In those cases, you can write the exact SQL you need, using dbt's compilation context, and pass it into a `pre-` or `post-` hook to run before or after your model, seed, or snapshot. ###### The render method The `.render()` method is generally used to resolve or evaluate Jinja expressions (such as `{{ source(...) }}`) during runtime. When using the `--empty flag`, dbt may skip processing `ref()` or `source()` for optimization. To avoid compilation errors and to explicitly tell dbt to process a specific relation (`ref()` or `source()`), use the `.render()` method in your model file. For example: models.sql ```jinja {{ config( pre_hook = [ "alter external table {{ source('sys', 'customers').render() }} refresh" ] ) }} select ... ``` #### Examples ##### \[Redshift] Unload one model to S3 model.sql ```sql {{ config( post_hook = "unload ('select from {{ this }}') to 's3:/bucket_name/{{ this }}" ) }} select ... ``` See: [Redshift docs on `UNLOAD`](https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html) ##### \[Apache Spark] Analyze tables after creation dbt\_project.yml ```yml models: jaffle_shop: # this is the project name marts: finance: +post-hook: # this can be a list - "analyze table {{ this }} compute statistics for all columns" # or call a macro instead - "{{ analyze_table() }}" ``` See: [Apache Spark docs on `ANALYZE TABLE`](https://spark.apache.org/docs/latest/sql-ref-syntax-aux-analyze-table.html) ##### Additional examples We've compiled some more in-depth examples [here](https://docs.getdbt.com/docs/build/hooks-operations.md#additional-examples). #### Usage notes ##### Hooks are cumulative If you define hooks in both your `dbt_project.yml` and in the `config` block of a model, both sets of hooks will be applied to your model. ##### Execution ordering If multiple instances of any hooks are defined, dbt will run each hook using the following ordering: 1. Hooks from dependent packages will be run before hooks in the active package. 2. Hooks defined within the model itself will be run after hooks defined in `dbt_project.yml`. 3. Hooks within a given context will be run in the order in which they are defined. ##### Transaction behavior If you're using an adapter that uses transactions (namely Postgres or Redshift), it's worth noting that by default hooks are executed inside of the same transaction as your model being created. There may be occasions where you need to run these hooks *outside* of a transaction, for example: * You want to run a `VACUUM` in a `post-hook`, however, this cannot be executed within a transaction ([Redshift docs](https://docs.aws.amazon.com/redshift/latest/dg/r_VACUUM_command.html#r_VACUUM_usage_notes)) * You want to insert a record into an audit table at the start of a run and do not want that statement rolled back if the model creation fails. To achieve this behavior, you can use one of the following syntaxes: * Important note: Do not use this syntax if you are using a database where dbt does not support transactions. This includes databases like Snowflake, BigQuery, and Spark or Databricks. ##### Use before\_begin and after\_commit ###### Config block: use the `before_begin` and `after_commit` helper macros models/\<modelname>.sql ```sql {{ config( pre_hook=before_begin("SQL-statement"), post_hook=after_commit("SQL-statement") ) }} select ... ``` ##### Use a dictionary ###### Config block: use a dictionary models/\<modelname>.sql ```sql {{ config( pre_hook={ "sql": "SQL-statement", "transaction": False }, post_hook={ "sql": "SQL-statement", "transaction": False } ) }} select ... ``` ##### Use dbt\_project.yml ###### `dbt_project.yml`: Use a dictionary dbt\_project.yml ```yml models: +pre-hook: sql: "SQL-statement" transaction: false +post-hook: sql: "SQL-statement" transaction: false ``` --- ### Print output ##### Suppress `print()` messages in stdout By default, dbt includes [`print()`](https://docs.getdbt.com/reference/dbt-jinja-functions/print.md) messages in standard out (stdout). You can use the (Applies to dbt v1.11 and later) `DBT_ENGINE_PRINT` environment variable to prevent these messages from showing up in stdout. Syntax deprecation The original `DBT_NO_PRINT` environment variable has been deprecated, starting with dbt v1.5. Backward compatibility is supported but will be removed in an as-of-yet-undetermined future release. Supply `--no-print` flag to `dbt run` to suppress `print()` messages from showing in stdout. ```text dbt run --no-print ``` ##### Printer width By default, dbt will print out lines padded to 80 characters wide. You can change this setting by adding the following to your `profiles.yml` file: profiles.yml ```yaml config: printer_width: 120 ``` ##### Print color By default, dbt will colorize the output it prints in your terminal. You can turn this off by adding the following to your `profiles.yml` file: profiles.yml ```yaml config: use_colors: False ``` ```text dbt run --use-colors dbt run --no-use-colors ``` You can set the color preferences for the file logs only within `profiles.yml` or using the `--use-colors-file / --no-use-colors-file` flags. profiles.yml ```yaml config: use_colors_file: False ``` ```text dbt run --use-colors-file dbt run --no-use-colors-file ``` --- ### profile dbt\_project.yml ```yml profile: string ``` #### Definition The profile your dbt project should use to connect to your data warehouse. * If you are developing in dbt: This configuration is not applicable * If you are developing locally: This configuration is required, unless a command-line option like [`--profile`](https://docs.getdbt.com/docs/local/profiles.yml.md#overriding-profiles-and-targets) is supplied. The `--profile` flag overrides the profile set in `dbt_project.yml`. #### Related guides * [Connecting to your warehouse using the command line](https://docs.getdbt.com/docs/local/profiles.yml.md#connecting-to-your-warehouse-using-the-command-line) #### Recommendation Often an organization has only one data warehouse, so it is sensible to use your organization's name as a profile name, in `snake_case`. For example: * `profile: acme` * `profile: jaffle_shop` It is also reasonable to include the name of your warehouse technology in your profile name, particularly if you have multiple warehouses. For example: * `profile: acme_snowflake` * `profile: jaffle_shop_bigquery` * `profile: jaffle_shop_redshift` --- ### Programmatic invocations Programmatic invocations let you call dbt commands from Python scripts and applications, instead of running them in a shell. This is useful when you want to embed dbt runs into a larger application or workflow, while still using the same command surface area as the dbt CLI. Common use cases include: * Running dbt as part of a Python application or service * Integrating dbt runs into orchestration workflows * Building internal tools that need to run dbt commands and inspect results Refer to the [dbt v1 package on PyPI](https://pypi.org/project/dbt-core/) to install the official Python package for dbt v1 if you haven't done so already. (Applies to dbt v2.0 and later) ```python from dbt.cli.main import dbtRunner, dbtRunnerResult # initialize dbt = dbtRunner() # create CLI args as a list of strings cli_args = ["run", "--select", "tag:my_tag"] # run the command res: dbtRunnerResult = dbt.invoke(cli_args) # inspect the results for r in res.result: print(f"{r.unique_id}: {r.status}") ``` For implementation details, refer to the [`dbt-python` crate](https://github.com/dbt-labs/dbt/tree/main/crates/dbt-python) in the dbt v1 repository. #### Supported arguments `dbtRunner.invoke` accepts the same arguments as the dbt CLI. The first positional argument is the command (for example, `run`, `build`, `test`), followed by any flags and options you would normally pass on the command line. For example, `dbt.invoke(["run", "--select", "tag:my_tag"])` is equivalent to running `dbt run --select tag:my_tag`. There is no separate, dbtRunner‑specific list of arguments; the authoritative source for available options is the CLI help reference (`dbt --help`, `dbt run --help`, and so on) and the [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md) documentation. ```python from dbt.cli.main import dbtRunner dbt = dbtRunner() # equivalent ways to pass arguments dbt.invoke(["run", "--select", "tag:my_tag"]) dbt.invoke(["run"], select="tag:my_tag") ``` #### Parallel execution not supported dbt doesn't support [safe parallel execution](https://docs.getdbt.com/reference/dbt-commands.md#parallel-execution) for multiple invocations in the same process. Running multiple dbt commands concurrently in one process is unsafe and officially discouraged, and requires a wrapping process to manage subprocesses. This is because: * Running concurrent commands can unexpectedly interact with the data platform. For example, running `dbt run` and `dbt build` for the same models simultaneously could lead to unpredictable results. * Each `dbt` command interacts with global Python variables. To ensure safe operation, commands need to be executed in separate processes, for example by spawning subprocesses or using Celery for orchestration. For [safe parallel execution](https://docs.getdbt.com/reference/dbt-commands.md#available-commands), you can use the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), both of which do that additional work to manage concurrency (multiple processes) on your behalf. (Applies to dbt v2.0 and later) In v2, invocations are serialized through thread-level locks, so multiple invocations can't run concurrently within the same process. (In v1, parallel execution was unsupported but there was no locking, so invocations could still run in multithreaded mode.) As in v1, you can still parallelize by using multiprocessing to run each invocation in a separate process. #### `dbtRunnerResult` Each command returns a `dbtRunnerResult` object with the following attributes: * `success` (bool): Whether the command succeeded. * `result`: When the command completes (successfully or with handled errors), it returns the command's result(s). The return type varies by command. * `exception`: When the dbt invocation encounters an unhandled error and does not complete, the exception that was raised. * `catalog` (v2 only): The catalog that the command produces when you request catalog generation. (Applies to dbt v2.0 and later) The v2 engine is implemented in Rust, so `exception` no longer contains the exact Python exception object raised by dbt. Instead, the caught error message is forwarded under an exception type: * If the invocation fails at a foreign function interface (FFI) boundary before the engine picks up the invocation, `exception` contains an unwrapped exception type, such as `ValueError` or `RuntimeError`. * If the invocation fails inside the engine, `exception` is a `DbtRunnerError`. v2 also adds a top-level `catalog` attribute to `dbtRunnerResult` when catalog generation is requested. In v1, `catalog.json` was only created when you ran `dbt docs generate`. In v2, you can generate the catalog as part of any command by passing the [`--write-catalog` flag](https://docs.getdbt.com/reference/commands/cmd-docs.md?version=2.0#--write-catalog-flag). For example, `dbt run --write-catalog` populates both `dbtRunnerResult.result` and `dbtRunnerResult.catalog`. There is a one-to-one correspondence between [CLI exit codes](https://docs.getdbt.com/reference/exit-codes.md) and the `dbtRunnerResult` returned by a programmatic invocation: | Scenario | CLI Exit Code | `success` | `result` | `exception` | | ----------------------------------------------------------------------------------------------------- | ------------- | --------- | ----------------- | ----------- | | Invocation completed without error | 0 | `True` | varies by command | `None` | | Invocation completed with at least one handled error (for example, test failure or model build error) | 1 | `False` | varies by command | `None` | | Unhandled error. Invocation did not complete, and returns no results. | 2 | `False` | `None` | Exception | #### Commitments and caveats We're making an ongoing commitment to providing a Python entry point at functional parity with dbt v1's CLI. We reserve the right to change the underlying implementation used to achieve that goal. We expect that the current implementation will unlock real use cases in the short- and medium-term while we work on a set of stable, long-term interfaces that will ultimately replace it. In particular, the objects returned by each command in `dbtRunnerResult.result` are not fully contracted, and therefore liable to change. Some of the returned objects are partially documented, because they overlap in part with the contents of [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md). As Python objects, they contain many more fields and methods than what's available in the serialized JSON artifacts. These additional fields and methods should be considered **internal and liable to change in future versions of dbt.** #### Advanced usage patterns caution The syntax and support for these patterns are liable to change in future versions of dbt. The goal of `dbtRunner` is to offer parity with CLI workflows within a programmatic environment. There are a few advanced usage patterns that extend what's possible with the CLI. (Applies to dbt v2.0 and later) ##### Reusing objects Manifest injection isn't supported in v2. You can't pass a pre-constructed `Manifest` into `dbtRunner`. ##### Registering callbacks Registering callbacks on dbt's `EventManager` isn't supported in v2. ##### Overriding parameters Pass in parameters as keyword arguments, instead of a list of CLI-style strings. At present, dbt will not do any validation or type coercion on your inputs. The command must be specified, in a list, as the first positional argument. ```python from dbt.cli.main import dbtRunner dbt = dbtRunner() # these are equivalent dbt.invoke(["--fail-fast", "run", "--select", "tag:my_tag"]) dbt.invoke(["run"], select=["tag:my_tag"], fail_fast=True) ``` --- ### Project flags dbt\_project.yml ```yaml flags: <global_config>: <value> ``` Reference the [table of all flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags) to see which global configs are available for setting in [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md). The `flags` dictionary is the *only* place you can opt out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md), while the legacy behavior is still supported. #### Config precedence There are multiple ways of setting flags, which depend on the use case: * **[CLI options](https://docs.getdbt.com/reference/global-configs/command-line-options.md):** Define behavior specific to *this invocation*. Supported for all dbt commands. * **[Environment variables](https://docs.getdbt.com/reference/global-configs/environment-variable-configs.md):** Define different behavior in different runtime environments (development vs. production vs. [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md)), or different behavior for different users in development (based on personal preferences). * **[Project-level `flags` in `dbt_project.yml`](https://docs.getdbt.com/reference/global-configs/project-flags.md):** Define version-controlled defaults for everyone running this project. Also, opt in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) to manage your migration off legacy functionality. * **[User settings (`~/.dbt/user_settings.yml`)](https://docs.getdbt.com/reference/global-configs/user-settings.md):** Define personal preferences that apply across all projects on your machine. Written automatically by `dbt login`. The most specific setting "wins." CLI options take the highest precedence, followed by environment variables, then `dbt_project.yml`, and finally `user_settings.yml`. If you set the flag in none of those places, it will use the default value defined within dbt. Most flags can be set in all three places: ```yaml # dbt_project.yml flags: # set default for running this project -- anywhere, anytime, by anyone fail_fast: true ``` (Applies to dbt v1.11 and later) ```bash # set this environment variable to 'True' (bash syntax) export DBT_ENGINE_FAIL_FAST=1 dbt run ``` ```bash dbt run --fail-fast # set to True for this specific invocation dbt run --no-fail-fast # set to False ``` There are two categories of exceptions: 1. **Flags setting file paths:** Flags for file paths that are relevant to runtime execution (for example, `--log-path` or `--state`) cannot be set in `dbt_project.yml`. To override defaults, pass CLI options or set environment variables ((Applies to dbt v1.11 and later) `DBT_ENGINE_LOG_PATH` and `DBT_ENGINE_STATE`). Flags that tell dbt where to find project resources (for example, `model-paths`) are set in `dbt_project.yml`, but as a top-level key, outside the `flags` dictionary; these configs are expected to be fully static and never vary based on the command or execution environment. 2. **Opt-in flags:** Flags opting in or out of [behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) can *only* be defined in `dbt_project.yml`. These are intended to be set in version control and migrated via pull/merge request. Their values should not diverge indefinitely across invocations, environments, or users. --- ### Project hooks with source freshness Removed in v2 This flag was removed in v2. The new behavior is always enabled. If you're upgrading, remove this flag from your `dbt_project.yml`. | source\_freshness\_run\_project\_hooks | dbt **v1 Latest** | dbt v1 | | -------------------------------------- | ----------------- | ------ | | Introduced | 2024.03 | 1.8.0 | | Matured (default → `true`) | 2025.05 | 1.10.0 | | Removed | — | v2.0 | Project hooks ([`on-run-start` / `on-run-end`](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md)) now run as part of the `dbt source freshness` command by default. Previously, hooks did not execute during `dbt source freshness`. If you have hooks that should not run before or after `dbt source freshness`, add a conditional check: dbt\_project.yml ```yaml on-run-start: - "{{ ... if flags.WHICH != 'freshness' }}" ``` --- ### Project Parsing #### Related documentation * The `dbt parse` [command](https://docs.getdbt.com/reference/commands/parse.md) * Partial parsing [profile config](https://docs.getdbt.com/docs/local/profiles.yml.md#partial_parse) and [CLI flags](https://docs.getdbt.com/reference/global-configs/parsing.md) * Parsing [CLI flags](https://docs.getdbt.com/reference/global-configs/parsing.md) #### What is parsing? At the start of every dbt invocation, dbt reads all the files in your project, extracts information, and constructs a manifest containing every object (model, source, macro, etc). Among other things, dbt uses the `ref()`, `source()`, and `config()` macro calls within models to set properties, infer dependencies, and construct your project's DAG. Parsing projects can be slow, especially as projects get bigger—hundreds of models, thousands of files—which is frustrating in development. There are a handful of ways to optimize dbt performance today: * LibYAML bindings for PyYAML * Partial parsing, which avoids re-parsing unchanged files between invocations * A static parser, which extracts information from simple models much more quickly * [RPC server](https://docs.getdbt.com/reference/commands/rpc.md), which keeps a manifest in memory, and re-parses the project at server startup/hangup These optimizations can be used in combination to reduce parse time from minutes to seconds. At the same time, each has some known limitations, so they are disabled by default. #### PyYAML + LibYAML dbt uses [PyYAML](https://pyyaml.org/wiki/PyYAML) to read and validate YAML files in your project. PyYAML is written in pure Python, but it can leverage [LibYAML](https://pyyaml.org/wiki/LibYAML) (written in C, much faster) if it's available in your system. Whenever it parses your project, dbt will always check first to see if LibYAML is available. You can test to see if LibYAML is installed by running this command in the environment where you've installed dbt: ```text python -c "from yaml import CLoader" ``` #### Partial parsing After parsing your project, dbt stores an internal project manifest in a file called `partial_parse.msgpack`. When partial parsing is enabled, dbt will use that internal manifest to determine which files have been changed (if any) since it last parsed the project. Then, it will *only* parse the changed files, or files related to those changes. Starting in v1.0, partial parsing is **on** by default. In development, partial parsing can significantly reduce the time spent waiting at the start of a run, which translates to faster dev cycles and iteration. The [`PARTIAL_PARSE` global config](https://docs.getdbt.com/reference/global-configs/parsing.md) can be enabled or disabled via `profiles.yml`, environment variable, or CLI flag. (Applies to dbt v2.0 and later) dbt v2 and partial parsing dbt v2 job runs no longer support the `--partial-parse` and `--no-partial-parse` CLI flags. If you pass them (for example, from a dbt v1 command or script), dbt logs deprecation warning `dbt1700`. Remove these flags from your dbt v2 job commands. For more information, refer to [Deprecated flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) in the guide to upgrading to dbt v2. ##### Known limitations Parse-time attributes (dependencies, configs, and resource properties) are resolved using the parse-time context. When partial parsing is enabled, and certain context variables change, those attributes will *not* be re-resolved, and are likely to become stale. In particular, you may see incorrect results if these attributes depend on "volatile" context variables, such as [`run_started_at`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_started_at.md), [`invocation_id`](https://docs.getdbt.com/reference/dbt-jinja-functions/invocation_id.md), or [flags](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md). These variables are likely (or even guaranteed!) to change in each invocation. dbt Labs *strongly discourages* you from using these variables to set parse-time attributes (dependencies, configs, and resource properties). Starting in v1.0, dbt *will* detect changes in environment variables. It will selectively re-parse only the files that depend on that [`env_var`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) value. (If the env var is used in `profiles.yml` or `dbt_project.yml`, a full re-parse is needed.) However, dbt will *not* re-render **descriptions** that include env vars. If your descriptions include frequently changing env vars (this is highly uncommon), we recommend that you fully re-parse when generating documentation: `dbt docs generate --no-partial-parse`. If certain inputs change between runs, dbt will trigger a full re-parse. The results will be correct, but the full re-parse may be quite slow. Today those inputs are: * `--vars` * `profiles.yml` content (or `env_var` values used within) * `dbt_project.yml` content (or `env_var` values used within) * installed packages * dbt version * certain widely-used macros (for example, [builtins](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md), overrides, or `generate_x_name` for `database`/`schema`/`alias`) If you're triggering [CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md) job runs, the benefits of partial parsing are not applicable to new pull requests (PR) or new branches. However, they are applied on subsequent commits to the new PR or branch. When partial parsing is enabled, dbt may occasionally fail or incorrectly parse the project causing: * Nodes (for example, models, sources) to not be found. * Configurations to be set incorrectly (for example, different from what is defined in a model's `schema.yml` file). If you get into this state, you can trigger a full re-parse using any of the following options: * Run the dbt command with `--no-partial-parse`. * Delete the `target/partial_parse.msgpack` file by running `dbt clean`. You can disable partial parsing entirely by setting the `PARTIAL_PARSE` global config to `false`. #### Static parser At parse time, dbt needs to extract the contents of `ref()`, `source()`, and `config()` from all models in the project. Traditionally, dbt has extracted those values by rendering the Jinja in every model file, which can be slow. We statically analyze model files leveraging [`tree-sitter`](https://github.com/tree-sitter/tree-sitter). You can see the code for an initial Jinja2 grammar [here](https://github.com/dbt-labs/tree-sitter-jinja2). The static parser is **on** by default. We believe it can offer *some* speed up to 95% of projects. You may optionally turn it off using the [`STATIC_PARSER` global config](https://docs.getdbt.com/reference/global-configs/parsing.md). For now, the static parser only works with models, and models whose Jinja is limited to those three special macros (`ref`, `source`, `config`). The static parser is at least 3x faster than a full Jinja render. Based on testing with data from dbt, we believe the current grammar can statically parse 60% of models in the wild. So for the average project, we'd hope to see a 40% speedup in the model parser. --- ### Project-level configuration for analyses | require\_corrected\_analysis\_fqns | dbt **v1 Latest** | dbt v1 | | ---------------------------------- | ----------------- | ------ | | Introduced | 2026.3 | 1.12.0 | | Matured (default → `true`) | — | — | | Removed | — | — | Previously, project-level configuration for [analyses](https://docs.getdbt.com/docs/build/analyses.md) in `dbt_project.yml` was silently ignored. Fully qualified names (FQNs) for analyses also contained an extra `analyses` path segment that was inconsistent with other resource types. When `require_corrected_analysis_fqns` is set to `true`, dbt: * Routes analysis configurations from the `analyses` block in `dbt_project.yml`, enabling project-level configurations to take effect. * Removes the extra FQN segment so that analysis FQNs are consistent with other resource types (for example, `your_project.my_analysis` instead of `your_project.analyses.my_analysis`). To configure analyses at the project level, set the [`require_corrected_analysis_fqns`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md) flag to `true` and add an `analyses` block in your `dbt_project.yml`. The project-level configuration applies to existing analyses in the `analyses/` folder — for example, setting `+enabled: false` disables them all. dbt\_project.yml ```yaml flags: require_corrected_analysis_fqns: true analyses: +enabled: true | false ``` For more information, refer to [Analyses](https://docs.getdbt.com/docs/build/analyses.md) and [Analysis properties](https://docs.getdbt.com/reference/analysis-properties.md). --- ### Putting it together ```bash dbt run --select "my_package.*+" # select all models in my_package and their children dbt run --select "+some_model+" # select some_model and all parents and children dbt run --select "tag:nightly+" # select "nightly" models and all children dbt run --select "+tag:nightly+" # select "nightly" models and all parents and children dbt run --select "@source:snowplow" # build all models that select from snowplow sources, plus their parents dbt test --select "config.incremental_strategy:insert_overwrite,test_name:unique" # execute all `unique` tests that select from models using the `insert_overwrite` incremental strategy ``` This can get complex! Let's say I want a nightly run of models that build off snowplow data and feed exports, while *excluding* the biggest incremental models (and one other model, to boot). ```bash dbt run --select "@source:snowplow,tag:nightly models/export" --exclude "package:snowplow,config.materialized:incremental export_performance_timing" ``` This command selects all models that: * Select from snowplow sources, plus their parents, *and* are tagged "nightly" * Are defined in the `export` model subfolder Except for models that are: * Defined in the snowplow package and materialized incrementally * Named `export_performance_timing` --- ### Python model configurations Use `dbt.config()` to set configs for a Python model. This page lists the parameters needed. For how to write a Python model, refer to [Python models](https://docs.getdbt.com/docs/build/python-models.md). Literal values only `dbt.config()` only accepts literal values (strings, booleans, and numbers). For anything more complex, set the config in a [properties YAML file](https://docs.getdbt.com/reference/resource-properties/config.md). Refer to [Configuring Python models](https://docs.getdbt.com/docs/build/python-models.md#configuring-python-models). #### Common parameters The following parameters work on every platform that supports Python models. | Parameter | Type | Description | | ------------------------------------------------------------------------------------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`materialized`](https://docs.getdbt.com/reference/resource-configs/materialized.md) | `<string>` | How dbt builds the model, such as `table` or `incremental`. Python models don't support `view`. | | [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md) | `<list>` | Tags to apply to the model. | | [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) | `<dictionary>` | Custom metadata. Read it with `dbt.config.meta_get()`. | | `packages` | `<list>` | Packages to install, with optional version pins such as `numpy==1.23.1`. Required on some platforms. Refer to [Configuring packages](https://docs.getdbt.com/docs/build/python-models.md#configuring-packages). | Read any config back inside your model with `dbt.config.get()`. #### Platform-specific parameters ##### Snowflake | Parameter | Type | Default | Description | | ------------------------------ | -------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `python_version` | `<string>` | `3.10` | Python version the Snowpark model runs on. | | `packages` | `<list>` | `[]` | Anaconda packages to install when the model runs. dbt always adds `snowflake-snowpark-python`. | | `imports` | `<list>` | `[]` | Staged files, such as zip files, to make available to your model. Refer to [Third-party Snowflake packages](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#third-party-snowflake-packages). | | `external_access_integrations` | `<list>` | `[]` | Snowflake external access integrations that let your model reach external APIs. | | `secrets` | `<dictionary>` | `{}` | Maps a variable name in your code to a Snowflake secret. Use it with `external_access_integrations`. | | `use_anonymous_sproc` | `<boolean>` | `True` | When `True`, dbt runs your model as an anonymous stored procedure. When `False`, dbt creates a named procedure (`<model_name>__dbt_sp`), calls it, then drops it. | For more details and a more comprehensive list, refer to [Snowflake configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#additional-parameters). ##### Databricks | Parameter | Type | Default | Description | | --------------------- | -------------- | --------------------- | ------------------------------------------------------------------------------------------------------- | | `submission_method` | `<string>` | `all_purpose_cluster` | How dbt submits the job: `all_purpose_cluster`, `job_cluster`, `serverless_cluster`, or `workflow_job`. | | `create_notebook` | `<boolean>` | `false` | With `all_purpose_cluster`, upload a notebook and run it as a job instead of using the Command API. | | `cluster_id` | `<string>` | `None` | ID of an existing all-purpose cluster to run on. | | `http_path` | `<string>` | `None` | Path of an existing all-purpose cluster to run on. | | `job_cluster_config` | `<dictionary>` | `{}` | Settings for the new cluster used by `job_cluster` and `workflow_job`. | | `timeout` | `<integer>` | `0` (no timeout) | Maximum time to wait for the command or job to run. | | `packages` | `<list>` | `[]` | Packages to install on the cluster. | | `index_url` | `<string>` | `None` (uses PyPI) | URL to install `packages` from. | | `additional_libs` | `<list>` | `[]` | Libraries to attach to the job. | | `access_control_list` | `<list>` | none | Access control for the job. | | `python_job_config` | `<dictionary>` | `{}` | Extra job and workflow settings. | Not every parameter works with every submission method. For more details and a more comprehensive list, refer to [Databricks configurations](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#python-submission-methods). ##### BigQuery | Parameter | Type | Default | Description | | ----------------------- | ----------- | --------- | ------------------------------------------------------------ | | `submission_method` | `<string>` | none | Where the job runs: `serverless`, `bigframes`, or `cluster`. | | `dataproc_cluster_name` | `<string>` | none | Dataproc cluster to run on. Required for `cluster`. | | `dataproc_region` | `<string>` | none | Google Cloud region where Dataproc is enabled. | | `gcs_bucket` | `<string>` | none | Cloud Storage bucket for job artifacts. | | `compute_region` | `<string>` | none | Region of the job. | | `notebook_template_id` | `<integer>` | none | Colab Enterprise runtime template. | | `packages` | `<list>` | none | Packages to `pip install`. `bigframes` only. | | `timeout` | `<integer>` | none | Maximum run time in seconds. | | `enable_list_inference` | `<boolean>` | `True` | Let PySpark read multiple records in one operation. | | `intermediate_format` | `<string>` | `parquet` | File format for intermediate writes: `parquet` or `orc`. | For more details and a more comprehensive list, refer to [BigQuery configurations](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#additional-parameters). --- ### query-comment dbt\_project.yml ```yml query-comment: string ``` The `query-comment` configuration also accepts a dictionary input, like so: dbt\_project.yml ```yml models: my_dbt_project: +materialized: table query-comment: comment: string append: true | false job-label: true | false # BigQuery only ``` #### Definition A string to inject as a comment in each query that dbt runs against your database. This comment can attribute SQL statements to specific dbt resources like models and tests. The `query-comment` configuration can also call a macro that returns a string. #### Default By default, dbt automatically inserts a JSON comment in each query it runs. This comment includes metadata such as the dbt version, profile and target names, and node ID for the resource generating the query. * For Snowflake, the comment appears at the *end* of the query. This prevents the comment from being stripped during processing. * For other adapters, the comment appears at the *beginning* of the query. For example: ```sql /* {"app": "dbt", "dbt_version": "1.10.0rc2", "profile_name": "debug", "target_name": "dev", "node_id": "model.dbt2.my_model"} */ create view analytics.analytics.orders as ( select ... ); ``` #### Using the dictionary syntax The dictionary syntax includes two keys: * `comment` (optional, for more information, refer to the [default](#default) section): The string to be injected into a query as a comment. * `append` (optional, default=`false`): Whether a comment should be appended (added to the bottom of a query) or not (i.e. added to the top of a query). By default, comments are added to the top of queries (i.e. `append: false`). This syntax is useful on databases like Snowflake which [remove leading SQL comments](https://docs.snowflake.com/en/release-notes/2017-04.html#queries-leading-comments-removed-during-execution). #### Examples ##### Prepend a static comment The following example injects a comment that reads `/* executed by dbt */` into the header of the SQL queries that dbt runs. dbt\_project.yml ```yml query-comment: "executed by dbt" ``` **Example output:** ```sql /* executed by dbt */ select ... ``` ##### Disable query comments dbt\_project.yml ```yml query-comment: ``` Or: dbt\_project.yml ```yml query-comment: null ``` ##### Prepend a dynamic comment The following example injects a comment that varies based on the configured `user` specified in the active dbt target. dbt\_project.yml ```yml query-comment: "run by {{ target.user }} in dbt" ``` **Example output:** ```sql /* run by drew in dbt */ select ... ``` ##### Append the default comment The following example uses the dictionary syntax to append (rather than prepend) the default comment. Note that the `comment:` field is omitted to allow the default to be appended. dbt\_project.yml ```yaml query-comment: append: True ``` **Example output:** ```sql select ... /* {"app": "dbt", "dbt_version": "1.6.0rc2", "profile_name": "debug", "target_name": "dev", "node_id": "model.dbt2.my_model"} */ ; ``` ##### BigQuery: include query comment items as job labels If `query-comment.job-label` is set to true, dbt will include the query comment items, if a dictionary, or the comment string, as job labels on the query it executes. These will be included in addition to labels specified in the [BigQuery-specific config](https://docs.getdbt.com/reference/project-configs/query-comment.md#bigquery-include-query-comment-items-as-job-labels). dbt\_project.yml ```yaml query-comment: job-label: True ``` ##### Append a custom comment The following example uses the dictionary syntax to append (rather than prepend) a comment that varies based on the configured `user` specified in the active dbt target. dbt\_project.yml ```yaml query-comment: comment: "run by {{ target.user }} in dbt" append: True ``` **Example output:** ```sql select ... /* run by drew in dbt */ ; ``` ##### Intermediate: Use a macro to generate a comment The `query-comment` config can reference macros in your dbt project. Simply create a macro with any name (`query_comment` is a good start!) in your `macros` directory, like so: macros/query\_comment.sql ```jinja2 {% macro query_comment() %} dbt {{ dbt_version }}: running {{ node.unique_id }} for target {{ target.name }} {% endmacro %} ``` Then call the macro in your `dbt_project.yml` file. Make sure you quote the macro to avoid the YAML parser from trying to interpret the `{` as the start of a dictionary. dbt\_project.yml ```yaml query-comment: "{{ query_comment() }}" ``` ##### Advanced: Use a macro to generate a comment The following example shows a JSON query comment which can be parsed to understand the performance characteristics of your dbt project. macros/query\_comment.sql ```jinja2 {% macro query_comment(node) %} {%- set comment_dict = {} -%} {%- do comment_dict.update( app='dbt', dbt_version=dbt_version, profile_name=target.get('profile_name'), target_name=target.get('target_name'), ) -%} {%- if node is not none -%} {%- do comment_dict.update( file=node.original_file_path, node_id=node.unique_id, node_name=node.name, resource_type=node.resource_type, package_name=node.package_name, relation={ "database": node.database, "schema": node.schema, "identifier": node.identifier } ) -%} {% else %} {%- do comment_dict.update(node_id='internal') -%} {%- endif -%} {% do return(tojson(comment_dict)) %} {% endmacro %} ``` As above, call this macro as follows: dbt\_project.yml ```yaml query-comment: "{{ query_comment(node) }}" ``` #### Compilation context The following context variables are available when generating a query comment: | Context Variable | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | dbt\_version | The version of dbt being used. For details about release versioning, refer to [Versioning](https://docs.getdbt.com/reference/commands/version.md#versioning). | | env\_var | See [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) | | modules | See [modules](https://docs.getdbt.com/reference/dbt-jinja-functions/modules.md) | | run\_started\_at | When the dbt invocation began | | invocation\_id | A unique ID for the dbt invocation | | fromjson | See [fromjson](https://docs.getdbt.com/reference/dbt-jinja-functions/fromjson.md) | | tojson | See [tojson](https://docs.getdbt.com/reference/dbt-jinja-functions/tojson.md) | | log | See [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) | | var | See [var](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) | | target | See [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) | | connection\_name | A string representing the internal name for the connection. This string is generated by dbt. | | node | A dictionary representation of the parsed node object. Use `node.unique_id`, `node.database`, `node.schema`, and so on. | Note: The `var()` function in `query-comment` macros only access variables passed through the `--vars` argument in the CLI. Variables defined in the vars block of your `dbt_project.yml` are not accessible when generating query comments. --- ### quote_args Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> config: snowflake: quote_args: true | false # optional, default: true ``` #### Definition When creating JavaScript user-defined functions (UDFs) on Snowflake, `quote_args` controls whether argument names are quoted in the `CREATE FUNCTION` statement. Defaults to `true`. When `quote_args` is `true`, Snowflake quotes argument names, which preserves their exact casing. Inside the function body, reference arguments using the same case as defined in the YAML (for example, `a_string`). When `quote_args` is `false`, argument names are not quoted. Snowflake uppercases them automatically, so you must reference them in uppercase inside the function body (for example, `A_STRING`). This config applies to JavaScript UDFs on Snowflake only. It has no effect on other adapters or languages. #### Example In this example, `quote_args` is set to `false` so argument names are not quoted in the generated `CREATE FUNCTION` statement: functions/is\_positive\_int.yml ```yaml functions: - name: is_positive_int description: Returns 1 if a string represents a positive integer, else 0 config: snowflake: quote_args: false arguments: - name: a_string data_type: string returns: data_type: integer ``` functions/is\_positive\_int.js ```js return /^[0-9]+$/.test(A_STRING) ? 1 : 0; ``` With `quote_args: false`, dbt generates: ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INTEGER LANGUAGE JAVASCRIPT AS $$ return /^[0-9]+$/.test(A_STRING) ? 1 : 0; $$; ``` With the default `quote_args: true`, dbt generates: ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int("a_string" STRING) RETURNS INTEGER LANGUAGE JAVASCRIPT AS $$ return /^[0-9]+$/.test(a_string) ? 1 : 0; $$; ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [volatility](https://docs.getdbt.com/reference/resource-configs/volatility.md) * [type](https://docs.getdbt.com/reference/resource-configs/type.md) * [arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) * [returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### quote_columns #### Definition An optional seed configuration, used to determine whether column names in the seed file should be quoted when the table is created. * When `True`, dbt will quote the column names defined in the seed file when building a table for the seed, preserving casing. * When `False`, dbt will not quote the column names defined in the seed file. * When not set, it will vary by adapter whether or not column names are quoted. #### Usage ##### Globally quote all seed columns dbt\_project.yml ```yml seeds: +quote_columns: true ``` ##### Only quote seeds in the `seeds/mappings` directory. For a project with: * `name: jaffle_shop` in the `dbt_project.yml` file * `seed-paths: ["seeds"]` in the `dbt_project.yml` file dbt\_project.yml ```yml seeds: jaffle_shop: mappings: +quote_columns: true ``` Or: seeds/properties.yml ```yml seeds: - name: mappings config: quote_columns: true ``` #### Recommended configuration * Explicitly set this value if using seed files. * Apply the configuration globally rather than to individual projects/seeds. * Set `quote_columns: false` *unless* your column names include a special character or casing needs to be preserved. In that case, consider renaming your seed columns (this will simplify code downstream) --- ### Record timing info The `-r` or `--record-timing-info` flag saves performance profiling information to a file. This file can be visualized with `snakeviz` to understand the performance characteristics of a dbt invocation. Usage ```text $ dbt run -r timing.txt ... $ snakeviz timing.txt ``` Alternatively, you can use [`py-spy`](https://github.com/benfred/py-spy) to collect [speedscope](https://github.com/jlfwong/speedscope) profiles of dbt commands like this: ```shell python -m pip install py-spy sudo py-spy record -s -f speedscope -- dbt parse ``` --- ### Redshift configurations #### Incremental materialization strategies In dbt-redshift, the following incremental materialization strategies are supported: * `append` (default when `unique_key` is not defined) * `merge` * `delete+insert` (default when `unique_key` is defined) * [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md) All of these strategies are inherited from dbt-postgres. #### Performance optimizations ##### Using sortkey and distkey Tables in Amazon Redshift have two powerful optimizations to improve query performance: distkeys and sortkeys. Supplying these values as model-level configurations apply the corresponding settings in the generated `CREATE TABLE` DDL. Note that these settings will have no effect on models set to `view` or `ephemeral` models. * `dist` can have a setting of `all`, `even`, `auto`, or the name of a key. * `sort` accepts a list of sort keys, for example: `['reporting_day', 'category']`. dbt will build the sort key in the same order the fields are supplied. * `sort_type` can have a setting of `interleaved` or `compound`. if no setting is specified, sort\_type defaults to `compound`. When working with sort keys, it's highly recommended you follow [Redshift's best practices](https://docs.aws.amazon.com/prescriptive-guidance/latest/query-best-practices-redshift/best-practices-tables.html#sort-keys) on sort key effectiveness and cardinality. Sort and dist keys should be added to the `{{ config(...) }}` block in model `.sql` files, eg: my\_model.sql ```sql -- Example with one sort key {{ config(materialized='table', sort='reporting_day', dist='unique_id') }} select ... -- Example with multiple sort keys {{ config(materialized='table', sort=['category', 'region', 'reporting_day'], dist='received_at') }} select ... -- Example with interleaved sort keys {{ config(materialized='table', sort_type='interleaved' sort=['category', 'region', 'reporting_day'], dist='unique_id') }} select ... ``` For more information on distkeys and sortkeys, view Amazon's docs: * [AWS Documentation » Amazon Redshift » Database Developer Guide » Designing Tables » Choosing a Data Distribution Style](https://docs.aws.amazon.com/redshift/latest/dg/t_Distributing_data.html) * [AWS Documentation » Amazon Redshift » Database Developer Guide » Designing Tables » Choosing Sort Keys](https://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html) (Applies to dbt v1.12 and later) ##### Session configuration The Redshift adapter supports the `query_group` session parameter, enabling dbt runs to tag queries for Redshift Workload Manager (WLM) and query logging (for example, `STL_QUERY` and `SVL_QLOG`). You can set `query_group` at the profile level (default for the connection) and override it at the model level. * **Profile-level configuration** Configure `query_group` in your `profiles.yml` to apply a default value to all queries executed using that profile. dbt sets the `query_group` when opening a connection. profiles.yml ```yml outputs: dev: type: redshift host: CLUSTER_ENDPOINT user: REDSHIFT_USER password: REDSHIFT_PASSWORD dbname: REDSHIFT_DBNAME port: 5439 schema: analytics threads: 4 query_group: QUERY_GROUP_NAME ``` ```sql -- models/a_default_group.sql -- Runs under query_group = 'QUERY_GROUP_NAME' (from the profile) select 1 as id ``` * **Model-level configuration** Set `query_group` in a model's `config()` block to temporarily override the default value for that model’s execution. dbt applies the model-level value while the model runs and then restores the default value after model materialization. ```sql -- models/b_override_group.sql -- dbt temporarily sets query_group = 'dbt_finance' for this model, then restores the default value {{ config(query_group='dbt_finance') }} select 1 as id ``` #### Datasharing [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Previously, the Redshift adapter used PostgreSQL-compatible catalog tables (for example, `pg_*`, `information_schema`) for metadata operations such as listing relations, schemas, and columns. These tables only surface objects within the currently connected database, which prevents cross-database operations needed for [Redshift Datasharing](https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html). Starting `dbt-redshift` v1.11.0rc1, you can set `datasharing: true` in your `profiles.yml` to enable cross-database and cross-cluster access. When enabled, `dbt-redshift` switches metadata queries to Redshift's native `SHOW` system commands. You can then materialize models into a database or cluster other than the one specified in your profile using `{{ config(database='other_db') }}`. Example configuration: profiles.yml ```yml company-name: target: dev outputs: dev: type: redshift host: hostname.region.redshift.amazonaws.com ... datasharing: true # default: false ``` Once enabled, you can materialize a model into a different database by setting `database` in the model config. For example: ```sql {{ config(database='other_db') }} select * from {{ ref('my_model') }} ``` The following macros switch to `SHOW` commands when `datasharing: true`: | Macro | Without `datasharing` | With `datasharing` | | ------------------------------------- | ----------------------------------- | -------------------------------------------------- | | `list_relations_without_caching` | `information_schema.tables` | `SHOW TABLES FROM SCHEMA` | | `list_schemas`, `check_schema_exists` | `pg_namespace` | `SHOW SCHEMAS FROM DATABASE` | | `get_columns_in_relation` | `information_schema.columns` | `SHOW COLUMNS FROM TABLE` | | Catalog queries | `pg_class`, `pg_tables`, `pg_views` | `SHOW TABLES FROM SCHEMA` + `SVV_REDSHIFT_COLUMNS` | | `get_relation_last_modified` | `information_schema.tables` | `SHOW TABLES FROM SCHEMA` | | Grants | `pg_user`, `has_table_privilege()` | `SHOW GRANTS ON TABLE` | `ra3_node: true` also enables this behavior and is supported for backward compatibility. For new projects, use `datasharing: true` instead. note If you see `pg_*` queries running with `datasharing` enabled, this is not necessarily a bug. Only the macros listed above are migrated — some macros remain on `pg_*` by design (for example, `get_relations` for dependency tracking and `list_function_relations_without_caching` for function discovery). Custom macro overrides in your project are not affected. To confirm whether a `pg_*` query is expected, check which macro is being overridden and which metadata operation is running. The following limitations apply when using `datasharing`: * Creating views (including materialized views) in another database is not supported. * Cross-database grants on objects are not supported. * Source freshness checks can have a lag of up to 5 minutes. * Metadata queries are limited to 10,000 rows. If a database has more than 10,000 schemas, or a schema has more than 10,000 tables, dbt runs can result in unexpected scenarios. * Cross-database writes require the `SNAPSHOT` transaction isolation level. * For views that reference tables in another database, define them as [late-binding views](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#late-binding-views). For setup instructions, see [Redshift setup](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md). #### Late binding views Redshift supports views unbound from their dependencies, or [late binding views](https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_VIEW.html#late-binding-views). This DDL option "unbinds" a view from the data it selects from. In practice, this means that if upstream views or tables are dropped with a cascade qualifier, the late-binding view does not get dropped as well. Using late-binding views in a production deployment of dbt can vastly improve the availability of data in the warehouse, especially for models that are materialized as late-binding views and are queried by end-users, since they won’t be dropped when upstream models are updated. Additionally, late binding views can be used with [external tables](https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_EXTERNAL_TABLE.html) via Redshift Spectrum. To materialize a dbt model as a late binding view, use the `bind: false` configuration option: my\_view.sql ```sql {{ config(materialized='view', bind=False) }} select * from source.data ``` To make all views late-binding, configure your `dbt_project.yml` file like this: dbt\_project.yml ```yaml models: +bind: false # Materialize all views as late-binding project_name: .... ``` #### Materialized views The Redshift adapter supports [materialized views](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) with the following configuration parameters: | Parameter | Type | Required | Default | Change Monitoring Support | | ---------------------------------------------------------------------------------------------------------- | ------------ | -------- | --------------------------------------------- | ------------------------- | | [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) | `<string>` | no | `apply` | n/a | | [`dist`](#using-sortkey-and-distkey) | `<string>` | no | `even` | drop/create | | [`sort`](#using-sortkey-and-distkey) | `[<string>]` | no | `none` | drop/create | | [`sort_type`](#using-sortkey-and-distkey) | `<string>` | no | `auto` if no `sort`<br />`compound` if `sort` | drop/create | | [`auto_refresh`](#auto-refresh) | `<boolean>` | no | `false` | alter | | [`backup`](#backup) | `<string>` | no | `true` | n/a | ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: materialized_view +on_configuration_change: apply | continue | fail +dist: all | auto | even | <field-name> +sort: <field-name> | [<field-name>] +sort_type: auto | compound | interleaved +auto_refresh: true | false +backup: true | false ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: materialized_view on_configuration_change: apply | continue | fail dist: all | auto | even | <field-name> sort: <field-name> | [<field-name>] sort_type: auto | compound | interleaved auto_refresh: true | false backup: true | false ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized="materialized_view", on_configuration_change="apply" | "continue" | "fail", dist="all" | "auto" | "even" | "<field-name>", sort=["<field-name>"], sort_type="auto" | "compound" | "interleaved", auto_refresh=true | false, backup=true | false, ) }} ``` Many of these parameters correspond to their table counterparts and have been linked above. The parameters unique to materialized views are the [auto-refresh](#auto-refresh) and [backup](#backup) functionality, which are covered below. Learn more about these parameters in Redshift's [docs](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html). ###### Auto-refresh | Parameter | Type | Required | Default | Change Monitoring Support | | -------------- | ----------- | -------- | ------- | ------------------------- | | `auto_refresh` | `<boolean>` | no | `false` | alter | Redshift supports [automatic refresh](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh.html#materialized-view-auto-refresh) configuration for materialized views. By default, a materialized view does not automatically refresh. dbt monitors this parameter for changes and applies them using an `ALTER` statement. Learn more information about the [parameters](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-parameters) in the Redshift docs. ###### Backup | Parameter | Type | Required | Default | Change Monitoring Support | | --------- | ----------- | -------- | ------- | ------------------------- | | `backup` | `<boolean>` | no | `true` | n/a | Redshift supports [backup](https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-snapshots.html) configuration of clusters at the object level. This parameter identifies if the materialized view should be backed up as part of the cluster snapshot. By default, a materialized view will be backed up during a cluster snapshot. dbt cannot monitor this parameter as it is not queryable within Redshift. If the value changes, the materialized view will need to go through a `--full-refresh` to set it. Learn more about these parameters in Redshift's [docs](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-parameters). ##### Limitations As with most data platforms, there are limitations associated with materialized views. Some worth noting include: * Materialized views cannot reference views, temporary tables, user-defined functions, or late-binding tables. * Auto-refresh cannot be used if the materialized view references mutable functions, external schemas, or another materialized view. Find more information about materialized view limitations in Redshift's [docs](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-limitations). #### Unit test limitations * Redshift doesn't support [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) when the SQL in the common table expression (CTE) contains functions such as `LISTAGG`, `MEDIAN`, `PERCENTILE_CONT`, and so on. These functions must be executed against a user-created table. dbt combines given rows to be part of the CTE, which Redshift does not support. In order to support this pattern in the future, dbt would need to "materialize" the input fixtures as tables, rather than interpolating them as CTEs. If you are interested in this functionality, we'd encourage you to participate in this issue in GitHub: [dbt-labs/dbt#8499](https://github.com/dbt-labs/dbt/issues/8499) * Redshift doesn't support unit tests that rely on sources in a database that differs from the models. See this issue in GitHub for more detail: <https://github.com/dbt-labs/dbt-redshift/issues/995> --- ### Redshift permissions In Redshift, permissions are used to control who can perform certain actions on different database objects. Use SQL statements to manage permissions in a Redshift database. #### Example Redshift permissions The following example provides you with the SQL statements you can use to manage permissions. **Note** that `database_name`, `database.schema_name`, and `user_name` are placeholders and you can replace them as needed for your organization's naming convention. ```text grant create schema on database database_name to user_name; grant usage on schema database.schema_name to user_name; grant create table on schema database.schema_name to user_name; grant create view on schema database.schema_name to user_name; grant usage for schemas in database database_name to role role_name; grant select on all tables in database database_name to user_name; grant select on all views in database database_name to user_name; ``` To connect to the database, confirm with an admin that your user role or group has been added to the database. Note that Redshift permissions differ from Postgres, and commands like [`grant connect`](https://www.postgresql.org/docs/current/sql-grant.html) aren't supported in Redshift. Check out the [official documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) for more information. --- ### require_fresh_data_from ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +state: require_fresh_data_from: any | all ``` ##### Properties YAML file models/\<filename>.yml ```yaml models: - name: my_model config: state: require_fresh_data_from: any | all ``` ##### SQL file config models/\<filename>.sql ```sql {{ config( state={ "require_fresh_data_from": "any" | "all" } ) }} ``` #### Definition `require_fresh_data_from` controls how many direct parent nodes need fresh data before dbt State considers this node eligible for a rebuild. dbt State still looks for an object to reuse before triggering an actual rebuild. | Value | Behavior | | --------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `any` (default) | The node becomes eligible for a rebuild when *any* direct parent has fresh data. Rebuilds more frequently; may increase spend. | | `all` | The node only becomes eligible for a rebuild when *all* direct parents have fresh data. Rebuilds less frequently; reduces spend. | #### Default `any` #### Examples ##### Require all parents to have fresh data Use `all` for a model that joins multiple sources and should only rebuild when every upstream dependency has updated: models/fct\_orders.yml ```yaml models: - name: fct_orders config: state: require_fresh_data_from: all ``` ##### Apply a project-wide default dbt\_project.yml ```yaml models: +state: require_fresh_data_from: all ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) --- ### require-dbt-version dbt\_project.yml ```yml require-dbt-version: version-range | [version-range] ``` #### Definition You can use `require-dbt-version` to restrict your project to only work with a range of dbt versions. When you set this configuration: * If you have installed packages from the [dbt Packages hub](https://hub.getdbt.com/) that specify a `require_dbt_version` that doesn't match, running dbt commands will result in an error. * It helps package maintainers (such as [dbt-utils](https://github.com/dbt-labs/dbt-utils)) ensure that users' dbt version is compatible with the package. * It signals [compatibility with dbt v2](#dbt-v2-compatibility) (`2.0.0` and higher). * It might also help your whole team remain synchronized on the same version of dbt for local development, to avoid compatibility issues from changed behavior. You should pin to a major release. See [pin to a range](#pin-to-a-range) for more details. If this configuration isn't specified, no version check will occur. dbt release tracks When you select a [release track in dbt](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing dbt version upgrades, dbt will ignore the `require-dbt-version` config. dbt Labs is committed to zero breaking changes for code in dbt projects, with ongoing releases to dbt. We also recommend these best practices:  Installing dbt packages If you install dbt packages for use in your project, whether the package is maintained by your colleagues or a member of the open source dbt community, we recommend pinning the package to a specific revision or `version` boundary. dbt manages this out-of-the-box by *locking* the version/revision of packages in development in order to guarantee predictable builds in production. To learn more, refer to [Predictable package installs](https://docs.getdbt.com/reference/commands/deps.md#predictable-package-installs).  Maintaining dbt packages If you maintain dbt packages, whether on behalf of your colleagues or members of the open source community, we recommend writing defensive code that checks to verify that other required packages and global macros are available. For example, if your package depends on the availability of a `date_spine` macro in the global `dbt` namespace, you can write: models/some\_days.sql ```sql {% macro a_few_days_in_september() %} {% if not dbt.get('date_spine') %} {{ exceptions.raise_compiler_error("Expected to find the dbt.date_spine macro, but it could not be found") }} {% endif %} {{ date_spine("day", "cast('2020-01-01' as date)", "cast('2030-12-31' as date)") }} {% endmacro %} ``` #### YAML quoting This configuration needs to be interpolated by the YAML parser as a string. As such, you should quote the value of the configuration, taking care to avoid whitespace. For example: ```yml # ✅ These will work require-dbt-version: ">=1.0.0" # Double quotes are OK require-dbt-version: '>=1.0.0' # So are single quotes # ❌ These will not work require-dbt-version: >=1.0.0 # No quotes? No good require-dbt-version: ">= 1.0.0" # Don't put whitespace after the equality signs ``` ###### Avoid unbounded upper limits We recommend [defining both lower and upper bounds](#pin-to-a-range), such as `">=1.0.0,<3.0.0"`, to ensure stability across releases. We don't recommend having an unbounded `require-dbt-version` (for example, `">=1.0.0"`). Without an upper limit, a project may break when dbt releases a new major version. #### dbt v2 compatibility The `require-dbt-version` also signals whether a project or package supports the [dbt v2](https://docs.getdbt.com/docs/introduction.md) (`2.0.0` and higher). * If it excludes `2.0.0`, dbt v2 will warn today and error in a future release, matching dbt v1 behavior. * You can [bypass version checks](#disabling-version-checks) with `--no-version-check`. Refer to [pin to a range](#pin-to-a-range) for more info on how to define a version range.  Use dbt-autofix to update dbt projects and packages [`dbt-autofix` tool](https://github.com/dbt-labs/dbt-autofix) automatically scans your dbt project for deprecated configurations and updates them to align with the latest best practices and prepare for dbt v2 migration. When it runs, `dbt-autofix` will: * Check your `packages.yml` to determine which packages it can automatically upgrade. * Look for packages that list `require-dbt-version: 2.0.0` or higher (indicating dbt v2 support). * Upgrade those packages to the lowest version that supports dbt v2. This ensures that `dbt-autofix` only updates packages that are confirmed to work with dbt v2 and avoids updating packages that are known to be incompatible with dbt v2. #### Examples The following examples showcase how to use the `require-dbt-version`: * [Specify a minimum dbt version](#specify-a-minimum-dbt-version) — Use a `>=` operator for a minimum boundary. * [Pin to a range](#pin-to-a-range) — Use a comma separated list to specify an upper and lower bound. * [Require a specific dbt version](#require-a-specific-dbt-version) — Restrict your project to run only with an exact version of dbt. ##### Specify a minimum dbt version Use a `>=` operator to specify a lower and an upper limit. For example: dbt\_project.yml ```yml require-dbt-version: ">=1.9.0" # project will only work with versions 1.9 and higher. require-dbt-version: ">=2.0.0" # project will only work with dbt v2 (v2.0.0 and higher). ``` Remember, having an unbounded upper limit isn't recommended. Instead, check out the [pin to a range](#pin-to-a-range) example to define a range with both a lower and upper limit to ensure stability across releases. ##### Pin to a range Use a comma separated list for an upper and lower bound. You can define a version range either as a YAML list (using square brackets) or as a comma-delimited string — both forms are valid and work. To signal compatibility with dbt v2, include `2.0.0` or higher in your version range. Both of the following formats are valid: dbt\_project.yml ```yaml require-dbt-version: [">=1.10.0", "<3.0.0"] # or require-dbt-version: ">=1.10.0,<3.0.0" ``` If your range excludes 2.0.0 (for example, `>=1.6.0,<2.0.0`), dbt v2 will show a warning now and error in a future release. You can [bypass version checks](#disabling-version-checks) with `--no-version-check`. ##### Require a specific dbt version Not recommended Pinning to a specific dbt version is discouraged because it limits project flexibility and can cause compatibility issues, especially with dbt packages. It's recommended to [pin to a major release](#pin-to-a-range), using a version range (for example, `">=1.0.0", "<2.0.0"`) for broader compatibility and to benefit from updates. While you can restrict your project to run only with an exact version of dbt, we do not recommend this for dbt v1.0.0 and higher. In the following example, the project will only run with dbt v1.5: dbt\_project.yml ```yml require-dbt-version: "1.5.0" ``` #### Invalid dbt versions If the version of dbt used to invoke a project disagrees with the specified `require-dbt-version` in the project or *any* of the included packages, then dbt will fail immediately with the following error: ```text $ dbt compile Running with dbt=1.5.0 Encountered an error while reading the project: Runtime Error This version of dbt is not supported with the 'my_project' package. Installed version of dbt: =1.5.0 Required version of dbt for 'my_project': ['>=1.6.0', '<2.0.0'] Check the requirements for the 'my_project' package, or run dbt again with --no-version-check ``` #### Disabling version checks To suppress failures to incompatible dbt versions, supply the `--no-version-check` flag to `dbt run`. ```text $ dbt run --no-version-check Running with dbt=1.5.0 Found 13 models, 2 tests, 1 archives, 0 analyses, 204 macros, 2 operations.... ``` See [global configs](https://docs.getdbt.com/reference/global-configs/version-compatibility.md) for usage details. --- ### Resource path The `<resource-path>` nomenclature is used in this documentation when documenting how to configure resource types like models, seeds, snapshots, tests, sources, and others, from your `dbt_project.yml` file. It represents the nested dictionary keys that provide the path to a directory of that resource type, or a single instance of that resource type by name. ```yml resource_type: project_name: directory_name: subdirectory_name: instance_of_resource_type (by name): ... ``` #### Example The following examples are mostly for models and a source, but the same concepts apply for seeds, snapshots, tests, sources, and other resource types. ##### Apply config to all models To apply a configuration to all models, do not use a `<resource-path>`: dbt\_project.yml ```yml models: +enabled: false # this will disable all models (not a thing you probably want to do) ``` ##### Apply config to all models in your project To apply a configuration to all models in *your* project only, use your [project name](https://docs.getdbt.com/reference/project-configs/name.md) as the `<resource-path>`: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: +enabled: false # this will apply to all models in your project, but not any installed packages ``` ##### Apply config to all models in a subdirectory To apply a configuration to all models in a subdirectory of your project, e.g. `staging`, nest the directory under the project name: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: staging: +enabled: false # this will apply to all models in the `staging/` directory of your project ``` In the following project, this would apply to models in the `staging/` directory, but not the `marts/` directory: ```text . ├── dbt_project.yml └── models    ├── marts    └── staging ``` ##### Apply config to a specific model To apply a configuration to a specific model, nest the full path under the project name. For a model at `/staging/stripe/payments.sql`, this would look like: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: staging: stripe: payments: +enabled: false # this will apply to only one model ``` In the following project, this would only apply to the `payments` model: ```text . ├── dbt_project.yml └── models    ├── marts    │   └── core    │   ├── dim_customers.sql    │   └── fct_orders.sql    └── staging    ├── jaffle_shop    │   ├── customers.sql    │   └── orders.sql    └── stripe    └── payments.sql ``` ##### Apply config to a source nested in a subfolder To disable a source table nested in a YAML file in a subfolder, you will need to supply the subfolder(s) within the path to that YAML file, as well as the source name and the table name in the `dbt_project.yml` file.<br /><br />The following example shows how to disable a source table nested in a YAML file in a subfolder: dbt\_project.yml ```yaml sources: your_project_name: subdirectory_name: source_name: source_table_name: +enabled: false ``` --- ### Resource type (Applies to dbt v1.9 and later) The `--resource-type` and `--exclude-resource-type` flags include or exclude resource types from the `dbt build`, `dbt clone`, `dbt test`, and `dbt list` commands.  Filtering test types with dbt test Use the `test_type` selector to filter test types with `dbt test` — it works across all engines: * `dbt test --select test_type:unit` — runs only unit tests * `dbt test --select test_type:data` — runs only data tests If you're using dbt v1.9+, you can also use `--resource-type`: * `dbt test --resource-type unit_test` — runs only unit tests * `dbt test --resource-type test` — runs only data tests For more examples, see [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md). This means the flags enable you to specify which types of resources to include or exclude when running the commands, instead of targeting specific resources. Note The `--exclude-resource-type` flag is only available in dbt version 1.8 and higher. If you're using older versions, this flag won't be available. The available resource types are: (Applies to dbt v1.11 and later) * [`analysis`](https://docs.getdbt.com/docs/build/analyses.md) * [`exposure`](https://docs.getdbt.com/docs/build/exposures.md) * [`function`](https://docs.getdbt.com/docs/build/udfs.md) * [`metric`](https://docs.getdbt.com/docs/build/build-metrics-intro.md) * [`model`](https://docs.getdbt.com/docs/build/models.md) * [`saved_query`](https://docs.getdbt.com/docs/build/saved-queries.md) * [`seed`](https://docs.getdbt.com/docs/build/seeds.md) * [`semantic_model`](https://docs.getdbt.com/docs/build/semantic-models.md) * [`snapshot`](https://docs.getdbt.com/docs/build/snapshots.md) * [`source`](https://docs.getdbt.com/docs/build/sources.md) * [`test`](https://docs.getdbt.com/docs/build/data-tests.md) * [`unit_test`](https://docs.getdbt.com/docs/build/unit-tests.md) #### Positive vs negative filters * `--resource-type` is a positive filter — dbt only runs the resource types selected in the command, implicitly skipping every other type. * `--exclude-resource-type` is a negative filter — dbt starts with the full catalog of resource types and then omits the types selected in the command. dbt runs everything *except* those resource types. You can use both flags in a command; dbt first applies the positive filter (`--resource-type`) and then removes the types listed in the negative filter (`--exclude-resource-type`). For example: ```text dbt build --resource-type model test snapshot --exclude-resource-type snapshot ``` Note that the list of dbt resource types is mutually exclusive and collectively exhaustive (MECE). This means that any `--resource-type` selection can also be achieved by excluding the other resource types using `--exclude-resource-type`, and vice versa. #### Examples Instead of targeting specific resources, use the `--resource-type` or `--exclude-resource-type` flags to target all resources of a certain type: `dbt build --resource-type RESOURCE_TYPE`, replacing `RESOURCE_TYPE` with the resource type you want to include. See the following sample commands for including or excluding resource types. Note that the `--exclude-resource-type` flag is only available in dbt version 1.8 and higher.  Include resource types ##### Include multiple resource types Use the following command to include multiple resource types such as data tests and models in your build process: Usage ```text dbt build --resource-type test model ``` ##### Include all snapshots Use the following command to only include snapshots in your dbt build process: Usage ```text dbt build --resource-type snapshot ``` ##### Include all saved queries Use the following command to only include saved queries with the `--resource-type` flag: Usage ```text dbt build --resource-type saved_query ``` ##### Include all data tests Use the following command to only include data tests in your build process: Usage ```text dbt build --resource-type test ``` (Applies to dbt v1.9 and later) ##### Include only data tests during testing To run only data tests (excluding unit tests) when using `dbt test`, use the `test_type` selector: Usage ```text dbt test --select test_type:data -- dbt v1 and dbt v2 dbt test --resource-type test -- dbt v1 only ```  Exclude resource types ##### Exclude multiple resource types Use the following command to exclude multiple resource types such as data tests and models from your build process: Usage ```text dbt build --exclude-resource-type test model ``` ##### Exclude all unit tests Use the following command to exclude unit tests from your dbt build process. Usage ```text dbt build --exclude-resource-type unit_test ``` (Applies to dbt v1.9 and later) ##### Exclude unit tests during testing To exclude unit tests when using `dbt test`, use the `test_type` selector to run only data tests: Usage ```text dbt test --select test_type:data -- dbt v1 and dbt v2 dbt test --exclude-resource-type unit_test -- dbt v1 only ``` --- ### returns Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> returns: data_type: <string> # required, warehouse-specific description: <markdown_string> # optional ``` #### Definition The `returns` property defines the output of a user-defined function (UDF). This is a required property for all functions and specifies what data type the function will return when executed. #### Properties ##### data\_type (required) The `data_type` field specifies the data type that the function returns. This is a required field and must match the data types supported by your specific data platform. Warehouse-specific data types The `data_type` values are warehouse-specific. Use the data type syntax that your warehouse requires: * **Snowflake**: `STRING`, `NUMBER`, `BOOLEAN`, `TIMESTAMP_NTZ`, `VARIANT`, etc. * **BigQuery**: `STRING`, `INT64`, `BOOL`, `TIMESTAMP`, `ARRAY<STRING>`, `STRUCT`, etc. * **Redshift**: `VARCHAR`, `INTEGER`, `BOOLEAN`, `TIMESTAMP`, `SUPER`, etc. * **Postgres**: `TEXT`, `INTEGER`, `BOOLEAN`, `TIMESTAMP`, `JSONB`, etc. Refer to your warehouse documentation for the complete list of supported data types and their syntax. ##### description An optional markdown string describing what the function returns. This is helpful for documentation purposes. #### Examples ##### Simple scalar function functions/schema.yml ```yml functions: - name: is_valid_email description: Validates if a string is a properly formatted email address arguments: - name: email_string data_type: STRING description: The email address to validate returns: data_type: BOOLEAN description: Returns true if the string is a valid email format, false otherwise ``` ##### Function with complex return type functions/schema.yml ```yml functions: - name: calculate_metrics description: Calculates revenue and profit metrics arguments: - name: revenue data_type: DECIMAL(18,2) - name: cost data_type: DECIMAL(18,2) returns: data_type: DECIMAL(18,2) description: The calculated profit margin as a percentage ``` ##### BigQuery function with ARRAY return type functions/schema.yml ```yml functions: - name: split_tags description: Splits a comma-separated string into an array of tags arguments: - name: tag_string data_type: STRING returns: data_type: ARRAY<STRING> description: An array of individual tag strings ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) --- ### Run results JSON file **Current schema**: [`v6`](https://schemas.getdbt.com/dbt/run-results/v6/index.html) **Produced by:** [`build`](https://docs.getdbt.com/reference/commands/build.md) [`clone`](https://docs.getdbt.com/reference/commands/clone.md) [`compile`](https://docs.getdbt.com/reference/commands/compile.md) [`docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) [`retry`](https://docs.getdbt.com/reference/commands/retry.md) [`run`](https://docs.getdbt.com/reference/commands/run.md) [`seed`](https://docs.getdbt.com/reference/commands/seed.md) [`show`](https://docs.getdbt.com/reference/commands/show.md) [`snapshot`](https://docs.getdbt.com/reference/commands/snapshot.md) [`test`](https://docs.getdbt.com/reference/commands/test.md) [`run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) This file contains information about a completed invocation of dbt, including timing and status info for each node (model, test, etc) that was executed. In aggregate, many `run_results.json` can be combined to calculate average model runtime, test failure rates, the number of record changes captured by snapshots, etc. Note that only executed nodes appear in the run results. If you have multiple run or test steps with different criteria, each will produce different run results. (Applies to dbt v2.0 and later) Note: `dbt freshness` produces different artifacts — [`freshness.json`](https://docs.getdbt.com/reference/artifacts/freshness-json.md) for both sources and models, and [`sources.json`](https://docs.getdbt.com/reference/artifacts/sources-json.md) for sources only (retained for backward compatibility). `dbt source freshness` is still supported but is considered a legacy command. ##### Top-level keys * [`metadata`](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md#common-metadata) * `args`: Dictionary of arguments passed to the CLI command or RPC method that produced this artifact. Most useful is `which` (command) or `rpc_method`. This dict excludes null values, and includes default values if they are not null. Equivalent to [`invocation_args_dict`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#invocation_args_dict) in the dbt-Jinja context. * `elapsed_time`: Total invocation time in seconds. * `results`: Array of node execution details. Each entry in `results` is a [`Result` object](https://docs.getdbt.com/reference/dbt-classes.md#result-objects), with one difference: Instead of including the entire `node` object, only the `unique_id` is included. (The full `node` object is recorded in [`manifest.json`](https://docs.getdbt.com/reference/artifacts/manifest-json.md).) * `unique_id`: Unique node identifier, which maps results to `nodes` in the [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) * `status`: dbt's interpretation of runtime success, failure, or error * `thread_id`: Which thread executed this node? E.g. `Thread-1` * `execution_time`: Total time spent executing this node * `timing`: Array that breaks down execution time into steps (often `compile` + `execute`) * `message`: How dbt will report this result on the CLI, based on information returned from the database - `adapter_response`: Dictionary of metadata returned from the database, which varies by adapter. For example, success `code`, number of `rows_affected`, total `bytes_processed`, and so on. Not applicable for [data tests](https://docs.getdbt.com/docs/build/data-tests.md). * `rows_affected` returns the number of rows modified by the last statement executed. In cases where the query's row count can't be determined or isn't applicable (such as when creating a view), a [standard value](https://peps.python.org/pep-0249/#rowcount) of `-1` is returned for `rowcount`. The run\_results.json includes three attributes related to the `applied` state that complement `unique_id`: * `compiled`: Boolean entry of the node compilation status (`False` after parsing, but `True` after compiling). * `compiled_code`: Rendered string of the code that was compiled (empty after parsing, but full string after compiling). * `relation_name`: The fully-qualified name of the object that was (or will be) created/updated within the database. Continue to look up additional information about the `logical` state of nodes using the full node object in manifest.json via the `unique_id`. #### Examples Here are a few examples and the resulting output to the `run_results.json` file. ##### Compile model results Let's say that you have a model that looks like this: models/my\_model.sql ```sql select {{ dbt.current_timestamp() }} as created_at ``` Compile the model: ```shell dbt compile -s my_model ``` Here's a printed snippet from the `run_results.json`: ```json { "status": "success", "timing": [ { "name": "compile", "started_at": "2023-10-12T16:35:28.510434Z", "completed_at": "2023-10-12T16:35:28.519086Z" }, { "name": "execute", "started_at": "2023-10-12T16:35:28.521633Z", "completed_at": "2023-10-12T16:35:28.521641Z" } ], "thread_id": "Thread-2", "execution_time": 0.0408780574798584, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.my_project.my_model", "compiled": true, "compiled_code": "select now() as created_at", "relation_name": "\"postgres\".\"dbt_dbeatty\".\"my_model\"" } ``` ##### Run generic data tests Use the [`store_failures_as`](https://docs.getdbt.com/reference/resource-configs/store_failures_as.md) config to store failures for only one data test in the database: models/\_models.yml ```yaml models: - name: my_model columns: - name: created_at data_tests: - not_null: config: store_failures_as: view - unique: config: store_failures_as: ephemeral ``` Run the built-in `unique` test and store the failures as a table: ```shell dbt test -s my_model ``` Here's a printed snippet from the `run_results.json`: ```json "results": [ { "status": "pass", "timing": [ { "name": "compile", "started_at": "2023-10-12T17:20:51.279437Z", "completed_at": "2023-10-12T17:20:51.317312Z" }, { "name": "execute", "started_at": "2023-10-12T17:20:51.319812Z", "completed_at": "2023-10-12T17:20:51.441967Z" } ], "thread_id": "Thread-2", "execution_time": 0.1807551383972168, "adapter_response": { "_message": "SELECT 1", "code": "SELECT", "rows_affected": 1 }, "message": null, "failures": 0, "unique_id": "test.my_project.unique_my_model_created_at.a9276afbbb", "compiled": true, "compiled_code": "\n \n \n\nselect\n created_at as unique_field,\n count(*) as n_records\n\nfrom \"postgres\".\"dbt_dbeatty\".\"my_model\"\nwhere created_at is not null\ngroup by created_at\nhaving count(*) > 1\n\n\n", "relation_name": null }, { "status": "pass", "timing": [ { "name": "compile", "started_at": "2023-10-12T17:20:51.274049Z", "completed_at": "2023-10-12T17:20:51.295237Z" }, { "name": "execute", "started_at": "2023-10-12T17:20:51.296361Z", "completed_at": "2023-10-12T17:20:51.491327Z" } ], "thread_id": "Thread-1", "execution_time": 0.22345590591430664, "adapter_response": { "_message": "SELECT 1", "code": "SELECT", "rows_affected": 1 }, "message": null, "failures": 0, "unique_id": "test.my_project.not_null_my_model_created_at.9b412fbcc7", "compiled": true, "compiled_code": "\n \n \n\n\n\nselect *\nfrom \"postgres\".\"dbt_dbeatty\".\"my_model\"\nwhere created_at is null\n\n\n", "relation_name": "\"postgres\".\"dbt_dbeatty_dbt_test__audit\".\"not_null_my_model_created_at\"" } ], ``` --- ### runtime_version Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> config: runtime_version: <string> # required for Snowflake and BigQuery; optional and ignored on Databricks ``` #### Definition When creating Python UDFs, specify the Python version to run in `runtime_version`. #### Supported values * [Snowflake](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-introduction): `3.10`, `3.11`, `3.12`, and `3.13` * [BigQuery](https://cloud.google.com/bigquery/docs/user-defined-functions-python): `3.11` * Databricks: Accepted for compatibility but has no effect. Databricks manages the Python runtime internally, so dbt displays a warning if you set it. Available in `dbt-databricks` v1.12+. #### Example In this example, we're using the Python version `3.11` for the UDF. functions/schema.yml ```yaml functions: - name: is_positive_int config: runtime_version: "3.11" ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [type](https://docs.getdbt.com/reference/resource-configs/type.md) * [volatility](https://docs.getdbt.com/reference/resource-configs/volatility.md) * [entry\_point](https://docs.getdbt.com/reference/resource-configs/entry-point.md) * [packages](https://docs.getdbt.com/reference/resource-configs/packages.md) * [arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) * [returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### Salesforce Data 360 configurationsBeta ### Salesforce Data 360 configurations[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") #### Supported materializations | Materialization | Supported | Notes | | ----------------- | --------- | ----------------------------------------------------------- | | View | ❌ | | | Table | ✅ | Creates a batch data transform and a Data Lake Object (DLO) | | Incremental | ❌ | Coming soon | | Ephemeral | ❌ | | | Seeds | ❌ | | | Sources | ✅ | Required | | Custom data tests | ❌ | | | Snapshots | ❌ | | ##### Sources For models that query raw Data 360 data, reference the table though a dbt source. Selecting a DLO directly is not supported. For example: ```yml sources: - name: default tables: - name: raw_customers__dll description: "Customers raw table stored in default dataspace" columns: - name: id__c description: "Customer ID" data_tests: - not_null - unique - name: first_name__c description: "Customer first name" - name: last_name__c description: "Customer last name" - name: email__c description: "Customer email address" data_tests: - not_null - unique ``` ##### Table materialization dbt v2 supports Table materialization on Salesforce Data 360. Execution of the materialization results in the creation of a [batch data transform](https://help.salesforce.com/s/articleView?id=data.c360_a_batch_transform_overview.htm\&language=en_US\&type=5) and a [Data Lake Object (DLO)](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_concepts_data_cloud_objects.htm) for querying. Currently, only the `profile` type DLO is supported. Support for `engagement` DLOs is coming soon. Profile DLOs must define a `primary_key` in the model config. For example: ```sql {{ config( materialized='table', primary_key='customer_id__c', category='Profile' ) }} select id__c as customer_id__c, first_name__c, last_name__c, email__c as customer_email__c from {{ source('default', 'raw_customers__dll') }} ``` #### Naming rules and required configs * All dbt model names must end with `__dll`. If you omit this suffix in your file name, it is appended automatically during execution (for example, `model_name` becomes `model_name__dll`). This will break downstream dbt references because dbt will look for a DLO named `model_name` when Data 360 has `model_name__dll`. * Columns must end with `__c`. Omitting the suffix causes a Data 360 “unknown syntax” error. * Model names cannot contain double under scores (`__`) outside of the final `__dll`. For example, `supplies__agg__dll` will build as `agg__dll`, which can cause confusion for downstream refs. * All dbt models must be configured with `primary_key` and `category='Profile'` in the model configuration. You can also apply the configurations in the `resources.yml` and `dbt_project.yml`. #### Known limitations * **Reruns of dbt models**: Due to the Data 360 architecture of metadata and dependency management, dbt cannot rerun the same model if a data transform and a DLO already exist. This is because dbt can't drop the DLO during subsequent runs of table materializations, as expected in data warehouses. If you change your logic between runs, you have to delete the dependencies of the data transform and DLO manually in the UI before executing a `dbtf run`. A fix is in progress. * **Static analysis in VS Code**: Column-level lineage and dbt buttons (`Build` and `Test`) are affected. You can either turn off static analysis temporarily by running all commands with `--static-analysis off` or set up your environment variables with `DBT_STATIC_ANALYSIS=off`. * **Arbitrary queries** (for example, `SELECT 1 AS foo`): All queries must be tied to a defined dbt source before building a dbt model on it. * **`select *`** Metadata queries may fail because Data 360 injects system columns into every DLO. Bug fix is in progress. --- ### schema ##### Model Specify a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md#understanding-custom-schemas) for a group of models in your project YAML file (`dbt_project.yml`), in a [SQL file config](https://docs.getdbt.com/reference/resource-configs/schema.md#models), or in a property YAML file. For example, if you have a group of marketing-related models and want to place them in a separate schema called `marketing`, you can configure it like this: dbt\_project.yml ```yml models: your_project: marketing: # Grouping or folder for set of models +schema: marketing ``` You can also set the schema for an individual model in its property file: models/properties.yml ```yml models: - name: my_model config: schema: marketing ``` This would result in the generated relations for these models being located in the `marketing` schema, so the full relation names would be `analytics.target_schema_marketing.model_name`. This is because the schema of the relation is `{{ target.schema }}_{{ schema }}`. The [definition](#definition) section explains this in more detail. ##### Seeds Configure a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md#understanding-custom-schemas) in your `dbt_project.yml` file. For example, if you have a seed that should be placed in a separate schema called `mappings`, you can configure it like this: dbt\_project.yml ```yml seeds: your_project: product_mappings: +schema: mappings ``` This would result in the generated relation being located in the `mappings` schema, so the full relation name would be `analytics.mappings.seed_name`. ##### Snapshots (Applies to dbt v1.9 and later) Specify a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md#understanding-custom-schemas) for a snapshot in your `dbt_project.yml` or property file. For example, if you have a snapshot that you want to load into a schema other than the target schema, you can configure it like this: In a `dbt_project.yml` file: dbt\_project.yml ```yml snapshots: your_project: your_snapshot: +schema: snapshots ``` In a `snapshots/snapshot_name.yml` file: snapshots/snapshot\_name.yml ```yaml snapshots: - name: snapshot_name config: schema: snapshots ``` This results in the generated relation being located in the `snapshots` schema so the full relation name would be `analytics.snapshots.your_snapshot` instead of the default target schema. ##### Saved queries Specify a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md#understanding-custom-schemas) for a [saved query](https://docs.getdbt.com/docs/build/saved-queries.md#parameters) in your `dbt_project.yml` or property file. dbt\_project.yml ```yml saved-queries: +schema: metrics ``` This would result in the saved query being stored in the `metrics` schema. ##### Test Customize a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md#understanding-custom-schemas) for storing test results in your `dbt_project.yml` file. For example, to save test results in a specific schema, you can configure it like this: dbt\_project.yml ```yml data_tests: +store_failures: true +schema: test_results ``` This would result in the test results being stored in the `test_results` schema. Refer to [Usage](#usage) for more examples. #### Definition Optionally specify a custom schema for a [model](https://docs.getdbt.com/docs/build/sql-models.md), [seed](https://docs.getdbt.com/docs/build/seeds.md), [snapshot](https://docs.getdbt.com/docs/build/snapshots.md), [saved query](https://docs.getdbt.com/docs/build/saved-queries.md), or [test](https://docs.getdbt.com/docs/build/data-tests.md). For users on dbt v1.8 or earlier, use the [`target_schema` config](https://docs.getdbt.com/reference/resource-configs/target_schema.md) to specify a custom schema for a snapshot. When dbt creates a relation (table/view) in a database, it creates it as: `{{ database }}.{{ schema }}.{{ identifier }}`, e.g. `analytics.finance.payments` The standard behavior of dbt is: * If a custom schema is *not* specified, the schema of the relation is the target schema (`{{ target.schema }}`). * If a custom schema is specified, by default, the schema of the relation is `{{ target.schema }}_{{ schema }}`. To learn more about changing the way that dbt generates a relation's `schema`, read [Using Custom Schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) #### Usage ##### Models Configure groups of models from the `dbt_project.yml` file. dbt\_project.yml ```yml models: jaffle_shop: # the name of a project marketing: +schema: marketing ``` Configure individual models using a config block: models/my\_model.sql ```sql {{ config( schema='marketing' ) }} ``` Or configure individual models in a property file: models/properties.yml ```yml models: - name: my_model config: schema: marketing ``` ##### Seeds dbt\_project.yml ```yml seeds: +schema: mappings ``` ##### Data tests Customize the name of the schema in which tests [configured to store failures](https://docs.getdbt.com/reference/resource-configs/store_failures.md) will save their results. The resulting schema is `{{ profile.schema }}_{{ tests.schema }}`, with a default suffix of `dbt_test__audit`. To use the same profile schema, set `+schema: null`. dbt\_project.yml ```yml data_tests: +store_failures: true +schema: _sad_test_failures # Will write tables to my_database.my_schema__sad_test_failures ``` Ensure you have the authorization to create or access schemas for your work. To ensure that the required schemas have the correct permissions, run a SQL statement in your respective data platform environment. For example, run the following command if using Redshift (exact authorization query may differ from one data platform to another): ```sql create schema if not exists dev_username_dbt_test__audit authorization username; ``` *Replace `dev_username` with your specific development schema name and `username` with the appropriate user who should have the permissions.* This command grants the appropriate permissions to create and access the `dbt_test__audit` schema, which is often used with the `store_failures` configuration. #### Warehouse specific information * BigQuery: `dataset` and `schema` are interchangeable --- ### Seed configurations #### Available configurations ##### Seed-specific configurations Resource-specific configurations are applicable to only one dbt resource type rather than multiple resource types. You can define these settings in the project file (`dbt_project.yml`), a property file (`models/properties.yml` for models, similarly for other resources), or within the resource’s file using the `{{ config() }}` macro.<br /> The following resource-specific configurations are only available to Seeds: ##### Project file dbt\_project.yml ```yml seeds: <resource-path>: +quote_columns: true | false +column_types: {column_name: datatype} +delimiter: <string> ``` ##### Property file seeds/properties.yml ```yaml seeds: - name: [<seed-name>] config: quote_columns: true | false column_types: {column_name: datatype} delimiter: <string> ``` ##### General configurations General configurations provide broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project file, property files, or within resource-specific files. ##### Project file dbt\_project.yml (Applies to dbt v1.9 and later) ```yaml seeds: <resource-path>: +enabled: true | false +tags: <string> | [<string>] +pre-hook: <sql-statement> | [<sql-statement>] +post-hook: <sql-statement> | [<sql-statement>] +database: <string> +schema: <string> +alias: <string> +persist_docs: <dict> +full_refresh: <boolean> +meta: {<dictionary>} +grants: {<dictionary>} +event_time: my_time_field ``` ##### Property file seeds/properties.yml (Applies to dbt v1.9 and later) ```yaml seeds: - name: [<seed-name>] config: enabled: true | false tags: <string> | [<string>] pre_hook: <sql-statement> | [<sql-statement>] post_hook: <sql-statement> | [<sql-statement>] database: <string> schema: <string> alias: <string> persist_docs: <dict> full_refresh: <boolean> meta: {<dictionary>} grants: {<dictionary>} event_time: my_time_field ``` #### Configuring seeds Seeds can only be configured from YAML files, either in `dbt_project.yml` or within an individual seed's YAML properties. It is not possible to configure a seed from within its CSV file. Seed configurations, like model configurations, are applied hierarchically — configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project, and configurations defined in a specific seed's properties will override configurations defined in `dbt_project.yml`. ##### Examples ###### Apply the `schema` configuration to all seeds To apply a configuration to all seeds, including those in any installed [packages](https://docs.getdbt.com/docs/build/packages.md), nest the configuration directly under the `seeds` key: dbt\_project.yml ```yml seeds: +schema: seed_data ``` ###### Apply the `schema` configuration to all seeds in your project To apply a configuration to all seeds in your project only (i.e. *excluding* any seeds in installed packages), provide your [project name](https://docs.getdbt.com/reference/project-configs/name.md) as part of the resource path. For a project named `jaffle_shop`: dbt\_project.yml ```yml seeds: jaffle_shop: +schema: seed_data ``` Similarly, you can use the name of an installed package to configure seeds in that package. ###### Apply the `schema` configuration to one seed only To apply a configuration to one seed only, provide the full resource path (including the project name, and subdirectories). seeds/marketing/properties.yml ```yml seeds: - name: utm_parameters config: schema: seed_data ``` In older versions of dbt, you must define configurations in `dbt_project.yml` and include the full resource path (including the project name, and subdirectories). For a project named `jaffle_shop`, with a seed file at `seeds/marketing/utm_parameters.csv`, this would look like: dbt\_project.yml ```yml seeds: jaffle_shop: marketing: utm_parameters: +schema: seed_data ``` #### Example seed configuration The following is a valid seed configuration for a project with: * `name: jaffle_shop` * A seed file at `seeds/country_codes.csv`, and * A seed file at `seeds/marketing/utm_parameters.csv` dbt\_project.yml ```yml name: jaffle_shop ... seeds: jaffle_shop: +enabled: true +schema: seed_data # This configures seeds/country_codes.csv country_codes: # Override column types +column_types: country_code: varchar(2) country_name: varchar(32) marketing: +schema: marketing # this will take precedence ``` --- ### Seed properties Seed properties can be declared in `.yml` files under a `seed` key. We recommend that you put them in the `seeds/` directory. You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within that directory. seeds/\<filename>.yml ```yml seeds: - name: <string> description: <markdown_string> config: <seed_config>: <config_value> docs: show: true | false node_color: <color_id> # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") data_tests: - <test> - ... # declare additional tests columns: - name: <column name> description: <markdown_string> quote: true | false data_tests: - <test> - ... # declare additional tests config: meta: {<dictionary>} tags: [<string>] - name: ... # declare properties of additional columns - name: ... # declare properties of additional seeds ``` --- ### seed-paths dbt\_project.yml ```yml seed-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [seed](https://docs.getdbt.com/docs/build/seeds.md) files are located. #### Default By default, dbt expects seeds to be located in the `seeds` directory. For example, `seed-paths: ["seeds"]`. Paths specified in `seed-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/seed`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml seed-paths: ["seed"] ``` * ❌ **Don't:** * Avoid absolute paths: ```yml seed-paths: ["/Users/username/project/seed"] ``` #### Examples ##### Use a directory named `custom_seeds` instead of `seeds` dbt\_project.yml ```yml seed-paths: ["custom_seeds"] ``` ##### Co-locate your models and seeds in the `models` directory Note: this works because dbt is looking for different file types for seeds (`.csv` files) and models (`.sql` files). dbt\_project.yml ```yml seed-paths: ["models"] model-paths: ["models"] ``` ##### Split your seeds across two directories Note: We recommend that you instead use two subdirectories within the `seeds/` directory to achieve a similar effect. dbt\_project.yml ```yml seed-paths: ["seeds", "custom_seeds"] ``` --- ### selection_filter_on Available in v2 When you pass a selector (`--select`, `--exclude`, `--selector`) to `dbt check` or `dbt build`, dbt uses it to scope which project resources each check evaluates. `selection_filter_on` tells dbt which column in the check's output contains the resource IDs to match against the selection. #### Values * **Default (not set):** If the query returns `unique_id`, dbt keeps only rows whose `unique_id` is in the selection. If the query does not return `unique_id`, the check runs against the whole project. Useful for aggregate checks like "the project has at least one model". * **`none`:** The check always runs against the whole project, ignoring any selector. Use this to make whole-project behavior explicit. * **A column name or list of column names:** dbt keeps a row if the ID in any of the named columns is in the selection. Each named column must exist in the result, or the check raises an error. Use this for checks that return relationships between resources (edges). #### When to set this config If your check returns a `unique_id` column, you don't need to set this config. Use `selection_filter_on` when your check returns rows with ID columns other than `unique_id`. Set it to the columns that contain resource IDs so selectors can scope rows by those columns. The following check queries the `edges` table and returns `child_unique_id` instead of `unique_id`, so you must set `selection_filter_on`: checks/multiple\_sources\_joined.sql ```sql select child_unique_id, count(*) as source_parents from {{ info_schema('edges') }} where parent_unique_id like 'source.%' group by child_unique_id having count(*) > 1 ``` Configure `selection_filter_on` for this check using one of the following methods: ##### Project YAML file dbt\_project.yml ```yaml checks: +selection_filter_on: child_unique_id ``` ##### Properties YAML file checks/\_checks.yml ```yaml version: 2 checks: - name: multiple_sources_joined description: "Fails if any model reads directly from more than one source." config: selection_filter_on: child_unique_id ``` ##### SQL file config checks/multiple\_sources\_joined.sql ```sql {{ config( selection_filter_on = "child_unique_id" ) }} ``` #### Related docs * [Checks](https://docs.getdbt.com/docs/build/checks.md) * [Check configurations](https://docs.getdbt.com/reference/check-configs.md) * [Using selectors with checks](https://docs.getdbt.com/docs/build/checks.md#using-selectors-with-checks) --- ### Semantic Layer configurations The Semantic Layer YAML spec defines every property and option for semantic models, metrics, and dimensions. Use it when you need the complete, authoritative list of what you can configure. For example, configurations when defining new objects, validating YAML, or troubleshooting config errors. Because this reference provides information about both the *latest spec* (model-embedded) and the *legacy spec* (standalone YAML), you'll need to select the appropriate version from the version picker. [Read the build docs](https://docs.getdbt.com/docs/build/semantic-models.md) to find out which applies to your environment. To convert from the legacy spec, see [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). Availability The latest YAML spec is supported in the following environments: * **dbt platform (v1 Latest release track)** * **dbt v2** * **dbt v1.12** #### Property reference The property reference pages document each resource type in detail so you can look up allowed values, syntax, and behavior for every property in the full YAML spec. Use the version picker on each page to see *latest spec (model YAML)* or *legacy spec (standalone YAML)* content: * [Semantic model properties](https://docs.getdbt.com/reference/semantic-model-properties.md) * [Metric properties](https://docs.getdbt.com/reference/metric-properties.md) * [Dimension properties](https://docs.getdbt.com/reference/dimension-properties.md) — for the latest spec, distinguishes **column-level** keys (`dimension:`, `granularity:`) from **properties inside** the `dimension:` block; use the version picker on that page for legacy standalone `dimensions:` fields. #### Where to define Semantic Layer objects | Object | Latest spec (model YAML) | Legacy spec (standalone YAML) | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | **Semantic models** | Top-level `semantic_model:` block under [models](https://docs.getdbt.com/reference/model-properties.md). | Top-level `semantic_models:` list. | | **Metrics** | For metrics that only depend on the same semantic model, nest them directly under each model in `models:` (but not nested under `semantic_model`).<br /><br />For metrics that depend on metrics or dimensions from a different semantic model, define them under a top-level `metrics:` block (For example, `outside models:`). This can be in the same YAML file or a separate one. | Top-level `metrics:` key in standalone YAML. | | **Dimensions** | `dimension:` blocks on model columns (and optional `derived_semantics.dimensions`). Defined within a model's semantic layer configuration. | `dimensions:` list on the semantic model. Defined within a model's semantic layer configuration. | #### Related documentation For the complete structure and examples, refer to these build docs: * [Build your metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md): Conceptual overview of the Semantic Layer, metric types, and how to get started. * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md): How to define semantic models (on a model or standalone), structure, and examples. * [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md): How to create and configure metrics; links to type-specific guides (simple, cumulative, ratio, derived, conversion). * [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md): How to define time and categorical dimensions within semantic models. * [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md): How to migrate from the legacy metrics YAML spec to the latest spec. --- ### Semantic manifest **Produced by:** Any command that parses your project. This includes all commands *except* [`deps`](https://docs.getdbt.com/reference/commands/deps.md), [`clean`](https://docs.getdbt.com/reference/commands/clean.md), [`debug`](https://docs.getdbt.com/reference/commands/debug.md), and [`init`](https://docs.getdbt.com/reference/commands/init.md). dbt creates an [artifact](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) file called the *Semantic Manifest* (`semantic_manifest.json`), which MetricFlow requires to build and run metric queries properly for the dbt Semantic Layer. This artifact contains comprehensive information about your dbt Semantic Layer. It is an internal file that acts as the integration point with MetricFlow. By using the semantic manifest produced by dbt, MetricFlow will instantiate a data flow plan and generate SQL from Semantic Layer query requests. It's a valuable reference that you can use to understand the structure and details of your data models. Similar to the [`manifest.json` file](https://docs.getdbt.com/reference/artifacts/manifest-json.md), the `semantic_manifest.json` file also lives in the [target directory](https://docs.getdbt.com/reference/global-configs/json-artifacts.md) of your dbt project where dbt stores various artifacts (such as compiled models and tests) generated during the execution of your project. There are two reasons why `semantic_manifest.json` exists alongside `manifest.json`: * Deserialization: dbt and MetricFlow use different libraries for handling data serialization. * Efficiency and performance: MetricFlow and the dbt Semantic Layer need specific semantic details from the manifest. By trimming down the information printed into `semantic_manifest.json`, the process becomes more efficient and enables faster data handling between `dbt` and MetricFlow. #### Top-level keys (Applies to dbt v1.12 and later) Top-level keys for the semantic manifest are: * `semantic_models` — Starting points of data with entities and dimensions, and correspond to models in your dbt project. * `metrics` — Functions combining entities, dimensions, and so on to define quantitative indicators. * `project_configuration` — Contains information around your project configurations * `saved_queries` — Saves commonly used queries in MetricFlow ##### Example target/semantic\_manifest.json ```json { "semantic_models": [ { "name": "semantic model name", "defaults": null, "description": "semantic model description", "node_relation": { "alias": "model alias", "schema_name": "model schema", "database": "model db", "relation_name": "Fully qualified relation name" }, "entities": ["entities in the semantic model"], "measures": ["measures in the semantic model"], "dimensions": ["dimensions in the semantic model" ], } ], "metrics": [ { "name": "name of the metric", "description": "metric description", "type": "metric type", "type_params": { "measure": { "name": "name for measure", "filter": "filter for measure", "alias": "alias for measure" }, "numerator": null, "denominator": null, "expr": null, "window": null, "grain_to_date": null, "metrics": ["metrics used in defining the metric. this is used in derived metrics"], "input_measures": [] }, "filter": null, "metadata": null } ], "project_configuration": { "time_spine_table_configurations": [ { "location": "fully qualified table name for timespine", "column_name": "date column", "grain": "day" } ], "metadata": null, "dsi_package_version": {} }, "saved_queries": [ { "name": "name of the saved query", "query_params": { "metrics": [ "metrics used in the saved query" ], "group_by": [ "TimeDimension('model_primary_key__date_column', 'day')", "Dimension('model_primary_key__metric_one')", "Dimension('model__dimension')" ], "where": null }, "description": "Description of the saved query", "metadata": null, "label": null, "exports": [ { "name": "saved_query_name", "config": { "export_as": "view", "schema_name": null, "alias": null } } ] } ] } ``` (Applies to dbt v1.12 and later) #### Apache Ossie document **Produced by:** Any command that parses your project (same as the semantic manifest). Starting in dbt v1.12, dbt also writes an `osi_document.json` file to your `target/` directory alongside `semantic_manifest.json` at parse time. This file represents your project's Semantic Layer in the [Apache Ossie](https://github.com/apache/ossie) format, a vendor-agnostic schema for describing semantic models and metrics. The Ossie document is generated by converting the full `PydanticSemanticManifest` to the Ossie format. Not all dbt semantic layer constructs have an Ossie equivalent, so dbt emits warnings (event code `I078`) when elements are dropped or degraded during conversion: | Warning | Cause | | --------------------------- | -------------------------------------------------------------------------------------------- | | `CONVERSION_METRIC_DROPPED` | Conversion metrics cannot be represented in Ossie and are excluded. | | `PRIVATE_METRIC_DROPPED` | Private metrics are not included in Ossie output. | | `NATURAL_ENTITY_DROPPED` | Natural entities have no Ossie equivalent and are excluded. | | `CUMULATIVE_SEMANTICS_LOSS` | The metric is included, but its cumulative window and grain semantics cannot be represented. | If the semantic manifest fails validation, dbt logs an `error` level `SemanticValidationFailure` and skips writing `osi_document.json` for that invocation. #### Related docs * [Semantic Layer API](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) * [About dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) --- ### Semantic model properties Semantic models define the structure that MetricFlow uses to build the semantic graph. In the *latest spec*, they can be declared as a top-level `semantic_model:` block on a [model](https://docs.getdbt.com/reference/model-properties.md). In the *legacy spec*, we used standalone YAML. For more information, refer to [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md). Availability The latest YAML spec is supported in the following environments: * **dbt platform (v1 Latest release track)** * **dbt v2** * **dbt v1.12** For more information, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). (Applies to dbt v1.12 and later) #### Latest spec (model YAML) In the latest spec, a model defines a semantic model with the `semantic_model` key, and that value can be: * **`true` or `false`**: Boolean shorthand. `true` enables a semantic model and uses the **model’s name** as the semantic model name (and other defaults where fields are unset). `false` turns off the semantic model for this model. * A **mapping**: An object with `enabled`, optional `name`, `group`, `config`, and so on, as in the table below. Dimensions and entities are defined with a column. ##### Available semantic model properties (latest spec) | Property / location | Type | Required | Description | | -------------------------------- | ----------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `semantic_model` | boolean or object | When using semantic layer for a model | `true` / `false`, or a mapping. When using a mapping, optional keys are in the following rows. | | `semantic_model.enabled` | boolean | No (mapping form) | Defaults to `true` when `semantic_model` is a mapping. Set `false` to disable. Omit when using `semantic_model: true` or `false`. | | `semantic_model.name` | string | No | Unique name; defaults to model name. Avoid double underscores (`__`) in the name; see [Name](https://docs.getdbt.com/docs/build/semantic-models.md#name) in **Semantic models**. | | `semantic_model.group` | string | No | Group for organization. | | `semantic_model.config` | object | No | Supports [meta](https://docs.getdbt.com/reference/resource-configs/meta.md), [group](https://docs.getdbt.com/reference/resource-configs/group.md), [enabled](https://docs.getdbt.com/reference/resource-configs/enabled.md). | | `agg_time_dimension` (top-level) | string | Yes | Default time dimension for metrics; references dimension name. | | `primary_entity` (top-level) | string | No | Primary entity name when no column has `type: primary`. | | `columns` | array | Yes | Column list; each can have `entity:` or `dimension:` block; time dimensions have `granularity` on the column. | | `derived_semantics` (top-level) | object | No | Optional dimensions and entities with `expr`. | | `metrics` (top-level) | array | No | Metrics derived from this semantic model; list is alongside (not under) `semantic_model`. | ##### Minimal structure example ```yaml models: - name: my_model semantic_model: enabled: true # explicit; defaults to true if omitted in mapping form name: optional_override # optional; defaults to model name group: optional_group config: meta: {} agg_time_dimension: my_time_dimension # top-level; references dimension name primary_entity: my_primary_entity # optional; use when no column has type: primary columns: - name: my_entity_column entity: type: primary | foreign | unique | natural name: optional_entity_name - name: my_time_dimension_column granularity: day # required for time dimensions dimension: type: time name: my_time_dimension - name: my_categorical_dimension_column dimension: type: categorical name: my_categorical_dimension derived_semantics: # optional dimensions: [] entities: [] metrics: - name: my_simple_metric type: simple agg: count expr: optional_expr ``` ##### Example with boolean instead of mapping Same layout as previous example, but with a boolean instead of a `semantic_model:` mapping (semantic model name defaults to the model name): ```yaml models: - name: my_model semantic_model: true agg_time_dimension: my_time_dimension primary_entity: my_primary_entity # optional columns: - name: my_entity_column entity: type: primary | foreign | unique | natural - name: my_time_dimension_column granularity: day dimension: type: time name: my_time_dimension metrics: - name: my_simple_metric type: simple agg: count ``` For the latest spec (model-embedded form with top-level `semantic_model:` and `metrics:` on the model), see [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md). --- ### Set operators Unions and intersections combine selector expressions when you list multiple arguments for `--select` or `--exclude`. A selector expression defines a subset of nodes in your project (models, tests, seeds, and other resource types) so you do not have to run the entire DAG. For the full selection language and for named selectors in `selectors.yml`, refer to [Syntax overview](https://docs.getdbt.com/reference/node-selection/syntax.md) and [YAML selectors](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md). #### Unions A union merges the node sets from several selector expressions; every resource that matches *any* of those expressions is selected. In other words, union behaves like OR across the arguments you pass to `--select` or `--exclude`. With unions, you can pass multiple arguments separated by spaces (space-delimited). dbt resolves each argument using the normal selection rules ([selection methods](https://docs.getdbt.com/reference/node-selection/methods.md), [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md), and other selection syntax), then combines the results. If more than one argument matches the same node, that node appears only once in the final selection. For example, the following command combines two selector expressions. Each uses `+` to include a model and its ancestors, and the space between them merges both sets: ```bash dbt run --select "+snowplow_sessions +fct_orders" ``` This behavior differs from an [intersection](#intersections), where comma-separated arguments with no spaces between them require a resource to satisfy *all* criteria at once. #### Intersections An intersection keeps only the nodes that match every selector expression; a resource must satisfy *all* comma-separated arguments to remain in the final set. Use commas with no spaces between arguments to define an intersection when you pass multiple arguments to `--select` or `--exclude`. Spaces between arguments still mean a [union](#unions). dbt resolves each argument using the normal selection rules ([selection methods](https://docs.getdbt.com/reference/node-selection/methods.md), [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md), and other selection syntax), then keeps only resources that satisfy all of them. The order of comma-separated arguments does not change the final set. The following examples show intersections: Select shared upstream nodes (common ancestors of `snowplow_sessions` and `fct_orders`): ```bash dbt run --select "+snowplow_sessions,+fct_orders" ``` Select shared downstream nodes (common descendants of `stg_invoices` and `stg_accounts`): ```bash dbt run --select "stg_invoices+,stg_accounts+" ``` Select models that are under the `marts/finance` path and tagged `nightly`: ```bash dbt run --select "marts.finance,tag:nightly" ``` #### Combining unions and intersections You can combine unions and intersections in a single `--select` or `--exclude` value to build complex selections in one command. dbt evaluates each space-delimited argument independently: commas with no spaces within an argument define an intersection, and a space between arguments combines their results as a union. Combining both operators lets you apply different filtering logic to different subsets of your project at once. For example, the following command contains two space-separated arguments, each defining an intersection: * `+snowplow_sessions,+fct_orders` selects nodes that are upstream of both `snowplow_sessions` and `fct_orders`. * `marts.finance,tag:nightly` selects models under the `marts/finance` path that are tagged `nightly`. The space between those arguments creates a union and combines both results into one selection: ```bash dbt run --select "+snowplow_sessions,+fct_orders marts.finance,tag:nightly" ``` --- ### severity, error_if, and warn_if Tests return a number of failures—most often, this is the count of rows returned by the test query, but it could be a [custom calculation](https://docs.getdbt.com/reference/resource-configs/fail_calc.md). Generally, if the number of failures is nonzero, the test returns an error. This makes sense, as test queries are designed to return all the rows you *don't* want: duplicate records, null values, and more. It's possible to configure tests to return warnings instead of errors, or to make the test status conditional on the number of failures returned. Maybe 1 duplicate record can count as a warning, but 10 duplicate records should count as an error. The relevant configs are: * `severity`: `error` or `warn` (default: `error`) * `error_if`: conditional expression (default: `!=0`) * `warn_if`: conditional expression (default: `!=0`) Conditional expressions can be any comparison logic that is supported by your SQL syntax with an integer number of failures: `> 5`, `= 0`, `between 5 and 10`, and so on. Here's how those play in practice: * If `severity: error`, dbt will check the `error_if` condition first. If the error condition is met, the test returns an error. If it's not met, dbt will then check the `warn_if` condition (defaulted to `!=0`). If it's not specified or the warn condition is met, the test warns; if it's not met, the test passes. * If `severity: warn`, dbt will skip the `error_if` condition entirely and jump straight to the `warn_if` condition. If the warn condition is met, the test warns; if it's not met, the test passes. By default, a test with `severity: warn` will only ever return a warning, and not cause errors. However, you can promote warnings to errors using: * `--warn-error`: Promotes *all* dbt warnings (including test warnings, Jinja warnings, deprecations, and so on.) to errors. * `--warn-error-options`: Promotes *only specific types* of warnings. For more information, refer to [Warnings](https://docs.getdbt.com/reference/global-configs/warnings.md). ##### Out-of-the-box generic tests Configure a specific instance of an out-of-the-box generic test: models/\<filename>.yml ```yaml models: - name: large_table columns: - name: slightly_unreliable_column data_tests: - unique: config: severity: error error_if: ">1000" warn_if: ">10" ``` ##### Singular tests Configure a singular test: tests/\<filename>.sql ```sql {{ config(error_if = '>50') }} select ... ``` ##### Custom generic tests Set the default for all instances of a custom generic test, by setting the config inside its test block (definition): macros/\<filename>.sql ```sql {% test <testname>(model, column_name) %} {{ config(severity = 'warn') }} select ... {% endtest %} ``` ##### Project level Set the default for all tests in a package or project: dbt\_project.yml ```yaml data_tests: +severity: warn # all tests <package_name>: +warn_if: >10 # tests in <package_name> ``` ##### Asserting an expected failure You can use `error_if` to assert an expected failure. This is useful in package integration tests, for example when validating that a generic test catches known-bad fixture data. In the following example, the test passes only when it returns one or more failing rows. If the test returns `0` rows, dbt raises an error because `0` satisfies `error_if: '<1'`. models/always\_bad.yml ```yaml models: - name: always_bad data_tests: - dbt_utils.expression_is_true: arguments: expression: "amount > 0" config: error_if: "<1" warn_if: "<0" ``` In this example, the test passes only when it returns one or more failing rows. If the test returns `0` rows, dbt raises an error because `0` satisfies `error_if: '<1'`. Set `warn_if: '<0'` to take warning behavior out of play for this pattern. Without overriding `warn_if`, the default warning condition (`!=0`) can still apply. --- ### SingleStore configurations #### Incremental materialization strategies The [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-models.md#about-incremental_strategy) controls how dbt builds incremental models. Currently, SingleStoreDB supports `delete+insert`, `append`, and `microbatch` configurations. The `delete+insert` incremental strategy directs dbt to follow a two-step incremental approach. Initially, it identifies and removes the records flagged by the configured `is_incremental()` block. Subsequently, it re-inserts these records. #### Performance Optimizations [SingleStore Physical Database Schema Design documentation](https://docs.singlestore.com/managed-service/en/create-a-database/physical-database-schema-design/concepts-of-physical-database-schema-design.html) is helpful if you want to use specific options (that are described below) in your dbt project. ##### Storage type SingleStore supports two storage types: **In-Memory Rowstore** and **Disk-based Columnstore** (the latter is default). See [the docs](https://docs.singlestore.com/managed-service/en/create-a-database/physical-database-schema-design/concepts-of-physical-database-schema-design/choosing-a-table-storage-type.html) for details. The dbt-singlestore adapter allows you to specify which storage type your table materialization would rely on using `storage_type` config parameter. rowstore\_model.sql ```sql {{ config(materialized='table', storage_type='rowstore') }} select ... ``` ##### Reference tables SingleStore supports **REFERENCE** tables (available starting from \[dbt-singlestore 1.10.0] (<https://pypi.org/project/dbt-singlestore/1.10.0/>)), which are replicated across the cluster and are useful for small/dimension tables that are frequently joined. To create a **REFERENCE** table from a dbt model, set `reference=true` on a `table` materialization: ```sql {{ config( materialized='table', reference=true, ) }} select ... ``` When `reference=true` (default `false`), the adapter generates `CREATE REFERENCE TABLE ...` rather than a regular `CREATE TABLE ...` ##### Rowstore reference tables If you want a rowstore reference table, set `storage_type='rowstore'`: ```sql {{ config( materialized='table', reference=true, storage_type='rowstore', ) }} select ... ``` This maps to `CREATE ROWSTORE REFERENCE TABLE ...` ###### Restrictions / validation To match SingleStore semantics, dbt-singlestore enforces: * No `shard_key` with `reference=true` — reference tables don’t use sharding; compilation fails if both are set. * SingleStore doesn’t support temporary reference tables. dbt-singlestore fails compilation if `reference=true` would result in a temporary table being created (for example, when `temporary=true` is set, or when a materialization strategy uses temporary tables internally). ##### Keys SingleStore tables are [sharded](https://docs.singlestore.com/managed-service/en/getting-started-with-managed-service/about-managed-service/sharding.html) and can be created with various column definitions. The following options are supported by the dbt-singlestore adapter, each of them accepts `column_list` (a list of column names) as an option value. Please refer to [Creating a Columnstore Table](https://docs.singlestore.com/managed-service/en/create-a-database/physical-database-schema-design/procedures-for-physical-database-schema-design/creating-a-columnstore-table.html) for more information on various key types in SingleStore. * `primary_key` (translated to `PRIMARY KEY (column_list)`) * `sort_key` (translated to `KEY (column_list) USING CLUSTERED COLUMNSTORE`) * `shard_key` (translated to `SHARD KEY (column_list)`) * `unique_table_key` (translated to `UNIQUE KEY (column_list)`) primary\_and\_shard\_model.sql ```sql {{ config( primary_key=['id', 'user_id'], shard_key=['id'] ) }} select ... ``` unique\_and\_sort\_model.sql ```sql {{ config( materialized='table', unique_table_key=['id'], sort_key=['status'], ) }} select ... ``` ##### Indexes Similarly to the Postgres adapter, table models, incremental models, seeds, and snapshots may have a list of `indexes` defined. Each index can have the following components: * `columns` (list, required): one or more columns on which the index is defined * `unique` (boolean, optional): whether the index should be declared unique * `type` (string, optional): a supported [index type](https://docs.singlestore.com/managed-service/en/reference/sql-reference/data-definition-language-ddl/create-index.html), `hash` or `btree` As SingleStore tables are sharded, there are certain limitations to indexes creation, see the [docs](https://docs.singlestore.com/managed-service/en/create-a-database/physical-database-schema-design/concepts-of-physical-database-schema-design/understanding-keys-and-indexes-in-singlestore.html) for more details. indexes\_model.sql ```sql {{ config( materialized='table', shard_key=['id'], indexes=[{'columns': ['order_date', 'id']}, {'columns': ['status'], 'type': 'hash'}] ) }} select ... ``` ##### Other options You can specify the character set and collation for the table using `charset` and/or `collation` options. Supported values for `charset` are `binary`, `utf8`, and `utf8mb4`. Supported values for `collation` can be viewed as the output of `SHOW COLLATION` SQL query. Default collations for the corresponding character sets are `binary`, `utf8_general_ci`, and `utf8mb4_general_ci`. utf8mb4\_model.sql ```sql {{ config( charset='utf8mb4', collation='utf8mb4_general_ci' ) }} select ... ``` #### Model contracts Starting from 1.5, the `dbt-singlestore` adapter supports model contracts. | Constraint type | Support | Platform enforcement | | --------------- | ---------------- | -------------------- | | not\_null | ✅ Supported | ✅ Enforced | | primary\_key | ✅ Supported | ❌ Not enforced | | foreign\_key | ❌ Not supported | ❌ Not enforced | | unique | ✅ Supported | ❌ Not enforced | | check | ❌ Not supported | ❌ Not enforced | Consider the following restrictions while using contracts with the `dbt-singlestore` adapter: ##### Model and Column Definitions: * The `unique` constraint can only be set at the model level. Hence, do not set it at the column level. * Repeating constraints will return an error. For example, setting `primary_key` in both column and model settings returns an error. ##### Overwriting Settings: The contract setting overrides the configuration setting. For example, if you define a `primary_key` or `unique_table_key` in the config and then also set it in the contract, the contract setting replaces the configuration setting. ##### Working with constants: dim\_customers.yml ```sql models: - name: dim_customers config: materialized: table contract: enforced: true columns: - name: customer_id data_type: int constraints: - type: not_null - name: customer_name data_type: text ``` Let's say your model is defined as: dim\_customers.sql ```sql select 'abc123' as customer_id, 'My Best Customer' as customer_name ``` When using constants, you must specify the data types directly. If not, SingleStoreDB will automatically choose what it thinks is the most appropriate data type. dim\_customers.sql ```sql select ('abc123' :> int) as customer_id, ('My Best Customer' :> text) as customer_name ``` ##### Misleading datatypes Using `model contracts` ensures that you don't accidentally add the wrong type of data into a column. For instance, if you expect a number in a column, but accidentally specify text to be added, the model contract catches it and returns an error. The error message may occasionally show a different data type name than expected, because of how the `singlestoredb-python` connector works. For instance, dim\_customers.sql ```sql select 'abc123' as customer_id, ('My Best Customer' :> text) as customer_name ``` will result in ```sql Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition. | column_name | definition_type | contract_type | mismatch_reason | | customer_id | LONGBLOB | LONG | data type mismatch | ``` It's important to note that certain data type mappings might show up differently in error messages, but this doesn't affect how they work. Here's a quick list of what you might see: | Data type | Data type returned by<br />singlestoredb-python | | ---------- | ----------------------------------------------- | | BOOL | TINY | | INT | LONG | | CHAR | BINARY | | VARCHAR | VARBINARY | | TEXT | BLOB | | TINYTEXT | TINYBLOB | | MEDIUMTEXT | MEDIUMBLOB | | LONGTEXT | LONGBLOB | Just keep these points in mind when setting up and using your `dbt-singlestore` adapter, and you'll avoid common pitfalls! --- ### skill-paths dbt\_project.yml ```yml skill-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [agent skills](https://docs.getdbt.com/docs/dbt-ai/package-skills.md) are located. Every subdirectory containing a `SKILL.md` file is treated as one skill. dbt scans `skill-paths` in your root project and in each installed package when it installs packages. #### Default By default, dbt looks for skills in a directory named `skills` in the root of your project. Paths specified in `skill-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/skills`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml skill-paths: ["skills"] ``` * ❌ **Don't** * Avoid absolute paths: ```yml skill-paths: ["/Users/username/project/skills"] ``` #### Examples ##### Use a subdirectory named `agent-skills` for skills dbt\_project.yml ```yml skill-paths: ["agent-skills"] ``` ##### Use more than one directory for skills dbt\_project.yml ```yml skill-paths: ["skills", "team-skills"] ``` --- ### Snapshot configurations Learn about using snapshot configurations in dbt, including snapshot-specific configurations and general configurations. #### Related documentation * [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) * The `dbt snapshot` [command](https://docs.getdbt.com/reference/commands/snapshot.md) Learn by video! For video tutorials on Snapshots, go to dbt Learn and check out the [Snapshots course](https://learn.getdbt.com/courses/snapshots). #### Available configurations ##### Snapshot-specific configurations Resource-specific configurations are applicable to only one dbt resource type rather than multiple resource types. You can define these settings in the project file (`dbt_project.yml`), a property file (`models/properties.yml` for models, similarly for other resources), or within the resource’s file using the `{{ config() }}` macro.<br /> The following resource-specific configurations are only available to Snapshots: ##### Project file (Applies to dbt v1.9 and later) dbt\_project.yml ```yaml snapshots: <resource-path>: +schema: <string> +database: <string> +alias: <string> +unique_key: <column_name_or_expression> +strategy: timestamp | check +updated_at: <column_name> +check_cols: [<column_name>] | all +snapshot_meta_column_names: {<dictionary>} +dbt_valid_to_current: <string> +hard_deletes: string ``` ##### Property file (Applies to dbt v1.9 and later) Refer to [configuring snapshots](https://docs.getdbt.com/docs/build/snapshots.md#configuring-snapshots) for the available configurations. snapshots/schema.yml ```yml snapshots: - name: <string> relation: ref() | source() config: database: <string> schema: <string> unique_key: <column_name_or_expression> strategy: timestamp | check updated_at: <column_name> check_cols: [<column_name>] | all snapshot_meta_column_names: {<dictionary>} hard_deletes: string dbt_valid_to_current: <string> ``` ##### SQL file config info Starting from [the dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and dbt v1.9, defining snapshots in a `.sql` file using a config block is a legacy method. You can define snapshots in properties YAML files using the latest [snapshot-specific configurations](https://docs.getdbt.com/docs/build/snapshots.md#configuring-snapshots). For new snapshots, we recommend using these latest configs. If applying them to existing snapshots, you'll need to [migrate](#snapshot-configuration-migration) over. ##### Snapshot configuration migration The latest snapshot configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md), [`dbt_valid_to_current`](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md), and `hard_deletes`) are best suited for new snapshots, but you can also adopt them in existing snapshots by migrating your table schema and configs carefully to avoid any inconsistencies in your snapshots. Here's how you can do it: 1. In your data platform, create a backup snapshot table. You can copy it to a new table: ```sql create table my_snapshot_table_backup as select * from my_snapshot_table; ``` This allows you to restore your snapshot if anything goes wrong during migration. 2. If you want to use the new configs, add required columns to your existing snapshot table using `alter` statements as needed. Here's an example of what to add if you're going to use `dbt_valid_to_current` and `snapshot_meta_column_names`: ```sql alter table my_snapshot_table add column dbt_valid_from timestamp, add column dbt_valid_to timestamp; ``` 3. Then update your snapshot config: ```yaml snapshots: - name: orders_snapshot relation: source('something','orders') config: strategy: timestamp updated_at: updated_at unique_key: id dbt_valid_to_current: "to_date('9999-12-31')" snapshot_meta_column_names: dbt_valid_from: start_date dbt_valid_to: end_date ``` 4. Test each change before adopting multiple new configs by running `dbt snapshot` in development or staging. 5. Confirm if the snapshot run completes without errors, the new columns are created, and historical logic behaves as you’d expect. The table should look like this: | `id` | `start_date` | `end_date` | `updated_at` | | ---- | ------------------- | ------------------- | ------------------- | | 1 | 2024-10-01 09:00:00 | 2024-10-03 08:00:00 | 2024-10-01 09:00:00 | | 2 | 2024-10-03 08:00:00 | 9999-12-31 00:00:00 | 2024-10-03 08:00:00 | | 3 | 2024-10-02 11:15:00 | 9999-12-31 00:00:00 | 2024-10-02 11:15:00 | Note: The `end_date` column (defined by `snapshot_meta_column_names`) uses the configured value from `dbt_valid_to_current` (9999-12-31) for newly inserted records, instead of the default `NULL`. Existing records will have `NULL` for `end_date`. warning If you use one of the latest configs, such as `dbt_valid_to_current`, without migrating your data, you may have mixed old and new data, leading to an incorrect downstream result. ##### General configurations General configurations provide broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project file, property files, or within resource-specific files. ##### Project file dbt\_project.yml (Applies to dbt v1.9 and later) ```yaml snapshots: <resource-path>: +enabled: true | false +tags: <string> | [<string>] +alias: <string> +pre-hook: <sql-statement> | [<sql-statement>] +post-hook: <sql-statement> | [<sql-statement>] +persist_docs: {<dict>} +grants: {<dict>} +event_time: my_time_field ``` ##### Property file (Applies to dbt v1.9 and later) snapshots/properties.yml ```yaml snapshots: - name: [<snapshot-name>] relation: source('my_source', 'my_table') config: enabled: true | false tags: <string> | [<string>] alias: <string> pre_hook: <sql-statement> | [<sql-statement>] post_hook: <sql-statement> | [<sql-statement>] persist_docs: {<dict>} grants: {<dictionary>} event_time: my_time_field ``` ##### SQL file config info Starting from [the dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and dbt v1.9, defining snapshots in a `.sql` file using a config block is a legacy method. You can define snapshots in properties YAML files using the latest [snapshot-specific configurations](https://docs.getdbt.com/docs/build/snapshots.md#configuring-snapshots). For new snapshots, we recommend using these latest configs. If applying them to existing snapshots, you'll need to [migrate](#snapshot-configuration-migration) over. #### Configuring snapshots Snapshots can be configured in multiple ways: (Applies to dbt v1.9 and later) 1. Defined in YAML files using the `config` [resource property](https://docs.getdbt.com/reference/model-properties.md), typically in your [snapshots directory](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md) or whichever folder you prefer. Available in [the dbt release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), dbt v1.9 and higher. 2. From the `dbt_project.yml` file, under the `snapshots:` key. To apply a configuration to a snapshot, or directory of snapshots, define the resource path as nested dictionary keys. Snapshot configurations are applied hierarchically in the order above with higher taking precedence. You can also apply [data tests](https://docs.getdbt.com/reference/snapshot-properties.md) to snapshots using the [`tests` property](https://docs.getdbt.com/reference/resource-properties/data-tests.md). ##### Examples (Applies to dbt v1.9 and later) The following examples demonstrate how to configure snapshots using the `dbt_project.yml` file and a `.yml` file. * ###### Apply configurations to all snapshots To apply a configuration to all snapshots, including those in any installed [packages](https://docs.getdbt.com/docs/build/packages.md), nest the configuration directly under the `snapshots` key: dbt\_project.yml ```yml snapshots: +unique_key: id ``` * ###### Apply configurations to all snapshots in your project To apply a configuration to all snapshots in your project only (for example, *excluding* any snapshots in installed packages), provide your project name as part of the resource path. For a project named `jaffle_shop`: dbt\_project.yml ```yml snapshots: jaffle_shop: +unique_key: id ``` Similarly, you can use the name of an installed package to configure snapshots in that package. * ###### Apply configurations to one snapshot only (Applies to dbt v1.9 and later) snapshots/postgres\_app/order\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: source('jaffle_shop', 'orders') config: unique_key: id strategy: timestamp updated_at: updated_at persist_docs: relation: true columns: true ``` Pro-tip: Use sources in snapshots: `select * from {{ source('jaffle_shop', 'orders') }}` You can also use the full resource path (including the project name, and subdirectories) to configure an individual snapshot from your `dbt_project.yml` file. For a project named `jaffle_shop`, with a snapshot file within the `snapshots/postgres_app/` directory, where the snapshot is named `orders_snapshot` (as above), this would look like: dbt\_project.yml ```yml snapshots: jaffle_shop: postgres_app: orders_snapshot: +unique_key: id +strategy: timestamp +updated_at: updated_at ``` You can also define some common configs in a snapshot's `config` block. However, we don't recommend this for a snapshot's required configuration. dbt\_project.yml ```yml snapshots: - name: orders_snapshot +persist_docs: relation: true columns: true ``` --- ### Snapshot properties Define snapshot properties in YAML to document snapshots, configure settings, add tests, and describe columns. (Applies to dbt v1.9 and later) In dbt v1.9 and later, snapshots are defined and configured in YAML files within your `snapshots/` directory (as defined by the [`snapshot-paths` config](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md)). Snapshot properties are declared within these YAML files, allowing you to define both the snapshot configurations and properties in one place. We recommend that you put them in the `snapshots/` directory. You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `snapshots/` or `models/` directory. Learn by video! For video tutorials on Snapshots, go to dbt Learn and check out the [Snapshots course](https://learn.getdbt.com/courses/snapshots). (Applies to dbt v1.9 and later) snapshots/\<filename>.yml ```yml snapshots: - name: <snapshot name> description: <markdown_string> config: <snapshot_config>: <config_value> meta: {<dictionary>} docs: show: true | false node_color: <color_id> # Use name (such as node_color: purple) or hex code with quotes (such as node_color: "#cd7f32") data_tests: - <test> - ... columns: - name: <column name> description: <markdown_string> quote: true | false data_tests: - <test> - ... # declare additional tests config: meta: {<dictionary>} tags: [<string>] - ... # declare properties of additional columns - name: ... # declare properties of additional snapshots ``` --- ### snapshot_meta_column_names Did you know... Available from dbt v1.9 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). snapshots/schema.yml ```yaml snapshots: - name: <snapshot_name> config: snapshot_meta_column_names: dbt_valid_from: <string> dbt_valid_to: <string> dbt_scd_id: <string> dbt_updated_at: <string> dbt_is_deleted: <string> ``` snapshots/\<filename>.sql ```jinja2 {{ config( snapshot_meta_column_names={ "dbt_valid_from": "<string>", "dbt_valid_to": "<string>", "dbt_scd_id": "<string>", "dbt_updated_at": "<string>", "dbt_is_deleted": "<string>", } ) }} ``` dbt\_project.yml ```yml snapshots: <resource-path>: +snapshot_meta_column_names: dbt_valid_from: <string> dbt_valid_to: <string> dbt_scd_id: <string> dbt_updated_at: <string> dbt_is_deleted: <string> ``` #### Description In order to align with an organization's naming conventions, the `snapshot_meta_column_names` config can be used to customize the names of the [metadata columns](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields) within each snapshot. #### Default By default, dbt snapshots use the following column names to track change history using [Type 2 slowly changing dimension](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) records: | Field | Meaning | Notes | Example | | ---------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `dbt_valid_from` | The timestamp when this snapshot row was first inserted and became valid. | The value is affected by the [`strategy`](https://docs.getdbt.com/reference/resource-configs/strategy.md). | `snapshot_meta_column_names: {dbt_valid_from: start_date}` | | `dbt_valid_to` | The timestamp when this row is no longer valid. | | `snapshot_meta_column_names: {dbt_valid_to: end_date}` | | `dbt_scd_id` | A unique key generated for each snapshot row. | This is used internally by dbt. | `snapshot_meta_column_names: {dbt_scd_id: scd_id}` | | `dbt_updated_at` | The `updated_at` timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt. | `snapshot_meta_column_names: {dbt_updated_at: modified_date}` | | `dbt_is_deleted` | A string value indicating if the record has been deleted. (`True` if deleted, `False` if not deleted). | Added when `hard_deletes='new_record'` is configured. | `snapshot_meta_column_names: {dbt_is_deleted: is_deleted}` | All of these column names can be customized using the `snapshot_meta_column_names` config. Refer to the [Example](#example) for more details. warning To avoid any unintentional data modification, dbt will **not** automatically apply any column renames. So if a user applies `snapshot_meta_column_names` config for a snapshot without updating the pre-existing table, they will get an error. We recommend either only using these settings for net-new snapshots, or arranging an update of pre-existing tables prior to committing a column name change. #### How [`dbt_scd_id`](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md#default) is calculated `dbt_scd_id` is a unique identifier generated for each row in a snapshot. dbt uses this identifier to detect changes in source records and manage versioning in slowly changing dimension (SCD) snapshots. dbt's snapshot macro handles `dbt_scd_id` in [the dbt-adapters repository](https://github.com/dbt-labs/dbt-adapters/blob/b12c9870d6134905ab09bfda609ce8f81bc4b40a/dbt/include/global_project/macros/materializations/snapshots/strategies.sql#L38). The hash is computed by concatenating values of the snapshot's [`unique_key`](https://docs.getdbt.com/reference/resource-configs/unique_key.md) and either the `updated_at` timestamp (for the timestamp strategy) or the values in `check_cols` (for the check strategy), and then hashing the resulting string using the `md5` function. This enables dbt to track whether the contents of a row have changed between runs. Here's an example of a custom hash calculation that combines multiple fields into a single string and hashes the result using `md5`. ```sql md5( coalesce(cast(unique_key1 as string), '') || '|' || coalesce(cast(unique_key2 as string), '') || '|' || coalesce(cast(updated_at as string), '') ) ``` The exact fields included in the hash depend on the snapshot strategy: * [`timestamp` strategy](https://docs.getdbt.com/reference/resource-configs/strategy.md#use-the-timestamp-strategy): The hash typically combines the [`unique_key`](https://docs.getdbt.com/reference/resource-configs/unique_key.md) columns and the `updated_at` value. * [`check` strategy](https://docs.getdbt.com/reference/resource-configs/strategy.md#use-the-check-strategy): The hash combines the `unique_key` columns and the values of the columns listed in [`check_cols`](https://docs.getdbt.com/reference/resource-configs/check_cols.md). If you don’t want to use `md5`, you can customize the [dispatched macro](https://github.com/dbt-labs/dbt-adapters/blob/4b3966efc50b1d013907a88bee4ab8ebd022d17a/dbt-adapters/src/dbt/include/global_project/macros/materializations/snapshots/strategies.sql#L42-L47). #### Example snapshots/schema.yml ```yaml snapshots: - name: orders_snapshot relation: ref("orders") config: unique_key: id strategy: check check_cols: all hard_deletes: new_record snapshot_meta_column_names: dbt_valid_from: start_date dbt_valid_to: end_date dbt_scd_id: scd_id dbt_updated_at: modified_date dbt_is_deleted: is_deleted ``` The resulting snapshot table contains the configured meta column names: | id | scd\_id | modified\_date | start\_date | end\_date | is\_deleted | | -- | -------------------- | -------------- | -------------- | -------------- | ----------- | | 1 | 60a1f1dbdf899a4dd... | 2024-10-02 ... | 2024-10-02 ... | 2024-10-03 ... | False | | 1 | 60a1f1dbdf899a4dd... | 2024-10-03 ... | 2024-10-03 ... | | True | | 2 | b1885d098f8bcff51... | 2024-10-02 ... | 2024-10-02 ... | | False | --- ### snapshot_name (Applies to dbt v1.9 and later) snapshots/\<filename>.yml ```yaml snapshots: - name: snapshot_name relation: source('my_source', 'my_table') config: schema: string database: string unique_key: column_name_or_expression strategy: timestamp | check updated_at: column_name # Required if strategy is 'timestamp' ``` #### Description The name of a snapshot, which is used when selecting from a snapshot using the [`ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) This name must not conflict with the name of any other "refable" resource (models, seeds, other snapshots) defined in this project or package. The name does not need to match the file name. As a result, snapshot filenames do not need to be unique. #### Examples ##### Name a snapshot `order_snapshot` (Applies to dbt v1.9 and later) snapshots/order\_snapshot.yml ```yaml snapshots: - name: order_snapshot relation: source('my_source', 'my_table') config: schema: string database: string unique_key: column_name_or_expression strategy: timestamp | check updated_at: column_name # Required if strategy is 'timestamp' ``` To select from this snapshot in a downstream model: ```sql select * from {{ ref('orders_snapshot') }} ``` --- ### snapshot-paths dbt\_project.yml ```yml snapshot-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) are located. (Applies to dbt v1.9 and later) In dbt v1.9+, you can co-locate your snapshots with models if they are [defined using the latest YAML syntax](https://docs.getdbt.com/docs/build/snapshots.md). #### Default By default, dbt will search for snapshots in the `snapshots` directory. For example, `snapshot-paths: ["snapshots"]`. Paths specified in `snapshot-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/snapshots`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml snapshot-paths: ["snapshots"] ``` * ❌ **Don't:** * Avoid absolute paths: ```yml snapshot-paths: ["/Users/username/project/snapshots"] ``` #### Examples ##### Use a subdirectory named `archives` instead of `snapshots` dbt\_project.yml ```yml snapshot-paths: ["archives"] ``` --- ### Snowflake adapter behavior changes The following are the current [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) that are specific to `dbt-snowflake`: | Flag | `dbt-snowflake`: Intro | `dbt-snowflake`: Maturity | Status | | ---------------------------------------------------------------------------------------------------- | ---------------------- | ------------------------- | ------ | | [`snowflake_default_transient_dynamic_tables`](#the-snowflake_default_transient_dynamic_tables-flag) | 1.12.0 | TBD | Active | | [`snowflake_managed_iceberg_default`](#the-snowflake_managed_iceberg_default-flag) | 1.12.0 | TBD | Active | #### The `snowflake_default_transient_dynamic_tables` flag Available starting `dbt-snowflake` v1.12. The `snowflake_default_transient_dynamic_tables` flag controls whether Snowflake dynamic tables are created as transient when the model config does not explicitly set the [`transient`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#transient-dynamic-tables) config. * When set to `false` (default): Dynamic tables are created as permanent tables with a [Fail-safe period](https://docs.snowflake.com/en/user-guide/data-failsafe) unless you set `transient: true` for a specific model. * When set to `true`: Dynamic tables are created as transient (no Fail-safe period) when `transient` is not specified in the model config. Transient dynamic tables can reduce storage costs. Set the `snowflake_default_transient_dynamic_tables` flag in your `dbt_project.yml` under the `flags` key. You can override the default setting using the [`transient`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#transient-dynamic-tables) config on dynamic table models. #### The `snowflake_managed_iceberg_default` flag dbt v2 In v2, this behavior is the default and there's no flag to set. `external_volume` defaults to `SNOWFLAKE_MANAGED`, and dbt omits `base_location` whenever the table uses Snowflake-managed storage. Available starting `dbt-snowflake` v1.12. The `snowflake_managed_iceberg_default` flag controls what dbt does when a model sets `table_format: iceberg` for an [Iceberg table](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) without an `external_volume`. Snowflake rejects `BASE_LOCATION` for Iceberg tables that use [Snowflake-managed storage](https://docs.snowflake.com/en/user-guide/tables-iceberg-internal-storage) and returns the error `BASE_LOCATION property is not supported for Iceberg tables using Snowflake Managed Storage`. This flag lets dbt generate data definition language (DDL) that Snowflake Horizon accepts without requiring you to configure a catalog. * When set to `false` (default): dbt doesn't emit `external_volume`. Snowflake falls back to any [default external volume](https://docs.snowflake.com/user-guide/tables-iceberg-configure-external-volume#set-a-default-external-volume-at-the-account-database-or-schema-level) set at the account, database, or schema level, and dbt still generates a `base_location`. * When set to `true`: dbt emits `external_volume = 'SNOWFLAKE_MANAGED'` and omits `base_location`, so the table uses Snowflake-managed storage. The flag defaults to `false` to avoid changing the target storage for projects that rely on a default external volume set at the account, database, or schema level. Two cases don't depend on this flag: * When you set `external_volume` to `SNOWFLAKE_MANAGED`, dbt always omits `base_location`. dbt resolves `external_volume` from the model config first and then from the catalog definition in `catalogs.yml`, so setting it in either place omits `base_location`. * When you set `external_volume` to a user-defined volume, such as Amazon S3, Azure, or Google Cloud Storage, dbt always emits both `external_volume` and `base_location`. Set the flag in your `dbt_project.yml` under the `flags` key: dbt\_project.yml ```yaml flags: snowflake_managed_iceberg_default: true ``` --- ### Snowflake configurations Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. #### Iceberg table format Our Snowflake Iceberg table content has moved to a [new page](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md)! #### Dynamic tables The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about). This materialization is specific to Snowflake, which means that any model configuration that would normally come along for the ride from `dbt` (like with a `view`) may not be available for dynamic tables. This gap will decrease in future patches and versions. While this materialization is specific to Snowflake, it very much follows the implementation of [materialized views](https://docs.getdbt.com/docs/build/materializations.md#Materialized-View). In particular, dynamic tables have access to the `on_configuration_change` setting. Dynamic tables are supported with the following configuration parameters: (Applies to dbt v1.12 and later) | Parameter | Type | Required | Default | Change Monitoring Support | | ---------------------------------------------------------------------------------------------------------- | ---------------------- | -------- | ----------- | ------------------------- | | [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) | `<string>` | no | `apply` | n/a | | [`target_lag`](#target-lag) | `<string>` | no | | alter | | [`scheduler`](#scheduler) | `<string>` | no | `DISABLE` | alter | | [`snowflake_warehouse`](#configuring-virtual-warehouses) | `<string>` | yes | | alter | | [`snowflake_initialization_warehouse`](#initialization-warehouse) | `<string>` | no | `None` | alter | | [`refresh_warehouse`](#refresh-warehouse) | `<string>` | no | `None` | alter | | [`refresh_mode`](#refresh-mode) | `<string>` | no | `AUTO` | refresh | | [`initialize`](#initialize) | `<string>` | no | `ON_CREATE` | n/a | | [`cluster_by`](#dynamic-table-clustering) | `<string>` or `<list>` | no | `None` | alter | | [`immutable_where`](#immutable-where) | `<string>` | no | `None` | alter | | [`copy_grants`](#copy-grants-dynamic-tables) | `<boolean>` | no | `false` | full refresh | | [`transient`](#transient-dynamic-tables) | `<boolean>` | no | `false` | full refresh | ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: dynamic_table +on_configuration_change: apply | continue | fail +target_lag: downstream | <time-delta> +scheduler: ENABLE | DISABLE +snowflake_warehouse: <warehouse-name> +snowflake_initialization_warehouse: <warehouse-name> +refresh_warehouse: <warehouse-name> +refresh_mode: AUTO | FULL | INCREMENTAL +initialize: ON_CREATE | ON_SCHEDULE +cluster_by: <column-name> | [<column-name>, <column-name>, ...] +immutable_where: <condition> +copy_grants: true | false +transient: true | false ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: dynamic_table on_configuration_change: apply | continue | fail target_lag: downstream | <time-delta> scheduler: ENABLE | DISABLE snowflake_warehouse: <warehouse-name> snowflake_initialization_warehouse: <warehouse-name> refresh_warehouse: <warehouse-name> refresh_mode: AUTO | FULL | INCREMENTAL initialize: ON_CREATE | ON_SCHEDULE cluster_by: <column-name> | [<column-name>, <column-name>, ...] immutable_where: <condition> copy_grants: true | false transient: true | false ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized="dynamic_table", on_configuration_change="apply" | "continue" | "fail", target_lag="downstream" | "<integer> seconds | minutes | hours | days", scheduler="ENABLE" | "DISABLE", snowflake_warehouse="<warehouse-name>", snowflake_initialization_warehouse="<warehouse-name>", refresh_warehouse="<warehouse-name>", refresh_mode="AUTO" | "FULL" | "INCREMENTAL", initialize="ON_CREATE" | "ON_SCHEDULE", cluster_by="<column-name>" | ["<column-name>", "<column-name>", ...], immutable_where="<condition>", copy_grants=true | false, transient=true | false, ) }} ``` Learn more about these parameters in Snowflake's [docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table): ##### Target lag Snowflake allows two configuration scenarios for scheduling automatic refreshes: * **Time-based** — Provide a value of the form `<int> { seconds | minutes | hours | days }`. For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`. * **Downstream** — Applicable when the dynamic table is referenced by other dynamic tables. In this scenario, `target_lag='downstream'` allows for refreshes to be controlled at the target, instead of at each layer. (Applies to dbt v1.12 and later) ###### How `target_lag` interacts with `scheduler` `target_lag` works with [`scheduler`](#scheduler) to determine how dynamic table refreshes are managed: | `target_lag` | `scheduler` | Behavior | | ------------ | -------------------- | --------------------------------------------------------------------------------------------------- | | Set | `ENABLE` or omitted | Snowflake manages refreshes automatically. If `scheduler` is omitted, dbt defaults to `ENABLE`. | | Not set | `DISABLE` or omitted | dbt manages refreshes during model execution. If `scheduler` is omitted, dbt defaults to `DISABLE`. | | Set | `DISABLE` | Invalid: `DISABLE` does not accept `target_lag`. dbt raises an error. | | Not set | `ENABLE` | Invalid: `ENABLE` requires `target_lag`. dbt raises an error. | Learn more about `target_lag` in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh#understanding-target-lag). Please note that Snowflake supports a target lag of 1 minute or longer. (Applies to dbt v1.12 and later) ##### Scheduler The `scheduler` parameter controls whether the dynamic table is refreshed by Snowflake's background scheduler or by an external orchestrator (for example, dbt). Snowflake accepts two options: * **ENABLE** — Snowflake's built-in scheduler automatically refreshes the dynamic table based on the defined `target_lag`. Refreshes cascade across the dependency graph to maintain snapshot consistency. Setting `target_lag` is *required* when using this option. * **DISABLE** — The dynamic table is excluded from Snowflake's automatic background refresh. You must trigger refreshes manually or through orchestration external to Snowflake (for example, by a `dbt run` that executes `ALTER DYNAMIC TABLE ... REFRESH`). When this option is explicitly set, specifying `target_lag` results in an error. dbt default differs from Snowflake's native default In Snowflake's native DDL, omitting `SCHEDULER` defaults to `ENABLE`, and `TARGET_LAG` is required. In dbt, the default value is `DISABLE`. If neither `scheduler` nor `target_lag` is specified, dbt creates the dynamic table with `scheduler: DISABLE` and manages refreshes directly. If you specify `target_lag` without explicitly setting `scheduler`, dbt sets `scheduler: ENABLE`. **Key points:** * Explicitly setting `scheduler: DISABLE` together with `target_lag` results in an error. If you omit `scheduler` and provide `target_lag`, dbt resolves the conflict by setting `scheduler: ENABLE` automatically. * When `scheduler: DISABLE`, a manual refresh does *not* automatically refresh upstream dynamic table dependencies. This creates an isolation boundary, allowing dbt to manage specific table refreshes without triggering the entire pipeline. In contrast, `ENABLE` cascades refreshes across the dependency graph. * If a dynamic table with `scheduler: DISABLE` depends on other dynamic tables, those upstream tables will not be refreshed when the downstream table is refreshed. dbt must manage the refresh order explicitly. For example, to let dbt manage refreshes (default behavior): ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='MY_WH', ) }} select * from {{ source('raw', 'events') }} ``` To enable Snowflake-managed scheduling with a target lag: ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='MY_WH', target_lag='5 minutes', ) }} select * from {{ source('raw', 'events') }} ``` Learn more about `scheduler` in [Snowflake's docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table#optional-parameters). (Applies to dbt v1.11 and later) ##### Refresh warehouse Starting `dbt-snowflake` v1.11, you can use the `refresh_warehouse` parameter in your model configuration to specify a separate warehouse for the dynamic table's self-refresh operations. This is separate from [`snowflake_warehouse`](#configuring-virtual-warehouses), which controls DDL execution. By setting `refresh_warehouse`, you can use a smaller warehouse for automatic refreshes while keeping a larger `snowflake_warehouse` for DDL operations. To configure the `refresh_warehouse` parameter in your model, refer to the following example: models/\<model\_name>.sql ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='LARGE_EXECUTION_WH', refresh_warehouse='SMALL_REFRESH_WH', target_lag='1 hour' ) }} select * from {{ source('raw', 'events') }} ``` **Key points:** * If `refresh_warehouse` is not set, `snowflake_warehouse` is used for both DDL execution and self-refresh operations. * You can change `refresh_warehouse` on an existing dynamic table without a full refresh. * To revert to the default behavior after setting a refresh warehouse, remove the `refresh_warehouse` parameter from your model configuration or explicitly set it to `None`. Learn more about the `WAREHOUSE` parameter in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-warehouses). (Applies to dbt v1.9 and later) ##### Refresh mode Snowflake allows three options for refresh mode: * **AUTO** — Enforces an incremental refresh of the dynamic table by default. If the `CREATE DYNAMIC TABLE` statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode. * **FULL** — Enforces a full refresh of the dynamic table, even if the dynamic table can be incrementally refreshed. * **INCREMENTAL** — Enforces an incremental refresh of the dynamic table. If the query that underlies the dynamic table can’t perform an incremental refresh, dynamic table creation fails and displays an error message. Learn more about `refresh_mode` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh). ##### Initialize Snowflake allows two options for initialize: * **ON\_CREATE** — Refreshes the dynamic table synchronously at creation. If this refresh fails, dynamic table creation fails and displays an error message. * **ON\_SCHEDULE** — Refreshes the dynamic table at the next scheduled refresh. Learn more about `initialize` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh). (Applies to dbt v1.11 and later) ##### Immutable where Snowflake allows you to mark certain rows of a dynamic table as immutable using the `IMMUTABLE WHERE` clause. This prevents Snowflake from applying updates or deletions to matching rows during refreshes, so historical data stays the same and refreshes run faster. From dbt v1.11, you can configure this using the `immutable_where` configuration. This config accepts a SQL condition expression and rows that match it are treated as immutable and won’t be updated or deleted during future refreshes. For example, to mark data older than 1 day as immutable since historical data typically doesn't change: ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='MY_WH', target_lag='1 hour', immutable_where='ts < CURRENT_TIMESTAMP() - INTERVAL \'1 DAY\'' ) }} select id, ts, value from {{ source('raw', 'events') }} ``` **Key points:** * The config supports Jinja rendering (for example, dbt variables and macros), as long as the rendered result is a valid Snowflake SQL condition. * To remove the immutable constraint from an existing dynamic table, set `immutable_where` to `None`. * You can alter changes to `immutable_where` without a full refresh. Learn more about `IMMUTABLE WHERE` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-immutability-constraints). ##### Copy grants (dynamic tables) Starting `dbt-snowflake` v1.11, you can use `copy_grants` to preserve existing object-level privileges when dbt generates a `CREATE OR REPLACE DYNAMIC TABLE` statement. When disabled, all previously granted permissions are dropped when the table is recreated, and downstream users or roles lose access until grants are manually re-applied. When you set `copy_grants: true` on a dynamic table, dbt adds the `COPY GRANTS` clause to the `CREATE OR REPLACE DYNAMIC TABLE` statement. This preserves existing object-level privileges on the table during `--full-refresh` runs, so you don't need to re-grant access after the table is recreated. To configure the `copy_grants` parameter, refer to the following example: ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='MY_WH', target_lag='1 hour', copy_grants=true ) }} select * from {{ source('raw', 'events') }} ``` Learn more about `COPY GRANTS` in [Snowflake's docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table). (Applies to dbt v1.12 and later) ##### Transient (dynamic tables) You can create dynamic tables as transient to reduce storage costs. Transient dynamic tables do not use Snowflake's [Fail-safe](https://docs.snowflake.com/en/user-guide/data-failsafe) period, so they consume less storage than permanent dynamic tables. To create a dynamic table as transient, set `transient: true` in the model configuration. If you want all dynamic tables to be transient by default (without setting `transient: true` on each one), enable the [`snowflake_default_transient_dynamic_tables`](https://docs.getdbt.com/reference/global-configs/snowflake-changes.md#the-snowflake_default_transient_dynamic_tables-flag) flag in your `dbt_project.yml`. This flag defaults to `false`, meaning dynamic tables are created as permanent by default. **Key points:** * Setting `transient: true` creates the dynamic table with the `TRANSIENT` keyword in the `CREATE DYNAMIC TABLE` statement. * Snowflake does not support changing the transient property on an existing dynamic table. Changing `transient` from `true` to `false` or vice versa triggers a full table recreation. * To make all new dynamic tables transient by default when `transient` is not specified, enable the `snowflake_default_transient_dynamic_tables` flag in your `dbt_project.yml`. For example: ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='MY_WH', target_lag='1 hour', transient=true ) }} select * from {{ source('raw', 'events') }} ``` ##### Initialization warehouse Snowflake supports an `INITIALIZATION_WAREHOUSE` parameter that specifies which virtual warehouse to use when initializing or reinitializing a dynamic table. Starting `dbt-snowflake` v1.12, you can use the `snowflake_initialization_warehouse` parameter to configure this. This is separate from the `snowflake_warehouse` parameter used for regular incremental refreshes. By setting `snowflake_initialization_warehouse`, you can use a larger warehouse for the initial build and reinitialization, while keeping `snowflake_warehouse` smaller for regular refreshes. To configure the `snowflake_initialization_warehouse` parameter, refer to the following example: ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='COMPUTE_WH', snowflake_initialization_warehouse='LARGE_WH', target_lag='1 minute' ) }} select * from {{ source('raw', 'events') }} ``` **Key points:** * If `snowflake_initialization_warehouse` is not set, Snowflake uses `snowflake_warehouse` for both initialization and regular refreshes. * You can change `snowflake_initialization_warehouse` on an existing dynamic table without a full refresh. * To revert to the default behavior after setting an initialization warehouse, either remove the `snowflake_initialization_warehouse` parameter from your model configuration or explicitly set it to `None`. Learn more about `INITIALIZATION_WAREHOUSE` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-warehouses). ##### Limitations As with materialized views on most data platforms, there are limitations associated with dynamic tables. Some worth noting include: * Dynamic table SQL has a [limited feature set](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#query-constructs-not-currently-supported-in-dynamic-tables). * Dynamic table SQL cannot be updated; the dynamic table must go through a `--full-refresh` (DROP/CREATE). * Dynamic tables cannot be downstream from: materialized views, external tables, streams. * Dynamic tables cannot reference a view that is downstream from another dynamic table. Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions). For dbt limitations, [Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) are not supported. ##### Troubleshooting dynamic tables If your dynamic table model fails to rerun with the following error message after the initial execution: ```sql SnowflakeDynamicTableConfig.__init__() missing 6 required positional arguments: 'name', 'schema_name', 'database_name', 'query', 'target_lag', and 'snowflake_warehouse' ``` Ensure that `QUOTED_IDENTIFIERS_IGNORE_CASE` on your account is set to `FALSE`. #### Interactive tables [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Starting in `dbt-snowflake` v1.12, the Snowflake adapter supports [interactive tables](https://docs.snowflake.com/en/user-guide/interactive), which are optimized for low-latency queries. This materialization is specific to Snowflake, which means that any model configuration that dbt usually provides by default (for example, for a `view`) may not apply to interactive tables. dbt's support for interactive tables is in beta Interactive tables are generally available in Snowflake, but dbt's support for the `interactive_table` materialization is in beta in both the v1 and v2 engines. Behavior and configuration options may change. * **dbt platform** * **dbt v2** * **dbt v1** (`dbt-snowflake` v1.12+) Setting [`target_lag`](#target-lag-interactive-tables) makes the table a *dynamic* interactive table that Snowflake refreshes automatically. Without it, the table is *static* and only rebuilds when you run dbt. Several configurations below behave differently depending on which form you're using. Like dynamic tables, interactive tables have access to the [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) setting. Interactive tables are supported with the following configuration parameters: | Parameter | Type | Required | Default | Change Monitoring Support | | ---------------------------------------------------------------------------------------------------------- | ---------------------- | -------- | ------- | ------------------------- | | [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) | `<string>` | no | `apply` | n/a | | [`cluster_by`](#cluster-by-interactive-tables) | `<string>` or `<list>` | yes | | full refresh | | [`target_lag`](#target-lag-interactive-tables) | `<string>` | no | `None` | alter | | [`snowflake_warehouse`](#configuring-virtual-warehouses) | `<string>` | no | `None` | alter | | [`refresh_warehouse`](#refresh-warehouse-interactive-tables) | `<string>` | no | `None` | alter | | [`snowflake_initialization_warehouse`](#initialization-warehouse-interactive-tables) | `<string>` | no | `None` | alter | ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +materialized: interactive_table +on_configuration_change: apply | continue | fail +cluster_by: <column-name> | [<column-name>, <column-name>, ...] +target_lag: <time-delta> +snowflake_warehouse: <warehouse-name> +refresh_warehouse: <warehouse-name> +snowflake_initialization_warehouse: <warehouse-name> ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: [<model-name>] config: materialized: interactive_table on_configuration_change: apply | continue | fail cluster_by: <column-name> | [<column-name>, <column-name>, ...] target_lag: <time-delta> snowflake_warehouse: <warehouse-name> refresh_warehouse: <warehouse-name> snowflake_initialization_warehouse: <warehouse-name> ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized="interactive_table", on_configuration_change="apply" | "continue" | "fail", cluster_by="<column-name>" | ["<column-name>", "<column-name>", ...], target_lag="<integer> seconds | minutes | hours | days", snowflake_warehouse="<warehouse-name>", refresh_warehouse="<warehouse-name>", snowflake_initialization_warehouse="<warehouse-name>", ) }} ``` Learn more about these parameters in Snowflake's [docs](https://docs.snowflake.com/en/sql-reference/sql/create-interactive-table). The warehouse parameters above refer to ordinary virtual warehouses that run DDL and refreshes. They do not attach the table to an [interactive warehouse](https://docs.snowflake.com/en/user-guide/interactive) — refer to [Limitations of interactive tables](#limitations-of-interactive-tables). ##### Cluster by (interactive tables) Unlike dynamic tables, where [`cluster_by`](#dynamic-table-clustering) is optional, interactive tables *require* it: ```sql {{ config( materialized='interactive_table', cluster_by=['order_id'], ) }} select * from {{ ref('stg_orders') }} ``` **Key points:** * dbt validates `cluster_by` when it parses your project, so an omitted value, an empty list, or a blank entry within a list (such as `["id", " "]`) fails at parse time instead of at execution. * Changing `cluster_by` on an existing interactive table triggers a full refresh. For guidance on choosing clustering columns, refer to [CREATE INTERACTIVE TABLE](https://docs.snowflake.com/en/sql-reference/sql/create-interactive-table) in Snowflake's docs. ##### Target lag (interactive tables) Set `target_lag` to make the table dynamic (auto-refreshing). Snowflake also requires a warehouse for those refreshes, so dbt raises an error at parse time if you set `target_lag` without either [`refresh_warehouse`](#refresh-warehouse-interactive-tables) or [`snowflake_warehouse`](#configuring-virtual-warehouses). ```sql {{ config( materialized='interactive_table', cluster_by=['order_id'], target_lag='30 minutes', snowflake_warehouse='MY_WH', ) }} select * from {{ ref('stg_orders') }} ``` **Key points:** * Changing `target_lag` from one value to another updates the table in place with an `alter` statement. * Adding `target_lag` to a static table, or removing it from a dynamic one, replaces the table. Snowflake can't switch between the static and dynamic forms with an `alter`, so dbt runs `create or replace` instead. You don't need to pass --full-refresh — dbt does this for you. Learn more about `TARGET_LAG` in [CREATE INTERACTIVE TABLE](https://docs.snowflake.com/en/sql-reference/sql/create-interactive-table) in Snowflake's docs. ##### Refresh warehouse (interactive tables) Use `refresh_warehouse` to run a dynamic interactive table's automatic refreshes on a different warehouse than the one dbt uses for DDL execution ([`snowflake_warehouse`](#configuring-virtual-warehouses)). This lets you keep a smaller warehouse for refreshes and a larger one for DDL. ```sql {{ config( materialized='interactive_table', cluster_by=['order_id'], target_lag='1 hour', snowflake_warehouse='LARGE_EXECUTION_WH', refresh_warehouse='SMALL_REFRESH_WH', ) }} select * from {{ ref('stg_orders') }} ``` **Key points:** * On a dynamic interactive table, if `refresh_warehouse` is not set, `snowflake_warehouse` is used for both DDL execution and self-refresh operations. * On a static interactive table, `refresh_warehouse` is ignored because the table never refreshes itself. `snowflake_warehouse` still controls which warehouse dbt uses to build the table. * You can change `refresh_warehouse` on an existing interactive table without a full refresh. * On a dynamic interactive table, you can change `refresh_warehouse` without replacing the table. dbt updates it in place with an `alter` statement. * To go back to using `snowflake_warehouse` for refreshes, remove `refresh_warehouse` from your model configuration. ##### Initialization warehouse (interactive tables) Use `snowflake_initialization_warehouse` to specify which virtual warehouse Snowflake uses when initializing or reinitializing a dynamic interactive table. This lets you use a larger warehouse for the initial build while keeping `snowflake_warehouse` smaller for regular refreshes. ```sql {{ config( materialized='interactive_table', cluster_by=['order_id'], target_lag='1 hour', snowflake_warehouse='COMPUTE_WH', snowflake_initialization_warehouse='LARGE_WH', ) }} select * from {{ ref('stg_orders') }} ``` **Key points:** * This parameter only applies when the table refreshes itself. If you set it on a static interactive table (one without `target_lag`), dbt ignores it and warns you when you run the model. * You can change `snowflake_initialization_warehouse` on an existing interactive table without a full refresh. * To revert to the default behavior, remove the parameter from your model configuration or explicitly set it to `None`. ##### Change monitoring for interactive tables Interactive tables support [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md). Snowflake determines which configuration changes dbt can apply in place, and which force a full rebuild: | Changed configuration | dbt behavior | | ------------------------------------------------------------------------- | ------------------------------------------- | | `target_lag` value to value | Alters the table in place | | `refresh_warehouse` | Alters the table in place | | `snowflake_initialization_warehouse` | Alters the table in place | | `cluster_by` | Rebuilds the table with `CREATE OR REPLACE` | | Adding or removing `target_lag` (static to dynamic, or dynamic to static) | Rebuilds the table with `CREATE OR REPLACE` | ##### Unsupported configurations for interactive tables The following configurations are not supported on interactive tables. dbt rejects the first two when it parses your project, rather than letting Snowflake fail the run: * `table_format: iceberg`: Interactive tables have no Iceberg variant. * `transient: true`: Snowflake does not accept a transient interactive table (error 001003). If you set `transient: true` at the project level in dbt\_project.yml, your interactive table models inherit it and fail. Set `transient: false` on those models to override it. * `change_tracking`: Accepted but inert. `CREATE INTERACTIVE TABLE` does not take a change tracking option. ##### Limitations of interactive tables Limitations worth noting when you build interactive tables with dbt: * dbt does not attach or detach interactive tables to or from [interactive warehouses](https://docs.snowflake.com/en/user-guide/interactive) at this time. Manage that association in Snowflake. The two are independent objects; Snowflake can attach an interactive warehouse to [other relation types](https://docs.snowflake.com/en/user-guide/interactive#all-table-format-support-preview) as well. * Dropping a column from an interactive table is not supported. This is reachable if you have an `incremental` model with [`on_schema_change: sync_all_columns`](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change) running against a relation that was previously an interactive table. * Converting an interactive table to an `incremental` model requires a `--full-refresh`. * Interactive tables cannot be cloned or created in a personal database. * [Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) are not supported. Find more information about interactive table and interactive warehouse limitations, including per-warehouse table limits, in the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/interactive). ##### Troubleshooting interactive tables A dynamic interactive table downstream of a `table` model can serve stale data When a dynamic interactive table reads from a dbt-managed table model, dbt's `create or replace` statement on that upstream model destroys Snowflake's change tracking history. The interactive table can then no longer refresh itself, but your `dbt run` still reports success, so a clean run can leave the table serving stale rows. To recover, run the interactive table with `--full-refresh`. To prevent it, add a post-hook to the upstream model that re-enables change tracking: ```sql {{ config( materialized='table', post_hook="alter table {{ this }} set change_tracking = true", ) }} ``` Note that a `change_tracking` model config is *not* a substitute here, since that config does not reach the SQL for plain tables. Plain dynamic tables have the same exposure. If your interactive table model fails to rerun after the initial execution with an error about missing positional arguments, ensure that `QUOTED_IDENTIFIERS_IGNORE_CASE` on your account is set to `FALSE`. #### Semantic Views [Snowflake Semantic Views](https://docs.snowflake.com/en/user-guide/views-semantic/overview) provide a native schema-level object for centralizing metric definitions and reducing fragmented metric logic across BI and analytics tools. Use the [`dbt_semantic_view` package](https://hub.getdbt.com/Snowflake-Labs/dbt_semantic_view/latest/) to define and manage Snowflake Semantic Views in your dbt project. This lets you keep Semantic View definitions in version control and apply your existing testing and CI/CD workflows to your Semantic Layer. ##### Install the package Prerequisite * This package requires `dbt` version `>=1.0.0, <2.0.0`. For the latest compatibility details, refer to the [`dbt_semantic_view` package](https://hub.getdbt.com/Snowflake-Labs/dbt_semantic_view/latest/). * Your Snowflake account supports Semantic Views. * Your role has permission to create Semantic Views. * You can write to a database and schema where you have create privileges. Add `dbt_semantic_view` to your `packages.yml` file: ```yaml packages: - package: Snowflake-Labs/dbt_semantic_view version: 1.0.3 ``` Run `dbt deps` to install package dependencies: ```shell dbt deps ``` Verify the package was installed by confirming `dbt_semantic_view` is present in your `dbt_packages/` directory. ##### Highlighted features The `dbt_semantic_view` package includes the following features for defining and managing Snowflake Semantic Views in dbt projects. ###### Materialize models as Snowflake Semantic Views Use the `semantic_view` materialization to define Snowflake Semantic Views in dbt, including tables, relationships, facts, dimensions, and metrics. Semantic view models use Snowflake’s semantic view syntax (for example, `TABLES`, `DIMENSIONS`, and `METRICS`) rather than a standard `SELECT` query. The example below is adapted from [Getting Started with Snowflake Semantic View](https://quickstarts.snowflake.com/guide/snowflake-semantic-view/index.html?index=..%2F..index#3). ```sql {{ config(materialized='semantic_view') }} tables ( CUSTOMER as {{ SOURCE('<SOURCE_NAME>', 'CUSTOMER') }} primary key (C_CUSTOMER_SK), DATE as {{ SOURCE('<SOURCE_NAME>', 'DATE_DIM') }} primary key (D_DATE_SK), DEMO as {{ SOURCE('<SOURCE_NAME>', 'CUSTOMER_DEMOGRAPHICS') }} primary key (CD_DEMO_SK), ITEM as {{ SOURCE('<SOURCE_NAME>', 'ITEM') }} primary key (I_ITEM_SK), STORE as {{ SOURCE('<SOURCE_NAME>', 'STORE') }} primary key (S_STORE_SK), STORESALES as {{ SOURCE('<SOURCE_NAME>', 'STORESALES') }} primary key (SS_SOLD_DATE_SK,SS_CDEMO_SK,SS_ITEM_SK,SS_STORE_SK,SS_CUSTOMER_SK) ) relationships ( SALESTOCUSTOMER as STORESALES(SS_CUSTOMER_SK) references CUSTOMER(C_CUSTOMER_SK), SALESTODATE as STORESALES(SS_SOLD_DATE_SK) references DATE(D_DATE_SK), SALESTODEMO as STORESALES(SS_CDEMO_SK) references DEMO(CD_DEMO_SK), SALESTOITEM as STORESALES(SS_ITEM_SK) references ITEM(I_ITEM_SK), SALETOSTORE as STORESALES(SS_STORE_SK) references STORE(S_STORE_SK) ) facts ( ITEM.COST as i_wholesale_cost, ITEM.PRICE as i_current_price, STORE.TAX_RATE as S_TAX_PERCENTAGE, STORESALES.SALES_QUANTITY as SS_QUANTITY ) dimensions ( CUSTOMER.BIRTHYEAR as C_BIRTH_YEAR, CUSTOMER.COUNTRY as C_BIRTH_COUNTRY, CUSTOMER.C_CUSTOMER_SK as c_customer_sk, DATE.DATE as D_DATE, DATE.D_DATE_SK as d_date_sk, DATE.MONTH as D_MOY, DATE.WEEK as D_WEEK_SEQ, DATE.YEAR as D_YEAR, DEMO.CD_DEMO_SK as cd_demo_sk, DEMO.CREDIT_RATING as CD_CREDIT_RATING, DEMO.MARITAL_STATUS as CD_MARITAL_STATUS, ITEM.BRAND as I_BRAND, ITEM.CATEGORY as I_CATEGORY, ITEM.CLASS as I_CLASS, ITEM.I_ITEM_SK as i_item_sk, STORE.MARKET as S_MARKET_ID, STORE.SQUAREFOOTAGE as S_FLOOR_SPACE, STORE.STATE as S_STATE, STORE.STORECOUNTRY as S_COUNTRY, STORE.S_STORE_SK as s_store_sk, STORESALES.SS_CDEMO_SK as ss_cdemo_sk, STORESALES.SS_CUSTOMER_SK as ss_customer_sk, STORESALES.SS_ITEM_SK as ss_item_sk, STORESALES.SS_SOLD_DATE_SK as ss_sold_date_sk, STORESALES.SS_STORE_SK as ss_store_sk ) metrics ( STORESALES.TOTALCOST as SUM(item.cost), STORESALES.TOTALSALESPRICE as SUM(SS_SALES_PRICE), STORESALES.TOTALSALESQUANTITY as SUM(SS_QUANTITY) WITH SYNONYMS = ('total sales quantity', 'total sales amount') ) ``` When you run dbt, this model compiles to a Snowflake `CREATE SEMANTIC VIEW` statement. ###### Reference Semantic Views in other dbt models Use `ref()` for Semantic Views defined in your dbt project, and use `source()` for existing external Semantic Views. ```sql {{ config(materialized='view') }} select * from semantic_view( {{ ref('<semantic_view_model_name>') }} METRICS ... DIMENSIONS ... WHERE ... ) ``` ```sql {{ config(materialized='table') }} select * from semantic_view( {{ source('<source_name>', '<semantic_view>') }} METRICS ... DIMENSIONS ... WHERE ... ) ``` #### Temporary tables To save compile time and avoid the database write step initiated by a temporary table, incremental table merges for Snowflake prefer to utilize a `view` rather than a `temporary table` . Sometimes a temporary table achieves results faster or more safely. You can opt in to temporary or transient tables for incremental builds by using the `tmp_relation_type` configuration This is defined as part of the model configuration. To guarantee accuracy, an incremental model using the `delete+insert` strategy with a `unique_key` defined requires a temporary table; trying to change this to a view will result in an error. `tmp_relation_type` accepts these values: * `view` (default): Skips intermediate step of creating a temporary physical table for the tmp relation; fastest but not suitable for all strategies. * `table`: A session-scoped temporary table; not visible in the Snowflake catalog and isolated per session. * `transient`: A transient table; persists in the catalog, enabling [Snowflake native lineage tracking](https://docs.snowflake.com/en/user-guide/ui-snowsight-lineage), while avoiding the 7-day fail-safe storage costs of permanent tables. **Note:** This value is distinct from the separate model-level `transient` config described later, which controls the final model relation. Defined in the project YAML: dbt\_project.yml ```yaml name: my_project ... models: <resource-path>: +tmp_relation_type: table | view | transient ## If not defined, view is the default. ``` In the configuration format for the model SQL file: dbt\_model.sql ```sql {{ config( tmp_relation_type="table | view | transient", -- If not defined, view is the default. ) }} ``` Concurrent run conflicts with `transient` When `tmp_relation_type` is set to `transient`, the tmp relation becomes a real table that persists in the target schema under a deterministic name. If multiple runs of the same incremental model execute concurrently in the same schema, they can overwrite each other's tmp relation, causing data duplication or incorrect results. For example, this might happen when developers share a target schema or when CI and production runs overlap. This risk depends on how you configure schemas and databases for your dbt models. To prevent conflicts, use `snowflake__resolve_incremental_tmp_relation` to route tmp relations to a schema that is unique per run or environment. For more information, refer to [Avoiding tmp relation conflicts](#avoiding-tmp-relation-conflicts). ##### Avoiding tmp relation conflicts To prevent name collisions across concurrent runs, override the `snowflake__resolve_incremental_tmp_relation` dispatch macro to redirect the tmp relation to a dedicated schema: macros/snowflake\_incremental.sql ```sql {% macro snowflake__resolve_incremental_tmp_relation(tmp_relation) %} {{ return(tmp_relation.incorporate(schema='scratch')) }} {% endmacro %} ``` This macro receives the default tmp relation object and returns a modified version. Common overrides include appending a developer username, a CI job ID, or a target name to the schema to ensure isolation across concurrent runs. To append a target name to the schema: macros/snowflake\_incremental.sql ```sql {% macro snowflake__resolve_incremental_tmp_relation(tmp_relation) %} {%- set scratch_schema = target.schema ~ '_scratch_' ~ env_var('DBT_JOB_ID', target.name) -%} {{ return(tmp_relation.incorporate(schema=scratch_schema)) }} {% endmacro %} ``` #### Transient tables Snowflake supports the creation of [transient tables](https://docs.snowflake.net/manuals/user-guide/tables-temp-transient.html). Snowflake does not preserve a history for these tables, which can result in a measurable reduction of your Snowflake storage costs. Transient tables participate in time travel to a limited degree with a retention period of 1 day by default with no fail-safe period. Weigh these tradeoffs when deciding whether or not to configure your dbt models as `transient`. **By default, all Snowflake tables created by dbt are `transient`.** ##### Configuring transient tables in dbt\_project.yml A whole folder (or package) can be configured to be transient (or not) by adding a line to the `dbt_project.yml` file. This config works just like all of the [model configs](https://docs.getdbt.com/reference/model-configs.md) defined in `dbt_project.yml`. dbt\_project.yml ```yaml name: my_project ... models: +transient: false my_project: ... ``` ##### Configuring transience for a specific model A specific model can be configured to be transient by setting the `transient` model config to `true`. my\_table.sql ```sql {{ config(materialized='table', transient=true) }} select * from ... ``` #### Query tags [Query tags](https://docs.snowflake.com/en/sql-reference/parameters.html#query-tag) are a Snowflake parameter that can be quite useful later on when searching in the [QUERY\_HISTORY view](https://docs.snowflake.com/en/sql-reference/account-usage/query_history.html). dbt supports setting a default query tag for the duration of its Snowflake connections in [your profile](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md). You can set more precise values (and override the default) for subsets of models by setting a `query_tag` model config or by overriding the default `set_query_tag` macro: dbt\_project.yml ```yaml models: <resource-path>: +query_tag: dbt_special ``` models/\<modelname>.sql ```sql {{ config( query_tag = 'dbt_special' ) }} select ... ``` In this example, you can set up a query tag to be applied to every query with the model's name. ```sql {% macro set_query_tag() -%} {% set new_query_tag = model.name %} {% if new_query_tag %} {% set original_query_tag = get_current_query_tag() %} {{ log("Setting query_tag to '" ~ new_query_tag ~ "'. Will reset to '" ~ original_query_tag ~ "' after materialization.") }} {% do run_query("alter session set query_tag = '{}'".format(new_query_tag)) %} {{ return(original_query_tag)}} {% endif %} {{ return(none)}} {% endmacro %} ``` **Note:** query tags are set at the *session* level. At the start of each model materialization, if the model has a custom `query_tag` configured, dbt will run `alter session set query_tag` to set the new value. At the end of the materialization, dbt will run another `alter` statement to reset the tag to its default value. As such, build failures midway through a materialization may result in subsequent queries running with an incorrect tag. #### Merge behavior (incremental models) The [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-strategy.md) controls how dbt builds incremental models. By default, dbt will use a [merge statement](https://docs.snowflake.net/manuals/sql-reference/sql/merge.html) on Snowflake to refresh incremental tables. Snowflake supports the following incremental strategies: * [`merge`](https://docs.getdbt.com/docs/build/incremental-strategy.md#merge) (default) * [`append`](https://docs.getdbt.com/docs/build/incremental-strategy.md#append) * [`delete+insert`](https://docs.getdbt.com/docs/build/incremental-strategy.md#deleteinsert) * [`insert_overwrite`](https://docs.getdbt.com/docs/build/incremental-strategy.md#insert_overwrite) * Note: This is not a standard dbt incremental strategy. `insert_overwrite` behaves like `truncate` + re-`insert` commands on Snowflake. It doesn't support partition-based overwrites, which means it'll overwrite the entire table intentionally. It's implemented as an incremental strategy because it aligns with dbt's workflow of not dropping existing tables. You can use [`overwrite_columns`](#overwrite_columns) to control which columns are included in the `INSERT OVERWRITE` statement. * [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md) Snowflake's `merge` statement fails with a "nondeterministic merge" error if the `unique_key` specified in your model config is not actually unique. If you encounter this error, you can instruct dbt to use a two-step incremental approach by setting the `incremental_strategy` config for your model to `delete+insert`. ##### `overwrite_columns` When using `incremental_strategy='insert_overwrite'` on Snowflake, you can set `overwrite_columns` to control how dbt generates the `INSERT OVERWRITE` statement for your incremental model. For example: models/my\_model.sql ```sql {{ config( materialized='incremental', incremental_strategy='insert_overwrite', overwrite_columns=['id', 'value', 'event_date'] ) }} select id, value, event_date from {{ ref('my_source') }} ``` * If you set `overwrite_columns`, dbt generates SQL that explicitly lists the columns in both the `INSERT` target and the `SELECT` projection: ```sql insert overwrite into my_schema.my_table (id, value, event_date) select id, value, event_date from staging_table ``` * If you don't set `overwrite_columns`, dbt currently defaults to `SELECT *`: ```sql insert overwrite into my_schema.my_table select * from staging_table ``` #### Configuring table clustering dbt supports [table clustering](https://docs.snowflake.net/manuals/user-guide/tables-clustering-keys.html) on Snowflake. To control clustering for a table or incremental model, use the `cluster_by` config. When this configuration is applied, dbt will do two things: 1. It will implicitly order the table results by the specified `cluster_by` fields. 2. It will add the specified clustering keys to the target table. By using the specified `cluster_by` fields to order the table, dbt minimizes the amount of work required by Snowflake's automatic clustering functionality. If an incremental model is configured to use table clustering, then dbt will also order the staged dataset before merging it into the destination table. As such, the dbt-managed table should always be in a mostly clustered state. ##### Using cluster\_by The `cluster_by` config accepts either a string, or a list of strings to use as clustering keys. The following example will create a sessions table that is clustered by the `session_start` column. models/events/sessions.sql ```sql {{ config( materialized='table', cluster_by=['session_start'] ) }} select session_id, min(event_time) as session_start, max(event_time) as session_end, count(*) as count_pageviews from {{ source('snowplow', 'event') }} group by 1 ``` The code above will be compiled to SQL that looks (approximately) like this: ```sql create or replace table my_database.my_schema.my_table as ( select * from ( select session_id, min(event_time) as session_start, max(event_time) as session_end, count(*) as count_pageviews from {{ source('snowplow', 'event') }} group by 1 ) -- this order by is added by dbt in order to create the -- table in an already-clustered manner. order by session_start ); alter table my_database.my_schema.my_table cluster by (session_start); ``` ##### Dynamic table clustering Starting in dbt v1.11, dynamic tables support the `cluster_by` configuration. When set, dbt includes the clustering specification in the `CREATE DYNAMIC TABLE` statement. For example: ```sql {{ config( materialized='dynamic_table', snowflake_warehouse='COMPUTE_WH', target_lag='1 minute', cluster_by=['session_start', 'user_id'] ) }} select session_id, user_id, min(event_time) as session_start, max(event_time) as session_end, count(*) as count_pageviews from {{ source('snowplow', 'event') }} group by 1, 2 ``` This config generates the following SQL when compiled: ```sql create or replace dynamic table my_database.my_schema.my_table target_lag = '1 minute' warehouse = COMPUTE_WH cluster by (session_start, user_id) as ( select session_id, user_id, min(event_time) as session_start, max(event_time) as session_end, count(*) as count_pageviews from source_table group by 1, 2 ); ``` You can specify clustering for dynamic tables when you create them using `CLUSTER BY` in the `CREATE DYNAMIC TABLE` statement. You don’t need to run a separate `ALTER TABLE` statement. ##### Automatic clustering Automatic clustering is [enabled by default in Snowflake today](https://docs.snowflake.com/en/user-guide/tables-auto-reclustering.html), no action is needed to make use of it. Though there is an `automatic_clustering` config, it has no effect except for accounts with (deprecated) manual clustering enabled. If [manual clustering is still enabled for your account](https://docs.snowflake.com/en/user-guide/tables-clustering-manual.html), you can use the `automatic_clustering` config to control whether or not automatic clustering is enabled for dbt models. When `automatic_clustering` is set to `true`, dbt will run an `alter table <table name> resume recluster` query after building the target table. The `automatic_clustering` config can be specified in the `dbt_project.yml` file, or in a model `config()` block. dbt\_project.yml ```yaml models: +automatic_clustering: true ``` #### Python model configuration The Snowflake adapter supports Python models. Snowflake uses its own framework, Snowpark, which has many similarities to PySpark. **Additional setup:** You will need to [acknowledge and accept Snowflake Third Party Terms](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages.html#getting-started) to use Anaconda packages. **Installing packages:** Snowpark supports several popular packages via Anaconda. Refer to the [complete list](https://repo.anaconda.com/pkgs/snowflake/) for more details. Packages are installed when your model is run. Different models can have different package dependencies. If you use third-party packages, Snowflake recommends using a dedicated virtual warehouse for best performance rather than one with many concurrent users. **Python version:** To specify a different Python version, use the following configuration: ```python def model(dbt, session): dbt.config( materialized = "table", python_version="3.11" ) ``` You can use the `python_version` config to run a Snowpark model with [Python versions](https://docs.snowflake.com/en/developer-guide/snowpark/python/setup) 3.9, 3.10, or 3.11. **External access integrations and secrets**: To query external APIs within dbt Python models, use Snowflake’s [external access](https://docs.snowflake.com/en/developer-guide/external-network-access/external-network-access-overview) together with [secrets](https://docs.snowflake.com/en/developer-guide/external-network-access/secret-api-reference). Here are some additional configurations you can use: ```python import pandas import snowflake.snowpark as snowpark def model(dbt, session: snowpark.Session): dbt.config( materialized="table", secrets={"secret_variable_name": "test_secret"}, external_access_integrations=["test_external_access_integration"], ) import _snowflake return session.create_dataframe( pandas.DataFrame( [{"secret_value": _snowflake.get_generic_secret_string('secret_variable_name')}] ) ) ``` **Docs:** ["Developer Guide: Snowpark Python"](https://docs.snowflake.com/en/developer-guide/snowpark/python/index.html) ##### Additional parameters Snowflake Python models accept these parameters in `dbt.config()`. Refer to [Configuring Python models](https://docs.getdbt.com/docs/build/python-models.md#configuring-python-models) for more info. | Parameter | Type | Default | Description | | ------------------------------ | -------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `python_version` | `<string>` | `3.10` | Python version the Snowpark model runs on. | | `packages` | `<list>` | `[]` | Anaconda packages to install when the model runs. dbt always adds `snowflake-snowpark-python`. | | `imports` | `<list>` | `[]` | Staged files, such as zip files, to make available to your model. Refer to [Third-party Snowflake packages](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#third-party-snowflake-packages). | | `external_access_integrations` | `<list>` | `[]` | Snowflake external access integrations that let your model reach external APIs. | | `secrets` | `<dictionary>` | `{}` | Maps a variable name in your code to a Snowflake secret. Use it with `external_access_integrations`. | | `use_anonymous_sproc` | `<boolean>` | `True` | When `True`, dbt runs your model as an anonymous stored procedure. When `False`, dbt creates a named procedure (`<model_name>__dbt_sp`), calls it, then drops it. | ##### Third-party Snowflake packages To use a third-party Snowflake package that isn't available in Snowflake Anaconda, upload your package by following [this example](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages#importing-packages-through-a-snowflake-stage), and then configure the `imports` setting in the dbt Python model to reference to the zip file in your Snowflake staging. Here’s a complete example configuration using a zip file, including using `imports` in a Python model: ```python def model(dbt, session): # Configure the model dbt.config( materialized="table", imports=["@mystage/mycustompackage.zip"], # Specify the external package location ) # Example data transformation using the imported package # (Assuming `some_external_package` has a function we can call) data = { "name": ["Alice", "Bob", "Charlie"], "score": [85, 90, 88] } df = pd.DataFrame(data) # Process data with the external package df["adjusted_score"] = df["score"].apply(lambda x: some_external_package.adjust_score(x)) # Return the DataFrame as the model output return df ``` For more information on using this configuration, refer to [Snowflake's documentation](https://community.snowflake.com/s/article/how-to-use-other-python-packages-in-snowpark) on uploading and using other python packages in Snowpark not published on Snowflake's Anaconda channel. #### Configuring virtual warehouses The default warehouse that dbt uses can be configured in your [Profile](https://docs.getdbt.com/docs/local/profiles.yml.md) for Snowflake connections. To override the warehouse that is used for specific models (or groups of models), use the `snowflake_warehouse` model configuration. This configuration can be used to specify a larger warehouse for certain models in order to control Snowflake costs and project build times. [Tests](https://docs.getdbt.com/docs/build/data-tests.md) also supports the `snowflake_warehouse` configuration. This can be useful when you want to you run tests on a different Snowflake virtual warehouse than the one used to build models, for example, using a smaller warehouse for lightweight data tests while models run on a larger warehouse. ##### Project file The following example changes the warehouse for a group of models with a config argument in the YAML. dbt\_project.yml ```yaml name: my_project version: 1.0.0 ... models: +snowflake_warehouse: "EXTRA_SMALL" # default Snowflake virtual warehouse for all models in the project. my_project: clickstream: +snowflake_warehouse: "EXTRA_LARGE" # override the default Snowflake virtual warehouse for all models under the `clickstream` directory. snapshots: +snowflake_warehouse: "EXTRA_LARGE" # all Snapshot models are configured to use the `EXTRA_LARGE` warehouse. data_tests: +snowflake_warehouse: "EXTRA_SMALL" # all data tests are configured to use the `EXTRA_SMALL` warehouse. ``` ##### Property file The following example overrides the Snowflake warehouse for a single model and a specific test using a config argument in the property file. models/my\_model.yml ```yaml models: - name: my_model config: snowflake_warehouse: "EXTRA_LARGE" # override the Snowflake virtual warehouse just for this model columns: - name: id data_tests: - unique: config: snowflake_warehouse: "EXTRA_SMALL" # use a smaller warehouse for this test ``` ##### SQL file config The following example changes the warehouse for a single model with a config() block in the SQL model. models/events/sessions.sql ```sql # override the Snowflake virtual warehouse for just this model {{ config( materialized='table', snowflake_warehouse='EXTRA_LARGE' ) }} with aggregated_page_events as ( select session_id, min(event_time) as session_start, max(event_time) as session_end, count(*) as count_page_views from {{ source('snowplow', 'event') }} group by 1 ), index_sessions as ( select *, row_number() over ( partition by session_id order by session_start ) as page_view_in_session_index from aggregated_page_events ) select * from index_sessions ``` #### Copying grants When the `copy_grants` config is set to `true`, dbt will add the `copy grants` DDL qualifier when rebuilding tables, views, and [dynamic tables](#copy-grants-dynamic-tables) (`dbt-snowflake` v1.11 and later). The default value is `false`. dbt\_project.yml ```yaml models: +copy_grants: true ``` (Applies to dbt v1.10 and later) #### Setting row access policies Configure [row access policies](https://docs.snowflake.com/en/user-guide/security-row-intro) on tables, views, and dynamic tables by using the `row_access_policy` config for models. The policy must already exist in Snowflake before you apply it to the model. models/\<modelname>.sql ```sql {{ config( row_access_policy = 'my_database.my_schema.my_row_access_policy_name on (id)' ) }} select ... ``` #### Configuring table tags To add tags to tables, views, and dynamic tables, use the `table_tag` config. Note, the tag must already exist in Snowflake before you apply it. models/\<modelname>.sql ```sql {{ config( table_tag = "my_tag_name = 'my_tag_value'" ) }} select ... ``` #### Secure views To create a Snowflake [secure view](https://docs.snowflake.net/manuals/user-guide/views-secure.html), use the `secure` config for view models. Secure views can be used to limit access to sensitive data. Note: secure views may incur a performance penalty, so you should only use them if you need them. The following example configures the models in the `sensitive/` folder to be configured as secure views. dbt\_project.yml ```yaml name: my_project version: 1.0.0 models: my_project: sensitive: +materialized: view +secure: true ``` #### Source freshness known limitation Snowflake calculates source freshness using information from the `LAST_ALTERED` column, meaning it relies on a field updated whenever any object undergoes modification, not only data updates. No action must be taken, but analytics teams should note this caveat. Per the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/info-schema/tables#usage-notes): > The `LAST_ALTERED` column is updated when the following operations are performed on an object: > > * DDL operations. > * DML operations (for tables only). > * Background maintenance operations on metadata performed by Snowflake. (Applies to dbt v1.9 and later) #### Pagination for object results By default, when dbt encounters a schema with up to 100,000 objects, it will paginate the results from `show objects` at 10,000 per page for up to 10 pages. Environments with more than 100,000 objects in a schema can customize the number of results per page and the page limit using the following [flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) in the `dbt_project.yml`: * `list_relations_per_page` — The number of relations on each page (Max 10k as this is the most Snowflake allows). * `list_relations_page_limit` — The maximum number of pages to include in the results. For example, if you wanted to include 10,000 objects per page and include up to 100 pages (1 million objects), configure the flags as follows: ```yml flags: list_relations_per_page: 10000 list_relations_page_limit: 100 ``` --- ### Snowflake permissions In Snowflake, permissions are used to control who can perform certain actions on different database objects. Use SQL statements to manage permissions in a Snowflake database. #### Set up Snowflake account This section explains how to set up permissions and roles within Snowflake. In Snowflake, you would perform these actions using SQL commands and set up your data warehouse and access control within Snowflake's ecosystem. 1. Set up databases ```text use role sysadmin; create database raw; create database analytics; ``` 2. Set up warehouses ```text create warehouse loading warehouse_size = xsmall auto_suspend = 3600 auto_resume = false initially_suspended = true; create warehouse transforming warehouse_size = xsmall auto_suspend = 60 auto_resume = true initially_suspended = true; create warehouse reporting warehouse_size = xsmall auto_suspend = 60 auto_resume = true initially_suspended = true; ``` 3. Set up roles and warehouse permissions ```text use role securityadmin; create role loader; grant all on warehouse loading to role loader; create role transformer; grant all on warehouse transforming to role transformer; create role reporter; grant all on warehouse reporting to role reporter; ``` 4. Create users, assigning them to their roles Every person and application gets a separate user and is assigned to the correct role. ```text create user stitch_user -- or fivetran_user password = '_generate_this_' default_warehouse = loading default_role = loader; create user claire -- or amy, jeremy, etc. password = '_generate_this_' default_warehouse = transforming default_role = transformer must_change_password = true; create user dbt_cloud_user password = '_generate_this_' default_warehouse = transforming default_role = transformer; create user looker_user -- or mode_user etc. password = '_generate_this_' default_warehouse = reporting default_role = reporter; -- then grant these roles to each user grant role loader to user stitch_user; -- or fivetran_user grant role transformer to user dbt_cloud_user; grant role transformer to user claire; -- or amy, jeremy grant role reporter to user looker_user; -- or mode_user, periscope_user ``` 5. Let loader load data Give the role unilateral permission to operate on the raw database ```text use role sysadmin; grant all on database raw to role loader; ``` 6. Let transformer transform data The transformer role needs to be able to read raw data. If you do this before you have any data loaded, you can run: ```text grant usage on database raw to role transformer; grant usage on future schemas in database raw to role transformer; grant select on future tables in database raw to role transformer; grant select on future views in database raw to role transformer; ``` If you already have data loaded in the raw database, make sure also you run the following to update the permissions ```text grant usage on all schemas in database raw to role transformer; grant select on all tables in database raw to role transformer; grant select on all views in database raw to role transformer; ``` transformer also needs to be able to create in the analytics database: ```text grant all on database analytics to role transformer; ``` 7. Let reporter read the transformed data A previous version of this article recommended this be implemented through hooks in dbt, but this way lets you get away with a one-off statement. ```text grant usage on database analytics to role reporter; grant usage on future schemas in database analytics to role reporter; grant select on future tables in database analytics to role reporter; grant select on future views in database analytics to role reporter; ``` Again, if you already have data in your analytics database, make sure you run: ```text grant usage on all schemas in database analytics to role reporter; grant select on all tables in database analytics to role reporter; grant select on all views in database analytics to role reporter; ``` 8. Maintain When new users are added, make sure you add them to the right role! Everything else should be inherited automatically thanks to those `future` grants. For more discussion and legacy information, refer to [this Discourse article](https://discourse.getdbt.com/t/setting-up-snowflake-the-exact-grant-statements-we-run/439). #### Example Snowflake permissions The following example provides you with the SQL statements you can use to manage permissions. **Note** that `warehouse_name`, `database_name`, and `role_name` are placeholders and you can replace them as needed for your organization's naming convention. ```text grant all on warehouse warehouse_name to role role_name; grant usage on database database_name to role role_name; grant create schema on database database_name to role role_name; grant usage on schema database.an_existing_schema to role role_name; grant create table on schema database.an_existing_schema to role role_name; grant create view on schema database.an_existing_schema to role role_name; grant usage on future schemas in database database_name to role role_name; grant monitor on future schemas in database database_name to role role_name; grant select on future tables in database database_name to role role_name; grant select on future views in database database_name to role role_name; grant usage on all schemas in database database_name to role role_name; grant monitor on all schemas in database database_name to role role_name; grant select on all tables in database database_name to role role_name; grant select on all views in database database_name to role role_name; ``` --- ### Source configurations #### Available configurations (Applies to dbt v1.9 and later) Sources configurations support [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md), [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md), and [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) ##### General configurations General configurations provide broader operational settings applicable across multiple resource types. Like resource-specific configurations, these can also be set in the project file, property files, or within resource-specific files. ##### Project YAML file dbt\_project.yml (Applies to dbt v1.9 and later) ```yaml sources: <resource-path>: +enabled: true | false +event_time: my_time_field +freshness: warn_after: count: <positive_integer> period: minute | hour | day +meta: key: value ``` ##### Properties YAML file models/properties.yml (Applies to dbt v1.9 and later) ```yaml sources: - name: [<source-name>] database: <database-name> schema: <schema-name> config: enabled: true | false event_time: my_time_field meta: {<dictionary>} freshness: warn_after: count: <positive_integer> period: minute | hour | day tables: - name: [<source-table-name>] config: enabled: true | false event_time: my_time_field meta: {<dictionary>} ``` #### Configuring sources Sources can be configured via a `config:` block within their `.yml` definitions, or from the `dbt_project.yml` file under the `sources:` key. This configuration is most useful for configuring sources imported from [a package](https://docs.getdbt.com/docs/build/packages.md). You can disable sources imported from a package to prevent them from rendering in the documentation, or to prevent [source freshness checks](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) from running on source tables imported from packages. * **Note**: To disable a source table nested in a properties YAML file in a subfolder, you will need to supply the subfolder(s) within the path to that properties YAML file, as well as the source name and the table name in the project YAML file (`dbt_project.yml`).<br /><br />The following example shows how to disable a source table nested in a properties YAML file in a subfolder: dbt\_project.yml (Applies to dbt v1.9 and later) ```yaml sources: your_project_name: subdirectory_name: source_name: source_table_name: +enabled: false +event_time: my_time_field ``` ##### Examples The following examples show how to configure sources in your dbt project. — [Disable all sources imported from a package](#disable-all-sources-imported-from-a-package)<br />— [Conditionally enable a single source](#conditionally-enable-a-single-source)<br />— [Disable a single source from a package](#disable-a-single-source-from-a-package)<br />— [Configure a source with an `event_time`](#configure-a-source-with-an-event_time)<br />— [Configure meta to a source](#configure-meta-to-a-source)<br />— [Configure source freshness](#configure-source-freshness)<br /> ###### Disable all sources imported from a package To apply a configuration to all sources included from a [package](https://docs.getdbt.com/docs/build/packages.md), state your configuration under the [project name](https://docs.getdbt.com/reference/project-configs/name.md) in the `sources:` config as a part of the resource path. dbt\_project.yml ```yml sources: events: +enabled: false ``` ###### Conditionally enable a single source When defining a source, you can disable the entire source, or specific source tables, using the inline `config` property. You can also specify `database` and `schema` to override the target database and schema: models/sources.yml ```yml sources: - name: my_source database: raw schema: my_schema config: enabled: true tables: - name: my_source_table # enabled - name: ignore_this_one # not enabled config: enabled: false ``` You can configure specific source tables, and use [variables](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) as the input to that configuration: models/sources.yml ```yml sources: - name: my_source tables: - name: my_source_table config: enabled: "{{ var('my_source_table_enabled', false) }}" ``` ###### Disable a single source from a package To disable a specific source from another package, qualify the resource path for your configuration with both a package name and a source name. In this case, we're disabling the `clickstream` source from the `events` package. dbt\_project.yml ```yml sources: events: clickstream: +enabled: false ``` Similarly, you can disable a specific table from a source by qualifying the resource path with a package name, source name, and table name: dbt\_project.yml ```yml sources: events: clickstream: pageviews: +enabled: false ``` ###### Configure a source with an `event_time` (Applies to dbt v1.9 and later) To configure a source with an `event_time`, specify the `event_time` field in the source configuration. This field is used to represent the actual timestamp of the event, rather than something like a loading date. For example, if you had a source table called `clickstream` in the `events` source, you can use the timestamp for each event in the `event_timestamp` column as follows: dbt\_project.yml ```yaml sources: events: clickstream: +event_time: event_timestamp ``` In this example, the `event_time` is set to `event_timestamp`, which has the exact time each clickstream event happened. Not only is this required for the [incremental microbatching strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md), but when you compare data across [CI and production](https://docs.getdbt.com/docs/deploy/advanced-ci.md#speeding-up-comparisons) environments, dbt will use `event_timestamp` to filter and match data by this event-based timeframe, ensuring that only overlapping timeframes are compared. ###### Configure meta to a source Use the `meta` field to assign metadata information to sources. This is useful for tracking additional context, documentation, logging, and more. For example, you can add `meta` information to a `clickstream` source to include information about the data source system: dbt\_project.yml ```yaml sources: events: clickstream: +meta: source_system: "Google analytics" data_owner: "marketing_team" ``` ###### Configure source freshness Use a `freshness` block to define expectations about how frequently a table is updated with new data, and to raise warnings and errors when those expectation are not met. dbt compares the most recently updated timestamp calculated from a column, warehouse metadata, or custom query against the current timestamp when the freshness check is running. You can provide one or both of the `warn_after` and `error_after` parameters. If neither is provided, then dbt will not calculate freshness snapshots for the tables in this source. For more information, see [freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md). See the following example of a `dbt_project.yml` file using the `freshness` config: dbt\_project.yml ```yml sources: <resource-path>: +freshness: warn_after: count: 4 period: hour ``` #### Example source configuration The following is a valid source configuration for a project with: * `name: jaffle_shop` * A package called `events` containing multiple source tables dbt\_project.yml ```yml name: jaffle_shop config-version: 2 ... sources: # project names jaffle_shop: +enabled: true events: # source names clickstream: # table names pageviews: +enabled: false link_clicks: +enabled: true ``` --- ### Source definitions for state:modified Removed in v2 This flag was removed in v2. The new behavior is always enabled. If you're upgrading, remove this flag from your `dbt_project.yml`. | state\_modified\_compare\_more\_unrendered\_values | dbt **v1 Latest** | dbt v1 | | -------------------------------------------------- | ----------------- | ------ | | Introduced | 2024.10 | 1.9.0 | | Matured (default → `true`) | 2026.09 | 1.12.0 | | Removed | — | v2.0 | info You need to build the state directory using dbt v1.9 or higher, or [the dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), and you need to set `state_modified_compare_more_unrendered_values` to `true` within your dbt\_project.yml. If the state directory was built with an older dbt version or if the `state_modified_compare_more_unrendered_values` behavior change flag was either not set or set to `false`, you need to rebuild the state directory to avoid false positives during state comparison with `state:modified`. In dbt v1.12, `state_modified_compare_more_unrendered_values` defaults to `true`, reducing false positives during `state:modified` checks, especially when configs differ by target environment (such as `prod` vs. `dev`). The flag changes the `state:modified` comparison from using rendered values to unrendered values instead, by persisting `unrendered_config` during model parsing and `unrendered_database` and `unrendered_schema` configs during source parsing. note This flag requires rebuilding the state directory (manifest) to take effect. #### Impact This flag silently changes the `state:modified` selection set that most CI, Slim CI, and `dbt build --defer` workflows rely on. There are two ways this surfaces: * **False "modified" on the first run after the flag is set to `true`.** If the baseline manifest was captured before the flag is set (rendered values stored) and the current parse runs after the setting change (literal text stored), every node whose YAML config contains Jinja will appear as `state:modified`, even if nothing has changed. This causes a full rebuild on the first CI run after the upgrade. * **New positives going forward.** After both manifests are captured with the flag enabled, `state:modified` will catch cases where two equivalent Jinja expressions render to the same value (for example, switching from `"{{ env_var('MAT', 'view') }}"` to `view`).  What to expect On the first CI or Slim CI run with this flag enabled, any node whose YAML config uses Jinja (`env_var`, `var`, conditional materialization) may appear as `state:modified` even if nothing changed. This is because the baseline manifest stored rendered values while the new parse stores literal Jinja text — the two sides of the comparison differ on serialization, not on real changes. Once your production job runs once on the **v1 Latest** release track and generates a new baseline manifest, both sides of the `state:modified` comparison use the same format and the extra diffs disappear. No code change is required. The cost is one extra rebuild cycle for affected nodes. This new behavior fixes the legacy behavior. Previously, the comparison silently missed real changes to Jinja expressions (for example, switching from `"{{ env_var('MAT', 'view') }}"` to `view` renders identically and was never caught). With this flag enabled, `state:modified` is more accurate going forward. To opt out, set the flag to `false` in `dbt_project.yml`: dbt\_project.yml ```yaml flags: state_modified_compare_more_unrendered_values: false ``` Opting out is recommended during a production deploy freeze or if your project has heavy Jinja in YAML configs and the one-time rebuild would exceed your CI compute budget. Otherwise, enabling this flag is recommended. --- ### Source properties #### Related documentation * [Using sources](https://docs.getdbt.com/docs/build/sources.md) * [Declaring resource properties](https://docs.getdbt.com/reference/configs-and-properties.md) #### Overview Source properties can be declared in any `properties.yml` file in your `models/` directory (as defined by the [`model-paths` config](https://docs.getdbt.com/reference/project-configs/model-paths.md)). Source properties are "special properties" in that you can't configure them in the `dbt_project.yml` file or using `config()` blocks. Refer to [Configs and properties](https://docs.getdbt.com/reference/define-properties#which-properties-are-not-also-configs) for more info.<br /> You can name these files `whatever_you_want.yml`, and nest them arbitrarily deeply in subfolders within the `models/` directory: models/\<filename>.yml ```yml sources: - name: <string> # required description: <markdown_string> database: <database_name> schema: <schema_name> loader: <string> config: # requires v1.1+ <source_config>: <config_value> loaded_at_field: <column_name> # moved under config in v1.10 freshness: # moved under config in v1.10 warn_after: count: <positive_integer> period: minute | hour | day error_after: count: <positive_integer> period: minute | hour | day filter: <where-condition> meta: {<dictionary>} # moved under config in v1.10 tags: [<string>] # moved under config in v1.10 overrides: <string> # deprecated in v1.10 quoting: database: true | false schema: true | false identifier: true | false tables: - name: <string> #required description: <markdown_string> identifier: <table_name> data_tests: - <test> - ... # declare additional tests config: loaded_at_field: <column_name> meta: {<dictionary>} tags: [<string>] freshness: warn_after: count: <positive_integer> period: minute | hour | day error_after: count: <positive_integer> period: minute | hour | day filter: <where-condition> quoting: database: true | false schema: true | false identifier: true | false external: {<dictionary>} columns: - name: <column_name> # required description: <markdown_string> quote: true | false data_tests: - <test> - ... # declare additional tests config: meta: {<dictionary>} tags: [<string>] - name: ... # declare properties of additional columns - name: ... # declare properties of additional source tables - name: ... # declare properties of additional sources ``` #### Example models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: public loader: emr # informational only (free text) config: # changed to config in v1.10 loaded_at_field: _loaded_at # configure for all sources # meta fields are rendered in auto-generated documentation meta: # changed to config in v1.10 contains_pii: true owner: "@alice" # Add tags to this source tags: # changed to config in v1.10 - ecom - pii quoting: database: false schema: false identifier: false tables: - name: orders identifier: Orders_ config: # changed to config in v1.10 loaded_at_field: updated_at # override source defaults columns: - name: id data_tests: - unique - name: price_in_usd data_tests: - not_null - name: customers quoting: identifier: true # override source defaults columns: data_tests: - unique ``` --- ### Sources JSON file **Current schema:** [`v3`](https://schemas.getdbt.com/dbt/sources/v3/index.html) (Applies to dbt v2.0 and later) **Produced by:** [`dbt freshness`](https://docs.getdbt.com/reference/commands/freshness.md) (when sources are included in the run) This file contains information about [sources with freshness checks](https://docs.getdbt.com/docs/build/sources.md#evaluate-source-freshness). dbt uses this file to power its [Source Freshness visualization](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness). (Applies to dbt v2.0 and later) `sources.json` is a legacy artifact that contains freshness results for sources only. For results that includes both sources and models in one file, use [`freshness.json`](https://docs.getdbt.com/reference/artifacts/freshness-json.md). If a `dbt freshness` run checks only models and no sources are included, dbt does not overwrite `sources.json`. ##### Top-level keys * [`metadata`](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md#common-metadata) * `elapsed_time`: Total invocation time in seconds. * `results`: Array of freshness-check execution details. Each entry in `results` is a dictionary with the following keys: * `unique_id`: Unique source node identifier, which map results to `sources` in the [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) * `max_loaded_at`: Max value of `loaded_at_field` timestamp in the source table when queried. * `snapshotted_at`: Current timestamp when querying. * `max_loaded_at_time_ago_in_s`: Interval between `max_loaded_at` and `snapshotted_at`, calculated in python to handle timezone complexity. * `criteria`: The freshness threshold(s) for this source, defined in the project. * `status`: The freshness status of this source, based on `max_loaded_at_time_ago_in_s` + `criteria`, reported on the CLI. One of `pass`, `warn`, or `error` if the query succeeds, `runtime error` if the query fails. * `execution_time`: Total time spent checking freshness for this source * `timing`: Array that breaks down execution time into steps (`compile` + `execute`) - `adapter_response`: Dictionary of metadata returned from the database, which varies by adapter. For example, success `code`, number of `rows_affected`, total `bytes_processed`, and so on. Not applicable for [data tests](https://docs.getdbt.com/docs/build/data-tests.md). * `rows_affected` returns the number of rows modified by the last statement executed. In cases where the query's row count can't be determined or isn't applicable (such as when creating a view), a [standard value](https://peps.python.org/pep-0249/#rowcount) of `-1` is returned for `rowcount`. --- ### SQL parse dbt uses the Python [`sqlparse`](https://pypi.org/project/sqlparse/) library when it parses SQL. For large or complex queries, you may need to adjust how `sqlparse` groups tokens to avoid performance issues or parsing limits. You can use the `--sqlparse` flag to adjust how `sqlparse` groups tokens. This maps to the grouping limits described in the [sqlparse Security and Performance Considerations](https://sqlparse.readthedocs.io/en/latest/api.html#security-and-performance-considerations). Supported keys: | Key | What it does | | --------------------- | ------------------------------------------------------------- | | `MAX_GROUPING_DEPTH` | Sets the maximum nesting depth when grouping tokens | | `MAX_GROUPING_TOKENS` | Sets the maximum number of tokens that can be grouped at once | <br /> Each key accepts a single value: an integer or `null`. The default value for both keys is `null`, which means dbt does not apply a custom limit. `sqlparse` enforces built-in limits (`MAX_GROUPING_DEPTH = 100` and `MAX_GROUPING_TOKENS = 10000`) to prevent excessive resource usage. Setting a value to `null` disables the limit entirely. For example, the following command sets both grouping limits: ```bash dbt compile --sqlparse '{"MAX_GROUPING_DEPTH": 200, "MAX_GROUPING_TOKENS": 20000}' ``` You can configure only one key to set a single limit. Any unspecified key defaults to `null`. In the following example, only the depth limit is raised to `500`; `MAX_GROUPING_TOKENS` is not set, so `sqlparse` does not apply a token limit for this run. ```bash dbt compile --sqlparse '{"MAX_GROUPING_DEPTH": 500}' ``` If no keys are provided, both limits remain unset (`null`), and `sqlparse` does not apply any grouping limits. You can use `--sqlparse` with the following commands: * [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) * [`dbt run`](https://docs.getdbt.com/reference/commands/run.md) * [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) * [`dbt test`](https://docs.getdbt.com/reference/commands/test.md) * [`dbt seed`](https://docs.getdbt.com/reference/commands/seed.md) * [`dbt snapshot`](https://docs.getdbt.com/reference/commands/snapshot.md) * [`dbt source freshness`](https://docs.getdbt.com/reference/commands/source.md#freshness) * [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate) * [`dbt show`](https://docs.getdbt.com/reference/commands/show.md) --- ### sql_header `sql_header` does not support Jinja or macros like `ref` and `source` Unlike [pre-hooks and post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md), macros like [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md), and references like [`{{ this }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md), aren't supported. The primary function of `set_sql_header` is fairly limited. It's intended to: * [Create UDFs](https://docs.getdbt.com/reference/resource-configs/sql_header.md#create-a-bigquery-temporary-udf) * [Set script variables](https://cloud.google.com/bigquery/docs/reference/standard-sql/procedural-language) (BigQuery) * [Set temporary session parameters](https://docs.getdbt.com/reference/resource-configs/sql_header.md#set-snowflake-session-parameters-for-a-particular-model) (Snowflake) ##### Models models/\<modelname>.sql ```sql {{ config( sql_header="<sql-statement>" ) }} select ... ``` dbt\_project.yml ```yml config-version: 2 models: <resource-path>: +sql_header: <sql-statement> ``` ##### Seeds This config is not implemented for seeds ##### Snapshots snapshots/\<filename>.sql ```sql {% snapshot snapshot_name %} {{ config( sql_header="<sql-statement>" ) }} select ... {% endsnapshot %} ``` dbt\_project.yml ```yml snapshots: <resource-path>: +sql_header: <sql-statement> ``` ##### Property file Setting `sql_header` in the `config` of a [generic data test](https://docs.getdbt.com/docs/build/data-tests.md) is available starting in dbt v1.12. Enable the [`require_sql_header_in_test_configs`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_sql_header_in_test_configs.md) flag to use `sql_header` in `properties.yml` for generic data tests. Here's an example of a model-level configuration: models/properties.yml ```yaml models: - name: orders data_tests: - unique: name: unique_orders_order_id arguments: column_name: order_id config: sql_header: "-- SQL_HEADER_TEST_MARKER" ``` You can also use `sql_header` for column-level data tests: models/properties.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - not_null: name: not_null_orders_order_id config: sql_header: "-- SQL_HEADER_TEST_MARKER" ``` #### Definition An optional configuration to inject SQL above the `create table as` and `create view as` statements that dbt executes when building models and snapshots. `sql_header`s can be set using the config, or by `call`-ing the `set_sql_header` macro (example below). (Applies to dbt v1.12 and later) You can also set `sql_header` in the `config` of a [generic data test](https://docs.getdbt.com/docs/build/data-tests.md) at the model or column level in your `properties.yml` file. Use `sql_header` to define SQL that should run before the test executes (for example, to create temporary functions, set session parameters, or declare variables required by the test query). dbt runs this SQL before executing the test. Enable the [`require_sql_header_in_test_configs`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_sql_header_in_test_configs.md) flag to use `sql_header` for data tests. For more information, refer to [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md). #### Comparison to pre-hooks [Pre-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) also provide an opportunity to execute SQL before model creation, as a *preceding* query. In comparison, SQL in a `sql_header` is run in the same *query* as the `create table|view as` statement. As a result, this makes it more useful for [Snowflake session parameters](https://docs.snowflake.com/en/sql-reference/parameters.html) and [BigQuery Temporary UDFs](https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions#sql-udf-examples). #### Examples ##### Set Snowflake session parameters for a particular model This uses the config block syntax: models/my\_model.sql ```sql {{ config( sql_header="alter session set timezone = 'Australia/Sydney';" ) }} select * from {{ ref('other_model') }} ``` ##### Set Snowflake session parameters for all models dbt\_project.yml ```yml config-version: 2 models: +sql_header: "alter session set timezone = 'Australia/Sydney';" ``` ##### Create a BigQuery Temporary UDF This example calls the `set_sql_header` macro. This macro is a convenience wrapper which you may choose to use if you have a multi-line SQL statement to inject. You do not need to use the `sql_header` configuration key in this case. models/my\_model.sql ```sql -- Supply a SQL header: {% call set_sql_header(config) %} CREATE TEMPORARY FUNCTION yes_no_to_boolean(answer STRING) RETURNS BOOLEAN AS ( CASE WHEN LOWER(answer) = 'yes' THEN True WHEN LOWER(answer) = 'no' THEN False ELSE NULL END ); {%- endcall %} -- Supply your model code: select yes_no_to_boolean(yes_no) from {{ ref('other_model') }} ``` --- ### sql_header in data tests | require\_sql\_header\_in\_test\_configs | dbt **v1 Latest** | dbt v1 | | --------------------------------------- | ----------------- | ------ | | Introduced | 2026.3 | 1.12.0 | | Matured (default → `true`) | — | — | | Removed | — | — | Set the `require_sql_header_in_test_configs` flag to `true` to enable support for the [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) config for generic data tests. When enabled, you can set `sql_header` in the `config` of a generic data test at the model or column level in your `properties.yml` file. You can use `sql_header` to define SQL that should run before the test executes (for example, to create temporary functions, to set session parameters, or to declare variables required by the test query). dbt runs this SQL before executing the test. For example: models/properties.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - not_null: name: not_null_orders_order_id config: sql_header: "-- SQL_HEADER_TEST_MARKER" ``` For more information, refer to [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md). --- ### Starburst/Trino configurations #### Cluster requirements The designated cluster must have an attached catalog where objects such as tables and views can be created, renamed, altered, and dropped. Any user connecting to the cluster with dbt must also have these same permissions for the target catalog. #### Session properties With a Starburst Enterprise, Starburst Galaxy, or Trino cluster, you can [set session properties](https://trino.io/docs/current/sql/set-session.html) to modify the current configuration for your user session. The standard way to define session properties is with the `session_properties` field of your `profiles.yml`. This ensures that all dbt connections use these settings by default. However, to temporarily adjust these session properties for a specific dbt model or group of models, you can use a [dbt hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) to set session properties on a specific dbt model. For example: ```sql {{ config( pre_hook="set session query_max_run_time='10m'" ) }} ``` #### Connector properties You can use Starburst/Trino table properties to configure how you want your data to be represented. For details on what's supported for each supported data source, refer to either the [Trino Connectors](https://trino.io/docs/current/connector.html) or [Starburst Catalog](https://docs.starburst.io/starburst-galaxy/catalogs/). ##### Hive catalogs At target catalog that uses the Hive connector and a metastore service (HMS) is typical when working with Starburst and dbt. The following settings are recommended for working with dbt. The intent is to ensure that dbt can perform the frequently executed `DROP` and `RENAME` statements. ```java hive.metastore-cache-ttl=0s hive.metastore-refresh-interval=5s ``` #### File format configuration When using file-based connectors such as Hive, a user can customize aspects of the connector such as the format that is used as well the type of materialization The below configures the table to be materializes as a set of partitioned [Parquet](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html) files. ```sql {{ config( materialized='table', properties= { "format": "'PARQUET'", "partitioning": "ARRAY['bucket(id, 2)']", } ) }} ``` #### Seeds and prepared statements The [dbt seed](https://docs.getdbt.com/docs/build/seeds.md) command makes use of prepared statements in [Starburst](https://docs.starburst.io/latest/sql/prepare.html)/[Trino](https://trino.io/docs/current/sql/prepare.html). Prepared statements are templated SQL statements that you can execute repeatedly with high efficiency. The values are sent in a separate field rather than hard coded in the SQL string itself. This is often how application frontends structure their record `INSERT` statements in the OLTP database backend. Because of this, it's common for prepared statements to have as many placeholder variables (parameters) as there are columns in the destination table. Most seed files have more than one row, and often thousands of rows. This makes the size of the client request as large as there are parameters. ##### Header line length limit in Python HTTP client You might run into an error message about header line limit if your prepared statements have too many parameters. This is because the header line limit in Python's HTTP client is `65536` bytes. You can avoid this upper limit by converting the large prepared statement into smaller statements. dbt already does this by batching an entire seed file into groups of rows — one group for a number of rows in the CSV. Let's say you have a seed file with 20 columns, 600 rows, and 12,000 parameters. Instead of creating a single prepared statement for this, you can have dbt create four prepared `INSERT` statements with 150 rows and 3,000 parameters. There's a drawback to grouping your table rows. When there are many columns (parameters) in a seed file, the batch size needs to be very small. For the `dbt-trino` adapter, the macro for batch size is `trino__get_batch_size()` and its default value is `1000`. To change this default behavior, you can add this macro to your dbt project: macros/YOUR\_MACRO\_NAME.sql ```sql {% macro trino__get_batch_size() %} {{ return(10000) }} -- Adjust this number as you see fit {% endmacro %} ``` Another way to avoid the header line length limit is to set `prepared_statements_enabled` to `true` in your dbt profile; however, this is considered legacy behavior and can be removed in a future release. #### Materializations ##### Table The `dbt-trino` adapter supports these modes in `table` materialization (and [full-refresh runs](https://docs.getdbt.com/reference/commands/run.md#refresh-incremental-models) in `incremental` materialization), which you can configure with `on_table_exists`: * `rename` — Creates an intermediate table, renames the target table to the backup one, and renames the intermediate table to the target one. * `drop` — Drops and re-creates a table. This overcomes the table rename limitation in AWS Glue. * `replace` — Replaces a table using CREATE OR REPLACE clause. Support for table replacement varies across connectors. Refer to the connector documentation for details. * `skip` — Skips table materialization altogether using a CREATE TABLE IF NOT EXISTS clause. If CREATE OR REPLACE is supported in underlying connector, `replace` is recommended option. Otherwise, the recommended `table` materialization uses `on_table_exists = 'rename'` and is also the default. You can change this default configuration by editing *one* of these files: * the SQL file for your model * the `dbt_project.yml` configuration file The following examples configure `table` materialization to be `drop`: models/YOUR\_MODEL\_NAME.sql ```sql {{ config( materialized = 'table', on_table_exists = 'drop` ) }} ``` dbt\_project.yml ```yaml models: path: materialized: table +on_table_exists: drop ``` If you use `table` materialization and `on_table_exists = 'rename'` with AWS Glue, you might encounter this error message. You can overcome the table rename limitation by using `drop`: ```sh TrinoUserError(type=USER_ERROR, name=NOT_SUPPORTED, message="Table rename is not yet supported by Glue service") ``` ##### View The `dbt-trino` adapter supports these security modes in `view` materialization, which you can configure with `view_security`: * `definer` * `invoker` For more details about security modes in views, see [Security](https://trino.io/docs/current/sql/create-view.html#security) in the Trino docs. By default, `view` materialization uses `view_security = 'definer'`. You can change this default configuration by editing *one* of these files: * the SQL file for your model * the `dbt_project.yml` configuration file For example, these configure the security mode to `invoker`: models/YOUR\_MODEL\_NAME.sql ```sql {{ config( materialized = 'view', view_security = 'invoker' ) }} ``` dbt\_project.yml ```yaml models: path: materialized: view +view_security: invoker ``` ##### Incremental Using an incremental model limits the amount of data that needs to be transformed, which greatly reduces the runtime of your transformations. This improves performance and reduces compute costs. ```sql {{ config( materialized = 'incremental', unique_key='<optional>', incremental_strategy='<optional>',) }} select * from {{ ref('events') }} {% if is_incremental() %} where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ``` Use the `+on_schema_change` property to define how dbt-trino should handle column changes. For more details about this property, see [column changes](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change). If your connector doesn't support views, set the `+views_enabled` property to `false`. You can decide how model should be rebuilt in a `full-refresh` run by specifying `on_table_exists` config. Options are the same as described in [table materialization section](https://docs.getdbt.com/reference/resource-configs/trino-configs.md#table) ###### append strategy The default incremental strategy is `append`. `append` only adds new records based on the condition specified in the `is_incremental()` conditional block. ```sql {{ config( materialized = 'incremental') }} select * from {{ ref('events') }} {% if is_incremental() %} where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ``` ###### delete+insert strategy With the `delete+insert` incremental strategy, you can instruct dbt to use a two-step incremental approach. First, it deletes the records detected through the configured `is_incremental()` block, then re-inserts them. ```sql {{ config( materialized = 'incremental', unique_key='user_id', incremental_strategy='delete+insert', ) }} select * from {{ ref('users') }} {% if is_incremental() %} where updated_ts > (select max(updated_ts) from {{ this }}) {% endif %} ``` ###### merge strategy With the `merge` incremental strategy, dbt-trino constructs a [Trino MERGE statement](https://trino.io/docs/current/sql/merge.html) to `insert` new records and `update` existing records, based on the `unique_key` property. If `unique_key` is not unique, you can use the `delete+insert` strategy instead. ```sql {{ config( materialized = 'incremental', unique_key='user_id', incremental_strategy='merge', ) }} select * from {{ ref('users') }} {% if is_incremental() %} where updated_ts > (select max(updated_ts) from {{ this }}) {% endif %} ``` Be aware that there are some Trino connectors that don't support `MERGE` or have limited support. ###### Incremental overwrite on Hive models If there's a [Hive connector](https://trino.io/docs/current/connector/hive.html) accessing your target incremental model, you can simulate an `INSERT OVERWRITE` statement by using the `insert-existing-partitions-behavior` setting on the Hive connector configuration in Trino: ```ini <hive-catalog-name>.insert-existing-partitions-behavior=OVERWRITE ``` Below is an example Hive configuration that sets the `OVERWRITE` functionality for a Hive connector called `minio`: ```yaml trino-incremental-hive: target: dev outputs: dev: type: trino method: none user: admin password: catalog: minio schema: tiny host: localhost port: 8080 http_scheme: http session_properties: minio.insert_existing_partitions_behavior: OVERWRITE threads: 1 ``` `dbt-trino` overwrites existing partitions in the target model that match the staged data. It appends the remaining partitions to the target model. This functionality works on incremental models that use partitioning. For example: ```sql {{ config( materialized = 'incremental', properties={ "format": "'PARQUET'", "partitioned_by": "ARRAY['day']", } ) }} ``` ##### Materialized view The `dbt-trino` adapter supports [materialized views](https://trino.io/docs/current/sql/create-materialized-view.html) and refreshes them for every subsequent `dbt run` that you execute. For more information, see [REFRESH MATERIALIZED VIEW](https://trino.io/docs/current/sql/refresh-materialized-view.html) in the Trino docs. You can also define custom properties for the materialized view through the `properties` config. This materialization supports the [full\_refresh](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) config and flag. Whenever you want to rebuild your materialized view (for example, when changing underlying SQL query) run `dbt run --full-refresh`. You can create a materialized view by editing *one* of these files: * the SQL file for your model * the `dbt_project.yml` configuration file The following examples create a materialized view in Parquet format: models/YOUR\_MODEL\_NAME.sql ```sql {{ config( materialized = 'materialized_view', properties = { 'format': "'PARQUET'" }, ) }} ``` dbt\_project.yml ```yaml models: path: materialized: materialized_view properties: format: "'PARQUET'" ``` #### Snapshots [Snapshots in dbt](https://docs.getdbt.com/docs/build/snapshots.md) depend on the `current_timestamp` macro, which returns a timestamp with millisecond precision (3 digits) by default. There are some connectors for Trino that don't support this timestamp precision (`TIMESTAMP(3) WITH TIME ZONE`), like Iceberg. To change timestamp precision, you can define your own [macro](https://docs.getdbt.com/docs/build/jinja-macros.md). For example, this defines a new `trino__current_timestamp()` macro with microsecond precision (6 digits): macros/YOUR\_MACRO\_NAME.sql ```sql {% macro trino__current_timestamp() %} current_timestamp(6) {% endmacro %} ``` #### Grants Use [grants](https://docs.getdbt.com/reference/resource-configs/grants.md) to manage access to the datasets you're producing with dbt. You can use grants with [Starburst Enterprise](https://docs.starburst.io/latest/security/biac-overview.html), [Starburst Galaxy](https://docs.starburst.io/starburst-galaxy/security/access-control.html), and Hive ([sql-standard](https://trino.io/docs/current/connector/hive-security.html)). To implement access permissions, define grants as resource configs on each model, seed, and snapshot. Define the default grants that apply to the entire project in your `dbt_project.yml` and define model-specific grants within each model's SQL or YAML file. dbt\_project.yml ```yaml models: - name: NAME_OF_YOUR_MODEL config: grants: select: ['reporter', 'bi'] ``` #### persist\_docs *Available in `dbt-trino` v1.10.3 and later* By default, [`persist_docs`](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) writes model and column descriptions to the underlying catalog using `COMMENT ON` SQL statements. If you set `starburst_url` along with `starburst_client_id` and `starburst_secret_key` in your connection profile, enabling `persist_docs` also syncs the model and column descriptions to Starburst's Data Discovery catalog through its REST API. This is opt-in and additive — projects that don't set `starburst_url` are unaffected. For details on configuring these profile fields, refer to [Additional parameters](https://docs.getdbt.com/docs/local/connect-data-platform/trino-setup.md#additional-parameters) in the Starburst/Trino setup guide. Column descriptions sync to Data Discovery in batches. The `starburst_max_column_batch_size` parameter controls the batch size with a default of `100`. The `starburst_metadata_failure_strategy` controls how dbt handles errors from the Data Discovery API: * `continue_on_error` (default) — Logs a warning and continues the run. * `fail_fast` — Aborts the run. #### Model contracts The `dbt-trino` adapter supports [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md). Currently, only [constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md) with `type` as `not_null` are supported. Before using `not_null` constraints in your model, make sure the underlying connector supports `not null`, to avoid running into errors. #### Query routing A Trino router picks the cluster that runs each statement from the request headers, so setting `client_tags` and `http_headers` lets a single dbt run send individual models to different clusters. For example, a large incremental model can run on a fault-tolerant cluster while the rest of the project stays on the default one. With [Starburst Galaxy](https://docs.starburst.io/starburst-galaxy/working-with-data/query-routing/user-role-based-routing.html), the routing decision is made from the client tags and the user's role. Use the following configs to set the request headers: * `client_tags` — A list of strings sent to Trino in the `X-Trino-Client-Tags` header. Trino routers use these tags to select the cluster that runs each statement. Tag values can't contain commas. * `http_headers` — HTTP headers sent alongside requests to Trino, specified as a YAML dictionary of header names and values. You can use custom headers for routers that support them, but you can't override reserved headers or set `X-Trino-Client-Tags` here. Refer to [Limitations](#limitations) for details. ##### Connection-level routing Set `client_tags` and `http_headers` in your connection profile to apply them to every statement dbt runs. \~/.dbt/profiles.yml ```yaml my_project: target: dev outputs: dev: type: trino host: mycluster.mydomain.com database: my_catalog schema: my_schema port: 443 client_tags: - analytics http_headers: X-Trino-Client-Info: dbt-trino ``` ##### Model-level routing *Available in `dbt-trino` v1.10.4 and later* Set `client_tags` and `http_headers` on a model to override the profile's values for the statements that model runs. models/\<model\_name>.sql ```sql {{ config( materialized = 'table', client_tags = ['fault-tolerant'] ) }} ``` dbt\_project.yml ```yaml models: my_project: heavy_models: +client_tags: ['fault-tolerant'] ``` ##### Precedence and merging The two configs interact with the profile's values differently: * `client_tags` — A model's tags *replace* the profile's tags entirely. They aren't added to them. * `http_headers` — A model's headers *merge with* the profile's headers. If both define the same header, the model's value takes precedence. Any headers the model doesn't define are inherited from the profile. ##### Limitations * If your tags don't match any routing rule and the router has no default rule configured, the query is rejected. Ensure your profile's `client_tags` match at least one routing rule. * Statements that dbt runs outside of a model context use the profile's `client_tags` and `http_headers`. For example, startup bookkeeping, such as listing schemas, creating schemas, and populating the relation cache, runs before models execute and doesn't use a model's values. * Tests are separate nodes with their own config — they don't inherit the routing config of the model they test. To route a test to the same cluster as its model, set `client_tags` or `http_headers` on the test directly, either in the test's `config` block or path-scoped under `data_tests:` in `dbt_project.yml`. * You can't set `X-Trino-Client-Tags` through `http_headers` because Trino builds that header itself from `client_tags`. * You can't override headers managed by the Trino Python client through `http_headers`. For header descriptions, refer to Trino's [client request headers](https://trino.io/docs/current/develop/client-protocol.html#client-request-headers). For the reserved-header list enforced by `dbt-trino`, refer to `RESERVED_HTTP_HEADERS` in the adapter's [connection implementation](https://github.com/starburstdata/dbt-trino/blob/master/dbt/adapters/trino/connections.py). --- ### Starrocks configurations #### Model Configuration A dbt model can be configured using the following syntax: ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: materialized: table // table or view or materialized_view keys: ['id', 'name', 'some_date'] table_type: 'PRIMARY' // PRIMARY or DUPLICATE or UNIQUE distributed_by: ['id'] buckets: 3 // default 10 partition_by: ['some_date'] partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"] properties: [{"replication_num":"1", "in_memory": "true"}] refresh_method: 'async' // only for materialized view default manual ``` ##### Properties YAML file models/properties.yml ```yaml models: - name: <model-name> config: materialized: table // table or view or materialized_view keys: ['id', 'name', 'some_date'] table_type: 'PRIMARY' // PRIMARY or DUPLICATE or UNIQUE distributed_by: ['id'] buckets: 3 // default 10 partition_by: ['some_date'] partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"] properties: [{"replication_num":"1", "in_memory": "true"}] refresh_method: 'async' // only for materialized view default manual ``` ##### SQL file config models/\<model\_name>.sql ```jinja {{ config( materialized = 'table', keys=['id', 'name', 'some_date'], table_type='PRIMARY', distributed_by=['id'], buckets=3, partition_by=['some_date'], .... ) }} ``` ##### Configuration Description | Option | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `materialized` | How the model will be materialized into Starrocks. Supports view, table, incremental, ephemeral, and materialized\_view. | | `keys` | Which columns serve as keys. | | `table_type` | Table type, supported are PRIMARY or DUPLICATE or UNIQUE. | | `distributed_by` | Specifies the column of data distribution. If not specified, it defaults to random. | | `buckets` | The bucket number in one partition. If not specified, it will be automatically inferred. | | `partition_by` | The partition column list. | | `partition_by_init` | The partition rule or some real partitions item. | | `properties` | The table properties configuration of Starrocks. ([Starrocks table properties](https://docs.starrocks.io/en-us/latest/sql-reference/sql-statements/data-definition/CREATE_TABLE#properties)) | | `refresh_method` | How to refresh materialized views. | #### Read From Catalog First you need to add this catalog to starrocks. The following is an example of hive. ```sql CREATE EXTERNAL CATALOG `hive_catalog` PROPERTIES ( "hive.metastore.uris" = "thrift://127.0.0.1:8087", "type"="hive" ); ``` How to add other types of catalogs can be found in the documentation. [Catalog Overview](https://docs.starrocks.io/en-us/latest/data_source/catalog/catalog_overview) Then write the sources.yaml file. ```yaml sources: - name: external_example schema: hive_catalog.hive_db tables: - name: hive_table_name ``` Finally, you might use below marco quote ```jinja {{ source('external_example', 'hive_table_name') }} ``` --- ### Static analysis Use the `--static-analysis` flag to override model-level `static_analysis` behavior for a single run. This flag applies to dbt v2 only; it is ignored by dbt v1. Values: * `baseline` (default): Statically analyze SQL for all models in the run. This is the recommended starting point for users transitioning from dbt v1. * `strict`: Statically analyze all SQL before execution begins. Provides maximum validation guarantees — nothing runs until the entire project is proven valid. * `off`: Disable static analysis for all models in the run. If not set, dbt v2 defaults to `baseline` mode, which provides a smooth transition from dbt v1 while still catching most SQL errors. See [Configuring `static_analysis`](https://docs.getdbt.com/docs/build/about-static-analysis.md#configuring-static_analysis) for more information on incrementally opting in to stricter analysis. Usage ```shell dbt run --static-analysis strict dbt run --static-analysis baseline dbt run --static-analysis off ``` #### Related docs * [`static_analysis` (resource config)](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) * [About flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) * [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md) --- ### static_analysis static\_analysis controls how dbt v2 analyzes SQL at compile time for models, tests, seeds, and snapshots. info The `static_analysis` config is available in dbt v2 only. It isn't available in dbt v1 and will be ignored. To upgrade to dbt v2, refer to [Get started with dbt v2](https://docs.getdbt.com/docs/dbt/get-started-dbt.md). The `static_analysis` config sets how dbt v2 validates SQL before execution—using `strict` analysis, a `baseline` that balances checks with compatibility, or `off` to skip analysis when needed. You can find supported configuration locations for each resource type. To check out which Snowflake functions are supported in dbt v2 in `strict` mode, refer to [Snowflake function support](https://docs.getdbt.com/reference/resource-configs/snowflake-function-support.md). For BigQuery, refer to [BigQuery function support](https://docs.getdbt.com/reference/resource-configs/bigquery-function-support.md). ##### Models dbt\_project.yml ```yml models: resource-path: +static_analysis: strict | baseline | off ``` models/filename.yml ```yml models: - name: model_name config: static_analysis: strict | baseline | off ``` models/model\_name.sql ```sql {{ config(static_analysis='strict' | 'baseline' | 'off') }} ``` ##### Tests dbt\_project.yml ```yml data_tests: +static_analysis: strict | baseline | off ``` models/filename.yml ```yml models: - name: model_name data_tests: - not_null: arguments: column_name: your_column_name config: static_analysis: strict | baseline | off ``` ##### Seeds dbt\_project.yml ```yml seeds: resource-path: +static_analysis: strict | baseline | off ``` seeds/filename.yml ```yml seeds: - name: seed_name config: static_analysis: strict | baseline | off ``` ##### Snapshots dbt\_project.yml ```yml snapshots: resource-path: +static_analysis: strict | baseline | off ``` snapshots/filename.yml ```yml snapshots: - name: snapshot_name config: static_analysis: strict | baseline | off ``` ##### Unit tests dbt\_project.yml ```yml unit_tests: resource-path: +static_analysis: strict | baseline | off ``` models/filename.yml ```yml unit_tests: - name: unit_test_name model: model_name config: static_analysis: strict | baseline | off ``` #### Definition You can configure `static_analysis` for [models](https://docs.getdbt.com/docs/build/sql-models.md), [data tests](https://docs.getdbt.com/docs/build/data-tests.md), [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md), [seeds](https://docs.getdbt.com/docs/build/seeds.md), and [snapshots](https://docs.getdbt.com/docs/build/snapshots.md). You can configure if and when dbt v2 performs static SQL analysis for a model. Configure the `static_analysis` config in your project YAML file (`dbt_project.yml`), model properties YAML file, or in a SQL config block in your model file. Refer to [Principles of static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md?version=1.12#principles-of-static-analysis) for more information on the different modes of static analysis. Setting a model to `strict` does not automatically set `strict` for downstream models; they keep the project default unless you configure them explicitly. For more information and examples, refer to [strict mode inheritance](https://docs.getdbt.com/docs/build/about-static-analysis.md#strict-mode-inheritance). The following values are available for `static_analysis`: * `baseline` (default): Statically analyze SQL. This is the recommended starting point for users transitioning from dbt v1, providing a smooth migration experience while still catching most SQL errors. You can incrementally opt-in to stricter analysis over time. * `strict`: Statically analyze all SQL before execution begins. Use this for maximum validation guarantees — nothing runs until the entire project is proven valid. * `off`: Skip SQL analysis for this model and its descendants. ##### User-defined functions (UDFs) in `strict` mode When `static_analysis: strict` is in effect, dbt v2 parses `CREATE FUNCTION` statements from [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) and from [`on-run-start`](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) project hooks, registers those UDFs in the compiler registry, and makes them available during strict static compilation. The `baseline` and `off` modes don't perform this UDF registration for static analysis. A model’s `sql_header` can include multiple statements. dbt v2 registers UDFs from `CREATE FUNCTION` statements and ignores other statements for this step. If strict analysis still cannot resolve a UDF, set [`static_analysis: off`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md#disable-static-analysis-in-sql-for-a-model-using-a-custom-udf) on the affected models. ##### How static analysis modes cascade Two rules determine how `static_analysis` modes apply in a lineage: * Eligibility rule: A model is eligible for static analysis only if all of its "parents" are eligible (by parents, we mean the models that are upstream of the current model in the lineage). * Strictness rule: A "child" model cannot be stricter than its parent (by child, we mean the models that are downstream of the current model in the lineage). The static analysis configuration cascades from most strict to least strict. Here's the strictness hierarchy: `strict` → `baseline` → `off` **Allowed downstream by parent mode**<br />When going downstream in your lineage, you can keep the same mode or relax it; but you cannot make a child stricter than its parent. The following table shows the allowed downstream modes by parent mode: | Parent mode | Child can be | | ----------- | ---------------------------------- | | `strict` | `strict`, `baseline`, or `off` | | `baseline` | `baseline` or `off` (not `strict`) | | `off` | `off` only | For example, for the lineage Model A → Model B → Model C: * If Model A is `baseline`, you *cannot* set Model B to `strict` * If Model A is `strict`, you *can* set Model B to `baseline` This makes sure that stricter validation requirements don't apply downstream when parent models haven't met those requirements. `baseline` doesn't produce the full analyzed schema that `strict` needs from its upstream models, so a downstream model of a `baseline` model can't run strict-level type checking. Refer to the v2 concepts page for deeper discussion and visuals: [New concepts](https://docs.getdbt.com/docs/build/about-static-analysis.md). For more info on the JSON schema, refer to the [dbt-jsonschema file](https://github.com/dbt-labs/dbt-jsonschema/blob/1e2c1536fbdd421e49c8b65c51de619e3cd313ff/schemas/latest_fusion/dbt_project-latest-fusion.json#L4689). ##### Custom materializations Models that use a [custom materialization](https://docs.getdbt.com/guides/create-new-materializations.md) follow the same `static_analysis` rules as any other model, whether you give the materialization a new name or reuse a built-in one like `table` or `incremental`. Downstream models follow the usual [cascading rules](#how-static-analysis-modes-cascade). If a custom materialization adds, renames, or retypes columns, dbt v2 can't predict those changes from the model's SQL. This mainly affects `strict`, which needs accurate upstream schemas. If you use `strict`, set `static_analysis: off` on the models that use that materialization. Because `off` cascades, every downstream model also loses static analysis and the features that depend on it, such as column-level lineage and type checking. To keep coverage across your DAG, use schema-changing materializations on models that nothing else depends on. For more information, refer to [Custom materializations](https://docs.getdbt.com/docs/build/about-static-analysis.md#custom-materializations). #### CLI override You can override model-level configuration for a run using the [`--static-analysis`](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) flag. For example, to disable static analysis for a run: ```bash dbt run --static-analysis off # disable static analysis for all models dbt run --static-analysis baseline # use baseline analysis for all models ``` #### Examples The following examples show how to disable or configure `static_analysis` for different scenarios: * [Enable strict analysis for all your models](#enable-strict-analysis-for-all-your-models) * [Disable static analysis for all models in a package](#disable-static-analysis-for-all-models-in-a-package) * [Disable static analysis in YAML for a single model](#disable-static-analysis-in-yaml-for-a-single-model) * [Disable static analysis in SQL for a model using a custom UDF](#disable-static-analysis-in-sql-for-a-model-using-a-custom-udf) * [Configure static analysis for data tests](#configure-static-analysis-for-data-tests) * [Configure static analysis for unit tests](#configure-static-analysis-for-unit-tests) * [Configure static analysis for seeds](#configure-static-analysis-for-seeds) * [Configure static analysis for snapshots](#configure-static-analysis-for-snapshots) ##### Enable strict analysis for all your models The recommended way to get maximum SQL validation for your entire project is to set `strict` in the top-level `models` configuration in your `dbt_project.yml`. This configuration applies strict analysis to every model in your project, so you don't need to configure each model individually: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: +static_analysis: strict staging: +materialized: view marts: +materialized: table ``` You can set individual subdirectories or models to `baseline` or `off` where needed (for example, models that use unsupported UDFs). Individual models can use less strict settings than the project-level config, but you can't set them to stricter settings. The project default is `baseline`. In this example, strict static analysis applies only to Jaffle Shop models. Installed packages keep the default `baseline` setting unless you explicitly configure them. ##### Disable static analysis for all models in a package This example shows how to disable static analysis for all models in a package. The [`+` prefix](https://docs.getdbt.com/reference/resource-configs/plus-prefix.md) applies the config to all models in the package. dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: marts: +materialized: table a_package_with_introspective_queries: +static_analysis: off ``` ##### Disable static analysis in YAML for a single model This example shows how to disable static analysis for a single model in YAML. models/my\_udf\_using\_model.yml ```yml models: - name: model_with_static_analysis_off config: static_analysis: off ``` ##### Disable static analysis in SQL for a model using a custom UDF This example shows how to disable static analysis for a model using a custom [user-defined function (UDF)](https://docs.getdbt.com/docs/build/udfs.md) in a SQL file. models/my\_udf\_using\_model.sql ```sql {{ config(static_analysis='off') }} select user_id, my_cool_udf(ip_address) as cleaned_ip from {{ ref('my_model') }} ``` ##### Configure static analysis for data tests This example shows how to set static analysis for all tests in a project using `dbt_project.yml`. dbt\_project.yml ```yaml # dbt_project.yml data_tests: +static_analysis: baseline ``` To configure static analysis for a specific data test on a model: models/filename.yml ```yaml # models/filename.yml models: - name: my_model data_tests: - not_null: arguments: column_name: order_id config: static_analysis: off ``` (Applies to dbt v2.0 and later) ##### Configure static analysis for unit tests You can set `static_analysis` on [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md), which validate your SQL logic against static fixtures before you build a model. This example shows how to set static analysis for all unit tests in a project using `dbt_project.yml`. dbt\_project.yml ```yaml # dbt_project.yml unit_tests: my_project: +static_analysis: baseline ``` To configure a single unit test in a properties file: models/filename.yml ```yaml # models/filename.yml unit_tests: - name: test_is_valid_email_address model: dim_customers config: static_analysis: off ``` Setting [`compute: local`](https://docs.getdbt.com/reference/resource-configs/compute.md) on a unit test (which runs the test with DuckDB instead of your data platform) promotes its `static_analysis` to `strict`, because local execution needs strict analysis to translate your SQL to DuckDB. This applies even if you set `baseline` on the test yourself. If you set `static_analysis: off` on a unit test that's configured to run locally, the test can't run and fails with `ExecutorFailed (dbt1401)`. Use `compute: remote` for that test instead. `compute: local` is in experimental and requires the `DBT_ENGINE_EXPERIMENTAL_LOCAL_UNIT_TESTS` environment variable set to `true`, so this interaction only applies once you've opted in. ##### Configure static analysis for seeds This example shows how to set static analysis for all seeds in a project. dbt\_project.yml ```yaml # dbt_project.yml seeds: your_project: +static_analysis: baseline ``` To configure a single seed in a properties file: seeds/filename.yml ```yaml # seeds/filename.yml seeds: - name: my_seed config: static_analysis: off ``` ##### Configure static analysis for snapshots This example shows how to set static analysis for all snapshots in a project. dbt\_project.yml ```yaml # dbt_project.yml snapshots: your_project: +static_analysis: baseline ``` To configure a single snapshot in a properties file: snapshots/filename.yml ```yaml # snapshots/filename.yml snapshots: - name: my_snapshot config: static_analysis: off ``` #### Considerations * For models, disabling static analysis means that features of the VS Code extension that depend on SQL comprehension will be unavailable. * For models, static analysis can fail in some cases (for example, dynamic SQL constructs or unrecognized UDFs) and you might need to set `static_analysis: off`. For more examples, refer to [When should I turn static analysis off?](https://docs.getdbt.com/docs/build/about-static-analysis.md#when-should-i-turn-static-analysis-off). * Because modes cascade, the mode you configure isn't always the mode in effect. To check the effective mode for a model, use the CodeLens in the dbt VS Code extension or the Studio IDE, which shows which models have static analysis disabled and why. #### Related docs * [About static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) * [`--static-analysis` flag](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) * [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md) --- ### store_failures The configured test(s) will store their failures when `dbt test --store-failures` is invoked. If you set this configuration as `false` but [`store_failures_as`](https://docs.getdbt.com/reference/resource-configs/store_failures_as.md) is configured, it will be overridden. #### Description Optionally set a test to always or never store its failures in the database. * If specified as `true` or `false`, the `store_failures` config will take precedence over the presence or absence of the `--store-failures` flag. * If the `store_failures` config is `none` or omitted, the resource will use the value of the `--store-failures` flag. * When true, `store_failures` saves all records (up to [limit](https://docs.getdbt.com/reference/resource-configs/limit.md)) that failed the test. Failures are saved in a new table with the name of the test. * A test's results will always **replace** previous failures for the same test, even if that test results in no failures. * By default, `store_failures` uses a schema named `{{ profile.schema }}_dbt_test__audit`, but, you can [configure](https://docs.getdbt.com/reference/resource-configs/schema.md#tests) the schema to a different value. Ensure you have the authorization to create or access schemas for your work. For more details, refer to the [FAQ](#faqs). This logic is encoded in the [`should_store_failures()`](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/configs.sql#L15) macro. ##### Specific test Configure a specific instance of a generic (schema) test: models/\<filename>.yml ```yaml models: - name: my_model columns: - name: my_column data_tests: - unique: config: store_failures: true # always store failures - not_null: config: store_failures: false # never store failures ``` ##### Singular test Configure a singular (data) test: tests/\<filename>.sql ```sql {{ config(store_failures = true) }} select ... ``` ##### Generic test block Set the default for all instances of a generic (schema) test, by setting the config inside its test block (definition): macros/\<filename>.sql ```sql {% test <testname>(model, column_name) %} {{ config(store_failures = false) }} select ... {% endtest %} ``` ##### Project level Set the default for all tests in a package or project: dbt\_project.yml ```yaml data_tests: +store_failures: true # all tests <package_name>: +store_failures: false # tests in <package_name> ``` #### FAQs  Receiving a 'permissions denied for schema' error If you're receiving a `Adapter name adapter: Adapter_name error: permission denied for schema dev_username_dbt_test__audit`, this is most likely due to your user not having permission to create new schemas, despite having owner access to your own development schema. To resolve this, you need proper authorization to create or access custom schemas. Run the following SQL command in your respective data platform environment. Note that the exact authorization query may differ from one data platform to another: ```sql create schema if not exists dev_username_dbt_test__audit authorization username; ``` *Replace `dev_username` with your specific development schema name and `username` with the appropriate user who should have the permissions.* This command grants the appropriate permissions to create and access the `dbt_test__audit` schema, which is often used with the `store_failures` configuration. --- ### store_failures_as For the `test` resource type, `store_failures_as` is an optional config that specifies how test failures should be stored in the database. If [`store_failures`](https://docs.getdbt.com/reference/resource-configs/store_failures.md) is also configured, `store_failures_as` takes precedence. The three supported values are: * `ephemeral` — nothing stored in the database (default) * `table` — test failures stored as a database table * `view` — test failures stored as a database view You can configure it in all the same places as `store_failures`, including singular tests (.sql files), generic tests (.yml files), and dbt\_project.yml. ##### Examples ###### Singular test [Singular test](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests) in `tests/singular/check_something.sql` file ```sql {{ config(store_failures_as="table") }} -- custom singular test select 1 as id where 1=0 ``` ###### Generic test [Generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests) in `models/_models.yml` file ```yaml models: - name: my_model columns: - name: id data_tests: - not_null: config: store_failures_as: view - unique: config: store_failures_as: ephemeral ``` ###### Project level Config in `dbt_project.yml` ```yaml name: "my_project" version: "1.0.0" config-version: 2 profile: "sandcastle" data_tests: my_project: +store_failures_as: table my_subfolder_1: +store_failures_as: view my_subfolder_2: +store_failures_as: ephemeral ``` ##### "Clobbering" configs As with most other configurations, `store_failures_as` is "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Additional resources: * [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md#related-documentation) * [Data test-specific configurations](https://docs.getdbt.com/reference/data-test-configs.md#test-data-specific-configurations) * [Configuring directories of models in dbt\_project.yml](https://docs.getdbt.com/reference/model-configs.md#configuring-directories-of-models-in-dbt_projectyml) * [Config inheritance](https://docs.getdbt.com/reference/define-configs.md#config-inheritance) --- ### strategy ##### timestamp (Applies to dbt v1.9 and later) snapshots/\<filename>.yml ```yaml snapshots: - name: snapshot_name: relation: source('my_source', 'my_table') config: strategy: timestamp updated_at: column_name ``` dbt\_project.yml ```yml snapshots: <resource-path>: +strategy: timestamp +updated_at: column_name ``` ##### check (Applies to dbt v1.9 and later) snapshots/\<filename>.yml ```yaml snapshots: - name: snapshot_name: relation: source('my_source', 'my_table') config: strategy: check check_cols: [column_name] | "all" ``` dbt\_project.yml ```yml snapshots: <resource-path>: +strategy: check +check_cols: [column_name] | all ``` #### Description The snapshot strategy dbt should use to detect record changes. Read the guide to [snapshots](https://docs.getdbt.com/docs/build/snapshots.md#detecting-row-changes) to understand the differences between the two. #### Default This is a **required configuration**. There is no default value. #### Examples ##### Use the timestamp strategy (Applies to dbt v1.9 and later) snapshots/timestamp\_example.yml ```yaml snapshots: - name: orders_snapshot_timestamp relation: source('jaffle_shop', 'orders') config: schema: snapshots strategy: timestamp unique_key: id updated_at: updated_at ``` ##### Use the check strategy (Applies to dbt v1.9 and later) snapshots/check\_example.yml ```yaml snapshots: - name: orders_snapshot_check relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: check check_cols: - status - is_cancelled ``` ##### Advanced: define and use custom snapshot strategy Behind the scenes, snapshot strategies are implemented as macros, named `snapshot_<strategy>_strategy` * [Source code](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/snapshots/strategies.sql#L52) for the timestamp strategy * [Source code](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/snapshots/strategies.sql#L136) for the check strategy It's possible to implement your own snapshot strategy by adding a macro with the same naming pattern to your project. For example, you might choose to create a strategy which records hard deletes, named `timestamp_with_deletes`. 1. Create a macro named `snapshot_timestamp_with_deletes_strategy`. Use the existing code as a guide and adjust as needed. 2. Use this strategy via the `strategy` configuration: (Applies to dbt v1.9 and later) snapshots/\<filename>.yml ```yaml snapshots: - name: my_custom_snapshot relation: source('my_source', 'my_table') config: strategy: timestamp_with_deletes updated_at: updated_at_column unique_key: id ``` --- ### Supported BigQuery functions in dbt v2 dbt v2 can validate that your BigQuery function arguments match expected types directly in dbt platform, the CLI, or using the dbt VS Code extension. To enable this, set [`static_analysis: strict`](https://docs.getdbt.com/docs/build/about-static-analysis.md?version=2.0) in your project configuration. The following table lists *every* BigQuery built-in function and its typechecking support status. Use the table's search and filters to find specific functions. Refreshed weekly This table is updated weekly from the [BigQuery SQL function reference](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/functions-all) and cross-referenced with dbt v2's support list. If you spot a discrepancy, we'd love for you to contribute and [open an issue](https://github.com/dbt-labs/dbt/issues) in the dbt v1 repository! | Function | Category | v2 typechecking | | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | --------------- | | [ABS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#abs) | Built-in | ✓ | | [ACOS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#acos) | Built-in | ✓ | | [ACOSH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#acosh) | Built-in | ✓ | | [AEAD.DECRYPT\_BYTES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#aeaddecrypt_bytes) | Built-in | ✓ | | [AEAD.DECRYPT\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#aeaddecrypt_string) | Built-in | ✓ | | [AEAD.ENCRYPT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#aeadencrypt) | Built-in | ✓ | | [AGG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#agg) | Built-in | — | | [ANY\_VALUE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#any_value) | Built-in | ✓ | | [APPENDS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time-series-functions#appends) | Built-in | — | | [APPROX\_COUNT\_DISTINCT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/approximate_aggregate_functions#approx_count_distinct) | Built-in | ✓ | | [APPROX\_QUANTILES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/approximate_aggregate_functions#approx_quantiles) | Built-in | ✓ | | [APPROX\_TOP\_COUNT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/approximate_aggregate_functions#approx_top_count) | Built-in | ✓ | | [APPROX\_TOP\_SUM](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/approximate_aggregate_functions#approx_top_sum) | Built-in | ✓ | | [ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array) | Built-in | — | | [ARRAY\_AGG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#array_agg) | Built-in | ✓ | | [ARRAY\_CONCAT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array_concat) | Built-in | ✓ | | [ARRAY\_CONCAT\_AGG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#array_concat_agg) | Built-in | ✓ | | [ARRAY\_FIRST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array_first) | Built-in | ✓ | | [ARRAY\_LAST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array_last) | Built-in | ✓ | | [ARRAY\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array_length) | Built-in | ✓ | | [ARRAY\_REVERSE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array_reverse) | Built-in | ✓ | | [ARRAY\_SLICE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array_slice) | Built-in | ✓ | | [ARRAY\_TO\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#array_to_string) | Built-in | ✓ | | [ASCII](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#ascii) | Built-in | ✓ | | [ASIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#asin) | Built-in | ✓ | | [ASINH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#asinh) | Built-in | ✓ | | [ATAN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#atan) | Built-in | ✓ | | [ATAN2](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#atan2) | Built-in | ✓ | | [ATANH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#atanh) | Built-in | ✓ | | [AVG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#avg) | Built-in | ✓ | | [AVG (Differential Privacy)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate-dp-functions#dp_avg) | Built-in | ✓ | | [BAG\_OF\_WORDS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/text-analysis-functions#bag_of_words) | Built-in | — | | [BIT\_AND](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_and) | Built-in | ✓ | | [BIT\_COUNT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bit_functions#bit_count) | Built-in | ✓ | | [BIT\_OR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_or) | Built-in | ✓ | | [BIT\_XOR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_xor) | Built-in | ✓ | | [BOOL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#bool_for_json) | Built-in | ✓ | | [BYTE\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#byte_length) | Built-in | ✓ | | [CAST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/conversion_functions#cast) | Built-in | — | | [CBRT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#cbrt) | Built-in | ✓ | | [CEIL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#ceil) | Built-in | ✓ | | [CEILING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#ceiling) | Built-in | — | | [CHANGES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time-series-functions#changes) | Built-in | — | | [CHAR\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#char_length) | Built-in | ✓ | | [CHARACTER\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#character_length) | Built-in | ✓ | | [CHR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#chr) | Built-in | ✓ | | [CODE\_POINTS\_TO\_BYTES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#code_points_to_bytes) | Built-in | ✓ | | [CODE\_POINTS\_TO\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#code_points_to_string) | Built-in | ✓ | | [COLLATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#collate) | Built-in | ✓ | | [CONCAT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#concat) | Built-in | ✓ | | [CONTAINS\_SUBSTR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#contains_substr) | Built-in | ✓ | | [CORR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#corr) | Built-in | ✓ | | [COS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#cos) | Built-in | ✓ | | [COSH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#cosh) | Built-in | ✓ | | [COSINE\_DISTANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#cosine_distance) | Built-in | — | | [COT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#cot) | Built-in | ✓ | | [COTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#coth) | Built-in | ✓ | | [COUNT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#count) | Built-in | ✓ | | [COUNT (Differential Privacy)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate-dp-functions#dp_count) | Built-in | ✓ | | [COUNTIF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#countif) | Built-in | ✓ | | [COVAR\_POP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#covar_pop) | Built-in | ✓ | | [COVAR\_SAMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#covar_samp) | Built-in | ✓ | | [CSC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#csc) | Built-in | ✓ | | [CSCH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#csch) | Built-in | ✓ | | [CUME\_DIST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#cume_dist) | Built-in | ✓ | | [CURRENT\_DATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#current_date) | Built-in | ✓ | | [CURRENT\_DATETIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#current_datetime) | Built-in | ✓ | | [CURRENT\_TIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#current_time) | Built-in | ✓ | | [CURRENT\_TIMESTAMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#current_timestamp) | Built-in | ✓ | | [DATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date) | Built-in | ✓ | | [DATE\_ADD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date_add) | Built-in | ✓ | | [DATE\_BUCKET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time-series-functions#date_bucket) | Built-in | ✓ | | [DATE\_DIFF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date_diff) | Built-in | ✓ | | [DATE\_FROM\_UNIX\_DATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date_from_unix_date) | Built-in | ✓ | | [DATE\_SUB](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date_sub) | Built-in | ✓ | | [DATE\_TRUNC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date_trunc) | Built-in | ✓ | | [DATETIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#datetime) | Built-in | ✓ | | [DATETIME\_ADD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#datetime_add) | Built-in | ✓ | | [DATETIME\_BUCKET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time-series-functions#datetime_bucket) | Built-in | ✓ | | [DATETIME\_DIFF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#datetime_diff) | Built-in | ✓ | | [DATETIME\_SUB](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#datetime_sub) | Built-in | ✓ | | [DATETIME\_TRUNC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#datetime_trunc) | Built-in | ✓ | | [DENSE\_RANK](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#dense_rank) | Built-in | ✓ | | [DESTINATION\_NODE\_ID](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#destination_node_id) | Built-in | — | | [DETERMINISTIC\_DECRYPT\_BYTES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#deterministic_decrypt_bytes) | Built-in | ✓ | | [DETERMINISTIC\_DECRYPT\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#deterministic_decrypt_string) | Built-in | ✓ | | [DETERMINISTIC\_ENCRYPT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#deterministic_encrypt) | Built-in | ✓ | | [DIV](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#div) | Built-in | ✓ | | [DLP\_DETERMINISTIC\_DECRYPT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/dlp_functions#dlp_deterministic_decrypt) | Built-in | — | | [DLP\_DETERMINISTIC\_ENCRYPT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/dlp_functions#dlp_deterministic_encrypt) | Built-in | — | | [DLP\_KEY\_CHAIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/dlp_functions#dlp_key_chain) | Built-in | — | | [EDGES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#edges) | Built-in | — | | [EDIT\_DISTANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#edit_distance) | Built-in | ✓ | | [ELEMENT\_ID](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#element_id) | Built-in | — | | [ENDS\_WITH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#ends_with) | Built-in | ✓ | | [ERROR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/debugging_functions#error) | Built-in | ✓ | | [EUCLIDEAN\_DISTANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#euclidean_distance) | Built-in | — | | [EXP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#exp) | Built-in | ✓ | | [EXTERNAL\_OBJECT\_TRANSFORM](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/table-functions-built-in#external_object_transform) | Built-in | — | | [EXTERNAL\_QUERY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/federated_query_functions#external_query) | Built-in | ✓ | | [EXTRACT (Date)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#extract) | Built-in | — | | [EXTRACT (Datetime)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#extract) | Built-in | — | | [EXTRACT (Interval)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/interval_functions#extract) | Built-in | — | | [EXTRACT (Time)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#extract) | Built-in | — | | [EXTRACT (Timestamp)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#extract) | Built-in | — | | [FARM\_FINGERPRINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hash_functions#farm_fingerprint) | Built-in | ✓ | | [FIRST\_VALUE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions#first_value) | Built-in | ✓ | | [FLOAT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#double_for_json) | Built-in | ✓ | | [FLOOR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#floor) | Built-in | ✓ | | [FORMAT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#format_string) | Built-in | ✓ | | [FORMAT\_DATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#format_date) | Built-in | ✓ | | [FORMAT\_DATETIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#format_datetime) | Built-in | ✓ | | [FORMAT\_TIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#format_time) | Built-in | ✓ | | [FORMAT\_TIMESTAMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#format_timestamp) | Built-in | ✓ | | [FROM\_BASE32](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#from_base32) | Built-in | ✓ | | [FROM\_BASE64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#from_base64) | Built-in | ✓ | | [FROM\_HEX](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#from_hex) | Built-in | ✓ | | [GAP\_FILL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time-series-functions#gap_fill) | Built-in | ✓ | | [GENERATE\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#generate_array) | Built-in | ✓ | | [GENERATE\_DATE\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#generate_date_array) | Built-in | ✓ | | [GENERATE\_RANGE\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#generate_range_array) | Built-in | — | | [GENERATE\_TIMESTAMP\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#generate_timestamp_array) | Built-in | ✓ | | [GENERATE\_UUID](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/utility-functions#generate_uuid) | Built-in | ✓ | | [GREATEST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#greatest) | Built-in | ✓ | | [GROUPING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#grouping) | Built-in | ✓ | | [HLL\_COUNT.EXTRACT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions#hll_countextract) | Built-in | ✓ | | [HLL\_COUNT.INIT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions#hll_countinit) | Built-in | ✓ | | [HLL\_COUNT.MERGE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions#hll_countmerge) | Built-in | ✓ | | [HLL\_COUNT.MERGE\_PARTIAL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions#hll_countmerge_partial) | Built-in | ✓ | | [IEEE\_DIVIDE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#ieee_divide) | Built-in | ✓ | | [INITCAP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#initcap) | Built-in | ✓ | | [INSTR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#instr) | Built-in | ✓ | | [INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#int64_for_json) | Built-in | ✓ | | [IS\_ACYCLIC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#is_acyclic) | Built-in | — | | [IS\_INF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#is_inf) | Built-in | ✓ | | [IS\_NAN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#is_nan) | Built-in | ✓ | | [IS\_SIMPLE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#is_simple) | Built-in | — | | [IS\_TRAIL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#is_trail) | Built-in | — | | [JSON\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_array) | Built-in | ✓ | | [JSON\_ARRAY\_APPEND](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_array_append) | Built-in | ✓ | | [JSON\_ARRAY\_INSERT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_array_insert) | Built-in | ✓ | | [JSON\_EXTRACT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_extract) | Built-in | — | | [JSON\_EXTRACT\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_extract_array) | Built-in | ✓ | | [JSON\_EXTRACT\_SCALAR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_extract_scalar) | Built-in | ✓ | | [JSON\_EXTRACT\_STRING\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_extract_string_array) | Built-in | ✓ | | [JSON\_FLATTEN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_flatten) | Built-in | ✓ | | [JSON\_KEYS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_keys) | Built-in | ✓ | | [JSON\_OBJECT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_object) | Built-in | ✓ | | [JSON\_QUERY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_query) | Built-in | ✓ | | [JSON\_QUERY\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_query_array) | Built-in | ✓ | | [JSON\_REMOVE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_remove) | Built-in | ✓ | | [JSON\_SET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_set) | Built-in | ✓ | | [JSON\_STRIP\_NULLS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_strip_nulls) | Built-in | ✓ | | [JSON\_TYPE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_type) | Built-in | ✓ | | [JSON\_VALUE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_value) | Built-in | ✓ | | [JSON\_VALUE\_ARRAY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#json_value_array) | Built-in | ✓ | | [JUSTIFY\_DAYS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/interval_functions#justify_days) | Built-in | ✓ | | [JUSTIFY\_HOURS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/interval_functions#justify_hours) | Built-in | ✓ | | [JUSTIFY\_INTERVAL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/interval_functions#justify_interval) | Built-in | ✓ | | [KEYS.ADD\_KEY\_FROM\_RAW\_BYTES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keysadd_key_from_raw_bytes) | Built-in | ✓ | | [KEYS.KEYSET\_CHAIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keyskeyset_chain) | Built-in | ✓ | | [KEYS.KEYSET\_FROM\_JSON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keyskeyset_from_json) | Built-in | ✓ | | [KEYS.KEYSET\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keyskeyset_length) | Built-in | ✓ | | [KEYS.KEYSET\_TO\_JSON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keyskeyset_to_json) | Built-in | ✓ | | [KEYS.NEW\_KEYSET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keysnew_keyset) | Built-in | ✓ | | [KEYS.NEW\_WRAPPED\_KEYSET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keysnew_wrapped_keyset) | Built-in | ✓ | | [KEYS.REWRAP\_KEYSET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keysrewrap_keyset) | Built-in | ✓ | | [KEYS.ROTATE\_KEYSET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keysrotate_keyset) | Built-in | ✓ | | [KEYS.ROTATE\_WRAPPED\_KEYSET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aead_encryption_functions#keysrotate_wrapped_keyset) | Built-in | ✓ | | [KLL\_QUANTILES.EXTRACT\_FLOAT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesextract_double) | Built-in | — | | [KLL\_QUANTILES.EXTRACT\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesextract_int64) | Built-in | — | | [KLL\_QUANTILES.EXTRACT\_POINT\_FLOAT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesextract_point_double) | Built-in | — | | [KLL\_QUANTILES.EXTRACT\_POINT\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesextract_point_int64) | Built-in | — | | [KLL\_QUANTILES.INIT\_FLOAT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesinit_double) | Built-in | — | | [KLL\_QUANTILES.INIT\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesinit_int64) | Built-in | — | | [KLL\_QUANTILES.MERGE\_FLOAT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesmerge_double) | Built-in | — | | [KLL\_QUANTILES.MERGE\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesmerge_int64) | Built-in | — | | [KLL\_QUANTILES.MERGE\_PARTIAL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesmerge_partial) | Built-in | — | | [KLL\_QUANTILES.MERGE\_POINT\_FLOAT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesmerge_point_double) | Built-in | — | | [KLL\_QUANTILES.MERGE\_POINT\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/kll_functions#kll_quantilesmerge_point_int64) | Built-in | — | | [LABELS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#labels) | Built-in | — | | [LAG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions#lag) | Built-in | ✓ | | [LAST\_DAY (Date)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#last_day) | Built-in | ✓ | | [LAST\_DAY (Datetime)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#last_day) | Built-in | ✓ | | [LAST\_VALUE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions#last_value) | Built-in | ✓ | | [LAX\_BOOL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#lax_bool) | Built-in | ✓ | | [LAX\_FLOAT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#lax_double) | Built-in | ✓ | | [LAX\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#lax_int64) | Built-in | ✓ | | [LAX\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#lax_string) | Built-in | ✓ | | [LEAD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions#lead) | Built-in | ✓ | | [LEAST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#least) | Built-in | ✓ | | [LEFT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#left) | Built-in | ✓ | | [LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#length) | Built-in | ✓ | | [LN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#ln) | Built-in | ✓ | | [LOG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#log) | Built-in | ✓ | | [LOG10](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#log10) | Built-in | ✓ | | [LOGICAL\_AND](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#logical_and) | Built-in | ✓ | | [LOGICAL\_OR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#logical_or) | Built-in | ✓ | | [LOWER](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#lower) | Built-in | ✓ | | [LPAD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#lpad) | Built-in | ✓ | | [LTRIM](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#ltrim) | Built-in | ✓ | | [MAKE\_INTERVAL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/interval_functions#make_interval) | Built-in | ✓ | | [MAX](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#max) | Built-in | ✓ | | [MAX\_BY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#max_by) | Built-in | ✓ | | [MD5](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hash_functions#md5) | Built-in | ✓ | | [MIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#min) | Built-in | ✓ | | [MIN\_BY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#min_by) | Built-in | ✓ | | [MOD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#mod) | Built-in | ✓ | | [NET.HOST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#nethost) | Built-in | ✓ | | [NET.IP\_FROM\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netip_from_string) | Built-in | ✓ | | [NET.IP\_NET\_MASK](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netip_net_mask) | Built-in | ✓ | | [NET.IP\_TO\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netip_to_string) | Built-in | ✓ | | [NET.IP\_TRUNC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netip_trunc) | Built-in | ✓ | | [NET.IPV4\_FROM\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netipv4_from_int64) | Built-in | ✓ | | [NET.IPV4\_TO\_INT64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netipv4_to_int64) | Built-in | ✓ | | [NET.PUBLIC\_SUFFIX](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netpublic_suffix) | Built-in | ✓ | | [NET.REG\_DOMAIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netreg_domain) | Built-in | ✓ | | [NET.SAFE\_IP\_FROM\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/net_functions#netsafe_ip_from_string) | Built-in | ✓ | | [NODES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#nodes) | Built-in | — | | [NORMALIZE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#normalize) | Built-in | — | | [NORMALIZE\_AND\_CASEFOLD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#normalize_and_casefold) | Built-in | ✓ | | [NTH\_VALUE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions#nth_value) | Built-in | ✓ | | [NTILE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#ntile) | Built-in | ✓ | | [OBJ.FETCH\_METADATA](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions#objfetch_metadata) | Built-in | — | | [OBJ.GET\_ACCESS\_URL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions#objget_access_url) | Built-in | — | | [OBJ.GET\_READ\_URL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions#objget_read_url) | Built-in | — | | [OBJ.LIST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions#objlist) | Built-in | — | | [OBJ.MAKE\_REF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions#objmake_ref) | Built-in | — | | [OCTET\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#octet_length) | Built-in | ✓ | | [PARSE\_BIGNUMERIC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/conversion_functions#parse_bignumeric) | Built-in | ✓ | | [PARSE\_DATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#parse_date) | Built-in | ✓ | | [PARSE\_DATETIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#parse_datetime) | Built-in | ✓ | | [PARSE\_JSON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#parse_json) | Built-in | ✓ | | [PARSE\_NUMERIC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/conversion_functions#parse_numeric) | Built-in | ✓ | | [PARSE\_TIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#parse_time) | Built-in | ✓ | | [PARSE\_TIMESTAMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#parse_timestamp) | Built-in | ✓ | | [PATH\_FIRST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#path_first) | Built-in | — | | [PATH\_LAST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#path_last) | Built-in | — | | [PATH\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#path_length) | Built-in | — | | [PERCENT\_RANK](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#percent_rank) | Built-in | ✓ | | [PERCENTILE\_CONT (Differential Privacy)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate-dp-functions#dp_percentile_cont) | Built-in | ✓ | | [PERCENTILE\_CONT (Navigation)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions#percentile_cont) | Built-in | ✓ | | [PERCENTILE\_DISC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/navigation_functions#percentile_disc) | Built-in | ✓ | | [POW](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#pow) | Built-in | ✓ | | [POWER](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#power) | Built-in | ✓ | | [RAND](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#rand) | Built-in | ✓ | | [RANGE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#range) | Built-in | ✓ | | [RANGE\_BUCKET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#range_bucket) | Built-in | ✓ | | [RANGE\_CONTAINS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#range_contains) | Built-in | — | | [RANGE\_END](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#range_end) | Built-in | — | | [RANGE\_INTERSECT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#range_intersect) | Built-in | — | | [RANGE\_OVERLAPS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#range_overlaps) | Built-in | — | | [RANGE\_SESSIONIZE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#range_sessionize) | Built-in | — | | [RANGE\_START](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/range-functions#range_start) | Built-in | — | | [RANK](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#rank) | Built-in | ✓ | | [REGEXP\_CONTAINS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_contains) | Built-in | ✓ | | [REGEXP\_EXTRACT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_extract) | Built-in | ✓ | | [REGEXP\_EXTRACT\_ALL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_extract_all) | Built-in | ✓ | | [REGEXP\_INSTR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_instr) | Built-in | ✓ | | [REGEXP\_REPLACE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_replace) | Built-in | ✓ | | [REGEXP\_SUBSTR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_substr) | Built-in | ✓ | | [REPEAT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#repeat) | Built-in | ✓ | | [REPLACE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#replace) | Built-in | ✓ | | [REVERSE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#reverse) | Built-in | ✓ | | [RIGHT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#right) | Built-in | ✓ | | [ROUND](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#round) | Built-in | ✓ | | [ROW\_NUMBER](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#row_number) | Built-in | ✓ | | [RPAD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#rpad) | Built-in | ✓ | | [RTRIM](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#rtrim) | Built-in | ✓ | | [S2\_CELLIDFROMPOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#s2_cellidfrompoint) | Built-in | ✓ | | [S2\_COVERINGCELLIDS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#s2_coveringcellids) | Built-in | ✓ | | [SAFE\_ADD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#safe_add) | Built-in | ✓ | | [SAFE\_CAST](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/conversion_functions#safe_casting) | Built-in | — | | [SAFE\_CONVERT\_BYTES\_TO\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#safe_convert_bytes_to_string) | Built-in | ✓ | | [SAFE\_DIVIDE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#safe_divide) | Built-in | ✓ | | [SAFE\_MULTIPLY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#safe_multiply) | Built-in | ✓ | | [SAFE\_NEGATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#safe_negate) | Built-in | ✓ | | [SAFE\_SUBTRACT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#safe_subtract) | Built-in | ✓ | | [SEARCH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#search) | Built-in | ✓ | | [SEC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#sec) | Built-in | ✓ | | [SECH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#sech) | Built-in | ✓ | | [SESSION\_USER](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/security_functions#session_user) | Built-in | ✓ | | [SHA1](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hash_functions#sha1) | Built-in | ✓ | | [SHA256](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hash_functions#sha256) | Built-in | ✓ | | [SHA512](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/hash_functions#sha512) | Built-in | ✓ | | [SIGN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#sign) | Built-in | ✓ | | [SIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#sin) | Built-in | ✓ | | [SINH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#sinh) | Built-in | ✓ | | [SOUNDEX](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#soundex) | Built-in | ✓ | | [SOURCE\_NODE\_ID](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/graph-sql-functions#source_node_id) | Built-in | — | | [SPLIT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#split) | Built-in | ✓ | | [SQRT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#sqrt) | Built-in | ✓ | | [ST\_ANGLE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_angle) | Built-in | ✓ | | [ST\_AREA](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_area) | Built-in | ✓ | | [ST\_ASBINARY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_asbinary) | Built-in | ✓ | | [ST\_ASGEOJSON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_asgeojson) | Built-in | ✓ | | [ST\_ASTEXT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_astext) | Built-in | ✓ | | [ST\_AZIMUTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_azimuth) | Built-in | ✓ | | [ST\_BOUNDARY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_boundary) | Built-in | ✓ | | [ST\_BOUNDINGBOX](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_boundingbox) | Built-in | ✓ | | [ST\_BUFFER](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_buffer) | Built-in | ✓ | | [ST\_BUFFERWITHTOLERANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_bufferwithtolerance) | Built-in | ✓ | | [ST\_CENTROID](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_centroid) | Built-in | ✓ | | [ST\_CENTROID\_AGG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_centroid_agg) | Built-in | ✓ | | [ST\_CLOSESTPOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_closestpoint) | Built-in | ✓ | | [ST\_CLUSTERDBSCAN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_clusterdbscan) | Built-in | ✓ | | [ST\_CONTAINS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_contains) | Built-in | ✓ | | [ST\_CONVEXHULL](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_convexhull) | Built-in | ✓ | | [ST\_COVEREDBY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_coveredby) | Built-in | ✓ | | [ST\_COVERS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_covers) | Built-in | ✓ | | [ST\_DIFFERENCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_difference) | Built-in | ✓ | | [ST\_DIMENSION](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_dimension) | Built-in | ✓ | | [ST\_DISJOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_disjoint) | Built-in | ✓ | | [ST\_DISTANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_distance) | Built-in | ✓ | | [ST\_DUMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_dump) | Built-in | ✓ | | [ST\_DWITHIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_dwithin) | Built-in | ✓ | | [ST\_ENDPOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_endpoint) | Built-in | ✓ | | [ST\_EQUALS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_equals) | Built-in | ✓ | | [ST\_EXTENT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_extent) | Built-in | — | | [ST\_EXTERIORRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_exteriorring) | Built-in | ✓ | | [ST\_GEOGFROM](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geogfrom) | Built-in | ✓ | | [ST\_GEOGFROMGEOJSON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geogfromgeojson) | Built-in | ✓ | | [ST\_GEOGFROMTEXT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geogfromtext) | Built-in | ✓ | | [ST\_GEOGFROMWKB](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geogfromwkb) | Built-in | ✓ | | [ST\_GEOGPOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geogpoint) | Built-in | ✓ | | [ST\_GEOGPOINTFROMGEOHASH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geogpointfromgeohash) | Built-in | ✓ | | [ST\_GEOHASH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geohash) | Built-in | ✓ | | [ST\_GEOMETRYTYPE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_geometrytype) | Built-in | ✓ | | [ST\_HAUSDORFFDISTANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_hausdorffdistance) | Built-in | — | | [ST\_HAUSDORFFDWITHIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_hausdorffdwithin) | Built-in | — | | [ST\_INTERIORRINGS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_interiorrings) | Built-in | ✓ | | [ST\_INTERSECTION](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_intersection) | Built-in | ✓ | | [ST\_INTERSECTS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_intersects) | Built-in | ✓ | | [ST\_INTERSECTSBOX](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_intersectsbox) | Built-in | ✓ | | [ST\_ISCLOSED](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_isclosed) | Built-in | ✓ | | [ST\_ISCOLLECTION](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_iscollection) | Built-in | ✓ | | [ST\_ISEMPTY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_isempty) | Built-in | ✓ | | [ST\_ISRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_isring) | Built-in | ✓ | | [ST\_LENGTH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_length) | Built-in | ✓ | | [ST\_LINEINTERPOLATEPOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_lineinterpolatepoint) | Built-in | — | | [ST\_LINELOCATEPOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_linelocatepoint) | Built-in | ✓ | | [ST\_LINESUBSTRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_linesubstring) | Built-in | — | | [ST\_MAKELINE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_makeline) | Built-in | ✓ | | [ST\_MAKEPOLYGON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_makepolygon) | Built-in | ✓ | | [ST\_MAKEPOLYGONORIENTED](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_makepolygonoriented) | Built-in | ✓ | | [ST\_MAXDISTANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_maxdistance) | Built-in | ✓ | | [ST\_NPOINTS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_npoints) | Built-in | ✓ | | [ST\_NUMGEOMETRIES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_numgeometries) | Built-in | ✓ | | [ST\_NUMPOINTS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_numpoints) | Built-in | ✓ | | [ST\_PERIMETER](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_perimeter) | Built-in | ✓ | | [ST\_POINTN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_pointn) | Built-in | ✓ | | [ST\_REGIONSTATS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_regionstats) | Built-in | — | | [ST\_SIMPLIFY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_simplify) | Built-in | ✓ | | [ST\_SNAPTOGRID](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_snaptogrid) | Built-in | ✓ | | [ST\_STARTPOINT](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_startpoint) | Built-in | ✓ | | [ST\_TOUCHES](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_touches) | Built-in | ✓ | | [ST\_UNION](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_union) | Built-in | ✓ | | [ST\_UNION\_AGG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_union_agg) | Built-in | ✓ | | [ST\_WITHIN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_within) | Built-in | ✓ | | [ST\_X](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_x) | Built-in | ✓ | | [ST\_Y](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions#st_y) | Built-in | ✓ | | [STARTS\_WITH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#starts_with) | Built-in | ✓ | | [STDDEV](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#stddev) | Built-in | ✓ | | [STDDEV\_POP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#stddev_pop) | Built-in | ✓ | | [STDDEV\_SAMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#stddev_samp) | Built-in | ✓ | | [STRING (JSON)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#string_for_json) | Built-in | ✓ | | [STRING (Timestamp)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#string) | Built-in | ✓ | | [STRING\_AGG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#string_agg) | Built-in | ✓ | | [STRPOS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#strpos) | Built-in | ✓ | | [SUBSTR](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#substr) | Built-in | ✓ | | [SUBSTRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#substring) | Built-in | — | | [SUM](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#sum) | Built-in | ✓ | | [SUM (Differential Privacy)](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/aggregate-dp-functions#dp_sum) | Built-in | ✓ | | [TAN](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#tan) | Built-in | ✓ | | [TANH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#tanh) | Built-in | ✓ | | [TEXT\_ANALYZE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/text-analysis-functions#text_analyze) | Built-in | — | | [TF\_IDF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/text-analysis-functions#tf_idf) | Built-in | — | | [TIME](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#time) | Built-in | ✓ | | [TIME\_ADD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#time_add) | Built-in | ✓ | | [TIME\_DIFF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#time_diff) | Built-in | ✓ | | [TIME\_SUB](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#time_sub) | Built-in | ✓ | | [TIME\_TRUNC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time_functions#time_trunc) | Built-in | ✓ | | [TIMESTAMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp) | Built-in | ✓ | | [TIMESTAMP\_ADD](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_add) | Built-in | ✓ | | [TIMESTAMP\_BUCKET](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/time-series-functions#timestamp_bucket) | Built-in | ✓ | | [TIMESTAMP\_DIFF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_diff) | Built-in | ✓ | | [TIMESTAMP\_MICROS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_micros) | Built-in | ✓ | | [TIMESTAMP\_MILLIS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_millis) | Built-in | ✓ | | [TIMESTAMP\_SECONDS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_seconds) | Built-in | ✓ | | [TIMESTAMP\_SUB](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_sub) | Built-in | ✓ | | [TIMESTAMP\_TRUNC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_trunc) | Built-in | ✓ | | [TO\_BASE32](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#to_base32) | Built-in | ✓ | | [TO\_BASE64](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#to_base64) | Built-in | ✓ | | [TO\_CODE\_POINTS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#to_code_points) | Built-in | ✓ | | [TO\_HEX](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#to_hex) | Built-in | ✓ | | [TO\_JSON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#to_json) | Built-in | ✓ | | [TO\_JSON\_STRING](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/json_functions#to_json_string) | Built-in | ✓ | | [TRANSLATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#translate) | Built-in | ✓ | | [TRIM](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#trim) | Built-in | ✓ | | [TRUNC](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/mathematical_functions#trunc) | Built-in | ✓ | | [TYPEOF](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/utility-functions#typeof) | Built-in | ✓ | | [UNICODE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#unicode) | Built-in | ✓ | | [UNIX\_DATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#unix_date) | Built-in | ✓ | | [UNIX\_MICROS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#unix_micros) | Built-in | ✓ | | [UNIX\_MILLIS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#unix_millis) | Built-in | ✓ | | [UNIX\_SECONDS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#unix_seconds) | Built-in | ✓ | | [UPPER](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#upper) | Built-in | ✓ | | [VAR\_POP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#var_pop) | Built-in | ✓ | | [VAR\_SAMP](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#var_samp) | Built-in | ✓ | | [VARIANCE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#variance) | Built-in | ✓ | | [VECTOR\_INDEX.STATISTICS](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/vectorindex_functions#vector_indexstatistics) | Built-in | — | | [VECTOR\_SEARCH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#vector_search) | Built-in | — | --- ### Supported data formats for unit tests Currently, mock data for unit testing in dbt supports three formats: * `dict` (default): Inline dictionary values. * `csv`: Inline CSV values or a CSV file. * `sql`: Inline SQL query or a SQL file. Note: For this format you must supply mock data for *all columns*. #### dict The `dict` data format is the default if no `format` is defined. `dict` requires an inline dictionary for `rows`: models/schema.yml ```yml unit_tests: - name: test_my_model model: my_model given: - input: ref('my_model_a') format: dict rows: - {id: 1, name: gerda} - {id: 2, name: michelle} ``` #### CSV When using the `csv` format, you can use either an inline CSV string for `rows`: models/schema.yml ```yml unit_tests: - name: test_my_model model: my_model given: - input: ref('my_model_a') format: csv rows: | id,name 1,gerda 2,michelle ``` Or, you can provide the name of a CSV file in the `tests/fixtures` directory (or the configured `test-paths` location) of your project for `fixture`: models/schema.yml ```yml unit_tests: - name: test_my_model model: my_model given: - input: ref('my_model_a') format: csv fixture: my_model_a_fixture ``` #### sql Using this format: * Provides more flexibility for the types of data you can unit test * Allows you to unit test a model that depends on an ephemeral model However, when using `format: sql` you must supply mock data for *all rows*. When using the `sql` format, you can use either an inline SQL query for `rows`: models/schema.yml ```yml unit_tests: - name: test_my_model model: my_model given: - input: ref('my_model_a') format: sql rows: | select 1 as id, 'gerda' as name, null as loaded_at union all select 2 as id, 'michelle', null as loaded_at as name ``` Or, you can provide the name of a SQL file in the `tests/fixtures` directory (or the configured `test-paths` location) of your project for `fixture`: models/schema.yml ```yml unit_tests: - name: test_my_model model: my_model given: - input: ref('my_model_a') format: sql fixture: my_model_a_fixture ``` **Note:** Jinja is unsupported in SQL fixtures for unit tests. --- ### Supported Snowflake functions in dbt v2 dbt v2 can validate that your Snowflake function arguments match expected types directly in dbt platform, the CLI, or using dbt VS Code extension. To enable this, set [`static_analysis: strict`](https://docs.getdbt.com/docs/build/about-static-analysis.md?version=2.0) in your project configuration. The following table lists *every* Snowflake built-in function and its typechecking support status. Use the table's search and filters to find specific functions. Refreshed weekly This table is updated weekly from the [Snowflake SQL function reference](https://docs.snowflake.com/en/sql-reference/functions-all) and cross-referenced with dbt v2's support list. If you spot a discrepancy, we'd love for you to contribute and [open an issue](https://github.com/dbt-labs/dbt/issues) in the dbt v1 repository! | Function | Category | v2 typechecking | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | [TAG\_REFERENCES\_WITH\_LINEAGE](https://docs.snowflake.com/en/sql-reference/functions/tag_references_with_lineage) | Account Usage table | ✓ | | [ACCUMULATE](https://docs.snowflake.com/en/sql-reference/functions/accumulate) | Aggregate | — | | [AGG](https://docs.snowflake.com/en/sql-reference/functions/agg) | Aggregate | — | | [AI\_AGG](https://docs.snowflake.com/en/sql-reference/functions/ai_agg) | Aggregate | — | | [AI\_SUMMARIZE\_AGG](https://docs.snowflake.com/en/sql-reference/functions/ai_summarize_agg) | Aggregate | — | | [ANY\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/any_value) | Aggregate | ✓ | | [APPROX\_COUNT\_DISTINCT](https://docs.snowflake.com/en/sql-reference/functions/approx_count_distinct) | Aggregate | ✓ | | [APPROX\_PERCENTILE](https://docs.snowflake.com/en/sql-reference/functions/approx_percentile) | Aggregate | ✓ | | [APPROX\_PERCENTILE\_ACCUMULATE](https://docs.snowflake.com/en/sql-reference/functions/approx_percentile_accumulate) | Aggregate | ✓ | | [APPROX\_PERCENTILE\_COMBINE](https://docs.snowflake.com/en/sql-reference/functions/approx_percentile_combine) | Aggregate | ✓ | | [APPROX\_PERCENTILE\_ESTIMATE](https://docs.snowflake.com/en/sql-reference/functions/approx_percentile_estimate) | Aggregate | ✓ | | [APPROX\_TOP\_K](https://docs.snowflake.com/en/sql-reference/functions/approx_top_k) | Aggregate | ✓ | | [APPROX\_TOP\_K\_ACCUMULATE](https://docs.snowflake.com/en/sql-reference/functions/approx_top_k_accumulate) | Aggregate | ✓ | | [APPROX\_TOP\_K\_COMBINE](https://docs.snowflake.com/en/sql-reference/functions/approx_top_k_combine) | Aggregate | ✓ | | [APPROX\_TOP\_K\_ESTIMATE](https://docs.snowflake.com/en/sql-reference/functions/approx_top_k_estimate) | Aggregate | ✓ | | [APPROXIMATE\_JACCARD\_INDEX](https://docs.snowflake.com/en/sql-reference/functions/approximate_jaccard_index) | Aggregate | ✓ | | [APPROXIMATE\_SIMILARITY](https://docs.snowflake.com/en/sql-reference/functions/approximate_similarity) | Aggregate | ✓ | | [ARRAY\_AGG](https://docs.snowflake.com/en/sql-reference/functions/array_agg) | Aggregate | ✓ | | [ARRAY\_UNION\_AGG](https://docs.snowflake.com/en/sql-reference/functions/array_union_agg) | Aggregate | ✓ | | [ARRAY\_UNIQUE\_AGG](https://docs.snowflake.com/en/sql-reference/functions/array_unique_agg) | Aggregate | ✓ | | [AVG](https://docs.snowflake.com/en/sql-reference/functions/avg) | Aggregate | ✓ | | [BITAND\_AGG](https://docs.snowflake.com/en/sql-reference/functions/bitand_agg) | Aggregate | ✓ | | [BITMAP\_ABSOLUTE\_POSITION](https://docs.snowflake.com/en/sql-reference/functions/bitmap_absolute_position) | Aggregate | ✓ | | [BITMAP\_AND](https://docs.snowflake.com/en/sql-reference/functions/bitmap_and) | Aggregate | — | | [BITMAP\_AND\_AGG](https://docs.snowflake.com/en/sql-reference/functions/bitmap_and_agg) | Aggregate | — | | [BITMAP\_BIT\_POSITION](https://docs.snowflake.com/en/sql-reference/functions/bitmap_bit_position) | Aggregate | ✓ | | [BITMAP\_BUCKET\_NUMBER](https://docs.snowflake.com/en/sql-reference/functions/bitmap_bucket_number) | Aggregate | ✓ | | [BITMAP\_CONSTRUCT\_AGG](https://docs.snowflake.com/en/sql-reference/functions/bitmap_construct_agg) | Aggregate | ✓ | | [BITMAP\_COUNT](https://docs.snowflake.com/en/sql-reference/functions/bitmap_count) | Aggregate | ✓ | | [BITMAP\_OR](https://docs.snowflake.com/en/sql-reference/functions/bitmap_or) | Aggregate | ✓ | | [BITMAP\_OR\_AGG](https://docs.snowflake.com/en/sql-reference/functions/bitmap_or_agg) | Aggregate | ✓ | | [BITMAP\_TO\_ARRAY](https://docs.snowflake.com/en/sql-reference/functions/bitmap_to_array) | Aggregate | ✓ | | [BITOR\_AGG](https://docs.snowflake.com/en/sql-reference/functions/bitor_agg) | Aggregate | ✓ | | [BITXOR\_AGG](https://docs.snowflake.com/en/sql-reference/functions/bitxor_agg) | Aggregate | ✓ | | [BOOLAND\_AGG](https://docs.snowflake.com/en/sql-reference/functions/booland_agg) | Aggregate | ✓ | | [BOOLOR\_AGG](https://docs.snowflake.com/en/sql-reference/functions/boolor_agg) | Aggregate | ✓ | | [BOOLXOR\_AGG](https://docs.snowflake.com/en/sql-reference/functions/boolxor_agg) | Aggregate | ✓ | | [CORR](https://docs.snowflake.com/en/sql-reference/functions/corr) | Aggregate | ✓ | | [COUNT](https://docs.snowflake.com/en/sql-reference/functions/count) | Aggregate | ✓ | | [COUNT\_IF](https://docs.snowflake.com/en/sql-reference/functions/count_if) | Aggregate | ✓ | | [COVAR\_POP](https://docs.snowflake.com/en/sql-reference/functions/covar_pop) | Aggregate | ✓ | | [COVAR\_SAMP](https://docs.snowflake.com/en/sql-reference/functions/covar_samp) | Aggregate | ✓ | | [DATASKETCHES\_HLL](https://docs.snowflake.com/en/sql-reference/functions/datasketches_hll) | Aggregate | — | | [DATASKETCHES\_HLL\_ACCUMULATE](https://docs.snowflake.com/en/sql-reference/functions/datasketches_hll_accumulate) | Aggregate | — | | [DATASKETCHES\_HLL\_COMBINE](https://docs.snowflake.com/en/sql-reference/functions/datasketches_hll_combine) | Aggregate | — | | [DATASKETCHES\_HLL\_ESTIMATE](https://docs.snowflake.com/en/sql-reference/functions/datasketches_hll_estimate) | Aggregate | — | | [GROUPING](https://docs.snowflake.com/en/sql-reference/functions/grouping) | Aggregate | ✓ | | [GROUPING\_ID](https://docs.snowflake.com/en/sql-reference/functions/grouping_id) | Aggregate | ✓ | | [HASH\_AGG](https://docs.snowflake.com/en/sql-reference/functions/hash_agg) | Aggregate | ✓ | | [HLL](https://docs.snowflake.com/en/sql-reference/functions/hll) | Aggregate | — | | [HLL\_ACCUMULATE](https://docs.snowflake.com/en/sql-reference/functions/hll_accumulate) | Aggregate | ✓ | | [HLL\_COMBINE](https://docs.snowflake.com/en/sql-reference/functions/hll_combine) | Aggregate | ✓ | | [HLL\_ESTIMATE](https://docs.snowflake.com/en/sql-reference/functions/hll_estimate) | Aggregate | ✓ | | [HLL\_EXPORT](https://docs.snowflake.com/en/sql-reference/functions/hll_export) | Aggregate | ✓ | | [HLL\_IMPORT](https://docs.snowflake.com/en/sql-reference/functions/hll_import) | Aggregate | ✓ | | [KURTOSIS](https://docs.snowflake.com/en/sql-reference/functions/kurtosis) | Aggregate | ✓ | | [LISTAGG](https://docs.snowflake.com/en/sql-reference/functions/listagg) | Aggregate | ✓ | | [MAX](https://docs.snowflake.com/en/sql-reference/functions/max) | Aggregate | ✓ | | [MAX\_BY](https://docs.snowflake.com/en/sql-reference/functions/max_by) | Aggregate | ✓ | | [MEDIAN](https://docs.snowflake.com/en/sql-reference/functions/median) | Aggregate | ✓ | | [MIN](https://docs.snowflake.com/en/sql-reference/functions/min) | Aggregate | ✓ | | [MIN\_BY](https://docs.snowflake.com/en/sql-reference/functions/min_by) | Aggregate | ✓ | | [MINHASH](https://docs.snowflake.com/en/sql-reference/functions/minhash) | Aggregate | ✓ | | [MINHASH\_COMBINE](https://docs.snowflake.com/en/sql-reference/functions/minhash_combine) | Aggregate | ✓ | | [MODE](https://docs.snowflake.com/en/sql-reference/functions/mode) | Aggregate | ✓ | | [OBJECT\_AGG](https://docs.snowflake.com/en/sql-reference/functions/object_agg) | Aggregate | ✓ | | [PERCENTILE\_CONT](https://docs.snowflake.com/en/sql-reference/functions/percentile_cont) | Aggregate | — | | [PERCENTILE\_DISC](https://docs.snowflake.com/en/sql-reference/functions/percentile_disc) | Aggregate | — | | [REGR\_AVGX](https://docs.snowflake.com/en/sql-reference/functions/regr_avgx) | Aggregate | ✓ | | [REGR\_AVGY](https://docs.snowflake.com/en/sql-reference/functions/regr_avgy) | Aggregate | ✓ | | [REGR\_COUNT](https://docs.snowflake.com/en/sql-reference/functions/regr_count) | Aggregate | ✓ | | [REGR\_INTERCEPT](https://docs.snowflake.com/en/sql-reference/functions/regr_intercept) | Aggregate | ✓ | | [REGR\_R2](https://docs.snowflake.com/en/sql-reference/functions/regr_r2) | Aggregate | ✓ | | [REGR\_SLOPE](https://docs.snowflake.com/en/sql-reference/functions/regr_slope) | Aggregate | ✓ | | [REGR\_SXX](https://docs.snowflake.com/en/sql-reference/functions/regr_sxx) | Aggregate | ✓ | | [REGR\_SXY](https://docs.snowflake.com/en/sql-reference/functions/regr_sxy) | Aggregate | ✓ | | [REGR\_SYY](https://docs.snowflake.com/en/sql-reference/functions/regr_syy) | Aggregate | ✓ | | [SKEW](https://docs.snowflake.com/en/sql-reference/functions/skew) | Aggregate | ✓ | | [STDDEV\_POP](https://docs.snowflake.com/en/sql-reference/functions/stddev_pop) | Aggregate | ✓ | | [STDDEV, STDDEV\_SAMP](https://docs.snowflake.com/en/sql-reference/functions/stddev) | Aggregate | ✓ | | [SUM](https://docs.snowflake.com/en/sql-reference/functions/sum) | Aggregate | ✓ | | [VAR\_POP](https://docs.snowflake.com/en/sql-reference/functions/var_pop) | Aggregate | ✓ | | [VAR\_SAMP](https://docs.snowflake.com/en/sql-reference/functions/var_samp) | Aggregate | ✓ | | [VARIANCE , VARIANCE\_SAMP](https://docs.snowflake.com/en/sql-reference/functions/variance) | Aggregate | ✓ | | [VARIANCE\_POP](https://docs.snowflake.com/en/sql-reference/functions/variance_pop) | Aggregate | — | | [BITAND](https://docs.snowflake.com/en/sql-reference/functions/bitand) | Bitwise expression | ✓ | | [BITNOT](https://docs.snowflake.com/en/sql-reference/functions/bitnot) | Bitwise expression | ✓ | | [BITOR](https://docs.snowflake.com/en/sql-reference/functions/bitor) | Bitwise expression | ✓ | | [BITSHIFTLEFT](https://docs.snowflake.com/en/sql-reference/functions/bitshiftleft) | Bitwise expression | ✓ | | [BITSHIFTRIGHT](https://docs.snowflake.com/en/sql-reference/functions/bitshiftright) | Bitwise expression | ✓ | | [BITXOR](https://docs.snowflake.com/en/sql-reference/functions/bitxor) | Bitwise expression | ✓ | | [GETBIT](https://docs.snowflake.com/en/sql-reference/functions/getbit) | Bitwise expression | ✓ | | [\[ NOT \] BETWEEN](https://docs.snowflake.com/en/sql-reference/functions/between) | Conditional expression | — | | [\[ NOT \] EQUAL\_NULL](https://docs.snowflake.com/en/sql-reference/functions/equal_null) | Conditional expression | ✓ | | [\[ NOT \] IN](https://docs.snowflake.com/en/sql-reference/functions/in) | Conditional expression | — | | [BOOLAND](https://docs.snowflake.com/en/sql-reference/functions/booland) | Conditional expression | ✓ | | [BOOLNOT](https://docs.snowflake.com/en/sql-reference/functions/boolnot) | Conditional expression | ✓ | | [BOOLOR](https://docs.snowflake.com/en/sql-reference/functions/boolor) | Conditional expression | ✓ | | [BOOLXOR](https://docs.snowflake.com/en/sql-reference/functions/boolxor) | Conditional expression | ✓ | | [CASE](https://docs.snowflake.com/en/sql-reference/functions/case) | Conditional expression | — | | [COALESCE](https://docs.snowflake.com/en/sql-reference/functions/coalesce) | Conditional expression | ✓ | | [DECODE](https://docs.snowflake.com/en/sql-reference/functions/decode) | Conditional expression | — | | [GREATEST](https://docs.snowflake.com/en/sql-reference/functions/greatest) | Conditional expression | ✓ | | [GREATEST\_IGNORE\_NULLS](https://docs.snowflake.com/en/sql-reference/functions/greatest_ignore_nulls) | Conditional expression | ✓ | | [IFF](https://docs.snowflake.com/en/sql-reference/functions/iff) | Conditional expression | ✓ | | [IFNULL](https://docs.snowflake.com/en/sql-reference/functions/ifnull) | Conditional expression | ✓ | | [IS \[ NOT \] DISTINCT FROM](https://docs.snowflake.com/en/sql-reference/functions/is-distinct-from) | Conditional expression | — | | [IS \[ NOT \] NULL](https://docs.snowflake.com/en/sql-reference/functions/is-null) | Conditional expression | — | | [IS\_NULL\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/is_null_value) | Conditional expression | ✓ | | [LEAST](https://docs.snowflake.com/en/sql-reference/functions/least) | Conditional expression | ✓ | | [LEAST\_IGNORE\_NULLS](https://docs.snowflake.com/en/sql-reference/functions/least_ignore_nulls) | Conditional expression | ✓ | | [NULLIF](https://docs.snowflake.com/en/sql-reference/functions/nullif) | Conditional expression | ✓ | | [NULLIFZERO](https://docs.snowflake.com/en/sql-reference/functions/nullifzero) | Conditional expression | ✓ | | [NVL](https://docs.snowflake.com/en/sql-reference/functions/nvl) | Conditional expression | ✓ | | [NVL2](https://docs.snowflake.com/en/sql-reference/functions/nvl2) | Conditional expression | ✓ | | [REGR\_VALX](https://docs.snowflake.com/en/sql-reference/functions/regr_valx) | Conditional expression | ✓ | | [REGR\_VALY](https://docs.snowflake.com/en/sql-reference/functions/regr_valy) | Conditional expression | ✓ | | [ZEROIFNULL](https://docs.snowflake.com/en/sql-reference/functions/zeroifnull) | Conditional expression | ✓ | | [ALL\_USER\_NAMES](https://docs.snowflake.com/en/sql-reference/functions/all_user_names) | Context | ✓ | | [CURRENT\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/current_account) | Context | ✓ | | [CURRENT\_ACCOUNT\_NAME](https://docs.snowflake.com/en/sql-reference/functions/current_account_name) | Context | ✓ | | [CURRENT\_AVAILABLE\_ROLES](https://docs.snowflake.com/en/sql-reference/functions/current_available_roles) | Context | ✓ | | [CURRENT\_CLIENT](https://docs.snowflake.com/en/sql-reference/functions/current_client) | Context | ✓ | | [CURRENT\_DATABASE](https://docs.snowflake.com/en/sql-reference/functions/current_database) | Context | ✓ | | [CURRENT\_DATE](https://docs.snowflake.com/en/sql-reference/functions/current_date) | Context | ✓ | | [CURRENT\_IP\_ADDRESS](https://docs.snowflake.com/en/sql-reference/functions/current_ip_address) | Context | ✓ | | [CURRENT\_ORGANIZATION\_NAME](https://docs.snowflake.com/en/sql-reference/functions/current_organization_name) | Context | ✓ | | [CURRENT\_ORGANIZATION\_USER](https://docs.snowflake.com/en/sql-reference/functions/current_organization_user) | Context | — | | [CURRENT\_REGION](https://docs.snowflake.com/en/sql-reference/functions/current_region) | Context | ✓ | | [CURRENT\_ROLE](https://docs.snowflake.com/en/sql-reference/functions/current_role) | Context | ✓ | | [CURRENT\_ROLE\_TYPE](https://docs.snowflake.com/en/sql-reference/functions/current_role_type) | Context | ✓ | | [CURRENT\_SCHEMA](https://docs.snowflake.com/en/sql-reference/functions/current_schema) | Context | ✓ | | [CURRENT\_SCHEMAS](https://docs.snowflake.com/en/sql-reference/functions/current_schemas) | Context | ✓ | | [CURRENT\_SECONDARY\_ROLES](https://docs.snowflake.com/en/sql-reference/functions/current_secondary_roles) | Context | ✓ | | [CURRENT\_SESSION](https://docs.snowflake.com/en/sql-reference/functions/current_session) | Context | ✓ | | [CURRENT\_STATEMENT](https://docs.snowflake.com/en/sql-reference/functions/current_statement) | Context | ✓ | | [CURRENT\_TIME](https://docs.snowflake.com/en/sql-reference/functions/current_time) | Context | ✓ | | [CURRENT\_TIMESTAMP](https://docs.snowflake.com/en/sql-reference/functions/current_timestamp) | Context | ✓ | | [CURRENT\_TRANSACTION](https://docs.snowflake.com/en/sql-reference/functions/current_transaction) | Context | ✓ | | [CURRENT\_USER](https://docs.snowflake.com/en/sql-reference/functions/current_user) | Context | ✓ | | [CURRENT\_VERSION](https://docs.snowflake.com/en/sql-reference/functions/current_version) | Context | ✓ | | [CURRENT\_WAREHOUSE](https://docs.snowflake.com/en/sql-reference/functions/current_warehouse) | Context | ✓ | | [GET\_CONDITION\_QUERY\_UUID](https://docs.snowflake.com/en/sql-reference/functions/get_condition_query_uuid) | Context | — | | [GET\_CONFIGURATION\_VALUE (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/get_configuration_value) | Context | — | | [GETDATE](https://docs.snowflake.com/en/sql-reference/functions/getdate) | Context | ✓ | | [GETVARIABLE](https://docs.snowflake.com/en/sql-reference/functions/getvariable) | Context | ✓ | | [INVOKER\_ROLE](https://docs.snowflake.com/en/sql-reference/functions/invoker_role) | Context | ✓ | | [INVOKER\_SHARE](https://docs.snowflake.com/en/sql-reference/functions/invoker_share) | Context | ✓ | | [IS\_AGENT\_ACTIVATED (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_agent_activated) | Context | — | | [IS\_APPLICATION\_ROLE\_ACTIVATED (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_application_role_activated) | Context | — | | [IS\_APPLICATION\_ROLE\_IN\_SESSION](https://docs.snowflake.com/en/sql-reference/functions/is_application_role_in_session) | Context | ✓ | | [IS\_CONFIGURATION\_SET (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_configuration_set) | Context | — | | [IS\_DATABASE\_ROLE\_ACTIVATED (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_database_role_activated) | Context | — | | [IS\_DATABASE\_ROLE\_IN\_SESSION](https://docs.snowflake.com/en/sql-reference/functions/is_database_role_in_session) | Context | ✓ | | [IS\_GRANTED\_TO\_INVOKER\_ROLE](https://docs.snowflake.com/en/sql-reference/functions/is_granted_to_invoker_role) | Context | ✓ | | [IS\_GROUP\_ACTIVATED (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_group_activated) | Context | — | | [IS\_GROUP\_IMPORTED (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_group_imported) | Context | — | | [IS\_INSTANCE\_ROLE\_IN\_SESSION](https://docs.snowflake.com/en/sql-reference/functions/is_instance_role_in_session) | Context | — | | [IS\_ORGANIZATION\_USER\_GROUP\_IN\_SESSION](https://docs.snowflake.com/en/sql-reference/functions/is_organization_user_group_in_session) | Context | — | | [IS\_ROLE\_ACTIVATED (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_role_activated) | Context | — | | [IS\_ROLE\_IN\_SESSION](https://docs.snowflake.com/en/sql-reference/functions/is_role_in_session) | Context | ✓ | | [IS\_USER\_IMPORTED (SYS\_CONTEXT FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/is_user_imported) | Context | — | | [LAST\_QUERY\_ID](https://docs.snowflake.com/en/sql-reference/functions/last_query_id) | Context | ✓ | | [LAST\_TRANSACTION](https://docs.snowflake.com/en/sql-reference/functions/last_transaction) | Context | ✓ | | [LOCALTIME](https://docs.snowflake.com/en/sql-reference/functions/localtime) | Context | ✓ | | [LOCALTIMESTAMP](https://docs.snowflake.com/en/sql-reference/functions/localtimestamp) | Context | ✓ | | [POLICY\_CONTEXT](https://docs.snowflake.com/en/sql-reference/functions/policy_context) | Context | — | | [SET\_SYS\_CONTEXT](https://docs.snowflake.com/en/sql-reference/functions/set_sys_context) | Context | — | | [SYS\_CONTEXT](https://docs.snowflake.com/en/sql-reference/functions/sys_context) | Context | — | | [SYS\_CONTEXT (SNOWFLAKE$APPLICATION NAMESPACE)](https://docs.snowflake.com/en/sql-reference/functions/sys_context_snowflake_application) | Context | — | | [SYS\_CONTEXT (SNOWFLAKE$CURRENT NAMESPACE)](https://docs.snowflake.com/en/sql-reference/functions/sys_context_snowflake_current) | Context | — | | [SYS\_CONTEXT (SNOWFLAKE$ENVIRONMENT NAMESPACE)](https://docs.snowflake.com/en/sql-reference/functions/sys_context_snowflake_environment) | Context | — | | [SYS\_CONTEXT (SNOWFLAKE$ORGANIZATION NAMESPACE)](https://docs.snowflake.com/en/sql-reference/functions/sys_context_snowflake_organization) | Context | — | | [SYS\_CONTEXT (SNOWFLAKE$ORGANIZATION\_SESSION NAMESPACE)](https://docs.snowflake.com/en/sql-reference/functions/sys_context_snowflake_organization_session) | Context | — | | [SYS\_CONTEXT (SNOWFLAKE$SESSION NAMESPACE)](https://docs.snowflake.com/en/sql-reference/functions/sys_context_snowflake_session) | Context | — | | [SYS\_CONTEXT (SNOWFLAKE$SESSION\_ATTRIBUTES NAMESPACE)](https://docs.snowflake.com/en/sql-reference/functions/sys_context_snowflake_session_attributes) | Context | — | | [SYSDATE](https://docs.snowflake.com/en/sql-reference/functions/sysdate) | Context | ✓ | | [SYSTIMESTAMP](https://docs.snowflake.com/en/sql-reference/functions/systimestamp) | Context | ✓ | | [CAST , ::](https://docs.snowflake.com/en/sql-reference/functions/cast) | Conversion | — | | [TO\_ARRAY](https://docs.snowflake.com/en/sql-reference/functions/to_array) | Conversion | ✓ | | [TO\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/to_binary) | Conversion | ✓ | | [TO\_BOOLEAN](https://docs.snowflake.com/en/sql-reference/functions/to_boolean) | Conversion | ✓ | | [TO\_CHAR , TO\_VARCHAR](https://docs.snowflake.com/en/sql-reference/functions/to_char) | Conversion | ✓ | | [TO\_DATE , DATE](https://docs.snowflake.com/en/sql-reference/functions/to_date) | Conversion | ✓ | | [TO\_DECFLOAT](https://docs.snowflake.com/en/sql-reference/functions/to_decfloat) | Conversion | — | | [TO\_DECIMAL , TO\_NUMBER , TO\_NUMERIC](https://docs.snowflake.com/en/sql-reference/functions/to_decimal) | Conversion | ✓ | | [TO\_DOUBLE](https://docs.snowflake.com/en/sql-reference/functions/to_double) | Conversion | ✓ | | [TO\_JSON](https://docs.snowflake.com/en/sql-reference/functions/to_json) | Conversion | ✓ | | [TO\_OBJECT](https://docs.snowflake.com/en/sql-reference/functions/to_object) | Conversion | ✓ | | [TO\_TIME , TIME](https://docs.snowflake.com/en/sql-reference/functions/to_time) | Conversion | ✓ | | [TO\_TIMESTAMP / TO\_TIMESTAMP\_\*](https://docs.snowflake.com/en/sql-reference/functions/to_timestamp) | Conversion | ✓ | | [TO\_UUID](https://docs.snowflake.com/en/sql-reference/functions/to_uuid) | Conversion | — | | [TO\_VARIANT](https://docs.snowflake.com/en/sql-reference/functions/to_variant) | Conversion | ✓ | | [TO\_XML](https://docs.snowflake.com/en/sql-reference/functions/to_xml) | Conversion | ✓ | | [TRY\_CAST](https://docs.snowflake.com/en/sql-reference/functions/try_cast) | Conversion | — | | [TRY\_TO\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/try_to_binary) | Conversion | ✓ | | [TRY\_TO\_BOOLEAN](https://docs.snowflake.com/en/sql-reference/functions/try_to_boolean) | Conversion | ✓ | | [TRY\_TO\_DATE](https://docs.snowflake.com/en/sql-reference/functions/try_to_date) | Conversion | ✓ | | [TRY\_TO\_DECFLOAT](https://docs.snowflake.com/en/sql-reference/functions/try_to_decfloat) | Conversion | — | | [TRY\_TO\_DECIMAL, TRY\_TO\_NUMBER, TRY\_TO\_NUMERIC](https://docs.snowflake.com/en/sql-reference/functions/try_to_decimal) | Conversion | ✓ | | [TRY\_TO\_DOUBLE](https://docs.snowflake.com/en/sql-reference/functions/try_to_double) | Conversion | ✓ | | [TRY\_TO\_TIME](https://docs.snowflake.com/en/sql-reference/functions/try_to_time) | Conversion | ✓ | | [TRY\_TO\_TIMESTAMP / TRY\_TO\_TIMESTAMP\_\*](https://docs.snowflake.com/en/sql-reference/functions/try_to_timestamp) | Conversion | ✓ | | [TRY\_TO\_UUID](https://docs.snowflake.com/en/sql-reference/functions/try_to_uuid) | Conversion | — | | [NORMAL](https://docs.snowflake.com/en/sql-reference/functions/normal) | Data generation | ✓ | | [RANDOM](https://docs.snowflake.com/en/sql-reference/functions/random) | Data generation | ✓ | | [RANDSTR](https://docs.snowflake.com/en/sql-reference/functions/randstr) | Data generation | ✓ | | [SEQ1 / SEQ2 / SEQ4 / SEQ8](https://docs.snowflake.com/en/sql-reference/functions/seq1) | Data generation | ✓ | | [UNIFORM](https://docs.snowflake.com/en/sql-reference/functions/uniform) | Data generation | ✓ | | [ZIPF](https://docs.snowflake.com/en/sql-reference/functions/zipf) | Data generation | ✓ | | [ACCEPTED\_VALUES (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_accepted_values) | Data metric | — | | [ALL\_SPACE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_all_space_count) | Data metric | — | | [ALL\_SPACE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_all_space_percent) | Data metric | — | | [APPROX\_QUANTILE\_20 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_20) | Data metric | — | | [APPROX\_QUANTILE\_25 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_25) | Data metric | — | | [APPROX\_QUANTILE\_40 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_40) | Data metric | — | | [APPROX\_QUANTILE\_50 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_50) | Data metric | — | | [APPROX\_QUANTILE\_60 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_60) | Data metric | — | | [APPROX\_QUANTILE\_80 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_80) | Data metric | — | | [APPROX\_QUANTILE\_95 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_95) | Data metric | — | | [APPROX\_QUANTILE\_99 (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_approx_quantile_99) | Data metric | — | | [AVG (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_avg) | Data metric | ✓ | | [BLANK\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_blank_count) | Data metric | — | | [BLANK\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_blank_percent) | Data metric | — | | [CASE\_FORMAT\_VIOLATION\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_case_format_violation_count) | Data metric | — | | [CASE\_FORMAT\_VIOLATION\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_case_format_violation_percent) | Data metric | — | | [CONTAINS\_SPACES\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_contains_spaces_count) | Data metric | — | | [CONTAINS\_SPACES\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_contains_spaces_percent) | Data metric | — | | [DATA\_METRIC\_SCHEDULED\_TIME (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_data_metric_schedule_time) | Data metric | — | | [DUPLICATE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_duplicate_count) | Data metric | — | | [DUPLICATE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_duplicate_percent) | Data metric | — | | [EMPTY\_STRING\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_empty_string_count) | Data metric | — | | [EMPTY\_STRING\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_empty_string_percent) | Data metric | — | | [EXTREME\_OUTLIER\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_extreme_outlier_count) | Data metric | — | | [EXTREME\_OUTLIER\_IQR\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_extreme_outlier_iqr_count) | Data metric | — | | [EXTREME\_OUTLIER\_IQR\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_extreme_outlier_iqr_percent) | Data metric | — | | [EXTREME\_OUTLIER\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_extreme_outlier_percent) | Data metric | — | | [EXTREME\_OUTLIER\_ZSCORE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_extreme_outlier_zscore_count) | Data metric | — | | [EXTREME\_OUTLIER\_ZSCORE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_extreme_outlier_zscore_percent) | Data metric | — | | [FALSE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_false_count) | Data metric | — | | [FALSE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_false_percent) | Data metric | — | | [FRESHNESS (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_freshness) | Data metric | — | | [FUTURE\_TIMESTAMP\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_future_timestamp_count) | Data metric | — | | [FUTURE\_TIMESTAMP\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_future_timestamp_percent) | Data metric | — | | [GEOMETRIC\_MEAN (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_geometric_mean) | Data metric | — | | [HARMONIC\_MEAN (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_harmonic_mean) | Data metric | — | | [INVALID\_CA\_POSTAL\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_ca_postal_code_count) | Data metric | — | | [INVALID\_CA\_POSTAL\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_ca_postal_code_percent) | Data metric | — | | [INVALID\_CA\_SIN\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_ca_sin_count) | Data metric | — | | [INVALID\_CA\_SIN\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_ca_sin_percent) | Data metric | — | | [INVALID\_CH\_OASI\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_ch_oasi_count) | Data metric | — | | [INVALID\_CH\_OASI\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_ch_oasi_percent) | Data metric | — | | [INVALID\_CUSIP\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_cusip_count) | Data metric | — | | [INVALID\_CUSIP\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_cusip_percent) | Data metric | — | | [INVALID\_DE\_POSTAL\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_de_postal_code_count) | Data metric | — | | [INVALID\_DE\_POSTAL\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_de_postal_code_percent) | Data metric | — | | [INVALID\_DE\_TAX\_ID\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_de_tax_id_count) | Data metric | — | | [INVALID\_DE\_TAX\_ID\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_de_tax_id_percent) | Data metric | — | | [INVALID\_EMAIL\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_email_count) | Data metric | — | | [INVALID\_EMAIL\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_email_percent) | Data metric | — | | [INVALID\_ES\_DNI\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_es_dni_count) | Data metric | — | | [INVALID\_ES\_DNI\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_es_dni_percent) | Data metric | — | | [INVALID\_FIGI\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_figi_count) | Data metric | — | | [INVALID\_FIGI\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_figi_percent) | Data metric | — | | [INVALID\_FR\_INSEE\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_fr_insee_code_count) | Data metric | — | | [INVALID\_FR\_INSEE\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_fr_insee_code_percent) | Data metric | — | | [INVALID\_FR\_POSTAL\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_fr_postal_code_count) | Data metric | — | | [INVALID\_FR\_POSTAL\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_fr_postal_code_percent) | Data metric | — | | [INVALID\_INTEGER\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_integer_count) | Data metric | — | | [INVALID\_INTEGER\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_integer_percent) | Data metric | — | | [INVALID\_ISIN\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_isin_count) | Data metric | — | | [INVALID\_ISIN\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_isin_percent) | Data metric | — | | [INVALID\_ISO8601\_FORMAT\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_iso8601_format_count) | Data metric | — | | [INVALID\_ISO8601\_FORMAT\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_iso8601_format_percent) | Data metric | — | | [INVALID\_IT\_FISCAL\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_it_fiscal_code_count) | Data metric | — | | [INVALID\_IT\_FISCAL\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_it_fiscal_code_percent) | Data metric | — | | [INVALID\_JSON\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_json_count) | Data metric | — | | [INVALID\_JSON\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_json_percent) | Data metric | — | | [INVALID\_LATITUDE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_latitude_count) | Data metric | — | | [INVALID\_LATITUDE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_latitude_percent) | Data metric | — | | [INVALID\_LEI\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_lei_count) | Data metric | — | | [INVALID\_LEI\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_lei_percent) | Data metric | — | | [INVALID\_LONGITUDE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_longitude_count) | Data metric | — | | [INVALID\_LONGITUDE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_longitude_percent) | Data metric | — | | [INVALID\_NL\_BSN\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_nl_bsn_count) | Data metric | — | | [INVALID\_NL\_BSN\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_nl_bsn_percent) | Data metric | — | | [INVALID\_NUMERIC\_TYPE\_CAST\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_numeric_type_cast_count) | Data metric | — | | [INVALID\_NUMERIC\_TYPE\_CAST\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_numeric_type_cast_percent) | Data metric | — | | [INVALID\_PERCENTAGE\_RANGE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_percentage_range_count) | Data metric | — | | [INVALID\_PERCENTAGE\_RANGE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_percentage_range_percent) | Data metric | — | | [INVALID\_PERM\_ID\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_perm_id_count) | Data metric | — | | [INVALID\_PERM\_ID\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_perm_id_percent) | Data metric | — | | [INVALID\_PL\_PESEL\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_pl_pesel_count) | Data metric | — | | [INVALID\_PL\_PESEL\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_pl_pesel_percent) | Data metric | — | | [INVALID\_PROBABILITY\_RANGE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_probability_range_count) | Data metric | — | | [INVALID\_PROBABILITY\_RANGE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_probability_range_percent) | Data metric | — | | [INVALID\_SEDOL\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_sedol_count) | Data metric | — | | [INVALID\_SEDOL\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_sedol_percent) | Data metric | — | | [INVALID\_TIMESTAMP\_STRING\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_timestamp_string_count) | Data metric | — | | [INVALID\_TIMESTAMP\_STRING\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_timestamp_string_percent) | Data metric | — | | [INVALID\_TR\_ID\_NO\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_tr_id_no_count) | Data metric | — | | [INVALID\_TR\_ID\_NO\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_tr_id_no_percent) | Data metric | — | | [INVALID\_UK\_NINO\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_uk_nino_count) | Data metric | — | | [INVALID\_UK\_NINO\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_uk_nino_percent) | Data metric | — | | [INVALID\_UK\_POSTAL\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_uk_postal_code_count) | Data metric | — | | [INVALID\_UK\_POSTAL\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_uk_postal_code_percent) | Data metric | — | | [INVALID\_USA\_PHONE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_usa_phone_count) | Data metric | — | | [INVALID\_USA\_PHONE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_usa_phone_percent) | Data metric | — | | [INVALID\_USA\_STATE\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_usa_state_code_count) | Data metric | — | | [INVALID\_USA\_STATE\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_usa_state_code_percent) | Data metric | — | | [INVALID\_USA\_ZIP\_CODE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_usa_zip_code_count) | Data metric | — | | [INVALID\_USA\_ZIP\_CODE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_usa_zip_code_percent) | Data metric | — | | [INVALID\_UUID\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_uuid_count) | Data metric | — | | [INVALID\_UUID\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_invalid_uuid_percent) | Data metric | — | | [KURTOSIS (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_kurtosis) | Data metric | ✓ | | [MAX (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_max) | Data metric | ✓ | | [MEDIAN (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_median) | Data metric | ✓ | | [MIN (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_min) | Data metric | ✓ | | [NAN\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_nan_count) | Data metric | — | | [NAN\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_nan_percent) | Data metric | — | | [NEGATIVE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_negative_count) | Data metric | — | | [NEGATIVE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_negative_percent) | Data metric | — | | [NOT\_FRIDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_friday_count) | Data metric | — | | [NOT\_FRIDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_friday_percent) | Data metric | — | | [NOT\_IN\_FUTURE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_in_future_count) | Data metric | — | | [NOT\_IN\_FUTURE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_in_future_percent) | Data metric | — | | [NOT\_MONDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_monday_count) | Data metric | — | | [NOT\_MONDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_monday_percent) | Data metric | — | | [NOT\_NULL\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_null_count) | Data metric | — | | [NOT\_NULL\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_null_percent) | Data metric | — | | [NOT\_SATURDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_saturday_count) | Data metric | — | | [NOT\_SATURDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_saturday_percent) | Data metric | — | | [NOT\_SUNDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_sunday_count) | Data metric | — | | [NOT\_SUNDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_sunday_percent) | Data metric | — | | [NOT\_THURSDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_thursday_count) | Data metric | — | | [NOT\_THURSDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_thursday_percent) | Data metric | — | | [NOT\_TODAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_today_count) | Data metric | — | | [NOT\_TODAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_today_percent) | Data metric | — | | [NOT\_TUESDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_tuesday_count) | Data metric | — | | [NOT\_TUESDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_tuesday_percent) | Data metric | — | | [NOT\_WEDNESDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_wednesday_count) | Data metric | — | | [NOT\_WEDNESDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_wednesday_percent) | Data metric | — | | [NOT\_WEEKDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_weekday_count) | Data metric | — | | [NOT\_WEEKDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_weekday_percent) | Data metric | — | | [NOT\_YESTERDAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_yesterday_count) | Data metric | — | | [NOT\_YESTERDAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_not_yesterday_percent) | Data metric | — | | [NULL\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_null_count) | Data metric | — | | [NULL\_KEYWORD\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_null_keyword_count) | Data metric | — | | [NULL\_KEYWORD\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_null_keyword_percent) | Data metric | — | | [NULL\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_null_percent) | Data metric | — | | [OUTLIER\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_outlier_count) | Data metric | — | | [OUTLIER\_IQR\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_outlier_iqr_count) | Data metric | — | | [OUTLIER\_IQR\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_outlier_iqr_percent) | Data metric | — | | [OUTLIER\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_outlier_percent) | Data metric | — | | [OUTLIER\_ZSCORE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_outlier_zscore_count) | Data metric | — | | [OUTLIER\_ZSCORE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_outlier_zscore_percent) | Data metric | — | | [REFERENTIAL\_INTEGRITY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_referential_integrity_count) | Data metric | — | | [ROW\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_row_count) | Data metric | — | | [ROWS\_IN\_PAST\_DAY\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_day_count) | Data metric | — | | [ROWS\_IN\_PAST\_DAY\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_day_percent) | Data metric | — | | [ROWS\_IN\_PAST\_HOUR\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_hour_count) | Data metric | — | | [ROWS\_IN\_PAST\_HOUR\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_hour_percent) | Data metric | — | | [ROWS\_IN\_PAST\_MONTH\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_month_count) | Data metric | — | | [ROWS\_IN\_PAST\_MONTH\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_month_percent) | Data metric | — | | [ROWS\_IN\_PAST\_WEEK\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_week_count) | Data metric | — | | [ROWS\_IN\_PAST\_WEEK\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_week_percent) | Data metric | — | | [ROWS\_IN\_PAST\_YEAR\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_year_count) | Data metric | — | | [ROWS\_IN\_PAST\_YEAR\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_rows_in_past_year_percent) | Data metric | — | | [SCHEMA\_CHANGE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_schema_change_count) | Data metric | — | | [SKEW (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_skew) | Data metric | ✓ | | [SPECIAL\_CHARACTER\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_special_character_count) | Data metric | — | | [SPECIAL\_CHARACTER\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_special_character_percent) | Data metric | — | | [STDDEV (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_stddev) | Data metric | ✓ | | [STRING\_LENGTH\_AVG (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_string_length_avg) | Data metric | — | | [STRING\_LENGTH\_MAX (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_string_length_max) | Data metric | — | | [STRING\_LENGTH\_MIN (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_string_length_min) | Data metric | — | | [STRING\_LENGTH\_STDDEV (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_string_length_stddev) | Data metric | — | | [SUM (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_sum) | Data metric | ✓ | | [TRUE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_true_count) | Data metric | — | | [TRUE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_true_percent) | Data metric | — | | [UNIQUE\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_unique_count) | Data metric | — | | [UNIQUE\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_unique_percent) | Data metric | — | | [UNIX\_ZERO\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_unix_zero_count) | Data metric | — | | [UNIX\_ZERO\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_unix_zero_percent) | Data metric | — | | [UNTRIMMED\_STRING\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_untrimmed_string_count) | Data metric | — | | [UNTRIMMED\_STRING\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_untrimmed_string_percent) | Data metric | — | | [VARIANCE (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_variance) | Data metric | ✓ | | [ZERO\_COUNT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_zero_count) | Data metric | — | | [ZERO\_PERCENT (SYSTEM DATA METRIC FUNCTION)](https://docs.snowflake.com/en/sql-reference/functions/dmf_zero_percent) | Data metric | — | | [ADD\_MONTHS](https://docs.snowflake.com/en/sql-reference/functions/add_months) | Date & time | ✓ | | [CONVERT\_TIMEZONE](https://docs.snowflake.com/en/sql-reference/functions/convert_timezone) | Date & time | ✓ | | [DATE\_FROM\_PARTS](https://docs.snowflake.com/en/sql-reference/functions/date_from_parts) | Date & time | ✓ | | [DATE\_PART](https://docs.snowflake.com/en/sql-reference/functions/date_part) | Date & time | ✓ | | [DATE\_TRUNC](https://docs.snowflake.com/en/sql-reference/functions/date_trunc) | Date & time | ✓ | | [DATEADD](https://docs.snowflake.com/en/sql-reference/functions/dateadd) | Date & time | ✓ | | [DATEDIFF](https://docs.snowflake.com/en/sql-reference/functions/datediff) | Date & time | ✓ | | [DAYNAME](https://docs.snowflake.com/en/sql-reference/functions/dayname) | Date & time | ✓ | | [EXTRACT](https://docs.snowflake.com/en/sql-reference/functions/extract) | Date & time | — | | [HOUR / MINUTE / SECOND](https://docs.snowflake.com/en/sql-reference/functions/hour-minute-second) | Date & time | ✓ | | [LAST\_DAY](https://docs.snowflake.com/en/sql-reference/functions/last_day) | Date & time | ✓ | | [LAST\_SUCCESSFUL\_SCHEDULED\_TIME](https://docs.snowflake.com/en/sql-reference/functions/last_successful_scheduled_time) | Date & time | — | | [MONTHNAME](https://docs.snowflake.com/en/sql-reference/functions/monthname) | Date & time | ✓ | | [MONTHS\_BETWEEN](https://docs.snowflake.com/en/sql-reference/functions/months_between) | Date & time | ✓ | | [NEXT\_DAY](https://docs.snowflake.com/en/sql-reference/functions/next_day) | Date & time | ✓ | | [PREVIOUS\_DAY](https://docs.snowflake.com/en/sql-reference/functions/previous_day) | Date & time | ✓ | | [SCHEDULED\_TIME](https://docs.snowflake.com/en/sql-reference/functions/scheduled_time) | Date & time | — | | [TIME\_FROM\_PARTS](https://docs.snowflake.com/en/sql-reference/functions/time_from_parts) | Date & time | ✓ | | [TIME\_SLICE](https://docs.snowflake.com/en/sql-reference/functions/time_slice) | Date & time | ✓ | | [TIMEADD](https://docs.snowflake.com/en/sql-reference/functions/timeadd) | Date & time | — | | [TIMEDIFF](https://docs.snowflake.com/en/sql-reference/functions/timediff) | Date & time | — | | [TIMESTAMP\_FROM\_PARTS](https://docs.snowflake.com/en/sql-reference/functions/timestamp_from_parts) | Date & time | ✓ | | [TIMESTAMPADD](https://docs.snowflake.com/en/sql-reference/functions/timestampadd) | Date & time | — | | [TIMESTAMPDIFF](https://docs.snowflake.com/en/sql-reference/functions/timestampdiff) | Date & time | — | | [TRUNCATE, TRUNC](https://docs.snowflake.com/en/sql-reference/functions/trunc2) | Date & time | — | | [YEAR\* / DAY\* / WEEK\* / MONTH / QUARTER](https://docs.snowflake.com/en/sql-reference/functions/year) | Date & time | ✓ | | [DP\_INTERVAL\_HIGH](https://docs.snowflake.com/en/sql-reference/functions/dp_interval_high) | Differential privacy | — | | [DP\_INTERVAL\_LOW](https://docs.snowflake.com/en/sql-reference/functions/dp_interval_low) | Differential privacy | — | | [ESTIMATE\_REMAINING\_DP\_AGGREGATES](https://docs.snowflake.com/en/sql-reference/functions/estimate_remaining_dp_aggregates) | Differential privacy | — | | [DECRYPT](https://docs.snowflake.com/en/sql-reference/functions/decrypt) | Encryption | ✓ | | [DECRYPT\_RAW](https://docs.snowflake.com/en/sql-reference/functions/decrypt_raw) | Encryption | ✓ | | [ENCRYPT](https://docs.snowflake.com/en/sql-reference/functions/encrypt) | Encryption | ✓ | | [ENCRYPT\_RAW](https://docs.snowflake.com/en/sql-reference/functions/encrypt_raw) | Encryption | ✓ | | [TRY\_DECRYPT](https://docs.snowflake.com/en/sql-reference/functions/try_decrypt) | Encryption | ✓ | | [TRY\_DECRYPT\_RAW](https://docs.snowflake.com/en/sql-reference/functions/try_decrypt_raw) | Encryption | ✓ | | [AI\_TRANSCRIBE](https://docs.snowflake.com/en/sql-reference/functions/ai_transcribe) | File | ✓ | | [BUILD\_SCOPED\_FILE\_URL](https://docs.snowflake.com/en/sql-reference/functions/build_scoped_file_url) | File | — | | [BUILD\_STAGE\_FILE\_URL](https://docs.snowflake.com/en/sql-reference/functions/build_stage_file_url) | File | — | | [FL\_GET\_CONTENT\_TYPE](https://docs.snowflake.com/en/sql-reference/functions/fl_get_content_type) | File | — | | [FL\_GET\_ETAG](https://docs.snowflake.com/en/sql-reference/functions/fl_get_etag) | File | — | | [FL\_GET\_FILE\_TYPE](https://docs.snowflake.com/en/sql-reference/functions/fl_get_file_type) | File | — | | [FL\_GET\_LAST\_MODIFIED](https://docs.snowflake.com/en/sql-reference/functions/fl_get_last_modified) | File | — | | [FL\_GET\_RELATIVE\_PATH](https://docs.snowflake.com/en/sql-reference/functions/fl_get_relative_path) | File | — | | [FL\_GET\_SCOPED\_FILE\_URL](https://docs.snowflake.com/en/sql-reference/functions/fl_get_scoped_file_url) | File | — | | [FL\_GET\_SIZE](https://docs.snowflake.com/en/sql-reference/functions/fl_get_size) | File | — | | [FL\_GET\_STAGE](https://docs.snowflake.com/en/sql-reference/functions/fl_get_stage) | File | — | | [FL\_GET\_STAGE\_FILE\_URL](https://docs.snowflake.com/en/sql-reference/functions/fl_get_stage_file_url) | File | — | | [FL\_IS\_AUDIO](https://docs.snowflake.com/en/sql-reference/functions/fl_is_audio) | File | — | | [FL\_IS\_COMPRESSED](https://docs.snowflake.com/en/sql-reference/functions/fl_is_compressed) | File | — | | [FL\_IS\_DOCUMENT](https://docs.snowflake.com/en/sql-reference/functions/fl_is_document) | File | — | | [FL\_IS\_IMAGE](https://docs.snowflake.com/en/sql-reference/functions/fl_is_image) | File | — | | [FL\_IS\_VIDEO](https://docs.snowflake.com/en/sql-reference/functions/fl_is_video) | File | — | | [GET\_ABSOLUTE\_PATH](https://docs.snowflake.com/en/sql-reference/functions/get_absolute_path) | File | — | | [GET\_PRESIGNED\_URL](https://docs.snowflake.com/en/sql-reference/functions/get_presigned_url) | File | — | | [GET\_RELATIVE\_PATH](https://docs.snowflake.com/en/sql-reference/functions/get_relative_path) | File | — | | [GET\_STAGE\_LOCATION](https://docs.snowflake.com/en/sql-reference/functions/get_stage_location) | File | — | | [PARSE\_DOCUMENT (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/parse_document-snowflake-cortex) | File | — | | [TO\_FILE](https://docs.snowflake.com/en/sql-reference/functions/to_file) | File | ✓ | | [TRY\_TO\_FILE](https://docs.snowflake.com/en/sql-reference/functions/try_to_file) | File | ✓ | | [GENERATE\_POSTGRES\_ACCESS\_TOKEN\_FOR\_USER](https://docs.snowflake.com/en/sql-reference/functions/generate_postgres_access_token_for_user) | Generates a short-lived access token for a Snowflake user to use as a password when logging into a Snowflake Postgres instance that has the AUTHENTICATION\_AUTHORITY attribute set to POSTGRES\_OR\_SNOWFLAKE. | — | | [H3\_CELL\_TO\_BOUNDARY](https://docs.snowflake.com/en/sql-reference/functions/h3_cell_to_boundary) | Geospatial | ✓ | | [H3\_CELL\_TO\_CHILDREN](https://docs.snowflake.com/en/sql-reference/functions/h3_cell_to_children) | Geospatial | ✓ | | [H3\_CELL\_TO\_CHILDREN\_STRING](https://docs.snowflake.com/en/sql-reference/functions/h3_cell_to_children_string) | Geospatial | ✓ | | [H3\_CELL\_TO\_PARENT](https://docs.snowflake.com/en/sql-reference/functions/h3_cell_to_parent) | Geospatial | ✓ | | [H3\_CELL\_TO\_POINT](https://docs.snowflake.com/en/sql-reference/functions/h3_cell_to_point) | Geospatial | ✓ | | [H3\_COMPACT\_CELLS](https://docs.snowflake.com/en/sql-reference/functions/h3_compact_cells) | Geospatial | — | | [H3\_COMPACT\_CELLS\_STRINGS](https://docs.snowflake.com/en/sql-reference/functions/h3_compact_cells_strings) | Geospatial | — | | [H3\_COVERAGE](https://docs.snowflake.com/en/sql-reference/functions/h3_coverage) | Geospatial | ✓ | | [H3\_COVERAGE\_STRINGS](https://docs.snowflake.com/en/sql-reference/functions/h3_coverage_strings) | Geospatial | ✓ | | [H3\_GET\_RESOLUTION](https://docs.snowflake.com/en/sql-reference/functions/h3_get_resolution) | Geospatial | ✓ | | [H3\_GRID\_DISK](https://docs.snowflake.com/en/sql-reference/functions/h3_grid_disk) | Geospatial | ✓ | | [H3\_GRID\_DISTANCE](https://docs.snowflake.com/en/sql-reference/functions/h3_grid_distance) | Geospatial | ✓ | | [H3\_GRID\_PATH](https://docs.snowflake.com/en/sql-reference/functions/h3_grid_path) | Geospatial | ✓ | | [H3\_INT\_TO\_STRING](https://docs.snowflake.com/en/sql-reference/functions/h3_int_to_string) | Geospatial | ✓ | | [H3\_IS\_PENTAGON](https://docs.snowflake.com/en/sql-reference/functions/h3_is_pentagon) | Geospatial | — | | [H3\_IS\_VALID\_CELL](https://docs.snowflake.com/en/sql-reference/functions/h3_is_valid_cell) | Geospatial | — | | [H3\_LATLNG\_TO\_CELL](https://docs.snowflake.com/en/sql-reference/functions/h3_latlng_to_cell) | Geospatial | ✓ | | [H3\_LATLNG\_TO\_CELL\_STRING](https://docs.snowflake.com/en/sql-reference/functions/h3_latlng_to_cell_string) | Geospatial | ✓ | | [H3\_POINT\_TO\_CELL](https://docs.snowflake.com/en/sql-reference/functions/h3_point_to_cell) | Geospatial | ✓ | | [H3\_POINT\_TO\_CELL\_STRING](https://docs.snowflake.com/en/sql-reference/functions/h3_point_to_cell_string) | Geospatial | ✓ | | [H3\_POLYGON\_TO\_CELLS](https://docs.snowflake.com/en/sql-reference/functions/h3_polygon_to_cells) | Geospatial | ✓ | | [H3\_POLYGON\_TO\_CELLS\_STRINGS](https://docs.snowflake.com/en/sql-reference/functions/h3_polygon_to_cells_strings) | Geospatial | ✓ | | [H3\_STRING\_TO\_INT](https://docs.snowflake.com/en/sql-reference/functions/h3_string_to_int) | Geospatial | ✓ | | [H3\_TRY\_COVERAGE](https://docs.snowflake.com/en/sql-reference/functions/h3_try_coverage) | Geospatial | — | | [H3\_TRY\_COVERAGE\_STRINGS](https://docs.snowflake.com/en/sql-reference/functions/h3_try_coverage_strings) | Geospatial | — | | [H3\_TRY\_GRID\_DISTANCE](https://docs.snowflake.com/en/sql-reference/functions/h3_try_grid_distance) | Geospatial | — | | [H3\_TRY\_GRID\_PATH](https://docs.snowflake.com/en/sql-reference/functions/h3_try_grid_path) | Geospatial | — | | [H3\_TRY\_POLYGON\_TO\_CELLS](https://docs.snowflake.com/en/sql-reference/functions/h3_try_polygon_to_cells) | Geospatial | — | | [H3\_TRY\_POLYGON\_TO\_CELLS\_STRINGS](https://docs.snowflake.com/en/sql-reference/functions/h3_try_polygon_to_cells_strings) | Geospatial | — | | [H3\_UNCOMPACT\_CELLS](https://docs.snowflake.com/en/sql-reference/functions/h3_uncompact_cells) | Geospatial | — | | [H3\_UNCOMPACT\_CELLS\_STRINGS](https://docs.snowflake.com/en/sql-reference/functions/h3_uncompact_cells_strings) | Geospatial | — | | [HAVERSINE](https://docs.snowflake.com/en/sql-reference/functions/haversine) | Geospatial | ✓ | | [ST\_AREA](https://docs.snowflake.com/en/sql-reference/functions/st_area) | Geospatial | ✓ | | [ST\_ASEWKB](https://docs.snowflake.com/en/sql-reference/functions/st_asewkb) | Geospatial | ✓ | | [ST\_ASEWKT](https://docs.snowflake.com/en/sql-reference/functions/st_asewkt) | Geospatial | ✓ | | [ST\_ASGEOJSON](https://docs.snowflake.com/en/sql-reference/functions/st_asgeojson) | Geospatial | ✓ | | [ST\_ASWKB , ST\_ASBINARY](https://docs.snowflake.com/en/sql-reference/functions/st_aswkb) | Geospatial | ✓ | | [ST\_ASWKT , ST\_ASTEXT](https://docs.snowflake.com/en/sql-reference/functions/st_aswkt) | Geospatial | ✓ | | [ST\_AZIMUTH](https://docs.snowflake.com/en/sql-reference/functions/st_azimuth) | Geospatial | ✓ | | [ST\_BUFFER](https://docs.snowflake.com/en/sql-reference/functions/st_buffer) | Geospatial | ✓ | | [ST\_CENTROID](https://docs.snowflake.com/en/sql-reference/functions/st_centroid) | Geospatial | ✓ | | [ST\_COLLECT](https://docs.snowflake.com/en/sql-reference/functions/st_collect) | Geospatial | ✓ | | [ST\_CONTAINS](https://docs.snowflake.com/en/sql-reference/functions/st_contains) | Geospatial | ✓ | | [ST\_COVEREDBY](https://docs.snowflake.com/en/sql-reference/functions/st_coveredby) | Geospatial | ✓ | | [ST\_COVERS](https://docs.snowflake.com/en/sql-reference/functions/st_covers) | Geospatial | ✓ | | [ST\_DIFFERENCE](https://docs.snowflake.com/en/sql-reference/functions/st_difference) | Geospatial | ✓ | | [ST\_DIMENSION](https://docs.snowflake.com/en/sql-reference/functions/st_dimension) | Geospatial | ✓ | | [ST\_DISJOINT](https://docs.snowflake.com/en/sql-reference/functions/st_disjoint) | Geospatial | ✓ | | [ST\_DISTANCE](https://docs.snowflake.com/en/sql-reference/functions/st_distance) | Geospatial | ✓ | | [ST\_DWITHIN](https://docs.snowflake.com/en/sql-reference/functions/st_dwithin) | Geospatial | ✓ | | [ST\_ENDPOINT](https://docs.snowflake.com/en/sql-reference/functions/st_endpoint) | Geospatial | ✓ | | [ST\_ENVELOPE](https://docs.snowflake.com/en/sql-reference/functions/st_envelope) | Geospatial | ✓ | | [ST\_GEOGFROMGEOHASH](https://docs.snowflake.com/en/sql-reference/functions/st_geogfromgeohash) | Geospatial | ✓ | | [ST\_GEOGPOINTFROMGEOHASH](https://docs.snowflake.com/en/sql-reference/functions/st_geogpointfromgeohash) | Geospatial | ✓ | | [ST\_GEOGRAPHYFROMWKB](https://docs.snowflake.com/en/sql-reference/functions/st_geographyfromwkb) | Geospatial | ✓ | | [ST\_GEOGRAPHYFROMWKT](https://docs.snowflake.com/en/sql-reference/functions/st_geographyfromwkt) | Geospatial | ✓ | | [ST\_GEOHASH](https://docs.snowflake.com/en/sql-reference/functions/st_geohash) | Geospatial | ✓ | | [ST\_GEOMETRYFROMWKB](https://docs.snowflake.com/en/sql-reference/functions/st_geometryfromwkb) | Geospatial | ✓ | | [ST\_GEOMETRYFROMWKT](https://docs.snowflake.com/en/sql-reference/functions/st_geometryfromwkt) | Geospatial | ✓ | | [ST\_GEOMFROMGEOHASH](https://docs.snowflake.com/en/sql-reference/functions/st_geomfromgeohash) | Geospatial | — | | [ST\_GEOMPOINTFROMGEOHASH](https://docs.snowflake.com/en/sql-reference/functions/st_geompointfromgeohash) | Geospatial | — | | [ST\_HAUSDORFFDISTANCE](https://docs.snowflake.com/en/sql-reference/functions/st_hausdorffdistance) | Geospatial | ✓ | | [ST\_INTERPOLATE](https://docs.snowflake.com/en/sql-reference/functions/st_interpolate) | Geospatial | — | | [ST\_INTERSECTION](https://docs.snowflake.com/en/sql-reference/functions/st_intersection) | Geospatial | ✓ | | [ST\_INTERSECTION\_AGG](https://docs.snowflake.com/en/sql-reference/functions/st_intersection_agg) | Geospatial | — | | [ST\_INTERSECTS](https://docs.snowflake.com/en/sql-reference/functions/st_intersects) | Geospatial | ✓ | | [ST\_ISVALID](https://docs.snowflake.com/en/sql-reference/functions/st_isvalid) | Geospatial | ✓ | | [ST\_LENGTH](https://docs.snowflake.com/en/sql-reference/functions/st_length) | Geospatial | ✓ | | [ST\_MAKEGEOMPOINT , ST\_GEOMPOINT](https://docs.snowflake.com/en/sql-reference/functions/st_makegeompoint) | Geospatial | ✓ | | [ST\_MAKELINE](https://docs.snowflake.com/en/sql-reference/functions/st_makeline) | Geospatial | ✓ | | [ST\_MAKEPOINT , ST\_POINT](https://docs.snowflake.com/en/sql-reference/functions/st_makepoint) | Geospatial | ✓ | | [ST\_MAKEPOLYGON , ST\_POLYGON](https://docs.snowflake.com/en/sql-reference/functions/st_makepolygon) | Geospatial | ✓ | | [ST\_MAKEPOLYGONORIENTED](https://docs.snowflake.com/en/sql-reference/functions/st_makepolygonoriented) | Geospatial | ✓ | | [ST\_NPOINTS , ST\_NUMPOINTS](https://docs.snowflake.com/en/sql-reference/functions/st_npoints) | Geospatial | ✓ | | [ST\_PERIMETER](https://docs.snowflake.com/en/sql-reference/functions/st_perimeter) | Geospatial | ✓ | | [ST\_POINTN](https://docs.snowflake.com/en/sql-reference/functions/st_pointn) | Geospatial | ✓ | | [ST\_SETSRID](https://docs.snowflake.com/en/sql-reference/functions/st_setsrid) | Geospatial | ✓ | | [ST\_SIMPLIFY](https://docs.snowflake.com/en/sql-reference/functions/st_simplify) | Geospatial | ✓ | | [ST\_SRID](https://docs.snowflake.com/en/sql-reference/functions/st_srid) | Geospatial | ✓ | | [ST\_STARTPOINT](https://docs.snowflake.com/en/sql-reference/functions/st_startpoint) | Geospatial | ✓ | | [ST\_SYMDIFFERENCE](https://docs.snowflake.com/en/sql-reference/functions/st_symdifference) | Geospatial | ✓ | | [ST\_TRANSFORM](https://docs.snowflake.com/en/sql-reference/functions/st_transform) | Geospatial | ✓ | | [ST\_UNION](https://docs.snowflake.com/en/sql-reference/functions/st_union) | Geospatial | ✓ | | [ST\_UNION\_AGG](https://docs.snowflake.com/en/sql-reference/functions/st_union_agg) | Geospatial | ✓ | | [ST\_WITHIN](https://docs.snowflake.com/en/sql-reference/functions/st_within) | Geospatial | ✓ | | [ST\_X](https://docs.snowflake.com/en/sql-reference/functions/st_x) | Geospatial | ✓ | | [ST\_XMAX](https://docs.snowflake.com/en/sql-reference/functions/st_xmax) | Geospatial | ✓ | | [ST\_XMIN](https://docs.snowflake.com/en/sql-reference/functions/st_xmin) | Geospatial | ✓ | | [ST\_Y](https://docs.snowflake.com/en/sql-reference/functions/st_y) | Geospatial | ✓ | | [ST\_YMAX](https://docs.snowflake.com/en/sql-reference/functions/st_ymax) | Geospatial | ✓ | | [ST\_YMIN](https://docs.snowflake.com/en/sql-reference/functions/st_ymin) | Geospatial | ✓ | | [TO\_GEOGRAPHY](https://docs.snowflake.com/en/sql-reference/functions/to_geography) | Geospatial | ✓ | | [TO\_GEOMETRY](https://docs.snowflake.com/en/sql-reference/functions/to_geometry) | Geospatial | ✓ | | [TRY\_TO\_GEOGRAPHY](https://docs.snowflake.com/en/sql-reference/functions/try_to_geography) | Geospatial | ✓ | | [TRY\_TO\_GEOMETRY](https://docs.snowflake.com/en/sql-reference/functions/try_to_geometry) | Geospatial | ✓ | | [HASH](https://docs.snowflake.com/en/sql-reference/functions/hash) | Hash | ✓ | | [ALERT\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/alert_history) | Information Schema | ✓ | | [APPLICATION\_CALLBACK\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/application_callback_history) | Information Schema | — | | [APPLICATION\_SPECIFICATION\_STATUS\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/application_specification_status_history) | Information Schema | — | | [AUTO\_REFRESH\_REGISTRATION\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/auto_refresh_registration_history) | Information Schema | ✓ | | [AUTOMATIC\_CLUSTERING\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/automatic_clustering_history) | Information Schema | ✓ | | [AVAILABLE\_LISTING\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/available_listing_refresh_history) | Information Schema | — | | [AVAILABLE\_LISTINGS](https://docs.snowflake.com/en/sql-reference/functions/available_listings) | Information Schema | — | | [BIND\_VALUES](https://docs.snowflake.com/en/sql-reference/functions/bind_values) | Information Schema | — | | [COMPLETE\_TASK\_GRAPHS](https://docs.snowflake.com/en/sql-reference/functions/complete_task_graphs) | Information Schema | ✓ | | [COPY\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/copy_history) | Information Schema | ✓ | | [CORTEX\_SEARCH\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/cortex_search_refresh_history) | Information Schema | — | | [CURRENT\_TASK\_GRAPHS](https://docs.snowflake.com/en/sql-reference/functions/current_task_graphs) | Information Schema | ✓ | | [DATA\_METRIC\_FUNCTION\_EXPECTATIONS](https://docs.snowflake.com/en/sql-reference/functions/data_metric_function_expectations) | Information Schema | — | | [DATA\_METRIC\_FUNCTION\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/data_metric_function_references) | Information Schema | — | | [DATA\_TRANSFER\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/data_transfer_history) | Information Schema | ✓ | | [DATABASE\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/database_refresh_history) | Information Schema | ✓ | | [DATABASE\_REFRESH\_PROGRESS , DATABASE\_REFRESH\_PROGRESS\_BY\_JOB](https://docs.snowflake.com/en/sql-reference/functions/database_refresh_progress) | Information Schema | ✓ | | [DATABASE\_REPLICATION\_USAGE\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/database_replication_usage_history) | Information Schema | ✓ | | [DATABASE\_STORAGE\_USAGE\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/database_storage_usage_history) | Information Schema | ✓ | | [DBT\_PROJECT\_EXECUTION\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/dbt_project_execution_history) | Information Schema | — | | [DYNAMIC\_TABLE\_GRAPH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/dynamic_table_graph_history) | Information Schema | ✓ | | [DYNAMIC\_TABLE\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/dynamic_table_refresh_history) | Information Schema | ✓ | | [DYNAMIC\_TABLES](https://docs.snowflake.com/en/sql-reference/functions/dynamic_tables) | Information Schema | — | | [EXTERNAL\_FUNCTIONS\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/external_functions_history) | Information Schema | ✓ | | [EXTERNAL\_TABLE\_FILE\_REGISTRATION\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/external_table_registration_history) | Information Schema | ✓ | | [EXTERNAL\_TABLE\_FILES](https://docs.snowflake.com/en/sql-reference/functions/external_table_files) | Information Schema | ✓ | | [LISTING\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/listing_refresh_history) | Information Schema | — | | [LOGIN\_HISTORY , LOGIN\_HISTORY\_BY\_USER](https://docs.snowflake.com/en/sql-reference/functions/login_history) | Information Schema | ✓ | | [MATERIALIZED\_VIEW\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/materialized_view_refresh_history) | Information Schema | ✓ | | [NETWORK\_RULE\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/network_rule_references) | Information Schema | — | | [NOTIFICATION\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/notification_history) | Information Schema | ✓ | | [ONLINE\_FEATURE\_TABLE\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/online-feature-table-refresh-history) | Information Schema | — | | [PIPE\_USAGE\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/pipe_usage_history) | Information Schema | ✓ | | [POLICY\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/policy_references) | Information Schema | ✓ | | [QUERY\_ACCELERATION\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/query_acceleration_history) | Information Schema | ✓ | | [QUERY\_HISTORY , QUERY\_HISTORY\_BY\_\*](https://docs.snowflake.com/en/sql-reference/functions/query_history) | Information Schema | ✓ | | [REPLICATION\_GROUP\_DANGLING\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/replication_group_dangling_references) | Information Schema | — | | [REPLICATION\_GROUP\_REFRESH\_HISTORY, REPLICATION\_GROUP\_REFRESH\_HISTORY\_ALL](https://docs.snowflake.com/en/sql-reference/functions/replication_group_refresh_history) | Information Schema | ✓ | | [REPLICATION\_GROUP\_REFRESH\_PROGRESS, REPLICATION\_GROUP\_REFRESH\_PROGRESS\_BY\_JOB, REPLICATION\_GROUP\_REFRESH\_PROGRESS\_ALL](https://docs.snowflake.com/en/sql-reference/functions/replication_group_refresh_progress) | Information Schema | ✓ | | [REPLICATION\_GROUP\_USAGE\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/replication_group_usage_history) | Information Schema | ✓ | | [REPLICATION\_USAGE\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/replication_usage_history) | Information Schema | ✓ | | [SEARCH\_OPTIMIZATION\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/search_optimization_history) | Information Schema | ✓ | | [SERVERLESS\_ALERT\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/serverless_alert_history) | Information Schema | — | | [SERVERLESS\_TASK\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/serverless_task_history) | Information Schema | ✓ | | [STAGE\_DIRECTORY\_FILE\_REGISTRATION\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/stage_directory_file_registration_history) | Information Schema | ✓ | | [STAGE\_STORAGE\_USAGE\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/stage_storage_usage_history) | Information Schema | ✓ | | [TAG\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/tag_references) | Information Schema | ✓ | | [TAG\_REFERENCES\_ALL\_COLUMNS](https://docs.snowflake.com/en/sql-reference/functions/tag_references_all_columns) | Information Schema | ✓ | | [TASK\_DEPENDENTS](https://docs.snowflake.com/en/sql-reference/functions/task_dependents) | Information Schema | ✓ | | [TASK\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/task_history) | Information Schema | ✓ | | [VALIDATE\_PIPE\_LOAD](https://docs.snowflake.com/en/sql-reference/functions/validate_pipe_load) | Information Schema | ✓ | | [WAREHOUSE\_LOAD\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/warehouse_load_history) | Information Schema | ✓ | | [WAREHOUSE\_METERING\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/warehouse_metering_history) | Information Schema | ✓ | | [DATA\_QUALITY\_MONITORING\_EXPECTATION\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/data_quality_monitoring_expectation_status) | LOCAL schema | — | | [DATA\_QUALITY\_MONITORING\_RESULTS](https://docs.snowflake.com/en/sql-reference/functions/data_quality_monitoring_results) | LOCAL schema | — | | [GENERATE\_COLUMN\_DESCRIPTION](https://docs.snowflake.com/en/sql-reference/functions/generate_column_description) | Metadata | ✓ | | [GET\_DDL](https://docs.snowflake.com/en/sql-reference/functions/get_ddl) | Metadata | ✓ | | [MODEL\_MONITOR\_DRIFT\_METRIC](https://docs.snowflake.com/en/sql-reference/functions/model-monitor-drift-metric) | Model monitor | — | | [MODEL\_MONITOR\_PERFORMANCE\_METRIC](https://docs.snowflake.com/en/sql-reference/functions/model-monitor-performance-metric) | Model monitor | — | | [MODEL\_MONITOR\_STAT\_METRIC](https://docs.snowflake.com/en/sql-reference/functions/model-monitor-stat-metric) | Model monitor | — | | [APPLICATION\_JSON](https://docs.snowflake.com/en/sql-reference/functions/application_json) | Notification | — | | [EMAIL\_INTEGRATION\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/email_integration_config) | Notification | — | | [INTEGRATION](https://docs.snowflake.com/en/sql-reference/functions/integration) | Notification | — | | [SANITIZE\_WEBHOOK\_CONTENT](https://docs.snowflake.com/en/sql-reference/functions/sanitize_webhook_content) | Notification | — | | [TEXT\_HTML](https://docs.snowflake.com/en/sql-reference/functions/text_html) | Notification | — | | [TEXT\_PLAIN](https://docs.snowflake.com/en/sql-reference/functions/text_plain) | Notification | — | | [ABS](https://docs.snowflake.com/en/sql-reference/functions/abs) | Numeric | ✓ | | [ACOS](https://docs.snowflake.com/en/sql-reference/functions/acos) | Numeric | ✓ | | [ACOSH](https://docs.snowflake.com/en/sql-reference/functions/acosh) | Numeric | ✓ | | [ASIN](https://docs.snowflake.com/en/sql-reference/functions/asin) | Numeric | ✓ | | [ASINH](https://docs.snowflake.com/en/sql-reference/functions/asinh) | Numeric | ✓ | | [ATAN](https://docs.snowflake.com/en/sql-reference/functions/atan) | Numeric | ✓ | | [ATAN2](https://docs.snowflake.com/en/sql-reference/functions/atan2) | Numeric | ✓ | | [ATANH](https://docs.snowflake.com/en/sql-reference/functions/atanh) | Numeric | ✓ | | [CBRT](https://docs.snowflake.com/en/sql-reference/functions/cbrt) | Numeric | ✓ | | [CEIL](https://docs.snowflake.com/en/sql-reference/functions/ceil) | Numeric | ✓ | | [COS](https://docs.snowflake.com/en/sql-reference/functions/cos) | Numeric | ✓ | | [COSH](https://docs.snowflake.com/en/sql-reference/functions/cosh) | Numeric | ✓ | | [COT](https://docs.snowflake.com/en/sql-reference/functions/cot) | Numeric | ✓ | | [DEGREES](https://docs.snowflake.com/en/sql-reference/functions/degrees) | Numeric | ✓ | | [DIV0](https://docs.snowflake.com/en/sql-reference/functions/div0) | Numeric | ✓ | | [DIV0NULL](https://docs.snowflake.com/en/sql-reference/functions/div0null) | Numeric | ✓ | | [EXP](https://docs.snowflake.com/en/sql-reference/functions/exp) | Numeric | ✓ | | [FACTORIAL](https://docs.snowflake.com/en/sql-reference/functions/factorial) | Numeric | ✓ | | [FLOOR](https://docs.snowflake.com/en/sql-reference/functions/floor) | Numeric | ✓ | | [LN](https://docs.snowflake.com/en/sql-reference/functions/ln) | Numeric | ✓ | | [LOG](https://docs.snowflake.com/en/sql-reference/functions/log) | Numeric | ✓ | | [MOD](https://docs.snowflake.com/en/sql-reference/functions/mod) | Numeric | — | | [PI](https://docs.snowflake.com/en/sql-reference/functions/pi) | Numeric | ✓ | | [POW, POWER](https://docs.snowflake.com/en/sql-reference/functions/pow) | Numeric | ✓ | | [RADIANS](https://docs.snowflake.com/en/sql-reference/functions/radians) | Numeric | ✓ | | [ROUND](https://docs.snowflake.com/en/sql-reference/functions/round) | Numeric | ✓ | | [SIGN](https://docs.snowflake.com/en/sql-reference/functions/sign) | Numeric | ✓ | | [SIN](https://docs.snowflake.com/en/sql-reference/functions/sin) | Numeric | ✓ | | [SINH](https://docs.snowflake.com/en/sql-reference/functions/sinh) | Numeric | ✓ | | [SQRT](https://docs.snowflake.com/en/sql-reference/functions/sqrt) | Numeric | ✓ | | [SQUARE](https://docs.snowflake.com/en/sql-reference/functions/square) | Numeric | ✓ | | [TAN](https://docs.snowflake.com/en/sql-reference/functions/tan) | Numeric | ✓ | | [TANH](https://docs.snowflake.com/en/sql-reference/functions/tanh) | Numeric | ✓ | | [TRUNCATE , TRUNC](https://docs.snowflake.com/en/sql-reference/functions/trunc) | Numeric | ✓ | | [WIDTH\_BUCKET](https://docs.snowflake.com/en/sql-reference/functions/width_bucket) | Numeric | ✓ | | [IS\_ORGANIZATION\_USER](https://docs.snowflake.com/en/sql-reference/functions/is_organization_user) | Organization user and organization user group | — | | [IS\_ORGANIZATION\_USER\_GROUP](https://docs.snowflake.com/en/sql-reference/functions/is_organization_user_group) | Organization user and organization user group | — | | [PERIOD\_BEGIN](https://docs.snowflake.com/en/sql-reference/functions/period_begin) | Period | — | | [PERIOD\_CONSTRUCT](https://docs.snowflake.com/en/sql-reference/functions/period_construct) | Period | — | | [PERIOD\_CONTAINS](https://docs.snowflake.com/en/sql-reference/functions/period_contains) | Period | — | | [PERIOD\_END](https://docs.snowflake.com/en/sql-reference/functions/period_end) | Period | — | | [PERIOD\_EQUALS](https://docs.snowflake.com/en/sql-reference/functions/period_equals) | Period | — | | [PERIOD\_IMMEDIATELY\_PRECEDES](https://docs.snowflake.com/en/sql-reference/functions/period_immediately_precedes) | Period | — | | [PERIOD\_IMMEDIATELY\_SUCCEEDS](https://docs.snowflake.com/en/sql-reference/functions/period_immediately_succeeds) | Period | — | | [PERIOD\_INTERSECT](https://docs.snowflake.com/en/sql-reference/functions/period_intersect) | Period | — | | [PERIOD\_LDIFF](https://docs.snowflake.com/en/sql-reference/functions/period_ldiff) | Period | — | | [PERIOD\_MEETS](https://docs.snowflake.com/en/sql-reference/functions/period_meets) | Period | — | | [PERIOD\_OVERLAPS](https://docs.snowflake.com/en/sql-reference/functions/period_overlaps) | Period | — | | [PERIOD\_PRECEDES](https://docs.snowflake.com/en/sql-reference/functions/period_precedes) | Period | — | | [PERIOD\_RDIFF](https://docs.snowflake.com/en/sql-reference/functions/period_rdiff) | Period | — | | [PERIOD\_SUCCEEDS](https://docs.snowflake.com/en/sql-reference/functions/period_succeeds) | Period | — | | [ARRAY\_APPEND](https://docs.snowflake.com/en/sql-reference/functions/array_append) | Semi-structured and structured data | ✓ | | [ARRAY\_CAT](https://docs.snowflake.com/en/sql-reference/functions/array_cat) | Semi-structured and structured data | ✓ | | [ARRAY\_COMPACT](https://docs.snowflake.com/en/sql-reference/functions/array_compact) | Semi-structured and structured data | ✓ | | [ARRAY\_CONSTRUCT](https://docs.snowflake.com/en/sql-reference/functions/array_construct) | Semi-structured and structured data | ✓ | | [ARRAY\_CONSTRUCT\_COMPACT](https://docs.snowflake.com/en/sql-reference/functions/array_construct_compact) | Semi-structured and structured data | ✓ | | [ARRAY\_CONSTRUCT\_STRUCTURED](https://docs.snowflake.com/en/sql-reference/functions/array_construct_structured) | Semi-structured and structured data | — | | [ARRAY\_CONTAINS](https://docs.snowflake.com/en/sql-reference/functions/array_contains) | Semi-structured and structured data | ✓ | | [ARRAY\_DISTINCT](https://docs.snowflake.com/en/sql-reference/functions/array_distinct) | Semi-structured and structured data | ✓ | | [ARRAY\_EXCEPT](https://docs.snowflake.com/en/sql-reference/functions/array_except) | Semi-structured and structured data | ✓ | | [ARRAY\_FLATTEN](https://docs.snowflake.com/en/sql-reference/functions/array_flatten) | Semi-structured and structured data | ✓ | | [ARRAY\_GENERATE\_RANGE](https://docs.snowflake.com/en/sql-reference/functions/array_generate_range) | Semi-structured and structured data | ✓ | | [ARRAY\_INSERT](https://docs.snowflake.com/en/sql-reference/functions/array_insert) | Semi-structured and structured data | ✓ | | [ARRAY\_INTERSECTION](https://docs.snowflake.com/en/sql-reference/functions/array_intersection) | Semi-structured and structured data | ✓ | | [ARRAY\_MAX](https://docs.snowflake.com/en/sql-reference/functions/array_max) | Semi-structured and structured data | ✓ | | [ARRAY\_MIN](https://docs.snowflake.com/en/sql-reference/functions/array_min) | Semi-structured and structured data | ✓ | | [ARRAY\_POSITION](https://docs.snowflake.com/en/sql-reference/functions/array_position) | Semi-structured and structured data | ✓ | | [ARRAY\_PREPEND](https://docs.snowflake.com/en/sql-reference/functions/array_prepend) | Semi-structured and structured data | ✓ | | [ARRAY\_REMOVE](https://docs.snowflake.com/en/sql-reference/functions/array_remove) | Semi-structured and structured data | ✓ | | [ARRAY\_REMOVE\_AT](https://docs.snowflake.com/en/sql-reference/functions/array_remove_at) | Semi-structured and structured data | ✓ | | [ARRAY\_REPEAT](https://docs.snowflake.com/en/sql-reference/functions/array_repeat) | Semi-structured and structured data | ✓ | | [ARRAY\_REVERSE](https://docs.snowflake.com/en/sql-reference/functions/array_reverse) | Semi-structured and structured data | ✓ | | [ARRAY\_SIZE](https://docs.snowflake.com/en/sql-reference/functions/array_size) | Semi-structured and structured data | ✓ | | [ARRAY\_SLICE](https://docs.snowflake.com/en/sql-reference/functions/array_slice) | Semi-structured and structured data | ✓ | | [ARRAY\_SORT](https://docs.snowflake.com/en/sql-reference/functions/array_sort) | Semi-structured and structured data | ✓ | | [ARRAY\_TO\_STRING](https://docs.snowflake.com/en/sql-reference/functions/array_to_string) | Semi-structured and structured data | ✓ | | [ARRAYS\_OVERLAP](https://docs.snowflake.com/en/sql-reference/functions/arrays_overlap) | Semi-structured and structured data | ✓ | | [ARRAYS\_TO\_OBJECT](https://docs.snowflake.com/en/sql-reference/functions/arrays_to_object) | Semi-structured and structured data | ✓ | | [ARRAYS\_ZIP](https://docs.snowflake.com/en/sql-reference/functions/arrays_zip) | Semi-structured and structured data | ✓ | | [AS\_*\<OBJECT\_TYPE>*](https://docs.snowflake.com/en/sql-reference/functions/as) | Semi-structured and structured data | — | | [AS\_ARRAY](https://docs.snowflake.com/en/sql-reference/functions/as_array) | Semi-structured and structured data | ✓ | | [AS\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/as_binary) | Semi-structured and structured data | ✓ | | [AS\_BOOLEAN](https://docs.snowflake.com/en/sql-reference/functions/as_boolean) | Semi-structured and structured data | ✓ | | [AS\_CHAR , AS\_VARCHAR](https://docs.snowflake.com/en/sql-reference/functions/as_char-varchar) | Semi-structured and structured data | ✓ | | [AS\_DATE](https://docs.snowflake.com/en/sql-reference/functions/as_date) | Semi-structured and structured data | ✓ | | [AS\_DECIMAL , AS\_NUMBER](https://docs.snowflake.com/en/sql-reference/functions/as_decimal-number) | Semi-structured and structured data | ✓ | | [AS\_DOUBLE , AS\_REAL](https://docs.snowflake.com/en/sql-reference/functions/as_double-real) | Semi-structured and structured data | ✓ | | [AS\_INTEGER](https://docs.snowflake.com/en/sql-reference/functions/as_integer) | Semi-structured and structured data | ✓ | | [AS\_OBJECT](https://docs.snowflake.com/en/sql-reference/functions/as_object) | Semi-structured and structured data | ✓ | | [AS\_TIME](https://docs.snowflake.com/en/sql-reference/functions/as_time) | Semi-structured and structured data | ✓ | | [AS\_TIMESTAMP\_\*](https://docs.snowflake.com/en/sql-reference/functions/as_timestamp) | Semi-structured and structured data | ✓ | | [CHECK\_JSON](https://docs.snowflake.com/en/sql-reference/functions/check_json) | Semi-structured and structured data | ✓ | | [CHECK\_XML](https://docs.snowflake.com/en/sql-reference/functions/check_xml) | Semi-structured and structured data | ✓ | | [FILTER](https://docs.snowflake.com/en/sql-reference/functions/filter) | Semi-structured and structured data | ✓ | | [GET](https://docs.snowflake.com/en/sql-reference/functions/get) | Semi-structured and structured data | ✓ | | [GET\_IGNORE\_CASE](https://docs.snowflake.com/en/sql-reference/functions/get_ignore_case) | Semi-structured and structured data | ✓ | | [GET\_PATH , :](https://docs.snowflake.com/en/sql-reference/functions/get_path) | Semi-structured and structured data | ✓ | | [IS\_*\<OBJECT\_TYPE>*](https://docs.snowflake.com/en/sql-reference/functions/is) | Semi-structured and structured data | — | | [IS\_ARRAY](https://docs.snowflake.com/en/sql-reference/functions/is_array) | Semi-structured and structured data | ✓ | | [IS\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/is_binary) | Semi-structured and structured data | ✓ | | [IS\_BOOLEAN](https://docs.snowflake.com/en/sql-reference/functions/is_boolean) | Semi-structured and structured data | ✓ | | [IS\_CHAR , IS\_VARCHAR](https://docs.snowflake.com/en/sql-reference/functions/is_char-varchar) | Semi-structured and structured data | ✓ | | [IS\_DATE , IS\_DATE\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/is_date-value) | Semi-structured and structured data | ✓ | | [IS\_DECIMAL](https://docs.snowflake.com/en/sql-reference/functions/is_decimal) | Semi-structured and structured data | ✓ | | [IS\_DOUBLE , IS\_REAL](https://docs.snowflake.com/en/sql-reference/functions/is_double-real) | Semi-structured and structured data | ✓ | | [IS\_INTEGER](https://docs.snowflake.com/en/sql-reference/functions/is_integer) | Semi-structured and structured data | ✓ | | [IS\_OBJECT](https://docs.snowflake.com/en/sql-reference/functions/is_object) | Semi-structured and structured data | ✓ | | [IS\_TIME](https://docs.snowflake.com/en/sql-reference/functions/is_time) | Semi-structured and structured data | ✓ | | [IS\_TIMESTAMP\_\*](https://docs.snowflake.com/en/sql-reference/functions/is_timestamp) | Semi-structured and structured data | ✓ | | [JSON\_EXTRACT\_PATH\_TEXT](https://docs.snowflake.com/en/sql-reference/functions/json_extract_path_text) | Semi-structured and structured data | ✓ | | [MAP\_CAT](https://docs.snowflake.com/en/sql-reference/functions/map_cat) | Semi-structured and structured data | ✓ | | [MAP\_CONSTRUCT](https://docs.snowflake.com/en/sql-reference/functions/map_construct) | Semi-structured and structured data | — | | [MAP\_CONTAINS\_KEY](https://docs.snowflake.com/en/sql-reference/functions/map_contains_key) | Semi-structured and structured data | ✓ | | [MAP\_DELETE](https://docs.snowflake.com/en/sql-reference/functions/map_delete) | Semi-structured and structured data | ✓ | | [MAP\_ENTRIES](https://docs.snowflake.com/en/sql-reference/functions/map_entries) | Semi-structured and structured data | — | | [MAP\_INSERT](https://docs.snowflake.com/en/sql-reference/functions/map_insert) | Semi-structured and structured data | ✓ | | [MAP\_KEYS](https://docs.snowflake.com/en/sql-reference/functions/map_keys) | Semi-structured and structured data | ✓ | | [MAP\_PICK](https://docs.snowflake.com/en/sql-reference/functions/map_pick) | Semi-structured and structured data | ✓ | | [MAP\_SIZE](https://docs.snowflake.com/en/sql-reference/functions/map_size) | Semi-structured and structured data | ✓ | | [OBJECT\_CONSTRUCT](https://docs.snowflake.com/en/sql-reference/functions/object_construct) | Semi-structured and structured data | ✓ | | [OBJECT\_CONSTRUCT\_KEEP\_NULL](https://docs.snowflake.com/en/sql-reference/functions/object_construct_keep_null) | Semi-structured and structured data | ✓ | | [OBJECT\_CONSTRUCT\_KEEP\_NULL\_STRUCTURED](https://docs.snowflake.com/en/sql-reference/functions/object_construct_keep_null_structured) | Semi-structured and structured data | — | | [OBJECT\_DELETE](https://docs.snowflake.com/en/sql-reference/functions/object_delete) | Semi-structured and structured data | ✓ | | [OBJECT\_INSERT](https://docs.snowflake.com/en/sql-reference/functions/object_insert) | Semi-structured and structured data | ✓ | | [OBJECT\_KEYS](https://docs.snowflake.com/en/sql-reference/functions/object_keys) | Semi-structured and structured data | ✓ | | [OBJECT\_PICK](https://docs.snowflake.com/en/sql-reference/functions/object_pick) | Semi-structured and structured data | ✓ | | [PARSE\_JSON](https://docs.snowflake.com/en/sql-reference/functions/parse_json) | Semi-structured and structured data | ✓ | | [PARSE\_XML](https://docs.snowflake.com/en/sql-reference/functions/parse_xml) | Semi-structured and structured data | ✓ | | [PROMPT](https://docs.snowflake.com/en/sql-reference/functions/prompt) | Semi-structured and structured data | ✓ | | [REDUCE](https://docs.snowflake.com/en/sql-reference/functions/reduce) | Semi-structured and structured data | ✓ | | [STRIP\_NULL\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/strip_null_value) | Semi-structured and structured data | ✓ | | [TRANSFORM](https://docs.snowflake.com/en/sql-reference/functions/transform) | Semi-structured and structured data | ✓ | | [TRY\_PARSE\_JSON](https://docs.snowflake.com/en/sql-reference/functions/try_parse_json) | Semi-structured and structured data | ✓ | | [TYPEOF](https://docs.snowflake.com/en/sql-reference/functions/typeof) | Semi-structured and structured data | ✓ | | [XMLGET](https://docs.snowflake.com/en/sql-reference/functions/xmlget) | Semi-structured and structured data | ✓ | | [\<SERVICE\_NAME>!SPCS\_WAIT\_FOR](https://docs.snowflake.com/en/sql-reference/functions/spcs_wait_for) | Snowpark Container Services | — | | [\[ NOT \] ILIKE](https://docs.snowflake.com/en/sql-reference/functions/ilike) | String & binary | ✓ | | [\[ NOT \] LIKE](https://docs.snowflake.com/en/sql-reference/functions/like) | String & binary | ✓ | | [AGENT\_RUN (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/agent_run-snowflake-cortex) | String & binary | — | | [AI\_CLASSIFY](https://docs.snowflake.com/en/sql-reference/functions/ai_classify) | String & binary | ✓ | | [AI\_COMPLETE](https://docs.snowflake.com/en/sql-reference/functions/ai_complete) | String & binary | ✓ | | [AI\_COMPLETE (PROMPT OBJECT)](https://docs.snowflake.com/en/sql-reference/functions/ai_complete-prompt-object) | String & binary | ✓ | | [AI\_COMPLETE (SINGLE IMAGE)](https://docs.snowflake.com/en/sql-reference/functions/ai_complete-single-file) | String & binary | ✓ | | [AI\_COMPLETE (SINGLE STRING)](https://docs.snowflake.com/en/sql-reference/functions/ai_complete-single-string) | String & binary | ✓ | | [AI\_COMPLETE (STRUCTURED OUTPUTS)](https://docs.snowflake.com/en/sql-reference/functions/ai_complete-structured-outputs) | String & binary | ✓ | | [AI\_COUNT\_TOKENS](https://docs.snowflake.com/en/sql-reference/functions/ai_count_tokens) | String & binary | ✓ | | [AI\_EMBED](https://docs.snowflake.com/en/sql-reference/functions/ai_embed) | String & binary | ✓ | | [AI\_EXTRACT](https://docs.snowflake.com/en/sql-reference/functions/ai_extract) | String & binary | ✓ | | [AI\_EXTRACT (DOCUMENT AI LEGACY MODELS)](https://docs.snowflake.com/en/sql-reference/functions/ai_extract-document-ai) | String & binary | ✓ | | [AI\_FILTER](https://docs.snowflake.com/en/sql-reference/functions/ai_filter) | String & binary | ✓ | | [AI\_MULTI\_EMBED](https://docs.snowflake.com/en/sql-reference/functions/ai_multi_embed) | String & binary | — | | [AI\_PARSE\_DOCUMENT](https://docs.snowflake.com/en/sql-reference/functions/ai_parse_document) | String & binary | ✓ | | [AI\_REDACT](https://docs.snowflake.com/en/sql-reference/functions/ai_redact) | String & binary | ✓ | | [AI\_SENTIMENT](https://docs.snowflake.com/en/sql-reference/functions/ai_sentiment) | String & binary | ✓ | | [AI\_SIMILARITY](https://docs.snowflake.com/en/sql-reference/functions/ai_similarity) | String & binary | ✓ | | [AI\_SUMMARIZE](https://docs.snowflake.com/en/sql-reference/functions/ai_summarize) | String & binary | — | | [AI\_TRANSLATE](https://docs.snowflake.com/en/sql-reference/functions/ai_translate) | String & binary | ✓ | | [ASCII](https://docs.snowflake.com/en/sql-reference/functions/ascii) | String & binary | ✓ | | [BASE64\_DECODE\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/base64_decode_binary) | String & binary | ✓ | | [BASE64\_DECODE\_STRING](https://docs.snowflake.com/en/sql-reference/functions/base64_decode_string) | String & binary | ✓ | | [BASE64\_ENCODE](https://docs.snowflake.com/en/sql-reference/functions/base64_encode) | String & binary | ✓ | | [BIT\_LENGTH](https://docs.snowflake.com/en/sql-reference/functions/bit_length) | String & binary | ✓ | | [CHARINDEX](https://docs.snowflake.com/en/sql-reference/functions/charindex) | String & binary | ✓ | | [CHR , CHAR](https://docs.snowflake.com/en/sql-reference/functions/chr) | String & binary | ✓ | | [CLASSIFY\_TEXT (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/classify_text-snowflake-cortex) | String & binary | ✓ | | [COLLATE](https://docs.snowflake.com/en/sql-reference/functions/collate) | String & binary | ✓ | | [COLLATION](https://docs.snowflake.com/en/sql-reference/functions/collation) | String & binary | ✓ | | [COMPLETE (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex) | String & binary | ✓ | | [COMPLETE (SNOWFLAKE.CORTEX) (MULTIMODAL)](https://docs.snowflake.com/en/sql-reference/functions/complete-snowflake-cortex-multimodal) | String & binary | — | | [COMPRESS](https://docs.snowflake.com/en/sql-reference/functions/compress) | String & binary | ✓ | | [CONCAT , ||](https://docs.snowflake.com/en/sql-reference/functions/concat) | String & binary | ✓ | | [CONCAT\_WS](https://docs.snowflake.com/en/sql-reference/functions/concat_ws) | String & binary | ✓ | | [CONTAINS](https://docs.snowflake.com/en/sql-reference/functions/contains) | String & binary | ✓ | | [COUNT\_TOKENS (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/count_tokens-snowflake-cortex) | String & binary | ✓ | | [DATA\_AGENT\_RUN (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/data_agent_run-snowflake-cortex) | String & binary | ✓ | | [DECOMPRESS\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/decompress_binary) | String & binary | ✓ | | [DECOMPRESS\_STRING](https://docs.snowflake.com/en/sql-reference/functions/decompress_string) | String & binary | ✓ | | [EDITDISTANCE](https://docs.snowflake.com/en/sql-reference/functions/editdistance) | String & binary | ✓ | | [EMBED\_TEXT\_1024 (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/embed_text_1024-snowflake-cortex) | String & binary | ✓ | | [EMBED\_TEXT\_768 (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/embed_text-snowflake-cortex) | String & binary | — | | [ENDSWITH](https://docs.snowflake.com/en/sql-reference/functions/endswith) | String & binary | ✓ | | [ENTITY\_SENTIMENT (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/entity_sentiment-snowflake-cortex) | String & binary | ✓ | | [EVALUATE AN AI FUNCTION (AI\_FUNCTION\_EVALUATION)](https://docs.snowflake.com/en/sql-reference/functions/ai_function_evaluation) | String & binary | — | | [EXTRACT\_ANSWER (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/extract_answer-snowflake-cortex) | String & binary | ✓ | | [FINETUNE (‘CANCEL’) (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/finetune-cancel) | String & binary | — | | [FINETUNE (‘CREATE’) (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/finetune-create) | String & binary | — | | [FINETUNE (‘DESCRIBE’) (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/finetune-describe) | String & binary | — | | [FINETUNE (‘SHOW’) (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/finetune-show) | String & binary | ✓ | | [FINETUNE (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/finetune-snowflake-cortex) | String & binary | ✓ | | [FORMAT\_STRING\_TEMPLATE](https://docs.snowflake.com/en/sql-reference/functions/format_string_template) | String & binary | — | | [HEX\_DECODE\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/hex_decode_binary) | String & binary | ✓ | | [HEX\_DECODE\_STRING](https://docs.snowflake.com/en/sql-reference/functions/hex_decode_string) | String & binary | ✓ | | [HEX\_ENCODE](https://docs.snowflake.com/en/sql-reference/functions/hex_encode) | String & binary | ✓ | | [ILIKE ANY](https://docs.snowflake.com/en/sql-reference/functions/ilike_any) | String & binary | ✓ | | [INITCAP](https://docs.snowflake.com/en/sql-reference/functions/initcap) | String & binary | ✓ | | [INSERT](https://docs.snowflake.com/en/sql-reference/functions/insert) | String & binary | ✓ | | [JAROWINKLER\_SIMILARITY](https://docs.snowflake.com/en/sql-reference/functions/jarowinkler_similarity) | String & binary | ✓ | | [LEFT](https://docs.snowflake.com/en/sql-reference/functions/left) | String & binary | ✓ | | [LENGTH, LEN](https://docs.snowflake.com/en/sql-reference/functions/length) | String & binary | ✓ | | [LIKE ALL](https://docs.snowflake.com/en/sql-reference/functions/like_all) | String & binary | ✓ | | [LIKE ANY](https://docs.snowflake.com/en/sql-reference/functions/like_any) | String & binary | ✓ | | [LOWER](https://docs.snowflake.com/en/sql-reference/functions/lower) | String & binary | ✓ | | [LPAD](https://docs.snowflake.com/en/sql-reference/functions/lpad) | String & binary | ✓ | | [LTRIM](https://docs.snowflake.com/en/sql-reference/functions/ltrim) | String & binary | ✓ | | [MD5 , MD5\_HEX](https://docs.snowflake.com/en/sql-reference/functions/md5) | String & binary | ✓ | | [MD5\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/md5_binary) | String & binary | ✓ | | [MD5\_NUMBER — OBSOLETED](https://docs.snowflake.com/en/sql-reference/functions/md5_number) | String & binary | — | | [MD5\_NUMBER\_LOWER64](https://docs.snowflake.com/en/sql-reference/functions/md5_number_lower64) | String & binary | ✓ | | [MD5\_NUMBER\_UPPER64](https://docs.snowflake.com/en/sql-reference/functions/md5_number_upper64) | String & binary | ✓ | | [OCTET\_LENGTH](https://docs.snowflake.com/en/sql-reference/functions/octet_length) | String & binary | ✓ | | [OPTIMIZE AN AI FUNCTION (AI\_FUNCTION\_OPTIMIZATION)](https://docs.snowflake.com/en/sql-reference/functions/ai_function_optimization) | String & binary | — | | [PARSE\_IP](https://docs.snowflake.com/en/sql-reference/functions/parse_ip) | String & binary | ✓ | | [PARSE\_URL](https://docs.snowflake.com/en/sql-reference/functions/parse_url) | String & binary | ✓ | | [POSITION](https://docs.snowflake.com/en/sql-reference/functions/position) | String & binary | ✓ | | [REPEAT](https://docs.snowflake.com/en/sql-reference/functions/repeat) | String & binary | ✓ | | [REPLACE](https://docs.snowflake.com/en/sql-reference/functions/replace) | String & binary | ✓ | | [REVERSE](https://docs.snowflake.com/en/sql-reference/functions/reverse) | String & binary | ✓ | | [RIGHT](https://docs.snowflake.com/en/sql-reference/functions/right) | String & binary | ✓ | | [RPAD](https://docs.snowflake.com/en/sql-reference/functions/rpad) | String & binary | ✓ | | [RTRIM](https://docs.snowflake.com/en/sql-reference/functions/rtrim) | String & binary | ✓ | | [RTRIMMED\_LENGTH](https://docs.snowflake.com/en/sql-reference/functions/rtrimmed_length) | String & binary | ✓ | | [SEARCH](https://docs.snowflake.com/en/sql-reference/functions/search) | String & binary | ✓ | | [SEARCH\_IP](https://docs.snowflake.com/en/sql-reference/functions/search_ip) | String & binary | — | | [SEARCH\_PREVIEW (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/search_preview-snowflake-cortex) | String & binary | — | | [SENTIMENT (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/sentiment-snowflake-cortex) | String & binary | ✓ | | [SHA1 , SHA1\_HEX](https://docs.snowflake.com/en/sql-reference/functions/sha1) | String & binary | ✓ | | [SHA1\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/sha1_binary) | String & binary | ✓ | | [SHA2 , SHA2\_HEX](https://docs.snowflake.com/en/sql-reference/functions/sha2) | String & binary | ✓ | | [SHA2\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/sha2_binary) | String & binary | ✓ | | [SOUNDEX](https://docs.snowflake.com/en/sql-reference/functions/soundex) | String & binary | ✓ | | [SOUNDEX\_P123](https://docs.snowflake.com/en/sql-reference/functions/soundex_p123) | String & binary | ✓ | | [SPACE](https://docs.snowflake.com/en/sql-reference/functions/space) | String & binary | ✓ | | [SPLIT](https://docs.snowflake.com/en/sql-reference/functions/split) | String & binary | ✓ | | [SPLIT\_PART](https://docs.snowflake.com/en/sql-reference/functions/split_part) | String & binary | ✓ | | [SPLIT\_TEXT\_MARKDOWN\_HEADER (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/split_text_markdown_header-snowflake-cortex) | String & binary | ✓ | | [SPLIT\_TEXT\_RECURSIVE\_CHARACTER (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/split_text_recursive_character-snowflake-cortex) | String & binary | ✓ | | [SPLIT\_TO\_TABLE](https://docs.snowflake.com/en/sql-reference/functions/split_to_table) | String & binary | ✓ | | [STARTSWITH](https://docs.snowflake.com/en/sql-reference/functions/startswith) | String & binary | ✓ | | [STRTOK](https://docs.snowflake.com/en/sql-reference/functions/strtok) | String & binary | ✓ | | [STRTOK\_SPLIT\_TO\_TABLE](https://docs.snowflake.com/en/sql-reference/functions/strtok_split_to_table) | String & binary | ✓ | | [STRTOK\_TO\_ARRAY](https://docs.snowflake.com/en/sql-reference/functions/strtok_to_array) | String & binary | ✓ | | [SUBSTR , SUBSTRING](https://docs.snowflake.com/en/sql-reference/functions/substr) | String & binary | ✓ | | [THREAD\_MESSAGES (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/thread_messages-snowflake-cortex) | String & binary | — | | [TRANSLATE](https://docs.snowflake.com/en/sql-reference/functions/translate) | String & binary | ✓ | | [TRANSLATE (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/translate-snowflake-cortex) | String & binary | ✓ | | [TRIM](https://docs.snowflake.com/en/sql-reference/functions/trim) | String & binary | ✓ | | [TRY\_BASE64\_DECODE\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/try_base64_decode_binary) | String & binary | ✓ | | [TRY\_BASE64\_DECODE\_STRING](https://docs.snowflake.com/en/sql-reference/functions/try_base64_decode_string) | String & binary | ✓ | | [TRY\_COMPLETE (SNOWFLAKE.CORTEX)](https://docs.snowflake.com/en/sql-reference/functions/try_complete-snowflake-cortex) | String & binary | ✓ | | [TRY\_HEX\_DECODE\_BINARY](https://docs.snowflake.com/en/sql-reference/functions/try_hex_decode_binary) | String & binary | ✓ | | [TRY\_HEX\_DECODE\_STRING](https://docs.snowflake.com/en/sql-reference/functions/try_hex_decode_string) | String & binary | ✓ | | [UNICODE](https://docs.snowflake.com/en/sql-reference/functions/unicode) | String & binary | ✓ | | [UPPER](https://docs.snowflake.com/en/sql-reference/functions/upper) | String & binary | ✓ | | [UUID\_STRING](https://docs.snowflake.com/en/sql-reference/functions/uuid_string) | String & binary | ✓ | | [\[ NOT \] REGEXP](https://docs.snowflake.com/en/sql-reference/functions/regexp) | String functions (regular expressions) | ✓ | | [\[ NOT \] RLIKE](https://docs.snowflake.com/en/sql-reference/functions/rlike) | String functions (regular expressions) | ✓ | | [REGEXP\_COUNT](https://docs.snowflake.com/en/sql-reference/functions/regexp_count) | String functions (regular expressions) | ✓ | | [REGEXP\_INSTR](https://docs.snowflake.com/en/sql-reference/functions/regexp_instr) | String functions (regular expressions) | ✓ | | [REGEXP\_LIKE](https://docs.snowflake.com/en/sql-reference/functions/regexp_like) | String functions (regular expressions) | ✓ | | [REGEXP\_REPLACE](https://docs.snowflake.com/en/sql-reference/functions/regexp_replace) | String functions (regular expressions) | ✓ | | [REGEXP\_SUBSTR](https://docs.snowflake.com/en/sql-reference/functions/regexp_substr) | String functions (regular expressions) | ✓ | | [REGEXP\_SUBSTR\_ALL](https://docs.snowflake.com/en/sql-reference/functions/regexp_substr_all) | String functions (regular expressions) | ✓ | | [EXECUTE\_AI\_EVALUATION](https://docs.snowflake.com/en/sql-reference/functions/execute_ai_evaluation) | System | — | | [EXPLAIN\_GRANTABLE\_PRIVILEGES](https://docs.snowflake.com/en/sql-reference/functions/explain_grantable_privileges) | System | — | | [EXPLAIN\_JSON](https://docs.snowflake.com/en/sql-reference/functions/explain_json) | System | — | | [EXPLAIN\_PRIVILEGES](https://docs.snowflake.com/en/sql-reference/functions/explain_privileges) | System | — | | [EXTRACT\_SEMANTIC\_CATEGORIES](https://docs.snowflake.com/en/sql-reference/functions/extract_semantic_categories) | System | — | | [GET\_ANACONDA\_PACKAGES\_REPODATA](https://docs.snowflake.com/en/sql-reference/functions/get_anaconda_packages_repodata) | System | — | | [GET\_PYTHON\_PROFILER\_OUTPUT (SNOWFLAKE.CORE)](https://docs.snowflake.com/en/sql-reference/functions/get_python_profiler_output) | System | — | | [GET\_QUERY\_OPERATOR\_STATS](https://docs.snowflake.com/en/sql-reference/functions/get_query_operator_stats) | System | ✓ | | [SHOW\_PYTHON\_PACKAGES\_DEPENDENCIES](https://docs.snowflake.com/en/sql-reference/functions/show_python_packages_dependencies) | System | — | | [SYSTEM$ABORT\_SESSION](https://docs.snowflake.com/en/sql-reference/functions/system_abort_session) | System | ✓ | | [SYSTEM$ABORT\_TRANSACTION](https://docs.snowflake.com/en/sql-reference/functions/system_abort_transaction) | System | ✓ | | [SYSTEM$ACTIVATE\_CMK\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_activate_cmk_info) | System | — | | [SYSTEM$ACTIVATE\_CMK\_INFO\_POSTGRES](https://docs.snowflake.com/en/sql-reference/functions/system_activate_cmk_info_postgres) | System | — | | [SYSTEM$ADD\_EVENT (FOR SNOWFLAKE SCRIPTING)](https://docs.snowflake.com/en/sql-reference/functions/system_add_event) | System | — | | [SYSTEM$ADD\_PRIVATELINK\_ENDPOINT\_HOSTNAME](https://docs.snowflake.com/en/sql-reference/functions/system_add_privatelink_endpoint_hostname) | System | — | | [SYSTEM$ADD\_REFERENCE](https://docs.snowflake.com/en/sql-reference/functions/system_add_reference) | System | ✓ | | [SYSTEM$ALLOWLIST](https://docs.snowflake.com/en/sql-reference/functions/system_allowlist) | System | — | | [SYSTEM$ALLOWLIST\_PRIVATELINK](https://docs.snowflake.com/en/sql-reference/functions/system_allowlist_privatelink) | System | — | | [SYSTEM$APP\_COMPATIBILITY\_CHECK](https://docs.snowflake.com/en/sql-reference/functions/system_app_compatibility_check) | System | — | | [SYSTEM$APPLICATION\_GET\_LOG\_LEVEL](https://docs.snowflake.com/en/sql-reference/functions/system_application_get_log_level) | System | — | | [SYSTEM$APPLICATION\_GET\_METRIC\_LEVEL](https://docs.snowflake.com/en/sql-reference/functions/system_application_get_metric_level) | System | — | | [SYSTEM$APPLICATION\_GET\_TRACE\_LEVEL](https://docs.snowflake.com/en/sql-reference/functions/system_application_get_trace_level) | System | — | | [SYSTEM$AUTHORIZE\_PRIVATELINK](https://docs.snowflake.com/en/sql-reference/functions/system_authorize_privatelink) | System | — | | [SYSTEM$AUTHORIZE\_SNOWFLAKE\_MANAGED\_STORAGE\_VOLUME\_PRIVATELINK\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_authorize_snowflake_managed_storage_volume_privatelink_access) | System | — | | [SYSTEM$AUTHORIZE\_STAGE\_PRIVATELINK\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_authorize_stage_privatelink_access) | System | — | | [SYSTEM$AUTO\_REFRESH\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_auto_refresh_status) | System | ✓ | | [SYSTEM$BEGIN\_DEBUG\_APPLICATION](https://docs.snowflake.com/en/sql-reference/functions/system_begin_debug_application) | System | — | | [SYSTEM$BEHAVIOR\_CHANGE\_BUNDLE\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_behavior_change_bundle_status) | System | ✓ | | [SYSTEM$BLOCK\_INTERNAL\_STAGES\_PUBLIC\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_block_internal_stages_public_access) | System | — | | [SYSTEM$BLOCK\_INTERNAL\_STAGES\_PUBLIC\_ACCESS\_WITH\_EXCEPTION](https://docs.snowflake.com/en/sql-reference/functions/system_block_internal_stages_public_access_with_exception) | System | — | | [SYSTEM$BLOCK\_SNOWFLAKE\_MANAGED\_STORAGE\_VOLUME\_PUBLIC\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_block_snowflake_managed_storage_volume_public_access) | System | — | | [SYSTEM$BLOCK\_SNOWFLAKE\_MANAGED\_STORAGE\_VOLUME\_PUBLIC\_ACCESS\_WITH\_EXCEPTION](https://docs.snowflake.com/en/sql-reference/functions/system_block_snowflake_managed_storage_volume_public_access_with_exception) | System | — | | [SYSTEM$CANCEL\_ALL\_QUERIES](https://docs.snowflake.com/en/sql-reference/functions/system_cancel_all_queries) | System | ✓ | | [SYSTEM$CANCEL\_QUERY](https://docs.snowflake.com/en/sql-reference/functions/system_cancel_query) | System | ✓ | | [SYSTEM$CAST\_MULTI\_PARTY\_APPROVAL\_VOTE](https://docs.snowflake.com/en/sql-reference/functions/system_cast_multi_party_approval_vote) | System | — | | [SYSTEM$CATALOG\_LINK\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_catalog_link_status) | System | — | | [SYSTEM$CKE\_HASH\_FUNCTION](https://docs.snowflake.com/en/sql-reference/functions/system_cke_hash_function) | System | — | | [SYSTEM$CLEANUP\_DATABASE\_ROLE\_GRANTS](https://docs.snowflake.com/en/sql-reference/functions/system_cleanup_database_role_grants) | System | — | | [SYSTEM$CLIENT\_VERSION\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_client_version_info) | System | — | | [SYSTEM$CLIENT\_VULNERABILITY\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_client_vulnerability_info) | System | — | | [SYSTEM$CLUSTERING\_DEPTH](https://docs.snowflake.com/en/sql-reference/functions/system_clustering_depth) | System | ✓ | | [SYSTEM$CLUSTERING\_INFORMATION](https://docs.snowflake.com/en/sql-reference/functions/system_clustering_information) | System | ✓ | | [SYSTEM$CLUSTERING\_RATIO — DEPRECATED](https://docs.snowflake.com/en/sql-reference/functions/system_clustering_ratio) | System | ✓ | | [SYSTEM$COMMIT\_MOVE\_ORGANIZATION\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_commit_move_organization_account) | System | — | | [SYSTEM$CONVERT\_PIPES\_SQS\_TO\_SNS](https://docs.snowflake.com/en/sql-reference/functions/system_convert_pipes_sqs_to_sns) | System | ✓ | | [SYSTEM$CREATE\_BILLING\_EVENT](https://docs.snowflake.com/en/sql-reference/functions/system_create_billing_event) | System | — | | [SYSTEM$CREATE\_BILLING\_EVENTS](https://docs.snowflake.com/en/sql-reference/functions/system_create_billing_events) | System | — | | [SYSTEM$CREATE\_EVALUATION\_DATASET](https://docs.snowflake.com/en/sql-reference/functions/system_create_evaluation_dataset) | System | — | | [SYSTEM$CURRENT\_USER\_TASK\_NAME](https://docs.snowflake.com/en/sql-reference/functions/system_current_user_task_name) | System | — | | [SYSTEM$DATA\_METRIC\_SCAN](https://docs.snowflake.com/en/sql-reference/functions/system_data_metric_scan) | System | — | | [SYSTEM$DATABASE\_REFRESH\_HISTORY — DEPRECATED](https://docs.snowflake.com/en/sql-reference/functions/system_database_refresh_history) | System | — | | [SYSTEM$DATABASE\_REFRESH\_PROGRESS , SYSTEM$DATABASE\_REFRESH\_PROGRESS\_BY\_JOB — DEPRECATED](https://docs.snowflake.com/en/sql-reference/functions/system_database_refresh_progress) | System | — | | [SYSTEM$DBT\_GET\_LAST\_FAILED\_RUN\_TARGET](https://docs.snowflake.com/en/sql-reference/functions/system_dbt_get_last_failed_run_target) | System | — | | [SYSTEM$DBT\_GET\_LAST\_RUN\_TARGET](https://docs.snowflake.com/en/sql-reference/functions/system_dbt_get_last_run_target) | System | — | | [SYSTEM$DBT\_GET\_LAST\_SUCCESSFUL\_RUN\_TARGET](https://docs.snowflake.com/en/sql-reference/functions/system_dbt_get_last_successful_run_target) | System | — | | [SYSTEM$DEACTIVATE\_CMK\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_deactivate_cmk_info) | System | — | | [SYSTEM$DECODE\_PAT](https://docs.snowflake.com/en/sql-reference/functions/system_decode_pat) | System | — | | [SYSTEM$DEMIGRATE\_DBT\_PROJECT](https://docs.snowflake.com/en/sql-reference/functions/system_demigrate_dbt_project) | System | — | | [SYSTEM$DEPROVISION\_PRIVATELINK\_ENDPOINT](https://docs.snowflake.com/en/sql-reference/functions/system_deprovision_privatelink_endpoint) | System | — | | [SYSTEM$DEPROVISION\_PRIVATELINK\_ENDPOINT\_TSS](https://docs.snowflake.com/en/sql-reference/functions/system_deprovision_privatelink_endpoint_tss) | System | — | | [SYSTEM$DEREGISTER\_CMK\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_deregister_cmk_info) | System | — | | [SYSTEM$DEREGISTER\_CMK\_INFO\_POSTGRES](https://docs.snowflake.com/en/sql-reference/functions/system_deregister_cmk_info_postgres) | System | — | | [SYSTEM$DESC\_ICEBERG\_ACCESS\_IDENTITY](https://docs.snowflake.com/en/sql-reference/functions/system_desc_iceberg_access_identity) | System | — | | [SYSTEM$DISABLE\_BEHAVIOR\_CHANGE\_BUNDLE](https://docs.snowflake.com/en/sql-reference/functions/system_disable_behavior_change_bundle) | System | — | | [SYSTEM$DISABLE\_DATABASE\_REPLICATION](https://docs.snowflake.com/en/sql-reference/functions/system_disable_database_replication) | System | ✓ | | [SYSTEM$DISABLE\_FEATURE\_GROUPS](https://docs.snowflake.com/en/sql-reference/functions/system_disable_feature_groups) | System | — | | [SYSTEM$DISABLE\_FEATURE\_GROUPS\_PREVIEW](https://docs.snowflake.com/en/sql-reference/functions/system_disable_feature_groups_preview) | System | — | | [SYSTEM$DISABLE\_GLOBAL\_DATA\_SHARING\_FOR\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_disable_global_data_sharing_for_account) | System | — | | [SYSTEM$DISABLE\_PREVIEW\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_disable_preview_access) | System | — | | [SYSTEM$DISABLE\_PRIVATELINK\_ACCESS\_ONLY](https://docs.snowflake.com/en/sql-reference/functions/system_disable_privatelink_access_only) | System | — | | [SYSTEM$ENABLE\_BEHAVIOR\_CHANGE\_BUNDLE](https://docs.snowflake.com/en/sql-reference/functions/system_enable_behavior_change_bundle) | System | — | | [SYSTEM$ENABLE\_FEATURE\_GROUPS](https://docs.snowflake.com/en/sql-reference/functions/system_enable_feature_groups) | System | — | | [SYSTEM$ENABLE\_GLOBAL\_DATA\_SHARING\_FOR\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_enable_global_data_sharing_for_account) | System | — | | [SYSTEM$ENABLE\_PREVIEW\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_enable_preview_access) | System | — | | [SYSTEM$ENCODE\_CKE\_PRIMARY\_KEY](https://docs.snowflake.com/en/sql-reference/functions/system_encode_cke_primary_key) | System | — | | [SYSTEM$END\_DEBUG\_APPLICATION](https://docs.snowflake.com/en/sql-reference/functions/system_end_debug_application) | System | — | | [SYSTEM$ENFORCE\_PRIVATELINK\_ACCESS\_ONLY](https://docs.snowflake.com/en/sql-reference/functions/system_enforce_privatelink_access_only) | System | — | | [SYSTEM$ESTIMATE\_AUTOMATIC\_CLUSTERING\_COSTS](https://docs.snowflake.com/en/sql-reference/functions/system_estimate_automatic_clustering_costs) | System | — | | [SYSTEM$ESTIMATE\_QUERY\_ACCELERATION](https://docs.snowflake.com/en/sql-reference/functions/system_estimate_query_acceleration) | System | ✓ | | [SYSTEM$ESTIMATE\_SEARCH\_OPTIMIZATION\_COSTS](https://docs.snowflake.com/en/sql-reference/functions/system_estimate_search_optimization_costs) | System | ✓ | | [SYSTEM$EVALUATE\_DATA\_QUALITY\_EXPECTATIONS](https://docs.snowflake.com/en/sql-reference/functions/system_evaluate_data_quality_expectations) | System | — | | [SYSTEM$EVALUATE\_DATA\_QUALITY\_EXPECTATIONS\_PERSIST\_RESULT](https://docs.snowflake.com/en/sql-reference/functions/system_evaluate_data_quality_expectations_persist_result) | System | ✓ | | [SYSTEM$EXECUTE\_CATALOG\_OPERATION](https://docs.snowflake.com/en/sql-reference/functions/system_execute_catalog_operation) | System | — | | [SYSTEM$EXPLAIN\_JSON\_TO\_TEXT](https://docs.snowflake.com/en/sql-reference/functions/system_explain_json_to_text) | System | ✓ | | [SYSTEM$EXPLAIN\_PLAN\_JSON](https://docs.snowflake.com/en/sql-reference/functions/system_explain_plan_json) | System | ✓ | | [SYSTEM$EXPORT\_TDS\_FROM\_SEMANTIC\_VIEW](https://docs.snowflake.com/en/sql-reference/functions/system_export_tds_from_semantic_view) | System | — | | [SYSTEM$EXTERNAL\_TABLE\_PIPE\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_external_table_pipe_status) | System | ✓ | | [SYSTEM$FETCH\_EXTERNAL\_SECRET\_FROM\_INTEGRATION](https://docs.snowflake.com/en/sql-reference/functions/system_fetch_external_secret_from_integration) | System | — | | [SYSTEM$FINISH\_OAUTH\_FLOW](https://docs.snowflake.com/en/sql-reference/functions/system_finish_oauth_flow) | System | — | | [SYSTEM$GENERATE\_SAML\_CSR](https://docs.snowflake.com/en/sql-reference/functions/system_generate_saml_csr) | System | ✓ | | [SYSTEM$GENERATE\_SCIM\_ACCESS\_TOKEN](https://docs.snowflake.com/en/sql-reference/functions/system_generate_scim_access_token) | System | — | | [SYSTEM$GET\_ALERT\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_alert_config) | System | — | | [SYSTEM$GET\_ALERT\_TEMPLATE](https://docs.snowflake.com/en/sql-reference/functions/system_get_alert_template) | System | — | | [SYSTEM$GET\_ALL\_DEFAULT\_COLUMNS\_OVERRIDES](https://docs.snowflake.com/en/sql-reference/functions/system_get_all_default_columns_overrides) | System | — | | [SYSTEM$GET\_ALL\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/system_get_all_references) | System | ✓ | | [SYSTEM$GET\_APPLICATION\_SERVICE\_LOGS](https://docs.snowflake.com/en/sql-reference/functions/system_get_application_service_logs) | System | — | | [SYSTEM$GET\_AWS\_SNS\_IAM\_POLICY](https://docs.snowflake.com/en/sql-reference/functions/system_get_aws_sns_iam_policy) | System | ✓ | | [SYSTEM$GET\_CATALOG\_LINKED\_DATABASE\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_catalog_linked_database_config) | System | — | | [SYSTEM$GET\_CLASSIFICATION\_RESULT](https://docs.snowflake.com/en/sql-reference/functions/system_get_classification_result) | System | — | | [SYSTEM$GET\_CMK\_AKV\_CONSENT\_URL](https://docs.snowflake.com/en/sql-reference/functions/system_get_cmk_akv_consent_url) | System | — | | [SYSTEM$GET\_CMK\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_cmk_config) | System | — | | [SYSTEM$GET\_CMK\_CONFIG\_POSTGRES](https://docs.snowflake.com/en/sql-reference/functions/system_get_cmk_config_postgres) | System | — | | [SYSTEM$GET\_CMK\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_get_cmk_info) | System | — | | [SYSTEM$GET\_CMK\_INFO\_POSTGRES](https://docs.snowflake.com/en/sql-reference/functions/system_get_cmk_info_postgres) | System | — | | [SYSTEM$GET\_CMK\_KMS\_KEY\_POLICY](https://docs.snowflake.com/en/sql-reference/functions/system_get_cmk_kms_key_policy) | System | — | | [SYSTEM$GET\_COLUMN\_COMMENT\_PROPAGATION\_SOURCE](https://docs.snowflake.com/en/sql-reference/functions/system_get_column_comment_propagation_source) | System | — | | [SYSTEM$GET\_COMPUTE\_POOL\_PENDING\_MAINTENANCE](https://docs.snowflake.com/en/sql-reference/functions/system_get_compute_pool_pending_maintenance) | System | — | | [SYSTEM$GET\_DBT\_LOG](https://docs.snowflake.com/en/sql-reference/functions/system_get_dbt_log) | System | — | | [SYSTEM$GET\_DEBUG\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_get_debug_status) | System | — | | [SYSTEM$GET\_DEFAULT\_COLUMNS\_OVERRIDE\_FOR\_SHOW\_COMMAND](https://docs.snowflake.com/en/sql-reference/functions/system_get_default_columns_override_for_show_command) | System | — | | [SYSTEM$GET\_DEFAULT\_COLUMNS\_OVERRIDE\_FOR\_SYSTEM\_OBJECT](https://docs.snowflake.com/en/sql-reference/functions/system_get_default_columns_override_for_system_object) | System | — | | [SYSTEM$GET\_DIRECTORY\_TABLE\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_get_directory_table_status) | System | ✓ | | [SYSTEM$GET\_GCP\_KMS\_CMK\_GRANT\_ACCESS\_CMD](https://docs.snowflake.com/en/sql-reference/functions/system_get_gcp_kms_cmk_grant_access_cmd) | System | — | | [SYSTEM$GET\_HASH\_FOR\_APPLICATION](https://docs.snowflake.com/en/sql-reference/functions/system_get_hash_for_application) | System | — | | [SYSTEM$GET\_ICEBERG\_TABLE\_INFORMATION](https://docs.snowflake.com/en/sql-reference/functions/system_get_iceberg_table_information) | System | ✓ | | [SYSTEM$GET\_INSTANCE\_FAMILY\_PLACEMENT\_GROUPS](https://docs.snowflake.com/en/sql-reference/functions/system_get_instance_family_placement_groups) | System | — | | [SYSTEM$GET\_LISTING\_URL\_FOR\_EXTERNAL\_CONSUMER](https://docs.snowflake.com/en/sql-reference/functions/system_get_listing_url_for_external_consumer) | System | — | | [SYSTEM$GET\_LOGIN\_FAILURE\_DETAILS](https://docs.snowflake.com/en/sql-reference/functions/system_get_login_failure_details) | System | — | | [SYSTEM$GET\_PREDECESSOR\_RETURN\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/system_get_predecessor_return_value) | System | ✓ | | [SYSTEM$GET\_PREVIEW\_ACCESS\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_get_preview_access_status) | System | — | | [SYSTEM$GET\_PRIVATELINK](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink) | System | — | | [SYSTEM$GET\_PRIVATELINK\_AUTHORIZED\_ENDPOINTS](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_authorized_endpoints) | System | — | | [SYSTEM$GET\_PRIVATELINK\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config) | System | — | | [SYSTEM$GET\_PRIVATELINK\_ENDPOINT\_REGISTRATIONS](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_endpoint_registrations) | System | — | | [SYSTEM$GET\_PRIVATELINK\_ENDPOINTS\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_endpoints_info) | System | — | | [SYSTEM$GET\_PURCHASE\_ATTRIBUTES](https://docs.snowflake.com/en/sql-reference/functions/system_get_purchase_attributes) | System | — | | [SYSTEM$GET\_REFERENCED\_OBJECT\_ID\_HASH](https://docs.snowflake.com/en/sql-reference/functions/system_get_referenced_object_id_hash) | System | ✓ | | [SYSTEM$GET\_RESULTSET\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_get_resultset_status) | System | — | | [SYSTEM$GET\_SECURITY\_INTEGRATIONS\_FOR\_API\_PROVIDER](https://docs.snowflake.com/en/sql-reference/functions/system_get_security_integrations_for_api_provider) | System | — | | [SYSTEM$GET\_SERVICE\_CALLER\_TOKEN\_EXPIRY](https://docs.snowflake.com/en/sql-reference/functions/system_get_service_caller_token_expiry) | System | — | | [SYSTEM$GET\_SERVICE\_DNS\_DOMAIN](https://docs.snowflake.com/en/sql-reference/functions/system_get_service_dns_domain) | System | — | | [SYSTEM$GET\_SERVICE\_LOGS](https://docs.snowflake.com/en/sql-reference/functions/system_get_service_logs) | System | ✓ | | [SYSTEM$GET\_SERVICE\_STATUS — DEPRECATED](https://docs.snowflake.com/en/sql-reference/functions/system_get_service_status) | System | ✓ | | [SYSTEM$GET\_SNOWFLAKE\_EGRESS\_IP\_RANGES](https://docs.snowflake.com/en/sql-reference/functions/system_get_snowflake_egress_ip_ranges) | System | — | | [SYSTEM$GET\_SNOWFLAKE\_PLATFORM\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_get_snowflake_platform_info) | System | — | | [SYSTEM$GET\_STAGE\_PRIVATELINK\_AUTHORIZED\_ENDPOINTS](https://docs.snowflake.com/en/sql-reference/functions/system_get_stage_privatelink_authorized_endpoints) | System | — | | [SYSTEM$GET\_TABLE\_ARCHIVE\_METADATA](https://docs.snowflake.com/en/sql-reference/functions/system_get_table_archive_metadata) | System | — | | [SYSTEM$GET\_TAG](https://docs.snowflake.com/en/sql-reference/functions/system_get_tag) | System | ✓ | | [SYSTEM$GET\_TAG\_ALLOWED\_VALUES](https://docs.snowflake.com/en/sql-reference/functions/system_get_tag_allowed_values) | System | ✓ | | [SYSTEM$GET\_TAG\_ON\_CURRENT\_COLUMN](https://docs.snowflake.com/en/sql-reference/functions/system_get_tag_on_current_column) | System | ✓ | | [SYSTEM$GET\_TAG\_ON\_CURRENT\_TABLE](https://docs.snowflake.com/en/sql-reference/functions/system_get_tag_on_current_table) | System | ✓ | | [SYSTEM$GET\_TASK\_GRAPH\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_task_graph_config) | System | ✓ | | [SYSTEM$GET\_TZDB\_VERSION](https://docs.snowflake.com/en/sql-reference/functions/system_get_tzdb_version) | System | — | | [SYSTEM$GLOBAL\_ACCOUNT\_SET\_PARAMETER](https://docs.snowflake.com/en/sql-reference/functions/system_global_account_set_parameter) | System | ✓ | | [SYSTEM$HOLD\_PRIVILEGE\_ON\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_hold_privilege_on_account) | System | ✓ | | [SYSTEM$INGEST\_REBIND\_PIPE](https://docs.snowflake.com/en/sql-reference/functions/system_ingest_rebind_pipe) | System | — | | [SYSTEM$INITIATE\_MOVE\_ORGANIZATION\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_initiate_move_organization_account) | System | — | | [SYSTEM$INTERNAL\_STAGES\_PUBLIC\_ACCESS\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_internal_stages_public_access_status) | System | — | | [SYSTEM$IS\_APPLICATION\_ALL\_MANDATORY\_TELEMETRY\_EVENT\_DEFINITIONS\_ENABLED](https://docs.snowflake.com/en/sql-reference/functions/system_is_application_all_mandatory_telemetry_event_definitions_enabled) | System | — | | [SYSTEM$IS\_APPLICATION\_AUTHORIZED\_FOR\_TELEMETRY\_EVENT\_SHARING](https://docs.snowflake.com/en/sql-reference/functions/system_is_application_authorized_for_telemetry_event_sharing) | System | — | | [SYSTEM$IS\_APPLICATION\_INSTALLED\_FROM\_SAME\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_is_application_installed_from_same_account) | System | — | | [SYSTEM$IS\_APPLICATION\_SHARING\_EVENTS\_WITH\_PROVIDER](https://docs.snowflake.com/en/sql-reference/functions/system_is_application_sharing_events_with_provider) | System | — | | [SYSTEM$IS\_GLOBAL\_DATA\_SHARING\_ENABLED\_FOR\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_is_global_data_sharing_enabled_for_account) | System | — | | [SYSTEM$IS\_LISTING\_PURCHASED](https://docs.snowflake.com/en/sql-reference/functions/system_is_listing_purchased) | System | — | | [SYSTEM$IS\_LISTING\_TRIAL](https://docs.snowflake.com/en/sql-reference/functions/system_is_listing_trial) | System | — | | [SYSTEM$ISSUE\_PER\_ACCOUNT\_APP\_SERVICE\_CERTIFICATE](https://docs.snowflake.com/en/sql-reference/functions/system_issue_per_account_app_service_certificate) | System | — | | [SYSTEM$ISSUE\_PER\_ACCOUNT\_CERTIFICATES](https://docs.snowflake.com/en/sql-reference/functions/system_issue_per_account_certificates) | System | — | | [SYSTEM$ISSUE\_WORKLOAD\_IDENTITY\_FEDERATION\_TOKEN](https://docs.snowflake.com/en/sql-reference/functions/system_issue_workload_identity_federation_token) | System | — | | [SYSTEM$LAST\_CHANGE\_COMMIT\_TIME](https://docs.snowflake.com/en/sql-reference/functions/system_last_change_commit_time) | System | ✓ | | [SYSTEM$LINK\_ACCOUNT\_OBJECTS\_BY\_NAME](https://docs.snowflake.com/en/sql-reference/functions/system_link_account_objects_by_name) | System | — | | [SYSTEM$LINK\_ORGANIZATION\_USER](https://docs.snowflake.com/en/sql-reference/functions/system_link_organization_user) | System | — | | [SYSTEM$LINK\_ORGANIZATION\_USER\_GROUP](https://docs.snowflake.com/en/sql-reference/functions/system_link_organization_user_group) | System | — | | [SYSTEM$LIST\_ALERT\_TEMPLATES](https://docs.snowflake.com/en/sql-reference/functions/system_list_alert_templates) | System | — | | [SYSTEM$LIST\_APPLICATION\_RESTRICTED\_FEATURES](https://docs.snowflake.com/en/sql-reference/functions/system_list_application_restricted_features) | System | — | | [SYSTEM$LIST\_EXTERNAL\_SECRETS](https://docs.snowflake.com/en/sql-reference/functions/system_list_external_secrets) | System | — | | [SYSTEM$LIST\_ICEBERG\_TABLES\_FROM\_CATALOG](https://docs.snowflake.com/en/sql-reference/functions/system_list_iceberg_tables_from_catalog) | System | — | | [SYSTEM$LIST\_NAMESPACES\_FROM\_CATALOG](https://docs.snowflake.com/en/sql-reference/functions/system_list_namespaces_from_catalog) | System | — | | [SYSTEM$LOCATE\_DBT\_ARCHIVE](https://docs.snowflake.com/en/sql-reference/functions/system_locate_dbt_archive) | System | — | | [SYSTEM$LOCATE\_DBT\_ARTIFACTS](https://docs.snowflake.com/en/sql-reference/functions/system_locate_dbt_artifacts) | System | — | | [SYSTEM$LOG, SYSTEM$LOG\_\<LEVEL> (FOR SNOWFLAKE SCRIPTING)](https://docs.snowflake.com/en/sql-reference/functions/system_log) | System | — | | [SYSTEM$MIGRATE\_DBT\_PROJECT](https://docs.snowflake.com/en/sql-reference/functions/system_migrate_dbt_project) | System | — | | [SYSTEM$MIGRATE\_SAML\_IDP\_REGISTRATION](https://docs.snowflake.com/en/sql-reference/functions/system_migrate_saml_idp_registration) | System | ✓ | | [SYSTEM$OPT\_IN\_INTERNAL\_STAGE\_NETWORK\_LOGS](https://docs.snowflake.com/en/sql-reference/functions/system_opt_in_internal_stage_network_logs) | System | — | | [SYSTEM$OPT\_OUT\_INTERNAL\_STAGE\_NETWORK\_LOGS](https://docs.snowflake.com/en/sql-reference/functions/system_opt_out_internal_stage_network_logs) | System | — | | [SYSTEM$OPT\_OUT\_MALICIOUS\_IP\_PROTECTION\_BY\_CATEGORY](https://docs.snowflake.com/en/sql-reference/functions/system_opt_out_malicious_ip_protection_by_category) | System | — | | [SYSTEM$PIPE\_FORCE\_RESUME](https://docs.snowflake.com/en/sql-reference/functions/system_pipe_force_resume) | System | ✓ | | [SYSTEM$PIPE\_REBINDING\_WITH\_NOTIFICATION\_CHANNEL](https://docs.snowflake.com/en/sql-reference/functions/system_pipe_rebinding_with_notification_channel) | System | ✓ | | [SYSTEM$PIPE\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_pipe_status) | System | ✓ | | [SYSTEM$PROVISION\_PRIVATELINK\_ENDPOINT](https://docs.snowflake.com/en/sql-reference/functions/system_provision_privatelink_endpoint) | System | — | | [SYSTEM$PROVISION\_PRIVATELINK\_ENDPOINT\_TSS](https://docs.snowflake.com/en/sql-reference/functions/system_provision_privatelink_endpoint_tss) | System | — | | [SYSTEM$QUERY\_REFERENCE](https://docs.snowflake.com/en/sql-reference/functions/system_query_reference) | System | ✓ | | [SYSTEM$READ\_OSI\_YAML\_FROM\_SEMANTIC\_VIEW](https://docs.snowflake.com/en/sql-reference/functions/system_read_osi_yaml_from_semantic_view) | System | — | | [SYSTEM$READ\_OSSIE\_YAML\_FROM\_SEMANTIC\_VIEW](https://docs.snowflake.com/en/sql-reference/functions/system_read_ossie_yaml_from_semantic_view) | System | — | | [SYSTEM$READ\_YAML\_FROM\_SEMANTIC\_VIEW](https://docs.snowflake.com/en/sql-reference/functions/system_read_yaml_from_semantic_view) | System | — | | [SYSTEM$REFERENCE](https://docs.snowflake.com/en/sql-reference/functions/system_reference) | System | ✓ | | [SYSTEM$REGISTER\_CMK\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_register_cmk_info) | System | — | | [SYSTEM$REGISTER\_CMK\_INFO\_POSTGRES](https://docs.snowflake.com/en/sql-reference/functions/system_register_cmk_info_postgres) | System | — | | [SYSTEM$REGISTER\_PRIVATELINK\_ENDPOINT](https://docs.snowflake.com/en/sql-reference/functions/system_register_privatelink_endpoint) | System | — | | [SYSTEM$REGISTRY\_LIST\_IMAGES — DEPRECATED](https://docs.snowflake.com/en/sql-reference/functions/system_registry_list_images) | System | — | | [SYSTEM$REMOTE\_APP\_OPERATION](https://docs.snowflake.com/en/sql-reference/functions/system_remote_app_operation) | System | — | | [SYSTEM$REMOVE\_ALL\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/system_remove_all_references) | System | ✓ | | [SYSTEM$REMOVE\_PRIVATELINK\_ENDPOINT\_HOSTNAME](https://docs.snowflake.com/en/sql-reference/functions/system_remove_privatelink_endpoint_hostname) | System | — | | [SYSTEM$REMOVE\_REFERENCE](https://docs.snowflake.com/en/sql-reference/functions/system_remove_reference) | System | ✓ | | [SYSTEM$REPORT\_HEALTH\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_report_health_status) | System | — | | [SYSTEM$RESOLVE\_PYTHON\_PACKAGES](https://docs.snowflake.com/en/sql-reference/functions/system_resolve_python_packages) | System | — | | [SYSTEM$RESTORE\_PRIVATELINK\_ENDPOINT](https://docs.snowflake.com/en/sql-reference/functions/system_restore_privatelink_endpoint) | System | — | | [SYSTEM$RESTORE\_PRIVATELINK\_ENDPOINT\_TSS](https://docs.snowflake.com/en/sql-reference/functions/system_restore_privatelink_endpoint_tss) | System | — | | [SYSTEM$REVOKE\_PRIVATELINK](https://docs.snowflake.com/en/sql-reference/functions/system_revoke_privatelink) | System | — | | [SYSTEM$REVOKE\_SNOWFLAKE\_MANAGED\_STORAGE\_VOLUME\_PRIVATELINK\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_revoke_snowflake_managed_storage_volume_privatelink_access) | System | — | | [SYSTEM$REVOKE\_STAGE\_PRIVATELINK\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_revoke_stage_privatelink_access) | System | — | | [SYSTEM$SCHEDULE\_ASYNC\_REPLICATION\_GROUP\_REFRESH](https://docs.snowflake.com/en/sql-reference/functions/system_schedule_async_replication_group_refresh) | System | — | | [SYSTEM$SEND\_NOTIFICATIONS\_TO\_CATALOG](https://docs.snowflake.com/en/sql-reference/functions/system_send_notifications_to_catalog) | System | — | | [SYSTEM$SET\_APPLICATION\_RESTRICTED\_FEATURE\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_set_application_restricted_feature_access) | System | — | | [SYSTEM$SET\_CATALOG\_INTEGRATION](https://docs.snowflake.com/en/sql-reference/functions/system_set_catalog_integration) | System | — | | [SYSTEM$SET\_DEFAULT\_COLUMNS\_OVERRIDE\_FOR\_SHOW\_COMMAND](https://docs.snowflake.com/en/sql-reference/functions/system_set_default_columns_override_for_show_command) | System | — | | [SYSTEM$SET\_DEFAULT\_COLUMNS\_OVERRIDE\_FOR\_SYSTEM\_OBJECT](https://docs.snowflake.com/en/sql-reference/functions/system_set_default_columns_override_for_system_object) | System | — | | [SYSTEM$SET\_EVENT\_SHARING\_ACCOUNT\_FOR\_REGION](https://docs.snowflake.com/en/sql-reference/functions/system_set_event_sharing_account_for_region) | System | — | | [SYSTEM$SET\_PRIVATELINK\_ENDPOINT\_HOSTNAME](https://docs.snowflake.com/en/sql-reference/functions/system_set_privatelink_endpoint_hostname) | System | — | | [SYSTEM$SET\_REFERENCE](https://docs.snowflake.com/en/sql-reference/functions/system_set_reference) | System | ✓ | | [SYSTEM$SET\_RETURN\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/system_set_return_value) | System | ✓ | | [SYSTEM$SET\_ROW\_TIMESTAMP\_ON\_ALL\_SUPPORTED\_TABLES](https://docs.snowflake.com/en/sql-reference/functions/system_set_row_timestamp_on_all_supported_tables) | System | — | | [SYSTEM$SET\_SPAN\_ATTRIBUTES (FOR SNOWFLAKE SCRIPTING)](https://docs.snowflake.com/en/sql-reference/functions/system_set_span_attributes) | System | — | | [SYSTEM$SHOW\_ACTIVE\_BEHAVIOR\_CHANGE\_BUNDLES](https://docs.snowflake.com/en/sql-reference/functions/system_show_active_behavior_change_bundles) | System | ✓ | | [SYSTEM$SHOW\_BUDGET\_SHARED\_RESOURCE\_CANDIDATES](https://docs.snowflake.com/en/sql-reference/functions/system_show_budget_shared_resource_candidates) | System | — | | [SYSTEM$SHOW\_BUDGETS\_FOR\_RESOURCE](https://docs.snowflake.com/en/sql-reference/functions/system_show_budgets_for_resource) | System | — | | [SYSTEM$SHOW\_BUDGETS\_IN\_ACCOUNT](https://docs.snowflake.com/en/sql-reference/functions/system_show_budgets_in_account) | System | — | | [SYSTEM$SHOW\_DYNAMIC\_TABLES\_CREATED\_FOR\_RESHARING](https://docs.snowflake.com/en/sql-reference/functions/system_show_dynamic_tables_created_for_resharing) | System | — | | [SYSTEM$SHOW\_EVENT\_SHARING\_ACCOUNTS](https://docs.snowflake.com/en/sql-reference/functions/system_show_event_sharing_accounts) | System | — | | [SYSTEM$SHOW\_MOVE\_ORGANIZATION\_ACCOUNT\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_show_move_organization_account_status) | System | — | | [SYSTEM$SHOW\_OAUTH\_CLIENT\_SECRETS](https://docs.snowflake.com/en/sql-reference/functions/system_show_oauth_client_secrets) | System | — | | [SYSTEM$SHOW\_SENSITIVE\_DATA\_MONITORED\_ENTITIES](https://docs.snowflake.com/en/sql-reference/functions/system_show_sensitive_data_monitored_entities) | System | — | | [SYSTEM$SNOWFLAKE\_MANAGED\_STORAGE\_VOLUME\_PUBLIC\_ACCESS\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_snowflake_managed_storage_volume_public_access_status) | System | — | | [SYSTEM$SNOWPIPE\_STREAMING\_UPDATE\_CHANNEL\_OFFSET\_TOKEN](https://docs.snowflake.com/en/sql-reference/functions/system_snowpipe_streaming_update_channel_offset_token) | System | ✓ | | [SYSTEM$START\_OAUTH\_FLOW](https://docs.snowflake.com/en/sql-reference/functions/system_start_oauth_flow) | System | — | | [SYSTEM$START\_USER\_EMAIL\_VERIFICATION](https://docs.snowflake.com/en/sql-reference/functions/system_start_user_email_verification) | System | — | | [SYSTEM$STREAM\_GET\_TABLE\_TIMESTAMP](https://docs.snowflake.com/en/sql-reference/functions/system_stream_get_table_timestamp) | System | — | | [SYSTEM$STREAM\_HAS\_DATA](https://docs.snowflake.com/en/sql-reference/functions/system_stream_has_data) | System | — | | [SYSTEM$SUPPORTED\_DBT\_VERSIONS](https://docs.snowflake.com/en/sql-reference/functions/system_supported_dbt_versions) | System | — | | [SYSTEM$TAG\_VALUE\_CONTAINS](https://docs.snowflake.com/en/sql-reference/functions/system_tag_value_contains) | System | — | | [SYSTEM$TASK\_DEPENDENTS\_ENABLE](https://docs.snowflake.com/en/sql-reference/functions/system_task_dependents_enable) | System | ✓ | | [SYSTEM$TASK\_RUNTIME\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_task_runtime_info) | System | ✓ | | [SYSTEM$TRIGGER\_LISTING\_REFRESH](https://docs.snowflake.com/en/sql-reference/functions/system_trigger_listing_refresh) | System | — | | [SYSTEM$TYPEOF](https://docs.snowflake.com/en/sql-reference/functions/system_typeof) | System | ✓ | | [SYSTEM$UNBLOCK\_INTERNAL\_STAGES\_PUBLIC\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_unblock_internal_stages_public_access) | System | — | | [SYSTEM$UNBLOCK\_SNOWFLAKE\_MANAGED\_STORAGE\_VOLUME\_PUBLIC\_ACCESS](https://docs.snowflake.com/en/sql-reference/functions/system_unblock_snowflake_managed_storage_volume_public_access) | System | — | | [SYSTEM$UNLINK\_ORGANIZATION\_USER](https://docs.snowflake.com/en/sql-reference/functions/system_unlink_organization_user) | System | — | | [SYSTEM$UNLINK\_ORGANIZATION\_USER\_GROUP](https://docs.snowflake.com/en/sql-reference/functions/system_unlink_organization_user_group) | System | — | | [SYSTEM$UNREGISTER\_PRIVATELINK\_ENDPOINT](https://docs.snowflake.com/en/sql-reference/functions/system_unregister_privatelink_endpoint) | System | — | | [SYSTEM$UNSET\_DEFAULT\_COLUMNS\_OVERRIDE\_FOR\_SHOW\_COMMAND](https://docs.snowflake.com/en/sql-reference/functions/system_unset_default_columns_override_for_show_command) | System | — | | [SYSTEM$UNSET\_DEFAULT\_COLUMNS\_OVERRIDE\_FOR\_SYSTEM\_OBJECT](https://docs.snowflake.com/en/sql-reference/functions/system_unset_default_columns_override_for_system_object) | System | — | | [SYSTEM$UNSET\_EVENT\_SHARING\_ACCOUNT\_FOR\_REGION](https://docs.snowflake.com/en/sql-reference/functions/system_unset_event_sharing_account_for_region) | System | — | | [SYSTEM$UNVERIFY\_DNS\_DOMAIN](https://docs.snowflake.com/en/sql-reference/functions/system_unverify_dns_domain) | System | — | | [SYSTEM$UPDATE\_MULTI\_PARTY\_APPROVAL\_JUSTIFICATION](https://docs.snowflake.com/en/sql-reference/functions/system_update_multi_party_approval_justification) | System | — | | [SYSTEM$USER\_TASK\_CANCEL\_ONGOING\_EXECUTIONS](https://docs.snowflake.com/en/sql-reference/functions/system_user_task_cancel_ongoing_executions) | System | ✓ | | [SYSTEM$VALIDATE\_STORAGE\_INTEGRATION](https://docs.snowflake.com/en/sql-reference/functions/system_validate_storage_integration) | System | — | | [SYSTEM$VERIFY\_CATALOG\_INTEGRATION](https://docs.snowflake.com/en/sql-reference/functions/system_verify_catalog_integration) | System | — | | [SYSTEM$VERIFY\_CMK\_INFO](https://docs.snowflake.com/en/sql-reference/functions/system_verify_cmk_info) | System | — | | [SYSTEM$VERIFY\_CMK\_INFO\_POSTGRES](https://docs.snowflake.com/en/sql-reference/functions/system_verify_cmk_info_postgres) | System | — | | [SYSTEM$VERIFY\_DNS\_DOMAIN](https://docs.snowflake.com/en/sql-reference/functions/system_verify_dns_domain) | System | — | | [SYSTEM$VERIFY\_EXTERNAL\_OAUTH\_TOKEN](https://docs.snowflake.com/en/sql-reference/functions/system_verify_ext_oauth_token) | System | ✓ | | [SYSTEM$VERIFY\_EXTERNAL\_SECRET\_INTEGRATION](https://docs.snowflake.com/en/sql-reference/functions/system_verify_external_secret_integration) | System | — | | [SYSTEM$VERIFY\_EXTERNAL\_VOLUME](https://docs.snowflake.com/en/sql-reference/functions/system_verify_external_volume) | System | — | | [SYSTEM$WAIT](https://docs.snowflake.com/en/sql-reference/functions/system_wait) | System | ✓ | | [SYSTEM$WAIT\_FOR\_OPENFLOW\_CONNECTOR\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_wait_for_openflow_connector_status) | System | — | | [SYSTEM$WAIT\_FOR\_OPENFLOW\_DEPLOYMENT\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_wait_for_openflow_deployment_status) | System | — | | [SYSTEM$WAIT\_FOR\_OPENFLOW\_RUNTIME\_STATUS](https://docs.snowflake.com/en/sql-reference/functions/system_wait_for_openflow_runtime_status) | System | — | | [SYSTEM$WAIT\_FOR\_SERVICES](https://docs.snowflake.com/en/sql-reference/functions/system_wait_for_services) | System | — | | [SYSTEM$WAIT\_FOR\_STABLE\_OPENFLOW\_CONNECTORS](https://docs.snowflake.com/en/sql-reference/functions/system_wait_for_stable_openflow_connectors) | System | — | | [SYSTEM$WAIT\_FOR\_STABLE\_OPENFLOW\_DEPLOYMENTS](https://docs.snowflake.com/en/sql-reference/functions/system_wait_for_stable_openflow_deployments) | System | — | | [SYSTEM$WAIT\_FOR\_STABLE\_OPENFLOW\_RUNTIMES](https://docs.snowflake.com/en/sql-reference/functions/system_wait_for_stable_openflow_runtimes) | System | — | | [SYSTEM$WHITELIST — DEPRECATED](https://docs.snowflake.com/en/sql-reference/functions/system_whitelist) | System | — | | [SYSTEM$WHITELIST\_PRIVATELINK — DEPRECATED](https://docs.snowflake.com/en/sql-reference/functions/system_whitelist_privatelink) | System | — | | [\<SERVICE\_NAME>!SPCS\_CANCEL\_JOB](https://docs.snowflake.com/en/sql-reference/functions/spcs_cancel_job) | Table | — | | [\<SERVICE\_NAME>!SPCS\_GET\_EVENTS](https://docs.snowflake.com/en/sql-reference/functions/spcs_get_events) | Table | — | | [\<SERVICE\_NAME>!SPCS\_GET\_LOGS](https://docs.snowflake.com/en/sql-reference/functions/spcs_get_logs) | Table | — | | [\<SERVICE\_NAME>!SPCS\_GET\_METRICS](https://docs.snowflake.com/en/sql-reference/functions/spcs_get_metrics) | Table | — | | [APPLICATION\_CONFIGURATION\_VALUE\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/application_configuration_value_history) | Table | — | | [CORTEX\_SEARCH\_BATCH](https://docs.snowflake.com/en/sql-reference/functions/cortex_search_batch) | Table | — | | [CORTEX\_SEARCH\_DATA\_SCAN](https://docs.snowflake.com/en/sql-reference/functions/cortex_search_data_scan) | Table | — | | [CUMULATIVE\_PRIVACY\_LOSSES](https://docs.snowflake.com/en/sql-reference/functions/cumulative_privacy_losses) | Table | — | | [ESTIMATE\_HYBRID\_TABLE\_STORAGE\_USAGE](https://docs.snowflake.com/en/sql-reference/functions/estimate_hybrid_table_storage_usage) | Table | — | | [FLATTEN](https://docs.snowflake.com/en/sql-reference/functions/flatten) | Table | ✓ | | [GENERATOR](https://docs.snowflake.com/en/sql-reference/functions/generator) | Table | ✓ | | [GET\_AI\_EVALUATION\_DATA (SNOWFLAKE.LOCAL)](https://docs.snowflake.com/en/sql-reference/functions/get_ai_evaluation_data-snowflake-local) | Table | — | | [GET\_AI\_OBSERVABILITY\_EVENTS (SNOWFLAKE.LOCAL)](https://docs.snowflake.com/en/sql-reference/functions/get_ai_observability_events-snowflake-local) | Table | — | | [GET\_AI\_OBSERVABILITY\_LOGS (SNOWFLAKE.LOCAL)](https://docs.snowflake.com/en/sql-reference/functions/get_ai_observability_logs-snowflake-local) | Table | — | | [GET\_AI\_RECORD\_TRACE (SNOWFLAKE.LOCAL)](https://docs.snowflake.com/en/sql-reference/functions/get_ai_record_trace-snowflake-local) | Table | — | | [GET\_CONTACTS](https://docs.snowflake.com/en/sql-reference/functions/get_contacts) | Table | — | | [GET\_JOB\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/get_job_history) | Table | — | | [GET\_LINEAGE (SNOWFLAKE.CORE)](https://docs.snowflake.com/en/sql-reference/functions/get_lineage-snowflake-core) | Table | — | | [GET\_OBJECT\_REFERENCES](https://docs.snowflake.com/en/sql-reference/functions/get_object_references) | Table | ✓ | | [ICEBERG\_TABLE\_FILES](https://docs.snowflake.com/en/sql-reference/functions/iceberg_table_files) | Table | — | | [ICEBERG\_TABLE\_SNAPSHOT\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/iceberg_table_snapshot_refresh_history) | Table | — | | [INFER\_SCHEMA](https://docs.snowflake.com/en/sql-reference/functions/infer_schema) | Table | ✓ | | [REPLICATION\_GROUP\_LAG\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/replication_group_lag_history) | Table | — | | [REST\_EVENT\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/rest_event_history) | Table | ✓ | | [RESULT\_SCAN](https://docs.snowflake.com/en/sql-reference/functions/result_scan) | Table | ✓ | | [SEMANTIC\_VIEW\_MATERIALIZATION\_REFRESH\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/semantic_view_materialization_refresh_history) | Table | — | | [STORAGE\_LIFECYCLE\_POLICY\_HISTORY](https://docs.snowflake.com/en/sql-reference/functions/storage_lifecycle_policy_history) | Table | — | | [SYSTEM$STREAM\_BACKLOG](https://docs.snowflake.com/en/sql-reference/functions/system_stream_backlog) | Table | — | | [TO\_QUERY](https://docs.snowflake.com/en/sql-reference/functions/to_query) | Table | — | | [VALIDATE](https://docs.snowflake.com/en/sql-reference/functions/validate) | Table | ✓ | | [VECTOR\_AVG](https://docs.snowflake.com/en/sql-reference/functions/vector_avg) | Vector | ✓ | | [VECTOR\_COSINE\_SIMILARITY](https://docs.snowflake.com/en/sql-reference/functions/vector_cosine_similarity) | Vector | ✓ | | [VECTOR\_INNER\_PRODUCT](https://docs.snowflake.com/en/sql-reference/functions/vector_inner_product) | Vector | ✓ | | [VECTOR\_L1\_DISTANCE](https://docs.snowflake.com/en/sql-reference/functions/vector_l1_distance) | Vector | ✓ | | [VECTOR\_L2\_DISTANCE](https://docs.snowflake.com/en/sql-reference/functions/vector_l2_distance) | Vector | ✓ | | [VECTOR\_MAX](https://docs.snowflake.com/en/sql-reference/functions/vector_max) | Vector | ✓ | | [VECTOR\_MIN](https://docs.snowflake.com/en/sql-reference/functions/vector_min) | Vector | ✓ | | [VECTOR\_NORMALIZE](https://docs.snowflake.com/en/sql-reference/functions/vector_normalize) | Vector | — | | [VECTOR\_SUM](https://docs.snowflake.com/en/sql-reference/functions/vector_sum) | Vector | ✓ | | [VECTOR\_TRUNCATE](https://docs.snowflake.com/en/sql-reference/functions/vector_truncate) | Vector | — | | [CONDITIONAL\_CHANGE\_EVENT](https://docs.snowflake.com/en/sql-reference/functions/conditional_change_event) | Window | ✓ | | [CONDITIONAL\_TRUE\_EVENT](https://docs.snowflake.com/en/sql-reference/functions/conditional_true_event) | Window | ✓ | | [CUME\_DIST](https://docs.snowflake.com/en/sql-reference/functions/cume_dist) | Window | ✓ | | [INTERPOLATE\_BFILL, INTERPOLATE\_FFILL, INTERPOLATE\_LINEAR](https://docs.snowflake.com/en/sql-reference/functions/interpolate_bfill) | Window | — | | [PERCENT\_RANK](https://docs.snowflake.com/en/sql-reference/functions/percent_rank) | Window | ✓ | | [RANK](https://docs.snowflake.com/en/sql-reference/functions/rank) | Window | ✓ | | [RATIO\_TO\_REPORT](https://docs.snowflake.com/en/sql-reference/functions/ratio_to_report) | Window | ✓ | | [DENSE\_RANK](https://docs.snowflake.com/en/sql-reference/functions/dense_rank) | Window function syntax and usage | ✓ | | [FIRST\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/first_value) | Window function syntax and usage | ✓ | | [LAG](https://docs.snowflake.com/en/sql-reference/functions/lag) | Window function syntax and usage | ✓ | | [LAST\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/last_value) | Window function syntax and usage | ✓ | | [LEAD](https://docs.snowflake.com/en/sql-reference/functions/lead) | Window function syntax and usage | ✓ | | [NTH\_VALUE](https://docs.snowflake.com/en/sql-reference/functions/nth_value) | Window function syntax and usage | ✓ | | [NTILE](https://docs.snowflake.com/en/sql-reference/functions/ntile) | Window function syntax and usage | ✓ | | [ROW\_NUMBER](https://docs.snowflake.com/en/sql-reference/functions/row_number) | Window function syntax and usage | ✓ | --- ### Syntax overview dbt's node selection syntax makes it possible to run only specific resources in a given invocation of dbt. This selection syntax is used for the following subcommands: (Applies to dbt v1.12 and later) | command | argument(s) | | ----------------------------------------------------------------------- | ----------------------------------------------------- | | [run](https://docs.getdbt.com/reference/commands/run.md) | `--select`, `--exclude`, `--defer` | | [test](https://docs.getdbt.com/reference/commands/test.md) | `--select`, `--exclude`, `--defer` | | [seed](https://docs.getdbt.com/reference/commands/seed.md) | `--select`, `--exclude` | | [snapshot](https://docs.getdbt.com/reference/commands/snapshot.md) | `--select`, `--exclude` | | [ls (list)](https://docs.getdbt.com/reference/commands/list.md) | `--select`, `--exclude`, `--resource-type` | | [compile](https://docs.getdbt.com/reference/commands/compile.md) | `--select`, `--exclude`, `--inline` | | [freshness](https://docs.getdbt.com/reference/commands/source.md) | `--select`, `--exclude` | | [build](https://docs.getdbt.com/reference/commands/build.md) | `--select`, `--exclude`, `--resource-type`, `--defer` | | [docs generate](https://docs.getdbt.com/reference/commands/cmd-docs.md) | `--select`, `--exclude` | Nodes and resources We use the terms ["nodes"](https://en.wikipedia.org/wiki/Vertex_\(graph_theory\)) and "resources" interchangeably. These encompass all the models, tests, sources, seeds, snapshots, exposures, and analyses in your project. They are the objects that make up dbt's DAG (directed acyclic graph). (Applies to dbt v1.12 and later) You can reference a named selector using the `selector:` method with `--select` or `--exclude`. To learn more, refer to [`selector:` method](https://docs.getdbt.com/reference/node-selection/methods.md#selector). #### Specifying resources By default, `dbt run` executes *all* of the models in the dependency graph; `dbt seed` creates all seeds, `dbt snapshot` performs every snapshot. The `--select` flag is used to specify a subset of nodes to execute. To follow [POSIX standards](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html) and make things easier to understand, we recommend CLI users use quotes when passing arguments to the `--select` or `--exclude` option (including single or multiple space-delimited, or comma-delimited arguments). Not using quotes might not work reliably on all operating systems, terminals, and user interfaces. For example, `dbt run --select "my_dbt_project_name"` runs all models in your project. ##### How does selection work? 1. dbt gathers all the resources that are matched by one or more of the `--select` criteria, in the order of [selection methods](https://docs.getdbt.com/reference/node-selection/methods.md) (e.g. `tag:`), then [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md) (e.g. `+`), then finally set operators ([unions](https://docs.getdbt.com/reference/node-selection/set-operators.md#unions), [intersections](https://docs.getdbt.com/reference/node-selection/set-operators.md#intersections), [exclusions](https://docs.getdbt.com/reference/node-selection/exclude.md)). tip You can combine multiple selector methods in one `--select` command by separating them with commas (`,`) without whitespace (for example, `dbt run --select "marts.finance,tag:nightly"`). This only selects resources that satisfy *all* arguments. In this example, the command runs models that are in the `marts/finance` subdirectory and tagged `nightly`. For more information, see [Set operators](https://docs.getdbt.com/reference/node-selection/set-operators.md). 2. The selected resources may be models, sources, seeds, snapshots, tests. (Tests can also be selected "indirectly" via their parents; see [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) for details.) 3. dbt now has a list of still-selected resources of varying types. As a final step, it tosses away any resource that does not match the resource type of the current task. (Only seeds are kept for `dbt seed`, only models for `dbt run`, only tests for `dbt test`, and so on.) #### Shorthand Select resources to build (run, test, seed, snapshot) or check freshness: `--select`, `-s` ##### Examples By default, `dbt run` will execute *all* of the models in the dependency graph. During development (and deployment), it is useful to specify only a subset of models to run. Use the `--select` flag with `dbt run` to select a subset of models to run. (Applies to dbt v1.12 and later) Note that the following arguments (`--select` and `--exclude`) also apply to other dbt tasks, such as `test` and `build`. ##### Examples of select flag The `--select` flag accepts one or more arguments. Each argument can be one of: 1. a package name 2. a model name 3. a fully-qualified path to a directory of models 4. a selection method (`path:`, `tag:`, `config:`, `test_type:`, `test_name:`, `selector:`) Examples: ```bash dbt run --select "my_dbt_project_name" # runs all models in your project dbt run --select "my_dbt_model" # runs a specific model dbt run --select "path/to/my/models" # runs all models in a specific directory dbt run --select "my_package.some_model" # run a specific model in a specific package dbt run --select "tag:nightly" # run models with the "nightly" tag dbt run --select "path/to/models" # run models contained in path/to/models dbt run --select "path/to/my_model.sql" # run a specific model by its path dbt run --select "selector:my_selector" # run the node set defined by the named selector in selectors.yml; available starting v1.12 ``` ##### Examples of subsets of nodes dbt supports a shorthand language for defining subsets of nodes. This language uses the following characters: * plus operator [(`+`)](https://docs.getdbt.com/reference/node-selection/graph-operators.md#the-plus-operator) * at operator [(`@`)](https://docs.getdbt.com/reference/node-selection/graph-operators.md#the-at-operator) * asterisk operator (`*`) * comma operator (`,`) Examples: ```bash # multiple arguments can be provided to --select dbt run --select "my_first_model my_second_model" # select my_model and all of its children dbt run --select "my_model+" # select my_model, its children, and the parents of its children dbt run --select @my_model # these arguments can be projects, models, directory paths, tags, or sources dbt run --select "tag:nightly my_model finance.base.*" # use methods and intersections for more complex selectors dbt run --select "path:marts/finance,tag:nightly,config.materialized:table" # combine a named selector with another method dbt run --select "selector:staging,tag:nightly" ``` As your selection logic gets more complex, and becomes unwieldy to type out as command-line arguments, consider using a [yaml selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md). (Applies to dbt v1.12 and later) You can reference a predefined selector using the `selector:` method when using `--select` or `--exclude`. To learn more, refer to [`selector:` method](https://docs.getdbt.com/reference/node-selection/methods.md#selector). ##### Troubleshoot with the `ls` command Constructing and debugging your selection syntax can be challenging. To get a "preview" of what will be selected, we recommend using the [`list` command](https://docs.getdbt.com/reference/commands/list.md). This command, when combined with your selection syntax, will output a list of the nodes that meet that selection criteria. The `dbt ls` command supports all types of selection syntax arguments, for example: ```bash dbt ls --select "path/to/my/models" # Lists all models in a specific directory. dbt ls --select "source_status:fresher+" # Shows sources updated since the last dbt source freshness run. dbt ls --select state:modified+ # Displays nodes modified in comparison to a previous state. dbt ls --select "result:<status>+" state:modified+ --state ./<dbt-artifact-path> # Lists nodes that match certain result statuses and are modified. ``` --- ### tags ##### Models dbt\_project.yml (Applies to dbt v1.9 and later) ```yml models: <resource-path>: +tags: <string> | [<string>] ``` models/properties.yml ```yaml models: - name: model_name config: tags: <string> | [<string>] columns: - name: column_name config: tags: <string> | [<string>] # changed to config in v1.10 and backported to 1.9 data_tests: - test_name: config: tags: <string> | [<string>] ``` models/\<modelname>.sql ```sql {{ config( tags="<string>" | ["<string>"] ) }} select ... ``` ##### Seeds dbt\_project.yml (Applies to dbt v1.9 and later) ```yml seeds: <resource-path>: +tags: <string> | [<string>] ``` seeds/properties.yml ```yaml seeds: - name: seed_name config: tags: <string> | [<string>] columns: - name: column_name config: tags: <string> | [<string>] # changed to config in v1.10 and backported to 1.9 data_tests: - test_name: config: tags: <string> | [<string>] ``` ##### Snapshots dbt\_project.yml (Applies to dbt v1.9 and later) ```yml snapshots: <resource-path>: +tags: <string> | [<string>] ``` (Applies to dbt v1.9 and later) snapshots/properties.yml ```yaml snapshots: - name: snapshot_name config: tags: <string> | [<string>] ``` snapshots/\<filename>.sql ```sql {% snapshot snapshot_name %} {{ config( tags="<string>" | ["<string>"] ) }} select ... {% endsnapshot %} ``` ##### Saved queries dbt\_project.yml (Applies to dbt v1.9 and later) ```yml saved-queries: <resource-path>: +tags: <string> | [<string>] ``` models/semantic\_models.yml ```yaml saved_queries: - name: saved_query_name config: tags: <string> | [<string>] ``` ##### Sources dbt\_project.yml (Applies to dbt v1.9 and later) ```yml sources: <resource-path>: +tags: <string> | [<string>] ``` models/properties.yml ```yaml sources: - name: source_name config: tags: <string> | [<string>] # changed to config in v1.10 tables: - name: table_name config: tags: <string> | [<string>] # changed to config in v1.10 columns: - name: column_name config: tags: <string> | [<string>] # changed to config in v1.10 and backported to 1.9 data_tests: - test_name: config: tags: <string> | [<string>] ``` Note that for backwards compatibility, `tags` is supported as a top-level key for sources, but without the capabilities of config inheritance. ##### Exposures dbt\_project.yml (Applies to dbt v1.9 and later) ```yml exposures: <resource-path>: +tags: <string> | [<string>] ``` models/exposures.yml ```yaml exposures: - name: exposure_name config: tags: <string> | [<string>] # changed to config in v1.10 ``` Note that for backwards compatibility, `tags` is supported as a top-level key for exposures, but without the capabilities of config inheritance. ##### Tests dbt\_project.yml (Applies to dbt v1.9 and later) ```yml data_tests: <resource-path>: +tags: <string> | [<string>] ``` models/properties.yml ```yaml models: - name: model_name columns: - name: column_name data_tests: - test_name: config: tags: <string> | [<string>] ``` tests/\<filename>.sql ```sql {% test test_name() %} {{ config( tags="<string>" | ["<string>"] ) }} select ... {% endtest %} ``` #### Definition Apply a tag (or list of tags) to a resource. These tags can be used as part of the [resource selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md), when running the following commands: * `dbt run --select tag:my_tag` — Run all models tagged with a specific tag. * `dbt build --select tag:my_tag` — Build all resources tagged with a specific tag. * `dbt seed --select tag:my_tag` — Seed all resources tagged with a specific tag. * `dbt snapshot --select tag:my_tag` — Snapshot all resources tagged with a specific tag. * `dbt test --select tag:my_tag` — Indirectly runs all tests associated with the models that are tagged. Effect on state comparison Changes to `tags`, including at the column level, don't trigger [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state). dbt treats `tags` (and `meta`) as metadata only, since they don't affect how a resource is materialized. Refer to [caveats to state comparison](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md#tags-and-meta) for more detail. ###### Using tags with the `+` operator You can use the [`+` operator](https://docs.getdbt.com/reference/node-selection/graph-operators.md#the-plus-operator) to include upstream or downstream dependencies in your `tag` selection: * `dbt run --select tag:my_tag+` — Run models tagged with `my_tag` and all their downstream dependencies. * `dbt run --select +tag:my_tag` — Run models tagged with `my_tag` and all their upstream dependencies. * `dbt run --select +tag:my_tag+` — Run models tagged with `my_tag`, their upstream dependencies, and their downstream dependencies. * `dbt run --select tag:my_tag+ --exclude tag:exclude_tag` — Run models tagged with `my_tag` and their downstream dependencies, and exclude models tagged with `exclude_tag`, regardless of their dependencies. Usage notes about tags When using tags, consider the following: * Each individual tag must be a string. * Tags are additive across project hierarchy. * Some resource types (like sources, exposures) require tags at the top level. Refer to [usage notes](#usage-notes) for more information. #### Examples The following examples show how to apply tags to resources in your project. You can configure tags in the `dbt_project.yml`, property files, or SQL files. ##### Use tags to run parts of your project Apply tags in your `dbt_project.yml` as a single value or a string. In the following example, one of the models, the `jaffle_shop` model, is tagged with `contains_pii`. dbt\_project.yml ```yml models: jaffle_shop: +tags: "contains_pii" staging: +tags: - "hourly" marts: +tags: - "hourly" - "published" metrics: +tags: - "daily" - "published" ``` ##### Apply tags to models This section demonstrates applying tags to models in the `dbt_project.yml`, `schema.yml`, and SQL files. To apply tags to a model in your `dbt_project.yml` file, you would add the following: dbt\_project.yml ```yaml models: jaffle_shop: +tags: finance # jaffle_shop model is tagged with 'finance'. ``` To apply tags to a model in your `models/` directory YAML property file, you would add the following using the `config` property: models/stg\_customers.yml ```yaml models: - name: stg_customers description: Customer data with basic cleaning and transformation applied, one row per customer. config: tags: ['santi'] # stg_customers.yml model is tagged with 'santi'. columns: - name: customer_id description: The unique key for each customer. data_tests: - not_null - unique ``` To apply tags to a model in your SQL file, you would add the following: models/staging/stg\_payments.sql ```sql {{ config( tags=["finance"] # stg_payments.sql model is tagged with 'finance'. ) }} select ... ``` Run resources with specific tags (or exclude resources with specific tags) using the following commands: ```shell # Run all models tagged "daily" dbt run --select tag:daily # Run all models tagged "daily", except those that are tagged hourly dbt run --select tag:daily --exclude tag:hourly ``` ##### Apply tags to seeds dbt\_project.yml ```yml seeds: jaffle_shop: utm_mappings: +tags: marketing ``` dbt\_project.yml ```yml seeds: jaffle_shop: utm_mappings: +tags: - marketing - hourly ``` ##### Apply tags to saved queries This following example shows how to apply a tag to a saved query in the `dbt_project.yml` file. The saved query is then tagged with `order_metrics`. dbt\_project.yml ```yml saved-queries: jaffle_shop: customer_order_metrics: +tags: order_metrics ``` Then run resources with a specific tag using the following commands: ```shell # Run all resources tagged "order_metrics" dbt run --select tag:order_metrics ``` The second example shows how to apply multiple tags to a saved query in the `semantic_model.yml` file. The saved query is then tagged with `order_metrics` and `hourly`. semantic\_model.yml ```yaml saved_queries: - name: test_saved_query description: "{{ doc('saved_query_description') }}" label: Test saved query config: tags: - order_metrics - hourly ``` Run resources with multiple tags using the following commands: ```shell # Run all resources tagged "order_metrics" and "hourly" dbt build --select tag:order_metrics tag:hourly ``` #### Usage notes ##### Tags must be strings Each individual tag must be a string value (for example, `marketing` or `daily`). In the following example, `my_tag: "my_value"` is invalid because it is a key-value pair. ```yml sources: - name: ecom schema: raw description: E-commerce data for the Jaffle Shop config: tags: my_tag: "my_value". # invalid tables: - name: raw_customers config: tags: my_tag: "my_value". # invalid ``` A warning is raised when the `tags` value is not a string. For example: ```text Field config.tags: {'my_tag': 'my_value'} is not valid for source (ecom) ``` ##### Tags are additive Tags accumulate hierarchically. The [earlier example](https://docs.getdbt.com/reference/resource-configs/tags.md#use-tags-to-run-parts-of-your-project) would result in: | Model | Tags | | --------------------------------- | ------------------------------------- | | models/staging/stg\_customers.sql | `contains_pii`, `hourly` | | models/staging/stg\_payments.sql | `contains_pii`, `hourly`, `finance` | | models/marts/dim\_customers.sql | `contains_pii`, `hourly`, `published` | | models/metrics/daily\_metrics.sql | `contains_pii`, `daily`, `published` | ##### Applying tags to specific columns and tests You can also apply tags to specific columns in a resource, and to tests. models/properties.yml ```yml models: - name: my_model columns: - name: column_name config: tags: ['column_level'] # changed to config in v1.10 and backported to 1.9 data_tests: - unique: config: tags: ['test_level'] # changed to config in v1.10 ``` In the example above, the `unique` test would be selected by either of these tags: ```bash dbt test --select tag:column_level dbt test --select tag:test_level ``` ##### Backwards compatibility for sources and exposures For backwards compatibility, `tags` is supported as a top-level key for sources and exposures (prior to dbt v1.10), but without the capabilities of config inheritance. models/properties.yml ```yml exposures: - name: my_exposure tags: ['exposure_tag'] # top-level key (legacy) # OR use config (v1.10+) config: tags: ['exposure_tag'] sources: - name: source_name tags: ['top_level'] # top-level key (legacy) # OR use config (v1.10+) config: tags: ['top_level'] tables: - name: table_name tags: ['table_level'] # top-level key (legacy) # OR use config (v1.10+) config: tags: ['table_level'] columns: - name: column_name config: tags: ['column_level'] # changed to config in v1.10 and backported to 1.9 ``` --- ### target_database note Starting in dbt v1.9+, this functionality is no longer utilized. Use the [database](https://docs.getdbt.com/reference/resource-configs/database.md) config as an alternative to define a custom database while still respecting the `generate_database_name` macro. Try it now in the [dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). dbt\_project.yml ```yml snapshots: <resource-path>: +target_database: string ``` snapshots/\<filename>.sql ```jinja2 {{ config( target_database="string" ) }} ``` #### Description The database that dbt should build a [snapshot](https://docs.getdbt.com/docs/build/snapshots.md) table into. Notes: * The specified database must already exist * On **BigQuery**, this is analogous to a `project`. * On **Redshift**, cross-database queries are not possible. If you use this parameter, you will receive the following error. As such, **do not use** this parameter on Redshift: ```text Encountered an error: Runtime Error Cross-db references not allowed in redshift (raw vs analytics) ``` #### Default By default, dbt will use the [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) database associated with your profile/connection. #### Examples ##### Build all snapshots in a database named `snapshots` dbt\_project.yml ```yml snapshots: +target_database: snapshots ``` ##### Use a target-aware database Use the [`{{ target }}` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) to change which database a snapshot table is built in. Note: consider whether this use-case is right for you, as downstream `refs` will select from the `dev` version of a snapshot, which can make it hard to validate models that depend on snapshots. dbt\_project.yml ```yml snapshots: +target_database: "{% if target.name == 'dev' %}dev{% else %}{{ target.database }}{% endif %}" ``` ##### Use the same database-naming behavior as models Leverage the [`generate_database_name` macro](https://docs.getdbt.com/docs/build/custom-databases.md) to build snapshots in databases that follow the same naming behavior as your models. Notes: * This macro is not available when configuring from the `dbt_project.yml` file, so it must be configured in a snapshot SQL file config. * Consider whether this use-case is right for you, as downstream `refs` will select from the `dev` version of a snapshot, which can make it hard to validate models that depend on snapshots. snapshots/orders\_snaphot.sql ```sql {{ config( target_database=generate_database_name('snapshots') ) }} ``` --- ### target_schema note Starting in dbt v1.9+, this functionality is no longer utilized. Use the [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) config as an alternative to define a custom schema while still respecting the `generate_schema_name` macro. Try it now in the [dbt **v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). dbt\_project.yml ```yml snapshots: <resource-path>: +target_schema: string ``` snapshots/\<filename>.sql ```jinja2 {{ config( target_schema="string" ) }} ``` #### Description The schema that dbt should build a [snapshot](https://docs.getdbt.com/docs/build/snapshots.md) table into. When `target_schema` is provided, snapshots build into the same `target_schema`, no matter who is running them. On **BigQuery**, this is analogous to a `dataset`. #### Default (Applies to dbt v1.9 and later) In dbt v1.9+, this is not a required parameter. #### Examples ##### Build all snapshots in a schema named `snapshots` dbt\_project.yml ```yml snapshots: +target_schema: snapshots ``` --- ### Teradata configurations #### General * *Set `quote_columns`* - to prevent a warning, make sure to explicitly set a value for `quote_columns` in your `dbt_project.yml`. See the [doc on quote\_columns](https://docs.getdbt.com/reference/resource-configs/quote_columns.md) for more information. ```yaml seeds: +quote_columns: false #or `true` if you have CSV column headers with spaces ``` #### Models ##### table * `table_kind` - define the table kind. Legal values are `MULTISET` (default for ANSI transaction mode required by `dbt-teradata`) and `SET`, e.g.: * in SQL materialization definition file: ```yaml {{ config( materialized="table", table_kind="SET" ) }} ``` * in seed configuration: ```yaml seeds: <project-name>: table_kind: "SET" ``` For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). * `table_option` - defines table options. The config supports multiple statements. The definition below uses the Teradata syntax definition to explain what statements are allowed. Square brackets `[]` denote optional parameters. The pipe symbol `|` separates statements. Use commas to combine multiple statements as shown in the examples below: ```text { MAP = map_name [COLOCATE USING colocation_name] | [NO] FALLBACK [PROTECTION] | WITH JOURNAL TABLE = table_specification | [NO] LOG | [ NO | DUAL ] [BEFORE] JOURNAL | [ NO | DUAL | LOCAL | NOT LOCAL ] AFTER JOURNAL | CHECKSUM = { DEFAULT | ON | OFF } | FREESPACE = integer [PERCENT] | mergeblockratio | datablocksize | blockcompression | isolated_loading } ``` where: * mergeblockratio: ```text { DEFAULT MERGEBLOCKRATIO | MERGEBLOCKRATIO = integer [PERCENT] | NO MERGEBLOCKRATIO } ``` * datablocksize: ```text DATABLOCKSIZE = { data_block_size [ BYTES | KBYTES | KILOBYTES ] | { MINIMUM | MAXIMUM | DEFAULT } DATABLOCKSIZE } ``` * blockcompression: ```text BLOCKCOMPRESSION = { AUTOTEMP | MANUAL | ALWAYS | NEVER | DEFAULT } [, BLOCKCOMPRESSIONALGORITHM = { ZLIB | ELZS_H | DEFAULT } ] [, BLOCKCOMPRESSIONLEVEL = { value | DEFAULT } ] ``` * isolated\_loading: ```text WITH [NO] [CONCURRENT] ISOLATED LOADING [ FOR { ALL | INSERT | NONE } ] ``` Examples: * In SQL materialization definition file: ```yaml {{ config( materialized="table", table_option="NO FALLBACK" ) }} ``` ```yaml {{ config( materialized="table", table_option="NO FALLBACK, NO JOURNAL" ) }} ``` ```yaml {{ config( materialized="table", table_option="NO FALLBACK, NO JOURNAL, CHECKSUM = ON, NO MERGEBLOCKRATIO, WITH CONCURRENT ISOLATED LOADING FOR ALL" ) }} ``` * in seed configuration: ```yaml seeds: <project-name>: table_option:"NO FALLBACK" ``` ```yaml seeds: <project-name>: table_option:"NO FALLBACK, NO JOURNAL" ``` ```yaml seeds: <project-name>: table_option: "NO FALLBACK, NO JOURNAL, CHECKSUM = ON, NO MERGEBLOCKRATIO, WITH CONCURRENT ISOLATED LOADING FOR ALL" ``` For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). * `with_statistics` - should statistics be copied from the base table. For example: ```yaml {{ config( materialized="table", with_statistics="true" ) }} ``` For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). * `index` - defines table indices: ```text [UNIQUE] PRIMARY INDEX [index_name] ( index_column_name [,...] ) | NO PRIMARY INDEX | PRIMARY AMP [INDEX] [index_name] ( index_column_name [,...] ) | PARTITION BY { partitioning_level | ( partitioning_level [,...] ) } | UNIQUE INDEX [ index_name ] [ ( index_column_name [,...] ) ] [loading] | INDEX [index_name] [ALL] ( index_column_name [,...] ) [ordering] [loading] [,...] ``` where: * partitioning\_level: ```text { partitioning_expression | COLUMN [ [NO] AUTO COMPRESS | COLUMN [ [NO] AUTO COMPRESS ] [ ALL BUT ] column_partition ] } [ ADD constant ] ``` * ordering: ```text ORDER BY [ VALUES | HASH ] [ ( order_column_name ) ] ``` * loading: ```text WITH [NO] LOAD IDENTITY ``` Examples: * In SQL materialization definition file: ```yaml {{ config( materialized="table", index="UNIQUE PRIMARY INDEX ( GlobalID )" ) }} ``` > ℹ️ Note, unlike in `table_option`, there are no commas between index statements! ```yaml {{ config( materialized="table", index="PRIMARY INDEX(id) PARTITION BY RANGE_N(create_date BETWEEN DATE '2020-01-01' AND DATE '2021-01-01' EACH INTERVAL '1' MONTH)" ) }} ``` ```yaml {{ config( materialized="table", index="PRIMARY INDEX(id) PARTITION BY RANGE_N(create_date BETWEEN DATE '2020-01-01' AND DATE '2021-01-01' EACH INTERVAL '1' MONTH) INDEX index_attrA (attrA) WITH LOAD IDENTITY" ) }} ``` * in seed configuration: ```yaml seeds: <project-name>: index: "UNIQUE PRIMARY INDEX ( GlobalID )" ``` > ℹ️ Note, unlike in `table_option`, there are no commas between index statements! ```yaml seeds: <project-name>: index: "PRIMARY INDEX(id) PARTITION BY RANGE_N(create_date BETWEEN DATE '2020-01-01' AND DATE '2021-01-01' EACH INTERVAL '1' MONTH)" ``` ```yaml seeds: <project-name>: index: "PRIMARY INDEX(id) PARTITION BY RANGE_N(create_date BETWEEN DATE '2020-01-01' AND DATE '2021-01-01' EACH INTERVAL '1' MONTH) INDEX index_attrA (attrA) WITH LOAD IDENTITY" ``` #### Seeds Using seeds to load raw data As explained in [dbt seeds documentation](https://docs.getdbt.com/docs/build/seeds.md), seeds should not be used to load raw data (for example, large CSV exports from a production database). Since seeds are version controlled, they are best suited to files that contain business-specific logic, for example a list of country codes or user IDs of employees. Loading CSVs using dbt's seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your data warehouse. * `use_fastload` - use [fastload](https://github.com/Teradata/python-driver#FastLoad) when handling `dbt seed` command. The option will likely speed up loading when your seed files have hundreds of thousands of rows. You can set this seed configuration option in your `project.yml` file, e.g.: ```yaml seeds: <project-name>: +use_fastload: true ``` #### Snapshots Snapshots use the [HASHROW function](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Functions-Expressions-and-Predicates/Hash-Related-Functions/HASHROW/HASHROW-Function-Syntax) of the Teradata database to generate a unique hash value for the `dbt_scd_id` column. To use your own hash UDF, there is a configuration option in the snapshot model called `snapshot_hash_udf`, which defaults to HASHROW. You can provide a value like `<database_name.hash_udf_name>`. If you only provide `hash_udf_name`, it uses the same schema as the model runs. For example, in the `snapshots/snapshot_example.sql` file: ```sql {% snapshot snapshot_example %} {{ config( target_schema='snapshots', unique_key='id', strategy='check', check_cols=["c2"], snapshot_hash_udf='GLOBAL_FUNCTIONS.hash_md5' ) }} select * from {{ ref('order_payments') }} {% endsnapshot %} ``` ###### Grants Grants are supported in dbt-teradata adapter with release version 1.2.0 and above. You can use grants to manage access to the datasets you're producing with dbt. To implement these permissions, define grants as resource configs on each model, seed, or snapshot. Define the default grants that apply to the entire project in your `dbt_project.yml`, and define model-specific grants within each model's SQL or property file. For example: models/schema.yml ```yaml models: - name: model_name config: grants: select: ['user_a', 'user_b'] ``` Another example for adding multiple grants: ```yaml models: - name: model_name config: materialized: table grants: select: ["user_b"] insert: ["user_c"] ``` > ℹ️ `copy_grants` is not supported in Teradata. Refer to [grants](https://docs.getdbt.com/reference/resource-configs/grants.md) for more information on Grants. #### Query band Query band in dbt-teradata can be set on three levels: 1. Profiles level: In the `profiles.yml` file, the user can provide `query_band` using the following example: ```yaml query_band: 'application=dbt;' ``` 2. Project level: In the `dbt_project.yml` file, the user can provide `query_band` using the following example: ```yaml models: Project_name: +query_band: "app=dbt;model={model};" ``` 3. Model level: It can be set on the model SQL file or model level configuration on YAML files: ```sql {{ config( query_band='sql={model};' ) }} ``` Users can set `query_band` at any level or on all levels. With profiles-level `query_band`, dbt-teradata will set the `query_band` for the first time for the session, and subsequently for model and project level query band will be updated with respective configuration. If a user sets some key-value pair with value as `'{model}'`, internally this `'{model}'` will be replaced with model name, which can be useful for telemetry tracking of sql/ dbql logging. ```yaml models: Project_name: +query_band: "app=dbt;model={model};" ``` * For example, if the model the user is running is `stg_orders`, `{model}` will be replaced with `stg_orders` in runtime. * If no `query_band` is set by the user, the default query\_band used will be: `org=teradata-internal-telem;appname=dbt;` #### Unit testing * Unit testing is supported in dbt-teradata, allowing users to write and execute unit tests using the dbt test command. * For detailed guidance, refer to the [dbt unit tests documentation](https://docs.getdbt.com/docs/build/documentation.md). > In Teradata, reusing the same alias across multiple common table expressions (CTEs) or subqueries within a single model is not permitted, as it results in parsing errors; therefore, it is essential to assign unique aliases to each CTE or subquery to ensure proper query execution. #### valid\_history incremental materialization strategy *This is available in early access* This strategy is designed to manage historical data efficiently within a Teradata environment, leveraging dbt features to ensure data quality and optimal resource usage. In temporal databases, valid time is crucial for applications like historical reporting, ML training datasets, and forensic analysis. ```yaml {{ config( materialized='incremental', unique_key='id', on_schema_change='fail', incremental_strategy='valid_history', valid_period='valid_period_col', use_valid_to_time='no', ) }} ``` The `valid_history` incremental strategy requires the following parameters: * `unique_key`: The primary key of the model (excluding the valid time components), specified as a column name or list of column names. * `valid_period`: Name of the model column indicating the period for which the record is considered to be valid. The datatype must be `PERIOD(DATE)` or `PERIOD(TIMESTAMP)`. * `use_valid_to_time`: Whether the end bound value of the valid period in the input is considered by the strategy when building the valid timeline. Use `no` if you consider your record to be valid until changed (and supply any value greater to the begin bound for the end bound of the period. A typical convention is `9999-12-31` of \`\`9999-12-31 23:59:59.999999`). Use `yes\` if you know until when the record is valid (typically this is a correction in the history timeline). The valid\_history strategy in dbt-teradata involves several critical steps to ensure the integrity and accuracy of historical data management: * Remove duplicates and conflicting values from the source data: * This step ensures that the data is clean and ready for further processing by eliminating any redundant or conflicting records. * The process of removing primary key duplicates (two or more records with the same value for the `unique_key` and BEGIN() bond of the `valid_period` fields) in the dataset produced by the model. If such duplicates exist, the row with the lowest value is retained for all non-primary-key fields (in the order specified in the model). Full-row duplicates are always de-duplicated. * Identify and adjust overlapping time slices: * Overlapping or adjacent time periods in the data are corrected to maintain a consistent and non-overlapping timeline. To achieve this, the macro adjusts the valid period end bound of a record to align with the begin bound of the next record (if they overlap or are adjacent) within the same `unique_key` group. If `use_valid_to_time = 'yes'`, the valid period end bound provided in the source data is used. Otherwise, a default end date is applied for missing bounds, and adjustments are made accordingly. * Manage records needing to be adjusted, deleted, or split based on the source and target data: * This involves handling scenarios where records in the source data overlap with or need to replace records in the target data, ensuring that the historical timeline remains accurate. * Compact history: * Normalize and compact the history by merging records of adjacent time periods with the same value, optimizing database storage and performance. We use the function TD\_NORMALIZE\_MEET for this purpose. * Delete existing overlapping records from the target table: * Before inserting new or updated records, any existing records in the target table that overlap with the new data are removed to prevent conflicts. * Insert the processed data into the target table: * Finally, the cleaned and adjusted data is inserted into the target table, ensuring that the historical data is up-to-date and accurately reflects the intended timeline. These steps collectively ensure that the valid\_history strategy effectively manages historical data, maintaining its integrity and accuracy while optimizing performance. ```sql An illustration demonstrating the source sample data and its corresponding target data: -- Source data pk | valid_from | value_txt1 | value_txt2 ====================================================================== 1 | 2024-03-01 00:00:00.0000 | A | x1 1 | 2024-03-12 00:00:00.0000 | B | x1 1 | 2024-03-12 00:00:00.0000 | B | x2 1 | 2024-03-25 00:00:00.0000 | A | x2 2 | 2024-03-01 00:00:00.0000 | A | x1 2 | 2024-03-12 00:00:00.0000 | C | x1 2 | 2024-03-12 00:00:00.0000 | D | x1 2 | 2024-03-13 00:00:00.0000 | C | x1 2 | 2024-03-14 00:00:00.0000 | C | x1 -- Target data pk | valid_period | value_txt1 | value_txt2 =================================================================================================== 1 | PERIOD(TIMESTAMP)[2024-03-01 00:00:00.0, 2024-03-12 00:00:00.0] | A | x1 1 | PERIOD(TIMESTAMP)[2024-03-12 00:00:00.0, 2024-03-25 00:00:00.0] | B | x1 1 | PERIOD(TIMESTAMP)[2024-03-25 00:00:00.0, 9999-12-31 23:59:59.9999] | A | x2 2 | PERIOD(TIMESTAMP)[2024-03-01 00:00:00.0, 2024-03-12 00:00:00.0] | A | x1 2 | PERIOD(TIMESTAMP)[2024-03-12 00:00:00.0, 9999-12-31 23:59:59.9999] | C | x1 ``` #### Common Teradata-specific tasks * *collect statistics* - when a table is created or modified significantly, there might be a need to tell Teradata to collect statistics for the optimizer. It can be done using `COLLECT STATISTICS` command. You can perform this step using dbt's `post-hooks`, e.g.: ```yaml {{ config( post_hook=[ "COLLECT STATISTICS ON {{ this }} COLUMN (column_1, column_2 ...);" ] )}} ``` See [Collecting Statistics documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/RAyUdGfvREwbO9J0DMNpLw) for more information. #### The external tables package The [dbt-external-tables](https://github.com/dbt-labs/dbt-external-tables) package is supported with the dbt-teradata adapter from v1.9.3 onwards. Under the hood, dbt-teradata uses the concept of foreign tables to create tables from external sources. More information can be found in the [Teradata documentation](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Data-Definition-Language-Syntax-and-Examples/Table-Statements/CREATE-FOREIGN-TABLE). You need to add the `dbt-external-tables` package as a dependency: ```yaml packages: - package: dbt-labs/dbt_external_tables version: [">=0.9.0", "<1.0.0"] ``` You need to add the dispatch config for the project to pick the overridden macros from the dbt-teradata package: ```yaml dispatch: - macro_namespace: dbt_external_tables search_order: ['dbt', 'dbt_external_tables'] ``` To define `STOREDAS` and `ROWFORMAT` for external tables, one of the following options can be used: * You can use the standard dbt-external-tables config `file_format` and `row_format` respectively. * Or you can add it in the `USING` config as mentioned in the [Teradata documentation](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Data-Definition-Language-Syntax-and-Examples/Table-Statements/CREATE-FOREIGN-TABLE/CREATE-FOREIGN-TABLE-Syntax-Elements/USING-Clause). For the external sources, which require authentication, you need to create an authentication object and pass it in `tbl_properties` as `EXTERNAL SECURITY` object. For more information on authentication objects, check out the [Teradata documentation](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Data-Definition-Language-Syntax-and-Examples/Authorization-Statements-for-External-Routines/CREATE-AUTHORIZATION-and-REPLACE-AUTHORIZATION). The following are examples of external sources configured for Teradata: ```yaml sources: - name: teradata_external schema: "{{ target.schema }}" loader: S3 tables: - name: people_csv_partitioned external: location: "/s3/s3.amazonaws.com/dbt-external-tables-testing/csv/" file_format: "TEXTFILE" row_format: '{"field_delimiter":",","record_delimiter":"\n","character_set":"LATIN"}' using: | PATHPATTERN ('$var1/$section/$var3') tbl_properties: | MAP = TD_MAP1 ,EXTERNAL SECURITY MyAuthObj partitions: - name: section data_type: CHAR(1) columns: - name: id data_type: int - name: first_name data_type: varchar(64) - name: last_name data_type: varchar(64) - name: email data_type: varchar(64) ``` ```yaml sources: - name: teradata_external schema: "{{ target.schema }}" loader: S3 tables: - name: people_json_partitioned external: location: '/s3/s3.amazonaws.com/dbt-external-tables-testing/json/' using: | STOREDAS('TEXTFILE') ROWFORMAT('{"record_delimiter":"\n", "character_set":"cs_value"}') PATHPATTERN ('$var1/$section/$var3') tbl_properties: | MAP = TD_MAP1 ,EXTERNAL SECURITY MyAuthObj partitions: - name: section data_type: CHAR(1) ``` ##### `temporary_metadata_generation_schema` (previously `fallback_schema`) The dbt-teradata adapter internally creates temporary tables to fetch the metadata of views for manifest and catalog creation. If you lack permission to create tables on the schema you are working with, you can define a `temporary_metadata_generation_schema` (to which you have the proper `create`/`drop` privileges) in the `dbt_project.yml` as a variable. ```yaml vars: temporary_metadata_generation_schema: <schema-name> ``` --- ### Test selection examples Test selection works a little differently from other resource selection. This makes it very easy to: * run tests on a particular model * run tests on all models in a subdirectory * run tests on all models upstream / downstream of a model, etc. Like all resource types, tests can be selected **directly**, by methods and operators that capture one of their attributes: their name, properties, tags, etc. Unlike other resource types, tests can also be selected *indirectly* through relationships in your DAG. If a selection method or operator includes a model that a test depends on, dbt will also select that test. For example, when you run `dbt test --select model_b`, dbt includes tests defined on `model_b` as well as tests on related models (like `model_a`) that reference `model_b`. Refer to [Indirect selection](#indirect-selection) for more details on controlling this behavior. Test selection is powerful, and we know it can be tricky. To that end, we've included lots of examples below: ##### Direct selection Use the `test_type` selector to run a specific category of tests without relying on model selection. This is useful when you want to isolate unit tests from data tests — for example, running only unit tests during development for fast feedback, or running only data tests in production where unit test compute isn't needed. The `test_type` selector works across all engines (dbt v1 and dbt v2). The following configurations can be used with `test_type` selector to compile matching test nodes without executing them. For command behavior, refer to [About dbt compile command](https://docs.getdbt.com/reference/commands/compile.md). **Run only unit tests** — use this during development or CI to validate SQL logic before materializing models: ```bash dbt test --select "test_type:unit" ``` **Run all data tests** (includes both generic and singular) — use this to skip unit tests entirely, for example in production pipelines: ```bash dbt test --select "test_type:data" ``` **Run only generic data tests** — use this to run schema-level assertions defined in `.yml` files (such as `not_null` and `unique`), without running custom SQL test files: ```bash dbt test --select "test_type:generic" ``` **Run only singular data tests** — use this to run only custom SQL test files from your `tests/` directory, without running generic schema tests: ```bash dbt test --select "test_type:singular" ``` In all cases, `test_type` checks a property of the test itself — these are forms of "direct" test selection. ##### Indirect selection Indirect selection modes control which tests run based on the models you select and their relationships in your DAG. These modes determine how dbt handles tests that reference your selected models, either directly or through upstream/downstream relationships. You can use the following modes (with `eager` as the default). Test exclusion is always greedy: if ANY parent is explicitly excluded, the test will be excluded as well. Building subsets of a DAG The `buildable` and `cautious` modes can be useful when you're only building a subset of your DAG, and you want to avoid test failures in `eager` mode caused by unbuilt resources. You can also achieve this with [deferral](https://docs.getdbt.com/reference/node-selection/defer.md). ###### Eager mode (default) Most inclusive and runs tests if *any* of the parent nodes are selected, regardless of whether all dependencies are met. This includes *any* tests that reference the selected nodes, even if they also reference other unselected nodes. For example, if you run `dbt test --select model_b`, eager mode will run: * Tests directly on `model_b` * Tests in upstream models (like `model_a`) that reference `model_b` * Tests in downstream models that reference `model_b` dbt builds models that depend on the selected model. In this mode, any tests depending on unbuilt resources will raise an error. ###### Buildable mode Buildable mode is a middle ground between `cautious` and `eager`, running only tests that reference selected nodes (or their ancestors). This mode is slightly more inclusive than `cautious` by including tests whose references are each within the selected nodes (or their ancestors). This mode is useful when a test depends on a model *and* a direct ancestor of that model, like confirming an aggregation has the same totals as its input. ###### Cautious mode Cautious is the most exclusive mode and ensures that tests are executed and models are built only when all necessary dependencies of the selected models are met. Restricts tests to only those that exclusively reference selected nodes. Tests will only be executed if all the nodes they depend on are selected, which prevents tests from running if one or more of its parent nodes are unselected and, consequently, unbuilt. ###### Empty mode Empty mode runs no tests and restricts the build to the selected node, ignoring all indirect dependencies. It doesn't execute any tests, whether they are directly attached to the selected node or not. The empty mode is automatically used for [interactive compilation](https://docs.getdbt.com/reference/commands/compile.md#interactive-compile). ##### Indirect selection examples To visualize these methods, suppose you have `model_a`, `model_b`, and `model_c` and associated data tests. The following illustrates which tests will be run when you execute `dbt build` with the various indirect selection modes: [![dbt build](/img/docs/reference/indirect-selection-dbt-build.png?v=2 "dbt build")](#)dbt build [![Eager (default)](/img/docs/reference/indirect-selection-eager.png?v=2 "Eager (default)")](#)Eager (default) [![Buildable](/img/docs/reference/indirect-selection-buildable.png?v=2 "Buildable")](#)Buildable [![Cautious](/img/docs/reference/indirect-selection-cautious.png?v=2 "Cautious")](#)Cautious [![Empty](/img/docs/reference/indirect-selection-empty.png?v=2 "Empty")](#)Empty ##### Eager mode (default) In this example, during the build process, any test that depends on the selected "orders" model or its dependent models will be executed, even if it depends other models as well. ```shell dbt test --select "orders" dbt build --select "orders" ``` ##### Buildable mode In this example, dbt executes tests that reference "orders" within the selected nodes (or their ancestors). ```shell dbt test --select "orders" --indirect-selection=buildable dbt build --select "orders" --indirect-selection=buildable ``` ##### Cautious mode In this example, only tests that depend *exclusively* on the "orders" model will be executed: ```shell dbt test --select "orders" --indirect-selection=cautious dbt build --select "orders" --indirect-selection=cautious ``` ##### Empty mode This mode does not execute any tests, whether they are directly attached to the selected node or not. ```shell dbt test --select "orders" --indirect-selection=empty dbt build --select "orders" --indirect-selection=empty ``` ##### Test selection syntax examples Setting `indirect_selection` can also be specified in a [yaml selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md#indirect-selection). The following examples use *eager* mode by default for indirect selection, unless you specify another mode (like `--indirect-selection=cautious`). The selection operators (`+`, `tags`, and so on) determine which models are selected; the indirect selection mode determines which tests run for those models. ```bash # Run tests on a model (indirect selection) dbt test --select "customers" # Run tests on two or more specific models (indirect selection) dbt test --select "customers orders" # Run tests on all models in the models/staging/jaffle_shop directory (indirect selection) dbt test --select "staging.jaffle_shop" # Run tests downstream of a model (note this will select those tests directly!) dbt test --select "stg_customers+" # Run tests upstream of a model (indirect selection) dbt test --select "+stg_customers" # Run tests on all models with a particular tag (direct + indirect) dbt test --select "tag:my_model_tag" # Run tests on all models with a particular materialization (indirect selection) dbt test --select "config.materialized:table" # To change the indirect selection mode, add the flag: dbt test --select "customers" --indirect-selection=cautious ``` The same principle can be extended to tests defined on other resource types. In these cases, we will execute all tests defined on certain sources via the `source:` selection method: ```bash # tests on all sources dbt test --select "source:*" # tests on one source dbt test --select "source:jaffle_shop" # tests on two or more specific sources dbt test --select "source:jaffle_shop source:raffle_bakery" # tests on one source table dbt test --select "source:jaffle_shop.customers" # tests on everything _except_ sources dbt test --exclude "source:*" ``` ##### More complex selection Through the combination of direct and indirect selection, there are many ways to accomplish the same outcome. Let's say we have a data test named `assert_total_payment_amount_is_positive` that depends on a model named `payments`. All of the following would manage to select and execute that test specifically: ```bash dbt test --select "assert_total_payment_amount_is_positive" # directly select the test by name dbt test --select "payments,test_type:singular" # indirect selection, v1.2 ``` As long as you can select a common property of a group of resources, indirect selection allows you to execute all the tests on those resources, too. In the example above, we saw it was possible to test all table-materialized models. This principle can be extended to other resource types, too: ```bash # Run tests on all models with a particular materialization dbt test --select "config.materialized:table" # Run tests on all seeds, which use the 'seed' materialization dbt test --select "config.materialized:seed" # Run tests on all snapshots, which use the 'snapshot' materialization dbt test --select "config.materialized:snapshot" ``` Note that this functionality may change in future versions of dbt. ##### Run tests on tagged columns Because the column `order_id` is tagged `my_column_tag`, the test itself also receives the tag `my_column_tag`. Because of that, this is an example of direct selection. models/\<filename>.yml ```yml models: - name: orders columns: - name: order_id config: tags: [my_column_tag] # changed to config in v1.10 and backported to 1.9 data_tests: - unique ``` ```bash dbt test --select "tag:my_column_tag" ``` Currently, tests "inherit" tags applied to columns, sources, and source tables. They do *not* inherit tags applied to models, seeds, or snapshots. In all likelihood, those tests would still be selected indirectly, because the tag selects its parent. This is a subtle distinction, and it may change in future versions of dbt. ##### Run tagged tests only This is an even clearer example of direct selection: the test itself is tagged `my_test_tag`, and selected accordingly. models/\<filename>.yml ```yml models: - name: orders columns: - name: order_id data_tests: - unique: config: tags: [my_test_tag] # changed to config in v1.10 ``` ```bash dbt test --select "tag:my_test_tag" ``` --- ### test-paths dbt\_project.yml ```yml test-paths: [directorypath] ``` #### Definition Optionally specify a custom list of directories where [singular tests](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests) and [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests) are located. #### Default Without specifying this config, dbt will search for tests in the `tests` directory, i.e. `test-paths: ["tests"]`. Specifically, it will look for `.sql` files containing: * Generic test definitions in the `tests/generic` subdirectory * Singular tests (all other files) Paths specified in `test-paths` must be relative to the location of your `dbt_project.yml` file. Avoid using absolute paths like `/Users/username/project/test`, as it will lead to unexpected behavior and outcomes. * ✅ **Do** * Use relative path: ```yml test-paths: ["test"] ``` * ❌ **Don't:** * Avoid absolute paths: ```yml test-paths: ["/Users/username/project/test"] ``` #### Examples ##### Use a subdirectory named `custom_tests` instead of `tests` for data tests dbt\_project.yml ```yml test-paths: ["custom_tests"] ``` --- ### type Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: function_name config: type: scalar | aggregate ``` In the future, we're considering adding support for `table` type. Refer to [this issue](https://github.com/dbt-labs/dbt/issues/11917) to track the progress and provide any feedback. #### Definition The `type` config specifies the type of user-defined function (UDF) you're creating. This config is optional and defaults to `scalar` if not specified. #### Supported function types The following function types are supported: * [scalar (default)](#scalar-default) * [aggregate](#aggregate) Support for `type` differs based on the warehouse and language (SQL or Python) you're using: | Adapter | scalar SQL | scalar Python | aggregate SQL | aggregate Python | | -------------- | ---------- | ------------- | ------------- | ---------------- | | dbt-bigquery | ✅ | ✅ | ✅ | ❌ | | dbt-snowflake | ✅ | ✅ | ❌ | ✅ | | dbt-databricks | ✅ | ❌ | ❌ | ❌ | | dbt-postgres | ✅ | ❌ | ❌ | ❌ | | dbt-redshift | ✅ | ❌ | ❌ | ❌ | ##### scalar (default) A scalar function returns a single value for each row of input. This is the most common type of UDF. **Example use cases:** * Data validation (checking if a string matches a pattern) * Data transformation (converting formats, cleaning strings) * Custom calculations (complex mathematical operations) functions/schema.yml ```yml functions: - name: is_positive_int description: Determines if a string represents a positive integer config: type: scalar arguments: - name: input_string data_type: STRING returns: data_type: BOOLEAN ``` ##### aggregate Aggregate functions operate on multiple rows and return a single value — for example, they sum values or calculate an average for a group. Queries use these functions in `GROUP BY` operations. Aggregate functions are currently supported only for: * Python functions on Snowflake * SQL functions on BigQuery **Example use cases:** * Calculating totals or averages for groups of data (for example, total sales per customer) * Aggregating data over time (for example, daily, monthly, or yearly totals) functions/schema.yml ```yml functions: - name: double_total description: Sums values and doubles the result config: type: aggregate arguments: - name: values data_type: FLOAT description: A sequence of numbers to aggregate returns: data_type: FLOAT ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [volatility](https://docs.getdbt.com/reference/resource-configs/volatility.md) * [arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) * [returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### Unique project resource names | require\_unique\_project\_resource\_names | dbt **v1 Latest** | dbt v1 | | ----------------------------------------- | ----------------- | ------ | | Introduced | 2025.12 | 1.11.0 | | Matured (default → `true`) | — | — | | Removed | — | — | The `require_unique_project_resource_names` flag enforces uniqueness of resource names within the same package. dbt resources such as models, seeds, snapshots, analyses, tests, and functions share a common namespace. When two resources in the same package have the same name, dbt must decide which one a `ref()` or `source()` refers to. Previously, this check was not always enforced, which meant duplicate names could result in dbt referencing the wrong resource. The `require_unique_project_resource_names` flag is set to `false` by default. With this setting, if two unversioned resources in the same package share the same name, dbt continues to run and raises a [`DuplicateNameDistinctNodeTypesDeprecation`](https://docs.getdbt.com/reference/deprecations.md#duplicatenamedistinctnodetypesdeprecation) warning. When set to `true`, dbt raises a `DuplicateResourceNameError` error. For example, if your project contains a model and a seed named `sales`: ```text models/sales.sql seeds/sales.csv ``` And a model contains: ```sql select * from {{ ref('sales') }} ``` When the flag is set to `true`, dbt will raise: ```text DuplicateResourceNameError: Found resources with the same name 'sales' in package 'project': 'model.project.sales' and 'seed.project.sales'. Please update one of the resources to have a unique name. ``` When this error is raised, rename one of the resources or refactor the project structure to avoid name conflicts. --- ### unique_key unique\_key identifies records for incremental models or snapshots, ensuring changes are captured or updated correctly. ##### Models Configure the `unique_key` in the `config` block of your [incremental model's](https://docs.getdbt.com/docs/build/incremental-models.md) SQL file, in your `models/properties.yml` file, or in your `dbt_project.yml` file. models/my\_incremental\_model.sql ```sql {{ config( materialized='incremental', unique_key='id' ) }} ``` models/properties.yml ```yaml models: - name: my_incremental_model description: "An incremental model example with a unique key." config: materialized: incremental unique_key: id ``` dbt\_project.yml ```yaml name: jaffle_shop models: jaffle_shop: staging: +unique_key: id ``` ##### Snapshots (Applies to dbt v1.9 and later) For [snapshots](https://docs.getdbt.com/docs/build/snapshots.md), configure the `unique_key` in the your `snapshot/filename.yml` file or in your `dbt_project.yml` file. snapshots/\<filename>.yml ```yaml snapshots: - name: orders_snapshot relation: source('my_source', 'my_table') config: unique_key: order_id ``` dbt\_project.yml ```yml snapshots: <resource-path>: +unique_key: column_name_or_expression ``` #### Description A column name or expression that uniquely identifies each record in the inputs of a snapshot or incremental model. dbt uses this key to match incoming records to existing records in the target table (either a snapshot or an incremental model) so that changes can be captured or updated correctly: * In an incremental model, dbt replaces the old row (like a merge key or upsert). * In a snapshot, dbt keeps history, storing multiple rows for that same `unique_key` as it evolves over time. For best practices on choosing incremental models, snapshots, or both for change data capture, refer to [Choosing incremental models or snapshots](https://docs.getdbt.com/best-practices/how-we-handle-cdc/2-choosing-incremental-or-snapshots.md). In dbt **v1 Latest** release track and from dbt v1.9, [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) are defined and configured in YAML files within your `snapshots/` directory. You can specify one or multiple `unique_key` values within your snapshot YAML file's `config` key. caution Providing a non-unique key will result in unexpected snapshot results. dbt **will not** test the uniqueness of this key, consider [testing](https://docs.getdbt.com/blog/primary-key-testing#how-to-test-primary-keys-with-dbt) the source data to ensure that this key is indeed unique. #### Default This parameter is optional. If you don't provide a `unique_key`, your adapter will default to using `incremental_strategy: append`. If you leave out the `unique_key` parameter and use strategies like `merge`, `insert_overwrite`, `delete+insert`, or `microbatch`, the adapter will fall back to using `incremental_strategy: append`. This is different for BigQuery: * For `incremental_strategy = merge`, you must provide a `unique_key`; leaving it out leads to ambiguous or failing behavior. * For `insert_overwrite` or `microbatch`, `unique_key` is not required because they work by partition replacement rather than row-level upserts. #### Examples ##### Use an `id` column as a unique key ##### Models In this example, the `id` column is the unique key for an incremental model. models/my\_incremental\_model.sql ```sql {{ config( materialized='incremental', unique_key='id' ) }} select * from .. ``` ##### Snapshots In this example, the `id` column is used as a unique key for a snapshot. (Applies to dbt v1.9 and later) snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: timestamp updated_at: updated_at ``` You can also specify configurations in your `dbt_project.yml` file if multiple snapshots share the same `unique_key`: dbt\_project.yml ```yml snapshots: <resource-path>: +unique_key: id ``` (Applies to dbt v1.9 and later) ##### Use multiple unique keys ##### Models Configure multiple unique keys for an incremental model as a string representing a single column or a list of single-quoted column names that can be used together, for example, `['col1', 'col2', …]`. Columns must not contain null values, otherwise the incremental model will fail to match rows and generate duplicate rows. Refer to [Defining a unique key](https://docs.getdbt.com/docs/build/incremental-models.md#defining-a-unique-key-optional) for more information. models/my\_incremental\_model.sql ```sql {{ config( materialized='incremental', unique_key=['order_id', 'location_id'] ) }} with... ``` ##### Snapshots You can configure snapshots to use multiple unique keys for `primary_key` columns. snapshots/transaction\_items\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: - order_id - product_id strategy: timestamp updated_at: updated_at ``` --- ### Unit test overrides When configuring your unit test, you can override the output of [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros), [project variables](https://docs.getdbt.com/docs/build/project-variables.md), or [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) for a given unit test. models/schema.yml ```yml - name: test_my_model_overrides model: my_model given: - input: ref('my_model_a') rows: - {id: 1, a: 1} - input: ref('my_model_b') rows: - {id: 1, b: 2} - {id: 2, b: 2} overrides: macros: type_numeric: override invocation_id: 123 vars: my_test: var_override env_vars: MY_TEST: env_var_override expect: rows: - {macro_call: override, var_call: var_override, env_var_call: env_var_override, invocation_id: 123} ``` #### Macros You can override the output of any macro referenced directly by the model being unit tested in your unit test definition. Overrides apply only to macros, variables, and environment variables that are referenced directly within the model being unit tested. If a macro, variable, or environment variable is only referenced indirectly (for example, inside a macro that your model calls), the override will not be applied. If the model you're unit testing uses these macros, you must override them: * [`is_incremental`](https://docs.getdbt.com/docs/build/incremental-models.md#understand-the-is_incremental-macro): If you're unit testing an incremental model, you must explicitly set `is_incremental` to `true` or `false`. See more docs on unit testing incremental models [here](https://docs.getdbt.com/docs/build/unit-tests.md#unit-testing-incremental-models). models/schema.yml ```yml unit_tests: - name: my_unit_test model: my_incremental_model overrides: macros: # unit test this model in "full refresh" mode is_incremental: false ... ``` * [`dbt_utils.star`](https://docs.getdbt.com/blog/star-sql-love-letter): If you're unit testing a model that uses the `star` macro, you must explicitly set `star` to a list of columns. This is because the `star` only accepts a [relation](https://docs.getdbt.com/reference/dbt-classes.md#relation) for the `from` argument; the unit test mock input data is injected directly into the model SQL, replacing the `ref('')` or `source('')` function, causing the `star` macro to fail unless overridden. models/schema.yml ```yml unit_tests: - name: my_other_unit_test model: my_model_that_uses_star overrides: macros: # explicitly set star to relevant list of columns dbt_utils.star: col_a,col_b,col_c ... ``` --- ### Unit testing versioned SQL models If your model has multiple versions, the default unit test will run on *all* versions of your model. To specify version(s) of your model to unit test, use `include` or `exclude` for the desired versions in your model versions config: models/schema.yml ```yaml # my test_is_valid_email_address unit test will run on all versions of my_model unit_tests: - name: test_is_valid_email_address model: my_model ... # my test_is_valid_email_address unit test will run on ONLY version 2 of my_model unit_tests: - name: test_is_valid_email_address model: my_model versions: include: - 2 ... # my test_is_valid_email_address unit test will run on all versions EXCEPT 1 of my_model unit_tests: - name: test_is_valid_email_address model: my_model versions: exclude: - 1 ... ``` --- ### updated_at (Applies to dbt v1.9 and later) snapshots/snapshots.yml ```yaml snapshots: - name: snapshot relation: source('my_source', 'my_table') config: strategy: timestamp updated_at: column_name ``` dbt\_project.yml ```yml snapshots: <resource-path>: +strategy: timestamp +updated_at: column_name ``` (Applies to dbt v1.9 and later) caution You will get a warning if the data type of the `updated_at` column does not match the adapter-configured default. #### Description A column within the results of your snapshot query that represents when the record row was last updated. This parameter is **required if using the `timestamp` [strategy](https://docs.getdbt.com/reference/resource-configs/strategy.md)**. The `updated_at` field may support ISO date strings and unix epoch integers, depending on the data platform you use. #### Default No default is provided. #### Examples ##### Use a column name `updated_at` (Applies to dbt v1.9 and later) snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: timestamp updated_at: updated_at ``` ##### Coalesce two columns to create a reliable `updated_at` column Consider a data source that only has an `updated_at` column filled in when a record is updated (so a `null` value indicates that the record hasn't been updated after it was created). Since the `updated_at` configuration only takes a column name, rather than an expression, you should update your snapshot query to include the coalesced column. (Applies to dbt v1.9 and later) 1. Create an staging model to perform the transformation. In your `models/` directory, create a SQL file that configures an staging model to coalesce the `updated_at` and `created_at` columns into a new column `updated_at_for_snapshot`. models/staging\_orders.sql ```sql select * coalesce (updated_at, created_at) as updated_at_for_snapshot from {{ source('jaffle_shop', 'orders') }} ``` 2. Define the snapshot configuration in a YAML file. In your `snapshots/` directory, create a YAML file that defines your snapshot and references the `updated_at_for_snapshot` staging model you just created. snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: ref('staging_orders') config: schema: snapshots unique_key: id strategy: timestamp updated_at: updated_at_for_snapshot ``` 3. Run `dbt snapshot` to execute the snapshot. Alternatively, you can also create an ephemeral model to performs the required transformations. Then, you reference this model in your snapshot's `relation` key. --- ### Upsolver configurations #### Supported Upsolver SQLake functionality | COMMAND | STATE | MATERIALIZED | | ---------------------- | ------------- | ---------------- | | SQL compute cluster | not supported | - | | SQL connections | supported | connection | | SQL copy job | supported | incremental | | SQL merge job | supported | incremental | | SQL insert job | supported | incremental | | SQL materialized views | supported | materializedview | | Expectations | supported | incremental | #### Configs materialization | Config | Required | Materialization | Description | Example | | ---------------------- | -------- | ---------------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | connection\_type | Yes | connection | Connection identifier: S3/GLUE\_CATALOG/KINESIS | connection\_type='S3' | | connection\_options | Yes | connection | Dictionary of options supported by selected connection | connection\_options={ 'aws\_role': 'aws\_role', 'external\_id': 'SAMPLES', 'read\_only': True } | | incremental\_strategy | No | incremental | Define one of incremental strategies: merge/copy/insert. Default: copy | incremental\_strategy='merge' | | source | No | incremental | Define source to copy from: S3/KAFKA/KINESIS | source = 'S3' | | target\_type | No | incremental | Define target type REDSHIFT/ELASTICSEARCH/S3/SNOWFLAKE/POSTGRES. Default None for Data lake | target\_type='Snowflake' | | target\_prefix | False | incremental | Define PREFIX for ELASTICSEARCH target type | target\_prefix = 'orders' | | target\_location | False | incremental | Define LOCATION for S3 target type | target\_location = 's3://your-bucket-name/path/to/folder/' | | schema | Yes/No | incremental | Define target schema. Required if target\_type, no table created in a metastore connection | schema = 'target\_schema' | | database | Yes/No | incremental | Define target connection. Required if target\_type, no table created in a metastore connection | database = 'target\_connection' | | alias | Yes/No | incremental | Define target table. Required if target\_type, no table created in a metastore connection | alias = 'target\_table' | | delete\_condition | No | incremental | Records that match the ON condition and a delete condition can be deleted | delete\_condition='nettotal > 1000' | | partition\_by | No | incremental | List of dictionaries to define partition\_by for target metastore table | partition\_by=\[{'field':'$field\_name'}] | | primary\_key | No | incremental | List of dictionaries to define partition\_by for target metastore table | primary\_key=\[{'field':'customer\_email', 'type':'string'}] | | map\_columns\_by\_name | No | incremental | Maps columns from the SELECT statement to the table. Boolean. Default: False | map\_columns\_by\_name=True | | sync | No | incremental/materializedview | Boolean option to define if job is synchronized or non-msynchronized. Default: False | sync=True | | options | No | incremental/materializedview | Dictionary of job options | options={ 'START\_FROM': 'BEGINNING', 'ADD\_MISSING\_COLUMNS': True } | #### SQL connection Connections are used to provide Upsolver with the proper credentials to bring your data into SQLake as well as to write out your transformed data to various services. More details on ["Upsolver SQL connections"](https://docs.upsolver.com/sqlake/sql-command-reference/sql-connections) As a dbt model connection is a model with materialized='connection' ```sql {{ config( materialized='connection', connection_type={ 'S3' | 'GLUE_CATALOG' | 'KINESIS' | 'KAFKA'| 'SNOWFLAKE' }, connection_options={} ) }} ``` Running this model will compile CREATE CONNECTION(or ALTER CONNECTION if exists) SQL and send it to Upsolver engine. Name of the connection will be name of the model. #### SQL copy job A COPY FROM job allows you to copy your data from a given source into a table created in a metastore connection. This table then serves as your staging table and can be used with SQLake transformation jobs to write to various target locations. More details on ["Upsolver SQL copy-from"](https://docs.upsolver.com/sqlake/sql-command-reference/sql-jobs/create-job/copy-from) As a dbt model copy job is model with materialized='incremental' ```sql {{ config( materialized='incremental', sync=True|False, source = 'S3'| 'KAFKA' | ... , options={ 'option_name': 'option_value' }, partition_by=[{}] ) }} SELECT * FROM {{ ref(<model>) }} ``` Running this model will compile CREATE TABLE SQL for target type Data lake (or ALTER TABLE if exists) and CREATE COPY JOB(or ALTER COPY JOB if exists) SQL and send it to Upsolver engine. Name of the table will be name of the model. Name of the job will be name of the model plus '\_job' #### SQL insert job An INSERT job defines a query that pulls in a set of data based on the given SELECT statement and inserts it into the designated target. This query is then run periodically based on the RUN\_INTERVAL defined within the job. More details on ["Upsolver SQL insert"](https://docs.upsolver.com/sqlake/sql-command-reference/sql-jobs/create-job/sql-transformation-jobs/insert). As a dbt model insert job is model with materialized='incremental' and incremental\_strategy='insert' ```sql {{ config( materialized='incremental', sync=True|False, map_columns_by_name=True|False, incremental_strategy='insert', options={ 'option_name': 'option_value' }, primary_key=[{}] ) }} SELECT ... FROM {{ ref(<model>) }} WHERE ... GROUP BY ... HAVING COUNT(DISTINCT orderid::string) ... ``` Running this model will compile CREATE TABLE SQL for target type Data lake(or ALTER TABLE if exists) and CREATE INSERT JOB(or ALTER INSERT JOB if exists) SQL and send it to Upsolver engine. Name of the table will be name of the model. Name of the job will be name of the model plus '\_job' #### SQL merge job A MERGE job defines a query that pulls in a set of data based on the given SELECT statement and inserts into, replaces, or deletes the data from the designated target based on the job definition. This query is then run periodically based on the RUN\_INTERVAL defined within the job. More details on ["Upsolver SQL merge"](https://docs.upsolver.com/sqlake/sql-command-reference/sql-jobs/create-job/sql-transformation-jobs/merge). As a dbt model merge job is model with materialized='incremental' and incremental\_strategy='merge' ```sql {{ config( materialized='incremental', sync=True|False, map_columns_by_name=True|False, incremental_strategy='merge', options={ 'option_name': 'option_value' }, primary_key=[{}] ) }} SELECT ... FROM {{ ref(<model>) }} WHERE ... GROUP BY ... HAVING COUNT ... ``` Running this model will compile CREATE TABLE SQL for target type Data lake(or ALTER TABLE if exists) and CREATE MERGE JOB(or ALTER MERGE JOB if exists) SQL and send it to Upsolver engine. Name of the table will be name of the model. Name of the job will be name of the model plus '\_job' #### SQL materialized views When transforming your data, you may find that you need data from multiple source tables in order to achieve your desired result. In such a case, you can create a materialized view from one SQLake table in order to join it with your other table (which in this case is considered the main table). More details on ["Upsolver SQL materialized views"](https://docs.upsolver.com/sqlake/sql-command-reference/sql-jobs/create-job/sql-transformation-jobs/sql-materialized-views). As a dbt model materialized views is model with materialized='materializedview'. ```sql {{ config( materialized='materializedview', sync=True|False, options={'option_name': 'option_value'} ) }} SELECT ... FROM {{ ref(<model>) }} WHERE ... GROUP BY ... ``` Running this model will compile CREATE MATERIALIZED VIEW SQL(or ALTER MATERIALIZED VIEW if exists) and send it to Upsolver engine. Name of the materializedview will be name of the model. #### Expectations/constraints Data quality conditions can be added to your job to drop a row or trigger a warning when a column violates a predefined condition. ```sql WITH EXPECTATION <expectation_name> EXPECT <sql_predicate> ON VIOLATION WARN ``` Expectations can be implemented with dbt constraints Supported constraints: check and not\_null ```yaml models: - name: <model name> # required config: contract: enforced: true # model-level constraints constraints: - type: check columns: ['<column1>', '<column2>'] expression: "column1 <= column2" name: <constraint_name> - type: not_null columns: ['column1', 'column2'] name: <constraint_name> columns: - name: <column3> data_type: string # column-level constraints constraints: - type: not_null - type: check expression: "REGEXP_LIKE(<column3>, '^[0-9]{4}[a-z]{5}$')" name: <constraint_name> ``` #### Projects examples > projects examples link: [github.com/dbt-upsolver/examples/](https://github.com/Upsolver/dbt-upsolver/tree/main/examples) #### Connection options | Option | Storage | Editable | Optional | Config Syntax | | ---------------------------------- | ------------- | -------- | -------- | ------------------------------------------------------------------- | | aws\_role | s3 | True | True | 'aws\_role': `'<aws_role>'` | | external\_id | s3 | True | True | 'external\_id': `'<external_id>'` | | aws\_access\_key\_id | s3 | True | True | 'aws\_access\_key\_id': `'<aws_access_key_id>'` | | aws\_secret\_access\_key | s3 | True | True | 'aws\_secret\_access\_key\_id': `'<aws_secret_access_key_id>'` | | path\_display\_filter | s3 | True | True | 'path\_display\_filter': `'<path_display_filter>'` | | path\_display\_filters | s3 | True | True | 'path\_display\_filters': (`'<filter>'`, ...) | | read\_only | s3 | True | True | 'read\_only': True/False | | encryption\_kms\_key | s3 | True | True | 'encryption\_kms\_key': `'<encryption_kms_key>'` | | encryption\_customer\_managed\_key | s3 | True | True | 'encryption\_customer\_kms\_key': `'<encryption_customer_kms_key>'` | | comment | s3 | True | True | 'comment': `'<comment>'` | | host | kafka | False | False | 'host': `'<host>'` | | hosts | kafka | False | False | 'hosts': (`'<host>'`, ...) | | consumer\_properties | kafka | True | True | 'consumer\_properties': `'<consumer_properties>'` | | version | kafka | False | True | 'version': `'<value>'` | | require\_static\_ip | kafka | True | True | 'require\_static\_ip': True/False | | ssl | kafka | True | True | 'ssl': True/False | | topic\_display\_filter | kafka | True | True | 'topic\_display\_filter': `'<topic_display_filter>'` | | topic\_display\_filters | kafka | True | True | 'topic\_display\_filter': (`'<filter>'`, ...) | | comment | kafka | True | True | 'comment': `'<comment>'` | | aws\_role | glue\_catalog | True | True | 'aws\_role': `'<aws_role>'` | | external\_id | glue\_catalog | True | True | 'external\_id': `'<external_id>'` | | aws\_access\_key\_id | glue\_catalog | True | True | 'aws\_access\_key\_id': `'<aws_access_key_id>'` | | aws\_secret\_access\_key | glue\_catalog | True | True | 'aws\_secret\_access\_key': `'<aws_secret_access_key>'` | | default\_storage\_connection | glue\_catalog | False | False | 'default\_storage\_connection': `'<default_storage_connection>'` | | default\_storage\_location | glue\_catalog | False | False | 'default\_storage\_location': `'<default_storage_location>'` | | region | glue\_catalog | False | True | 'region': `'<region>'` | | database\_display\_filter | glue\_catalog | True | True | 'database\_display\_filter': `'<database_display_filter>'` | | database\_display\_filters | glue\_catalog | True | True | 'database\_display\_filters': (`'<filter>'`, ...) | | comment | glue\_catalog | True | True | 'comment': `'<comment>'` | | aws\_role | kinesis | True | True | 'aws\_role': `'<aws_role>'` | | external\_id | kinesis | True | True | 'external\_id': `'<external_id>'` | | aws\_access\_key\_id | kinesis | True | True | 'aws\_access\_key\_id': `'<aws_access_key_id>'` | | aws\_secret\_access\_key | kinesis | True | True | 'aws\_secret\_access\_key': `'<aws_secret_access_key>'` | | region | kinesis | False | False | 'region': `'<region>'` | | read\_only | kinesis | False | True | 'read\_only': True/False | | max\_writers | kinesis | True | True | 'max\_writers': `<integer>` | | stream\_display\_filter | kinesis | True | True | 'stream\_display\_filter': `'<stream_display_filter>'` | | stream\_display\_filters | kinesis | True | True | 'stream\_display\_filters': (`'<filter>'`, ...) | | comment | kinesis | True | True | 'comment': `'<comment>'` | | connection\_string | snowflake | True | False | 'connection\_string': `'<connection_string>'` | | user\_name | snowflake | True | False | 'user\_name': `'<user_name>'` | | password | snowflake | True | False | 'password': `'<password>'` | | max\_concurrent\_connections | snowflake | True | True | 'max\_concurrent\_connections': `<integer>` | | comment | snowflake | True | True | 'comment': `'<comment>'` | | connection\_string | redshift | True | False | 'connection\_string': `'<connection_string>'` | | user\_name | redshift | True | False | 'user\_name': `'<user_name>'` | | password | redshift | True | False | 'password': `'<password>'` | | max\_concurrent\_connections | redshift | True | True | 'max\_concurrent\_connections': `<integer>` | | comment | redshift | True | True | 'comment': `'<comment>'` | | connection\_string | mysql | True | False | 'connection\_string': `'<connection_string>'` | | user\_name | mysql | True | False | 'user\_name': `'<user_name>'` | | password | mysql | True | False | 'password': `'<password>'` | | comment | mysql | True | True | 'comment': `'<comment>'` | | connection\_string | postgres | True | False | 'connection\_string': `'<connection_string>'` | | user\_name | postgres | True | False | 'user\_name': `'<user_name>'` | | password | postgres | True | False | 'password': `'<password>'` | | comment | postgres | True | True | 'comment': `'<comment>'` | | connection\_string | elasticsearch | True | False | 'connection\_string': `'<connection_string>'` | | user\_name | elasticsearch | True | False | 'user\_name': `'<user_name>'` | | password | elasticsearch | True | False | 'password': `'<password>'` | | comment | elasticsearch | True | True | 'comment': `'<comment>'` | | connection\_string | mongodb | True | False | 'connection\_string': `'<connection_string>'` | | user\_name | mongodb | True | False | 'user\_name': `'<user_name>'` | | password | mongodb | True | False | 'password': `'<password>'` | | timeout | mongodb | True | True | 'timeout': "INTERVAL 'N' SECONDS" | | comment | mongodb | True | True | 'comment': `'<comment>'` | | connection\_string | mssql | True | False | 'connection\_string': `'<connection_string>'` | | user\_name | mssql | True | False | 'user\_name': `'<user_name>'` | | password | mssql | True | False | 'password': `'<password>'` | | comment | mssql | True | True | 'comment': `'<comment>'` | #### Target options | Option | Storage | Editable | Optional | Config Syntax | | --------------------------- | ------------------ | -------- | -------- | ------------------------------------------------------------------------------- | | globally\_unique\_keys | datalake | False | True | 'globally\_unique\_keys': True/False | | storage\_connection | datalake | False | True | 'storage\_connection': `'<storage_connection>'` | | storage\_location | datalake | False | True | 'storage\_location': `'<storage_location>'` | | compute\_cluster | datalake | True | True | 'compute\_cluster': `'<compute_cluster>'` | | compression | datalake | True | True | 'compression': 'SNAPPY/GZIP' | | compaction\_processes | datalake | True | True | 'compaction\_processes': `<integer>` | | disable\_compaction | datalake | True | True | 'disable\_compaction': True/False | | retention\_date\_partition | datalake | False | True | 'retention\_date\_partition': `'<column>'` | | table\_data\_retention | datalake | True | True | 'table\_data\_retention': `'<N DAYS>'` | | column\_data\_retention | datalake | True | True | 'column\_data\_retention': ({'COLUMN' : `'<column>'`,'DURATION': `'<N DAYS>'`}) | | comment | datalake | True | True | 'comment': `'<comment>'` | | storage\_connection | materialized\_view | False | True | 'storage\_connection': `'<storage_connection>'` | | storage\_location | materialized\_view | False | True | 'storage\_location': `'<storage_location>'` | | max\_time\_travel\_duration | materialized\_view | True | True | 'max\_time\_travel\_duration': `'<N DAYS>'` | | compute\_cluster | materialized\_view | True | True | 'compute\_cluster': `'<compute_cluster>'` | | column\_transformations | snowflake | False | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | deduplicate\_with | snowflake | False | True | 'deduplicate\_with': {'COLUMNS' : \['col1', 'col2'],'WINDOW': 'N HOURS'} | | exclude\_columns | snowflake | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | create\_table\_if\_missing | snowflake | False | True | 'create\_table\_if\_missing': True/False} | | run\_interval | snowflake | False | True | 'run\_interval': `'<N MINUTES/HOURS/DAYS>'` | #### Transformation options | Option | Storage | Editable | Optional | Config Syntax | | ---------------------------------- | ------------- | -------- | -------- | ------------------------------------------------------------------------------------------------- | | run\_interval | s3 | False | True | 'run\_interval': `'<N MINUTES/HOURS/DAYS>'` | | start\_from | s3 | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | end\_at | s3 | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | s3 | True | True | 'compute\_cluster': `'<compute_cluster>'` | | comment | s3 | True | True | 'comment': `'<comment>'` | | skip\_validations | s3 | False | True | 'skip\_validations': ('ALLOW\_CARTESIAN\_PRODUCT', ...) | | skip\_all\_validations | s3 | False | True | 'skip\_all\_validations': True/False | | aggregation\_parallelism | s3 | True | True | 'aggregation\_parallelism': `<integer>` | | run\_parallelism | s3 | True | True | 'run\_parallelism': `<integer>` | | file\_format | s3 | False | False | 'file\_format': '(type = `<file_format>`)' | | compression | s3 | False | True | 'compression': 'SNAPPY/GZIP ...' | | date\_pattern | s3 | False | True | 'date\_pattern': `'<date_pattern>'` | | output\_offset | s3 | False | True | 'output\_offset': `'<N MINUTES/HOURS/DAYS>'` | | run\_interval | elasticsearch | False | True | 'run\_interval': `'<N MINUTES/HOURS/DAYS>'` | | routing\_field\_name | elasticsearch | True | True | 'routing\_field\_name': `'<routing_field_name>'` | | start\_from | elasticsearch | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | end\_at | elasticsearch | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | elasticsearch | True | True | 'compute\_cluster': `'<compute_cluster>'` | | skip\_validations | elasticsearch | False | True | 'skip\_validations': ('ALLOW\_CARTESIAN\_PRODUCT', ...) | | skip\_all\_validations | elasticsearch | False | True | 'skip\_all\_validations': True/False | | aggregation\_parallelism | elasticsearch | True | True | 'aggregation\_parallelism': `<integer>` | | run\_parallelism | elasticsearch | True | True | 'run\_parallelism': `<integer>` | | bulk\_max\_size\_bytes | elasticsearch | True | True | 'bulk\_max\_size\_bytes': `<integer>` | | index\_partition\_size | elasticsearch | True | True | 'index\_partition\_size': 'HOURLY/DAILY ...' | | comment | elasticsearch | True | True | 'comment': `'<comment>'` | | custom\_insert\_expressions | snowflake | True | True | 'custom\_insert\_expressions': {'INSERT\_TIME' : 'CURRENT\_TIMESTAMP()','MY\_VALUE': `'<value>'`} | | custom\_update\_expressions | snowflake | True | True | 'custom\_update\_expressions': {'UPDATE\_TIME' : 'CURRENT\_TIMESTAMP()','MY\_VALUE': `'<value>'`} | | keep\_existing\_values\_when\_null | snowflake | True | True | 'keep\_existing\_values\_when\_null': True/False | | add\_missing\_columns | snowflake | False | True | 'add\_missing\_columns': True/False | | run\_interval | snowflake | False | True | 'run\_interval': `'<N MINUTES/HOURS/DAYS>'` | | commit\_interval | snowflake | True | True | 'commit\_interval': `'<N MINUTE[S]/HOUR[S]/DAY[S]>'` | | start\_from | snowflake | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | end\_at | snowflake | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | snowflake | True | True | 'compute\_cluster': `'<compute_cluster>'` | | skip\_validations | snowflake | False | True | 'skip\_validations': ('ALLOW\_CARTESIAN\_PRODUCT', ...) | | skip\_all\_validations | snowflake | False | True | 'skip\_all\_validations': True/False | | aggregation\_parallelism | snowflake | True | True | 'aggregation\_parallelism': `<integer>` | | run\_parallelism | snowflake | True | True | 'run\_parallelism': `<integer>` | | comment | snowflake | True | True | 'comment': `'<comment>'` | | add\_missing\_columns | datalake | False | True | 'add\_missing\_columns': True/False | | run\_interval | datalake | False | True | 'run\_interval': `'<N MINUTES/HOURS/DAYS>'` | | start\_from | datalake | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | end\_at | datalake | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | datalake | True | True | 'compute\_cluster': `'<compute_cluster>'` | | skip\_validations | datalake | False | True | 'skip\_validations': ('ALLOW\_CARTESIAN\_PRODUCT', ...) | | skip\_all\_validations | datalake | False | True | 'skip\_all\_validations': True/False | | aggregation\_parallelism | datalake | True | True | 'aggregation\_parallelism': `<integer>` | | run\_parallelism | datalake | True | True | 'run\_parallelism': `<integer>` | | comment | datalake | True | True | 'comment': `'<comment>'` | | run\_interval | redshift | False | True | 'run\_interval': `'<N MINUTES/HOURS/DAYS>'` | | start\_from | redshift | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | end\_at | redshift | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | redshift | True | True | 'compute\_cluster': `'<compute_cluster>'` | | skip\_validations | redshift | False | True | 'skip\_validations': ('ALLOW\_CARTESIAN\_PRODUCT', ...) | | skip\_all\_validations | redshift | False | True | 'skip\_all\_validations': True/False | | aggregation\_parallelism | redshift | True | True | 'aggregation\_parallelism': `<integer>` | | run\_parallelism | redshift | True | True | 'run\_parallelism': `<integer>` | | skip\_failed\_files | redshift | False | True | 'skip\_failed\_files': True/False | | fail\_on\_write\_error | redshift | False | True | 'fail\_on\_write\_error': True/False | | comment | redshift | True | True | 'comment': `'<comment>'` | | run\_interval | postgres | False | True | 'run\_interval': `'<N MINUTES/HOURS/DAYS>'` | | start\_from | postgres | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | end\_at | postgres | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | postgres | True | True | 'compute\_cluster': `'<compute_cluster>'` | | skip\_validations | postgres | False | True | 'skip\_validations': ('ALLOW\_CARTESIAN\_PRODUCT', ...) | | skip\_all\_validations | postgres | False | True | 'skip\_all\_validations': True/False | | aggregation\_parallelism | postgres | True | True | 'aggregation\_parallelism': `<integer>` | | run\_parallelism | postgres | True | True | 'run\_parallelism': `<integer>` | | comment | postgres | True | True | 'comment': `'<comment>'` | #### Copy options | Option | Storage | Category | Editable | Optional | Config Syntax | | -------------------------- | -------- | --------------- | -------- | -------- | ------------------------------------------------------------------------ | | topic | kafka | source\_options | False | False | 'topic': `'<topic>'` | | exclude\_columns | kafka | job\_options | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | deduplicate\_with | kafka | job\_options | False | True | 'deduplicate\_with': {'COLUMNS' : \['col1', 'col2'],'WINDOW': 'N HOURS'} | | consumer\_properties | kafka | job\_options | True | True | 'consumer\_properties': `'<consumer_properties>'` | | reader\_shards | kafka | job\_options | True | True | 'reader\_shards': `<integer>` | | store\_raw\_data | kafka | job\_options | False | True | 'store\_raw\_data': True/False | | start\_from | kafka | job\_options | False | True | 'start\_from': 'BEGINNING/NOW' | | end\_at | kafka | job\_options | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | kafka | job\_options | True | True | 'compute\_cluster': `'<compute_cluster>'` | | run\_parallelism | kafka | job\_options | True | True | 'run\_parallelism': `<integer>` | | content\_type | kafka | job\_options | True | True | 'content\_type': 'AUTO/CSV/...' | | compression | kafka | job\_options | False | True | 'compression': 'AUTO/GZIP/...' | | column\_transformations | kafka | job\_options | False | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | commit\_interval | kafka | job\_options | True | True | 'commit\_interval': `'<N MINUTE[S]/HOUR[S]/DAY[S]>'` | | skip\_validations | kafka | job\_options | False | True | 'skip\_validations': ('MISSING\_TOPIC') | | skip\_all\_validations | kafka | job\_options | False | True | 'skip\_all\_validations': True/False | | comment | kafka | job\_options | True | True | 'comment': `'<comment>'` | | table\_include\_list | mysql | source\_options | True | True | 'table\_include\_list': (`'<regexFilter>'`, ...) | | column\_exclude\_list | mysql | source\_options | True | True | 'column\_exclude\_list': (`'<regexFilter>'`, ...) | | exclude\_columns | mysql | job\_options | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | column\_transformations | mysql | job\_options | False | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | skip\_snapshots | mysql | job\_options | True | True | 'skip\_snapshots': True/False | | end\_at | mysql | job\_options | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | mysql | job\_options | True | True | 'compute\_cluster': `'<compute_cluster>'` | | snapshot\_parallelism | mysql | job\_options | True | True | 'snapshot\_parallelism': `<integer>` | | ddl\_filters | mysql | job\_options | False | True | 'ddl\_filters': (`'<filter>'`, ...) | | comment | mysql | job\_options | True | True | 'comment': `'<comment>'` | | table\_include\_list | postgres | source\_options | False | False | 'table\_include\_list': (`'<regexFilter>'`, ...) | | column\_exclude\_list | postgres | source\_options | False | True | 'column\_exclude\_list': (`'<regexFilter>'`, ...) | | heartbeat\_table | postgres | job\_options | False | True | 'heartbeat\_table': `'<heartbeat_table>'` | | skip\_snapshots | postgres | job\_options | False | True | 'skip\_snapshots': True/False | | publication\_name | postgres | job\_options | False | False | 'publication\_name': `'<publication_name>'` | | end\_at | postgres | job\_options | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | postgres | job\_options | True | True | 'compute\_cluster': `'<compute_cluster>'` | | comment | postgres | job\_options | True | True | 'comment': `'<comment>'` | | parse\_json\_columns | postgres | job\_options | False | False | 'parse\_json\_columns': True/False | | column\_transformations | postgres | job\_options | False | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | snapshot\_parallelism | postgres | job\_options | True | True | 'snapshot\_parallelism': `<integer>` | | exclude\_columns | postgres | job\_options | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | location | s3 | source\_options | False | False | 'location': `'<location>'` | | date\_pattern | s3 | job\_options | False | True | 'date\_pattern': `'<date_pattern>'` | | file\_pattern | s3 | job\_options | False | True | 'file\_pattern': `'<file_pattern>'` | | initial\_load\_pattern | s3 | job\_options | False | True | 'initial\_load\_pattern': `'<initial_load_pattern>'` | | initial\_load\_prefix | s3 | job\_options | False | True | 'initial\_load\_prefix': `'<initial_load_prefix>'` | | delete\_files\_after\_load | s3 | job\_options | False | True | 'delete\_files\_after\_load': True/False | | deduplicate\_with | s3 | job\_options | False | True | 'deduplicate\_with': {'COLUMNS' : \['col1', 'col2'],'WINDOW': 'N HOURS'} | | end\_at | s3 | job\_options | True | True | 'end\_at': `'<timestamp>/NOW'` | | start\_from | s3 | job\_options | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | compute\_cluster | s3 | job\_options | True | True | 'compute\_cluster': `'<compute_cluster>'` | | run\_parallelism | s3 | job\_options | True | True | 'run\_parallelism': `<integer>` | | content\_type | s3 | job\_options | True | True | 'content\_type': 'AUTO/CSV...' | | compression | s3 | job\_options | False | True | 'compression': 'AUTO/GZIP...' | | comment | s3 | job\_options | True | True | 'comment': `'<comment>'` | | column\_transformations | s3 | job\_options | False | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | commit\_interval | s3 | job\_options | True | True | 'commit\_interval': `'<N MINUTE[S]/HOUR[S]/DAY[S]>'` | | skip\_validations | s3 | job\_options | False | True | 'skip\_validations': ('EMPTY\_PATH') | | skip\_all\_validations | s3 | job\_options | False | True | 'skip\_all\_validations': True/False | | exclude\_columns | s3 | job\_options | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | stream | kinesis | source\_options | False | False | 'stream': `'<stream>'` | | reader\_shards | kinesis | job\_options | True | True | 'reader\_shards': `<integer>` | | store\_raw\_data | kinesis | job\_options | False | True | 'store\_raw\_data': True/False | | start\_from | kinesis | job\_options | False | True | 'start\_from': `'<timestamp>/NOW/BEGINNING'` | | end\_at | kinesis | job\_options | False | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | kinesis | job\_options | True | True | 'compute\_cluster': `'<compute_cluster>'` | | run\_parallelism | kinesis | job\_options | False | True | 'run\_parallelism': `<integer>` | | content\_type | kinesis | job\_options | True | True | 'content\_type': 'AUTO/CSV...' | | compression | kinesis | job\_options | False | True | 'compression': 'AUTO/GZIP...' | | comment | kinesis | job\_options | True | True | 'comment': `'<comment>'` | | column\_transformations | kinesis | job\_options | True | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | deduplicate\_with | kinesis | job\_options | False | True | 'deduplicate\_with': {'COLUMNS' : \['col1', 'col2'],'WINDOW': 'N HOURS'} | | commit\_interval | kinesis | job\_options | True | True | 'commit\_interval': `'<N MINUTE[S]/HOUR[S]/DAY[S]>'` | | skip\_validations | kinesis | job\_options | False | True | 'skip\_validations': ('MISSING\_STREAM') | | skip\_all\_validations | kinesis | job\_options | False | True | 'skip\_all\_validations': True/False | | exclude\_columns | kinesis | job\_options | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | table\_include\_list | mssql | source\_options | True | True | 'table\_include\_list': (`'<regexFilter>'`, ...) | | column\_exclude\_list | mssql | source\_options | True | True | 'column\_exclude\_list': (`'<regexFilter>'`, ...) | | exclude\_columns | mssql | job\_options | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | column\_transformations | mssql | job\_options | False | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | skip\_snapshots | mssql | job\_options | True | True | 'skip\_snapshots': True/False | | end\_at | mssql | job\_options | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | mssql | job\_options | True | True | 'compute\_cluster': `'<compute_cluster>'` | | snapshot\_parallelism | mssql | job\_options | True | True | 'snapshot\_parallelism': `<integer>` | | parse\_json\_columns | mssql | job\_options | False | False | 'parse\_json\_columns': True/False | | comment | mssql | job\_options | True | True | 'comment': `'<comment>'` | | collection\_include\_list | mongodb | source\_options | True | True | 'collection\_include\_list': (`'<regexFilter>'`, ...) | | exclude\_columns | mongodb | job\_options | False | True | 'exclude\_columns': (`'<exclude_column>'`, ...) | | column\_transformations | mongodb | job\_options | False | True | 'column\_transformations': {`'<column>'` : `'<expression>'` , ...} | | skip\_snapshots | mongodb | job\_options | True | True | 'skip\_snapshots': True/False | | end\_at | mongodb | job\_options | True | True | 'end\_at': `'<timestamp>/NOW'` | | compute\_cluster | mongodb | job\_options | True | True | 'compute\_cluster': `'<compute_cluster>'` | | snapshot\_parallelism | mongodb | job\_options | True | True | 'snapshot\_parallelism': `<integer>` | | comment | mongodb | job\_options | True | True | 'comment': `'<comment>'` | --- ### User settings `~/.dbt/user_settings.yml` is a user-scoped configuration file for personal dbt preferences. Because it always lives at `~/.dbt/`, it applies globally across all projects on your machine, regardless of which project you're running or where your `profiles.yml` is located. Today, `user_settings.yml` is primarily written by [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-with-dbt-state) to configure [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) locally (`manage_state`). dbt may add other user-scoped flags to this file over time. Settings in this file are separate from project-level configuration ([`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md)) and connection configuration ([`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md)). Use this file for preferences that belong to you, not the project, as an alternative to setting environment variables or populating an `.env` file. \~/.dbt/user\_settings.yml ```yaml flags: manage_state: true ``` Some dbt commands write to this file automatically. You can also edit it manually. When you run [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-with-dbt-state), dbt writes to `user_settings.yml`: * **Log in with your dbt platform account** * In dbt v2, the CLI prompts you before writing to `user_settings.yml`. * In dbt v1.13, dbt writes `manage_state: true` automatically. If you previously set `manage_state: false`, the CLI prompts you before overwriting it. * Setting `manage_state: true` enables dbt State locally on every `dbt run` or `dbt build`. If `user_settings.yml` already contains a value, [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-with-dbt-state) prompts you before overwriting it. #### Config precedence `user_settings.yml` is the lowest precedence level in dbt's flag hierarchy. More specific settings always override it. For more on flag precedence, refer to [About flags (global configs)](https://docs.getdbt.com/reference/global-configs/about-global-configs.md). 1. CLI option (highest) 2. Environment variable 3. Project scope (`dbt_project.yml`) 4. User scope (`~/.dbt/user_settings.yml`) (lowest) For example, if `manage_state: true` is set in `user_settings.yml` but you run `dbt run --no-manage-state`, the CLI flag takes precedence, and dbt disables dbt State for that invocation. #### When dbt State is enabled locally but not in dbt platform If `manage_state: true` is set in `user_settings.yml` but dbt State is not enabled in your dbt platform account, dbt fails with an error on your next `dbt run` or `dbt build`. To resolve this, either: * [Enable dbt State in your dbt platform account](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). * Disable dbt State locally by setting `manage_state: false` in `user_settings.yml`, or run `dbt run --no-manage-state`. --- ### Using the + prefix Use the + prefix to help clarify the difference between resource paths and configs in dbt\_project.yml files. The `+` prefix is a dbt syntax feature which helps disambiguate between [resource paths](https://docs.getdbt.com/reference/resource-configs/resource-path.md) and configs in [`dbt_project.yml` files](https://docs.getdbt.com/reference/dbt_project.yml.md). * It is not compatible with `dbt_project.yml` files that use [`config-version`](https://docs.getdbt.com/reference/project-configs/config-version.md) 1. * It doesn't apply to: * `config()` Jinja macro within a resource file * config property in a `.yml` file. For example: dbt\_project.yml ```yml name: jaffle_shop config-version: 2 ... models: +materialized: view jaffle_shop: marts: +materialized: table ``` Throughout this documentation, we use the `+` prefix on configuration keys in `dbt_project.yml` files. For projects using [`config-version`](https://docs.getdbt.com/reference/project-configs/config-version.md) 2, dbt expects configuration keys to use the `+` prefix. Specifying configurations without the `+` prefix is [deprecated](https://docs.getdbt.com/reference/deprecations.md#missingplusprefixdeprecation). Folder and file names within resource configurations still do not use the `+` prefix. (Applies to dbt v2.0 and later) Because the `+` prefix is reserved for configs, folder and file names in a [resource path](https://docs.getdbt.com/reference/resource-configs/resource-path.md) must not start with `+`. In dbt v2, dbt raises a warning for any resource path that begins with `+`, and raises an error when you enable the [`require_resource_names_without_plus_prefix`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_plus_prefix.md) behavior change flag. If you have a folder named `+my_folder`, rename it to remove the `+` prefix. The `+` prefix is especially important when you need to disambiguate between [resource paths](https://docs.getdbt.com/reference/resource-configs/resource-path.md) and configs. For example, when: * A config accepts a dictionary as its input, such as [`persist_docs`](https://docs.getdbt.com/reference/resource-configs/persist_docs.md). * A config shares a key with part of a resource path, such as a directory of models named `tags`. dbt has deprecated specifying configurations without [the `+` prefix](https://docs.getdbt.com/reference/dbt_project.yml#the--prefix) in `dbt_project.yml`. Only folder and file names can be specified without the `+` prefix within resource configurations in `dbt_project.yml`. This deprecation warning is only raised for the following adapters: * Snowflake * Databricks * BigQuery * Redshift dbt\_project.yml ```yml name: jaffle_shop config-version: 2 ... models: +persist_docs: # this config is a dictionary, so needs a + prefix relation: true columns: true jaffle_shop: +schema: my_schema +tags: # this is the tags config - "hello" config: tags: # resource path (models/tags/), not the tags config # config: disambiguates path vs. config (required in v1.10+) # The below config applies to models in the models/tags/ directory. +materialized: view ``` **Note:** The use of the `+` prefix in `dbt_project.yml` is distinct from the use of `+` to control config merge behavior (clobber vs. add) in other config settings (specific resource `.yml` and `.sql` files). Currently, the only config which supports `+` for controlling config merge behavior is [`grants`](https://docs.getdbt.com/reference/resource-configs/grants.md#grant-config-inheritance). --- ### Valid schema from generate_schema_name | require\_valid\_schema\_from\_generate\_schema\_name | dbt **v1 Latest** | dbt v1 | | ---------------------------------------------------- | ----------------- | -------- | | Introduced | 2026.1 | 1.12.0a1 | | Matured (default → `true`) | — | — | | Removed | — | — | The `generate_schema_name` macro determines the schema where dbt creates models and other resources. Returning a `null` value from this macro can result in invalid schema names and lead to unpredictable behavior during dbt runs. The `require_valid_schema_from_generate_schema_name` behavior flag is set to `false` by default. When `false`, dbt raises the [`GenerateSchemaNameNullValueDeprecation`](https://docs.getdbt.com/reference/deprecations.md#generateschemanamenullvaluedeprecation) warning when a custom `generate_schema_name` macro returns a `null` value. When `require_valid_schema_from_generate_schema_name` is set to `true`, dbt enforces stricter validation and raises a parsing error. For example, if your project has a custom `generate_schema_name` macro that returns `null`: macros/get\_custom\_schema.sql ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- if custom_schema_name is none -%} {{ return(none) }} {%- else -%} {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` With the default behavior, dbt raises a deprecation warning. When `require_valid_schema_from_generate_schema_name` is set to `true`, dbt raises an error. To resolve this, update your macro to return a valid schema name (`target.schema` in this example): macros/get\_custom\_schema.sql ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- if custom_schema_name is none -%} {{ return(target.schema) }} {%- else -%} {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` --- ### version Model versions, dbt\_project.yml versions, and .yml versions The word "version" appears in multiple places in docs site and with different meanings: * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) — A dbt Mesh feature that enables better governance and data model management by allowing you to track changes and updates to models over time. * [dbt\_project.yml version](https://docs.getdbt.com/reference/project-configs/version.md#dbt_projectyml-versions)(optional) — `dbt_project.yml` version is unrelated to Mesh and refers to the compatibility of the dbt project with a specific version of dbt. * [.yml property file version](https://docs.getdbt.com/reference/project-configs/version.md#yml-property-file-versions)(optional) — Version numbers within .yml property files inform how dbt parses those YAML files. Unrelated to Mesh. dbt projects have two distinct types of `version` tags. This field has a different meaning depending on its location. #### `dbt_project.yml` versions The version tag in a `dbt_project` file represents the version of your dbt project. Starting in dbt version 1.5, `version` in the `dbt_project.yml` is an *optional parameter*. If used, the version must be in a [semantic version](https://semver.org/) format, such as `1.0.0`. The default value is `None` if not specified. For users on dbt version 1.4 or lower, this tag is required, though it isn't currently used meaningfully by dbt. For more on dbt v1 versions, see [About dbt versions](https://docs.getdbt.com/docs/dbt-versions.md). dbt\_project.yml ```yml version: version ``` #### `.yml` property file versions A version tag in a `.yml` property file provides the control tag, which informs how dbt processes property files. Starting from version 1.5, dbt will no longer require this configuration in your resource `.yml` files. If you want to know more about why this tag was previously required, you can refer to the [FAQs](#faqs). For users on dbt version 1.4 or lower, this tag is required, For more on property files, see their general [documentation](https://docs.getdbt.com/reference/define-properties.md) on the same page. ##### Resource property file with version specified \<any valid filename>.yml ```yml version: 2 # Only 2 is accepted by dbt versions up to 1.4.latest. models: ... ``` ##### Resource property file without version specified \<any valid filename>.yml ```yml models: ... ``` #### FAQS Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future --- ### versions Model versions, dbt\_project.yml versions, and .yml versions The word "version" appears in multiple places in docs site and with different meanings: * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) — A dbt Mesh feature that enables better governance and data model management by allowing you to track changes and updates to models over time. * [dbt\_project.yml version](https://docs.getdbt.com/reference/project-configs/version.md#dbt_projectyml-versions)(optional) — `dbt_project.yml` version is unrelated to Mesh and refers to the compatibility of the dbt project with a specific version of dbt. * [.yml property file version](https://docs.getdbt.com/reference/project-configs/version.md#yml-property-file-versions)(optional) — Version numbers within .yml property files inform how dbt parses those YAML files. Unrelated to Mesh. models/\<schema>.yml ```yml models: - name: model_name versions: - v: <version_identifier> # required defined_in: <file_name> # optional -- default is <model_name>_v<v> columns: # specify all columns, or include/exclude columns from the top-level model YAML definition - include: <include_value> exclude: <exclude_list> # specify additional columns - name: <column_name> # required - v: ... # optional latest_version: <version_identifier> ``` The standard convention for naming model versions is `<model_name>_v<v>`. This holds for the file where dbt expects to find the model's definition (SQL or Python), and the alias it will use by default when materializing the model in the database. ##### `v` The version identifier for a version of a model. This value can be numeric (integer or float), or any string. The value of the version identifier is used to order versions of a model relative to one another. If a versioned model does *not* explicitly configure a [`latest_version`](https://docs.getdbt.com/reference/resource-properties/latest_version.md), the highest version number is used as the latest version to resolve `ref` calls to the model without a `version` argument. In general, we recommend that you use a simple "major versioning" scheme for your models: `1`, `2`, `3`, and so on, where each version reflects a breaking change from previous versions. You are able to use other versioning schemes. dbt will sort your version identifiers alphabetically if the values are not all numeric. You should **not** include the letter `v` in the version identifier, as dbt will do that for you. To run a model with multiple versions, you can use the [`--select` flag](https://docs.getdbt.com/reference/node-selection/syntax.md). Refer to [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md#run-a-model-with-multiple-versions) for more information and syntax. ##### `defined_in` The name of the model file (excluding the file extension, e.g. `.sql` or `.py`) where the model version is defined. If `defined_in` is not specified, dbt searches for the definition of a versioned model in a model file named `<model_name>_v<v>`. The **latest** version of a model may also be defined in a file named `<model_name>`, without the version suffix. Model file names must be globally unique, even when defining versioned implementations of a model with a different name. ##### `alias` The default resolved `alias` for a versioned model is `<model_name>_v<v>`. The logic for this is encoded in the `generate_alias_name` macro. This default can be overwritten in two ways: * Configuring a custom `alias` within the version yaml, or the versioned model's definition * Overwriting dbt's `generate_alias_name` macro, to use different behavior based on `node.version` See ["Custom aliases"](https://docs.getdbt.com/docs/build/custom-aliases.md) for more details. Note that the value of `defined_in` and the `alias` configuration of a model are not coordinated, except by convention. The two are declared and determined independently. ##### `include` The specification of which columns are defined in a model's top-level `columns` property to include or exclude in a versioned implementation of that model. * `include` is either: * a list of specific column names to include * `'*'` or `'all'`, indicating that **all** columns from the top-level `columns` property should be included in the versioned model * `exclude` is a list of column names to exclude. It can only be declared if `include` is set to one of `'*'` or `'all'`. tip Not to be confused with the `--select/--exclude` [syntax](https://docs.getdbt.com/reference/node-selection/exclude.md), which is used for model selection. The `columns` list of a versioned model can have *at most one* `include/exclude` element. However, if none of your model versions specify columns, you don't need to define columns at all and can omit the `columns/include`/`exclude` keys from the versioned model. In this case, dbt will automatically use all top-level columns for all versions. You may declare additional columns within the version's `columns` list. If a version-specific column's `name` matches a column included from the top level, the version-specific entry will override that column for that version. models/\<schema>.yml ```yml models: # top-level model properties - name: <model_name> columns: - name: <column_name> # required # versions of this model versions: - v: <version_identifier> # required columns: - include: '*' | 'all' | [<column_name>, ...] exclude: - <column_name> - ... # declare additional column names to exclude # declare more columns -- can be overrides from top-level, or in addition - name: <column_name> ... ``` By default, `include` is "all", and `exclude` is the empty list. This has the effect of including all columns from the base model in the versioned model. ###### Example models/customers.yml ```yml models: - name: customers columns: - name: customer_id description: Unique identifier for this table data_type: text constraints: - type: not_null data_tests: - unique - name: customer_country data_type: text description: "Country where the customer currently lives" - name: first_purchase_date data_type: date versions: - v: 4 - v: 3 columns: - include: "*" - name: customer_country data_type: text description: "Country where the customer first lived at time of first purchase" - v: 2 columns: - include: "*" exclude: - customer_country - v: 1 columns: - include: [] - name: id data_type: int ``` Because `v4` has not specified any `columns`, it will include all of the top-level `columns`. Each other version has declared a modification from the top-level property: * `v3` will include all columns, but it reimplements the `customer_country` column with a different `description`. * `v2` will include all columns *except* `customer_country`. * `v1` doesn't include *any* of the top-level `columns`. Instead, it declares only a single integer column named `id`. ##### Our recommendations (Applies to dbt v1.12 and later) * Follow a consistent naming convention for model versions and aliases. * Use `defined_in` and `alias` only if you have good reason. * Use the [`latest_version_pointer`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) config to automatically create a view pointing to the latest version of your model. Enable it per model or globally with the [`latest_version_pointer_enabled_by_default`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) flag in `dbt_project.yml`. ##### Detecting breaking changes When you use the `state:modified` selection method in Slim CI, dbt will detect changes to versioned model contracts, and raise an error if any of those changes could be breaking for downstream consumers. Breaking changes include: * Removing an existing column * Changing the data\_type of an existing column * Removing or modifying one of the `constraints` on an existing column (dbt v1.6 or higher) * Changing unversioned, contracted models. * dbt also warns if a model has or had a contract but isn't versioned. ##### Example message for unversioned models ```text Breaking Change to Unversioned Contract for contracted_model (models/contracted_models/contracted_model.sql) While comparing to previous project state, dbt detected a breaking change to an unversioned model. - Contract enforcement was removed: Previously, this model's configuration included contract: {enforced: true}. It is no longer configured to enforce its contract, and this is a breaking change. - Columns were removed: - color - date_day - Enforced column level constraints were removed: - id (ConstraintType.not_null) - id (ConstraintType.primary_key) - Enforced model level constraints were removed: - ConstraintType.check -> ['id'] - Materialization changed with enforced constraints: - table -> view ``` ##### Example message for versioned models ```text Breaking Change to Contract Error in model sometable (models/sometable.sql) While comparing to previous project state, dbt detected a breaking change to an enforced contract. The contract's enforcement has been disabled. Columns were removed: - order_name Columns with data_type changes: - order_id (number -> int) Consider making an additive (non-breaking) change instead, if possible. Otherwise, create a new model version: https://docs.getdbt.com/docs/mesh/govern/model-versions ``` Additive changes are **not** considered breaking: * Adding a new column to a contracted model * Adding new `constraints` to an existing column in a contracted model --- ### Vertica configurations #### Configuration of Incremental Models ##### Using the on\_schema\_change config parameter You can use `on_schema_change` parameter with values `ignore`, `fail` and `append_new_columns`. Value `sync_all_columns` is not supported at this time. ###### Configuring the `ignore` (default) parameter ##### Source code vertica\_incremental.sql ```sql {{config(materialized = 'incremental',on_schema_change='ignore')}} select * from {{ ref('seed_added') }} ``` ##### Run code vertica\_incremental.sql ```sql insert into "VMart"."public"."merge" ("id", "name", "some_date") ( select "id", "name", "some_date" from "merge__dbt_tmp" ) ``` ###### Configuring the `fail` parameter ##### Source code vertica\_incremental.sql ```sql {{config(materialized = 'incremental',on_schema_change='fail')}} select * from {{ ref('seed_added') }} ``` ##### Run code vertica\_incremental.sql ```text The source and target schemas on this incremental model are out of sync! They can be reconciled in several ways: - set the `on_schema_change` config to either append_new_columns or sync_all_columns, depending on your situation. - Re-run the incremental model with `full_refresh: True` to update the target schema. - update the schema manually and re-run the process. Additional troubleshooting context: Source columns not in target: {{ schema_changes_dict['source_not_in_target'] }} Target columns not in source: {{ schema_changes_dict['target_not_in_source'] }} New column types: {{ schema_changes_dict['new_target_types'] }} ``` ###### Configuring the `append_new_columns` parameter ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='incremental', on_schema_change='append_new_columns') }} select * from public.seed_added ``` ##### Run code vertica\_incremental.sql ```sql insert into "VMart"."public"."over" ("id", "name", "some_date", "w", "w1", "t1", "t2", "t3") ( select "id", "name", "some_date", "w", "w1", "t1", "t2", "t3" from "over__dbt_tmp" ) ``` ##### Using the `incremental_strategy` config ​parameter **The `append` strategy (default)**: Insert new records without updating or overwriting any existing data. append only adds the new records based on the condition specified in the `is_incremental()` conditional block. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='incremental', incremental_strategy='append' ) }} select * from public.product_dimension {% if is_incremental() %} where product_key > (select max(product_key) from {{this }}) {% endif %} ``` ##### Run code vertica\_incremental.sql ```sql insert into "VMart"."public"."samp" ( "product_key", "product_version", "product_description", "sku_number", "category_description", "department_description", "package_type_description", "package_size", "fat_content", "diet_type", "weight", "weight_units_of_measure", "shelf_width", "shelf_height", "shelf_depth", "product_price", "product_cost", "lowest_competitor_price", "highest_competitor_price", "average_competitor_price", "discontinued_flag") ( select "product_key", "product_version", "product_description", "sku_number", "category_description", "department_description", "package_type_description", "package_size", "fat_content", "diet_type", "weight", "weight_units_of_measure", "shelf_width", "shelf_height", "shelf_depth", "product_price", "product_cost", "lowest_competitor_price", "highest_competitor_price", "average_competitor_price", "discontinued_flag" from "samp__dbt_tmp" ) ``` **The `merge` strategy**: Match records based on a unique\_key; update old records, insert new ones. (If no unique\_key is specified, all new data is inserted, similar to append.) The unique\_key config parameter is required for using the merge strategy, the value accepted by this parameter is a single table column. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', unique_key='promotion_key' ) }} select * FROM public.promotion_dimension ``` ##### Run code vertica\_incremental.sql ```sql merge into "VMart"."public"."samp" as DBT_INTERNAL_DEST using "samp__dbt_tmp" as DBT_INTERNAL_SOURCE on DBT_INTERNAL_DEST."promotion_key" = DBT_INTERNAL_SOURCE."promotion_key" when matched then update set "promotion_key" = DBT_INTERNAL_SOURCE."promotion_key", "price_reduction_type" = DBT_INTERNAL_SOURCE."price_reduction_type", "promotion_media_type" = DBT_INTERNAL_SOURCE."promotion_media_type", "display_type" = DBT_INTERNAL_SOURCE."display_type", "coupon_type" = DBT_INTERNAL_SOURCE."coupon_type", "ad_media_name" = DBT_INTERNAL_SOURCE."ad_media_name", "display_provider" = DBT_INTERNAL_SOURCE."display_provider", "promotion_cost" = DBT_INTERNAL_SOURCE."promotion_cost", "promotion_begin_date" = DBT_INTERNAL_SOURCE."promotion_begin_date", "promotion_end_date" = DBT_INTERNAL_SOURCE."promotion_end_date" when not matched then insert ("promotion_key", "price_reduction_type", "promotion_media_type", "display_type", "coupon_type", "ad_media_name", "display_provider", "promotion_cost", "promotion_begin_date", "promotion_end_date") values ( DBT_INTERNAL_SOURCE."promotion_key", DBT_INTERNAL_SOURCE."price_reduction_type", DBT_INTERNAL_SOURCE."promotion_media_type", DBT_INTERNAL_SOURCE."display_type", DBT_INTERNAL_SOURCE."coupon_type", DBT_INTERNAL_SOURCE."ad_media_name", DBT_INTERNAL_SOURCE."display_provider", DBT_INTERNAL_SOURCE."promotion_cost", DBT_INTERNAL_SOURCE."promotion_begin_date", DBT_INTERNAL_SOURCE."promotion_end_date" ) ``` ###### Using the `merge_update_columns` config parameter The `merge_update_columns` config parameter is passed to only update the columns specified and it accepts a list of table columns. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized = 'incremental', incremental_strategy='merge', unique_key = 'id', merge_update_columns = ["names", "salary"] )}} select * from {{ref('seed_tc1')}} ``` ##### Run code vertica\_incremental.sql ```sql merge into "VMart"."public"."test_merge" as DBT_INTERNAL_DEST using "test_merge__dbt_tmp" as DBT_INTERNAL_SOURCE on DBT_INTERNAL_DEST."id" = DBT_INTERNAL_SOURCE."id" when matched then update set "names" = DBT_INTERNAL_SOURCE."names", "salary" = DBT_INTERNAL_SOURCE."salary" when not matched then insert ("id", "names", "salary") values ( DBT_INTERNAL_SOURCE."id", DBT_INTERNAL_SOURCE."names", DBT_INTERNAL_SOURCE."salary" ) ``` **`delete+insert` strategy**: Through the `delete+insert` incremental strategy, you can instruct dbt to use a two-step incremental approach. It will first delete the records detected through the configured `is_incremental()` block and then re-insert them. The `unique_key` is a required parameter for using `delete+insert` strategy which specifies how to update the records when there is duplicate data. The value accepted by this parameter is a single table column. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized = 'incremental', incremental_strategy = 'delete+insert', unique_key='date_key' ) }} select * FROM public.date_dimension ``` ##### Run code vertica\_incremental.sql ```sql delete from "VMart"."public"."samp" where ( date_key) in ( select (date_key) from "samp__dbt_tmp" ); insert into "VMart"."public"."samp" ( "date_key", "date", "full_date_description", "day_of_week", "day_number_in_calendar_month", "day_number_in_calendar_year", "day_number_in_fiscal_month", "day_number_in_fiscal_year", "last_day_in_week_indicator", "last_day_in_month_indicator", "calendar_week_number_in_year", "calendar_month_name", "calendar_month_number_in_year", "calendar_year_month", "calendar_quarter", "calendar_year_quarter", "calendar_half_year", "calendar_year", "holiday_indicator", "weekday_indicator", "selling_season") ( select "date_key", "date", "full_date_description", "day_of_week", "day_number_in_calendar_month", "day_number_in_calendar_year", "day_number_in_fiscal_month", "day_number_in_fiscal_year", "last_day_in_week_indicator", "last_day_in_month_indicator", "calendar_week_number_in_year", "calendar_month_name", "calendar_month_number_in_year", "calendar_year_month", "calendar_quarter", "calendar_year_quarter", "calendar_half_year", "calendar_year", "holiday_indicator", "weekday_indicator", "selling_season" from "samp__dbt_tmp" ); ``` **`insert_overwrite` strategy**: The `insert_overwrite` strategy does not use a full-table scan to delete records. Instead of deleting records it drops entire partitions. This strategy may accept `partition_by_string` and `partitions` parameters. You provide these parameters when you want to overwrite a part of the table. `partition_by_string` accepts an expression based on which partitioning of the table takes place. This is the PARTITION BY clause in Vertica. `partitions` accepts a list of values in the partition column. The config parameter `partitions` must be used carefully. Two situations to consider: * Fewer partitions in the `partitions` parameter than in the where clause: destination table ends up with duplicates. * More partitions in the `partitions` parameter than in the where clause: destination table ends up missing rows. Less rows in destination than in source. To understand more about PARTITION BY clause check [here](https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Statements/partition-clause.htm) Note: The `partitions` parameter is optional, if the `partitions` parameter is not provided, the partitions in the where clause will be dropped from destination and inserted back from source. If you use a where clause, you might not need the `partitions` parameter. The where clause condition is also optional, but if not provided then all data in source is inserted in destination. If no where clause condition and no `partitions` parameter are provided, then it drops all partitions from the table and inserts all of them again. If the `partitions` parameter is provided but not where clause is provided, the destination table ends up with duplicates because the partitions in the `partitions` parameter are dropped but all data in the source table (no where clause) is inserted in destination. The `partition_by_string` config parameter is also optional. If no `partition_by_string` parameter is provided, then it behaves like `delete+insert`. It deletes all records from destination and then it inserts all records from source. It won’t use or drop partitions. If both the `partition_by_string` and `partitions` parameters are not provided then `insert_overwrite` strategy truncates the target table and insert the source table data into target. If you want to use `partitions` parameter then you have to partition the table by passing `partition_by_string` parameter. ##### Source code vertica\_incremental.sql ```sql {{config(materialized = 'incremental',incremental_strategy = 'insert_overwrite',partition_by_string='YEAR(cc_open_date)',partitions=['2023'])}} select * from online_sales.call_center_dimension ``` ##### Run code vertica\_incremental.sql ```sql select PARTITION_TABLE('online_sales.update_call_center_dimension'); SELECT DROP_PARTITIONS('online_sales.update_call_center_dimension', '2023', '2023'); SELECT PURGE_PARTITION('online_sales.update_call_center_dimension', '2023'); insert into "VMart"."online_sales"."update_call_center_dimension" ("call_center_key", "cc_closed_date", "cc_open_date", "cc_name", "cc_class", "cc_employees", "cc_hours", "cc_manager", "cc_address", "cc_city", "cc_state", "cc_region") ( select "call_center_key", "cc_closed_date", "cc_open_date", "cc_name", "cc_class", "cc_employees", "cc_hours", "cc_manager", "cc_address", "cc_city", "cc_state", "cc_region" from "update_call_center_dimension__dbt_tmp" ); ``` #### Optimization options for table materialization There are multiple optimizations that can be used when materializing models as tables. Each config parameter applies a Vertica specific clause in the generated `CREATE TABLE` DDL. For more information see [Vertica](https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/SQLReferenceManual/Statements/CREATETABLE.htm) options for table optimization. You can configure these optimizations in your model SQL file as described in the examples below: ##### Configuring the `ORDER BY` clause To leverage the `ORDER BY` clause of the `CREATE TABLE` statement use the `order_by` config param in your model. ###### Using the `order_by` config parameter ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='table', order_by='product_key') }} select * from public.product_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."order_s__dbt_tmp" as ( select * from public.product_dimension) order by product_key; ``` ##### Configuring the `SEGMENTED BY` clause To leverage the `SEGMENTED BY` clause of the `CREATE TABLE` statement, use the `segmented_by_string` or `segmented_by_all_nodes` config parameters in your model. By default ALL NODES are used to segment tables, so the ALL NODES clause in the SQL statement will be added when using `segmented_by_string` config parameter. You can disable ALL NODES using `no_segmentation` parameter. To learn more about segmented by clause check [here](https://www.vertica.com/docs/12.0.x/HTML/Content/Authoring/SQLReferenceManual/Statements/hash-segmentation-clause.htm). ###### Using the `segmented_by_string` config parameter `segmented_by_string` config parameter can be used to segment projection data using a SQL expression like hash segmentation. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='table', segmented_by_string='product_key' ) }} select * from public.product_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."segmented_by__dbt_tmp" as (select * from public.product_dimension) segmented by product_key ALL NODES; ``` ###### Using the `segmented_by_all_nodes` config parameter `segmented_by_all_nodes` config parameter can be used to segment projection data for distribution across all cluster nodes. Note: If you want to pass `segmented_by_all_nodes` parameter then you have to segment the table by passing `segmented_by_string` parameter. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='table', segmented_by_string='product_key' ,segmented_by_all_nodes='True' ) }} select * from public.product_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."segmented_by__dbt_tmp" as (select * from public.product_dimension) segmented by product_key ALL NODES; ``` ##### Configuring the UNSEGMENTED ALL NODES clause To leverage the`UNSEGMENTED ALL NODES` clause of the `CREATE TABLE` statement, use the `no_segmentation` config parameters in your model. ###### Using the `no_segmentation` config parameter ##### Source code vertica\_incremental.sql ```sql {{config(materialized='table',no_segmentation='true')}} select * from public.product_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."ww__dbt_tmp" INCLUDE SCHEMA PRIVILEGES as ( select * from public.product_dimension ) UNSEGMENTED ALL NODES ; ``` ##### Configuring the `PARTITION BY` clause To leverage the `PARTITION BY` clause of the `CREATE TABLE` statement, use the `partition_by_string`, `partition_by_active_count` or the `partition_by_group_by_string` config parameters in your model. To learn more about partition by clause check [here](https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Statements/partition-clause.htm) ###### Using the `partition_by_string` config parameter `partition_by_string` (optional) accepts a string value of a any one specific `column_name` based on which partitioning of the table data takes place. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='table', partition_by_string='employee_age' )}} select * FROM public.employee_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."test_partition__dbt_tmp" as ( select * FROM public.employee_dimension); alter table "VMart"."public"."test_partition__dbt_tmp" partition BY employee_age ``` ###### Using the `partition_by_active_count` config parameter `partition_by_active_count` (optional) specifies how many partitions are active for this table. It accepts an integer value. Note: If you want to pass `partition_by_active_count` parameter then you have to partition the table by passing `partition_by_string` parameter. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='table', partition_by_string='employee_age', partition_by_group_by_string=""" CASE WHEN employee_age < 5 THEN 1 WHEN employee_age>50 THEN 2 ELSE 3 END""", partition_by_active_count = 2) }} select * FROM public.employee_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."test_partition__dbt_tmp" as ( select * FROM public.employee_dimension ); alter table "VMart"."public"."test_partition__dbt_tmp" partition BY employee_ag group by CASE WHEN employee_age < 5 THEN 1 WHEN employee_age>50 THEN 2 ELSE 3 END SET ACTIVEPARTITIONCOUNT 2 ; ``` ###### Using the `partition_by_group_by_string` config parameter `partition_by_group_by_string` parameter(optional) accepts a string, in which user should specify each group cases as a single string. This is derived from the `partition_by_string` value. `partition_by_group_by_string` parameter is used to merge partitions into separate partition groups. Note: If you want to pass `partition_by_group_by_string` parameter then you have to partition the table by passing `partition_by_string` parameter. ##### Source code vertica\_incremental.sql ```sql {{config(materialized='table', partition_by_string='number_of_children', partition_by_group_by_string=""" CASE WHEN number_of_children <= 2 THEN 'small_family' ELSE 'big_family' END""")}} select * from public.customer_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."test_partition__dbt_tmp" INCLUDE SCHEMA PRIVILEGES as ( select * from public.customer_dimension ) ; alter table "VMart"."public"."test_partition__dbt_tmp" partition BY number_of_children group by CASE WHEN number_of_children <= 2 THEN 'small_family' ELSE 'big_family' END ; ``` ##### Configuring the KSAFE clause To leverage the `KSAFE` clause of the `CREATE TABLE` statement, use the `ksafe` config parameter in your model. ##### Source code vertica\_incremental.sql ```sql {{ config( materialized='table', ksafe='1' ) }} select * from public.product_dimension ``` ##### Run code vertica\_incremental.sql ```sql create table "VMart"."public"."segmented_by__dbt_tmp" as (select * from public.product_dimension ) ksafe 1; ``` --- ### volatility Did you know... Available from dbt v1.11 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). functions/\<filename>.yml ```yml functions: - name: <function name> config: volatility: deterministic | stable | non-deterministic ``` #### Definition You can optionally use the [`volatility` config](https://docs.getdbt.com/reference/resource-configs/volatility.md) for SQL, Python, or JavaScript UDFs to describe how predictable the function output is by using `deterministic`, `stable`, or `non-deterministic`. Warehouses use this information to decide if results can be cached, reordered, or inlined. Setting the appropriate volatility helps prevent incorrect results when a function isn’t safe to cache or reorder. For example: * A function that returns a random number (`random()`) should be set as `non-deterministic` because its output changes every time it’s called. * A function that returns today’s date (`current_date()`) is `stable`; its value remains consistent within a single query execution but may change between queries. If it were configured as `deterministic`, a warehouse might incorrectly cache the value and reuse it on subsequent days. By default, dbt does not specify a volatility value. If you don’t set volatility, dbt generates a `CREATE` statement without a volatility keyword, and the warehouse’s default behavior applies — except in Redshift. In Redshift, dbt sets `non-deterministic` (`VOLATILE`) by default if no volatility is specified, because Redshift requires an explicit volatility and `VOLATILE` is the safest assumption.  Warehouse-specific volatility keywords Different warehouses show volatility controls with different keywords and default values: | Warehouse | `deterministic` | `stable` | `non-deterministic` | | --------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------------- | ------------------------------------------- | | [Snowflake](https://docs.snowflake.com/en/sql-reference/sql/create-function#sql-handler) | `IMMUTABLE` | Not supported | `VOLATILE` (default) | | [Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_FUNCTION.html#r_CREATE_FUNCTION-synopsis) | `IMMUTABLE` | `STABLE` | `VOLATILE` (default) | | [Databricks](https://docs.databricks.com/aws/en/udf/unity-catalog#set-deterministic-if-your-function-produces-consistent-results) | `DETERMINISTIC` | Not supported | Assumed `non-deterministic` unless declared | | [Postgres](https://www.postgresql.org/docs/current/xfunc-volatility.html) | `IMMUTABLE` | `STABLE` | `VOLATILE` (default) | For SQL and Python UDFs, BigQuery does not support explicitly setting volatility and infers it based on the functions and expressions used. For JavaScript UDFs, BigQuery supports `deterministic` and `non-deterministic` only; `stable` is not supported. #### Supported volatility types In dbt, you can use the following values for the `volatility` config: | Value | Description | Example | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `deterministic` | Always returns the same output for the same input. Safe for aggressive optimizations and caching. | `substr()` — Produces the same substring when given the same string and parameters. | | `stable` | Returns the same value within a single query execution, but may change across executions. Not supported by all warehouses. For more information, see [Warehouse-specific volatility keywords](https://docs.getdbt.com/reference/resource-configs/volatility.md#warehouse-specific-volatility-keywords). | `now()` — Returns the current timestamp the moment a query starts; constant within a single query but different across runs. | | `non-deterministic` | May return different results for the same inputs. Warehouses shouldn't cache or reorder assuming stable results. | `first()` — May return different rows depending on query plan or ordering.<br />`random()` — Produces a random number that varies with each call, even with identical inputs. | #### Example In this example, we're using the `deterministic` volatility for the `is_positive_int` function: functions/schema.yml ```yaml functions: - name: is_positive_int description: Check whether a string is a positive integer config: volatility: deterministic # Optional: stable | non-deterministic | deterministic arguments: - name: a_string data_type: string returns: data_type: boolean ``` #### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Function properties](https://docs.getdbt.com/reference/function-properties.md) * [Function configurations](https://docs.getdbt.com/reference/function-configs.md) * [type](https://docs.getdbt.com/reference/resource-configs/type.md) * [arguments](https://docs.getdbt.com/reference/resource-properties/function-arguments.md) * [returns](https://docs.getdbt.com/reference/resource-properties/returns.md) --- ### Warn-error handler for all warnings | require\_all\_warnings\_handled\_by\_warn\_error | dbt **v1 Latest** | dbt v1 | | ------------------------------------------------ | ----------------- | ------ | | Introduced | 2025.06 | 1.10.0 | | Matured (default → `true`) | 2026.09 | 1.12.0 | | Removed | — | — | <br /> Starting in dbt v1.12, the `require_all_warnings_handled_by_warn_error` flag defaults to `true`. With this flag enabled, all warnings raised during a run are routed through the `--warn-error` / `--warn-error-options` handler. This ensures consistent behavior when promoting warnings to errors or silencing them. When the flag is `false`, only some warnings are processed by the handler while others may bypass it. Projects that use `--warn-error` (or `--warn-error-options='{"error":"all"}'`) may have builds fail on warnings that were previously ignored. #### Impact This only affects projects that use `warn_error: true` or `--warn-error` — common in CI or in dbt platform production jobs configured for strict mode. Projects without `--warn-error` are not affected. Example warnings that switch from "log only" to "error" for `warn_error: true` users: * `JinjaLogWarning` — emitted for column types declared for non-existent seed columns * `WarnStateTargetEqual` — emitted when the `--state` and `--target` directories are the same path * `SelectExcludeIgnoredWithSelectorWarning` — emitted when the legacy `--selector` flag is combined with `--select` or `--exclude` * `RunResultWarningMessage` — emitted when a test or model returns `WARN` status * Various adapter-level warnings not previously handled by `--warn-error`  Recommended actions If a job using `warn_error: true` or `--warn-error` fails with `EventCompilationError` at parse or run time for an event class that previously only appeared as a log line, you can silence specific warnings with `warn_error_options`: dbt\_project.yml ```yaml flags: warn_error_options: silence: - JinjaLogWarning - WarnStateTargetEqual - RunResultWarningMessage - SelectExcludeIgnoredWithSelectorWarning ``` You can also use the `warn` list to keep specific events as warnings even when `--warn-error` would otherwise escalate them to errors. To opt out of this behavior, set the flag to `false`: dbt\_project.yml ```yaml flags: require_all_warnings_handled_by_warn_error: false ``` --- ### Warnings Use the --warn-error flag to promote all warnings to errors or --warn-error-options for granular control through options. #### Use `--warn-error` to promote all warnings to errors Enabling `WARN_ERROR` config or setting the `--warn-error` flag will convert *all* dbt warnings into errors. Any time dbt would normally warn, it will instead raise an error. Examples include `--select` criteria that selects no resources, deprecations, configurations with no associated models, invalid test configurations, or tests and freshness checks that are configured to return warnings. Usage ```text dbt run --warn-error ``` Proceed with caution in production environments Using the `--warn-error` flag or `--warn-error-options '{"error": "all"}'` will treat *all* current and future warnings as errors. This means that if a new warning is introduced in a future version of dbt, your production job may start failing unexpectedly. We recommend proceeding with caution when doing this in production environments, and explicitly listing only the warnings you want to treat as errors in production. #### Use `--warn-error-options` for targeted warnings In some cases, you may want to convert *all* warnings to errors. However, when you want *some* warnings to stay as warnings and only promote or silence specific warnings you can instead use `--warn-error-options`. The `WARN_ERROR_OPTIONS` config or `--warn-error-options` flag gives you more granular control over *exactly which types of warnings* are treated as errors. `WARN_ERROR` and `WARN_ERROR_OPTIONS` are mutually exclusive `WARN_ERROR` and `WARN_ERROR_OPTIONS` are mutually exclusive. You can only specify one, even when you're specifying the config in multiple places (like env var or a flag), otherwise, you'll see a usage error. Warnings that should be treated as errors can be specified through the `error` parameter. Warning names can be found in: * [The types.py file in `dbt-labs/dbt`](https://github.com/dbt-labs/dbt/blob/1.latest/core/dbt/events/types.py), where each class name that inherits from `WarnLevel` corresponds to a warning name (e.g. `AdapterDeprecationWarning`, `NoNodesForSelectionCriteria`). * Using the `--log-format json` flag. (Applies to dbt v1.12 and later) Starting in v1.12, dbt v1 ignores [dbt v2-specific names](https://github.com/dbt-labs/dbt/blob/1.12.latest/core/dbt/events/fusion_warn_error_options.py) in `warn_error_options` (for example, `StaticAnalysis` and `PackageParsingCompatibility`) instead of raising an error, and emits a note: `<name> is not being used because it's specific to the dbt Fusion engine.` This lets you share `warn_error_options` configs across dbt v1 and dbt v2. Genuine typos still raise an error. (Applies to dbt v2.0 and later) In dbt v2, every warning has both a numeric code (for example, `1092`) and an event name (for example, `NoNodesForSelectionCriteria`). Runtime messages show both, but `warn_error_options` only accepts the *name*. Use the event name, v2-native name, or a supported group (`all`, `*`). Numeric codes aren't accepted and will cause an error. To find the name for a code you see in your logs, check out [Supported legacy dbt-Core event name aliases](#supported-legacy-dbt-core-event-name-aliases). The `error` parameter can be set to `"all"` or `"*"` to treat all warnings as errors (this behavior is the same as using the `--warn-error` flag), or to a list of specific warning names to treat as exceptions. * When `error` is set to `"all"` or `"*"`, the optional `warn` parameter can be set to exclude specific warnings from being treated as exceptions. * Use the `silence` parameter to ignore warnings. To silence certain warnings you want to ignore, you can specify them in the `silence` parameter. This is useful in large projects where certain warnings aren't critical and can be ignored to keep the noise low and logs clean. Here's how you can use the [`--warn-error-options`](#use---warn-error-options-for-targeted-warnings) flag to promote *specific* warnings to errors: * [Test warnings](https://docs.getdbt.com/reference/resource-configs/severity.md) with the `--warn-error-options '{"error": ["LogTestResult"]}'` flag. * Jinja [exception warnings](https://docs.getdbt.com/reference/dbt-jinja-functions/exceptions.md#warn) with `--warn-error-options '{"error": ["JinjaLogWarning"]}'`. * No nodes selected with `--warn-error-options '{"error": ["NoNodesForSelectionCriteria"]}'`. * Deprecation warnings with `--warn-error-options '{"error": ["Deprecations"]}'` (new in v1.10). ##### Configuration You can configure warnings as errors or which warnings to silence, by warn error options through command flag, environment variable, or `dbt_project.yml`. You can choose to: * Promote all warnings to errors using `{"error": "all"}` or `--warn-error` flag. * Promote specific warnings to errors using `error` and optionally exclude others from being treated as errors with `--warn-error-options` flag. `warn` tells dbt to continue treating the warnings as warnings. * Ignore warnings using `silence` with `--warn-error-options` flag. In the following example, we're silencing the [`NoNodesForSelectionCriteria` warning](https://github.com/dbt-labs/dbt/blob/main/core/dbt/events/types.py#L1227) in the `dbt_project.yml` file by adding it to the `silence` parameter: dbt\_project.yml ```yaml ... flags: warn_error_options: error: # Previously called "include" warn: # Previously called "exclude" silence: # To silence or ignore warnings - NoNodesForSelectionCriteria ``` ##### Examples Here are some examples that show you how to configure `warn_error_options` using flags or file-based configuration. ###### Target specific warnings Some of the examples use `NoNodesForSelectionCriteria`, which is a specific warning that occurs when your `--select` flag doesn't match any nodes/resources in your dbt project: * This command promotes all warnings to errors, except for `NoNodesForSelectionCriteria`: ```text dbt run --warn-error-options '{"error": "all", "warn": ["NoNodesForSelectionCriteria"]}' ``` * This command promotes all warnings to errors, except for deprecation warnings: ```text dbt run --warn-error-options '{"error": "all", "warn": ["Deprecations"]}' ``` * This command promotes only `NoNodesForSelectionCriteria` as an error: ```text dbt run --warn-error-options '{"error": ["NoNodesForSelectionCriteria"]}' ``` * This promotes only `NoNodesForSelectionCriteria` as an error, using an environment variable: (Applies to dbt v1.11 and later) ```text DBT_ENGINE_WARN_ERROR_OPTIONS='{"error": ["NoNodesForSelectionCriteria"]}' dbt run ``` Values for `error`, `warn`, and/or `silence` should be passed on as arrays. For example, `dbt run --warn-error-options '{"error": "all", "warn": ["NoNodesForSelectionCriteria"]}'` not `dbt run --warn-error-options '{"error": "all", "warn": "NoNodesForSelectionCriteria"}'`. The following example shows how to promote all warnings to errors, except for the `NoNodesForSelectionCriteria` warning using the `silence` and `warn` parameters in the `dbt_project.yml` file: dbt\_project.yml ```yaml ... flags: warn_error_options: error: all # Previously called "include" warn: # Previously called "exclude" - NoNodesForSelectionCriteria silence: # To silence or ignore warnings - NoNodesForSelectionCriteria ``` ###### Promote all warnings to errors Some examples of how to promote all warnings to errors: ###### using dbt command flags ```bash dbt run --warn-error dbt run --warn-error-options '{"error": "all"}' dbt run --warn-error-options '{"error": "*"}' ``` ###### using environment variables (Applies to dbt v1.11 and later) ```bash WARN_ERROR=true dbt run DBT_ENGINE_WARN_ERROR_OPTIONS='{"error": "all"}' dbt run DBT_ENGINE_WARN_ERROR_OPTIONS='{"error": "*"}' dbt run ``` caution Note, using `warn_error_options: error: "all"` will treat all current and future warnings as errors. This means that if a new warning is introduced in a future version of dbt, your production job may start failing unexpectedly. We recommend proceeding with caution when doing this in production environments, and explicitly listing only the warnings you want to treat as errors in production. (Applies to dbt v2.0 and later) #### dbt v2 behavior and warning codes dbt v2 fully supports `warn_error_options`. This section describes important differences from dbt v1 behavior. Existing dbt-core event names fall into three categories: * **Supported** Mapped to similar dbt v2 warning and behave approximately the same. * **Won't be supported:** Those that we deliberately decided to not ever support. * **Not supported yet:** Parsed, but do nothing yet. ##### Warning codes in dbt v2 In dbt v2, every warning has both a numeric code (for example, `1092`) and an event name (for example, `NoNodesForSelectionCriteria`). Warning messages at runtime show both, but `warn_error_options` only accepts the *name*, never the code: ```yaml flags: warn_error_options: error: - NothingToDo # by name silence: - FreshnessConfigProblem # by name ``` Any value that isn't a supported legacy event name, v2-native name, or supported group (`all`, `*`) causes dbt v2 to exit with an error at startup, including numeric codes. For example, `{error: [1092]}` fails, but `{error: [NoNodesForSelectionCriteria]}` works. Not every valid name appears in the tables on this page. dbt v2 also emits its own warnings (for example, `SemanticModelDeprecated`, code `dbt1157`) that aren't listed here. Use the name shown in the runtime message. ##### Supported legacy dbt-core event name aliases When you see a warning code in your logs, use the following table to find the matching event name to put in `warn_error_options`. The code column is only for looking up warnings you see at runtime — you can't use the code itself in your config: | dbt v2 code (runtime only) | dbt-core event name (use this in config) | Description | | -------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------- | | 1601 | `NoNodesSelected` | No nodes selected | | 1601 | `NothingToDo` | No nodes selected (alias) | | 1087 | `NodeNotFoundOrDisabled` | A test or exposure dependency references a missing or disabled node | | 1085 | `DeprecatedModel` | A model has passed its deprecation date and should be removed | | 1072 | `DeprecatedReference` | A reference to a model that has already been deprecated | | 1073 | `UpcomingReferenceDeprecation` | A reference to a model that will be deprecated on a future date | | 1074 | `JinjaLogWarning` | Jinja `exceptions.warn()` called in a macro | | 1075 | `SnapshotTimestampWarning` | Snapshot timestamp column type mismatch | | 1076 | `PackageRedirectDeprecation` | A package has been redirected to a new name; update your `packages.yml` | | 1077 | `DepsUnpinned` | A git-sourced package uses an unpinned revision (`HEAD`, `main`, or `master`) | | 1078 | `FreshnessConfigProblem` | A source has no freshness configuration; freshness check was skipped | | 1084 | `WarnStateTargetEqual` | The `--state` and `--target` directories are the same path | | 1086 | `WEOIncludeExcludeDeprecation` | Deprecated `include`/`exclude` keys were used in `warn_error_options`; use `error`/`warn` instead | | 1089 | `NoNodeForYamlKey` | A YAML key references a node that doesn't exist in the project | | 1090 | `MacroNotFoundForPatch` | A `patches:` entry in a YAML file references a macro that doesn't exist | | 1091 | `InvalidConcurrentBatchesConfig` | `concurrent_batches` is configured but not supported for this model | | 1092 | `NoNodesForSelectionCriteria` | `--select` criteria matched no nodes | | 1093 | `MicrobatchModelNoEventTimeInputs` | A microbatch model has no upstream inputs with `event_time` configured | | 1094 | `UnversionedBreakingChange` | A breaking change was made to an unversioned model | | 1095 | `UnsupportedConstraintMaterialization` | A constraint was defined on a materialization that doesn't support it | | 1097 | `UnusedResourceConfigPath` | A `+config` path in `dbt_project.yml` doesn't match any resources | | 1098 | `DepsScrubbedPackageName` | A package name contained characters that were scrubbed during install | | 1099 | `DepsFoundDuplicatePackage` | The same package was found more than once in `packages.yml` | | 1506 | `InvalidMacroAnnotation` | A macro's YAML annotation (argument name or type) doesn't match its Jinja definition | | *no code* | `LogTestResult` | A data test result (pass/warn/fail) | | *no code* | `RunResultWarning` | A model or test run completed with `warn` status | | *no code* | `RunResultWarningMessage` | The message accompanying a `warn`-status run result | ##### Unsupported dbt v1 event names Only the legacy names in [Supported legacy dbt-Core event name aliases](#supported-legacy-dbt-core-event-name-aliases) are valid string aliases in dbt v2. There are many other dbt v1 warning event names; if you put one of those in `warn_error_options`, dbt v2 will throw a warning at startup. The table below is not a complete list of unsupported names. It only includes dbt v1 event names that dbt v2 recognizes by name so it can emit a startup warning explaining why the entry has no effect and prompting you to remove it: the underlying dbt v1 behavior was removed, replaced, or made unconditional in dbt v2. Many other unsupported dbt v1 names are not listed here; they still warn during startup validation when used in `warn_error_options`. | dbt-core event name | Message | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | | `MicrobatchMacroOutsideOfBatchesDeprecation` | dbt v2 only supports the newer behavior-change flag, where this case is a hard error. | | `SeedExceedsLimitSamePath` | This warning comes from partial parsing in dbt v1, which dbt v2 does not support. | | `SeedIncreased` | This warning comes from partial parsing in dbt v1, which dbt v2 does not support. | | `GenerateSchemaNameNullValueDeprecation` | dbt v2 only supports the newer behavior-change flag, where this case is a hard error. | | `GenericSemanticLayerDeprecation` | dbt v2 already implements the new semantic layer spec, so this legacy warning no longer applies. | | `MFCumulativeTypeParamsDeprecation` | dbt v2 already implements the new semantic layer spec, so this legacy warning no longer applies. | | `MFTimespineWithoutYamlConfigurationDeprecation` | dbt v2 already implements the new semantic layer spec, so this legacy warning no longer applies. | | `MetricAttributesRenamed` | dbt v2 already implements the new semantic layer spec, so this legacy warning no longer applies. | | `TimeDimensionsRequireGranularityDeprecation` | dbt v2 already implements the new semantic layer spec, so this legacy warning no longer applies. | | `SourceFreshnessProjectHooksNotRun` | dbt v2 already uses the newer source freshness behavior, so this legacy warning does not apply. | | `SemanticValidationFailure` | dbt v2 does not support semantic models, so this warning does not apply. | | `ValidationWarning` | dbt v2 already validates allowed YAML keys strictly, so this warning would be redundant. | | `PackageMaterializationOverrideDeprecation` | dbt v2 already enforces the latest behavior, which prevents packages from overriding built-in materializations. | | `TestsConfigDeprecation` | dbt v2 does not surface this warning by default, which matches current dbt v1 behavior. | | `ProjectFlagsMovedDeprecation` | dbt v2 already errors on this configuration, which matches newer dbt v1 behavior. | | `ConfigSourcePathDeprecation` | This is now fully deprecated in dbt v2. | | `ConfigLogPathDeprecation` | This is now fully deprecated in dbt v2. | | `ConfigTargetPathDeprecation` | This is now fully deprecated in dbt v2. | | `ConfigDataPathDeprecation` | This is now fully deprecated in dbt v2. | | `EnvironmentVariableNamespaceDeprecation` | dbt v2 reserves the `DBT_ENGINE_` prefix and rejects unknown environment variables that use it. | | `UnusedTables` | dbt v2 does not allow source overrides, so packages must disable a source explicitly instead. | | `WrongResourceSchemaFile` | dbt v2 reports this case under `NoNodeForYamlKey` instead. | | `PackageNodeDependsOnRootProjectNode` | dbt v2 only supports the newer behavior-change flag `require_ref_searches_node_package_before_root`, where this case is a hard error. | ##### Warnings that are hard errors in dbt v2 Some dbt v1 warning names correspond to behaviors that dbt v2 enforces unconditionally as parse errors. If you reference these names in `warn_error_options`, dbt v2 emits a startup warning explaining that the entry has no effect. You can carry over your `warn_error_options` config from dbt v1 without breaking, but these configs do nothing (They will throw a warning as `unsupported` and should be removed from the config): | dbt-Core event name | dbt v2 behavior | dbt v2 error code | | ------------------------------------------- | -------------------------------------------------------------------------- | --------------------------- | | `DuplicateYAMLKeysDeprecation` | dbt v2's YAML parser rejects duplicate keys as hard parse errors | `DuplicateConfigKey` (1059) | | `CustomKeyInConfigDeprecation` | Unknown config keys are rejected via strict schema validation | `UnusedConfigKey` (1060) | | `CustomTopLevelKeyDeprecation` | Unknown top-level schema keys are hard parse errors | `UnusedConfigKey` (1060) | | `ResourceNamesWithSpacesDeprecation` | Resource names with spaces are rejected during name validation | `SchemaError` | | `InvalidValueForField` | Field value failures are surfaced as hard parse errors via deserialization | `SerializationError` | | `GenericJSONSchemaValidationDeprecation` | JSON schema validation failures are hard parse errors | `SerializationError` | | `DuplicateNameDistinctNodeTypesDeprecation` | Caught as a hard error during node resolution | `SchemaError` | ##### Enabling `--warn-error` with static analysis in `baseline` mode If your project emits static analysis warnings and you use `--warn-error` (which promotes all warnings to errors), your project may fail unexpectedly. We recommend explicitly listing the warning categories you want to enforce rather than using `error: all` when `baseline` mode is active. ##### Deprecated `include` and `exclude` keys The legacy `include` and `exclude` fields for `warn_error_options` were deprecated in dbt v1.8 but are still supported in dbt v2. If you use them, dbt v2 emits a `WEOIncludeExcludeDeprecation` warning (code 1086) and ignores the deprecated keys. Migrate to `error`, `warn`, and `silence` instead: ```yaml # Before (dbt ≤1.7) flags: warn_error_options: include: all exclude: - NoNodesForSelectionCriteria # After (dbt ≥1.8 and v2) flags: warn_error_options: error: all warn: - NoNodesForSelectionCriteria ``` --- ### where ##### Definition Filter the resource being tested (model, source, seed, or snapshot). The `where` condition is templated into the test query by replacing the resource reference with a subquery. For instance, a `not_null` test may look like: ```sql select * from my_model where my_column is null ``` If the `where` config is set to `where date_column = current_date`, then the test query will be updated to: ```sql select * from (select * from my_model where date_column = current_date) dbt_subquery where my_column is null ``` ##### Examples ##### Specific test Configure a specific instance of a generic (schema) test: models/\<filename>.yml ```yaml models: - name: large_table columns: - name: my_column data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ["a", "b", "c"] config: where: "date_column = current_date" - name: other_column data_tests: - not_null: config: where: "date_column < current_date" ``` ##### One-off test This config is ignored for one-off tests. ##### Generic test block Set the default for all instances of a generic (schema) test, by setting the config inside its test block (definition): macros/\<filename>.sql ```sql {% test <testname>(model, column_name) %} {{ config(where = "date_column = current_date") }} select ... {% endtest %} ``` ##### Project level Set the default for all tests in a package or project: dbt\_project.yml ```yaml data_tests: +where: "date_column = current_date" <package_name>: +where: > date_column = current_date and another_column is not null ``` ##### Custom logic The rendering context for the `where` config is the same as for all configurations defined in `.yml` files. You have access to `{{ var() }}` and `{{ env_var() }}`, but you **do not** have access to custom macros for setting this config. If you do want to use custom macros to template out the `where` filter for certain tests, there is a workaround. dbt defines a `get_where_subquery` macro. dbt replaces `{{ model }}` in generic test definitions with `{{ get_where_subquery(relation) }}`, where `relation` is a `ref()` or `source()` for the resource being tested. The default implementation of this macro returns: * `{{ relation }}` when the `where` config is not defined (`ref()` or `source()`) * `(select * from {{ relation }} where {{ where }}) dbt_subquery` when the `where` config is defined You can override this behavior by: * Defining a custom `get_where_subquery` in your root project * Defining a custom `<adapter>__get_where_subquery` [dispatch candidate](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) in your package or adapter plugin Within this macro definition, you can reference whatever custom macros you want, based on static inputs from the configuration. At simplest, this enables you to DRY up code that you'd otherwise need to repeat across many different `.yml` files. Because the `get_where_subquery` macro is resolved at runtime, your custom macros can also include [fetching the results of introspective database queries](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md). ###### Example Filter your test to the past N days of data, using dbt's cross-platform [`dateadd()`](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md#dateadd) utility macro. You can set the number of days in the placeholder string. models/config.yml ```yml models: - name: my_model columns: - name: id data_tests: - unique: config: where: "date_column > __3_days_ago__" # placeholder string for static config ``` macros/custom\_get\_where\_subquery.sql ```sql {% macro get_where_subquery(relation) -%} {% set where = config.get('where') %} {% if where %} {% if "_days_ago__" in where %} {# replace placeholder string with result of custom macro #} {% set where = replace_days_ago(where) %} {% endif %} {%- set filtered -%} (select * from {{ relation }} where {{ where }}) dbt_subquery {%- endset -%} {% do return(filtered) %} {%- else -%} {% do return(relation) %} {%- endif -%} {%- endmacro %} {% macro replace_days_ago(where_string) %} {# Use regex to search the pattern for the number days #} {# Default to 3 days when no number found #} {% set re = modules.re %} {% set days = 3 %} {% set pattern = '__(\d+)_days_ago__' %} {% set match = re.search(pattern, where_string) %} {% if match %} {% set days = match.group(1) | int %} {% endif %} {% set n_days_ago = dbt.dateadd('day', -days, current_timestamp()) %} {% set result = re.sub(pattern, n_days_ago, where_string) %} {{ return(result) }} {% endmacro %} ``` --- ### YAML Selectors (Applies to dbt v1.12 and later) Write resource selectors in YAML, save them with a human-friendly name, and reference them using the `selector:` method with `--select`. By recording selectors in a top-level `selectors.yml` file: * **Legibility:** complex selection criteria are composed of dictionaries and arrays * **Version control:** selector definitions are stored in the same git repository as the dbt project * **Reusability:** selectors can be referenced in multiple job definitions, and their definitions are extensible (via YAML anchors) Selectors live in a top-level file named `selectors.yml`. Each must have a `name` and a `definition`, and can optionally define a `description` and [`default` flag](#default). selectors.yml ```yml selectors: - name: nodes_to_joy definition: ... - name: nodes_to_a_grecian_urn description: Attic shape with a fair attitude default: true definition: ... ``` (Applies to dbt v1.12 and later) Starting in dbt v1.12, you can reference a named selector from `--select` or `--exclude` using the [`selector` method](https://docs.getdbt.com/reference/node-selection/methods.md#selector) (for example, `selector:my_selector`). #### Definitions Each `definition` is comprised of one or more arguments, which can be one of the following: * **CLI-style:** strings, representing CLI-style arguments * **Key-value:** pairs in the form `method: value` * **Full YAML:** fully specified dictionaries with items for `method`, `value`, operator-equivalent keywords, and support for `exclude` Use the `union` and `intersection` operator-equivalent keywords to organize multiple arguments. ##### CLI-style ```yml definition: 'tag:nightly' ``` This simple syntax supports use of the `+`, `@`, and `*` [graph](https://docs.getdbt.com/reference/node-selection/graph-operators.md) operators, [set](https://docs.getdbt.com/reference/node-selection/set-operators.md) operators, and `exclude`. ##### Key-value ```yml definition: tag: nightly ``` This simple syntax does not support any [graph](https://docs.getdbt.com/reference/node-selection/graph-operators.md) or [set](https://docs.getdbt.com/reference/node-selection/set-operators.md) operators or `exclude`. ##### Full YAML This is the most thorough syntax, which can include the operator-equivalent keywords for [graph](https://docs.getdbt.com/reference/node-selection/graph-operators.md) and [set](https://docs.getdbt.com/reference/node-selection/set-operators.md) operators. Review [methods](https://docs.getdbt.com/reference/node-selection/methods.md) for the available list. ```yml definition: method: tag value: nightly # Optional keywords map to the `+` and `@` graph operators: children: true | false parents: true | false children_depth: 1 # if children: true, degrees to include parents_depth: 1 # if parents: true, degrees to include childrens_parents: true | false # @ operator indirect_selection: eager | cautious | buildable | empty # include all tests selected indirectly? eager by default ``` The `*` operator to select all nodes can be written as: ```yml definition: method: fqn value: "*" ``` ###### Exclude The `exclude` keyword is only supported by fully-qualified dictionaries. It may be passed as an argument to each dictionary, or as an item in a `union`. The following are equivalent: ```yml - method: tag value: nightly exclude: - "@tag:daily" ``` ```yml - union: - method: tag value: nightly - exclude: - method: tag value: daily ``` Note: The `exclude` argument in YAML selectors is subtly different from the `--exclude` CLI argument. Here, `exclude` *always* returns a [set difference](https://en.wikipedia.org/wiki/Complement_\(set_theory\)), and it is always applied *last* within its scope. When more than one "yeslist" (`--select`) is passed, they are treated as a [union](https://docs.getdbt.com/reference/node-selection/set-operators.md#unions) rather than an [intersection](https://docs.getdbt.com/reference/node-selection/set-operators.md#intersections). Same thing when there is more than one "nolist" (`--exclude`). ###### Indirect selection As a general rule, dbt will indirectly select *all* tests if they touch *any* resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious", "buildable", or "empty" by setting `indirect_selection` for a specific criterion: ```yml - union: - method: fqn value: model_a indirect_selection: eager # default: will include all tests that touch model_a - method: fqn value: model_b indirect_selection: cautious # will not include tests touching model_b # if they have other unselected parents - method: fqn value: model_c indirect_selection: buildable # will not include tests touching model_c # if they have other unselected parents (unless they have an ancestor that is selected) - method: fqn value: model_d indirect_selection: empty # will include tests for only the selected node and ignore all tests attached to model_d ``` If provided, a YAML selector's `indirect_selection` value will take precedence over the CLI flag `--indirect-selection`. Because `indirect_selection` is defined separately for *each* selection criterion, it's possible to mix eager/cautious/buildable/empty modes within the same definition, to achieve the exact behavior that you need. (Applies to dbt v1.12 and later) You can always test out your criteria with `dbt ls --select selector:SELECTOR_NAME`. See [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) for more details about indirect selection. #### Example Here are two ways to represent: ```bash $ dbt run --select @source:snowplow,tag:nightly models/export --exclude package:snowplow,config.materialized:incremental export_performance_timing ``` ##### CLI-style selectors.yml ```yml selectors: - name: nightly_diet_snowplow description: "Non-incremental Snowplow models that power nightly exports" definition: # Optional `union` and `intersection` keywords map to the ` ` and `,` set operators: union: - intersection: - '@source:snowplow' - 'tag:nightly' - 'models/export' - exclude: - intersection: - 'package:snowplow' - 'config.materialized:incremental' - export_performance_timing ``` ##### Full YML selectors.yml ```yml selectors: - name: nightly_diet_snowplow description: "Non-incremental Snowplow models that power nightly exports" definition: # Optional `union` and `intersection` keywords map to the ` ` and `,` set operators: union: - intersection: - method: source value: snowplow childrens_parents: true - method: tag value: nightly - method: path value: models/export - exclude: - intersection: - method: package value: snowplow - method: config.materialized value: incremental - method: fqn value: export_performance_timing ``` Then in our job definition: (Applies to dbt v1.12 and later) ```bash dbt run --select selector:nightly_diet_snowplow ``` #### Default Selectors may define a boolean `default` property. If a selector has `default: true`, dbt will use this selector's criteria when tasks do not define their own selection criteria. Let's say we define a default selector that only selects resources defined in our root project: ```yml selectors: - name: root_project_only description: > Only resources from the root project. Excludes resources defined in installed packages. default: true definition: method: package value: <my_root_project_name> ``` If I run an "unqualified" command, dbt will use the selection criteria defined in `root_project_only`—that is, dbt will only build / freshness check / generate compiled SQL for resources defined in my root project. ```text dbt build dbt source freshness dbt docs generate ``` (Applies to dbt v1.12 and later) If I run a command that defines its own selection criteria (through `--select` or `--exclude`), dbt will ignore the default selector and use the flag criteria instead. It will not try to combine the two. If you want to apply both, use the `selector:` method with `--select` to reference the default selector alongside other criteria. For example: ```text dbt build --select selector:root_project_only tag:nightly ``` For more information, refer to [Using `selector:` with `--select`](#using-selector-with---select). Only one selector may set `default: true` for a given invocation; otherwise, dbt will return an error. You may use a Jinja expression to adjust the value of `default` depending on the environment, however: ```yml selectors: - name: default_for_dev default: "{{ target.name == 'dev' | as_bool }}" definition: ... - name: default_for_prod default: "{{ target.name == 'prod' | as_bool }}" definition: ... ``` ##### Selector inheritance Selectors can reuse and extend definitions from other selectors, via the `selector` method. ```yml selectors: - name: foo_and_bar definition: intersection: - tag: foo - tag: bar - name: foo_bar_less_buzz definition: intersection: # reuse the definition from above - method: selector value: foo_and_bar # with a modification! - exclude: - method: tag value: buzz ``` **Note:** While selector inheritance allows the logic from another selector to be *reused*, it doesn't allow the logic from that selector to be *modified* by means of `parents`, `children`, `indirect_selection`, and so on. The `selector` method returns the complete set of nodes returned by the named selector. (Applies to dbt v1.12 and later) If selector definitions reference each other in a cycle, dbt raises `DbtRecursionError` at runtime. For example, the following selector inheritance is invalid: ```yaml selectors: - name: selector_a definition: method: selector value: selector_b - name: selector_b definition: method: selector value: selector_a ``` (Applies to dbt v1.12 and later) #### Using `selector:` with `--select` Starting in dbt v1.12, dbt raises `SelectExcludeIgnoredWithSelectorWarning` when the legacy `--selector` flag is combined with `--select` or `--exclude`. Use the [`selector:` method](https://docs.getdbt.com/reference/node-selection/methods.md#selector) directly with `--select` to reference a predefined selector alongside other selection criteria. For example: ```text dbt run --select selector:nightly_diet_snowplow tag:nightly ``` --- ### Yellowbrick configurations #### Incremental materialization strategies The dbt-yellowbrick adapter supports the following incremental materialization strategies: * `append` (default when `unique_key` is not defined) * `delete+insert` (default when `unique_key` is defined) All of these strategies are inherited from the dbt-postgres adapter. #### Performance optimizations To improve query performance, tables in Yellowbrick Data support several optimizations that can be defined as model-level configurations in dbt. These will be applied to `CREATE TABLE` DDL statements generated at compile or run time. Note that these settings will have no effect on models set to `view` or `ephemeral`. dbt-yellowbrick supports the following Yellowbrick-specific features when defining tables: * `dist` - applies a single-column distribution key, or sets the distribution to `RANDOM` or `REPLICATE` * `sort_col` - applies the `SORT ON (column)` clause that names a single column to sort on before data is stored on media * `cluster_cols` - applies the `CLUSTER ON (column, column, ...)` clause that names up to four columns to cluster on before data is stored on the media A table that has sorted or clustered columns facilitates the skipping of blocks when tables are scanned with restrictions applied in the query. Further details can be found in the [Yellowbrick Data Warehouse](https://docs.yellowbrick.com/latest/ybd_sqlref/clustered_tables.html#clustered-tables) documentation. ##### Some example model configurations * `DISTRIBUTE REPLICATE` with a `SORT` column... ```sql {{ config( materialized = "table", dist = "replicate", sort_col = "stadium_capacity" ) }} select hash(stg.name) as team_key , stg.name as team_name , stg.nickname as team_nickname , stg.city as home_city , stg.stadium as stadium_name , stg.capacity as stadium_capacity , stg.avg_att as average_game_attendance , current_timestamp as md_create_timestamp from {{ source('premdb_public','team') }} stg where stg.name is not null ``` gives the following model output: ```sql create table if not exists marts.dim_team as ( select hash(stg.name) as team_key , stg.name as team_name , stg.nickname as team_nickname , stg.city as home_city , stg.stadium as stadium_name , stg.capacity as stadium_capacity , stg.avg_att as average_game_attendance , current_timestamp as md_create_timestamp from premdb.public.team stg where stg.name is not null ) distribute REPLICATE sort on (stadium_capacity); ``` <br /> * `DISTRIBUTE` on a single column and define up to four `CLUSTER` columns... ```sql {{ config( materialized = 'table', dist = 'match_key', cluster_cols = ['season_key', 'match_date_key', 'home_team_key', 'away_team_key'] ) }} select hash(concat_ws('||', lower(trim(s.season_name)), translate(left(m.match_ts,10), '-', ''), lower(trim(h."name")), lower(trim(a."name")))) as match_key , hash(lower(trim(s.season_name))) as season_key , cast(translate(left(m.match_ts,10), '-', '') as integer) as match_date_key , hash(lower(trim(h."name"))) as home_team_key , hash(lower(trim(a."name"))) as away_team_key , m.htscore , split_part(m.htscore, '-', 1) as home_team_goals_half_time , split_part(m.htscore , '-', 2) as away_team_goals_half_time , m.ftscore , split_part(m.ftscore, '-', 1) as home_team_goals_full_time , split_part(m.ftscore, '-', 2) as away_team_goals_full_time from {{ source('premdb_public','match') }} m inner join {{ source('premdb_public','team') }} h on (m.htid = h.htid) inner join {{ source('premdb_public','team') }} a on (m.atid = a.atid) inner join {{ source('premdb_public','season') }} s on (m.seasonid = s.seasonid) ``` gives the following model output: ```sql create table if not exists marts.fact_match as ( select hash(concat_ws('||', lower(trim(s.season_name)), translate(left(m.match_ts,10), '-', ''), lower(trim(h."name")), lower(trim(a."name")))) as match_key , hash(lower(trim(s.season_name))) as season_key , cast(translate(left(m.match_ts,10), '-', '') as integer) as match_date_key , hash(lower(trim(h."name"))) as home_team_key , hash(lower(trim(a."name"))) as away_team_key , m.htscore , split_part(m.htscore, '-', 1) as home_team_goals_half_time , split_part(m.htscore , '-', 2) as away_team_goals_half_time , m.ftscore , split_part(m.ftscore, '-', 1) as home_team_goals_full_time , split_part(m.ftscore, '-', 2) as away_team_goals_full_time from premdb.public.match m inner join premdb.public.team h on (m.htid = h.htid) inner join premdb.public.team a on (m.atid = a.atid) inner join premdb.public.season s on (m.seasonid = s.seasonid) ) distribute on (match_key) cluster on (season_key, match_date_key, home_team_key, away_team_key); ``` #### Cross-database materializations Yellowbrick supports cross-database queries and the dbt-yellowbrick adapter will permit cross-database reads into a specific target on the same appliance instance. #### Limitations This initial implementation of the dbt adapter for Yellowbrick Data Warehouse may not support some use cases. We strongly advise validating all records or transformations resulting from the adapter output. --- ## Search ### Search the documentation [Skip to main content](#__docusaurus_skipToContent_fallback) [Relive the best of dbt Summit! Get access to the keynotes and breakout sessions from dbt Summit 2026. See the latest product innovations from dbt, and hear how data teams are leveling up for the agentic era.](https://www.getdbt.com/dbt-summit/registration/online) [![dbt Logo](/img/dbt-logo.svg?v=2)![dbt Logo](/img/dbt-logo-light.svg?v=2)](https://docs.getdbt.com/index.md) * [v2]() * [v1]() * [1.13 (beta)]() * [1.12]() * [1.11]() * On dbt platform? <br /> v2 = [dbt v2 release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks?version=2.0#dbt-v2-release-tracks) <br /> v1 = [dbt v1 release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks?version=2.0#dbt-v1-release-tracks) [Docs](#) * [Product docs](https://docs.getdbt.com/docs/introduction.md) * [References](https://docs.getdbt.com/reference/references-overview.md) * [Best practices](https://docs.getdbt.com/best-practices.md) * [Developer blog](https://docs.getdbt.com/blog) [Guides](https://docs.getdbt.com/guides.md)[APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) [Help](#) * [Release notes](https://docs.getdbt.com/docs/dbt-versions/release-notes.md) * [FAQs](https://docs.getdbt.com/docs/faqs.md) * [Support and billing](https://docs.getdbt.com/docs/dbt-support.md) * [Licensing](https://docs.getdbt.com/docs/dbt-licensing.md) * [Courses](https://learn.getdbt.com) [Community](#) * [Join the dbt Community](https://docs.getdbt.com/community/join.md) * [Become a contributor](https://docs.getdbt.com/community/contribute.md) * [Community forum](https://discourse.getdbt.com/) * [Webinars](https://www.getdbt.com/resources/webinars) * [Events](https://www.getdbt.com/events) [Account](#) * [Log in to dbt](https://login.dbt.com/) * [Create a free account](https://www.getdbt.com/signup) [Get started with dbt](https://docs.getdbt.com/docs/dbt/get-started-dbt.md?version=2) Search Type your search here [](https://www.algolia.com/) Get started #### Start building with dbt. The free dbt VS Code extension is the best way to develop locally with dbt. [Install free extension](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt) [Request your demo](https://www.getdbt.com/contact) [![dbt Labs](/img/dbt-logo-light.svg?v=2)](https://docs.getdbt.com/index.md) ##### Resources [VS Code Extension](https://docs.getdbt.com/docs/about-dbt-extension.md) [Resource Hub](https://www.getdbt.com/resources) [dbt Learn](https://www.getdbt.com/dbt-learn) [Certification](https://www.getdbt.com/dbt-certification) [Developer Blog](https://docs.getdbt.com/blog) ##### Community [Join the Community](https://docs.getdbt.com/community/join.md) [Become a Contributor](https://docs.getdbt.com/community/contribute.md) [Open Source dbt Packages](https://hub.getdbt.com/) [Community Forum](https://discourse.getdbt.com/) ##### Support [Contact Support](https://docs.getdbt.com/docs/dbt-support.md) [Professional Services](https://www.getdbt.com/services) [Find a Partner](https://www.getdbt.com/partner-directory) [System Status](https://status.getdbt.com/) ##### Connect with Us [](https://github.com/dbt-labs/docs.getdbt.com "GitHub") [](https://www.linkedin.com/company/dbtlabs/mycompany/ "LinkedIn") [](https://www.youtube.com/channel/UCVpBwKK-ecMEV75y1dYLE5w "YouTube") [](https://www.instagram.com/dbt_labs/ "Instagram") [](https://x.com/dbt_labs "X") [](https://bsky.app/profile/getdbt.com "Bluesky") [](https://www.getdbt.com/community/join-the-community/ "Community Slack") © 2026 dbt Labs, LLC. All Rights Reserved. [Terms of Service](https://www.getdbt.com/terms-of-use/) [Privacy Policy](https://www.getdbt.com/cloud/privacy-policy/) [Security](https://www.getdbt.com/security/) Cookie Settings --- ## Set up dbt ### About environments In software engineering, environments are used to enable engineers to develop and test code without impacting the users of their software. Typically, there are two types of environments in dbt: * **Deployment or Production** (or *prod*) — Refers to the environment that end users interact with. * **Development** (or *dev*) — Refers to the environment that engineers work in. This means that engineers can work iteratively when writing and testing new code in *development*. Once they are confident in these changes, they can deploy their code to *production*. In traditional software engineering, different environments often use completely separate architecture. For example, the dev and prod versions of a website may use different servers and databases. Data warehouses can also be designed to have separate environments — the *production* environment refers to the relations (for example, schemas, tables, and views) that your end users query (often through a BI tool). Configure environments to tell dbt or dbt v1 how to build and execute your project in development and production: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-platform-environments.md) ###### [Environments in dbt platform](https://docs.getdbt.com/docs/dbt-platform-environments.md) [Seamlessly configure development and deployment environments in dbt to control how your project runs in both the Studio IDE, dbt platform CLI, and dbt jobs.](https://docs.getdbt.com/docs/dbt-platform-environments.md) [![](/img/icons/command-line.svg)](https://docs.getdbt.com/docs/local/dbt-environments.md) ###### [Environments in self-hosted dbt](https://docs.getdbt.com/docs/local/dbt-environments.md) [Setup and maintain separate deployment and development environments through the use of targets within a profile file](https://docs.getdbt.com/docs/local/dbt-environments.md) <br /> #### Related docs * [dbt environment best practices](https://docs.getdbt.com/guides/set-up-ci.md) * [Deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md) * [About dbt versions](https://docs.getdbt.com/docs/dbt-versions.md) * [Set Environment variables in dbt](https://docs.getdbt.com/docs/build/environment-variables.md#special-environment-variables) * [Use Environment variables in jinja](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) --- ### dbt setup and installation dbt compiles and runs your analytics code against your data platform, enabling you and your team to collaborate on a single source of truth for metrics, insights, and business definitions. There are two options for deploying dbt: * **dbt platform** (formerly dbt Cloud) runs dbt v2 or dbt v1 in a hosted (single or multi-tenant) environment with a browser-based interface. The intuitive user interface aids you in setting up the various components. dbt comes equipped with turnkey support for scheduling jobs, CI/CD, hosting documentation, monitoring, and alerting. It also offers an integrated development environment (Studio IDE) and allows you to develop and run dbt commands from your local command line (CLI) or code editor. * **Self-hosted dbt** is a free command line tool that you can install in a local environment, and communication with databases is facilitated through adapters. If you're not sure which is the right solution for you, read our [What is dbt?](https://docs.getdbt.com/docs/introduction.md) and our [dbt features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) articles to help you decide. If you still have questions, don't hesitate to [contact us](https://www.getdbt.com/contact/). To begin configuring dbt now, select the option that is right for you. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/about-platform-setup.md) ###### [dbt platform setup](https://docs.getdbt.com/docs/platform/about-platform-setup.md) [Learn how to connect to a data platform, integrate with secure authentication methods, and configure a sync with a git repo.](https://docs.getdbt.com/docs/platform/about-platform-setup.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/local/install-dbt.md) ###### [Self-hosted dbt setup](https://docs.getdbt.com/docs/local/install-dbt.md) [Learn how to set up self-hosted dbt using the dbt VS Code extension or CLI.](https://docs.getdbt.com/docs/local/install-dbt.md) Pair your setup with AI Whichever option you choose, you can pair it with dbt Wizard to build models faster. [Get started with dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). --- ### Run your dbt projects You can run your dbt projects locally or using the [dbt platform](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) with the dbt framework. #### Common commands In dbt, the commands you commonly use are: * [dbt run](https://docs.getdbt.com/reference/commands/run.md) — Run the models you defined in your project * [dbt build](https://docs.getdbt.com/reference/commands/build.md) — Build and test your selected resources such as models, seeds, snapshots, and tests * [dbt test](https://docs.getdbt.com/reference/commands/test.md) — Execute the tests you defined for your project For all dbt commands and their arguments (flags), see the [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md). To list all dbt commands from the command line, run `dbt --help`. To list a specific command's arguments, run `dbt COMMAND_NAME --help`.  New to the command line? If you're new to the command line: 1. Open your computer's terminal application (such as Terminal or iTerm) to access the command line. 2. Make sure you navigate to your dbt project directory before running any dbt commands. 3. These terminal commands help you navigate your file system: `cd` (change directory), `ls` (list directory contents), and `pwd` (present working directory). #### Where to run dbt Use the dbt framework to quickly and collaboratively transform data and deploy analytics code following software engineering best practices like version control, modularity, portability, CI/CD, and documentation. This means anyone on the data team familiar with SQL can safely contribute to production-grade data pipelines. The dbt framework is composed of a *language* and an *engine*: * The *dbt language* is the code you write in your dbt project — SQL select statements, Jinja templating, YAML configs, tests, and more. It's the standard for the data industry and the foundation of the dbt framework. * The *dbt engine* compiles your project, executes your transformation graph, and produces metadata. Today, the current Rust-based version generation is v2. By default, installing dbt gives you SQL comprehension, editor features, and richer development workflows. ##### dbt platform The dbt platform is a fully managed service that gives you a complete environment to build, test, deploy, and collaborate on dbt projects. You can develop in the browser or with a self-hosted dbt installation. * [Develop in your browser using the Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) * [Seamless drag-and-drop development with Canvas](https://docs.getdbt.com/docs/platform/canvas.md) * [Run dbt commands from your local command line](#dbt-local-development) using dbt VS Code extension or dbt platform CLI (both which integrate seamlessly with the dbt platform project(s)). For more details, see [About dbt plans](https://www.getdbt.com/pricing). ##### Self-hosted dbt development You can run dbt locally with dbt v2 or the dbt v1 engine: * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) — Get dbt v2 from the command line * [Install the dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) — Combines dbt v2 performance with visual features like autocomplete, inline errors, and lineage. Includes [LSP features](https://docs.getdbt.com/docs/about-dbt-lsp.md) and suitable for users with dbt platform projects or running dbt locally without a dbt platform project. *Recommended for local development.* * [Install the dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) — The dbt platform CLI, which allows you to run dbt commands against your dbt platform development environment from your local command line. Requires a dbt platform project. #### Related docs * [About the dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [dbt features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) * [Model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) * [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) * [Studio IDE features](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features) * [Does dbt offer extract and load functionality?](https://docs.getdbt.com/faqs/Project/transformation-tool.md) * [Why does dbt compile need a data platform connection](https://docs.getdbt.com/faqs/Warehouse/db-connection-dbt-compile.md) --- ### Using threads When dbt runs, it creates a directed acyclic graph (DAG) of links between models. The number of threads represents the maximum number of paths through the graph dbt may work on at once – increasing the number of threads can minimize the run time of your project. For example, if you specify `threads: 1`, dbt will start building only one model, and finish it, before moving on to the next. Specifying `threads: 4` means that dbt will work on *up to* 4 models at once without violating dependencies; the actual number of models it can work on will likely be constrained by the available paths through the dependency graph. Here's the difference between running a project with 1 thread and 4 threads. With 1 thread, dbt builds one model at a time in dependency order. With 4 threads, dbt builds as many ready models in parallel as the graph allows, finishing much sooner. [![Increase threads to parallelize builds and finish faster. The colors have different meanings: gray means waiting, orange means building, and green means done.](/img/docs/running-a-dbt-project/threads-1-vs-4-animated.gif?v=2 "Increase threads to parallelize builds and finish faster. The colors have different meanings: gray means waiting, orange means building, and green means done.")](#)Increase threads to parallelize builds and finish faster. The colors have different meanings: gray means waiting, orange means building, and green means done. There's no set limit of the maximum number of threads you can set – while increasing the number of threads generally decreases execution time, there are a number of things to consider: * Increasing the number of threads increases the load on your warehouse, which may impact other tools in your data stack. For example, if your BI tool uses the same compute resources as dbt, their queries may get queued during a dbt run. * The number of concurrent queries your database will allow you to run may be a limiting factor in how many models can be actively built – some models may queue while waiting for an available query slot. Generally the optimal number of threads depends on your data warehouse and its configuration. It’s best to test different values to find the best number of threads for your project. (Applies to dbt v2.0 and later) In dbt v2, start with the maximum and lower it if needed. For the recommended values, check out the [dbt v2 thread optimization](#dbt-v2-thread-optimization) section. You can use a different number of threads than the value defined in your target by using the `--threads` option when executing a dbt command. You will define the number of threads in your `profiles.yml` file (when developing locally with dbt v1 and dbt v2), dbt job definition, and dbt development credentials under your profile. #### dbt v2 thread optimization In the context of dbt v2, a thread is an open connection to your data warehouse, not the number of parallel threads on your local machine's CPU. Data platforms vary in how many concurrent connections they allow; exceeding those limits causes the platform to reject new connections. In dbt v2, `threads` sets the maximum number of SQL queries dbt runs on your warehouse at the same time. To let dbt v2 run as many queries at once as your project allows: * **When running in dbt platform**: Set `threads` to `256` in your [job settings](https://docs.getdbt.com/docs/deploy/deploy-jobs.md?version=2#create-and-schedule-jobs). * **When running locally**: Set `threads: 0` (or pass `--threads 0`). If your warehouse rejects connections or you hit rate limits, lower `threads` to reduce concurrent load. Project parsing runs separately and automatically uses all available CPUs. To disable parallel parsing and run one operation at a time, use the `--no-parallel` flag. This is useful for debugging parse errors and does not affect `threads`. ##### Adapter-specific behavior | Adapter | Behavior | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Snowflake** | dbt v2 automatically manages connection parallelism based on platform limits and backpressure. The `threads` setting acts as a maximum connection cap if set, but dbt v2 is designed to work optimally without it configured. If you're experiencing timeout or rate limit issues, setting `threads` to a lower value can help. | | **Databricks** | dbt v2 automatically manages connection parallelism based on platform limits and backpressure. The `threads` setting acts as a maximum connection cap if set, but dbt v2 is designed to work optimally without it configured. If you're experiencing timeout or rate limit issues, setting `threads` to a lower value can help. | | **BigQuery** | dbt v2 respects user-set threads to manage API rate limits.<br />Setting `--threads 0` (or omitting the setting) allows dbt v2 to dynamically optimize parallelism. | | **Redshift** | dbt v2 respects user-set threads to manage concurrency limits.<br />Setting `--threads 0` (or omitting the setting) allows dbt v2 to dynamically optimize parallelism. | For more information about dbt v2's approach to parallelism, refer to [the dbt v2](https://docs.getdbt.com/docs/introduction.md) page. #### Related docs * [About profiles.yml](https://docs.getdbt.com/docs/local/profiles.yml.md) * [dbt job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) --- ### dbt local installation #### dbt environments Local development dbt makes it easy to maintain separate production and development environments through the use of [targets](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) within a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md). A typical profile, when using dbt locally (for example, running from your command line), will have a target named `dev` and have this set as the default. This means that while making changes, your objects will be built in your *development* target without affecting production queries made by your end users. Once you are confident in your changes, you can deploy the code to *production*, by running your dbt project with a *prod* target. Running dbt in production You can learn more about different ways to run dbt in production in [this article](https://docs.getdbt.com/docs/deploy/deployments.md). Targets offer the flexibility to decide how to implement your separate environments – whether you want to use separate schemas, databases, or entirely different clusters altogether! We recommend using *different schemas within one database* to separate your environments. This is the easiest to set up and is the most cost-effective solution in a modern cloud-based data stack. In practice, this means that most of the details in a target will be consistent across all targets, except for the `schema` and user credentials. If you have multiple dbt users writing code, it often makes sense for *each user* to have their own *development* environment. A pattern we've found useful is to set your dev target schema to be `dbt_<username>`. User credentials should also differ across targets so that each dbt user is using their own data warehouse user. --- #### Install dbt Local development(Applies to dbt v2.0 and later) Get dbt running on your machine in a few minutes. Installing dbt gives you dbt v2 by default: the current, free-to-use experience for v2. Choose your preferred installation method: #### Install dbt ##### pip ```shell python -m pip install dbt ``` To upgrade later, run `python -m pip install --upgrade dbt`. ##### Homebrew (macOS) ```shell brew tap dbt-labs/dbt brew install dbt-labs/dbt/dbt ``` To upgrade later, run `brew upgrade dbt`. ##### curl (macOS/Linux) ```shell curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh -s -- --update ``` This installs the dbt binary to `~/.local/bin/dbt` and adds that directory to your `$PATH`. See [uninstalling a curl install](#faqs) if you ever need to remove it. Close and reopen your terminal (or run `exec $SHELL`) so the new `$PATH` is recognized. To upgrade later, run `dbt system update`. ##### winget (Windows) ```shell winget install --id dbtLabs.dbt --exact ``` To install a specific version, run `winget install --id dbtLabs.dbt --exact --version <version>`. ##### Windows (PowerShell) ```powershell irm https://public.cdn.getdbt.com/fs/install/install.ps1 | iex ``` Close and reopen your shell (or run `Start-Process powershell`) so the new `Path` is recognized. To upgrade later, run `dbt system update`. * Verify your installation: ```shell dbt --version ``` * With dbt v2, you can start using it right away. For the best v2 editor experience, install the dbt VS Code extension to use features like autocomplete, inline errors, and lineage. For full LSP features and other dbt platform-based capabilities, run `dbt login` to sign in with a free dbt platform account: ```shell dbt login ``` Refer to the [dbt VS Code extension docs](https://docs.getdbt.com/docs/about-dbt-extension.md) for more info. If you or your org has a strict requirement to use the open-source runtime, refer to [Install dbt OSS](https://docs.getdbt.com/docs/local/install-dbt-v2.md). Note that `dbt-oss` and `dbt` cannot share a Python environment. #### Troubleshooting Common issues and resolutions: * **dbt command not found:** Add the installation location to your `$PATH`. * **Version conflicts:** Check that no other self-hosted dbt or dbt platform CLI versions are installed or active on your machine. * **`dbt-oss` conflict:** `dbt-oss` and `dbt` cannot share a Python environment. If you need to run both, create an alias as described in [Run dbt-oss alongside other dbt installs](https://docs.getdbt.com/docs/local/install-dbt-v2.md#run-dbt-oss-alongside-other-dbt-installs). * **Installation permissions:** Make sure your user account can install software locally. #### FAQs *  Can I revert to my previous dbt installation? Yes. To test a new install without affecting your existing workflows, use a separate environment or virtual machine. *  Can I download the Apache 2.0 runtime only? Yes if you need to use the Apache 2.0 runtime, you can [install open source dbt v2](https://docs.getdbt.com/docs/local/install-dbt-v2.md), the open-source project behind dbt v2. *  How do I uninstall a curl (install.sh) install? These steps apply only if you installed dbt v2 with the curl (`install.sh`) script. If you used pip, Homebrew, or winget, remove dbt with that tool instead (for example, `pip uninstall dbt` or `brew uninstall dbt`). 1. **Uninstall dbt.** Run the built-in uninstall command to clear cached files and state. This also removes the binary for you: ```shell dbt system uninstall ``` 2. **Clean up your shell profile.** The installer adds a `$PATH` export and a `dbtf` alias to `~/.zshrc` or `~/.bashrc`, each under its own comment. Open that file and delete these lines: ```shell # Added by dbt installer export PATH="$PATH:$HOME/.local/bin" # dbt aliases alias dbtf=$HOME/.local/bin/dbt ``` Then reload your profile: ```shell source ~/.zshrc # or source ~/.bashrc ``` #### More information about dbt v2 * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-dbt.md) * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [dbt v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) #### Next steps * Configure [environment variables](https://docs.getdbt.com/docs/local/configure-environment-variables.md) to manage credentials. * Configure your [profiles.yml](https://docs.getdbt.com/docs/local/profiles.yml.md#location-of-profilesyml) file. * Configure your [data platform connection](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md). * Create your first [dbt project](https://docs.getdbt.com/docs/build/projects.md) using the [`dbt init`](https://docs.getdbt.com/reference/commands/init.md) command. --- #### Connect data platform ##### About data platform connections Local development dbt connects to your data platform to run SQL transformations against your data. (Applies to dbt v2.0 and later) #### Supported dbt v2 data platforms dbt v2 includes built-in support for: * [Snowflake](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md) * [Databricks](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) * [Amazon Redshift](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) * [Google BigQuery](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md) * [DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md) * [Apache Spark](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [ClickHouse](https://docs.getdbt.com/docs/local/connect-data-platform/clickhouse-setup.md) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") *Adapter lifecycle can differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* <br /> <br /> dbt v2 uses [ADBC (Arrow Database Connectivity)](https://arrow.apache.org/adbc/) drivers for high-performance connections to these platforms. No separate adapter installation is required. #### Connection profiles When you run dbt locally, it reads your `dbt_project.yml` file to find the profile name, then looks for a profile with the same name in your `profiles.yml` file. This profile contains the connection details for your data platform. For detailed configuration options, refer to [Connection profiles](https://docs.getdbt.com/docs/local/profiles.yml.md). #### Next steps For step-by-step setup instructions with demo project data, see our [Quickstart guides](https://docs.getdbt.com/guides.md). --- ##### Connect AlloyDB to dbt v1 Local development * **Maintained by**: dbt Labs * **Authors**: dbt Labs * **GitHub repo**: [dbt-labs/dbt-adapters](https://github.com/dbt-labs/dbt-adapters) [![](https://img.shields.io/github/stars/dbt-labs/dbt-adapters?style=for-the-badge)](https://github.com/dbt-labs/dbt-adapters) * **PyPI package**: `dbt-postgres` [![](https://badge.fury.io/py/dbt-postgres.svg)](https://badge.fury.io/py/dbt-postgres) * **Slack channel**: [#db-postgres](https://getdbt.slack.com/archives/C0172G2E273) * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: ? #### Installing dbt-postgres Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-postgres` #### Configuring dbt-postgres For AlloyDB-specific configuration, please refer to [AlloyDB configs.](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md) #### Profile Configuration AlloyDB targets are configured exactly the same as [Postgres targets](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md#profile-configuration). --- ##### Connect Apache Spark to dbt v2 Beta (Applies to dbt v2.0 and later) ### Connect Apache Spark to dbt v2 [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development dbt v2 supports Apache Spark, enabling faster compilation and execution for your Spark-based dbt projects. Currently, dbt v2 only supports Apache Spark 3.0. #### dbt v2 and Spark dbt v2 uses the Databricks SQL dialect for [static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md#principles-of-static-analysis) when working with Spark. Databricks SQL is a superset of Spark SQL, so your SQL is validated with Databricks semantics. This provides comprehensive error checking and SQL comprehension features. A dedicated Spark SQL dialect for static analysis is planned for a future release. #### Authentication The Spark adapter in dbt v2 supports: * Thrift * Simple Authentication and Security Layer (SASL) PLAIN * No SASL (NOSASL) * Livy * Basic authentication (username and password) * When deployed on Amazon Web Services (AWS): AWS Signature Version 4 * Supports authentication using single sign-on, service accounts, or user tokens #### Configure dbt v2 Configure your Spark connection in `profiles.yml`: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: spark method: CONNECTION_METHOD # thrift, http, or livy port: PORT_NUMBER auth: AUTH_METHOD schema: SCHEMA_NAME host: HOSTNAME platform_hint: PLATFORM_HINT # optional; aws_emr_serverless or aws_emr_eks server_side_parameters: # optional; required keys depend on platform_hint livy.server.session.ttl: SESSION_TTL_SECONDS # optional when using method: livy ``` | Profile field | Required | Description | Example | | ------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | | `method` | Yes | Connection method. Accepted values: `thrift`, `http`, or `livy`. | `thrift` | | `port` | Yes | Port number for the connection. | `443` | | `auth` | Yes | Authentication method. | `SASL PLAIN`, `NOSASL`, `AWS_SIGV4` | | `schema` | Yes | The database or schema name where dbt will create and query objects. | `analytics` | | `host` | Yes | Hostname of the Spark cluster or Databricks workspace. | `yourorg.sparkhost.com` | | `platform_hint` | No | Hints to dbt v2 which Spark platform you use. dbt v2 uses this to validate required `server_side_parameters`. Accepted values: `aws_emr_serverless`, `aws_emr_eks`. If omitted, dbt v2 assumes a generic Spark cluster. | `aws_emr_serverless` | | `server_side_parameters` | No | Spark session parameters passed to the cluster.<br /><br />Required keys when using `platform_hint`:<br />- For `aws_emr_serverless`, use `emr-serverless.session.executionRoleArn`.<br />- For `aws_emr_eks`, use `spark.kubernetes.namespace`.<br /><br />When using `method: livy`, you can set `livy.server.session.ttl` to configure how long a session can remain idle before it is terminated. | Refer to [example profiles](#example-profiles). | ##### Example profiles ##### Thrift (binary) \~/.dbt/profiles.yml ```yaml spark-local-thrift-binary: target: spark outputs: spark: type: spark method: thrift port: 10000 auth: NOSASL host: localhost schema: my_schema ``` ##### Thrift (HTTP) \~/.dbt/profiles.yml ```yaml spark-local-thrift-http: target: spark outputs: spark: type: spark method: http port: 443 auth: NOSASL # Omit the protocol scheme to use HTTPS. For HTTP, use a host like http://localhost host: localhost schema: my_schema ``` ##### AWS EMR Serverless \~/.dbt/profiles.yml ```yaml spark-emr-serverless: target: spark outputs: spark: type: spark method: livy auth: AWS_SIGV4 port: 443 host: "YOUR_APPLICATION_ID.livy.emr-serverless-services.us-east-1.amazonaws.com" schema: my_schema platform_hint: aws_emr_serverless server_side_parameters: "spark.driver.memory": "1g" "spark.driver.cores": 1 "spark.executor.memory": "1g" "spark.executor.cores": 1 # Required by EMR Serverless when using platform_hint: aws_emr_serverless "emr-serverless.session.executionRoleArn": "arn:aws:iam::YOUR_AWS_ACCOUNT:role/YOUR_ROLE" ``` ##### AWS EMR on EKS \~/.dbt/profiles.yml ```yaml spark-emr-eks: target: spark outputs: spark: type: spark method: livy auth: AWS_SIGV4 port: 443 host: "https://my-spark-cluster.com" schema: my_schema platform_hint: aws_emr_eks server_side_parameters: # Required by EMR on EKS when using platform_hint: aws_emr_eks "spark.kubernetes.namespace": "emr-jobs" "spark.driver.memory": "1g" "spark.driver.cores": 1 "spark.executor.memory": "8g" "spark.executor.cores": 4 ``` For detailed configuration options, refer to the [Spark configuration](https://docs.getdbt.com/reference/resource-configs/spark-configs.md) page. --- ##### Connect Athena to dbt v1 Local development * **Maintained by**: dbt Labs * **Authors**: dbt Labs * **GitHub repo**: [dbt-labs/dbt-adapters](https://github.com/dbt-labs/dbt-adapters) [![](https://img.shields.io/github/stars/dbt-labs/dbt-adapters?style=for-the-badge)](https://github.com/dbt-labs/dbt-adapters) * **PyPI package**: `dbt-athena` [![](https://badge.fury.io/py/dbt-athena.svg)](https://badge.fury.io/py/dbt-athena) * **Slack channel**: [#db-athena](https://getdbt.slack.com/archives/C013MLFR7BQ) * **Supported dbt version**: v1.3.0 and newer * **dbt support**: Supported * **Minimum data platform version**: engine version 2 and 3 #### Installing dbt-athena Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-athena` #### Configuring dbt-athena For Athena-specific configuration, please refer to [Athena configs.](https://docs.getdbt.com/reference/resource-configs/athena-configs.md) `dbt-athena` vs `dbt-athena-community` `dbt-athena-community` was the community-maintained adapter until dbt Labs took over maintenance in late 2024. Both `dbt-athena` and `dbt-athena-community` are maintained by dbt Labs, but `dbt-athena-community` is now simply a wrapper on `dbt-athena`, published for backwards compatibility. #### Connecting to Athena with dbt-athena This plugin does not accept any credentials directly. Instead, [credentials are determined automatically](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) based on AWS CLI/boto3 conventions and stored login info. You can configure the AWS profile name to use via `aws_profile_name`. Alternatively, you can set explicit credential keys in the profile: `aws_access_key_id`, `aws_secret_access_key`, and, for [AWS STS temporary credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html), `aws_session_token` (must match exactly). \~/.dbt/profiles.yml ```yaml default: outputs: dev: type: athena s3_staging_dir: [s3_staging_dir] s3_data_dir: [s3_data_dir] s3_data_naming: [table_unique] # the type of naming convention used when writing to S3 region_name: [region_name] database: [database name] schema: [dev_schema] aws_profile_name: [optional profile to use from your AWS shared credentials file.] threads: [1 or more] num_retries: [0 or more] # number of retries performed by the adapter. Defaults to 5 target: dev ``` ##### Example Config profiles.yml ```yaml default: outputs: dev: type: athena s3_staging_dir: s3://dbt_demo_bucket/athena-staging/ s3_data_dir: s3://dbt_demo_bucket/dbt-data/ s3_data_naming: schema_table region_name: us-east-1 database: warehouse schema: dev aws_profile_name: demo threads: 4 num_retries: 3 target: dev ``` --- ##### Connect AWS Glue to dbt v1 Local development Community plugin Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below. * **Maintained by**: Community * **Authors**: Benjamin Menuet, Moshir Mikael, Armando Segnini and Amine El Mallem * **GitHub repo**: [aws-samples/dbt-glue](https://github.com/aws-samples/dbt-glue) [![](https://img.shields.io/github/stars/aws-samples/dbt-glue?style=for-the-badge)](https://github.com/aws-samples/dbt-glue) * **PyPI package**: `dbt-glue` [![](https://badge.fury.io/py/dbt-glue.svg)](https://badge.fury.io/py/dbt-glue) * **Slack channel**: [#db-glue](https://getdbt.slack.com/archives/C02R4HSMBAT) * **Supported dbt version**: v0.24.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Glue 2.0 #### Installing dbt-glue Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-glue` #### Configuring dbt-glue For AWS Glue-specific configuration, please refer to [AWS Glue configs.](https://docs.getdbt.com/reference/resource-configs/glue-configs.md) For further (and more likely up-to-date) info, see the [README](https://github.com/aws-samples/dbt-glue#readme) #### Connection Methods ##### Configuring your AWS profile for Glue Interactive Session There are two IAM principals used with interactive sessions. * Client principal: The principal (either user or role) calling the AWS APIs (Glue, Lake Formation, Interactive Sessions) from the local client. This is the principal configured in the AWS CLI and is likely the same. * Service role: The IAM role that AWS Glue uses to execute your session. This is the same as AWS Glue ETL. Read [this documentation](https://docs.aws.amazon.com/glue/latest/dg/glue-is-security.html) to configure these principals. You will find below a least privileged policy to enjoy all features of **`dbt-glue`** adapter. Please to update variables between **`<>`**, here are explanations of these arguments: | Args | Description | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | region | The region where your Glue database is stored | | AWS Account | The AWS account where you run your pipeline | | dbt output database | The database updated by dbt (this is the schema configured in the profiles.yml of your dbt environment) | | dbt source database | All databases used as source | | dbt output bucket | The bucket name where the data will be generated by dbt (the location configured in the profiles.yml of your dbt environment) | | dbt source bucket | The bucket name of source databases (if they are not managed by Lake Formation) | sample\_IAM\_Policy.yml ```yml { "Version": "2012-10-17", "Statement": [ { "Sid": "Read_and_write_databases", "Action": [ "glue:SearchTables", "glue:BatchCreatePartition", "glue:CreatePartitionIndex", "glue:DeleteDatabase", "glue:GetTableVersions", "glue:GetPartitions", "glue:DeleteTableVersion", "glue:UpdateTable", "glue:DeleteTable", "glue:DeletePartitionIndex", "glue:GetTableVersion", "glue:UpdateColumnStatisticsForTable", "glue:CreatePartition", "glue:UpdateDatabase", "glue:CreateTable", "glue:GetTables", "glue:GetDatabases", "glue:GetTable", "glue:GetDatabase", "glue:GetPartition", "glue:UpdateColumnStatisticsForPartition", "glue:CreateDatabase", "glue:BatchDeleteTableVersion", "glue:BatchDeleteTable", "glue:DeletePartition", "glue:GetUserDefinedFunctions", "lakeformation:ListResources", "lakeformation:BatchGrantPermissions", "lakeformation:ListPermissions", "lakeformation:GetDataAccess", "lakeformation:GrantPermissions", "lakeformation:RevokePermissions", "lakeformation:BatchRevokePermissions", "lakeformation:AddLFTagsToResource", "lakeformation:RemoveLFTagsFromResource", "lakeformation:GetResourceLFTags", "lakeformation:ListLFTags", "lakeformation:GetLFTag", ], "Resource": [ "arn:aws:glue:<region>:<AWS Account>:catalog", "arn:aws:glue:<region>:<AWS Account>:table/<dbt output database>/*", "arn:aws:glue:<region>:<AWS Account>:database/<dbt output database>" ], "Effect": "Allow" }, { "Sid": "Read_only_databases", "Action": [ "glue:SearchTables", "glue:GetTableVersions", "glue:GetPartitions", "glue:GetTableVersion", "glue:GetTables", "glue:GetDatabases", "glue:GetTable", "glue:GetDatabase", "glue:GetPartition", "lakeformation:ListResources", "lakeformation:ListPermissions" ], "Resource": [ "arn:aws:glue:<region>:<AWS Account>:table/<dbt source database>/*", "arn:aws:glue:<region>:<AWS Account>:database/<dbt source database>", "arn:aws:glue:<region>:<AWS Account>:database/default", "arn:aws:glue:<region>:<AWS Account>:database/global_temp" ], "Effect": "Allow" }, { "Sid": "Storage_all_buckets", "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<dbt output bucket>", "arn:aws:s3:::<dbt source bucket>" ], "Effect": "Allow" }, { "Sid": "Read_and_write_buckets", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::<dbt output bucket>" ], "Effect": "Allow" }, { "Sid": "Read_only_buckets", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::<dbt source bucket>" ], "Effect": "Allow" } ] } ``` ##### Configuration of the local environment Because **`dbt`** and **`dbt-glue`** adapters are compatible with Python versions 3.9 or higher, check the version of Python: ```bash $ python3 --version ``` Configure a Python virtual environment to isolate package version and code dependencies: ```bash $ sudo yum install git $ python3 -m venv dbt_venv $ source dbt_venv/bin/activate $ python3 -m pip install --upgrade pip ``` Configure the last version of AWS CLI ```bash $ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" $ unzip awscliv2.zip $ sudo ./aws/install ``` Install boto3 package ```bash $ sudo yum install gcc krb5-devel.x86_64 python3-devel.x86_64 -y $ pip3 install —upgrade boto3 ``` Install the package: ```bash $ pip3 install dbt-glue ``` ##### Example config profiles.yml ```yml type: glue query-comment: This is a glue dbt example role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole region: us-east-1 workers: 2 worker_type: G.1X idle_timeout: 10 schema: "dbt_demo" session_provisioning_timeout_in_seconds: 120 location: "s3://dbt_demo_bucket/dbt_demo_data" ``` The table below describes all the options. | Option | Description | Mandatory | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | project\_name | The dbt project name. This must be the same as the one configured in the dbt project. | yes | | type | The driver to use. | yes | | query-comment | A string to inject as a comment in each query that dbt runs. | no | | role\_arn | The ARN of the glue interactive session IAM role. | yes | | region | The AWS Region where you run the data pipeline. | yes | | workers | The number of workers of a defined workerType that are allocated when a job runs. | yes | | worker\_type | The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X. | yes | | schema | The schema used to organize data stored in Amazon S3.Additionally, is the database in AWS Lake Formation that stores metadata tables in the Data Catalog. | yes | | session\_provisioning\_timeout\_in\_seconds | The timeout in seconds for AWS Glue interactive session provisioning. | yes | | location | The Amazon S3 location of your target data. | yes | | query\_timeout\_in\_minutes | The timeout in minutes for a single query. Default is 300 | no | | idle\_timeout | The AWS Glue session idle timeout in minutes. (The session stops after being idle for the specified amount of time) | no | | glue\_version | The version of AWS Glue for this session to use. Currently, the only valid options are 2.0 and 3.0. The default value is 3.0. | no | | security\_configuration | The security configuration to use with this session. | no | | connections | A comma-separated list of connections to use in the session. | no | | conf | Specific configuration used at the startup of the Glue Interactive Session (arg --conf) | no | | extra\_py\_files | Extra python Libs that can be used by the interactive session. | no | | delta\_athena\_prefix | A prefix used to create Athena-compatible tables for Delta tables (if not specified, then no Athena-compatible table will be created) | no | | tags | The map of key-value pairs (tags) belonging to the session. Ex: `KeyName1=Value1,KeyName2=Value2` | no | | seed\_format | By default `parquet`, can be Spark format compatible like `csv` or `json` | no | | seed\_mode | By default `overwrite`, the seed data will be overwritten, you can set it to `append` if you just want to add new data in your dataset | no | | default\_arguments | The map of key-value pairs parameters belonging to the session. More information on [Job parameters used by AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html). Ex: `--enable-continuous-cloudwatch-log=true,--enable-continuous-log-filter=true` | no | | glue\_session\_id | re-use the glue-session to run multiple dbt run commands: set a glue session id you need to use | no | | glue\_session\_reuse | Reuse the glue-session to run multiple dbt run commands: If set to true, the glue session will not be closed for re-use. If set to false, the session will be closed | no | | datalake\_formats | The ACID data lake format that you want to use if you are doing merge, can be `hudi`, `ìceberg` or `delta` | no | #### Configs ##### Configuring tables When materializing a model as `table`, you may include several optional configs that are specific to the dbt-spark plugin, in addition to the standard [model configs](https://docs.getdbt.com/reference/model-configs.md). | Option | Description | Required? | Example | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ------------------------------------------ | | file\_format | The file format to use when creating tables (`parquet`, `csv`, `json`, `text`, `jdbc` or `orc`). | Optional | `parquet` | | partition\_by | Partition the created table by the specified columns. A directory is created for each partition. | Optional | `date_day` | | clustered\_by | Each partition in the created table will be split into a fixed number of buckets by the specified columns. | Optional | `country_code` | | buckets | The number of buckets to create while clustering | Required if `clustered_by` is specified | `8` | | custom\_location | By default, the adapter will store your data in the following path: `location path`/`schema`/`table`. If you don't want to follow that default behaviour, you can use this parameter to set your own custom location on S3 | No | `s3://mycustombucket/mycustompath` | | hudi\_options | When using file\_format `hudi`, gives the ability to overwrite any of the default configuration options. | Optional | `{'hoodie.schema.on.read.enable': 'true'}` | #### Incremental models dbt seeks to offer useful and intuitive modeling abstractions by means of its built-in configurations and materializations. For that reason, the dbt-glue plugin leans heavily on the [`incremental_strategy` config](https://docs.getdbt.com/docs/build/incremental-models.md). This config tells the incremental materialization how to build models in runs beyond their first. It can be set to one of three values: * **`append`** (default): Insert new records without updating or overwriting any existing data. * **`insert_overwrite`**: If `partition_by` is specified, overwrite partitions in the table with new data. If no `partition_by` is specified, overwrite the entire table with new data. * **`merge`** (Apache Hudi and Apache Iceberg only): Match records based on a `unique_key`; update old records, and insert new ones. (If no `unique_key` is specified, all new data is inserted, similar to `append`.) Each of these strategies has its pros and cons, which we'll discuss below. As with any model config, `incremental_strategy` may be specified in `dbt_project.yml` or within a model file's `config()` block. **Notes:** The default strategy is **`insert_overwrite`** ##### The `append` strategy Following the `append` strategy, dbt will perform an `insert into` statement with all new data. The appeal of this strategy is that it is straightforward and functional across all platforms, file types, connection methods, and Apache Spark versions. However, this strategy *cannot* update, overwrite, or delete existing data, so it is likely to insert duplicate records for many data sources. ###### Source code ```sql {{ config( materialized='incremental', incremental_strategy='append', ) }} -- All rows returned by this query will be appended to the existing table select * from {{ ref('events') }} {% if is_incremental() %} where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ``` ###### Run Code ```sql create temporary view spark_incremental__dbt_tmp as select * from analytics.events where event_ts >= (select max(event_ts) from {{ this }}) ; insert into table analytics.spark_incremental select `date_day`, `users` from spark_incremental__dbt_tmp ``` ##### The `insert_overwrite` strategy This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://spark.apache.org/docs/latest/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select *all* of the relevant data for a partition when using this incremental strategy. If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead. ###### Source Code ```sql {{ config( materialized='incremental', partition_by=['date_day'], file_format='parquet' ) }} /* Every partition returned by this query will be overwritten when this model runs */ with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select date_day, count(*) as users from events group by 1 ``` ###### Run Code ```sql create temporary view spark_incremental__dbt_tmp as with new_events as ( select * from analytics.events where date_day >= date_add(current_date, -1) ) select date_day, count(*) as users from events group by 1 ; insert overwrite table analytics.spark_incremental partition (date_day) select `date_day`, `users` from spark_incremental__dbt_tmp ``` Specifying `insert_overwrite` as the incremental strategy is optional since it's the default strategy used when none is specified. ##### The `merge` strategy **Compatibility:** * Hudi : OK * Delta Lake : OK * Iceberg : OK * Lake Formation Governed Tables : On going NB: * For Glue 3: you have to set up a [Glue connectors](https://docs.aws.amazon.com/glue/latest/ug/connectors-chapter.html). * For Glue 4: use the `datalake_formats` option in your profiles.yml When using a connector be sure that your IAM role has these policies: ```text { "Sid": "access_to_connections", "Action": [ "glue:GetConnection", "glue:GetConnections" ], "Resource": [ "arn:aws:glue:<region>:<AWS Account>:catalog", "arn:aws:glue:<region>:<AWS Account>:connection/*" ], "Effect": "Allow" } ``` and that the managed policy `AmazonEC2ContainerRegistryReadOnly` is attached. Be sure that you follow the getting started instructions [here](https://docs.aws.amazon.com/glue/latest/ug/setting-up.html#getting-started-min-privs-connectors). This [blog post](https://aws.amazon.com/blogs/big-data/part-1-integrate-apache-hudi-delta-lake-apache-iceberg-datasets-at-scale-aws-glue-studio-notebook/) also explains how to set up and works with Glue Connectors ###### Hudi **Usage notes:** The `merge` with Hudi incremental strategy requires: * To add `file_format: hudi` in your table configuration * To add a datalake\_formats in your profile : `datalake_formats: hudi` * Alternatively, to add a connection in your profile: `connections: name_of_your_hudi_connector` * To add Kryo serializer in your Interactive Session Config (in your profile): `conf: spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false` dbt will run an [atomic `merge` statement](https://hudi.apache.org/docs/writing_data#spark-datasource-writer) which looks nearly identical to the default merge behavior on Snowflake and BigQuery. If a `unique_key` is specified (recommended), dbt will update old records with values from new records that match the key column. If a `unique_key` is not specified, dbt will forgo match criteria and simply insert all new records (similar to `append` strategy). ###### Profile config example ```yaml test_project: target: dev outputs: dev: type: glue query-comment: my comment role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole region: eu-west-1 glue_version: "4.0" workers: 2 worker_type: G.1X schema: "dbt_test_project" session_provisioning_timeout_in_seconds: 120 location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/" conf: spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false datalake_formats: hudi ``` ###### Source Code example ```sql {{ config( materialized='incremental', incremental_strategy='merge', unique_key='user_id', file_format='hudi', hudi_options={ 'hoodie.datasource.write.precombine.field': 'eventtime', } ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from events group by 1 ``` ###### Delta You can also use Delta Lake to be able to use merge feature on tables. **Usage notes:** The `merge` with Delta incremental strategy requires: * To add `file_format: delta` in your table configuration * To add a datalake\_formats in your profile : `datalake_formats: delta` * Alternatively, to add a connection in your profile: `connections: name_of_your_delta_connector` * To add the following config in your Interactive Session Config (in your profile): `conf: "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog` **Athena:** Athena is not compatible by default with delta tables, but you can configure the adapter to create Athena tables on top of your delta table. To do so, you need to configure the two following options in your profile: * For Delta Lake 2.1.0 supported natively in Glue 4.0: `extra_py_files: "/opt/aws_glue_connectors/selected/datalake/delta-core_2.12-2.1.0.jar"` * For Delta Lake 1.0.0 supported natively in Glue 3.0: `extra_py_files: "/opt/aws_glue_connectors/selected/datalake/delta-core_2.12-1.0.0.jar"` * `delta_athena_prefix: "the_prefix_of_your_choice"` * If your table is partitioned, then the addition of new partition is not automatic, you need to perform an `MSCK REPAIR TABLE your_delta_table` after each new partition adding ###### Profile config example ```yaml test_project: target: dev outputs: dev: type: glue query-comment: my comment role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole region: eu-west-1 glue_version: "4.0" workers: 2 worker_type: G.1X schema: "dbt_test_project" session_provisioning_timeout_in_seconds: 120 location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/" datalake_formats: delta conf: "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" extra_py_files: "/opt/aws_glue_connectors/selected/datalake/delta-core_2.12-2.1.0.jar" delta_athena_prefix: "delta" ``` ###### Source Code example ```sql {{ config( materialized='incremental', incremental_strategy='merge', unique_key='user_id', partition_by=['dt'], file_format='delta' ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen, current_date() as dt from events group by 1 ``` ###### Iceberg **Usage notes:** The `merge` with Iceberg incremental strategy requires: * To attach the AmazonEC2ContainerRegistryReadOnly Managed policy to your execution role : * To add the following policy to your execution role to enable commit locking in a dynamodb table (more info [here](https://iceberg.apache.org/docs/latest/aws/#dynamodb-lock-manager)). Note that the DynamoDB table specified in the resource field of this policy should be the one that is mentioned in your dbt profiles (`--conf spark.sql.catalog.glue_catalog.lock.table=myGlueLockTable`). By default, this table is named `myGlueLockTable` and is created automatically (with On-Demand Pricing) when running a dbt-glue model with Incremental Materialization and Iceberg file format. If you want to name the table differently or to create your own table without letting Glue do it on your behalf, please provide the `iceberg_glue_commit_lock_table` parameter with your table name (eg. `MyDynamoDbTable`) in your dbt profile. ```yaml iceberg_glue_commit_lock_table: "MyDynamoDbTable" ``` * the latest connector for iceberg in AWS marketplace uses Ver 0.14.0 for Glue 3.0, and Ver 1.2.1 for Glue 4.0 where Kryo serialization fails when writing iceberg, use "org.apache.spark.serializer.JavaSerializer" for spark.serializer instead, more info [here](https://github.com/apache/iceberg/pull/546) Make sure you update your conf with `--conf spark.sql.catalog.glue_catalog.lock.table=<YourDynamoDBLockTableName>` and, you change the below iam permission with your correct table name. ```text { "Version": "2012-10-17", "Statement": [ { "Sid": "CommitLockTable", "Effect": "Allow", "Action": [ "dynamodb:CreateTable", "dynamodb:BatchGetItem", "dynamodb:BatchWriteItem", "dynamodb:ConditionCheckItem", "dynamodb:PutItem", "dynamodb:DescribeTable", "dynamodb:DeleteItem", "dynamodb:GetItem", "dynamodb:Scan", "dynamodb:Query", "dynamodb:UpdateItem" ], "Resource": "arn:aws:dynamodb:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/myGlueLockTable" } ] } ``` * To add `file_format: Iceberg` in your table configuration * To add a datalake\_formats in your profile : `datalake_formats: iceberg` * Alternatively, to add connections in your profile: `connections: name_of_your_iceberg_connector` ( * For Athena version 3: * The adapter is compatible with the Iceberg Connector from AWS Marketplace with Glue 3.0 as Fulfillment option and 0.14.0 (Oct 11, 2022) as Software version) * the latest connector for iceberg in AWS marketplace uses Ver 0.14.0 for Glue 3.0, and Ver 1.2.1 for Glue 4.0 where Kryo serialization fails when writing iceberg, use "org.apache.spark.serializer.JavaSerializer" for spark.serializer instead, more info [here](https://github.com/apache/iceberg/pull/546) * For Athena version 2: The adapter is compatible with the Iceberg Connector from AWS Marketplace with Glue 3.0 as Fulfillment option and 0.12.0-2 (Feb 14, 2022) as Software version) * To add the following config in your Interactive Session Config (in your profile): ```--conf --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.warehouse=s3://<your-bucket-name> --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.dynamodb.DynamoDbLockManager --conf spark.sql.catalog.glue_catalog.lock.table=myGlueLockTable --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions ``` * For Glue 3.0, set `spark.sql.catalog.glue_catalog.lock-impl` to `org.apache.iceberg.aws.glue.DynamoLockManager` instead dbt will run an [atomic `merge` statement](https://iceberg.apache.org/docs/latest/spark-writes/) which looks nearly identical to the default merge behavior on Snowflake and BigQuery. You need to provide a `unique_key` to perform merge operation otherwise it will fail. This key is to provide in a Python list format and can contains multiple column name to create a composite unique\_key. ###### Notes * When using a custom\_location in Iceberg, avoid to use final trailing slash. Adding a final trailing slash lead to an un-proper handling of the location, and issues when reading the data from query engines like Trino. The issue should be fixed for Iceberg version > 0.13. Related Github issue can be find [here](https://github.com/apache/iceberg/issues/4582). * Iceberg also supports `insert_overwrite` and `append` strategies. * The `warehouse` conf must be provided, but it's overwritten by the adapter `location` in your profile or `custom_location` in model configuration. * By default, this materialization has `iceberg_expire_snapshots` set to 'True', if you need to have historical auditable changes, set: `iceberg_expire_snapshots='False'`. * Currently, due to some dbt internal, the iceberg catalog used internally when running glue interactive sessions with dbt-glue has a hardcoded name `glue_catalog`. This name is an alias pointing to the AWS Glue Catalog but is specific to each session. If you want to interact with your data in another session without using dbt-glue (from a Glue Studio notebook, for example), you can configure another alias (ie. another name for the Iceberg Catalog). To illustrate this concept, you can set in your configuration file : ```text --conf spark.sql.catalog.RandomCatalogName=org.apache.iceberg.spark.SparkCatalog ``` And then run in an AWS Glue Studio Notebook a session with the following config: ```text --conf spark.sql.catalog.AnotherRandomCatalogName=org.apache.iceberg.spark.SparkCatalog ``` In both cases, the underlying catalog would be the AWS Glue Catalog, unique in your AWS Account and Region, and you would be able to work with the exact same data. Also make sure that if you change the name of the Glue Catalog Alias, you change it in all the other `--conf` where it's used: ```text --conf spark.sql.catalog.RandomCatalogName=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.RandomCatalogName.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog ... --conf spark.sql.catalog.RandomCatalogName.lock-impl=org.apache.iceberg.aws.glue.DynamoLockManager ``` * A full reference to `table_properties` can be found [here](https://iceberg.apache.org/docs/latest/configuration/). * Iceberg Tables are natively supported by Athena. Therefore, you can query tables created and operated with dbt-glue adapter from Athena. * Incremental Materialization with Iceberg file format supports dbt snapshot. You are able to run a dbt snapshot command that queries an Iceberg Table and create a dbt fashioned snapshot of it. ###### Profile config example ```yaml test_project: target: dev outputs: dev: type: glue query-comment: my comment role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole region: eu-west-1 glue_version: "4.0" workers: 2 worker_type: G.1X schema: "dbt_test_project" session_provisioning_timeout_in_seconds: 120 location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/" datalake_formats: iceberg conf: --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.warehouse=s3://aws-dbt-glue-datalake-1234567890-eu-west-1/dbt_test_project --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.dynamodb.DynamoDbLockManager --conf spark.sql.catalog.glue_catalog.lock.table=myGlueLockTable --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions ``` ###### Source Code example ```sql {{ config( materialized='incremental', incremental_strategy='merge', unique_key=['user_id'], file_format='iceberg', iceberg_expire_snapshots='False', partition_by=['status'] table_properties={'write.target-file-size-bytes': '268435456'} ) }} with new_events as ( select * from {{ ref('events') }} {% if is_incremental() %} where date_day >= date_add(current_date, -1) {% endif %} ) select user_id, max(date_day) as last_seen from events group by 1 ``` ###### Iceberg Snapshot source code example (Applies to dbt v1.9 and later) ```sql {% snapshot demosnapshot %} {{ config( strategy='timestamp', schema='jaffle_db', updated_at='dt', file_format='iceberg' ) }} select * from {{ ref('customers') }} {% endsnapshot %} ``` #### Monitoring your Glue Interactive Session Monitoring is an important part of maintaining the reliability, availability, and performance of AWS Glue and your other AWS solutions. AWS provides monitoring tools that you can use to watch AWS Glue, identify the required number of workers required for your Glue Interactive Session, report when something is wrong and take action automatically when appropriate. AWS Glue provides Spark UI, and CloudWatch logs and metrics for monitoring your AWS Glue jobs. More information on: [Monitoring AWS Glue Spark jobs](https://docs.aws.amazon.com/glue/latest/dg/monitor-spark.html) **Usage notes:** Monitoring requires: * To add the following IAM policy to your IAM role: ```text { "Version": "2012-10-17", "Statement": [ { "Sid": "CloudwatchMetrics", "Effect": "Allow", "Action": "cloudwatch:PutMetricData", "Resource": "*", "Condition": { "StringEquals": { "cloudwatch:namespace": "Glue" } } }, { "Sid": "CloudwatchLogs", "Effect": "Allow", "Action": [ "s3:PutObject", "logs:CreateLogStream", "logs:CreateLogGroup", "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:*:*:/aws-glue/*", "arn:aws:s3:::bucket-to-write-sparkui-logs/*" ] } ] } ``` * To add monitoring parameters in your Interactive Session Config (in your profile). More information on [Job parameters used by AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html) ###### Profile config example ```yaml test_project: target: dev outputs: dev: type: glue query-comment: my comment role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole region: eu-west-1 glue_version: "4.0" workers: 2 worker_type: G.1X schema: "dbt_test_project" session_provisioning_timeout_in_seconds: 120 location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/" default_arguments: "--enable-metrics=true, --enable-continuous-cloudwatch-log=true, --enable-continuous-log-filter=true, --enable-spark-ui=true, --spark-event-logs-path=s3://bucket-to-write-sparkui-logs/dbt/" ``` If you want to use the Spark UI, you can launch the Spark history server using a AWS CloudFormation template that hosts the server on an EC2 instance, or launch locally using Docker. More information on [Launching the Spark history server](https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html#monitor-spark-ui-history-local) #### Enabling AWS Glue Auto Scaling Auto Scaling is available since AWS Glue version 3.0 or later. More information on the following AWS blog post: ["Introducing AWS Glue Auto Scaling: Automatically resize serverless computing resources for lower cost with optimized Apache Spark"](https://aws.amazon.com/blogs/big-data/introducing-aws-glue-auto-scaling-automatically-resize-serverless-computing-resources-for-lower-cost-with-optimized-apache-spark/) With Auto Scaling enabled, you will get the following benefits: * AWS Glue automatically adds and removes workers from the cluster depending on the parallelism at each stage or microbatch of the job run. * It removes the need for you to experiment and decide on the number of workers to assign for your AWS Glue Interactive sessions. * Once you choose the maximum number of workers, AWS Glue will choose the right size resources for the workload. * You can see how the size of the cluster changes during the Glue Interactive sessions run by looking at CloudWatch metrics. More information on [Monitoring your Glue Interactive Session](#Monitoring-your-Glue-Interactive-Session). **Usage notes:** AWS Glue Auto Scaling requires: * To set your AWS Glue version 3.0 or later. * To set the maximum number of workers (if Auto Scaling is enabled, the `workers` parameter sets the maximum number of workers) * To set the `--enable-auto-scaling=true` parameter on your Glue Interactive Session Config (in your profile). More information on [Job parameters used by AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html) ###### Profile config example ```yaml test_project: target: dev outputs: dev: type: glue query-comment: my comment role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole region: eu-west-1 glue_version: "3.0" workers: 2 worker_type: G.1X schema: "dbt_test_project" session_provisioning_timeout_in_seconds: 120 location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/" default_arguments: "--enable-auto-scaling=true" ``` #### Access Glue catalog in another AWS account In many cases, you may need to run you dbt jobs to read from another AWS account. Review the following link <https://repost.aws/knowledge-center/glue-tables-cross-accounts> to set up access policies in source and target accounts Add the following `"spark.hadoop.hive.metastore.glue.catalogid=<AWS-ACCOUNT-ID>"` to your conf in the dbt profile, as such, you can have multiple outputs for each of the accounts that you have access to. Note: The access cross-accounts need to be within the same AWS Region ###### Profile config example ```yaml test_project: target: dev outputsAccountB: dev: type: glue query-comment: my comment role_arn: arn:aws:iam::1234567890:role/GlueInteractiveSessionRole region: eu-west-1 glue_version: "3.0" workers: 2 worker_type: G.1X schema: "dbt_test_project" session_provisioning_timeout_in_seconds: 120 location: "s3://aws-dbt-glue-datalake-1234567890-eu-west-1/" conf: "--conf hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory --conf spark.hadoop.hive.metastore.glue.catalogid=<TARGET-AWS-ACCOUNT-ID-B>" ``` #### Persisting model descriptions Relation-level docs persistence is supported. For more information on configuring docs persistence, see [the docs](https://docs.getdbt.com/reference/resource-configs/persist_docs.md). When the `persist_docs` option is configured appropriately, you'll be able to see model descriptions in the `Comment` field of `describe [table] extended` or `show table extended in [database] like '*'`. #### Always `schema`, never `database` Apache Spark uses the terms "schema" and "database" interchangeably. dbt understands `database` to exist at a higher level than `schema`. As such, you should *never* use or set `database` as a node config or in the target profile when running dbt-glue. If you want to control the schema/database in which dbt will materialize models, use the `schema` config and `generate_schema_name` macro *only*. For more information, check the dbt documentation about [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). #### AWS Lakeformation integration The adapter supports AWS Lake Formation tags management enabling you to associate existing tags defined out of dbt-glue to database objects built by dbt-glue (database, table, view, snapshot, incremental models, seeds). * You can enable or disable lf-tags management via config, at model and dbt-project level (disabled by default) * If enabled, lf-tags will be updated on every dbt run. There are table level lf-tags configs and column-level lf-tags configs. * You can specify that you want to drop existing database, table column Lake Formation tags by setting the drop\_existing config field to True (False by default, meaning existing tags are kept) * Please note that if the tag you want to associate with the table does not exist, the dbt-glue execution will throw an error The adapter also supports AWS Lakeformation data cell filtering. * You can enable or disable data-cell filtering via config, at model and dbt-project level (disabled by default) * If enabled, data\_cell\_filters will be updated on every dbt run. * You can specify that you want to drop existing table data-cell filters by setting the drop\_existing config field to True (False by default, meaning existing filters are kept) * You can leverage excluded\_columns\_names **OR** columns config fields to perform Column level security as well. **Please note that you can use one or the other but not both**. * By default, if you don't specify any column or excluded\_columns, dbt-glue does not perform Column level filtering and let the principal access all the columns. The below configuration let the specified principal (lf-data-scientist IAM user) access rows that have a customer\_lifetime\_value > 15 and all the columns specified ('customer\_id', 'first\_order', 'most\_recent\_order', 'number\_of\_orders') ```sql lf_grants={ 'data_cell_filters': { 'enabled': True, 'drop_existing' : True, 'filters': { 'the_name_of_my_filter': { 'row_filter': 'customer_lifetime_value>15', 'principals': ['arn:aws:iam::123456789:user/lf-data-scientist'], 'column_names': ['customer_id', 'first_order', 'most_recent_order', 'number_of_orders'] } }, } } ``` The below configuration let the specified principal (lf-data-scientist IAM user) access rows that have a customer\_lifetime\_value > 15 and all the columns *except* the one specified ('first\_name') ```sql lf_grants={ 'data_cell_filters': { 'enabled': True, 'drop_existing' : True, 'filters': { 'the_name_of_my_filter': { 'row_filter': 'customer_lifetime_value>15', 'principals': ['arn:aws:iam::123456789:user/lf-data-scientist'], 'excluded_column_names': ['first_name'] } }, } } ``` See below some examples of how you can integrate LF Tags management and data cell filtering to your configurations : ###### At model level This way of defining your Lakeformation rules is appropriate if you want to handle the tagging and filtering policy at object level. Remember that it overrides any configuration defined at dbt-project level. ```sql {{ config( materialized='incremental', unique_key="customer_id", incremental_strategy='append', lf_tags_config={ 'enabled': true, 'drop_existing' : False, 'tags_database': { 'name_of_my_db_tag': 'value_of_my_db_tag' }, 'tags_table': { 'name_of_my_table_tag': 'value_of_my_table_tag' }, 'tags_columns': { 'name_of_my_lf_tag': { 'value_of_my_tag': ['customer_id', 'customer_lifetime_value', 'dt'] }}}, lf_grants={ 'data_cell_filters': { 'enabled': True, 'drop_existing' : True, 'filters': { 'the_name_of_my_filter': { 'row_filter': 'customer_lifetime_value>15', 'principals': ['arn:aws:iam::123456789:user/lf-data-scientist'], 'excluded_column_names': ['first_name'] } }, } } ) }} select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order, customer_orders.most_recent_order, customer_orders.number_of_orders, customer_payments.total_amount as customer_lifetime_value, current_date() as dt from customers left join customer_orders using (customer_id) left join customer_payments using (customer_id) ``` ###### At dbt-project level This way you can specify tags and data filtering policy for a particular path in your dbt project (eg. models, seeds, models/model\_group1, etc.) This is especially useful for seeds, for which you can't define configuration in the file directly. ```yml seeds: +lf_tags_config: enabled: true tags_table: name_of_my_table_tag: 'value_of_my_table_tag' tags_database: name_of_my_database_tag: 'value_of_my_database_tag' models: +lf_tags_config: enabled: true drop_existing: True tags_database: name_of_my_database_tag: 'value_of_my_database_tag' tags_table: name_of_my_table_tag: 'value_of_my_table_tag' ``` #### Tests To perform a functional test: 1. Install dev requirements: ```bash $ pip3 install -r dev-requirements.txt ``` 2. Install dev locally ```bash $ python3 setup.py build && python3 setup.py install_lib ``` 3. Export variables ```bash $ export DBT_S3_LOCATION=s3://mybucket/myprefix $ export DBT_ROLE_ARN=arn:aws:iam::1234567890:role/GlueInteractiveSessionRole ``` 4. Run the test ```bash $ python3 -m pytest tests/functional ``` For more information, check the dbt documentation about [testing a new adapter](https://docs.getdbt.com/guides/adapter-creation.md). #### Caveats ##### Supported Functionality Most dbt v1 functionality is supported, but some features are only available with Apache Hudi. Apache Hudi-only features: 1. Incremental model updates by `unique_key` instead of `partition_by` (see [`merge` strategy](https://docs.getdbt.com/reference/resource-configs/glue-configs.md#the-merge-strategy)) Some dbt features, available on the core adapters, are not yet supported on Glue: 1. [Persisting](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) column-level descriptions as database comments 2. [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) --- ##### Connect BigQuery to dbt v2 (Applies to dbt v2.0 and later) Local development You can configure the BigQuery adapter by running `dbt init` in your CLI or manually providing the `profiles.yml` file with the fields configured for your authentication type. The BigQuery adapter for dbt v2 supports the following [authentication methods](#supported-authentication-types): * Service account (JSON file) * gcloud OAuth * Workload Identity Federation (Microsoft Entra) #### Warehouse permissions The Google Cloud identity (service account or user) that dbt v2 uses must have IAM permissions to run jobs, read and write table data, and read metadata dbt v2 uses for introspection and source freshness. ##### Required Google Cloud objects Before connecting, these objects must exist or be accessible: | Object | BigQuery term | Purpose | | --------------------------- | ------------- | --------------------------------------------------- | | **Project** | GCP project | Your Google Cloud project ID | | **Dataset** | Dataset | Target dataset (equivalent to a schema) | | **Service account or user** | IAM identity | Identity for authentication | | **Location or region** | Location | Data location (for example, `US`, `EU`, `us-east1`) | ##### IAM permissions The following permissions are required for fundamental dbt features:: | Permission | Purpose | | ---------------------------- | ------------------------------ | | `bigquery.datasets.get` | Access dataset metadata | | `bigquery.tables.create` | Create tables | | `bigquery.tables.delete` | Drop or replace tables | | `bigquery.tables.get` | Read table metadata | | `bigquery.tables.getData` | Read table data | | `bigquery.tables.list` | List tables in dataset | | `bigquery.tables.update` | Update table schema | | `bigquery.tables.updateData` | Insert, update, or delete rows | | `bigquery.routines.get` | Read routine metadata | | `bigquery.routines.list` | List routines in a dataset | | `bigquery.jobs.create` | Run queries | The following are optional permissions for additional dbt features: | Permission | When required | | -------------------------- | -------------------------------- | | `bigquery.datasets.create` | Auto-create datasets | | `bigquery.routines.create` | Create UDFs or stored procedures | | `bigquery.jobs.get` | Monitor job status | | `storage.objects.create` | Python models (GCS staging) | | `dataproc.*` | Python models on Dataproc | ###### Predefined IAM roles The following roles represent the typical starting point for dbt access: | Role | Description | Use case | | --------------------------- | --------------------------------- | ----------------------- | | `roles/bigquery.dataEditor` | Read and write tables in datasets | Standard dbt operations | | `roles/bigquery.user` | Run jobs, create datasets | Job execution | | `roles/bigquery.jobUser` | Run jobs only | Minimal query execution | For Storage Read API access with dbt v2, also grant **BigQuery Read Session User** (`roles/bigquery.readSessionUser`) on the project, as noted in [Connect BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md#required-permissions). ##### Metadata operations The following are required for fundamental dbt features: | Query type | SQL or API used | Required permission | | ---------------- | --------------------------- | ------------------------------------------------ | | Get table schema | `get_table_schema` API | `bigquery.tables.get` | | List relations | Query against dataset | `bigquery.tables.list`, `bigquery.routines.list` | | Source freshness | Query `__TABLES__` metadata | `bigquery.tables.get` | | Get table stats | Query `INFORMATION_SCHEMA` | `bigquery.tables.get` | | Create dataset | `CREATE SCHEMA` | `bigquery.datasets.create` | ##### INFORMATION\_SCHEMA and metadata views dbt v2 queries these BigQuery system views: | View | Purpose | Scope | | ------------------------------- | ------------------------ | ----------------- | | `INFORMATION_SCHEMA.TABLES` | List tables | Dataset or region | | `INFORMATION_SCHEMA.COLUMNS` | Column metadata | Dataset or region | | `INFORMATION_SCHEMA.VIEWS` | View definitions | Dataset or region | | `INFORMATION_SCHEMA.PARTITIONS` | Partition information | Dataset only | | `__TABLES__` (deprecated) | Table modification times | Dataset | ##### BigQuery DataFrames (optional) For BigQuery DataFrames workflows, users typically need additional roles such as `BigQuery Job User`, `BigQuery Read Session User`, `Notebook Runtime User`, `Code Creator`, and `colabEnterpriseUser`. See your Google Cloud admin for exact role names in your organization. #### Configure dbt v2 Executing `dbt init` in your CLI will prompt for the following fields: * **Project ID:** The GCP BigQuery project ID * **Dataset:** The schema name * **Location:** The location for your GCP environment (for example, us-east1) Alternatively, you can manually create the `profiles.yml` file and configure the fields. See examples in [authentication](#supported-authentication-types) section for formatting. If there is an existing `profiles.yml` file, you have the option to retain the existing fields or overwrite them. Next, select your authentication method. Follow the on-screen prompts to provide the required information. #### Supported authentication types ##### Service account (JSON file) Selecting the **Service account (JSON file)** authentication type will prompt you for the path to your JSON file. You can also manually define the path in your `profiles.yml` file. ###### Example service account JSON file configuration profiles.yml ```yml default: target: dev outputs: dev: type: bigquery threads: 16 database: ABC123 schema: JAFFLE_SHOP method: service-account keyfile: /Users/youruser/Downloads/CustomRoleDefinition.json location: us-east1 dataproc_batch: null ``` ##### gcloud OAuth Prior to selecting this authentication method, you must first configure local OAuth for gcloud: ###### Local OAuth gcloud setup 1. Make sure the `gcloud` command is [installed on your computer](https://cloud.google.com/sdk/downloads). 2. Activate the application-default account with: ```shell gcloud auth application-default login \ --scopes=https://www.googleapis.com/auth/bigquery,\ https://www.googleapis.com/auth/drive.readonly,\ https://www.googleapis.com/auth/iam.test,\ https://www.googleapis.com/auth/cloud-platform # This command uses the `--scopes` flag to request access to Google Sheets. This makes it possible to transform data in Google Sheets using dbt. If your dbt project does not transform data in Google Sheets, then you may omit the `--scopes` flag. ``` A browser window should open, and you should be prompted to log into your Google account. Once you've done that, dbt will use your OAuth'd credentials to connect to BigQuery. ###### Example gcloud configuration profiles.yml ```yml default: target: dev outputs: dev: type: bigquery threads: 16 database: ABC123 schema: JAFFLE_SHOP method: oauth location: us-east1 dataproc_batch: null ``` ##### Workload Identity Federation (Microsoft Entra) Workload Identity Federation (WIF) lets you authenticate to BigQuery using credentials issued by an external OAuth identity provider — currently Microsoft Entra — without managing a Google service account key file. dbt v2 exchanges an Entra-issued token for short-lived Google credentials through a Google Cloud [workload identity pool](https://docs.cloud.google.com/iam/docs/workload-identity-federation#pools). Before selecting this method, an administrator must configure a workload identity pool and provider in GCP that trusts your Entra tenant. You'll need the resulting workload pool provider path. ###### Configuration fields Here are the required and optional fields for the WIF config in your `profiles.yml` file: | Field | Required | Description | | ----------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `method` | Yes | Set to `external-oauth-wif`. | | `workload_pool_provider_path` | Yes | The full resource path of the GCP workload identity pool provider that trusts your Entra tenant. | | `token_endpoint.type` | Yes | The external identity provider type. Only `entra` is currently supported. | | `token_endpoint.request_url` | Yes | The Entra OAuth token endpoint. | | `token_endpoint.request_data` | Yes | The URL-encoded request body used to fetch the Entra token (for example `grant_type`, `client_id`, `client_secret`, and `scope`). | | `service_account_impersonation_url` | No | The `generateAccessToken` URL of a service account to impersonate after federation, if you want BigQuery access scoped to that service account's permissions. | ###### WIF config example in `profiles.yml` The following code snippet is an example of a WIF config in your `profiles.yml`. Make sure to replace the values with your own: profiles.yml ```yaml default: target: dev outputs: dev: type: bigquery threads: 16 database: ABC123 schema: JAFFLE_SHOP method: external-oauth-wif location: us-east1 workload_pool_provider_path: //iam.googleapis.com/projects/123/locations/global/workloadIdentityPools/my-pool/providers/my-provider token_endpoint: type: entra request_url: https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token request_data: "grant_type=client_credentials&client_id={{ env_var('ENTRA_CLIENT_ID') }}&client_secret={{ env_var('ENTRA_CLIENT_SECRET') }}&scope=SCOPE/.default" # Optional: include only if your workload identity pool doesn't have direct resource access and needs to impersonate a service account. # service_account_impersonation_url: https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/sa@project.iam.gserviceaccount.com:generateAccessToken ``` #### More information Find BigQuery-specific configuration information in the [BigQuery adapter reference guide](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md). --- ##### Connect ClickHouse to dbt v2 Beta (Applies to dbt v2.0 and later) ### Connect ClickHouse to dbt v2 [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development The ClickHouse adapter for the dbt v2 connects to [ClickHouse](https://clickhouse.com) over HTTP or HTTPS. It supports self-managed single-node ClickHouse and [ClickHouse Cloud](https://clickhouse.com/cloud). #### Installing dbt The ClickHouse adapter is built into v2. To get started, [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). For connection examples and profile settings, refer to [Connecting to ClickHouse](#connecting-to-clickhouse). #### Authentication dbt v2 authenticates to ClickHouse with a username and password. Set `secure: true` in your profile to connect over HTTPS (default port 8443), or leave it unset to connect over plain HTTP (default port 8123). ClickHouse Cloud requires `secure: true`. #### Warehouse permissions The ClickHouse user that the dbt v2 connects as must be able to run dbt workloads in the target database and read the system tables used for introspection. ##### Required ClickHouse objects Before connecting, these objects must exist or be accessible: | Object | Purpose | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Service (ClickHouse Cloud) or server (self-managed, single node) | Compute resource | | Database | Target database. ClickHouse has no separate schema level, so the dbt `schema` maps to a database | | User | Database user for authentication | ##### Core permissions The following permissions are required for fundamental dbt features: | Permission | Object | Purpose | | -------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SELECT` | Tables and views | Read data | | `INSERT` | Tables | Load models, seeds, and snapshots | | `ALTER` | Tables | Schema changes (`on_schema_change`), indexes and projections, comments (`persist_docs`), `REPLACE PARTITION` (`insert_overwrite`), and lightweight deletes (`delete+insert`) | | `TRUNCATE` | Tables | Full-refresh seeds | | `CREATE TABLE`, `CREATE VIEW`, `CREATE DICTIONARY` | Database | Create materializations, including the intermediate and backup relations used for atomic rebuilds (`EXCHANGE TABLES`, `RENAME TABLE`) | | `DROP TABLE`, `DROP VIEW`, `DROP DICTIONARY` | Database | Drop or replace objects | ##### Metadata operations dbt v2 reads these ClickHouse system tables: | System table | Purpose | | ----------------------- | -------------------------------------------------------------------------------- | | `system.tables` | List relations, build the catalog, detect materialized views pointing at a table | | `system.columns` | Column metadata for the catalog and for schema-change detection | | `system.databases` | Check whether the target database exists | | `system.settings` | Capability probes (lightweight deletes, `insert_distributed_sync`) | | `system.view_refreshes` | Validate refreshable materialized view dependencies | ClickHouse filters system tables to the objects the user can access, so no separate grant is needed for them. ##### Database management Conditional permissions for database management: | Permission | Object | When required | | ----------------- | ------ | --------------------------------------------------------- | | `CREATE DATABASE` | Server | Auto-create the target database when it doesn't exist yet | #### Limitations The ClickHouse adapter for dbt v2 is in beta. Expect some minor bugs, and avoid using it in production environments for now. Some features available in the `dbt-clickhouse` adapter for dbt v1 are not yet supported. The ClickHouse adapter is under active development. If you encounter an issue, please [open it in dbt-core](https://github.com/dbt-labs/dbt-core/issues/new) and add the `adapter:clickhouse` label. ##### What works today On single-node ClickHouse and on ClickHouse Cloud (compatible with multi-node clusters): * All materializations: table, view, incremental (all strategies and `on_schema_change`), materialized view (including refreshable), dictionary, snapshot, seed, and ephemeral * Contracts and constraints, model and query `settings`, projections and indexes * Data tests, unit tests, catalog generation, and the `s3` table function ##### Not yet supported * Self-managed clusters that set `cluster:` in the profile aren't supported yet. `ON CLUSTER` isn't emitted in the data definition language (DDL) instructions, so Replicated engines and the `distributed_table` and `distributed_incremental` materializations don't work. * The `grants` config, and the `dbt clone` and `dbt source freshness` commands, aren't supported yet. * Smaller gaps remain: `query-comment: null` isn't honored, run results don't include the ClickHouse `query_id`, and `persist_docs` descriptions that contain `;` fail. * SQL comprehension features (static analysis and the rest of dbt's SQL intelligence) aren't available yet. Support is coming soon. * Minor issues may still be present in general ClickHouse functionality. #### Connecting to ClickHouse To connect to ClickHouse from dbt, you'll need to add a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) to your `profiles.yml` configuration file. Follow the reference configuration below to set up a ClickHouse profile: profiles.yml ```yaml clickhouse-service: target: dev outputs: dev: type: clickhouse schema: [ default ] # ClickHouse database for dbt models # optional host: [ <your-clickhouse-host> ] # Your clickhouse cluster url for example, abc123.clickhouse.cloud. Defaults to `localhost`. port: [ 8123 ] # Defaults to 8123, 8443, 9000, 9440 depending on the secure and driver settings user: [ default ] # User for all database operations password: [ <empty string> ] # Password for the user secure: [ False ] # Use TLS (native protocol) or HTTPS (http protocol). Must be set to true for ClickHouse Cloud. ``` For a complete list of configuration options, refer to the [ClickHouse documentation](https://clickhouse.com/docs/integrations/dbt). ##### Create a dbt project You can now use this profile in one of your existing projects or create a new one using: ```sh dbt init project_name ``` Navigate to the `project_name` directory and update your `dbt_project.yml` file to use the profile you configured to connect to ClickHouse. ```yaml profile: 'clickhouse-service' ``` ##### Test connection Execute `dbt debug` with the CLI tool to confirm whether dbt is able to connect to ClickHouse. Confirm the response includes `Connection test: [OK connection ok]`, indicating a successful connection. --- ##### Connect Cloudera Hive to dbt v1 Local development * **Maintained by**: Cloudera * **Authors**: Cloudera * **GitHub repo**: [cloudera/dbt-hive](https://github.com/cloudera/dbt-hive) [![](https://img.shields.io/github/stars/cloudera/dbt-hive?style=for-the-badge)](https://github.com/cloudera/dbt-hive) * **PyPI package**: `dbt-hive` [![](https://badge.fury.io/py/dbt-hive.svg)](https://badge.fury.io/py/dbt-hive) * **Slack channel**: [#db-hive](https://getdbt.slack.com/archives/C0401DTNSKW) * **Supported dbt version**: v1.1.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-hive Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-hive` #### Configuring dbt-hive For Hive-specific configuration, please refer to [Hive configs.](https://docs.getdbt.com/reference/resource-configs/hive-configs.md) #### Connection Methods dbt-hive can connect to Apache Hive and Cloudera Data Platform clusters. The [Impyla](https://github.com/cloudera/impyla/) library is used to establish connections to Hive. dbt-hive supports two transport mechanisms: * binary * HTTP(S) The default mechanism is `binary`. To use HTTP transport, use the boolean option. For example, `use_http_transport: true`. #### Authentication Methods dbt-hive supports two authentication mechanisms: * [`insecure`](#Insecure) No authentication is used, only recommended for testing. * [`ldap`](#ldap) Authentication via LDAP ##### Insecure This method is only recommended if you have a local install of Hive and want to test out the dbt-hive adapter. \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: hive host: localhost port: PORT # default value: 10000 schema: SCHEMA_NAME ``` ##### LDAP LDAP allows you to authenticate with a username and password when Hive is [configured with LDAP Auth](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2). LDAP is supported over Binary & HTTP connection mechanisms. This is the recommended authentication mechanism to use with Cloudera Data Platform (CDP). \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: hive host: HOST_NAME http_path: YOUR/HTTP/PATH # optional, http path to Hive default value: None port: PORT # default value: 10000 auth_type: ldap use_http_transport: BOOLEAN # default value: true use_ssl: BOOLEAN # TLS should always be used with LDAP to ensure secure transmission of credentials, default value: true user: USERNAME password: PASSWORD schema: SCHEMA_NAME ``` Note: When creating workload user in CDP, make sure the user has CREATE, SELECT, ALTER, INSERT, UPDATE, DROP, INDEX, READ, and WRITE permissions. If you need the user to execute GRANT statements, you should also configure the appropriate GRANT permissions for them. When using Apache Ranger, permissions for allowing GRANT are typically set using "Delegate Admin" option. For more information, see [`grants`](https://docs.getdbt.com/reference/resource-configs/grants.md) and [on-run-start & on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md). ##### Kerberos The Kerberos authentication mechanism uses GSSAPI to share Kerberos credentials when Hive is [configured with Kerberos Auth](https://ambari.apache.org/1.2.5/installing-hadoop-using-ambari/content/ambari-kerb-2-3-3.html). \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: hive host: HOSTNAME port: PORT # default value: 10000 auth_type: GSSAPI kerberos_service_name: KERBEROS_SERVICE_NAME # default value: None use_http_transport: BOOLEAN # default value: true use_ssl: BOOLEAN # TLS should always be used to ensure secure transmission of credentials, default value: true schema: SCHEMA_NAME ``` Note: A typical setup of Cloudera Private Cloud will involve the following steps to setup Kerberos before one can execute dbt commands: * Get the correct realm config file for your installation (krb5.conf) * Set environment variable to point to the config file (export KRB5\_CONFIG=/path/to/krb5.conf) * Set correct permissions for config file (sudo chmod 644 /path/to/krb5.conf) * Obtain keytab using kinit (kinit \<username@YOUR\_REALM.YOUR\_DOMAIN>) * The keytab is valid for certain period after which you will need to run kinit again to renew validity of the keytab. * User will need CREATE, DROP, INSERT permissions on the schema provided in profiles.yml ##### Instrumentation By default, the adapter will collect instrumentation events to help improve functionality and understand bugs. If you want to specifically switch this off, for instance, in a production environment, you can explicitly set the flag `usage_tracking: false` in your `profiles.yml` file. #### Installation and Distribution dbt's adapter for Cloudera Hive is managed in its own repository, [dbt-hive](https://github.com/cloudera/dbt-hive). To use it, you must install the `dbt-hive` plugin. ##### Using pip The following commands will install the latest version of `dbt-hive` as well as the requisite version of `dbt-core` and `impyla` driver used for connections. ```text python -m pip install dbt-hive ``` ##### Supported Functionality | Name | Supported | | ----------------------------------------------- | --------- | | Materialization: Table | Yes | | Materialization: View | Yes | | Materialization: Incremental - Append | Yes | | Materialization: Incremental - Insert+Overwrite | Yes | | Materialization: Incremental - Merge | No | | Materialization: Ephemeral | No | | Seeds | Yes | | Tests | Yes | | Snapshots | No | | Documentation | Yes | | Authentication: LDAP | Yes | | Authentication: Kerberos | Yes | --- ##### Connect Cloudera Impala to dbt v1 Local development * **Maintained by**: Cloudera * **Authors**: Cloudera * **GitHub repo**: [cloudera/dbt-impala](https://github.com/cloudera/dbt-impala) [![](https://img.shields.io/github/stars/cloudera/dbt-impala?style=for-the-badge)](https://github.com/cloudera/dbt-impala) * **PyPI package**: `dbt-impala` [![](https://badge.fury.io/py/dbt-impala.svg)](https://badge.fury.io/py/dbt-impala) * **Slack channel**: [#db-impala](https://getdbt.slack.com/archives/C01PWAH41A5) * **Supported dbt version**: v1.1.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-impala Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-impala` #### Configuring dbt-impala For Impala-specific configuration, please refer to [Impala configs.](https://docs.getdbt.com/reference/resource-configs/impala-configs.md) #### Connection Methods dbt-impala can connect to Apache Impala and Cloudera Data Platform clusters. The [Impyla](https://github.com/cloudera/impyla/) library is used to establish connections to Impala. Two transport mechanisms are supported: * binary * HTTP(S) The default mechanism is `binary`. To use HTTP transport, use the boolean option `use_http_transport: [true / false]`. #### Authentication Methods dbt-impala supports three authentication mechanisms: * [`insecure`](#Insecure) No authentication is used, only recommended for testing. * [`ldap`](#ldap) Authentication via LDAP * [`kerbros`](#kerbros) Authentication via Kerberos (GSSAPI) ##### Insecure This method is only recommended if you have a local install of Impala and want to test out the dbt-impala adapter. \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: impala host: [host] # default value: localhost port: [port] # default value: 21050 dbname: [db name] # this should be same as schema name provided below, starting with 1.1.2 this parameter is optional schema: [schema name] ``` ##### LDAP LDAP allows you to authenticate with a username & password when Impala is [configured with LDAP Auth](https://impala.apache.org/docs/build/html/topics/impala_ldap.html). LDAP is supported over Binary & HTTP connection mechanisms. This is the recommended authentication mechanism to use with Cloudera Data Platform (CDP). \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: impala host: [host name] http_path: [optional, http path to Impala] port: [port] # default value: 21050 auth_type: ldap use_http_transport: [true / false] # default value: true use_ssl: [true / false] # TLS should always be used with LDAP to ensure secure transmission of credentials, default value: true user: [username] password: [password] dbname: [db name] # this should be same as schema name provided below, starting with 1.1.2 this parameter is optional schema: [schema name] retries: [retries] # number of times Impala attempts retry connection to warehouse, default value: 3 ``` Note: When creating workload user in CDP ensure that the user has CREATE, SELECT, ALTER, INSERT, UPDATE, DROP, INDEX, READ, and WRITE permissions. If the user is required to execute GRANT statements, see for instance (/reference/resource-configs/grants) or (/reference/project-configs/on-run-start-on-run-end) appropriate GRANT permissions should be configured. When using Apache Ranger, permissions for allowing GRANT are typically set using "Delegate Admin" option. ##### Kerberos The Kerberos authentication mechanism uses GSSAPI to share Kerberos credentials when Impala is [configured with Kerberos Auth](https://impala.apache.org/docs/build/html/topics/impala_kerberos.html). \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: impala host: [hostname] port: [port] # default value: 21050 auth_type: [GSSAPI] kerberos_service_name: [kerberos service name] # default value: None use_http_transport: true # default value: true use_ssl: true # TLS should always be used with LDAP to ensure secure transmission of credentials, default value: true dbname: [db name] # this should be same as schema name provided below, starting with 1.1.2 this parameter is optional schema: [schema name] retries: [retries] # number of times Impala attempts retry connection to warehouse, default value: 3 ``` Note: A typical setup of Cloudera EDH will involve the following steps to setup Kerberos before one can execute dbt commands: * Get the correct realm config file for your installation (krb5.conf) * Set environment variable to point to the config file (export KRB5\_CONFIG=/path/to/krb5.conf) * Set correct permissions for config file (sudo chmod 644 /path/to/krb5.conf) * Obtain keytab using kinit (kinit \<username@YOUR\_REALM.YOUR\_DOMAIN>) * The keytab is valid for certain period after which you will need to run kinit again to renew validity of the keytab. ##### Instrumentation By default, the adapter will send instrumentation events to Cloudera to help improve functionality and understand bugs. If you want to specifically switch this off, for instance, in a production environment, you can explicitly set the flag `usage_tracking: false` in your `profiles.yml` file. To turn off dbt Labs anonymous usage tracking, refer to [Anonymous usage stats](https://docs.getdbt.com/reference/global-configs/usage-stats.md) for more details. ##### Supported Functionality | Name | Supported | | ----------------------------------------------- | --------- | | Materialization: Table | Yes | | Materialization: View | Yes | | Materialization: Incremental - Append | Yes | | Materialization: Incremental - Insert+Overwrite | Yes | | Materialization: Incremental - Merge | No | | Materialization: Ephemeral | No | | Seeds | Yes | | Tests | Yes | | Snapshots | Yes | | Documentation | Yes | | Authentication: LDAP | Yes | | Authentication: Kerberos | Yes | --- ##### Connect Confluent Cloud to dbt v1 Local development Community plugin Some features may be limited. To contribute, refer to the source repository below. * **Maintained by**: Confluent * **Authors**: Confluent * **GitHub repo**: [confluentinc/dbt-confluent](https://github.com/confluentinc/dbt-confluent) [![](https://img.shields.io/github/stars/confluentinc/dbt-confluent?style=for-the-badge)](https://github.com/confluentinc/dbt-confluent) * **PyPI package**: `dbt-confluent` [![](https://badge.fury.io/py/dbt-confluent.svg)](https://badge.fury.io/py/dbt-confluent) * **Slack channel**: [n/a](https://docs.getdbt.com/n/a) * **Supported dbt version**: v1.11.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-confluent Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-confluent` #### Configuring dbt-confluent For Confluent Cloud-specific configuration, please refer to [Confluent Cloud configs.](https://docs.getdbt.com/reference/resource-configs/confluent-configs.md) #### Connecting to Confluent Cloud with dbt-confluent Use the `dbt-confluent` adapter to connect to [Confluent Cloud for Apache Flink](https://docs.confluent.io/cloud/current/flink/overview.html), a fully managed stream processing service. The adapter [deploys dbt models as Flink SQL statements](https://docs.confluent.io/cloud/current/flink/operate-and-deploy/deploy-flink-dbt.html) that run continuously on Confluent Cloud. ##### Prerequisites * A [Confluent Cloud](https://confluent.cloud/) account * A [Flink compute pool](https://docs.confluent.io/cloud/current/flink/operate-and-deploy/compute-pools.html) in your environment * A [Flink API key](https://docs.confluent.io/cloud/current/flink/authenticate/authenticate-api-keys.html) for a service account with appropriate [RBAC permissions](https://docs.confluent.io/cloud/current/flink/operate-and-deploy/flink-rbac.html) * Python 3.10 or later ##### Installation Install `dbt-confluent` from PyPI: ```bash pip install dbt-confluent ``` ##### Configuring your profile Add the following configuration to your `profiles.yml` file to define Confluent Cloud targets. \~/.dbt/profiles.yml ```yaml my_confluent_project: target: dev outputs: dev: type: confluent cloud_provider: aws cloud_region: us-east-1 organization_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx compute_pool_id: lfcp-xxxxx environment_id: env-xxxxx dbname: my_kafka_cluster flink_api_key: "{{ env_var('CONFLUENT_FLINK_API_KEY') }}" flink_api_secret: "{{ env_var('CONFLUENT_FLINK_API_SECRET') }}" threads: 1 ``` ###### Description of profile fields | Field | Required | Description | | ----------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- | | `type` | Yes | Must be set to `confluent`. | | `cloud_provider` | Yes | The cloud provider for your Confluent Cloud environment (`aws`, `azure`, or `gcp`). | | `cloud_region` | Yes | The cloud region for your environment (for example, `us-east-1`, `us-west-2`). | | `organization_id` | Yes | Your Confluent Cloud organization ID (UUID format). Find this in the Confluent Cloud Console under **Settings**. | | `compute_pool_id` | Yes | The ID of the Flink compute pool to run statements on (for example, `lfcp-xxxxx`). | | `environment_id` | Yes | The Confluent Cloud environment ID (for example, `env-xxxxx`). Maps to the Flink SQL catalog. | | `dbname` | Yes | The Kafka cluster name within the environment. Maps to the Flink SQL database. | | `flink_api_key` | Yes | A Flink API key for authenticating with Confluent Cloud. Use `env_var` to avoid storing secrets in plain text. | | `flink_api_secret` | Yes | The corresponding Flink API secret. Use `env_var` to avoid storing secrets in plain text. | | `execution_mode` | No | Sets the default execution mode for statements. One of: `streaming_query` (default), `streaming_ddl`, `snapshot`, `snapshot_ddl`. | | `statement_name_prefix` | No | Adds a prefix to each Flink statement name. Default: `dbt-`. | | `statement_label` | No | Applies a label to all Flink statements so you can filter them in the Confluent Cloud Console. Default: `dbt-confluent`. | | `threads` | No | Sets how many models dbt runs concurrently. Default: `1`. | ##### Understanding Confluent Cloud concepts In Confluent Cloud for Apache Flink, dbt concepts map to Flink SQL as follows: | dbt concept | Confluent Cloud / Flink SQL concept | | --------------------------------- | ----------------------------------------------- | | `database` (via `environment_id`) | Flink SQL catalog (Confluent Cloud environment) | | `schema` (via `dbname`) | Flink SQL database (Kafka cluster) | | `model` | Flink SQL statement | | `table` | Kafka topic with a schema | ##### Environment variables Use environment variables to configure your profile, especially in CI/CD pipelines: ```bash export CONFLUENT_FLINK_API_KEY=your-api-key export CONFLUENT_FLINK_API_SECRET=your-api-secret ``` Then reference them in `profiles.yml` using the `env_var` Jinja function: ```yaml flink_api_key: "{{ env_var('CONFLUENT_FLINK_API_KEY') }}" flink_api_secret: "{{ env_var('CONFLUENT_FLINK_API_SECRET') }}" ``` --- ##### Connect CrateDB to dbt v1 Local development * **Maintained by**: Crate.io, Inc. * **Authors**: CrateDB maintainers * **GitHub repo**: [crate/dbt-cratedb2](https://github.com/crate/dbt-cratedb2) [![](https://img.shields.io/github/stars/crate/dbt-cratedb2?style=for-the-badge)](https://github.com/crate/dbt-cratedb2) * **PyPI package**: `dbt-cratedb2` [![](https://badge.fury.io/py/dbt-cratedb2.svg)](https://badge.fury.io/py/dbt-cratedb2) * **Slack channel**: [Community Forum](https://community.cratedb.com/) * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-cratedb2 Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-cratedb2` #### Configuring dbt-cratedb2 For CrateDB-specific configuration, please refer to [CrateDB configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) [CrateDB](https://cratedb.com/database) is compatible with PostgreSQL, so its dbt adapter strongly depends on dbt-postgres, documented at [PostgreSQL profile setup](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup). CrateDB targets are configured exactly the same way, see also [PostgreSQL configuration](https://docs.getdbt.com/reference/resource-configs/postgres-configs), with just a few things to consider which are special to CrateDB. Relevant details are outlined at [using dbt with CrateDB](https://cratedb.com/docs/guide/integrate/dbt/), which also includes up-to-date information. #### Profile configuration CrateDB targets should be set up using a configuration like this minimal sample of settings in your [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml) file. \~/.dbt/profiles.yml ```yaml cratedb_analytics: target: dev outputs: dev: type: cratedb host: [clustername].aks1.westeurope.azure.cratedb.net port: 5432 user: [username] pass: [password] dbname: crate # Do not change this value. CrateDB's only catalog is `crate`. schema: doc # Define the schema name. CrateDB's default schema is `doc`. ``` --- ##### Connect Databend Cloud to dbt v1 Local development Vendor-supported plugin Some [core functionality](https://github.com/databendcloud/dbt-databend#supported-features) may be limited. If you're interested in contributing, check out the source code repository listed below. * **Maintained by**: Databend Cloud * **Authors**: Shanjie Han * **GitHub repo**: [databendcloud/dbt-databend](https://github.com/databendcloud/dbt-databend) [![](https://img.shields.io/github/stars/databendcloud/dbt-databend?style=for-the-badge)](https://github.com/databendcloud/dbt-databend) * **PyPI package**: `dbt-databend-cloud` [![](https://badge.fury.io/py/dbt-databend-cloud.svg)](https://badge.fury.io/py/dbt-databend-cloud) * **Slack channel**:[]() * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-databend-cloud Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-databend-cloud` #### Configuring dbt-databend-cloud For Databend Cloud-specific configuration, please refer to [Databend Cloud configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) #### Connecting to Databend Cloud with **dbt-databend-cloud** ##### User / Password Authentication Configure your dbt profile for using Databend Cloud: ###### Databend Cloud connection profile profiles.yml ```yaml dbt-databend-cloud: target: dev outputs: dev: type: databend host: databend-cloud-host port: 443 schema: database_name user: username pass: password ``` ###### Description of Profile Fields | Option | Description | Required? | Example | | ------ | -------------------------------------------------- | --------- | --------------------------- | | type | The specific adapter to use | Required | `databend` | | host | The host (hostname) to connect to | Required | `yourorg.datafusecloud.com` | | port | The port to use | Required | `443` | | schema | Specify the schema (database) to build models into | Required | `default` | | user | The username to use to connect to the host | Required | `dbt_admin` | | pass | The password to use for authenticating to the host | Required | `awesome_password` | #### Database User Privileges Your database user would be able to have some abilities to read or write, such as `SELECT`, `CREATE`, and so on. You can find some help [here](https://docs.databend.com/using-databend-cloud/warehouses/connecting-a-warehouse) with Databend Cloud privileges management. | Required Privilege | | ---------------------- | | SELECT | | CREATE | | CREATE TEMPORARY TABLE | | CREATE VIEW | | INSERT | | DROP | | SHOW DATABASE | | SHOW VIEW | | SUPER | #### Supported features | ok | Feature | | -- | --------------------------- | | ✅ | Table materialization | | ✅ | View materialization | | ✅ | Incremental materialization | | ❌ | Ephemeral materialization | | ✅ | Seeds | | ✅ | Sources | | ✅ | Custom data tests | | ✅ | Docs generate | | ❌ | Snapshots | | ✅ | Connection retry | **Note:** * Databend does not support `Ephemeral` and `SnapShot`. You can find more detail [here](https://github.com/datafuselabs/databend/issues/8685) --- ##### Connect Databricks Lakebase to dbt v1 Local development * **Maintained by**: dbt Labs * **Authors**: dbt Labs * **GitHub repo**: [dbt-labs/dbt-adapters](https://github.com/dbt-labs/dbt-adapters) [![](https://img.shields.io/github/stars/dbt-labs/dbt-adapters?style=for-the-badge)](https://github.com/dbt-labs/dbt-adapters) * **PyPI package**: `dbt-postgres` [![](https://badge.fury.io/py/dbt-postgres.svg)](https://badge.fury.io/py/dbt-postgres) * **Slack channel**: [#db-postgres](https://getdbt.slack.com/archives/C0172G2E273) * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Supported * **Minimum data platform version**: ? #### Installing dbt-postgres Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-postgres` #### Configuring dbt-postgres For Lakebase-specific configuration, please refer to [Lakebase configs.](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md) #### Profile Configuration Databricks Lakebase targets are configured exactly the same as [Postgres targets](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md#profile-configuration). Use these key parameters to connect to Databricks Lakebase: * `host name`: Found in **Databricks** > **Compute** > **Database instances** > **Connect with PSQL** using the format `instance-123abcdef456.database.cloud.databricks.com` * `database name`: Use `databricks_postgres` by default * Authentication: dbt-postgres only supports username/password. You can generate a username/password by [enabling Native Postgres Role Login](https://docs.databricks.com/aws/en/oltp/oauth?language=UI#authenticate-with-databricks-identities) and use the role name as the username. To learn more about managing the Postgres roles and privileges, check out the [docs](https://docs.databricks.com/aws/en/oltp/pg-roles#create-postgres-roles-and-grant-privileges-for-databricks-identities). Alternatively you can [generate an OAuth token](https://docs.databricks.com/aws/en/oltp/oauth?language=UI#authenticate-with-databricks-identities) that will need to be refreshed every hour to use with your Databricks username. --- ##### Connect Databricks to dbt v2 (Applies to dbt v2.0 and later) Local development You can configure the Databricks adapter by running `dbt init` in your CLI or manually providing the `profiles.yml` file with the fields configured for your authentication type. The Databricks adapter for dbt v2 supports the following [authentication methods](#supported-authentication-types): * Personal access token (for individual users) * Service Principal token (for service users) * OAuth * OAuth token federation using OIDC #### Warehouse permissions for dbt v2 The Databricks user or service principal that dbt v2 uses must have privileges on the catalog and schemas where models run, plus access required for metadata queries. Requirements depend on whether you use Unity Catalog or the legacy Hive Metastore. ##### Required Databricks objects Before connecting, these objects must exist or be accessible: | Object | Purpose | | ----------------------------- | --------------------------------------- | | **Workspace** | Your Databricks workspace URL (host) | | **SQL warehouse or cluster** | Compute resource (using `http_path`) | | **Catalog** | Unity Catalog or Hive Metastore catalog | | **Schema** | Target schema within the catalog | | **User or service principal** | Identity for authentication | ##### Unity Catalog Required access for the Unity Catalog: | Permission | Object | Purpose | | --------------- | --------------- | ------------------------------------ | | `USE CATALOG` | Catalog | Access the catalog | | `USE SCHEMA` | Schema | Access schemas | | `SELECT` | Tables or views | Read existing data and sources | | `CREATE TABLE` | Schema | Create models materialized as tables | | `CREATE VIEW` | Schema | Create models materialized as views | | `MODIFY` | Tables | Insert, update, and delete data | | `CREATE SCHEMA` | Catalog | Auto-create schemas (if needed) | ##### Hive Metastore Required access for the legacy Hive Metastore: | Permission | Object | Purpose | | ---------- | -------- | ----------------------- | | `USAGE` | Database | Access the database | | `SELECT` | Tables | Read data | | `CREATE` | Database | Create tables and views | | `MODIFY` | Tables | Modify data | ##### Metadata operations The following are required for fundamental dbt features: | Query type | SQL used | Required permission | | ---------------------------- | ---------------------------------------------------- | -------------------------- | | Get table schema | `DESCRIBE TABLE` or `DESCRIBE TABLE EXTENDED` | SELECT on table | | Get table schema (DBR 16.2+) | `DESCRIBE TABLE EXTENDED ... AS JSON` | SELECT on table | | List relations | Query `INFORMATION_SCHEMA.TABLES` | USE CATALOG and USE SCHEMA | | Source freshness | Query `INFORMATION_SCHEMA.TABLES` for `last_altered` | USE CATALOG | | Get view definition | Query `SYSTEM.INFORMATION_SCHEMA.VIEWS` | Access to system catalog | | Create schemas | `CREATE SCHEMA IF NOT EXISTS` | CREATE SCHEMA on catalog | ##### Python models Optional permissions for environments using Python models | Permission | Object | Purpose | | -------------------- | ---------------------- | --------------------------- | | Workspace API access | `/api/2.0/workspace/*` | Create notebook directories | | Notebook import | Workspace | Import Python notebooks | | Job execution | Cluster or warehouse | Run Python models | #### Databricks configuration details The dbt v2 `dbt-databricks` adapter is the only supported connection method for Databricks. `dbt-databricks` can connect to Databricks SQL Warehouses. These warehouses are the recommended way to get started with Databricks. Refer to the [Databricks docs](https://docs.databricks.com/dev-tools/dbt.html#) for more info on how to obtain the credentials for configuring your profile. #### Configure dbt v2 Executing `dbt init` in your CLI will prompt for the following fields: * **Host:** Databricks instance hostname (excluding the `http` or `https` prefix) * **HTTP Path:** Path to your SQL server or cluster * **Schema:** The development/staging/deployment schema for the project * **Catalog (Optional):** The Databricks catalog containing your schemas and tables Alternatively, you can manually create the `profiles.yml` file and configure the fields. See examples in [authentication](#supported-authentication-types) section for formatting. If there is an existing `profiles.yml` file, you are given the option to retain the existing fields or overwrite them. Next, select your authentication method. Follow the on-screen prompts to provide the required information. #### Supported authentication types ##### Personal access token Enter your personal access token (PAT) for the Databricks environment. For more information about obtaining a PAT, refer to the [Databricks documentation](https://docs.databricks.com/aws/en/dev-tools/auth/pat). This is considered a legacy feature by Databricks and OAuth is recommended over PATs. ###### Example personal access token configuration profiles.yml ```yaml default: target: dev outputs: dev: type: databricks database: TRANSFORMING schema: JANE_SMITH host: YOUR.HOST.COM http_path: YOUR/PATH/HERE token: ABC123 auth_type: databricks_cli threads: 16 ``` ##### Service Principal token Enter your Service Principal token for the Databricks environment. For more information about obtaining a Service Principal token, refer to the [Databricks documentation](https://docs.databricks.com/aws/en/admin/users-groups/service-principals). ###### Example Service Principal token configuration profiles.yml ```yaml default: target: dev outputs: dev: type: databricks database: TRANSFORMING schema: JANE_SMITH host: YOUR.HOST.COM http_path: YOUR/PATH/HERE token: ABC123 auth_type: databricks_cli threads: 16 ``` ##### OAuth (Recommended) Selecting the OAuth option will create a connection to your Databricks environment and open a web browser so you can complete the authentication. Users will be prompted to re-authenticate with each new dbt session they initiate. ###### Example OAuth configuration profiles.yml ```yaml default: target: dev outputs: dev: type: databricks database: TRANSFORMING schema: JANE_SMITH host: YOUR.HOST.COM http_path: YOUR/PATH/HERE auth_type: oauth threads: 16 ``` ##### OAuth token federation To use OAuth token federation, you may provide an OIDC token either in an environment variable or a file. ###### Example OIDC configuration using an environment variable To provide the OIDC token using an environment variable, set `auth_type: env-oidc` in `profiles.yml`, and provide the OIDC token in `DATABRICKS_OIDC_TOKEN`. profiles.yml ```yaml target: dev outputs: dev: type: databricks database: TRANSFORMING schema: JANE_SMITH host: YOUR.HOST.COM http_path: YOUR/PATH/HERE auth_type: env-oidc client_id: CLIENT_ID threads: 16 ``` ###### Example OIDC configuration using a token file To provide the OIDC token using a token file, set `auth_type: file-oidc` in `profiles.yml` and provide the file path in `oidc_token_filepath`. You can instead leave `oidc_token_filepath` out of `profiles.yml` and set the path in the `DATABRICKS_OIDC_TOKEN_FILE` or `DATABRICKS_OIDC_TOKEN_FILEPATH` environment variable. dbt uses the environment variable only when `oidc_token_filepath` isn't in `profiles.yml`. When both are set, dbt only uses the `profiles.yml` setting and never falls back to the environment variable. We recommend setting the path in `profiles.yml`. profiles.yml ```yaml default: target: dev outputs: dev: type: databricks database: TRANSFORMING schema: JANE_SMITH host: YOUR.HOST.COM http_path: YOUR/PATH/HERE auth_type: file-oidc client_id: CLIENT_ID oidc_token_filepath: PATH/TO/TOKEN/FILE threads: 16 ``` #### Optional connection settings Add these optional fields to your Databricks output in `profiles.yml` to control connection timeouts and retries. | Profile field | Description | Default | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `connect_timeout` | How many seconds dbt waits for a connection to Databricks before giving up. Enter a number greater than `0`, or dbt returns an error. For example, `600` becomes 600s (10 mins). Raise this if your jobs start while your SQL warehouse is still starting up. | `200` | | `connect_retries` | The number of times dbt should retry the connection to Databricks after the first attempt fails. A retry runs only when Databricks marks the error as retryable, or when `retry_all` is `true`. | `1` | | `retry_all` | Set to `true` to retry every connection error. Set to `false` to retry only errors Databricks marks as retryable. | `false` | These settings apply only to connecting. They don't retry queries that fail after dbt has connected. ###### Example The following example sets `connect_timeout` to `600` (10 minutes) so dbt can wait for a SQL warehouse that is still starting. profiles.yml ```yaml outputs: dev: type: databricks # ...your other connection fields connect_timeout: 600 ``` #### More information Find Databricks-specific configuration information in the [Databricks adapter reference guide](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md). --- ##### Connect Decodable to dbt v1 Local development Community plugin Some core functionality may be limited. If you're interested in contributing, see the source code for the repository listed below. * **Maintained by**: Decodable * **Authors**: Decodable Team * **GitHub repo**: [decodableco/dbt-decodable](https://github.com/decodableco/dbt-decodable) [![](https://img.shields.io/github/stars/decodableco/dbt-decodable?style=for-the-badge)](https://github.com/decodableco/dbt-decodable) * **PyPI package**: `dbt-decodable` [![](https://badge.fury.io/py/dbt-decodable.svg)](https://badge.fury.io/py/dbt-decodable) * **Slack channel**: [#general](https://decodablecommunity.slack.com) * **Supported dbt version**: 1.3.1 and newer * **dbt support**: Not supported * **Minimum data platform version**: n/a #### Installing dbt-decodable Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-decodable` #### Configuring dbt-decodable For Decodable-specific configuration, please refer to [Decodable configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) #### Connecting to Decodable with **dbt-decodable** Do the following steps to connect to Decodable with dbt. ##### Prerequisites In order to properly connect to Decodable, you must have the Decodable CLI installed and have used it to login to Decodable at least once. See [Install the Decodable CLI](https://docs.decodable.co/docs/setup#install-the-cli-command-line-interface) for more information. ##### Steps To connect to Decodable with dbt, you'll need to add a Decodable profile to your `profiles.yml` file. A Decodable profile has the following fields. \~/.dbt/profiles.yml ```yaml dbt-decodable: target: dev outputs: dev: type: decodable database: None schema: None account_name: [your account] profile_name: [name of the profile] materialize_tests: [true | false] timeout: [ms] preview_start: [earliest | latest] local_namespace: [namespace prefix] ``` ###### Description of Profile Fields | Option | Description | Required? | Example | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------------ | | type | The specific adapter to use | Required | `decodable` | | database | Required but unused by this adapter. | Required | | | schema | Required but unused by this adapter. | Required | | | account\_name | The name of your Decodable account. | Required | `my_awesome_decodable_account` | | profile\_name | The name of your Decodable profile. | Required | `my_awesome_decodable_profile` | | materialize\_tests | Specify whether to materialize tests as a pipeline/stream pair. Defaults to false. | Optional | `false` | | timeout | The amount of time, in milliseconds, that a preview request runs. Defaults to 60000. | Optional | `60000` | | preview\_start | Specify where preview should start reading data from. If set to `earliest`, then preview will start reading from the earliest record possible. If set to `latest`, preview will start reading from the latest record. Defaults to `earliest`. | Optional | `latest` | | local\_namespace | Specify a prefix to add to all entities created on Decodable. Defaults to `none`, meaning that no prefix is added. | Optional | `none` | #### Supported features | Name | Supported | Notes | | --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Table materialization | Yes | Only table materialization are supported. A dbt table model translates to a pipeline/stream pair in Decodable, both sharing the same name. Pipelines for models are automatically activated upon materialization. To materialize your models, run the `dbt run` command which does the following: 1. Create a stream with the model's name and schema inferred by Decodable from the model's SQL.<br />2. Create a pipeline that inserts the SQL's results into the newly created stream.<br />3. Activate the pipeline. By default, the adapter does not tear down and recreate the model on Decodable if no changes to the model have been detected. Invoking dbt with the `--full-refresh` flag or setting that configuration option for a specific model causes the corresponding resources on Decodable to be destroyed and built from scratch. | | View materialization | No | | | Incremental materialization | No | | | Ephemeral materialization | No | | | Seeds | Yes | Running the `dbt seed` command performs the following steps for each specified seed: 1. Create a REST connection and an associated stream with the same name as the seed.<br />2. Activate the connection.<br />3. Send the data stored in the seed’s `.csv` file to the connection as events.<br />4. Deactivate the connection. After the `dbt seed` command has finished running, you can access the seed's data on the newly created stream. | | Tests | Yes | The `dbt test` command behaves differently depending on the `materialize_tests` option set for the specified target.<br /><br />If `materialize_tests = false`, then tests are only run after the preview job has completed and returned results. How long a preview job takes as well as what records are returned are defined by the `timeout` and `preview_start` configurations respectively.<br /><br />If `materialize_tests = true`, then dbt persists the specified tests as pipeline/stream pairs in Decodable. Use this configuration to allow for continuous testing of your models. You can run a preview on the created stream with the Decodable CLI or web interface to monitor the results. | | Sources | No | Sources in dbt correspond to Decodable source connections. However, the `dbt source` command is not supported. | | Docs generate | No | For details about your models, check your Decodable account. | | Snapshots | No | Snapshots and the `dbt snapshot` command are not supported. | #### Additional operations `dbt-decodable` provides a set of commands for managing the project’s resources on Decodable. Those commands can be run using `dbt run-operation {name} --args {args}`. For example, the following command runs the `delete_streams` operation ```text dbt run-operation delete_streams --args '{streams: [stream1, stream2], skip_errors: True}' ``` **stop\_pipelines(pipelines)** * pipelines: An optional list of pipeline names to deactivate. Defaults to none. Deactivate all pipelines for resources defined within the project. If the pipelines argument is provided, then only the specified pipelines are deactivated. <br /> <br /> **delete\_pipelines(pipelines)** * pipelines: An optional list of pipeline names to delete. Defaults to none. Delete all pipelines for resources defined within the project. If the pipelines argument is provided, then only the specified pipelines are deleted. <br /> <br /> **delete\_streams(streams, skip\_errors)** * streams: An optional list of stream names to delete. Defaults to none. * skip\_errors: Specify whether to treat errors as warnings. When set to true, any stream deletion failures are reported as warnings. When set to false, the operation stops when a stream cannot be deleted. Defaults to true. Delete all streams for resources defined within the project. If a pipeline is associated with a stream, then neither the pipeline nor stream are deleted. See the cleanup operation for a complete removal of stream/pipeline pairs.<br /><br /> **cleanup(list, models, seeds, tests)** * list: An optional list of resource entity names to delete. Defaults to none. * models: Specify whether to include models during cleanup. Defaults to true. * seeds: Specify whether to include seeds during cleanup. Defaults to true. * tests: Specify whether to include tests during cleanup. Defaults to true. <br /><br />Delete all Decodable entities resulting from the materialization of the project’s resources, i.e. connections, streams, and pipelines. If the list argument is provided, then only the specified resource entities are deleted. If the models, seeds, or test arguments are provided, then those resource types are also included in the cleanup. Tests that have not been materialized are not included in the cleanup. --- ##### Connect DeltaStream to dbt v1 Local development * **Maintained by**: Community * **Authors**: DeltaStream Team * **GitHub repo**: [deltastreaminc/dbt-deltastream](https://github.com/deltastreaminc/dbt-deltastream) [![](https://img.shields.io/github/stars/deltastreaminc/dbt-deltastream?style=for-the-badge)](https://github.com/deltastreaminc/dbt-deltastream) * **PyPI package**: `dbt-deltastream` [![](https://badge.fury.io/py/dbt-deltastream.svg)](https://badge.fury.io/py/dbt-deltastream) * **Slack channel**: [#db-deltastream]() * **Supported dbt version**: v1.10.0 and newer * **dbt support**: Not supported * **Minimum data platform version**: ? #### Installing dbt-deltastream Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-deltastream` #### Configuring dbt-deltastream For DeltaStream-specific configuration, please refer to [DeltaStream configs.](https://docs.getdbt.com/reference/resource-configs/deltastream-configs.md) #### Connecting to DeltaStream with **dbt-deltastream** To connect to DeltaStream from dbt, you'll need to add a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) to your `profiles.yml` file. A DeltaStream profile conforms to the following syntax: profiles.yml ```yaml <profile-name>: target: <target-name> outputs: <target-name>: type: deltastream # Required parameters token: [ your-api-token ] # Authentication token for DeltaStream API database: [ your-database ] # Target database name schema: [ your-schema ] # Target schema name organization_id: [ your-org-id ] # Organization identifier # Optional parameters url: [ https://api.deltastream.io/v2 ] # DeltaStream API URL, defaults to https://api.deltastream.io/v2 timezone: [ UTC ] # Timezone for operations, defaults to UTC session_id: [ <empty string> ] # Custom session identifier for debugging purpose role: [ <empty string> ] # User role store: [ <empty string> ] # Target store name compute_pool: [ <empty string> ] # Compute pool name to be used if any else use the default compute pool ``` ##### Description of DeltaStream profile fields | Field | Required | Description | | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ | | `type` | ✅ | This must be included either in `profiles.yml` or in the `dbt_project.yml` file. Must be set to `deltastream`. | | `token` | ✅ | Authentication token for DeltaStream API. This should be stored securely, preferably as an environment variable. | | `database` | ✅ | Target default database name in DeltaStream where your dbt models will be created. | | `schema` | ✅ | Target default schema name within the specified database. | | `organization_id` | ✅ | Organization identifier that determines which DeltaStream organization you're connecting to. | | `url` | ❌ | DeltaStream API URL. Defaults to `https://api.deltastream.io/v2` if not specified. | | `timezone` | ❌ | Timezone for operations. Defaults to `UTC` if not specified. | | `session_id` | ❌ | Custom session identifier for debugging purposes. Helps track operations in DeltaStream logs. | | `role` | ❌ | User role within the DeltaStream organization. If not specified, uses the default role associated with the token. | | `store` | ❌ | Target default store name. Stores represent external system connections (Kafka, PostgreSQL, etc.) in DeltaStream. | | `compute_pool` | ❌ | Compute pool name to be used for models that require computational resources. If not specified, uses the default compute pool. | #### Security best practices When configuring your project for production, it is strongly recommended to use environment variables to store sensitive information such as the authentication token: profiles.yml ```yaml your_profile_name: target: prod outputs: prod: type: deltastream token: "{{ env_var('DELTASTREAM_API_TOKEN') }}" database: "{{ env_var('DELTASTREAM_DATABASE') }}" schema: "{{ env_var('DELTASTREAM_SCHEMA') }}" organization_id: "{{ env_var('DELTASTREAM_ORG_ID') }}" ``` #### Troubleshooting connections If you encounter issues connecting to DeltaStream from dbt, verify the following: ##### Authentication issues * Ensure your API token is valid and has not expired * Verify the token has appropriate permissions for the target organization * Check that the `organization_id` matches your DeltaStream organization --- ##### Connect Doris to dbt v1 Local development * **Maintained by**: SelectDB * **Authors**: catpineapple,JNSimba * **GitHub repo**: [selectdb/dbt-doris](https://github.com/selectdb/dbt-doris) [![](https://img.shields.io/github/stars/selectdb/dbt-doris?style=for-the-badge)](https://github.com/selectdb/dbt-doris) * **PyPI package**: `dbt-doris` [![](https://badge.fury.io/py/dbt-doris.svg)](https://badge.fury.io/py/dbt-doris) * **Slack channel**: [#db-doris](https://www.getdbt.com/community) * **Supported dbt version**: v1.3.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: #### Installing dbt-doris Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-doris` #### Configuring dbt-doris For Apache Doris / SelectDB-specific configuration, please refer to [Apache Doris / SelectDB configs.](https://docs.getdbt.com/reference/resource-configs/doris-configs.md) #### Connecting to Doris/SelectDB with **dbt-doris** ##### User / Password Authentication Configure your dbt profile for using Doris: ###### Doris connection profile profiles.yml ```yaml dbt-doris: target: dev outputs: dev: type: doris host: 127.0.0.1 port: 9030 schema: database_name username: username password: password ``` ###### Description of Profile Fields | Option | Description | Required? | Example | | -------- | -------------------------------------------------------------------------------------------------------------------------------- | --------- | ----------- | | type | The specific adapter to use | Required | `doris` | | host | The hostname to connect to | Required | `127.0.0.1` | | port | The port to use | Required | `9030` | | schema | Specify the schema (database) to build models into, doris have not schema to make a collection of table or view' like PostgreSql | Required | `dbt` | | username | The username to use to connect to the doris | Required | `root` | | password | The password to use for authenticating to the doris | Required | `password` | #### Database User Privileges Your Doris/SelectDB database user would be able to have some abilities to read or write. You can find some help [here](https://doris.apache.org/docs/admin-manual/privilege-ldap/user-privilege) with Doris privileges management. | Required Privilege | | ------------------ | | Select\_priv | | Load\_priv | | Alter\_priv | | Create\_priv | | Drop\_priv | --- ##### Connect Dremio to dbt v1 Local development * **Maintained by**: Dremio * **Authors**: Dremio * **GitHub repo**: [dremio/dbt-dremio](https://github.com/dremio/dbt-dremio) [![](https://img.shields.io/github/stars/dremio/dbt-dremio?style=for-the-badge)](https://github.com/dremio/dbt-dremio) * **PyPI package**: `dbt-dremio` [![](https://badge.fury.io/py/dbt-dremio.svg)](https://badge.fury.io/py/dbt-dremio) * **Slack channel**: [db-dremio](https://docs.getdbt.com/[https://www.getdbt.com/community]\(https://getdbt.slack.com/archives/C049G61TKBK\)) * **Supported dbt version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Dremio 22.0 #### Installing dbt-dremio Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-dremio` #### Configuring dbt-dremio For Dremio-specific configuration, please refer to [Dremio configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) Follow the repository's link for OS dependencies. note [Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) are not supported. #### Prerequisites for Dremio Cloud Before connecting from project to Dremio Cloud, follow these prerequisite steps: * Ensure that you have the ID of the Sonar project that you want to use. See [Obtaining the ID of a Project](https://docs.dremio.com/cloud/cloud-entities/projects/#obtaining-the-id-of-a-project). * Ensure that you have a personal access token (PAT) for authenticating to Dremio Cloud. See [Creating a Token](https://docs.dremio.com/cloud/security/authentication/personal-access-token/#creating-a-token). * Ensure that Python 3.9.x or later is installed on the system that you are running dbt on. #### Prerequisites for Dremio Software * Ensure that you are using version 22.0 or later. * Ensure that Python 3.9.x or later is installed on the system that you are running dbt on. * If you want to use TLS to secure the connection between dbt and Dremio Software, configure full wire encryption in your Dremio cluster. For instructions, see [Configuring Wire Encryption](https://docs.dremio.com/software/deployment/wire-encryption-config/). #### Initializing a Project 1. Run the command `dbt init <project_name>`. 2. Select `dremio` as the database to use. 3. Select one of these options to generate a profile for your project: * `dremio_cloud` for working with Dremio Cloud * `software_with_username_password` for working with a Dremio Software cluster and authenticating to the cluster with a username and a password * `software_with_pat` for working with a Dremio Software cluster and authenticating to the cluster with a personal access token Next, configure the profile for your project. #### Profiles When you initialize a project, you create one of these three profiles. You must configure it before trying to connect to Dremio Cloud or Dremio Software. * Profile for Dremio Cloud * Profile for Dremio Software with Username/Password Authentication * Profile for Dremio Software with Authentication Through a Personal Access Token For descriptions of the configurations in these profiles, see [Configurations](#configurations). ##### Cloud ```yaml [project name]: outputs: dev: cloud_host: api.dremio.cloud cloud_project_id: [project ID] object_storage_source: [name] object_storage_path: [path] dremio_space: [name] dremio_space_folder: [path] pat: [personal access token] threads: [integer >= 1] type: dremio use_ssl: true user: [email address] target: dev ``` ##### Software (Username/Password) ```yaml [project name]: outputs: dev: password: [password] port: [port] software_host: [hostname or IP address] object_storage_source: [name object_storage_path: [path] dremio_space: [name] dremio_space_folder: [path] threads: [integer >= 1] type: dremio use_ssl: [true|false] user: [username] target: dev ``` ##### Software (Personal Access Token) ```yaml [project name]: outputs: dev: pat: [personal access token] port: [port] software_host: [hostname or IP address] object_storage_source: [name object_storage_path: [path] dremio_space: [name] dremio_space_folder: [path] threads: [integer >= 1] type: dremio use_ssl: [true|false] user: [username] target: dev ``` #### Configurations Common to Profiles for Dremio Cloud and Dremio Software | Configuration | Required? | Default Value | Description | | ----------------------- | --------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | Yes | dremio | Auto-populated when creating a Dremio project. Do not change this value. | | `threads` | Yes | 1 | The number of threads the dbt project runs on. | | `object_storage_source` | No | $scratch | The name of the filesystem in which to create tables, materialized views, tests, and other objects. The dbt alias is `datalake`. This name corresponds to the name of a source in the **Object Storage** section of the Datasets page in Dremio, which is "Samples" in the following image:[![dbt samples path](/img/reference/dremio-setup/dbt-Samples.png?v=2 "dbt samples path")](#)dbt samples path | | `object_storage_path` | No | `no_schema` | The path in the filesystem in which to create objects. The default is the root level of the filesystem. The dbt alias is `root_path`. Nested folders in the path are separated with periods. This value corresponds to the path in this location in the Datasets page in Dremio, which is "samples.dremio.com.Dremio University" in the following image:[![dbt samples path](/img/reference/dremio-setup/dbt-SamplesPath.png?v=2 "dbt samples path")](#)dbt samples path | | `dremio_space` | No | `@\<username>` | The value of the Dremio space in which to create views. The dbt alias is `database`. This value corresponds to the name in this location in the **Spaces** section of the Datasets page in Dremio:[![dbt spaces](/img/reference/dremio-setup/dbt-Spaces.png?v=2 "dbt spaces")](#)dbt spaces | | `dremio_space_folder` | No | `no_schema` | The folder in the Dremio space in which to create views. The default is the top level in the space. The dbt alias is `schema`. Nested folders are separated with periods. This value corresponds to the path in this location in the Datasets page in Dremio, which is `Folder1.Folder2` in the following image:[![Folder1.Folder2](/img/reference/dremio-setup/dbt-SpacesPath.png?v=2 "Folder1.Folder2")](#)Folder1.Folder2 | ##### Configurations in Profiles for Dremio Cloud | Configuration | Required? | Default Value | Description | | ------------------ | --------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cloud_host` | Yes | `api.dremio.cloud` | US Control Plane: `api.dremio.cloud`<br />EU Control Plane: `api.eu.dremio.cloud` | | `user` | Yes | None | Email address used as a username in Dremio Cloud | | `pat` | Yes | None | The personal access token to use for authentication. See [Personal Access Tokens](https://docs.dremio.com/cloud/security/authentication/personal-access-token/) for instructions about obtaining a token. | | `cloud_project_id` | Yes | None | The ID of the Sonar project in which to run transformations. | | `use_ssl` | Yes | `true` | The value must be `true`. | ##### Configurations in Profiles for Dremio Software | Configuration | Required? | Default Value | Description | | --------------- | --------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `software_host` | Yes | None | The hostname or IP address of the coordinator node of the Dremio cluster. | | `port` | Yes | `9047` | Port for Dremio Software cluster API endpoints. | | `user` | Yes | None | The username of the account to use when logging into the Dremio cluster. | | `password` | Yes, if you are not using the pat configuration. | None | The password of the account to use when logging into the Dremio cluster. | | `pat` | Yes, if you are not using the user and password configurations. | None | The personal access token to use for authenticating to Dremio. See [Personal Access Tokens](https://docs.dremio.com/software/security/personal-access-tokens/) for instructions about obtaining a token. The use of a personal access token takes precedence if values for the three configurations user, password, and pat are specified. | | `use_ssl` | Yes | `true` | Acceptable values are `true` and `false`. If the value is set to true, ensure that full wire encryption is configured in your Dremio cluster. See [Prerequisites for Dremio Software](#prerequisites-for-dremio-software). | --- ##### Connect DuckDB to dbt v2 (Applies to dbt v2.0 and later) Local development DuckDB with dbt v2 is the easiest way to get a dbt project running locally — no warehouse account or credentials required. [DuckDB](https://duckdb.org) is an embedded database that runs entirely in-process, so dbt connects directly to a local `.duckdb` file with no additional setup. DuckDB does not require authentication — it runs locally on your machine. ##### Installing dbt-duckdb The DuckDB adapter is built into v2. To get started, [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). We recommend using the [VS Code Extension](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) as the development interface. ###### DuckDB driver and extensions dbt v2 ships with a built-in DuckDB driver. However, this bundled driver does *not* support loading DuckDB extensions (for example, `httpfs`, `parquet`, or `spatial`). To use [DuckDB extensions](https://duckdb.org/docs/current/extensions/overview) with dbt v2, install the DuckDB driver with [`dbc`](https://docs.columnar.tech/dbc/#__tabbed_2_4). dbt v2 checks for a system-installed DuckDB driver first and falls back to the bundled driver if none is found. For connection examples and shared profile settings, refer to [Connecting to DuckDB](#connecting-to-duckdb). ##### Limitations Some features available in the `dbt-duckdb` adapter for dbt v1 are not yet supported. Current adapter feature parity work and SQL analysis gaps are tracked in [dbt#14393](https://github.com/dbt-labs/dbt/issues/14393). ###### Static analysis and local flat files dbt v2 performs static analysis on your SQL models to determine column types and lineage without executing queries. If your models reference local flat files (CSV, Parquet, or JSON) through DuckDB's `read_csv()`, `read_parquet()`, or `read_json()` functions, dbt v2 may not be able to infer the schema of those files at analysis time. As a result, you may see type-resolution warnings or compilation errors even when the query would succeed at runtime. To learn more, refer to [New concepts](https://docs.getdbt.com/docs/build/about-static-analysis.md). #### Connecting to DuckDB [DuckDB](https://duckdb.org) is an embedded database, similar to SQLite, but designed for OLAP-style analytics instead of OLTP. There are several ways to connect dbt to DuckDB depending on where you want your data to live. Configure your `profiles.yml` using the examples in the following sections: * [In-memory](#in-memory) * [Local file](#local-file) * [MotherDuck](#motherduck) * [Attaching additional databases](#attaching-additional-databases) Refer to the following table for the fields to use in your `profiles.yml`. `type: duckdb` is *always* required. Use `path` for local files or MotherDuck connection strings, and use `:memory:` or omit `path` for an in-memory database. | Profile field | Description | Example | | ------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------- | | `type` | The adapter type. | `duckdb` | | `path` | Path to a DuckDB database file, a MotherDuck `md:` connection string, or `:memory:` for an in-memory database. | `./jaffle_shop.duckdb` | | `schema` | The schema name where dbt creates objects. | `main` | | `threads` | Number of threads dbt uses when building models concurrently. | `4` | | `extensions` | List of [DuckDB extensions](https://duckdb.org/docs/extensions/overview) to load at startup. | `httpfs`, `parquet` | | `settings` | Map of [DuckDB configuration options](https://duckdb.org/docs/sql/configuration) to set at startup. | `s3_region: us-east-1` | If you're using dbt v2, loading extensions requires you to install the DuckDB driver with [`dbc`](https://docs.columnar.tech/dbc/#__tabbed_2_4). Refer to [DuckDB driver and extensions](#driver-and-extensions) for details. ##### In-memory The simplest configuration requires only `type: duckdb` in your profile. This runs an in-memory database — all data is lost after the run completes. This is useful for testing pipelines and for workflows that operate purely on external CSV, Parquet, or JSON files. profiles.yml ```yaml default: outputs: dev: type: duckdb target: dev ``` ##### Local file To persist data between runs, set `path` to a `.duckdb` file on your local filesystem. DuckDB creates the file automatically if it doesn't exist. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: duckdb path: './my_project.duckdb' schema: main # optional; defaults to main threads: 4 # optional ``` You can use a relative path (resolved relative to your `profiles.yml` file) or an absolute path. `dbt-duckdb` automatically sets the `database` property to the basename of the file with the suffix removed (for example, `/tmp/a/dbfile.duckdb` sets `database` to `dbfile`). ##### MotherDuck In `dbt-duckdb 1.5.2` and later, you can connect to a DuckDB instance running on [MotherDuck](https://motherduck.com) by setting `path` to an `md:` connection string: profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: duckdb path: "md:my_db?motherduck_token={{ env_var('MOTHERDUCK_TOKEN') }}" threads: 4 ``` MotherDuck databases generally work the same way as local DuckDB databases, with a few differences described in [MotherDuck's documentation](https://motherduck.com/docs/architecture-and-capabilities#considerations-and-limitations). MotherDuck preloads common DuckDB extensions but does not support loading custom extensions or user-defined functions. ##### Attaching additional databases DuckDB supports [attaching additional databases](https://duckdb.org/docs/sql/statements/attach.html) so you can read and write from multiple databases. Configure additional databases using the `attach` argument in your profile: profiles.yml ```yaml default: outputs: dev: type: duckdb path: /tmp/dbt.duckdb attach: - path: /tmp/other.duckdb - path: ./yet/another.duckdb alias: yet_another - path: s3://yep/even/this/works.duckdb read_only: true - path: sqlite.db type: sqlite - path: postgresql://username@hostname/dbname type: postgres target: dev ``` You can refer to attached databases by the basename of the file (without its suffix) or by an `alias` you specify. The `type` argument supports `duckdb`, `sqlite`, and `postgres`. You can also pass arbitrary options using the `options` dictionary — refer to [Arbitrary ATTACH options](https://docs.getdbt.com/reference/resource-configs/duckdb-configs.md#arbitrary-attach-options) for details. For DuckLake, use `ducklake:` for local databases. For MotherDuck-managed DuckLake, use `md:` with `is_ducklake: true`. Refer to the [DuckLake configuration](https://docs.getdbt.com/reference/resource-configs/duckdb-configs.md#ducklake) section for details. #### Extensions You can load any supported [DuckDB extensions](https://duckdb.org/docs/extensions/overview) by listing them in the `extensions` field in your profile. You can also set any additional [DuckDB configuration options](https://duckdb.org/docs/sql/configuration) in the `settings` field. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: duckdb path: 'file_path/database_name.duckdb' extensions: - httpfs - parquet settings: s3_region: my-aws-region s3_access_key_id: "{{ env_var('S3_ACCESS_KEY_ID') }}" s3_secret_access_key: "{{ env_var('S3_SECRET_ACCESS_KEY') }}" ``` You can also configure extensions from outside the core extension repository (such as a community extension) by specifying a `name`/`repo` pair: ```yml extensions: - httpfs - parquet - name: h3 repo: community - name: uc_catalog repo: core_nightly ``` For configuring cloud storage access using DuckDB's Secrets Manager or fsspec filesystems, refer to the [DuckDB configurations](https://docs.getdbt.com/reference/resource-configs/duckdb-configs.md) page. #### More information Find DuckDB-specific configuration information in the [DuckDB adapter reference guide](https://docs.getdbt.com/reference/resource-configs/duckdb-configs.md). For adapter source code, refer to the [`dbt-duckdb` repository](https://github.com/duckdb/dbt-duckdb). For adapter release notes, refer to the [`dbt-duckdb` releases page](https://github.com/duckdb/dbt-duckdb/releases). --- ##### Connect Exasol to dbt v1 Local development * **Maintained by**: Exasol * **Authors**: Torsten Glunde, Ilija Kutle * **GitHub repo**: [exasol/dbt-exasol](https://github.com/exasol/dbt-exasol) [![](https://img.shields.io/github/stars/exasol/dbt-exasol?style=for-the-badge)](https://github.com/exasol/dbt-exasol) * **PyPI package**: `dbt-exasol` [![](https://badge.fury.io/py/dbt-exasol.svg)](https://badge.fury.io/py/dbt-exasol) * **Slack channel**: [n/a]() * **Supported dbt version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Exasol 6.x #### Installing dbt-exasol Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-exasol` #### Configuring dbt-exasol For Exasol-specific configuration, please refer to [Exasol configs.](https://docs.getdbt.com/reference/resource-configs/exasol-configs.md) ##### Connecting to Exasol with **dbt-exasol** ###### User / password authentication Configure your dbt profile for using Exasol: ###### Exasol connection information profiles.yml ```yaml dbt-exasol: target: dev outputs: dev: type: exasol threads: 1 dsn: HOST:PORT user: USERNAME password: PASSWORD dbname: db schema: SCHEMA ``` ###### OpenID authentication (Exasol SaaS) For Exasol SaaS environments, you can authenticate using OpenID tokens instead of username and password: profiles.yml ```yaml dbt-exasol: target: dev outputs: dev: type: exasol threads: 1 dsn: HOST:PORT user: USERNAME access_token: YOUR_ACCESS_TOKEN # or use refresh_token dbname: db schema: SCHEMA encryption: True # required for SaaS ``` * **`access_token`** — Personal access token for OpenID authentication * **`refresh_token`** — Refresh token for OpenID authentication (alternative to `access_token`) info Use either `access_token` or `refresh_token`, not both. TLS encryption is required when using OpenID authentication with Exasol SaaS. ###### Optional parameters * **`connection_timeout`** — defaults to pyexasol default * **`socket_timeout`** — defaults to pyexasol default * **`query_timeout`** — defaults to pyexasol default * **`compression`** — default: False * **`encryption`** — default: True. Enables SSL/TLS encryption for secure connections. Required for Exasol SaaS * **`validate_server_certificate`** — default: True. Validates the SSL/TLS certificate when encryption is enabled. Set to False only for development/testing with self-signed certificates (not recommended for production) * **`protocol_version`** — default: v3 * **`row_separator`** — default: CRLF for windows - LF otherwise * **`timestamp_format`** — default: `YYYY-MM-DDTHH:MI:SS.FF6` SSL/TLS Certificate Validation By default, dbt-exasol validates SSL/TLS certificates when `encryption=True`. For development/testing with self-signed certificates, you can either: * Set `validate_server_certificate: False` (not recommended for production) * Use a certificate fingerprint in the DSN: `dsn: myhost/FINGERPRINT:8563` * Use `dsn: myhost/nocertcheck:8563` to skip validation (testing only) For more information, see the [PyExasol security documentation](https://exasol.github.io/pyexasol/master/user_guide/configuration/security.html). --- ##### Connect Extrica to dbt v1 Local development #### Overview of dbt-extrica * **Maintained by**: Extrica, Trianz * **Authors**: Gaurav Mittal, Viney Kumar, Mohammed Feroz, and Mrinal Mayank * **GitHub repo**: [extricatrianz/dbt-extrica](https://github.com/extricatrianz/dbt-extrica) * **PyPI package**: `dbt-extrica` [![](https://badge.fury.io/py/dbt-extrica.svg)](https://badge.fury.io/py/dbt-extrica) * **Supported dbt v1 version**: v1.7.2 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-extrica Use `pip` to install the adapter, which automatically installs `dbt-core` and any additional dependencies. Use the following command for installation: `python -m pip install dbt-extrica` #### Connecting to Extrica ###### Example profiles.yml Here is an example of dbt-extrica profiles. At a minimum, you need to specify `type`, `method`, `username`, `password` `host`, `port`, `schema`, `catalog` and `threads`. \~/.dbt/profiles.yml ```yaml <profile-name>: outputs: dev: type: extrica method: jwt username: [username for jwt auth] password: [password for jwt auth] host: [extrica hostname] port: [port number] schema: [dev_schema] catalog: [catalog_name] threads: [1 or more] prod: type: extrica method: jwt username: [username for jwt auth] password: [password for jwt auth] host: [extrica hostname] port: [port number] schema: [dev_schema] catalog: [catalog_name] threads: [1 or more] target: dev ``` ###### Description of Extrica Profile Fields | Parameter | Type | Description | | --------- | ------- | -------------------------------------------------------------------------------------------------------------------- | | type | string | Specifies the type of dbt adapter (Extrica). | | method | jwt | Authentication method for JWT authentication. | | username | string | Username for JWT authentication. The obtained JWT token is used to initialize a trino.auth.JWTAuthentication object. | | password | string | Password for JWT authentication. The obtained JWT token is used to initialize a trino.auth.JWTAuthentication object. | | host | string | The host parameter specifies the hostname or IP address of the Extrica's Trino server. | | port | integer | The port parameter specifies the port number on which the Extrica's Trino server is listening. | | schema | string | Schema or database name for the connection. | | catalog | string | Name of the catalog representing the data source. | | threads | integer | Number of threads for parallel execution of queries. (1 or more) | --- ##### Connect Firebird to dbt v1 Local development Community plugin Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below. * **Maintained by**: Community * **Authors**: Fernando Lozer (https://github.com/flozer) * **GitHub repo**: [flozer/dbt-firebird](https://github.com/flozer/dbt-firebird) [![](https://img.shields.io/github/stars/flozer/dbt-firebird?style=for-the-badge)](https://github.com/flozer/dbt-firebird) * **PyPI package**: `dbt-firebird` [![](https://badge.fury.io/py/dbt-firebird.svg)](https://badge.fury.io/py/dbt-firebird) * **Slack channel**: [n/a](https://www.getdbt.com/community) * **Supported dbt version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Firebird 3.0 #### Installing dbt-firebird Note: Installing an adapter doesn't install dbt v1 automatically. This is because adapters and dbt v1 versions are decoupled to avoid overwriting dbt v1 installations. Use the following command for installation: `python -m pip install dbt-core dbt-firebird` #### Configuring dbt-firebird For Firebird-specific configuration, please refer to [Firebird configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) The [dbt-firebird](https://pypi.org/project/dbt-firebird/) adapter connects dbt v1 to [Firebird](https://firebirdsql.org/) databases (3.0, 4.0, and 5.0) through the official [firebird-driver](https://pypi.org/project/firebird-driver/) package. It supports table, view, incremental, seed, and snapshot (SCD2) materializations, data tests, generated documentation, and an interactive `dbt init` experience. The adapter is validated against the official dbt v1 test suites and a CI matrix covering Firebird 3/4/5 on Python 3.9 to 3.12. #### Connecting to Firebird with dbt-firebird Firebird databases are single files. The `path` field is the database file path as seen by the server. If dbt runs on the same machine as the server, this is a normal local path; for a remote server, it is a server-side path (for example, `/data/prod.fdb` on Linux). ##### Local Use this profile when Firebird is running on the same computer as dbt. You only need the database file `path`. \~/.dbt/profiles.yml ```yaml # Replace PROFILE_NAME with the profile name in your dbt_project.yml PROFILE_NAME: target: dev outputs: dev: type: firebird path: C:/data/mydb.fdb user: SYSDBA password: masterkey threads: 1 ``` ##### Remote server Use this profile when Firebird is running on another computer. Include `host`, `port`, and the `path` as that server sees the file. \~/.dbt/profiles.yml ```yaml # Replace PROFILE_NAME with the profile name in your dbt_project.yml PROFILE_NAME: target: dev outputs: dev: type: firebird host: 192.168.0.10 port: 3050 path: /data/prod.fdb user: SYSDBA password: masterkey charset: WIN1252 threads: 1 ``` ##### Description of Firebird profile fields These fields apply to both local and remote profiles: | Field | Description | Required? | Example | | ------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------- | ------------------ | | type | The specific adapter to use | Required | `firebird` | | path | Database file path as seen by the server. Use forward slashes | Required | `C:/data/mydb.fdb` | | host | Server hostname or IP. Omit it to connect to a local file directly | Optional | `192.168.0.10` | | port | Server TCP port | Optional (default `3050`) | `3050` | | user | Database user | Required | `SYSDBA` | | password | Password | Required | `masterkey` | | charset | Connection character set. Refer to [Character sets](#character-sets) | Optional (default `UTF8`) | `WIN1252` | | role | Firebird role, if you use them | Optional | `RDB$ADMIN` | | lock\_timeout | Seconds a statement waits for a lock held by another transaction before failing. `0` = do not wait, `-1` = wait indefinitely | Optional (default `10`) | `30` | | threads | Parallel model executions. `1` is recommended | Optional (default `1`) | `1` | | database | Logical label used internally by dbt. Never rendered in SQL. Defaults to the file base name | Optional | `mydb` | | schema | Logical label used internally by dbt. Firebird has no schemas, so this is never rendered in SQL. Defaults to `main` | Optional | `main` | <br /> `dbt init` also works: it lists `firebird`, asks for every field above, writes the profile, and runs `dbt debug` at the end. #### Character sets Pick the `charset` value that matches how your Firebird database stores text: | Situation | `charset` | | -------------------------------------------------------------------------------------------- | ---------------- | | Database created with `UTF8`, or plain ASCII data | `UTF8` (default) | | Legacy/restored databases with character set `NONE` and accented data (for example, WIN1252) | `WIN1252` | <br /> If reads fail with `UnicodeDecodeError: 'utf-8' codec can't decode byte ...`, set `charset: WIN1252`. This is the most common case for legacy Brazilian Firebird databases. #### Object naming and case sensitivity Unquoted identifiers in Firebird SQL are stored as UPPERCASE. The dbt adapter creates its objects with quoted, lowercase names (the same convention as the Postgres adapter), so a model named `clientes` becomes the table `"clientes"`: ```sql select * from "clientes"; -- correct (quoted) select * from clientes; -- looks for CLIENTES, which does not exist ``` To expose objects in the unquoted/uppercase style, set a model or seed alias in uppercase: ```yaml seeds: my_project: store: +alias: STORE ``` #### Supported features * Materializations: `table` (atomic swap), `view` (`CREATE OR ALTER`), `incremental` (strategies `append`, `delete+insert`, `merge`), `seed` (atomic data replacement), and `snapshot` (strategies `check` and `timestamp`) * `on_schema_change` handling (`ignore`, `fail`, `append_new_columns`, `sync_all_columns`) * Data tests, generic and singular * `dbt docs generate` and `persist_docs` (writes `COMMENT ON` statements) * Automatic retries for lock conflicts and a configurable `lock_timeout` ##### Limitations * The `ephemeral` materialization is not supported: dbt generates inline CTEs with names starting with an underscore, which are invalid as unquoted Firebird identifiers * Firebird has no schemas: the `database` and `schema` profile fields are logical labels only * Model and seed names should be at most 22 characters on Firebird 3 (dbt appends suffixes like `__dbt_tmp` to the 31-byte identifier limit; Firebird 4+ allows 63) * Model contracts (`contract: enforced`) are best effort: declare Firebird-native data types in your YAML * Models without a `FROM` clause are invalid in Firebird SQL (for example, use `select 1 as id from rdb$database`) * `threads: 1` is recommended; the adapter is resilient to write conflicts, but Firebird applies table-level coordination across parallel writers in the same namespace #### Database privileges The profile user needs read and write access to the objects dbt manages, plus permission to create and alter tables and views. These grants cover all dbt work: tables, views, seeds, snapshots, and incremental models: | Privilege | Purpose | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SELECT` | Read tables and views | | `INSERT` | Load models, [seeds](https://docs.getdbt.com/docs/build/seeds.md), and [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) | | `UPDATE` | Change existing rows ([incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) and [snapshots](https://docs.getdbt.com/docs/build/snapshots.md)) | | `DELETE` | Replace data (table swap and [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md)) | | `CREATE` | Create tables and views | | `ALTER` | Change tables and views | | `DROP` | Replace objects dbt rebuilds | <br /> For SYSDBA or the database owner, no extra grants are needed. --- ##### Connect Firebolt to dbt v1 Local development Some core functionality may be limited. If you're interested in contributing, check out the source code for the repository listed below. * **Maintained by**: Firebolt * **Authors**: Firebolt * **GitHub repo**: [firebolt-db/dbt-firebolt](https://github.com/firebolt-db/dbt-firebolt) [![](https://img.shields.io/github/stars/firebolt-db/dbt-firebolt?style=for-the-badge)](https://github.com/firebolt-db/dbt-firebolt) * **PyPI package**: `dbt-firebolt` [![](https://badge.fury.io/py/dbt-firebolt.svg)](https://badge.fury.io/py/dbt-firebolt) * **Slack channel**: [#db-firebolt](https://getdbt.slack.com/archives/C03K2PTHHTP) * **Supported dbt version**: v1.1.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-firebolt Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-firebolt` #### Configuring dbt-firebolt For Firebolt-specific configuration, please refer to [Firebolt configs.](https://docs.getdbt.com/reference/resource-configs/firebolt-configs.md) For other information including Firebolt feature support, see the [GitHub README](https://github.com/firebolt-db/dbt-firebolt/blob/main/README.md) and the [changelog](https://github.com/firebolt-db/dbt-firebolt/blob/main/CHANGELOG.md). #### Connecting to Firebolt To connect to Firebolt from dbt, you'll need to add a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) to your `profiles.yml` file. A Firebolt profile conforms to the following syntax: profiles.yml ```yml <profile-name>: target: <target-name> outputs: <target-name>: type: firebolt client_id: "<id>" client_secret: "<secret>" database: "<database-name>" engine_name: "<engine-name>" account_name: "<account-name>" schema: <tablename-prefix> threads: 1 #optional fields host: "<hostname>" ``` ###### Description of Firebolt Profile Fields To specify values as environment variables, use the format `{{ env_var('<variable_name>' }}`. For example, `{{ env_var('DATABASE_NAME' }}`. | Field | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | This must be included either in `profiles.yml` or in the `dbt_project.yml` file. Must be set to `firebolt`. | | `client_id` | Required. Your [service account](https://docs.firebolt.io/godocs/Guides/managing-your-organization/service-accounts.html) id. | | `client_secret` | Required. The secret associated with the specified `client_id`. | | `database` | Required. The name of the Firebolt database to connect to. | | `engine_name` | Required. The name (not the URL) of the Firebolt engine to use in the specified `database`. This must be a general purpose read-write engine and the engine must be running. If omitted in earlier versions, the default engine for the specified `database` is used. | | `account_name` | Required. Specifies the account name under which the specified `database` exists. | | `schema` | Recommended. A string to add as a prefix to the names of generated tables when using the [custom schemas workaround](https://docs.getdbt.com/docs/local/connect-data-platform/firebolt-setup.md#supporting-concurrent-development). | | `threads` | Required. Set to higher number to improve performance. | | `host` | Optional. The host name of the connection. For all customers it is `api.app.firebolt.io`, which will be used if omitted. | ###### Troubleshooting Connections If you encounter issues connecting to Firebolt from dbt, make sure the following criteria are met: * You must have adequate permissions to access the engine and the database. * Your service account must be attached to a user. * The engine must be running. #### Supporting Concurrent Development In dbt, database schemas are used to compartmentalize developer environments so that concurrent development does not cause table name collisions. Firebolt, however, does not currently support database schemas (it is on the roadmap). To work around this, we recommend that you add the following macro to your project. This macro will take the `schema` field of your `profiles.yml` file and use it as a table name prefix. ```sql -- macros/generate_alias_name.sql {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {%- if custom_alias_name is none -%} {{ node.schema }}__{{ node.name }} {%- else -%} {{ node.schema }}__{{ custom_alias_name | trim }} {%- endif -%} {%- endmacro %} ``` For an example of how this works, let’s say Shahar and Eric are both working on the same project. In her `.dbt/profiles.yml`, Sharar sets `schema=sh`, whereas Eric sets `schema=er` in his. When each runs the `customers` model, the models will land in the database as tables named `sh_customers` and `er_customers`, respectively. When running dbt in production, you would use yet another `profiles.yml` with a string of your choice. --- ##### Connect Greenplum to dbt v1 Local development * **Maintained by**: Community * **Authors**: Mark Poroshin, Dmitry Bevz * **GitHub repo**: [markporoshin/dbt-greenplum](https://github.com/markporoshin/dbt-greenplum) [![](https://img.shields.io/github/stars/markporoshin/dbt-greenplum?style=for-the-badge)](https://github.com/markporoshin/dbt-greenplum) * **PyPI package**: `dbt-greenplum` [![](https://badge.fury.io/py/dbt-greenplum.svg)](https://badge.fury.io/py/dbt-greenplum) * **Slack channel**: [n/a](https://www.getdbt.com/community) * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Greenplum 6.0 #### Installing dbt-greenplum Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-greenplum` #### Configuring dbt-greenplum For Greenplum-specific configuration, please refer to [Greenplum configs.](https://docs.getdbt.com/reference/resource-configs/greenplum-configs.md) For further (and more likely up-to-date) info, see the [README](https://github.com/markporoshin/dbt-greenplum#README.md) #### Profile Configuration Greenplum targets should be set up using the following configuration in your `profiles.yml` file. \~/.dbt/profiles.yml ```yaml company-name: target: dev outputs: dev: type: greenplum host: [hostname] user: [username] password: [password] port: [port] dbname: [database name] schema: [dbt schema] threads: [1 or more] keepalives_idle: 0 # default 0, indicating the system default. See below connect_timeout: 10 # default 10 seconds search_path: [optional, override the default postgres search_path] role: [optional, set the role dbt assumes when executing queries] sslmode: [optional, set the sslmode used to connect to the database] ``` ##### Notes This adapter strongly depends on dbt-postgres, so you can read more about configurations here [Profile Setup](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md) --- ##### Connect Hologres to dbt v1 Local development * **Maintained by**: Alibaba Cloud Hologres Team * **Authors**: Alibaba Cloud Hologres Team * **GitHub repo**: [aliyun/dbt-hologres](https://github.com/aliyun/dbt-hologres) [![](https://img.shields.io/github/stars/aliyun/dbt-hologres?style=for-the-badge)](https://github.com/aliyun/dbt-hologres) * **PyPI package**: `dbt-alibaba-cloud-hologres` [![](https://badge.fury.io/py/dbt-alibaba-cloud-hologres.svg)](https://badge.fury.io/py/dbt-alibaba-cloud-hologres) * **Slack channel**:[]() * **Supported dbt version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: #### Installing dbt-alibaba-cloud-hologres Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-alibaba-cloud-hologres` #### Configuring dbt-alibaba-cloud-hologres For Hologres-specific configuration, please refer to [Hologres configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) #### Connecting to Hologres with **dbt-alibaba-cloud-hologres** `dbt-alibaba-cloud-hologres` enables dbt to work with Alibaba Cloud Hologres, a real-time data warehouse compatible with PostgreSQL. Check out the dbt profile configuration below for details. \~/.dbt/profiles.yml ```yaml dbt-alibaba-cloud-hologres: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: hologres host: HOST_NAME port: 80 user: USER_NAME password: PASSWORD database: DATABASE_NAME schema: SCHEMA_NAME threads: 4 ``` ##### Connection parameters Currently it supports the following parameters: | **Field** | **Description** | Required? | **Default** | **Example** | | ------------------ | ---------------------------------------------------------------------------------------------- | --------- | ------------------------ | --------------------------------- | | `type` | Specifies the type of database connection; must be set to "hologres" for Hologres connections. | Required | - | `hologres` | | `host` | The endpoint hostname for connecting to Hologres instance. | Required | - | `hgxxx-xxx.hologres.aliyuncs.com` | | `port` | Port number for Hologres connection. | Optional | `80` | `80` | | `user` | The username for authentication with Hologres (case-sensitive). | Required | - | `AccessKey ID` | | `password` | The password for authentication with Hologres (case-sensitive). | Required | - | `AccessKey Secret` | | `database` | The name of your Hologres database. | Required | - | `my_database` | | `schema` | The default schema that the models will use in Hologres (use empty string "" if not needed). | Required | - | `public` | | `threads` | Number of threads for parallel execution. | Optional | `1` | `4` | | `connect_timeout` | Connection timeout in seconds. | Optional | `10` | `10` | | `sslmode` | SSL mode for the connection. | Optional | `disable` | `disable` | | `application_name` | Application identifier for connection tracking. | Optional | `dbt_hologres_{version}` | `my_dbt_app` | | `retries` | Number of connection retries. | Optional | `1` | `3` | #### Authentication configuration `dbt-alibaba-cloud-hologres` uses the standard PostgreSQL-compatible authentication mechanism with username and password (Access Key). Hologres supports using Alibaba Cloud AccessKey or RAM user credentials for authentication. ##### Access key You can authenticate using your Alibaba Cloud account credentials. For security reasons, it is recommended to create a RAM sub-account with appropriate permissions rather than using the primary account AccessKey. ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: hologres host: hgxxx-cn-shanghai.hologres.aliyuncs.com # Replace with your Hologres endpoint port: 80 user: your_access_key_id # Replace with your AccessKeyId password: your_access_key_secret # Replace with your AccessKeySecret database: my_database # Replace with your database name schema: public # Replace with your schema name threads: 4 connect_timeout: 10 sslmode: disable ``` ##### Important notes 1. **Case sensitivity**: Hologres usernames and passwords are case-sensitive. Make sure to enter them exactly as configured. 2. **Default port**: The default port for Hologres is `80`, which is different from the standard PostgreSQL port `5432`. 3. **SSL mode**: SSL is disabled by default for Hologres connections. You can enable it by setting `sslmode` to an appropriate value if required. #### Testing your connection After configuring your `profiles.yml`, you can verify your connection by running: ```bash dbt debug ``` This [command](https://docs.getdbt.com/reference/commands/debug.md) will test the connection to your Hologres instance and report any configuration issues. #### Hologres-specific features ##### Dynamic tables in Hologres Dynamic tables are Hologres's implementation of materialized views with automatic refresh. When refreshing data, multiple modes are supported, including "full" (full mode) and "incremental" (incremental mode). For more information, please refer to the [reference manual](https://www.alibabacloud.com/help/en/hologres/user-guide/introduction-to-dynamic-table). You can configure them in your dbt models: ```yaml models: my_model: materialized: dynamic_table freshness: "30 minutes" auto_refresh_mode: auto computing_resource: serverless ``` Supported configurations for Dynamic tables: | **Configuration** | **Description** | **Example values** | | -------------------- | ----------------------------------------- | ------------------------------------- | | `freshness` | Data freshness requirement. | `"30 minutes"`, `"1 hours"` | | `auto_refresh_mode` | Refresh mode for the dynamic table. | `auto`, `incremental`, `full` | | `computing_resource` | Computing resource to use for refreshing. | `serverless`, `local`, warehouse name | ##### Incremental models with dbt `dbt-alibaba-cloud-hologres` supports multiple incremental strategies: * `append`: Simply append new records * `delete+insert`: Delete matching records and insert new ones * `merge`: Use MERGE statement for upsert operations * `microbatch`: Process data in small batches ##### Constraints Full support for database constraints including: * Primary keys * Not null constraints ##### Table properties Hologres supports the following table properties. For full details, see the [developer reference documentation](https://www.alibabacloud.com/help/en/hologres/developer-reference/create-tables). | Property | Best practices | | ----------------------------- | ---------------------------------------------------------------------------- | | `orientation` | Use `column` for olap workloads and `row` for key-value queries | | `distribution_key` | Choose frequently joined or grouped columns; prefer a single column | | `clustering_key` | Use for range filter columns; max 3 columns; follow the left-match principle | | `event_time_column` | Set for time-series data (timestamp columns) | | `bitmap_columns` | Use for equality filters | | `dictionary_encoding_columns` | Use for low-cardinality string columns | #### References * [dbt-alibaba-cloud-hologres GitHub repository](https://github.com/aliyun/dbt-hologres) * [Hologres documentation](https://www.alibabacloud.com/help/en/hologres/) * [Hologres dynamic table guide](https://www.alibabacloud.com/help/en/hologres/user-guide/introduction-to-dynamic-table) --- ##### Connect IBM Db2 to dbt v1 The `ibm-dbt-db2` adapter allows you to use dbt to transform and manage data on IBM Db2, leveraging its robust SQL capabilities across LUW (Linux, Unix, Windows), z/OS, and iSeries platforms. Before proceeding, ensure you have the following: * An active IBM Db2 database instance (LUW, z/OS, or iSeries) with connection details (host, port, database, schema, etc). * Authentication credentials: Username and password. * Python 3.10, 3.11, or 3.12 (Python 3.9 not supported due to dbt-core 1.11+ requirements). Note: This adapter is not compatible with dbt v2. Refer to [Configuring ibm-dbt-db2](https://github.com/IBM/db2-dbt) to learn more about obtaining and organizing these details. * **Maintained by**: IBM * **Authors**: Shubham Kapoor, Amit Kumar * **GitHub repo**: [IBM/db2-dbt](https://github.com/IBM/db2-dbt) [![](https://img.shields.io/github/stars/IBM/db2-dbt?style=for-the-badge)](https://github.com/IBM/db2-dbt) * **PyPI package**: `ibm-dbt-db2` [![](https://badge.fury.io/py/ibm-dbt-db2.svg)](https://badge.fury.io/py/ibm-dbt-db2) * **Slack channel**:[]() * **Supported dbt version**: v1.11.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: IBM Db2 LUW 9.7+, z/OS 11+, iSeries #### Installing ibm-dbt-db2 Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install ibm-dbt-db2` #### Configuring ibm-dbt-db2 For IBM Db2-specific configuration, please refer to [IBM Db2 configs.](https://docs.getdbt.com/reference/resource-configs/ibm-db2-config.md) #### Connecting to IBM Db2 To connect dbt with IBM Db2, you need to configure a profile in your `profiles.yml` file located in the `.dbt/` directory of your home folder. The following is an example configuration for connecting to IBM Db2 instances: \~/.dbt/profiles.yml ```yaml my_db2_project: outputs: dev: type: db2 host: [hostname] port: 50000 database: [database name] schema: [schema name] username: [username] password: [password] threads: [1 or more] target: dev ``` ##### Connection with SSL/TLS For secure connections, you can configure SSL/TLS parameters: \~/.dbt/profiles.yml ```yaml my_db2_project: outputs: prod: type: db2 host: [hostname] port: 50001 database: [database name] schema: [schema name] username: [username] password: [password] threads: [1 or more] # SSL/TLS settings security: SSL ssl_server_certificate: /path/to/server-ca.crt ssl_client_hostname_validation: true retries: 3 target: prod ``` ##### Connection using DSN Alternatively, you can use a Data Source Name (DSN) configured in your Db2 client: \~/.dbt/profiles.yml ```yaml my_db2_project: outputs: dev: type: db2 dsn: MY_DB2_DSN schema: [schema name] threads: [1 or more] target: dev ``` #### Host parameters Use the following required and optional profile fields to configure IBM Db2 connections. | Option | Required/Optional | Description | Example | | ---------- | ----------------- | ----------------------------------------------------- | ------------ | | `type` | Required | The adapter type. Must be `db2`. | `db2` | | `host` | Required\* | Hostname for connecting to Db2. | `localhost` | | `port` | Optional | The port for connecting to Db2. | `50000` | | `database` | Required | The database name in your Db2 instance. | `SAMPLE` | | `schema` | Required | The schema name within your Db2 instance database. | `ADMIN` | | `username` | Required | Username for authentication. | `db2user` | | `password` | Required | Password for authentication. | `password` | | `threads` | Optional | Number of threads for parallel execution. | `8` | | `dsn` | Optional | Data Source Name (alternative to host/port/database). | `MY_DB2_DSN` | \*Not required if using `dsn` ##### Optional SSL/TLS Parameters | Option | Required/Optional | Description | Example | | -------------------------------- | ----------------- | ------------------------------------------------------------ | ------------------------ | | `security` | Optional | Security protocol. Use `SSL` to enable SSL/TLS. | `SSL` | | `ssl_server_certificate` | Optional | Path to server CA certificate file for SSL verification. | `/path/to/server-ca.crt` | | `ssl_client_keystore` | Optional | Path to client keystore database (.kdb file) for mutual TLS. | `/path/to/client.kdb` | | `ssl_client_keystash` | Optional | Path to client keystash file (.sth file). | `/path/to/client.sth` | | `ssl_client_hostname_validation` | Optional | Enable hostname verification (true/false). | `true` | | `retries` | Optional | Number of connection retry attempts. | `3` | #### Schemas and databases When selecting the database and the schema, make sure you have read and write access to both. This selection does not limit your ability to query the database. Instead, they serve as the default location for where tables and views are materialized. #### Supported features | Feature | Supported | Notes | | --------------------------- | ---------- | ----------------------------------------------- | | Table materialization | ✅ Yes | | | View materialization | ✅ Yes | | | Incremental materialization | ✅ Yes | Supports `merge` and `delete+insert` strategies | | Ephemeral materialization | ✅ Yes | | | Seeds | ✅ Yes | Requires explicit data type configuration | | Snapshots | ✅ Yes | | | Tests | ✅ Yes | | | Documentation | ✅ Yes | | | Sources | ✅ Yes | | | Custom schemas | ✅ Yes | | | Grants | ✅ Yes | | | Constraints | ⚠️ Partial | NOT NULL enforced, others not enforced | #### Data types The adapter maps dbt data types to Db2 types: | dbt Type | Db2 Type | | --------- | -------------------- | | string | VARCHAR | | text | VARCHAR(max\_length) | | integer | INTEGER | | bigint | BIGINT | | float | FLOAT | | numeric | DECIMAL | | boolean | BOOLEAN | | timestamp | TIMESTAMP | | date | DATE | | time | TIME | #### Incremental models The `ibm-dbt-db2` adapter supports two incremental strategies: 1. **merge** (default) - Uses Db2's MERGE statement for efficient upserts 2. **delete+insert** - Deletes matching records then inserts new ones Example incremental model: ```sql {{ config( materialized='incremental', unique_key='id', incremental_strategy='merge' ) }} SELECT * FROM source_table {% if is_incremental() %} WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }}) {% endif %} ``` #### Case sensitivity Db2 uppercases unquoted identifiers by default. The adapter handles this automatically, but be aware: * Unquoted table/column names will be uppercased * Use quotes in your SQL to preserve case: `"MyTable"` vs `MYTABLE` #### Notes * The `ibm-dbt-db2` adapter is built on the `ibm_db` Python driver (version 3.2.8) which is automatically installed with the adapter. * **Python Version Requirements**: Requires Python 3.10, 3.11, or 3.12. Python 3.9 is not supported due to dbt v1.11+ dependency requirements. Python 3.13+ has not been tested yet. * **Constraints**: CHECK, UNIQUE, PRIMARY KEY, and FOREIGN KEY constraints are defined but not enforced by Db2 in the dbt context. Only NOT NULL constraints are enforced. * **LISTAGG Limitation**: Db2's LISTAGG function does not support limiting the number of aggregated values. --- ##### Connect IBM Netezza to dbt v1 Local development The dbt-ibm-netezza adapter allows you to use dbt to transform and manage data on IBM Netezza, leveraging its distributed SQL query engine capabilities. Before proceeding, ensure you have the following: * An active IBM Netezza engine with connection details (host, port, database, schema, etc) in SaaS/PaaS. * Authentication credentials: Username and password. Refer to [Configuring dbt-ibm-netezza](https://github.com/IBM/nz-dbt?tab=readme-ov-file#testing-sample-dbt-project) for guidance on obtaining and organizing these details. * **Maintained by**: IBM * **Authors**: Abhishek Jog, Sagar Soni, Ayush Mehrotra * **GitHub repo**: [IBM/nz-dbt](https://github.com/IBM/nz-dbt) [![](https://img.shields.io/github/stars/IBM/nz-dbt?style=for-the-badge)](https://github.com/IBM/nz-dbt) * **PyPI package**: `dbt-ibm-netezza` [![](https://badge.fury.io/py/dbt-ibm-netezza.svg)](https://badge.fury.io/py/dbt-ibm-netezza) * **Slack channel**:[]() * **Supported dbt version**: v1.9.2 and newer * **dbt support**: Not Supported * **Minimum data platform version**: 11.2.3.4 #### Installing dbt-ibm-netezza Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-ibm-netezza` #### Configuring dbt-ibm-netezza For IBM Netezza-specific configuration, please refer to [IBM Netezza configs.](https://docs.getdbt.com/reference/resource-configs/ibm-netezza-config.md) #### Connecting to IBM Netezza To connect dbt with IBM Netezza, you need to configure a profile in your `profiles.yml` file located in the `.dbt/` directory of your home folder. The following is an example configuration for connecting to IBM Netezza instances: \~/.dbt/profiles.yml ```yaml my_project: outputs: dev: type: netezza user: [user] password: [password] host: [hostname] database: [catalog name] schema: [schema name] port: 5480 threads: [1 or more] target: dev ``` ##### Setup external table options You also need to configure the `et_options.yml` file located in your project directory. Make sure the file is correctly setup before running the `dbt seed`. This ensures that data is inserted into your tables accurately as specified in the external data file. ./et\_options.yml ```yaml - !ETOptions SkipRows: "1" Delimiter: "','" DateDelim: "'-'" MaxErrors: " 0 " ``` Refer the [Netezza external tables option summary](https://www.ibm.com/docs/en/netezza?topic=eto-option-summary) for more options in the file. #### Host parameters The following profile fields are required to configure IBM Netezza connections. | Option | Required/Optional | Description | Example | | ---------- | ----------------- | ----------------------------------------------------- | ----------- | | `user` | Required | Username or email address for authentication. | `user` | | `password` | Required | Password or API key for authentication | `password` | | `host` | Required | Hostname for connecting to Netezza. | `127.0.0.1` | | `database` | Required | The catalog name in your Netezza instance. | `SYSTEM` | | `schema` | Required | The schema name within your Netezza instance catalog. | `my_schema` | | `port` | Required | The port for connecting to Netezza. | `5480` | ##### Schemas and databases When selecting the database and the schema, make sure the user has read and write access to both. This selection does not limit your ability to query the database. Instead, they serve as the default location for where tables and views are materialized. #### Notes: The `dbt-ibm-netezza` adapter is built on the IBM Netezza Python driver - [nzpy](https://pypi.org/project/nzpy/) and is a pre-requisite which gets installed along with the adapter. --- ##### Connect IBM watsonx.data Presto to dbt v1 Local development The dbt-watsonx-presto adapter allows you to use dbt to transform and manage data on IBM watsonx.data Presto(Java), leveraging its distributed SQL query engine capabilities. Before proceeding, ensure you have the following: * An active IBM watsonx.data Presto(Java) engine with connection details (host, port, catalog, schema) in SaaS/Software. * Authentication credentials: Username and password/apikey. * For watsonx.data instances, SSL verification is required for secure connections. If the instance host uses HTTPS, there is no need to specify the SSL certificate parameter. However, if the instance host uses an unsecured HTTP connection, ensure you provide the path to the SSL certificate file. Refer to [Configuring dbt-watsonx-presto](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=presto-configuration-setting-up-your-profile) for guidance on obtaining and organizing these details. * **Maintained by**: IBM * **Authors**: Karnati Naga Vivek, Hariharan Ashokan, Biju Palliyath, Gopikrishnan Varadarajulu, Rohan Pednekar * **GitHub repo**: [IBM/dbt-watsonx-presto](https://github.com/IBM/dbt-watsonx-presto) [![](https://img.shields.io/github/stars/IBM/dbt-watsonx-presto?style=for-the-badge)](https://github.com/IBM/dbt-watsonx-presto) * **PyPI package**: `dbt-watsonx-presto` [![](https://badge.fury.io/py/dbt-watsonx-presto.svg)](https://badge.fury.io/py/dbt-watsonx-presto) * **Slack channel**: [#db-watsonx-presto](https://getdbt.slack.com/archives/C08C7D53R40) * **Supported dbt version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-watsonx-presto Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-watsonx-presto` #### Configuring dbt-watsonx-presto For IBM watsonx.data-specific configuration, please refer to [IBM watsonx.data configs.](https://docs.getdbt.com/reference/resource-configs/watsonx-presto-config.md) #### Connecting to IBM watsonx.data presto To connect dbt with watsonx.data Presto(java), you need to configure a profile in your `profiles.yml` file located in the `.dbt/` directory of your home folder. The following is an example configuration for connecting to IBM watsonx.data SaaS and Software instances: \~/.dbt/profiles.yml ```yaml my_project: outputs: software: type: watsonx_presto method: BasicAuth user: [user] password: [password] host: [hostname] catalog: [catalog_name] schema: [your dbt schema] port: [port number] threads: [1 or more] ssl_verify: path/to/certificate saas: type: watsonx_presto method: BasicAuth user: [user] password: [api_key] host: [hostname] catalog: [catalog_name] schema: [your dbt schema] port: [port number] threads: [1 or more] target: software ``` #### Host parameters The following profile fields are required to configure watsonx.data Presto(java) connections. For IBM watsonx.data SaaS or Software instances, you can get the `hostname` and `port` details by clicking **View connect details** on the Presto(java) engine details page. | Option | Required/Optional | Description | Example | | ------------ | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | `method` | Required | Specifies the authentication method for secure connections. Use `BasicAuth` when connecting to IBM watsonx.data SaaS or Software instances. | `BasicAuth` | | `user` | Required | Username or email address for authentication. | `user` | | `password` | Required | Password or API key for authentication | `password` | | `host` | Required | Hostname for connecting to Presto. | `127.0.0.1` | | `catalog` | Required | The catalog name in your Presto instance. | `Analytics` | | `schema` | Required | The schema name within your Presto instance catalog. | `my_schema` | | `port` | Required | The port for connecting to Presto. | `443` | | `ssl_verify` | Optional (default: **true**) | Specifies the path to the SSL certificate or a boolean value. The SSL certificate path is required if the watsonx.data instance is not secure (HTTP). | `path/to/certificate` or `true` | ##### Schemas and databases When selecting the catalog and the schema, make sure the user has read and write access to both. This selection does not limit your ability to query the catalog. Instead, they serve as the default location for where tables and views are materialized. In addition, the Presto connector used in the catalog must support creating tables. This default can be changed later from within your dbt project. ##### SSL verification * If the Presto instance uses an unsecured HTTP connection, you must set `ssl_verify` to the path of the SSL certificate file. * If the instance uses `HTTPS`, this parameter is not required and can be omitted. #### Additional parameters The following profile fields are optional to set up. They let you configure your instance session and dbt for your connection. | Profile field | Description | Example | | -------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------- | | `threads` | How many threads dbt should use (default is `1`) | `8` | | `http_headers` | HTTP headers to send alongside requests to Presto, specified as a yaml dictionary of (header, value) pairs. | `X-Presto-Routing-Group: my-instance` | | `http_scheme` | The HTTP scheme to use for requests to (default: `http`, or `https` if `BasicAuth`) | `https` or `http` | --- ##### Connect IBM watsonx.data Spark to dbt v1 Local development * **Maintained by**: IBM * **Authors**: Bayan Albunayan, Reema Alzaid, Manjot Sidhu * **GitHub repo**: [IBM/dbt-watsonx-spark](https://github.com/IBM/dbt-watsonx-spark) [![](https://img.shields.io/github/stars/IBM/dbt-watsonx-spark?style=for-the-badge)](https://github.com/IBM/dbt-watsonx-spark) * **PyPI package**: `dbt-watsonx-spark` [![](https://badge.fury.io/py/dbt-watsonx-spark.svg)](https://badge.fury.io/py/dbt-watsonx-spark) * **Slack channel**:[]() * **Supported dbt version**: v0.0.8 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-watsonx-spark Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-watsonx-spark` #### Configuring dbt-watsonx-spark For IBM watsonx.data-specific configuration, please refer to [IBM watsonx.data configs.](https://docs.getdbt.com/reference/resource-configs/watsonx-Spark-config) The `dbt-watsonx-spark` adapter allows you to use dbt to transform and manage data on IBM watsonx.data Spark, leveraging its distributed SQL query engine capabilities. Before proceeding, ensure you have the following: * An active IBM watsonx.data, For [IBM Cloud (SaaS)](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started). For [Software](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=installing-watsonxdata-developer-version) * Provision **Native Spark engine** in watsonx.data, For [IBM Cloud (SaaS)](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-prov_nspark). For [Software](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=spark-native-engine) * An active **Spark query server** in your **Native Spark engine** Read the official documentation for using **watsonx.data** with `dbt-watsonx-spark` * [Documentation for IBM Cloud and SaaS offerings](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-dbt_watsonx_spark_inst) * [Documentation for IBM watsonx.data software](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=integration-data-build-tool-adapter-spark) #### Installing dbt-watsonx-spark Note: Installing an adapter doesn't install 'dbt v1' automatically. This is because adapters and dbt v1 versions are decoupled to avoid overwriting dbt v1 installations. Use the following command for installation: ```sh python -m pip install dbt-core dbt-watsonx-spark ``` #### Configuring `dbt-watsonx-spark` For IBM watsonx.data-specific configuration, refer to [IBM watsonx.data configs.](https://docs.getdbt.com/reference/resource-configs/watsonx-spark-config.md) #### Connecting to IBM watsonx.data Spark To connect dbt with watsonx.data Spark, configure a profile in your `profiles.yml` file located in the `.dbt/` directory of your home folder. The following is an example configuration for connecting to IBM watsonx.data SaaS and Software instances: \~/.dbt/profiles.yml ```yaml project_name: target: "dev" outputs: dev: type: watsonx_spark method: http schema: [schema name] host: [hostname] uri: [uri] catalog: [catalog name] use_ssl: false auth: instance: [Watsonx.data Instance ID] user: [username] apikey: [apikey] ``` #### Host parameters The following profile fields are required to configure watsonx.data Spark connections. For IBM watsonx.data SaaS or Software instances, To get the 'profile' details, click 'View connect details' when the 'query server' is in RUNNING status in watsonx.data (In watsonx.data (both SaaS or Software). The Connection details page opens with the profile configuration. Copy and paste the connection details in the profiles.yml file that is located in .dbt of your home directory The following profile fields are required to configure watsonx.data Spark connections: | Option | Required/Optional | Description | Example | | ---------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `method` | Required | Specifies the connection method to the spark query server. Use `http`. | `http` | | `schema` | Required | To choose an existing schema within spark engine or create a new schema. | `spark_schema` | | `host` | Required | Hostname of the watsonx.data console. For more information, see [Getting connection information](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=references-getting-connection-information#connection_info__conn_info_). | `https://dataplatform.cloud.ibm.com` | | `uri` | Required | URI of your query server that is running on watsonx.data. For more information, see [Getting connection information](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=references-getting-connection-information#connection_info__conn_info_). | `/lakehouse/api/v2/spark_engines/<sparkID>/query_servers/<queryID>/connect/cliservice` | | `catalog` | Required | The catalog that is associated with the Spark engine. | `my_catalog` | | `use_ssl` | Optional (default: **false**) | Specifies whether to use SSL. | `true` or `false` | | `instance` | Required | For **SaaS** set it as CRN of watsonx.data. As for **Software**, set it as instance ID of watsonx.data | `1726574045872688` | | `user` | Required | Username for the watsonx.data instance. for \[Saas] use email as username | `username` or `user@example.com` | | `apikey` | Required | Your API key. For more info on [SaaS](https://www.ibm.com/docs/en/software-hub/5.1.x?topic=started-generating-api-keys), For [Software](https://cloud.ibm.com/docs/account?topic=account-userapikey\&interface=ui#manage-user-keys) | `API key` | ##### Schemas and catalogs When selecting the catalog, ensure the user has read and write access. This selection does not limit your ability to query into the schema specified/created but also serves as the default location for materialized `tables`, `views`, and `incremental`. ##### SSL verification * If the Spark instance uses an unsecured HTTP connection, set `use_ssl` to `false`. * If the instance uses `HTTPS`, set it `true`. #### Additional parameters The following profile fields are optional. You can configure the instance session and dbt for the connection. | Profile field | Description | Example | | ----------------- | ------------------------------------------------------------ | ------- | | `threads` | How many threads dbt should use (default is `1`) | `8` | | `retry_all` | Enables automatic retries for transient connection failures. | `true` | | `connect_timeout` | Timeout for establishing a connection (in seconds). | `5` | | `connect_retries` | Number of retry attempts for connection failures. | `3` | #### Limitations and considerations * **Supports only HTTP**: No support for ODBC, Thrift, or session-based connections. * **Limited dbt Support**: Not fully compatible with dbt. * **Metadata Persistence**: Some dbt features, such as column descriptions, may not persist in all table formats. --- ##### Connect Infer to dbt v1 Local development Vendor-supported plugin Certain core functionality may vary. If you would like to report a bug, request a feature, or contribute, you can check out the linked repository and open an issue. * **Maintained by**: Infer * **Authors**: Erik Mathiesen-Dreyfus, Ryan Garland * **GitHub repo**: [inferlabs/dbt-infer](https://github.com/inferlabs/dbt-infer) [![](https://img.shields.io/github/stars/inferlabs/dbt-infer?style=for-the-badge)](https://github.com/inferlabs/dbt-infer) * **PyPI package**: `dbt-infer` [![](https://badge.fury.io/py/dbt-infer.svg)](https://badge.fury.io/py/dbt-infer) * **Slack channel**: [n/a]() * **Supported dbt version**: v1.2.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-infer Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-infer` #### Configuring dbt-infer For Infer-specific configuration, please refer to [Infer configs.](https://docs.getdbt.com/reference/resource-configs/infer-configs.md) #### Connecting to Infer with **dbt-infer** Infer allows you to perform advanced ML Analytics within SQL as if native to your data warehouse. To do this Infer uses a variant called SQL-inf, which defines as set of primitive ML commands from which you can build advanced analysis for any business use case. Read more about SQL-inf and Infer in the [Infer documentation](https://docs.getinfer.io/). The `dbt-infer` package allow you to use SQL-inf easily within your dbt models. You can read more about the `dbt-infer` package itself and how it connects to Infer in the [dbt-infer documentation](https://dbt.getinfer.io/). The dbt-infer adapter is maintained via PyPi and installed with pip. To install the latest dbt-infer package simply run the following within the same shell as you run dbt. ```python pip install dbt-infer ``` Versioning of dbt-infer follows the standard dbt versioning scheme - meaning if you are using dbt 1.2 the corresponding dbt-infer will be named 1.2.x where is the latest minor version number. Before using SQL-inf in your dbt models you need to setup an Infer account and generate an API-key for the connection. You can read how to do that in the [Getting Started Guide](https://docs.getinfer.io/docs/reference/integrations/dbt). The profile configuration in `profiles.yml` for `dbt-infer` should look something like this: \~/.dbt/profiles.yml ```yaml <profile-name>: target: <target-name> outputs: <target-name>: type: infer url: "<infer-api-endpoint>" username: "<infer-api-username>" apikey: "<infer-apikey>" data_config: [configuration for your underlying data warehouse] ``` Note that you need to also have installed the adapter package for your underlying data warehouse. For example, if your data warehouse is BigQuery then you need to also have installed the appropriate `dbt-bigquery` package. The configuration of this goes into the `data_config` field. ##### Description of Infer Profile Fields | Field | Required | Description | | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Yes | Must be set to `infer`. This must be included either in `profiles.yml` or in the `dbt_project.yml` file. | | `url` | Yes | The host name of the Infer server to connect to. Typically this is `https://app.getinfer.io`. | | `username` | Yes | Your Infer username - the one you use to login. | | `apikey` | Yes | Your Infer api key. | | `data_config` | Yes | The configuration for your underlying data warehouse. The format of this follows the format of the configuration for your data warehouse adapter. | ##### Example of Infer configuration To illustrate the above descriptions, here is an example of what a `dbt-infer` configuration might look like. In this case the underlying data warehouse is BigQuery, which we configure the adapter for inside the `data_config` field. ```yaml infer_bigquery: apikey: 1234567890abcdef username: my_name@example.com url: https://app.getinfer.io type: infer data_config: dataset: my_dataset job_execution_timeout_seconds: 300 job_retries: 1 keyfile: bq-user-creds.json location: EU method: service-account priority: interactive project: my-bigquery-project threads: 1 type: bigquery ``` #### Usage You do not need to change anything in your existing dbt models when switching to use SQL-inf – they will all work the same as before – but you now have the ability to use SQL-inf commands as native SQL functions. Infer supports a number of SQL-inf commands, including `PREDICT`, `EXPLAIN`, `CLUSTER`, `SIMILAR_TO`, `TOPICS`, `SENTIMENT`. You can read more about SQL-inf and the commands it supports in the [SQL-inf Reference Guide](https://docs.getinfer.io/docs/category/commands). To get you started we will give a brief example here of what such a model might look like. You can find other more complex examples in the [dbt-infer examples repo](https://github.com/inferlabs/dbt-infer-examples). In our simple example, we will show how to use a previous model 'user\_features' to predict churn by predicting the column `has_churned`. predict\_user\_churn.sql ```sql {{ config( materialized = "table" ) }} with predict_user_churn_input as ( select * from {{ ref('user_features') }} ) SELECT * FROM predict_user_churn_input PREDICT(has_churned, ignore=user_id) ``` Not that we ignore `user_id` from the prediction. This is because we think that the `user_id` might, and should, not influence our prediction of churn, so we remove it. We also use the convention of pulling together the inputs for our prediction in a CTE, named `predict_user_churn_input`. --- ##### Connect iomete to dbt v1 Local development * **Maintained by**: iomete * **Authors**: Namig Aliyev * **GitHub repo**: [iomete/dbt-iomete](https://github.com/iomete/dbt-iomete) [![](https://img.shields.io/github/stars/iomete/dbt-iomete?style=for-the-badge)](https://github.com/iomete/dbt-iomete) * **PyPI package**: `dbt-iomete` [![](https://badge.fury.io/py/dbt-iomete.svg)](https://badge.fury.io/py/dbt-iomete) * **Slack channel**: [##db-iomete](https://getdbt.slack.com/archives/C03JFG22EP9) * **Supported dbt version**: v0.18.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-iomete Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-iomete` #### Configuring dbt-iomete For iomete-specific configuration, please refer to [iomete configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) Set up a iomete Target iomete targets should be set up using the following configuration in your profiles.yml file. profiles.yml ```yaml iomete: target: dev outputs: dev: type: iomete cluster: cluster_name host: <region_name>.iomete.com port: 443 schema: database_name account_number: iomete_account_number user: iomete_user_name password: iomete_user_password ``` ###### Description of Profile Fields | Field | Description | Required | Example | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------------- | | type | The specific adapter to use | Required | `iomete` | | cluster | The cluster to connect | Required | `reporting` | | host | The host name of the connection. It is a combination of<br />`account_number` with the prefix `dwh-`<br />and the suffix `.iomete.com`. | Required | `dwh-12345.iomete.com` | | port | The port to use. | Required | `443` | | schema | Specify the schema (database) to build models into. | Required | `dbt_finance` | | account\_number | The iomete account number with single quotes. | Required | `'1234566789123'` | | username | The iomete username to use to connect to the server. | Required | `dbt_user` | | password | The iomete user password to use to connect to the server. | Required | `strong_password` | #### Supported Functionality Most dbt v1 functionality is supported. Iceberg specific improvements. 1. Joining the results of `show tables` and `show views`. --- ##### Connect Layer to dbt v1 Local development * **Maintained by**: Layer * **Authors**: Mehmet Ecevit * **GitHub repo**: [layerai/dbt-layer](https://github.com/layerai/dbt-layer) [![](https://img.shields.io/github/stars/layerai/dbt-layer?style=for-the-badge)](https://github.com/layerai/dbt-layer) * **PyPI package**: `dbt-layer-bigquery` [![](https://badge.fury.io/py/dbt-layer-bigquery.svg)](https://badge.fury.io/py/dbt-layer-bigquery) * **Slack channel**: [#tools-layer](https://getdbt.slack.com/archives/C03STA39TFE) * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-layer-bigquery Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-layer-bigquery` #### Configuring dbt-layer-bigquery For Layer-specific configuration, please refer to [Layer configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) ##### Profile Configuration Layer Bigquery targets should be set up using the following sections in your `profiles.yml` file. ###### Layer Authentication Add your `layer_api_key` to your `profiles.yaml` to authenticate with Layer. To get your Layer API Key: * First, [create your free Layer account](https://app.layer.ai/login?returnTo=%2Fgetting-started). * Go to [app.layer.ai](https://app.layer.ai) > **Settings** (Cog Icon by your profile photo) > **Developer** > **Create API key** to get your Layer API Key. ###### Bigquery Authentication You can use any [authentication method](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md) supported in the official dbt Bigquery adapter since Layer uses `dbt-bigquery` adapter to connect to your Bigquery instance. A sample profile: profiles.yml ```yaml layer-profile: target: dev outputs: dev: # Layer authentication type: layer_bigquery layer_api_key: [the API Key to access your Layer account (opt)] # Bigquery authentication method: service-account project: [GCP project id] dataset: [the name of your dbt dataset] threads: [1 or more] keyfile: [/path/to/bigquery/keyfile.json] ``` ###### Description of Layer Bigquery Profile Fields The following fields are required: | Parameter | Default | Type | Description | | --------------- | ------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | | string | Specifies the adapter you want to use. It should be `layer_bigquery`. | | `layer_api_key` | | string (opt) | Specifies your Layer API key. If you want to make predictions with public ML models from Layer, you don't need to have this key in your profile. It's required if you load ML models from your Layer account or train an AutoML model. | | `layer_project` | | string (opt) | Specifies your target Layer project. If you don't specify, Layer will use the project same name with your dbt project. | | `method` | | string | Specifies the authentication type to connect to your BigQuery. | Rest of the parameters depends on the BigQuery authentication method you specified. #### Usage ##### AutoML You can automatically build state-of-art ML models using your own dbt models with plain SQL. To train an AutoML model all you have to do is pass your model type, input data (features) and target column you want to predict to `layer.automl()` in your SQL. The Layer AutoML will pick the best performing model and enable you to call it by its dbt model name to make predictions as shown above. *Syntax:* ```text layer.automl("MODEL_TYPE", ARRAY[FEATURES], TARGET) ``` *Parameters:* | Syntax | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `MODEL_TYPE` | Type of the model your want to train. There are two options:<br />- `classifier`: A model to predict classes/labels or categories such as spam detection<br />- `regressor`: A model to predict continuous outcomes such as CLV prediction. | | `FEATURES` | Input column names as a list to train your AutoML model. | | `TARGET` | Target column that you want to predict. | *Requirements:* * You need to put `layer_api_key` to your dbt profile to make AutoML work. *Example:* Check out [Order Review AutoML Project](https://github.com/layerai/dbt-layer/tree/mecevit/update-docs/examples/order_review_prediction): ```sql SELECT order_id, layer.automl( -- This is a regression problem 'regressor', -- Data (input features) to train our model ARRAY[ days_between_purchase_and_delivery, order_approved_late, actual_delivery_vs_expectation_bucket, total_order_price, total_order_freight, is_multiItems_order,seller_shipped_late], -- Target column we want to predict review_score ) FROM {{ ref('training_data') }} ``` ##### Prediction You can make predictions using any Layer ML model within your dbt models. Layer dbt Adapter helps you score your data resides on your warehouse within your dbt DAG with SQL. *Syntax:* ```text layer.predict("LAYER_MODEL_PATH", ARRAY[FEATURES]) ``` *Parameters:* | Syntax | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `LAYER_MODEL_PATH` | This is the Layer model path in form of `/[organization_name]/[project_name]/models/[model_name]`. You can use only the model name if you want to use an AutoML model within the same dbt project. | | `FEATURES` | These are the columns that this model requires to make a prediction. You should pass the columns as a list like `ARRAY[column1, column2, column3]`. | *Example:* Check out [Cloth Detection Project](https://github.com/layerai/dbt-layer/tree/mecevit/update-docs/examples/cloth_detector): ```sql SELECT id, layer.predict("layer/clothing/models/objectdetection", ARRAY[image]) FROM {{ ref("products") }} ``` --- ##### Connect Materialize to dbt v1 Local development Vendor-supported plugin Certain core functionality may vary. If you would like to report a bug, request a feature, or contribute, you can check out the linked repository and open an issue. * **Maintained by**: Materialize Inc. * **Authors**: Materialize team * **GitHub repo**: [MaterializeInc/materialize](https://github.com/MaterializeInc/materialize) [![](https://img.shields.io/github/stars/MaterializeInc/materialize?style=for-the-badge)](https://github.com/MaterializeInc/materialize) * **PyPI package**: `dbt-materialize` [![](https://badge.fury.io/py/dbt-materialize.svg)](https://badge.fury.io/py/dbt-materialize) * **Slack channel**: [#db-materialize](https://getdbt.slack.com/archives/C01PWAH41A5) * **Supported dbt version**: v0.18.1 and newer * **dbt support**: Not Supported * **Minimum data platform version**: v0.28.0 #### Installing dbt-materialize Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-materialize` #### Configuring dbt-materialize For Materialize-specific configuration, please refer to [Materialize configs.](https://docs.getdbt.com/reference/resource-configs/materialize-configs.md) #### Connecting to Materialize Once you have set up a [Materialize account](https://materialize.com/register/), adapt your `profiles.yml` to connect to your instance using the following reference profile configuration: \~/.dbt/profiles.yml ```yaml materialize: target: dev outputs: dev: type: materialize host: [host] port: [port] user: [user@domain.com] pass: [password] dbname: [database] cluster: [cluster] # default 'default' schema: [dbt schema] sslmode: require keepalives_idle: 0 # default: 0, indicating the system default connect_timeout: 10 # default: 10 seconds retries: 1 # default: 1, retry on error/timeout when opening connections ``` ##### Configurations `cluster`: The default [cluster](https://materialize.com/docs/overview/key-concepts/#clusters) is used to maintain materialized views or indexes. A [`default` cluster](https://materialize.com/docs/sql/show-clusters/#default-cluster) is pre-installed in every environment, but we recommend creating dedicated clusters to isolate the workloads in your dbt project (for example, `staging` and `data_mart`). `keepalives_idle`: The number of seconds before sending a ping to keep the Materialize connection active. If you are encountering `SSL SYSCALL error: EOF detected`, you may want to lower the [keepalives\_idle](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md#keepalives_idle) value to prevent the database from closing its connection. To test the connection to Materialize, run: ```text dbt debug ``` If the output reads "All checks passed!", you’re good to go! Check the [dbt and Materialize guide](https://materialize.com/docs/guides/dbt/) to learn more and get started. #### Supported Features ##### Materializations Because Materialize is optimized for transformations on streaming data and the core of dbt is built around batch, the `dbt-materialize` adapter implements a few custom materialization types: | Type | Supported? | Details | | ------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `source` | YES | Creates a [source](https://materialize.com/docs/sql/create-source/). | | `view` | YES | Creates a [view](https://materialize.com/docs/sql/create-view/#main). | | `materializedview` | YES | Creates a [materialized view](https://materialize.com/docs/sql/create-materialized-view/#main). | | `table` | YES | Creates a [materialized view](https://materialize.com/docs/sql/create-materialized-view/#main). (Actual table support pending [#5266](https://github.com/MaterializeInc/materialize/issues/5266)) | | `sink` | YES | Creates a [sink](https://materialize.com/docs/sql/create-sink/#main). | | `ephemeral` | YES | Executes queries using CTEs. | | `incremental` | NO | Use the `materializedview` materialization instead. Materialized views will always return up-to-date results without manual or configured refreshes. For more information, check out [Materialize documentation](https://materialize.com/docs/). | ##### Indexes Materialized views (`materializedview`), views (`view`) and sources (`source`) may have a list of [`indexes`](https://docs.getdbt.com/reference/resource-configs/materialize-configs.md#indexes) defined. ##### Seeds Running [`dbt seed`](https://docs.getdbt.com/reference/commands/seed.md) will create a static materialized view from a CSV file. You will not be able to add to or update this view after it has been created. ##### Tests Running [`dbt test`](https://docs.getdbt.com/reference/commands/test.md) with the optional `--store-failures` flag or [`store_failures` config](https://docs.getdbt.com/reference/resource-configs/store_failures.md) will create a materialized view for each configured test that can keep track of failures over time. #### Resources * [dbt and Materialize guide](https://materialize.com/docs/guides/dbt/) --- ##### Connect MaxCompute to dbt v1 Local development * **Maintained by**: Alibaba Cloud MaxCompute Team * **Authors**: Alibaba Cloud MaxCompute Team * **GitHub repo**: [aliyun/dbt-maxcompute](https://github.com/aliyun/dbt-maxcompute) [![](https://img.shields.io/github/stars/aliyun/dbt-maxcompute?style=for-the-badge)](https://github.com/aliyun/dbt-maxcompute) * **PyPI package**: `dbt-maxcompute` [![](https://badge.fury.io/py/dbt-maxcompute.svg)](https://badge.fury.io/py/dbt-maxcompute) * **Slack channel**:[]() * **Supported dbt version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: #### Installing dbt-maxcompute Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-maxcompute` #### Configuring dbt-maxcompute For MaxCompute-specific configuration, please refer to [MaxCompute configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) #### Connecting to MaxCompute with **dbt-maxcompute** Check out the dbt profile configuration below for details. \~/.dbt/profiles.yml ```yaml dbt-maxcompute: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: PROJECT_ID schema: SCHEMA_NAME endpoint: ENDPOINT auth_type: access_key access_key_id: ACCESS_KEY_ID access_key_secret: ACCESS_KEY_SECRET ``` Currently it supports the following parameters: | **Field** | **Description** | Required? | **Example** | | ------------------- | -------------------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------ | | `type` | Specifies the type of database connection; must be set to "maxcompute" for MaxCompute connections. | Required | `maxcompute` | | `project` | The name of your MaxCompute project. | Required | `dbt-project` | | `endpoint` | The endpoint URL for connecting to MaxCompute. | Required | `http://service.cn-shanghai.maxcompute.aliyun.com/api` | | `schema` | The namespace schema that the models will use in MaxCompute. | Required | `default` | | `auth_type` | Authentication type for accessing MaxCompute | Required | `access_key` | | `access_key_id` | The Access ID for authentication with MaxCompute. | Required | `XXX` | | `access_key_secret` | The Access Key for authentication with MaxCompute. | Required | `XXX` | See the section below for other authentication type. #### Authentication Configuration `dbt-maxcompute` is a dbt adapter that allows you to seamlessly integrate with Alibaba Cloud's MaxCompute service, enabling you to build and manage your data transformations using dbt. To ensure secure and flexible access to MaxCompute, `dbt-maxcompute` leverages the [credentials-python](https://github.com/aliyun/credentials-python) library, which provides comprehensive support for various authentication methods supported by Alibaba Cloud. With `dbt-maxcompute`, you can utilize all the authentication mechanisms provided by `credentials-python`, ensuring that your credentials are managed securely and efficiently. Whether you're using Access Keys, STS Tokens, RAM Roles, or other advanced authentication methods, `dbt-maxcompute` has got you covered. ##### Key Notes on Configuration To avoid ambiguity in configuration options, some parameter names have been adjusted compared to those used in `credentials-python`. Specifically: * `type` becomes `auth_type` * `policy` becomes `auth_policy` * `host` becomes `auth_host` * `timeout` becomes `auth_timeout` * `connect_timeout` becomes `auth_connect_timeout` * `proxy` becomes `auth_proxy` These changes ensure clarity and consistency across different authentication methods while maintaining compatibility with the underlying `credentials-python` library. #### Usage Before you begin, you need to sign up for an Alibaba Cloud account and retrieve your [Credentials](https://usercenter.console.aliyun.com/#/manage/ak). ##### Credential Type ###### Access Key Setup access\_key credential through \[User Information Management]\[ak], it have full authority over the account, please keep it safe. Sometimes for security reasons, you cannot hand over a primary account AccessKey with full access to the developer of a project. You may create a sub-account \[RAM Sub-account]\[ram] , grant its \[authorization]\[permissions]，and use the AccessKey of RAM Sub-account. ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: access_key # credential type, Optional, default is 'access_key' access_key_id: accessKeyId # AccessKeyId access_key_secret: accessKeySecret # AccessKeySecret ``` ###### STS Create a temporary security credential by applying Temporary Security Credentials (TSC) through the Security Token Service (STS). ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: sts # credential type access_key_id: accessKeyId # AccessKeyId access_key_secret: accessKeySecret # AccessKeySecret security_token: securityToken # STS Token ``` ###### RAM Role ARN By specifying \[RAM Role]\[RAM Role], the credential will be able to automatically request maintenance of STS Token. If you want to limit the permissions(\[How to make a policy]\[policy]) of STS Token, you can assign value for `Policy`. ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: ram_role_arn # credential type access_key_id: accessKeyId # AccessKeyId access_key_secret: accessKeySecret # AccessKeySecret security_token: securityToken # STS Token role_arn: roleArn # Format: acs:ram::USER_ID:role/ROLE_NAME role_session_name: roleSessionName # Role Session Name auth_policy: policy # Not required, limit the permissions of STS Token role_session_expiration: 3600 # Not required, limit the Valid time of STS Token ``` ###### OIDC Role ARN By specifying \[OIDC Role]\[OIDC Role], the credential will be able to automatically request maintenance of STS Token. If you want to limit the permissions(\[How to make a policy]\[policy]) of STS Token, you can assign value for `Policy`. ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: oidc_role_arn # credential type access_key_id: accessKeyId # AccessKeyId access_key_secret: accessKeySecret # AccessKeySecret security_token: securityToken # STS Token role_arn: roleArn # Format: acs:ram::USER_ID:role/ROLE_NAME oidc_provider_arn: oidcProviderArn # Format: acs:ram::USER_Id:oidc-provider/OIDC Providers oidc_token_file_path: /Users/xxx/xxx # oidc_token_file_path can be replaced by setting environment variable: ALIBABA_CLOUD_OIDC_TOKEN_FILE role_session_name: roleSessionName # Role Session Name auth_policy: policy # Not required, limit the permissions of STS Token role_session_expiration: 3600 # Not required, limit the Valid time of STS Token ``` ###### ECS RAM Role Both ECS and ECI instances support binding instance RAM roles. When the Credentials tool is used in an instance, the RAM role bound to the instance will be automatically obtained, and the STS Token of the RAM role will be obtained by accessing the metadata service to complete the initialization of the credential client. The instance metadata server supports two access modes: hardened mode and normal mode. The Credentials tool uses hardened mode (IMDSv2) by default to obtain access credentials. If an exception occurs when using hardened mode, you can set disable\_imds\_v1 to perform different exception handling logic: * When the value is false (default value), the normal mode will continue to be used to obtain access credentials. * When the value is true, it means that only hardened mode can be used to obtain access credentials, and an exception will be thrown. Whether the server supports IMDSv2 depends on your configuration on the server. ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: ecs_ram_role # credential type role_name: roleName # `role_name` is optional. It will be retrieved automatically if not set. It is highly recommended to set it up to reduce requests. disable_imds_v1: True # Optional, whether to forcibly disable IMDSv1, that is, to use IMDSv2 hardening mode, which can be set by the environment variable ALIBABA_CLOUD_IMDSV1_DISABLED ``` ###### Credentials URI By specifying a credentials uri, get credential from the local or remote uri, the credential will be able to automatically request maintenance to keep it update. ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: credentials_uri # credential type credentials_uri: http://local_or_remote_uri/ # Credentials URI ``` ###### Bearer If credential is required by the Cloud Call Centre (CCC), please apply for Bearer Token maintenance by yourself. ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: bearer # credential type bearer_token: bearerToken # BearerToken ``` ##### Use the credential provider chain ```yaml jaffle_shop: # this needs to match the profile in your dbt_project.yml file target: dev outputs: dev: type: maxcompute project: dbt-example # Replace this with your project name schema: default # Replace this with schema name, for example, dbt_bilbo endpoint: http://service.cn-shanghai.maxcompute.aliyun.com/api # Replace this with your maxcompute endpoint auth_type: chain ``` The default credential provider chain looks for available credentials, with following order: 1. Environment Credentials Look for environment credentials in environment variable. If the `ALIBABA_CLOUD_ACCESS_KEY_ID` and `ALIBABA_CLOUD_ACCESS_KEY_SECRET` environment variables are defined and are not empty, the program will use them to create default credentials. If the `ALIBABA_CLOUD_ACCESS_KEY_ID`, `ALIBABA_CLOUD_ACCESS_KEY_SECRET` and `ALIBABA_CLOUD_SECURITY_TOKEN` environment variables are defined and are not empty, the program will use them to create temporary security credentials(STS). Note: This token has an expiration time, it is recommended to use it in a temporary environment. 2. Credentials File If there is `~/.alibabacloud/credentials.ini default file (Windows shows C:\Users\USER_NAME\.alibabacloud\credentials.ini)`, the program automatically creates credentials with the specified type and name. The default file is not necessarily exist, but a parse error will throw an exception. The name of configuration item is lowercase. This configuration file can be shared between different projects and between different tools. Because it is outside of the project and will not be accidentally committed to the version control. The path to the default file can be modified by defining the `ALIBABA_CLOUD_CREDENTIALS_FILE` environment variable. If not configured, use the default configuration `default`. You can also set the environment variables `ALIBABA_CLOUD_PROFILE` to use the configuration. ```ini [default] # default setting enable = true # Enable，Enabled by default if this option is not present type = access_key # Certification type: access_key access_key_id = foo # Key access_key_secret = bar # Secret [client1] # configuration that is named as `client1` type = ecs_ram_role # Certification type: ecs_ram_role role_name = EcsRamRoleTest # Role Name [client2] # configuration that is named as `client2` enable = false # Disable type = ram_role_arn # Certification type: ram_role_arn region_id = cn-test policy = test # optional Specify permissions access_key_id = foo access_key_secret = bar role_arn = role_arn role_session_name = session_name # optional [client3] # configuration that is named as `client3` enable = false # Disable type = oidc_role_arn # Certification type: oidc_role_arn region_id = cn-test policy = test # optional Specify permissions access_key_id = foo # optional access_key_secret = bar # optional role_arn = role_arn oidc_provider_arn = oidc_provider_arn oidc_token_file_path = /xxx/xxx # can be replaced by setting environment variable: ALIBABA_CLOUD_OIDC_TOKEN_FILE role_session_name = session_name # optional ``` 3. Instance RAM Role If there is no credential information with a higher priority, the Credentials tool will obtain the value of ALIBABA\_CLOUD\_ECS\_METADATA (ECS instance RAM role name) through the environment variable. If the value of this variable exists, the program will use the hardened mode (IMDSv2) to access the metadata service (Meta Data Server) of ECS to obtain the STS Token of the ECS instance RAM role as the default credential information. If an exception occurs when using the hardened mode, the normal mode will be used as a fallback to obtain access credentials. You can also set the environment variable ALIBABA\_CLOUD\_IMDSV1\_DISABLED to perform different exception handling logic: * When the value is false, the normal mode will continue to obtain access credentials. * When the value is true, it means that only the hardened mode can be used to obtain access credentials, and an exception will be thrown. Whether the server supports IMDSv2 depends on your configuration on the server. 4. Credentials URI If the environment variable `ALIBABA_CLOUD_CREDENTIALS_URI` is defined and not empty, the program will take the value of the environment variable as credentials uri to get the temporary Security credentials. #### References * [Credentials Python](https://github.com/aliyun/credentials-python) --- ##### Connect Microsoft Azure Synapse Analytics to dbt v1 Local development info The following is a guide to using Azure Synapse Analytics dedicated SQL pools (formerly SQL DW). For more info, refer to [What is dedicated SQL pool (formerly SQL DW) in Azure Synapse Analytics?](https://learn.microsoft.com/azure/synapse-analytics/sql-data-warehouse/sql-data-warehouse-overview-what-is) for more info. For Microsoft Fabric setup with dbt, refer to [Microsoft Fabric Data Warehouse](https://docs.getdbt.com/docs/local/connect-data-platform/fabric-setup.md). * **Maintained by**: Microsoft * **Authors**: Microsoft (https://github.com/Microsoft) * **GitHub repo**: [Microsoft/dbt-synapse](https://github.com/Microsoft/dbt-synapse) [![](https://img.shields.io/github/stars/Microsoft/dbt-synapse?style=for-the-badge)](https://github.com/Microsoft/dbt-synapse) * **PyPI package**: `dbt-synapse` [![](https://badge.fury.io/py/dbt-synapse.svg)](https://badge.fury.io/py/dbt-synapse) * **Slack channel**: [#db-synapse](https://getdbt.slack.com/archives/C01DRQ178LQ) * **Supported dbt version**: v0.18.0 and newer * **dbt support**: Supported * **Minimum data platform version**: Azure Synapse 10 #### Installing dbt-synapse Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-synapse` #### Configuring dbt-synapse For Synapse-specific configuration, please refer to [Synapse configs.](https://docs.getdbt.com/reference/resource-configs/azuresynapse-configs.md) Dedicated SQL only Azure Synapse Analytics offers both dedicated SQL pools and serverless SQL pools. \*\*Only Dedicated SQL Pools are supported by this adapter. ##### Prerequisites On Debian/Ubuntu make sure you have the ODBC header files before installing ```bash sudo apt install unixodbc-dev ``` Download and install the [Microsoft ODBC Driver 18 for SQL Server](https://docs.microsoft.com/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver15). If you already have ODBC Driver 17 installed, then that one will work as well. Default settings change in dbt-synapse v1.2 / ODBC Driver 18 Microsoft made several changes related to connection encryption. Read more about the changes [here](https://docs.getdbt.com/docs/local/connect-data-platform/mssql-setup.md). ##### Authentication methods This adapter is based on the adapter for Microsoft SQL Server. Therefore, the same authentication methods are supported. The configuration is the same except for 1 major difference: instead of specifying `type: sqlserver`, you specify `type: synapse`. Example: profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: synapse driver: 'ODBC Driver 17 for SQL Server' # (The ODBC Driver installed on your system) server: workspacename.sql.azuresynapse.net # (Dedicated SQL endpoint of your workspace here) port: 1433 database: exampledb schema: schema_name user: username password: password ``` You can find all the available options and the documentation and how to configure them on [the documentation page for the dbt-sqlserver adapter](https://docs.getdbt.com/docs/local/connect-data-platform/mssql-setup.md). --- ##### Connect Microsoft Fabric Data Warehouse to dbt v1 Local development `profiles.yml` file is for dbt v1 and dbt v2 only If you're using dbt platform, you don't need to create a `profiles.yml` file. This file is only necessary when you use v1 or v2 locally. To learn more about v2 prerequisites, refer to [Supported features](https://docs.getdbt.com/docs/dbt/supported-features.md). To connect your data platform to dbt, refer to [About data platforms](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). Below is a guide for use with [Fabric Data Warehouse](https://learn.microsoft.com/en-us/fabric/data-warehouse/data-warehousing#synapse-data-warehouse), a new product within Microsoft Fabric. The adapter currently supports connecting to a warehouse. To learn how to set up dbt using Fabric Lakehouse, refer to [Microsoft Fabric Lakehouse](https://docs.getdbt.com/docs/local/connect-data-platform/fabricspark-setup.md). To learn how to set up dbtAnalytics dedicated SQL pools, refer to [Microsoft Azure Synapse Analytics setup](https://docs.getdbt.com/docs/local/connect-data-platform/azuresynapse-setup.md). * **Maintained by**: Microsoft * **Authors**: Microsoft * **GitHub repo**: [Microsoft/dbt-fabric](https://github.com/Microsoft/dbt-fabric) [![](https://img.shields.io/github/stars/Microsoft/dbt-fabric?style=for-the-badge)](https://github.com/Microsoft/dbt-fabric) * **PyPI package**: `dbt-fabric` [![](https://badge.fury.io/py/dbt-fabric.svg)](https://badge.fury.io/py/dbt-fabric) * **Slack channel**:[]() * **Supported dbt version**: 1.4.0 and newer * **dbt support**: Supported * **Minimum data platform version**: #### Installing dbt-fabric Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-fabric` #### Configuring dbt-fabric For Microsoft Fabric-specific configuration, please refer to [Microsoft Fabric configs.](https://docs.getdbt.com/reference/resource-configs/fabric-configs.md) ##### Prerequisites On Debian/Ubuntu make sure you have the ODBC header files before installing ```bash sudo apt install unixodbc-dev ``` Download and install the [Microsoft ODBC Driver 18 for SQL Server](https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver15). If you already have ODBC Driver 17 installed, then that one will work as well. ###### Supported configurations * The adapter is tested with Microsoft Fabric Data Warehouse (also referred to as warehouses). * We test all combinations with Microsoft ODBC Driver 17 and Microsoft ODBC Driver 18. * The collations we run our tests on are `Latin1_General_100_BIN2_UTF8`. The adapter support is not limited to the matrix of the above configurations. If you notice an issue with any other configuration, let us know by opening an issue on [GitHub](https://github.com/microsoft/dbt-fabric). ###### Unsupported configurations SQL analytics endpoints are read-only and so are not appropriate for Transformation workloads, use a Warehouse instead. #### Authentication methods & profile configuration Supported authentication methods Microsoft Fabric supports two authentication types: * Microsoft Entra service principal * Microsoft Entra password To better understand the authentication mechanisms, read our [Connect Microsoft Fabric](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-microsoft-fabric.md) page. ##### Common configuration For all the authentication methods, refer to the following configuration options that can be set in your `profiles.yml` file. A complete reference of all options can be found [at the end of this page](#reference-of-all-connection-options). | Configuration option | Description | Type | Example | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------- | | `driver` | The ODBC driver to use | Required | `ODBC Driver 18 for SQL Server` | | `server` | The server hostname | Required | `localhost` | | `port` | The server port | Required | `1433` | | `database` | The database name | Required | Not applicable | | `schema` | The schema name | Required | `dbo` | | `retries` | The number of automatic times to retry a query before failing. Defaults to `1`. Queries with syntax errors will not be retried. This setting can be used to overcome intermittent network issues. | Optional | Not applicable | | `login_timeout` | The number of seconds used to establish a connection before failing. Defaults to `0`, which means that the timeout is disabled or uses the default system settings. | Optional | Not applicable | | `query_timeout` | The number of seconds used to wait for a query before failing. Defaults to `0`, which means that the timeout is disabled or uses the default system settings. | Optional | Not applicable | | `schema_authorization` | Optionally set this to the principal who should own the schemas created by dbt. [Read more about schema authorization](#schema-authorization). | Optional | Not applicable | | `encrypt` | Whether to encrypt the connection to the server. Defaults to `true`. Read more about [connection encryption](#connection-encryption). | Optional | Not applicable | | `trust_cert` | Whether to trust the server certificate. Defaults to `false`. Read more about [connection encryption](#connection-encryption). | Optional | Not applicable | ##### Connection encryption Microsoft made several changes in the release of ODBC Driver 18 that affects how connection encryption is configured. To accommodate these changes, starting in dbt-sqlserver 1.2.0 or newer the default values of `encrypt` and `trust_cert` have changed. Both of these settings will now **always** be included in the connection string to the server, regardless if you've left them out of your profile configuration or not. * The default value of `encrypt` is `true`, meaning that connections are encrypted by default. * The default value of `trust_cert` is `false`, meaning that the server certificate will be validated. By setting this to `true`, a self-signed certificate will be accepted. More details about how these values affect your connection and how they are used differently in versions of the ODBC driver can be found in the [Microsoft documentation](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16#encrypt). ##### Standard SQL Server authentication SQL Server and windows authentication are not supported by Microsoft Fabric Data Warehouse. ##### Microsoft Entra ID authentication Microsoft Entra ID (formerly Azure AD) authentication is a default authentication mechanism in Microsoft Fabric Data Warehouse. The following additional methods are available to authenticate to Azure SQL products: * Microsoft Entra ID username and password * Service principal * Environment-based authentication * Azure CLI authentication * VS Code authentication (available through the automatic option below) * Azure PowerShell module authentication (available through the automatic option below) * Automatic authentication The automatic authentication setting is in most cases the easiest choice and works for all of the above. ##### Microsoft Entra ID username & password profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabric driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ActiveDirectoryPassword user: bill.gates@microsoft.com password: iheartopensource ``` ##### Service principal Client ID is often also referred to as Application ID. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabric driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ServicePrincipal tenant_id: 00000000-0000-0000-0000-000000001234 client_id: 00000000-0000-0000-0000-000000001234 client_secret: S3cret! ``` ##### Managed Identity This authentication option allows you to dynamically select an authentication method depending on the available environment variables. [The Microsoft docs on EnvironmentCredential](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) explain the available combinations of environment variables you can use. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabric driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: environment ``` ##### Environment-based First, install the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli), then, log in: `az login` profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabric driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: CLI ``` ##### Azure CLI This authentication option will automatically try to use all available authentication methods. The following methods are tried in order: 1. Environment-based authentication 2. Managed Identity authentication. Managed Identity is not supported at this time. 3. Visual Studio authentication (*Windows only, ignored on other operating systems*) 4. Visual Studio Code authentication 5. Azure CLI authentication 6. Azure PowerShell module authentication profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabric driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: auto ``` ###### Additional options for Microsoft Entra ID on Windows On Windows systems, the following additional authentication methods are also available for Azure SQL: * Microsoft Entra ID interactive * Microsoft Entra ID integrated * Visual Studio authentication (available through the automatic option above) ##### Microsoft Entra ID interactive This setting can optionally show Multi-Factor Authentication prompts. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabric driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ActiveDirectoryInteractive user: bill.gates@microsoft.com ``` ##### Microsoft Entra ID integrated This uses the credentials you're logged in with on the current machine. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabric driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ActiveDirectoryIntegrated ``` ##### Automatic Microsoft Entra ID principal provisioning for grants Please note that automatic Microsoft Entra ID principal provisioning is not supported by Microsoft Fabric Data Warehouse at this time. Even though in dbtn use the [grants](https://docs.getdbt.com/reference/resource-configs/grants.md) config block to automatically grant/revoke permissions on your models to users or groups, the data warehouse does not support this feature at this time. You need to add the service principal or Microsoft Entra identity to a Fabric Workspace as an admin ##### Schema authorization You can optionally set the principal who should own all schemas created by dbt. This is then used in the `CREATE SCHEMA` statement like so: ```sql CREATE SCHEMA [schema_name] AUTHORIZATION [schema_authorization] ``` A common use case is to use this when you are authenticating with a principal who has permissions based on a group, such as a Microsoft Entra ID group. When that principal creates a schema, the server will first try to create an individual login for this principal and then link the schema to that principal. If you would be using Microsoft Entra ID in this case, then this would fail since Azure SQL can't create logins for individuals part of an AD group automatically. ##### Reference of all connection options | Configuration option | Description | Required | Default value | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------- | | `driver` | The ODBC driver to use. | ✅ | | | `host` | The hostname of the database server. | ✅ | | | `port` | The port of the database server. | | `1433` | | `database` | The name of the database to connect to. | ✅ | | | `schema` | The schema to use. | ✅ | | | `authentication` | The authentication method to use. This is not required for Windows authentication. | | `'sql'` | | `UID` | Username used to authenticate. This can be left out depending on the authentication method. | | | | `PWD` | Password used to authenticate. This can be left out depending on the authentication method. | | | | `tenant_id` | The tenant ID of the Microsoft Entra ID instance. This is only used when connecting to Azure SQL with a service principal. | | | | `client_id` | The client ID of the Microsoft Entra service principal. This is only used when connecting to Azure SQL with a Microsoft Entra service principal. | | | | `client_secret` | The client secret of the Microsoft Entra service principal. This is only used when connecting to Azure SQL with a Microsoft Entra service principal. | | | | `encrypt` | Set this to `false` to disable the use of encryption. See [above](#connection-encryption). | | `true` | | `trust_cert` | Set this to `true` to trust the server certificate. See [above](#connection-encryption). | | `false` | | `retries` | The number of times to retry a failed connection. | | `1` | | `schema_authorization` | Optionally set this to the principal who should own the schemas created by dbt. [Details above](#schema-authorization). | | | | `login_timeout` | The amount of seconds to wait until a response from the server is received when establishing a connection. `0` means that the timeout is disabled. | | `0` | | `query_timeout` | The amount of seconds to wait until a response from the server is received when executing a query. `0` means that the timeout is disabled. | | `0` | Valid values for `authentication`: * `ActiveDirectoryPassword`: Active Directory authentication using username and password * `ActiveDirectoryInteractive`: Active Directory authentication using a username and MFA prompts * `ActiveDirectoryIntegrated`: Active Directory authentication using the current user's credentials * `ServicePrincipal`: Microsoft Entra ID authentication using a service principal * `CLI`: Microsoft Entra ID authentication using the account you're logged in within the Azure CLI * `environment`: Microsoft Entra ID authentication using environment variables as documented [here](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) * `auto`: Microsoft Entra ID authentication trying the previous authentication methods until it finds one that works --- ##### Connect Microsoft Fabric Lakehouse to dbt v1 Local development `profiles.yml` file is for dbt v1 and dbt v2 only If you're using dbt platform, you don't need to create a `profiles.yml` file. This file is only necessary when you use v1 or v2 locally. To learn more about v2 prerequisites, refer to [Supported features](https://docs.getdbt.com/docs/dbt/supported-features.md). To connect your data platform to dbt, refer to [About data platforms](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). Below is a guide for use with [Fabric Data Engineering](https://learn.microsoft.com/en-us/fabric/data-engineering/data-engineering-overview), a new product within Microsoft Fabric. This adapter currently supports connecting to a lakehouse endpoint. To learn how to set up dbt using Fabric Warehouse, refer to [Microsoft Fabric Data Warehouse](https://docs.getdbt.com/docs/local/connect-data-platform/fabric-setup.md). * **Maintained by**: Microsoft * **Authors**: Microsoft * **GitHub repo**: [microsoft/dbt-fabricspark](https://github.com/microsoft/dbt-fabricspark) [![](https://img.shields.io/github/stars/microsoft/dbt-fabricspark?style=for-the-badge)](https://github.com/microsoft/dbt-fabricspark) * **PyPI package**: `dbt-fabricspark` [![](https://badge.fury.io/py/dbt-fabricspark.svg)](https://badge.fury.io/py/dbt-fabricspark) * **Slack channel**: [db-fabric-synapse](https://getdbt.slack.com/archives/C01DRQ178LQ) * **Supported dbt version**: v1.7 and newer * **dbt support**: Not supported * **Minimum data platform version**: n/a #### Installing dbt-fabricspark Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-fabricspark` #### Configuring dbt-fabricspark For Microsoft Fabric-specific configuration, please refer to [Microsoft Fabric configs.](https://docs.getdbt.com/reference/resource-configs/fabricspark-configs.md) For further info, refer to the GitHub repository: [microsoft/dbt-fabricspark](https://github.com/microsoft/dbt-fabricspark) #### Supported authentication types The Fabric Lakehouse adapter (`dbt-fabricspark`) connects to Fabric Spark through the Livy API. Choose the authentication method that matches where you run dbt v1. ##### Azure CLI Use Azure CLI for self-hosted dbt installations. This lets dbt v1 use credentials from an active `az login` session. To use this method, set `authentication: CLI` in your `profiles.yml` file and run `az login`. When you authenticate, Azure CLI may open a browser window or prompt you to complete sign-in on the [Microsoft device login](https://microsoft.com/devicelogin) page and enter a one-time code to complete sign-in. Once authentication is successful, dbt automatically reuses the active Azure CLI session for subsequent commands. ###### Example Azure CLI configuration profiles.yml ```yml default: target: dev outputs: dev: type: fabricspark method: livy endpoint: https://api.fabric.microsoft.com/v1 workspaceid: WORKSPACE_ID lakehouseid: LAKEHOUSE_ID lakehouse: LAKEHOUSE_NAME schema: LAKEHOUSE_OR_SCHEMA_NAME threads: 1 # Authentication (CLI for self-hosted dbt installations) authentication: CLI ``` ##### Service principal Use a Microsoft Entra ID service principal for CI/CD and other automated workloads. This method uses app registration credentials, not an active az login session. To use this method, set `authentication: SPN` in your `profiles.yml` file and provide the service principal's client ID, tenant ID, and client secret. Before you run with service principal authentication, make sure: * The service principal has access to the Fabric workspace and lakehouse. * Service principals can use Fabric APIs is enabled in the Fabric Admin port. ###### Example service principal configuration profiles.yml ```yml default: target: dev outputs: dev: type: fabricspark method: livy endpoint: https://api.fabric.microsoft.com/v1 workspaceid: WORKSPACE_ID lakehouseid: LAKEHOUSE_ID lakehouse: LAKEHOUSE_NAME schema: LAKEHOUSE_OR_SCHEMA_NAME threads: 1 # Authentication (SPN for CI/CD) authentication: SPN client_id: SPN_CLIENT_ID tenant_id: MICROSOFT_ENTRA_TENANT_ID client_secret: SPN_CLIENT_SECRET ``` ##### Fabric notebook You can authenticate through the Livy API from a Fabric notebook for production workloads and orchestration use cases within the Microsoft Fabric ecosystem. This method uses [`notebookutils.credentials`](https://learn.microsoft.com/en-us/fabric/data-engineering/notebookutils/notebookutils-credentials?tabs=python). Use Python notebooks instead of PySpark notebooks to minimize compute costs because the Livy API starts its own Spark sessions. ###### Example Fabric notebook configuration profiles.yml ```yml default: target: dev outputs: dev: type: fabricspark method: livy endpoint: https://api.fabric.microsoft.com/v1 workspaceid: WORKSPACE_ID lakehouseid: LAKEHOUSE_ID lakehouse: LAKEHOUSE_NAME schema: LAKEHOUSE_OR_SCHEMA_NAME threads: 1 # Authentication (fabric_notebook for execution from a Fabric notebook) authentication: fabric_notebook ``` ##### Custom TokenCredential Use `token_credential` when you need dbt v1 to load a custom [`azure.core.credentials.TokenCredential`](https://learn.microsoft.com/en-us/python/api/azure-core/azure.core.credentials.tokencredential) implementation, such as a workload identity federation or token broker flow. To use this method, set `authentication: token_credential` and provide the dotted path to your credential class. ###### Example custom TokenCredential configuration profiles.yml ```yml default: target: dev outputs: dev: type: fabricspark method: livy endpoint: https://api.fabric.microsoft.com/v1 workspaceid: WORKSPACE_ID lakehouseid: LAKEHOUSE_ID lakehouse: LAKEHOUSE_NAME schema: LAKEHOUSE_OR_SCHEMA_NAME threads: 1 # Authentication (custom TokenCredential) authentication: token_credential credential_class: my_package.auth.ExternalTokenCredential credential_kwargs: tenant_id: MICROSOFT_ENTRA_TENANT_ID ``` Replace `credential_class` and `credential_kwargs` with the class path and keyword arguments your custom credential implementation expects. #### Connection methods `dbt-fabricspark` can connect to Fabric Spark runtime using Fabric Livy API method. The Fabric Livy API allows submitting jobs in two different modes: * [`session-jobs`](#session-jobs) A Livy session job entails establishing a Spark session that remains active throughout the Spark session. A Spark session, can run multiple jobs (each job is an action), sharing state and cached data between jobs. * batch jobs entails submitting a Spark application for a single job execution. In contrast to a Livy session job, a batch job doesn't sustain an ongoing Spark session. With Livy batch jobs, each job initiates a new Spark session that ends when the job finishes. Supported mode To share the session state among jobs and reduce the overhead of session management, `dbt-fabricspark` adapter supports only `session-jobs` mode. ##### session-jobs `session-jobs` is the preferred method when connecting to Fabric Lakehouse. \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: fabricspark method: livy authentication: CLI endpoint: https://api.fabric.microsoft.com/v1 workspaceid: [Fabric Workspace GUID] lakehouseid: [Lakehouse GUID] lakehouse: [Lakehouse Name] schema: [Lakehouse Name] spark_config: name: [Application Name] # optional archives: - "example-archive.zip" conf: spark.executor.memory: "2g" spark.executor.cores: "2" tags: project: [Project Name] user: [User Email] driverMemory: "2g" driverCores: 2 executorMemory: "4g" executorCores: 4 numExecutors: 3 # optional connect_retries: 0 connect_timeout: 10 retry_all: true ``` #### Optional configurations ##### Retries Intermittent errors can crop up unexpectedly while running queries against Fabric Spark. If `retry_all` is enabled, `dbt-fabricspark` will naively retry any queries that fails, based on the configuration supplied by `connect_timeout` and `connect_retries`. It does not attempt to determine if the query failure was transient or likely to succeed on retry. This configuration is recommended in production environments, where queries ought to be succeeding. The default `connect_retries` configuration is 2. For instance, this will instruct dbt to retry all failed queries up to 3 times, with a 5 second delay between each retry: \~/.dbt/profiles.yml ```yaml retry_all: true connect_timeout: 5 connect_retries: 3 ``` ##### Spark configuration Spark can be customized using [Application Properties](https://spark.apache.org/docs/latest/configuration.html). Using these properties the execution can be customized, for example, to allocate more memory to the driver process. Also, the Spark SQL runtime can be set through these properties. For example, this allows the user to [set a Spark catalogs](https://spark.apache.org/docs/latest/configuration.html#spark-sql). ##### Supported functionality Most dbt v1 functionality is supported, Please refer to [Delta Lake interoperability](https://learn.microsoft.com/en-us/fabric/fundamentals/delta-lake-interoperability). Delta-only features: 1. Incremental model updates by `unique_key` instead of `partition_by` (see [`merge` strategy](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#the-merge-strategy)) 2. [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) 3. [Persisting](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) column-level descriptions as database comments ##### Limitations 1. Only Delta, CSV, and Parquet table data formats are supported by Fabric Lakehouse. --- ##### Connect Microsoft SQL Server to dbt v1 Local development Community plugin Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below. * **Maintained by**: Community * **Authors**: Mikael Ene & dbt-msft community (https://github.com/dbt-msft) * **GitHub repo**: [dbt-msft/dbt-sqlserver](https://github.com/dbt-msft/dbt-sqlserver) [![](https://img.shields.io/github/stars/dbt-msft/dbt-sqlserver?style=for-the-badge)](https://github.com/dbt-msft/dbt-sqlserver) * **PyPI package**: `dbt-sqlserver` [![](https://badge.fury.io/py/dbt-sqlserver.svg)](https://badge.fury.io/py/dbt-sqlserver) * **Slack channel**: [#db-sqlserver](https://getdbt.slack.com/archives/CMRMDDQ9W) * **Supported dbt version**: v0.14.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: SQL Server 2016 #### Installing dbt-sqlserver Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-sqlserver` #### Configuring dbt-sqlserver For SQL Server-specific configuration, please refer to [SQL Server configs.](https://docs.getdbt.com/reference/resource-configs/mssql-configs.md) Default settings change in dbt-sqlserver v1.2 / ODBC Driver 18 Microsoft made several changes related to connection encryption. Read more about the changes [below](#connection-encryption). ##### Prerequisites On Debian/Ubuntu make sure you have the ODBC header files before installing ```bash sudo apt install unixodbc-dev ``` Download and install the [Microsoft ODBC Driver 18 for SQL Server](https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver15). If you already have ODBC Driver 17 installed, then that one will work as well. ###### Supported configurations * The adapter is tested with SQL Server 2017, SQL Server 2019, SQL Server 2022 and Azure SQL Database. * We test all combinations with Microsoft ODBC Driver 17 and Microsoft ODBC Driver 18. * The collations we run our tests on are `SQL_Latin1_General_CP1_CI_AS` and `SQL_Latin1_General_CP1_CS_AS`. The adapter support is not limited to the matrix of the above configurations. If you notice an issue with any other configuration, let us know by opening an issue on [GitHub](https://github.com/dbt-msft/dbt-sqlserver). #### Authentication methods & profile configuration ##### Common configuration For all the authentication methods, refer to the following configuration options that can be set in your `profiles.yml` file. A complete reference of all options can be found [at the end of this page](#reference-of-all-connection-options). | Configuration option | Description | Type | Example | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------- | | `driver` | The ODBC driver to use | Required | `ODBC Driver 18 for SQL Server` | | `server` | The server hostname | Required | `localhost` | | `port` | The server port | Required | `1433` | | `database` | The database name | Required | Not applicable | | `schema` | The schema name | Required | `dbo` | | `retries` | The number of automatic times to retry a query before failing. Defaults to `1`. Queries with syntax errors will not be retried. This setting can be used to overcome intermittent network issues. | Optional | Not applicable | | `login_timeout` | The number of seconds used to establish a connection before failing. Defaults to `0`, which means that the timeout is disabled or uses the default system settings. | Optional | Not applicable | | `query_timeout` | The number of seconds used to wait for a query before failing. Defaults to `0`, which means that the timeout is disabled or uses the default system settings. | Optional | Not applicable | | `schema_authorization` | Optionally set this to the principal who should own the schemas created by dbt. [Read more about schema authorization](#schema-authorization). | Optional | Not applicable | | `encrypt` | Whether to encrypt the connection to the server. Defaults to `true`. Read more about [connection encryption](#connection-encryption). | Optional | Not applicable | | `trust_cert` | Whether to trust the server certificate. Defaults to `false`. Read more about [connection encryption](#connection-encryption). | Optional | Not applicable | ##### Connection encryption Microsoft made several changes in the release of ODBC Driver 18 that affects how connection encryption is configured. To accommodate these changes, starting in dbt-sqlserver 1.2.0 or newer the default values of `encrypt` and `trust_cert` have changed. Both of these settings will now **always** be included in the connection string to the server, regardless if you've left them out of your profile configuration or not. * The default value of `encrypt` is `true`, meaning that connections are encrypted by default. * The default value of `trust_cert` is `false`, meaning that the server certificate will be validated. By setting this to `true`, a self-signed certificate will be accepted. More details about how these values affect your connection and how they are used differently in versions of the ODBC driver can be found in the [Microsoft documentation](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16#encrypt). ##### Standard SQL Server authentication SQL Server credentials are supported for on-premise servers as well as Azure, and it is the default authentication method for `dbt-sqlserver`. When running on Windows, you can also use your Windows credentials to authenticate. ##### SQL Server credentials profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: database schema: schema_name user: username password: password ``` ##### Windows credentials profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name windows_login: True ``` ##### Microsoft Entra ID authentication While you can use the SQL username and password authentication as mentioned above, you might opt to use one of the authentication methods below for Azure SQL. The following additional methods are available to authenticate to Azure SQL products: * Microsoft Entra ID (formerly Azure AD) username and password * Service principal * Managed Identity * Environment-based authentication * Azure CLI authentication * VS Code authentication (available through the automatic option below) * Azure PowerShell module authentication (available through the automatic option below) * Automatic authentication The automatic authentication setting is in most cases the easiest choice and works for all of the above. ##### Microsoft Entra ID username & password profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ActiveDirectoryPassword user: bill.gates@microsoft.com password: iheartopensource ``` ##### Service principal Client ID is often also referred to as Application ID. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ServicePrincipal tenant_id: 00000000-0000-0000-0000-000000001234 client_id: 00000000-0000-0000-0000-000000001234 client_secret: S3cret! ``` ##### Managed Identity Both system-assigned and user-assigned managed identities will work. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ActiveDirectoryMsi ``` ##### Environment-based This authentication option allows you to dynamically select an authentication method depending on the available environment variables. [The Microsoft docs on EnvironmentCredential](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) explain the available combinations of environment variables you can use. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: environment ``` ##### Azure CLI First, install the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli), then, log in: `az login` profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: CLI ``` ##### Automatic This authentication option will automatically try to use all available authentication methods. The following methods are tried in order: 1. Environment-based authentication 2. Managed Identity authentication 3. Visual Studio authentication (*Windows only, ignored on other operating systems*) 4. Visual Studio Code authentication 5. Azure CLI authentication 6. Azure PowerShell module authentication profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: auto ``` ###### Additional options for Microsoft Entra ID on Windows On Windows systems, the following additional authentication methods are also available for Azure SQL: * Microsoft Entra ID interactive * Microsoft Entra ID integrated * Visual Studio authentication (available through the automatic option above) ##### Microsoft Entra ID interactive This setting can optionally show Multi-Factor Authentication prompts. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ActiveDirectoryInteractive user: bill.gates@microsoft.com ``` ##### Microsoft Entra ID integrated This uses the credentials you're logged in with on the current machine. profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlserver driver: 'ODBC Driver 18 for SQL Server' # (The ODBC Driver installed on your system) server: hostname or IP of your server port: 1433 database: exampledb schema: schema_name authentication: ActiveDirectoryIntegrated ``` ##### Automatic Microsoft Entra ID principal provisioning for grants In dbt 1.2 or newer you can use the [grants](https://docs.getdbt.com/reference/resource-configs/grants.md) config block to automatically grant/revoke permissions on your models to users or groups. This is fully supported in this adapter and comes with an additional feature. By setting `auto_provision_aad_principals` to `true` in your model configuration, you can automatically provision Microsoft Entra ID principals (users or groups) that don't exist yet. In Azure SQL, you can sign in using Microsoft Entra ID authentication, but to be able to grant a Microsoft Entra ID principal certain permissions, it needs to be linked in the database first. ([Microsoft documentation](https://learn.microsoft.com/en-us/azure/azure-sql/database/authentication-aad-configure?view=azuresql)) Note that principals will not be deleted automatically when they are removed from the `grants` block. ##### Schema authorization You can optionally set the principal who should own all schemas created by dbt. This is then used in the `CREATE SCHEMA` statement like so: ```sql CREATE SCHEMA [schema_name] AUTHORIZATION [schema_authorization] ``` A common use case is to use this when you are authenticating with a principal who has permissions based on a group, such as a Microsoft Entra ID group. When that principal creates a schema, the server will first try to create an individual login for this principal and then link the schema to that principal. If you would be using Microsoft Entra ID in this case, then this would fail since Azure SQL can't create logins for individuals part of an AD group automatically. ##### Reference of all connection options | Configuration option | Description | Required | Default value | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------- | | `driver` | The ODBC driver to use. | ✅ | | | `host` | The hostname of the database server. | ✅ | | | `port` | The port of the database server. | | `1433` | | `database` | The name of the database to connect to. | ✅ | | | `schema` | The schema to use. | ✅ | | | `authentication` | The authentication method to use. This is not required for Windows authentication. | | `'sql'` | | `UID` | Username used to authenticate. This can be left out depending on the authentication method. | | | | `PWD` | Password used to authenticate. This can be left out depending on the authentication method. | | | | `windows_login` | Set this to `true` to use Windows authentication. This is only available for SQL Server. | | | | `tenant_id` | The tenant ID of the Microsoft Entra ID instance. This is only used when connecting to Azure SQL with a service principal. | | | | `client_id` | The client ID of the Microsoft Entra service principal. This is only used when connecting to Azure SQL with a Microsoft Entra service principal. | | | | `client_secret` | The client secret of the Microsoft Entra service principal. This is only used when connecting to Azure SQL with a Microsoft Entra service principal. | | | | `encrypt` | Set this to `false` to disable the use of encryption. See [above](#connection-encryption). | | `true` | | `trust_cert` | Set this to `true` to trust the server certificate. See [above](#connection-encryption). | | `false` | | `retries` | The number of times to retry a failed connection. | | `1` | | `schema_authorization` | Optionally set this to the principal who should own the schemas created by dbt. [Details above](#schema-authorization). | | | | `login_timeout` | The amount of seconds to wait until a response from the server is received when establishing a connection. `0` means that the timeout is disabled. | | `0` | | `query_timeout` | The amount of seconds to wait until a response from the server is received when executing a query. `0` means that the timeout is disabled. | | `0` | Valid values for `authentication`: * `sql`: SQL authentication using username and password * `ActiveDirectoryPassword`: Active Directory authentication using username and password * `ActiveDirectoryInteractive`: Active Directory authentication using a username and MFA prompts * `ActiveDirectoryIntegrated`: Active Directory authentication using the current user's credentials * `ServicePrincipal`: Microsoft Entra ID authentication using a service principal * `CLI`: Microsoft Entra ID authentication using the account you're logged in with in the Azure CLI * `ActiveDirectoryMsi`: Microsoft Entra ID authentication using a managed identity available on the system * `environment`: Microsoft Entra ID authentication using environment variables as documented [here](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) * `auto`: Microsoft Entra ID authentication trying the previous authentication methods until it finds one that works --- ##### Connect MindsDB to dbt v1 Local development Vendor-supported plugin The dbt-mindsdb package allows dbt to connect to [MindsDB](https://github.com/mindsdb/mindsdb). * **Maintained by**: MindsDB * **Authors**: MindsDB team * **GitHub repo**: [mindsdb/dbt-mindsdb](https://github.com/mindsdb/dbt-mindsdb) [![](https://img.shields.io/github/stars/mindsdb/dbt-mindsdb?style=for-the-badge)](https://github.com/mindsdb/dbt-mindsdb) * **PyPI package**: `dbt-mindsdb` [![](https://badge.fury.io/py/dbt-mindsdb.svg)](https://badge.fury.io/py/dbt-mindsdb) * **Slack channel**: [n/a](https://www.getdbt.com/community) * **Supported dbt version**: v1.0.1 and newer * **dbt support**: Not Supported * **Minimum data platform version**: ? #### Installing dbt-mindsdb Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-mindsdb` #### Configuring dbt-mindsdb For MindsDB-specific configuration, please refer to [MindsDB configs.](https://docs.getdbt.com/reference/resource-configs/mindsdb-configs.md) #### Configurations Basic `profiles.yml` for connecting to MindsDB: ```yml mindsdb: outputs: dev: database: 'mindsdb' host: '127.0.0.1' password: '' port: 47335 schema: 'mindsdb' type: mindsdb username: 'mindsdb' target: dev ``` | Key | Required | Description | Example | | -------- | -------- | ---------------------------------------------------- | ------------------------------- | | type | ✔️ | The specific adapter to use | `mindsdb` | | host | ✔️ | The MindsDB (hostname) to connect to | `cloud.mindsdb.com` | | port | ✔️ | The port to use | `3306` or `47335` | | schema | ✔️ | Specify the schema (database) to build models into | The MindsDB datasource | | username | ✔️ | The username to use to connect to the server | `mindsdb` or mindsdb cloud user | | password | ✔️ | The password to use for authenticating to the server | \`pass | --- ##### Connect MySQL to dbt v1 Local development Community plugin Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below. * **Maintained by**: Community * **Authors**: Doug Beatty (https://github.com/dbeatty10) * **GitHub repo**: [dbeatty10/dbt-mysql](https://github.com/dbeatty10/dbt-mysql) [![](https://img.shields.io/github/stars/dbeatty10/dbt-mysql?style=for-the-badge)](https://github.com/dbeatty10/dbt-mysql) * **PyPI package**: `dbt-mysql` [![](https://badge.fury.io/py/dbt-mysql.svg)](https://badge.fury.io/py/dbt-mysql) * **Slack channel**: [#db-mysql-family](https://getdbt.slack.com/archives/C03BK0SHC64) * **Supported dbt version**: v0.18.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: MySQL 5.7 and 8.0 #### Installing dbt-mysql Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-mysql` #### Configuring dbt-mysql For MySQL-specific configuration, please refer to [MySQL configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) This is an experimental plugin: * It has not been tested extensively. * Storage engines other than the default of InnoDB are untested. * Only tested with [dbt-adapter-tests](https://github.com/dbt-labs/dbt-adapter-tests) with the following versions: * MySQL 5.7 * MySQL 8.0 * MariaDB 10.5 * Compatibility with other [dbt packages](https://hub.getdbt.com/) (like [dbt\_utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/)) are also untested. Please read these docs carefully and use at your own risk. [Issues](https://github.com/dbeatty10/dbt-mysql/issues/new) and [PRs](https://github.com/dbeatty10/dbt-mysql/blob/main/CONTRIBUTING.rst#contributing) welcome! #### Connecting to MySQL with dbt-mysql MySQL targets should be set up using the following configuration in your `profiles.yml` file. Example: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: mysql server: localhost port: 3306 schema: analytics username: your_mysql_username password: your_mysql_password ssl_disabled: True ``` ###### Description of MySQL Profile Fields | Option | Description | Required? | Example | | ------------- | ----------------------------------------------------- | --------- | ------------------------------ | | type | The specific adapter to use | Required | `mysql`, `mysql5` or `mariadb` | | server | The server (hostname) to connect to | Required | `yourorg.mysqlhost.com` | | port | The port to use | Optional | `3306` | | schema | Specify the schema (database) to build models into | Required | `analytics` | | username | The username to use to connect to the server | Required | `dbt_admin` | | password | The password to use for authenticating to the server | Required | `correct-horse-battery-staple` | | ssl\_disabled | Set to enable or disable TLS connectivity to mysql5.x | Optional | `True` or `False` | #### Supported features | MariaDB 10.5 | MySQL 5.7 | MySQL 8.0 | Feature | | ------------ | --------- | --------- | --------------------------- | | ✅ | ✅ | ✅ | Table materialization | | ✅ | ✅ | ✅ | View materialization | | ✅ | ✅ | ✅ | Incremental materialization | | ✅ | ❌ | ✅ | Ephemeral materialization | | ✅ | ✅ | ✅ | Seeds | | ✅ | ✅ | ✅ | Sources | | ✅ | ✅ | ✅ | Custom data tests | | ✅ | ✅ | ✅ | Docs generate | | 🤷 | 🤷 | ✅ | Snapshots | #### Notes * Ephemeral materializations rely upon [Common Table Expressions](https://en.wikipedia.org/wiki/Hierarchical_and_recursive_queries_in_SQL) (CTEs), which are not supported until MySQL 8.0. * MySQL 5.7 has some configuration gotchas that might affect dbt snapshots to not work properly due to [automatic initialization and updating for `TIMESTAMP`](https://dev.mysql.com/doc/refman/5.7/en/timestamp-initialization.html). * If the output of `SHOW VARIABLES LIKE 'sql_mode'` includes `NO_ZERO_DATE`. A solution is to include the following in a `*.cnf` file: ```text [mysqld] explicit_defaults_for_timestamp = true sql_mode = "ALLOW_INVALID_DATES,{other_sql_modes}" ``` * Where `{other_sql_modes}` is the rest of the modes from the `SHOW VARIABLES LIKE 'sql_mode'` output. --- ##### Connect Oracle to dbt v1 Local development * **Maintained by**: Oracle * **Authors**: Oracle * **GitHub repo**: [oracle/dbt-oracle](https://github.com/oracle/dbt-oracle) [![](https://img.shields.io/github/stars/oracle/dbt-oracle?style=for-the-badge)](https://github.com/oracle/dbt-oracle) * **PyPI package**: `dbt-oracle` [![](https://badge.fury.io/py/dbt-oracle.svg)](https://badge.fury.io/py/dbt-oracle) * **Slack channel**: [#db-oracle](https://getdbt.slack.com/archives/C01PWH4TXLY) * **Supported dbt version**: v1.2.1 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Oracle 12c and higher #### Installing dbt-oracle Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-oracle` #### Configuring dbt-oracle For Oracle-specific configuration, please refer to [Oracle configs.](https://docs.getdbt.com/reference/resource-configs/oracle-configs.md) ##### Configure the Python driver mode [python-oracledb](https://oracle.github.io/python-oracledb/) makes it optional to install the Oracle Client libraries. This driver supports 2 modes 1. **Thin mode (preferred)**: Python process directly connects to the Oracle database. This mode does not need the Oracle Client libraries 2. **Thick mode**: Python process links with the Oracle Client libraries. Some advanced Oracle database functionalities (for example: Advanced Queuing, LDAP connections, Scrollable cursors) are currently available via Oracle Client libraries You can configure the driver mode using the environment variable `ORA_PYTHON_DRIVER_TYPE`. Use the **thin** mode as it vastly simplifies installation. | Driver Mode | Oracle Client libraries required? | Configuration | | ----------- | --------------------------------- | ------------------------------ | | Thin | No | `ORA_PYTHON_DRIVER_TYPE=thin` | | Thick | Yes | `ORA_PYTHON_DRIVER_TYPE=thick` | The default value of `ORA_PYTHON_DRIVER_TYPE` is `thin` ##### Thin ```bash export ORA_PYTHON_DRIVER_TYPE=thin # default ``` ##### Thick ```bash export ORA_PYTHON_DRIVER_TYPE=thick ``` ##### Install Oracle Instant Client libraries In thick mode, you will need the [Oracle Instant Client libraries](https://www.oracle.com/database/technologies/instant-client.html) installed. These provide the necessary network connectivity allowing dbt-oracle to access an Oracle Database instance. Oracle Client versions 23, 21, 19, 18, 12 and 11.2 are supported. It is recommended to use the latest client possible: Oracle’s standard client-server version interoperability allows connection to both older and newer databases. ##### Linux 1. Download an Oracle 23, 21, 19, 18, 12, or 11.2 “Basic” or “Basic Light” zip file matching your Python 64-bit or 32-bit architecture: 1. [x86-64 64-bit](https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html) 2. [x86 32-bit](https://www.oracle.com/database/technologies/instant-client/linux-x86-32-downloads.html) 3. [ARM (aarch64) 64-bit](https://www.oracle.com/database/technologies/instant-client/linux-arm-aarch64-downloads.html) 2. Unzip the package into a single directory that is accessible to your application. For example: ```bash mkdir -p /opt/oracle cd /opt/oracle unzip instantclient-basic-linux.x64-21.6.0.0.0.zip ``` 3. Install the libaio package with sudo or as the root user. For example: ```bash sudo yum install libaio ``` On some Linux distributions this package is called `libaio1` instead. 4. If there is no other Oracle software on the machine that will be impacted, permanently add Instant Client to the runtime link path. For example, with sudo or as the root user: ```bash sudo sh -c "echo /opt/oracle/instantclient_21_6 > /etc/ld.so.conf.d/oracle-instantclient.conf" sudo ldconfig ``` Alternatively, set the environment variable `LD_LIBRARY_PATH` ```bash export LD_LIBRARY_PATH=/opt/oracle/instantclient_21_6:$LD_LIBRARY_PATH ``` 5. If you use optional Oracle configuration files such as tnsnames.ora, sqlnet.ora, or oraaccess.xml with Instant Client, then put the files in an accessible directory and set the environment variable TNS\_ADMIN to that directory name. ```bash export TNS_ADMIN=/opt/oracle/your_config_dir ``` ##### Windows 1. Download an Oracle 21, 19, 18, 12, or 11.2 “Basic” or “Basic Light” zip file: [64-bit](https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html) or [32-bit](https://www.oracle.com/database/technologies/instant-client/microsoft-windows-32-downloads.html), matching your Python architecture. Windows 7 users Note that Oracle Client versions 21c and 19c are not supported on Windows 7. 2. Unzip the package into a directory that is accessible to your application. For example, unzip `instantclient-basic-windows.x64-19.11.0.0.0dbru.zip` to `C:\oracle\instantclient_19_11`. 3. Oracle Instant Client libraries require a Visual Studio redistributable with a 64-bit or 32-bit architecture to match Instant Client’s architecture. 1. For Instant Client 21 install [VS 2019](https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-170) or later 2. For Instant Client 19 install [VS 2017](https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-170) 3. For Instant Client 18 or 12.2 install [VS 2013](https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-170#visual-studio-2013-vc-120) 4. For Instant Client 12.1 install [VS 2010](https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-170#visual-studio-2010-vc-100-sp1-no-longer-supported) 5. For Instant Client 11.2 install [VS 2005 64-bit](https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-170#visual-studio-2005-vc-80-sp1-no-longer-supported) 4. Add the Oracle Instant Client directory to the `PATH` environment variable. The directory must occur in `PATH` before any other Oracle directories. Restart any open command prompt windows. ```bash SET PATH=C:\oracle\instantclient_19_9;%PATH% ``` ##### MacOS Check the python-oracledb documentation for installation instructions on [MacOS ARM64](https://python-oracledb.readthedocs.io/en/latest/user_guide/installation.html#instant-client-scripted-installation-on-macos-arm64) or [MacOS Intel x84-64](https://python-oracledb.readthedocs.io/en/latest/user_guide/installation.html#instant-client-scripted-installation-on-macos-intel-x86-64) #### Configure wallet for Oracle Autonomous Database (ADB-S) in Cloud dbt can connect to Oracle Autonomous Database (ADB-S) in Oracle Cloud using either TLS (Transport Layer Security) or mutual TLS (mTLS). TLS and mTLS provide enhanced security for authentication and encryption. A database username and password is still required for dbt connections which can be configured as explained in the next section [Connecting to Oracle Database](#connecting-to-oracle-database). ##### TLS With TLS, dbt can connect to Oracle ADB without using a wallet. Both Thin and Thick modes of the python-oracledb driver support TLS. info In Thick mode, dbt can connect through TLS only when using Oracle Client library versions 19.14 (or later) or 21.5 (or later). Refer to Oracle documentation to [connect to an ADB instance using TLS authentication](https://docs.oracle.com/en/cloud/paas/autonomous-database/adbsa/connecting-nodejs-tls.html#GUID-B3809B88-D2FB-4E08-8F9B-65A550F93A07) and the blog post [Easy wallet-less connections to Oracle Autonomous Databases in Python](https://blogs.oracle.com/opal/post/easy-way-to-connect-python-applications-to-oracle-autonomous-databases) to enable TLS for your Oracle ADB instance. ##### Mutual TLS For mutual TLS connections, a wallet needs to be downloaded from the OCI console and the Python driver needs to be configured to use it. ###### Install the Wallet and Network Configuration Files From the Oracle Cloud console for the database, download the wallet zip file using the `DB Connection` button. The zip contains the wallet and network configuration files. Note Keep wallet files in a secure location and share them only with authorized users. Unzip the wallet zip file. ##### Thin In Thin mode, only two files from the zip are needed: * `tnsnames.ora` - Maps net service names used for application connection strings to your database services * `ewallet.pem` - Enables SSL/TLS connections in Thin mode. Keep this file secure After unzipping the files in a secure directory, set the **TNS\_ADMIN** and **WALLET\_LOCATION** environment variables to the directory name. ```bash export WALLET_LOCATION=/path/to/directory_containing_ewallet.pem export WALLET_PASSWORD=*** export TNS_ADMIN=/path/to/directory_containing_tnsnames.ora ``` Optionally, if `ewallet.pem` file is encrypted using a wallet password, specify the password using environment variable **WALLET\_PASSWORD** ##### Thick In Thick mode, the following files from the zip are needed: * `tnsnames.ora` - Maps net service names used for application connection strings to your database services * `sqlnet.ora` - Configures Oracle Network settings * `cwallet.sso` - Enables SSL/TLS connections After unzipping the files in a secure directory, set the **TNS\_ADMIN** environment variable to that directory name. ```bash export TNS_ADMIN=/path/to/directory_containing_tnsnames.ora ``` Next, edit the `sqlnet.ora` file to point to the wallet directory. sqlnet.ora ```text WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY="/path/to/wallet/directory"))) SSL_SERVER_DN_MATCH=yes ``` #### Connecting to Oracle Database Define the following mandatory parameters as environment variables and refer them in the connection profile using [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) Jinja function. Optionally, you can also define these directly in the `profiles.yml` file, but this is not recommended. ```bash export DBT_ORACLE_USER=<username> export DBT_ORACLE_PASSWORD=*** export DBT_ORACLE_SCHEMA=<username> export DBT_ORACLE_DATABASE=example_db2022adb ``` Use the following query to retrieve the database name: ```sql SELECT SYS_CONTEXT('userenv', 'DB_NAME') FROM DUAL ``` An Oracle connection profile for dbt can be set using any one of the following methods ##### Using TNS alias The `tnsnames.ora` file is a configuration file that contains network service names mapped to connect descriptors. The directory location of `tnsnames.ora` file can be specified using `TNS_ADMIN` environment variable tnsnames.ora ```text db2022adb_high = (description = (retry_count=20)(retry_delay=3) (address=(protocol=tcps) (port=1522) (host=adb.example.oraclecloud.com)) (connect_data=(service_name=example_high.adb.oraclecloud.com)) (security=(ssl_server_cert_dn="CN=adb.example.oraclecloud.com, OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US"))) ``` The TNS alias `db2022adb_high` can be defined as an environment variable and referred to in `profiles.yml` ```bash export DBT_ORACLE_TNS_NAME=db2022adb_high ``` \~/.dbt/profiles.yml ```yaml dbt_test: target: dev outputs: dev: type: oracle user: "{{ env_var('DBT_ORACLE_USER') }}" pass: "{{ env_var('DBT_ORACLE_PASSWORD') }}" database: "{{ env_var('DBT_ORACLE_DATABASE') }}" tns_name: "{{ env_var('DBT_ORACLE_TNS_NAME') }}" schema: "{{ env_var('DBT_ORACLE_SCHEMA') }}" threads: 4 ``` ##### Using Connect string The connection string identifies which database service to connect to. It can be one of the following * An [Oracle Easy Connect String](https://docs.oracle.com/en/database/oracle/oracle-database/21/netag/configuring-naming-methods.html#GUID-B0437826-43C1-49EC-A94D-B650B6A4A6EE) * An Oracle Net Connect Descriptor String * A Net Service Name mapping to a connect descriptor ```bash export DBT_ORACLE_CONNECT_STRING="(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522) (host=adb.example.oraclecloud.com))(connect_data=(service_name=example_high.adb.oraclecloud.com)) (security=(ssl_server_cert_dn=\"CN=adb.example.oraclecloud.com, OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US\")))" ``` \~/.dbt/profiles.yml (Applies to dbt v1.11 and later) ```yaml dbt_test: target: "{{ env_var('DBT_ENGINE_TARGET', 'dev') }}" outputs: dev: type: oracle user: "{{ env_var('DBT_ORACLE_USER') }}" pass: "{{ env_var('DBT_ORACLE_PASSWORD') }}" database: "{{ env_var('DBT_ORACLE_DATABASE') }}" schema: "{{ env_var('DBT_ORACLE_SCHEMA') }}" connection_string: "{{ env_var('DBT_ORACLE_CONNECT_STRING') }}" ``` ##### Using Database hostname To connect using the database hostname or IP address, you need to specify the following * host * port (1521 or 1522) * protocol (tcp or tcps) * service ```bash export DBT_ORACLE_HOST=adb.example.oraclecloud.com export DBT_ORACLE_SERVICE=example_high.adb.oraclecloud.com ``` \~/.dbt/profiles.yml (Applies to dbt v1.11 and later) ```yaml dbt_test: target: "{{ env_var('DBT_ENGINE_TARGET', 'dev') }}" outputs: dev: type: oracle user: "{{ env_var('DBT_ORACLE_USER') }}" pass: "{{ env_var('DBT_ORACLE_PASSWORD') }}" protocol: "tcps" host: "{{ env_var('DBT_ORACLE_HOST') }}" port: 1522 service: "{{ env_var('DBT_ORACLE_SERVICE') }}" database: "{{ env_var('DBT_ORACLE_DATABASE') }}" schema: "{{ env_var('DBT_ORACLE_SCHEMA') }}" retry_count: 1 retry_delay: 3 threads: 4 ``` Note Starting with `dbt-oracle==1.0.2`, it is **optional** to set the `database` name in `profiles.yml` Starting with `dbt-oracle==1.8.0` database key in `profiles.yml` is **still optional for all but one** of the dbt-oracle workflows. if `database` is missing in `profiles.yml` the generated catalog used for project documentation will be empty. From `dbt-oracle==1.8`, we detect that `database` key is missing from `profiles.yml` and issue a warning to add it for catalog generation. The warning message also shows the database name that dbt-oracle expects. That way users don't have to worry about "what" the database name is and "how" to get it. ##### Quoting configuration The default quoting configuration used by dbt-oracle is shown below: dbt\_project.yaml ```yaml quoting: database: false identifier: false schema: false ``` This is recommended and works for most cases. ##### Approximate relation match error Often users have complained about an approximate relation match as shown below: ```text Compilation Error in model <model> 19:09:40 When searching for a relation, dbt found an approximate match. Instead of guessing 19:09:40 which relation to use, dbt will move on. Please delete <model>, or rename it to be less ambiguous. Searched for: <model> ``` This is reported in multiple channels: * [StackOverFlow Approximate relation Match Error](https://stackoverflow.com/questions/75892325/approximate-relation-match-with-dbt-on-oracle) * [Github Issue #51](https://github.com/oracle/dbt-oracle/issues/51) * [Github Issue #143](https://github.com/oracle/dbt-oracle/issues/143) * [Github Issue #144](https://github.com/oracle/dbt-oracle/issues/144) In all cases, the solution was to enable quoting only for the database. To solve this issue of `approximate match` use the following quoting configuration dbt\_project.yaml ```yaml quoting: database: true ``` #### Python models using Oracle Autonomous Database (ADB-S) Oracle's Autonomous Database Serverless (ADB-S) users can run dbt-py models using Oracle Machine Learning (OML4PY) which is available without any extra setup required. ##### Features * User Defined Python function is run in an ADB-S spawned Python 3.12.1 runtime * Access to external Python packages available in the Python runtime. For example, `numpy`, `pandas`, `scikit_learn` etc * Integration with Conda 24.x to create environments with custom Python packages * Access to Database session in the Python function * DataFrame read API to read `TABLES`, `VIEWS`, and ad-hoc `SELECT` queries as DataFrames * DataFrame write API to write DataFrames as `TABLES` * Supports both table and incremental materialization ##### Setup ###### Required roles * User must be non-ADMIN to execute the Python function * User must be granted the `OML_DEVELOPER` role ###### OML Cloud Service URL OML Cloud Service URL is of the following format: ```text https://tenant1-dbt.adb.us-sanjose-1.oraclecloudapps.com ``` In this example: * `tenant1` is the tenancy ID * `dbt` is the database name * `us-sanjose-1` is the datacenter region * `oraclecloudapps.com` is the root domain Add `oml_cloud_service_url` to your existing `~/.dbt/profiles.yml` \~/.dbt/profiles.yml ```yaml dbt_test: target: dev outputs: dev: type: oracle user: "{{ env_var('DBT_ORACLE_USER') }}" pass: "{{ env_var('DBT_ORACLE_PASSWORD') }}" database: "{{ env_var('DBT_ORACLE_DATABASE') }}" tns_name: "{{ env_var('DBT_ORACLE_TNS_NAME') }}" schema: "{{ env_var('DBT_ORACLE_SCHEMA') }}" oml_cloud_service_url: "https://tenant1-dbt.adb.us-sanjose-1.oraclecloudapps.com" ``` ##### Python model configurations | Configuration | Datatype | Examples | | --------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------- | | Materialization | String | `dbt.config(materialized="incremental")` or `dbt.config(materialized="table")` | | Service | String | `dbt.config(service="HIGH")` or `dbt.config(service="MEDIUM")` or `dbt.config(service="LOW")` | | Async Mode | Boolean | `dbt.config(async_flag=True)` | | Timeout in seconds only to be used with ***async*** mode (`min: 1800` and `max: 43200`) | Integer | `dbt.config(timeout=1800)` | | Conda environment | String | `dbt.config(conda_env_name="dbt_py_env")` | In async mode, dbt-oracle will schedule a Python job, poll the job's status, and wait for it to complete. Without async mode, dbt-oracle will immediately invoke the Python job in a blocking manner. Note Use `dbt.config(async_flag=True)` for long-running Python jobs. ##### Python model examples ###### Refer other model Use `dbt.ref(model_name)` to refer to either SQL or Python model ```python def model(dbt, session): # Must be either table or incremental (view is not currently supported) dbt.config(materialized="table") # returns oml.core.DataFrame referring a dbt model s_df = dbt.ref("sales_cost") return s_df ``` ###### Refer a source Use `dbt.source(source_schema, table_name)` ```python def model(dbt, session): # Must be either table or incremental (view is not currently supported) dbt.config(materialized="table") # oml.core.DataFrame representing a datasource s_df = dbt.source("sh_database", "channels") return s_df ``` ###### Incremental materialization ```python def model(dbt, session): # Must be either table or incremental dbt.config(materialized="incremental") # oml.DataFrame representing a datasource sales_cost_df = dbt.ref("sales_cost") if dbt.is_incremental: cr = session.cursor() result = cr.execute(f"select max(cost_timestamp) from {dbt.this.identifier}") max_timestamp = result.fetchone()[0] # filter new rows sales_cost_df = sales_cost_df[sales_cost_df["COST_TIMESTAMP"] > max_timestamp] return sales_cost_df ``` ###### Concatenate a new column in Dataframe ```python def model(dbt, session): dbt.config(materialized="table") dbt.config(async_flag=True) dbt.config(timeout=1800) sql = f"""SELECT customer.cust_first_name, customer.cust_last_name, customer.cust_gender, customer.cust_marital_status, customer.cust_street_address, customer.cust_email, customer.cust_credit_limit, customer.cust_income_level FROM sh.customers customer, sh.countries country WHERE country.country_iso_code = ''US'' AND customer.country_id = country.country_id""" # session.sync(query) will run the sql query and returns a oml.core.DataFrame us_potential_customers = session.sync(query=sql) # Compute an ad-hoc anomaly score on the credit limit median_credit_limit = us_potential_customers["CUST_CREDIT_LIMIT"].median() mean_credit_limit = us_potential_customers["CUST_CREDIT_LIMIT"].mean() anomaly_score = (us_potential_customers["CUST_CREDIT_LIMIT"] - median_credit_limit)/(median_credit_limit - mean_credit_limit) # Add a new column "CUST_CREDIT_ANOMALY_SCORE" us_potential_customers = us_potential_customers.concat({"CUST_CREDIT_ANOMALY_SCORE": anomaly_score.round(3)}) # Return potential customers dataset as a oml.core.DataFrame return us_potential_customers ``` ##### Use Custom Conda environment 1. As ADMIN user, create a conda environment using [OML4PY Conda Notebook](https://docs.oracle.com/en/database/oracle/machine-learning/oml4py/1/mlpug/administrative-task-create-and-conda-environments.html): ```bash conda create -n dbt_py_env -c conda-forge --override-channels --strict-channel-priority python=3.12.1 nltk gensim ``` 2. Save this environment using the following command from the OML4PY Conda Notebook: ```bash conda upload --overwrite dbt_py_env -t application OML4PY ``` 3. Use the environment in dbt Python models: ```python # Import custom packages from Conda environments import nltk import gensim def model(dbt, session): dbt.config(materialized="table") dbt.config(conda_env_name="dbt_py_env") # Refer the conda environment dbt.config(async_flag=True) # Use async mode for long running Python jobs dbt.config(timeout=900) # oml.core.DataFrame referencing a dbt-sql model promotion_cost = dbt.ref("direct_sales_channel_promo_cost") return promotion_cost ``` #### Supported features * Table materialization * View materialization * Materialized View * Incremental materialization * Seeds * Data sources * Singular tests * Generic tests; Not null, Unique, Accepted values and Relationships * Operations * Analyses * Exposures * Document generation * Serve project documentation as a website * Python Models (from dbt-oracle version 1.5.1) * Integration with Conda to use any Python packages from Anaconda's repository * All dbt commands are supported #### Not supported features * Ephemeral materialization --- ##### Connect Postgres to dbt v1 Local development `profiles.yml` file is for dbt v1 and dbt v2 only If you're using dbt platform, you don't need to create a `profiles.yml` file. This file is only necessary when you use v1 or v2 locally. To learn more about v2 prerequisites, refer to [Supported features](https://docs.getdbt.com/docs/dbt/supported-features.md). To connect your data platform to dbt, refer to [About data platforms](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * **Maintained by**: dbt Labs * **Authors**: core dbt maintainers * **GitHub repo**: [dbt-labs/dbt-adapters](https://github.com/dbt-labs/dbt-adapters) [![](https://img.shields.io/github/stars/dbt-labs/dbt-adapters?style=for-the-badge)](https://github.com/dbt-labs/dbt-adapters) * **PyPI package**: `dbt-postgres` [![](https://badge.fury.io/py/dbt-postgres.svg)](https://badge.fury.io/py/dbt-postgres) * **Slack channel**: [#db-postgres](https://getdbt.slack.com/archives/C0172G2E273) * **Supported dbt version**: v0.4.0 and newer * **dbt support**: Supported * **Minimum data platform version**: n/a #### Installing dbt-postgres Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-postgres` #### Configuring dbt-postgres For Postgres-specific configuration, please refer to [Postgres configs.](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md) #### Profile Configuration Postgres targets should be set up using the following configuration in your `profiles.yml` file. \~/.dbt/profiles.yml ```yaml company-name: target: dev outputs: dev: type: postgres host: [hostname] user: [username] password: [password] port: [port] dbname: [database name] # or database instead of dbname schema: [dbt schema] threads: [optional, 1 or more] keepalives_idle: 0 # default 0, indicating the system default. See below connect_timeout: 10 # default 10 seconds retries: 1 # default 1 retry on error/timeout when opening connections search_path: [optional, override the default postgres search_path] role: [optional, set the role dbt assumes when executing queries] sslmode: [optional, set the sslmode used to connect to the database] sslcert: [optional, set the sslcert to control the certificate file location] sslkey: [optional, set the sslkey to control the location of the private key] sslrootcert: [optional, set the sslrootcert config value to a new file path in order to customize the file location that contain root certificates] ``` ##### Configurations ###### search\_path The `search_path` config controls the Postgres "search path" that dbt configures when opening new connections to the database. By default, the Postgres search path is `"$user, public"`, meaning that unqualified table names will be searched for in the `public` schema, or a schema with the same name as the logged-in user. **Note:** Setting the `search_path` to a custom value is not necessary or recommended for typical usage of dbt. ###### role The `role` config controls the Postgres role that dbt assumes when opening new connections to the database. ###### sslmode The `sslmode` config controls how dbt connects to Postgres databases using SSL. See [the Postgres docs](https://www.postgresql.org/docs/9.1/libpq-ssl.html) on `sslmode` for usage information. When unset, dbt will connect to databases using the Postgres default, `prefer`, as the `sslmode`. ###### sslcert The `sslcert` config controls the location of the certificate file used to connect to Postgres when using client SSL connections. To use a certificate file that is not in the default location, set that file path using this value. Without this config set, dbt uses the Postgres default locations. See [Client Certificates](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-CLIENTCERT) in the Postgres SSL docs for the default paths. ###### sslkey The `sslkey` config controls the location of the private key for connecting to Postgres using client SSL connections. If this config is omitted, dbt uses the default key location for Postgres. See [Client Certificates](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-CLIENTCERT) in the Postgres SSL docs for the default locations. ###### sslrootcert When connecting to a Postgres server using a client SSL connection, dbt verifies that the server provides an SSL certificate signed by a trusted root certificate. These root certificates are in the `~/.postgresql/root.crt` file by default. To customize the location of this file, set the `sslrootcert` config value to a new file path. ##### `keepalives_idle` If the database closes its connection while dbt is waiting for data, you may see the error `SSL SYSCALL error: EOF detected`. Lowering the [`keepalives_idle` value](https://www.postgresql.org/docs/9.3/libpq-connect.html) may prevent this, because the server will send a ping to keep the connection active more frequently. [dbt's default setting](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/src/dbt/adapters/postgres/connections.py#L32) is 0 (the server's default value), but can be configured lower (perhaps 120 or 60 seconds), at the cost of a chattier network connection. ###### retries If `dbt-postgres` encounters an operational error or timeout when opening a new connection, it will retry up to the number of times configured by `retries`. The default value is 1 retry. If set to 2+ retries, dbt will wait 1 second before retrying. If set to 0, dbt will not retry at all. ##### `psycopg2` vs `psycopg2-binary` `psycopg2-binary` is installed by default when installing `dbt-postgres`. Installing `psycopg2-binary` uses a pre-built version of `psycopg2` which may not be optimized for your particular machine. This is ideal for development and testing workflows where performance is less of a concern and speed and ease of install is more important. However, production environments will benefit from a version of `psycopg2` which is built from source for your particular operating system, and architecture. In this scenario, speed and ease of install is less important as the on-going usage is the focus. To use `psycopg2`: 1. Install `dbt-postgres` 2. Uninstall `psycopg2-binary` 3. Install the equivalent version of `psycopg2` ```bash pip install dbt-postgres if [[ $(pip show psycopg2-binary) ]]; then PSYCOPG2_VERSION=$(pip show psycopg2-binary | grep Version | cut -d " " -f 2) pip uninstall -y psycopg2-binary && pip install psycopg2==$PSYCOPG2_VERSION fi ``` Installing `psycopg2` often requires OS level dependencies. These dependencies may vary across operating systems and architectures. For example, on Ubuntu, you need to install `libpq-dev` and `python-dev`: ```bash sudo apt-get update sudo apt-get install libpq-dev python-dev ``` whereas on Mac, you need to install `postgresql`: ```bash brew install postgresql pip install psycopg2 ``` Your OS may have its own dependencies based on your particular scenario. --- ##### Connect Redshift to dbt v2 (Applies to dbt v2.0 and later) Local development You can configure the Redshift adapter by running `dbt init` in your CLI or manually providing the `profiles.yml` file with the fields configured for your authentication type. The Redshift adapter for dbt v2 supports the following [authentication methods](#supported-authentication-types): * Password * IAM profile * IAM Identity Center (browser) #### Warehouse permissions The Redshift database user that dbt v2 uses must be able to run dbt workloads and read catalog metadata used for introspection. ##### Required Redshift objects Before connecting, these objects must exist or be accessible: | Object | Purpose | | ------------------------------------------------------- | --------------------------------- | | **Cluster** (provisioned) or **workgroup** (serverless) | Compute resource | | **Database** | Target database | | **Schema** | Target schema within the database | | **User** | Database user for authentication | | **IAM role or profile** (optional) | For IAM-based authentication | ##### Core permissions The following permissions are required for fundamental dbt features: | Permission | Object | Purpose | | ---------- | --------------- | ------------------------------------- | | `USAGE` | Schema | Access the schema | | `CREATE` | Schema | Create tables and views in the schema | | `SELECT` | Tables or views | Read data | | `INSERT` | Tables | Insert data | | `UPDATE` | Tables | Update data | | `DELETE` | Tables | Delete data | | `DROP` | Tables or views | Drop or replace objects | | `TRUNCATE` | Tables | Truncate tables | ##### Metadata operations dbt v2 queries these Redshift system relations: | System relation | Purpose | Permission required | | ------------------ | ------------------------------------------------------ | ------------------------- | | `SVV_ALL_COLUMNS` | Column metadata | SELECT on the system view | | `pg_class` | List relations | Access to system catalog | | `pg_namespace` | Schema information | Access to system catalog | | `sys_query_detail` | Source freshness (last insert time) | SELECT on the system view | | `svv_table_info` | List materialized views when the project includes them | SELECT on the system view | | `svv_mv_info` | List materialized views when the project includes them | SELECT on the system view | ##### Schema management Conditional permissions for schema management | Permission | Object | When required | | --------------- | -------- | ------------------- | | `CREATE SCHEMA` | Database | Auto-create schemas | For example SQL grants in Redshift, refer to [Redshift permissions](https://docs.getdbt.com/reference/database-permissions/redshift-permissions.md). #### Configure dbt v2 Executing `dbt init` in your CLI will prompt for the following fields: * **Host:** The hostname of your Redshift cluster * **User:** Username of the account that will be connecting to the database * **Database:** The database name * **Schema:** The schema name * **Port (default: 5439):** Port for your Redshift environment Alternatively, you can manually create the `profiles.yml` file and configure the fields. See examples in [authentication](#supported-authentication-types) section for formatting. If there is an existing `profiles.yml` file, you are given the option to retain the existing fields or overwrite them. Next, select your authentication method. Follow the on-screen prompts to provide the required information. #### Supported authentication types ##### Password Use your Redshift user's password to authenticate. You can also manually enter it in plain text into the `profiles.yml` file configuration. ###### Example password configuration profiles.yml ```yml default: target: dev outputs: dev: type: redshift port: 5439 database: JAFFLE_SHOP schema: JAFFLE_TEST ra3_node: true method: database host: ABC123.COM user: JANE.SMITH@YOURCOMPANY.COM password: ABC123 threads: 16 ``` ##### IAM profile Specify the IAM profile to use to connect your v2 sessions. You will need to provide the following information: * **IAM Profile:** The profile name * **Cluster ID:** The unique identifier for your AWS cluster * **Region:** Your AWS region (for example, us-east-1) * **Use RA3 node type (y/n):** Use high performance AWS RA3 node ###### Example IAM profile configuration profiles.yml ```yml default: target: dev outputs: dev: type: redshift port: 5439 database: JAFFLE_SHOP schema: JAFFLE_TEST ra3_node: false method: iam host: YOURHOSTNAME.COM user: JANE.SMITH@YOURCOMPANY.COM iam_profile: YOUR_PROFILE_NAME cluster_id: ABC123 region: us-east-1 threads: 16 ``` ##### IAM Identity Center (browser) Sign in through [AWS IAM Identity Center](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html) and set `method` to `browser_identity_center`. dbt opens a browser window for you to authenticate without needing a password or IAM profile in your `profiles.yml`. Local only This method works when you run dbt locally from the command line. It isn't supported in the dbt platform yet. Before you start, make sure your Redshift cluster or workgroup is set up for [IAM Identity Center integration](https://docs.aws.amazon.com/redshift/latest/mgmt/redshift-iam-access-control-idp-connect.html). You need the following fields: | Profile field | Required | Default | Description | | ------------------------- | -------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | `method` | Yes | — | Set to `browser_identity_center`. | | `idc_region`\* | Yes | — | AWS region where your IAM Identity Center instance lives (for example, `us-east-1`). | | `issuer_url` | Yes | — | Issuer URL of your IAM Identity Center instance (for example, `https://identitycenter.amazonaws.com/ssoins-1234567890abcdef`). | | `idp_listen_port` | No | `7890` | Local port dbt listens on for the browser redirect after you sign in. | | `idc_client_display_name` | No | `Amazon Redshift driver` | App name shown in the browser consent prompt. | | `idp_response_timeout` | No | `60` | Seconds dbt waits for you to finish signing in before timing out. | \*`idc_region` isn't the same as your AWS account region. dbt doesn't support setting `region` alongside `browser_identity_center` in `profiles.yml` yet, so it reads your AWS region from your AWS config file (`~/.aws/config`). Make sure a default region is set there (for example, by running `aws configure`). ###### Example IAM Identity Center configuration profiles.yml ```yml default: target: dev outputs: dev: type: redshift method: browser_identity_center host: hostname.region.redshift.amazonaws.com port: 5439 dbname: analytics schema: analytics idc_region: us-east-1 issuer_url: https://identitycenter.amazonaws.com/ssoins-1234567890abcdef # Optional idp_listen_port: 7890 idc_client_display_name: Amazon Redshift driver idp_response_timeout: 60 threads: 4 ``` #### Identifier case Redshift lowercases names, including quoted names, unless `enable_case_sensitive_identifier` is set to `true` in the cluster's parameter group or the Serverless workgroup's configuration. dbt doesn't enable this setting. Refer to [Configuring quoting](https://docs.getdbt.com/reference/project-configs/quoting.md#redshift). #### More information Find Redshift-specific configuration information in the [Redshift adapter reference guide](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md). --- ##### Connect RisingWave to dbt v1 Local development Vendor-supported plugin Certain core functionality may vary. If you would like to report a bug, request a feature, or contribute, you can check out the linked repository and open an issue. * **Maintained by**: RisingWave * **Authors**: Dylan Chen * **GitHub repo**: [risingwavelabs/dbt-risingwave](https://github.com/risingwavelabs/dbt-risingwave) [![](https://img.shields.io/github/stars/risingwavelabs/dbt-risingwave?style=for-the-badge)](https://github.com/risingwavelabs/dbt-risingwave) * **PyPI package**: `dbt-risingwave` [![](https://badge.fury.io/py/dbt-risingwave.svg)](https://badge.fury.io/py/dbt-risingwave) * **Slack channel**: [N/A](https://www.risingwave.com/slack) * **Supported dbt version**: v1.6.1 and newer * **dbt support**: Not Supported * **Minimum data platform version**: #### Installing dbt-risingwave Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-risingwave` #### Configuring dbt-risingwave For RisingWave-specific configuration, please refer to [RisingWave configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) #### Connecting to RisingWave with dbt-risingwave Before connecting to RisingWave, ensure that RisingWave is installed and running. For more information about how to get RisingWave up and running, see the [RisingWave quick start guide](https://docs.risingwave.com/get-started/quickstart). To connect to RisingWave with dbt, you need to add a RisingWave profile to your dbt profile file (`~/.dbt/profiles.yml`). Below is an example RisingWave profile. Revise the field values when necessary. \~/.dbt/profiles.yml ```yaml default: outputs: dev: type: risingwave host: [host name] user: [user name] pass: [password] dbname: [database name] port: [port] schema: [dbt schema] target: dev ``` | Field | Description | | -------- | ------------------------------------------------------ | | `host` | The host name or IP address of the RisingWave instance | | `user` | The RisingWave database user you want to use | | `pass` | The password of the database user | | `dbname` | The RisingWave database name | | `port` | The port number that RisingWave listens on | | `schema` | The schema of the RisingWave database | To test the connection to RisingWave, run: ```bash dbt debug ``` #### Materializations The dbt models for managing data transformations in RisingWave are similar to typical dbt SQL models. In the `dbt-risingwave` adapter, we have customized some of the materializations to align with the streaming data processing model of RisingWave. | Materializations | Supported | Notes | | ---------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `table` | Yes | Creates a [table](https://docs.risingwave.com/sql/commands/sql-create-table). To use this materialization, add `{{ config(materialized='table') }}` to your model SQL files. | | `view` | Yes | Creates a [view](https://docs.risingwave.com/sql/commands/sql-create-view). To use this materialization, add `{{ config(materialized='view') }}` to your model SQL files. | | `ephemeral` | Yes | This materialization uses [common table expressions](https://docs.risingwave.com/sql/query-syntax/with-clause) in RisingWave under the hood. To use this materialization, add `{{ config(materialized='ephemeral') }}` to your model SQL files. | | `materializedview` | To be deprecated. | It is available only for backward compatibility purposes (for v1.5.1 of the dbt-risingwave adapter plugin). If you are using v1.6.0 and later versions of the dbt-risingwave adapter plugin, use `materialized_view` instead. | | `materialized_view` | Yes | Creates a [materialized view](https://docs.risingwave.com/sql/commands/sql-create-mv). This materialization corresponds the `incremental` one in dbt. To use this materialization, add `{{ config(materialized='materialized_view') }}` to your model SQL files. | | `incremental` | No | Please use `materialized_view` instead. Since RisingWave is designed to use materialized view to manage data transformation in an incremental way, you can just use the `materialized_view` materialization. | | `source` | Yes | Creates a [source](https://docs.risingwave.com/sql/commands/sql-create-source). To use this materialization, add {{ config(materialized='source') }} to your model SQL files. You need to provide your create source statement as a whole in this model. See [Example model files](https://docs.risingwave.com/integrations/other/dbt#example-model-files) for details. | | `table_with_connector` | Yes | Creates a table with connector settings. In RisingWave, a table with connector settings is similar to a source. The difference is that a table object with connector settings persists raw streaming data in the source, while a source object does not. To use this materialization, add `{{ config(materialized='table_with_connector') }}` to your model SQL files. You need to provide your create table with connector statement as a whole in this model (see [Example model files](https://docs.risingwave.com/integrations/other/dbt#example-model-files) for details). Because dbt tables have their own semantics, RisingWave use `table_with_connector` to distinguish itself from a dbt table. | | `sink` | Yes | Creates a [sink](https://docs.risingwave.com/sql/commands/sql-create-sink). To use this materialization, add `{{ config(materialized='sink') }}` to your SQL files. You need to provide your create sink statement as a whole in this model. See [Example model files](https://docs.risingwave.com/integrations/other/dbt#example-model-files) for details. | #### Resources * [RisingWave's guide about using dbt for data transformations](https://docs.risingwave.com/integrations/other/dbt) * [A demo project using dbt to manage Nexmark benchmark queries in RisingWave](https://github.com/risingwavelabs/dbt_rw_nexmark) --- ##### Connect Rockset to dbt v1 Local development Vendor-supported plugin Certain core functionality may vary. If you would like to report a bug, request a feature, or contribute, you can check out the linked repository and open an issue. * **Maintained by**: Rockset, Inc. * **Authors**: Rockset, Inc. * **GitHub repo**: [rockset/dbt-rockset](https://github.com/rockset/dbt-rockset) [![](https://img.shields.io/github/stars/rockset/dbt-rockset?style=for-the-badge)](https://github.com/rockset/dbt-rockset) * **PyPI package**: `dbt-rockset` [![](https://badge.fury.io/py/dbt-rockset.svg)](https://badge.fury.io/py/dbt-rockset) * **Slack channel**: [#dbt-rockset](https://getdbt.slack.com/archives/C02J7AZUAMN) * **Supported dbt version**: v0.19.2 and newer * **dbt support**: Not Supported * **Minimum data platform version**: ? #### Installing dbt-rockset Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-rockset` #### Configuring dbt-rockset For Rockset-specific configuration, please refer to [Rockset configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) #### Connecting to Rockset with **dbt-rockset** The dbt profile for Rockset is very simple and contains the following fields: profiles.yml ```yaml rockset: target: dev outputs: dev: type: rockset workspace: [schema] api_key: [api_key] api_server: [api_server] # (Default is api.rs2.usw2.rockset.com) ``` ##### Materializations | Type | Supported? | Details | | ----------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------- | | view | YES | Creates a [view](https://rockset.com/docs/views/#gatsby-focus-wrapper). | | table | YES | Creates a [collection](https://rockset.com/docs/collections/#gatsby-focus-wrapper). | | ephemeral | YES | Executes queries using CTEs. | | incremental | YES | Creates a [collection](https://rockset.com/docs/collections/#gatsby-focus-wrapper) if it doesn't exist, and then writes results to it. | #### Caveats 1. `unique_key` is not supported with incremental, unless it is set to [\_id](https://rockset.com/docs/special-fields/#the-_id-field), which acts as a natural `unique_key` in Rockset anyway. 2. The `table` materialization is slower in Rockset than most due to Rockset's architecture as a low-latency, real-time database. Creating new collections requires provisioning hot storage to index and serve fresh data, which takes about a minute. 3. Rockset queries have a two-minute timeout. Any model which runs a query that takes longer to execute than two minutes will fail. --- ##### Connect SingleStore to dbt v1 Local development Vendor-supported plugin Certain core functionality may vary. If you would like to report a bug, request a feature, or contribute, you can check out the linked repository and open an issue. * **Maintained by**: SingleStore, Inc. * **Authors**: SingleStore, Inc. * **GitHub repo**: [memsql/dbt-singlestore](https://github.com/memsql/dbt-singlestore) [![](https://img.shields.io/github/stars/memsql/dbt-singlestore?style=for-the-badge)](https://github.com/memsql/dbt-singlestore) * **PyPI package**: `dbt-singlestore` [![](https://badge.fury.io/py/dbt-singlestore.svg)](https://badge.fury.io/py/dbt-singlestore) * **Slack channel**: [db-singlestore](https://getdbt.slack.com/archives/C02V2QHFF7U) * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Not supported * **Minimum data platform version**: v7.5 #### Installing dbt-singlestore Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-singlestore` #### Configuring dbt-singlestore For SingleStore-specific configuration, please refer to [SingleStore configs.](https://docs.getdbt.com/reference/resource-configs/singlestore-configs.md) ##### Set up a SingleStore Target SingleStore targets should be set up using the following configuration in your `profiles.yml` file. If you are using SingleStore Managed Service, required connection details can be found on your Cluster Page under "Connect" -> "SQL IDE" tab. \~/.dbt/profiles.yml ```yaml singlestore: target: dev outputs: dev: type: singlestore host: [hostname] # optional, default localhost port: [port number] # optional, default 3306 user: [user] # optional, default root password: [password] # optional, default empty database: [database name] # required schema: [prefix for tables that dbt will generate] # required threads: [1 or more] # optional, default 1 ``` It is recommended to set optional parameters as well. ##### Description of SingleStore Profile Fields | Field | Required | Description | | ---------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Yes | Must be set to `singlestore`. This must be included either in `profiles.yml` or in the `dbt_project.yml` file. | | `host` | No | The host name of the SingleStore server to connect to. | | `user` | No | Your SingleStore database username. | | `password` | No | Your SingleStore database password. | | `database` | Yes | The name of your database. If you are using custom database names in your models config, they must be created prior to running those models. | | `schema` | Yes | The string to prefix the names of generated tables if `generate_alias_name` macro is added (see below). If you are using a custom schema name in your model config, it will be concatenated with the one specified in profile using `_`. | | `threads` | No | The number of threads available to dbt. | #### Schema and Concurrent Development SingleStore doesn't have a concept of `schema` that corresponds to one used in `dbt` (namespace within a database). `schema` in your profile is required for `dbt` to work correctly with your project metadata. For example, you will see it on "dbt docs" page, even though it's not present in the database. In order to support concurrent development, `schema` can be used to prefix table names that `dbt` is building within your database. In order to enable this, add the following macro to your project. This macro will take the `schema` field of your `profiles.yml` file and use it as a table name prefix. ```sql -- macros/generate_alias_name.sql {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {%- if custom_alias_name is none -%} {{ node.schema }}__{{ node.name }} {%- else -%} {{ node.schema }}__{{ custom_alias_name | trim }} {%- endif -%} {%- endmacro %} ``` Therefore, if you set `schema=dev` in your `.dbt/profiles.yml` file and run the `customers` model with the corresponding profile, `dbt` will create a table named `dev__customers` in your database. --- ##### Connect Snowflake to dbt v2 (Applies to dbt v2.0 and later) Local development You can configure the Snowflake adapter by running `dbt init` in your CLI or manually providing the `profiles.yml` file with the fields configured for your authentication type. To check out which Snowflake functions are supported in dbt v2 in `strict` mode, refer to [Snowflake function support](https://docs.getdbt.com/reference/resource-configs/snowflake-function-support.md). The Snowflake adapter for dbt v2 supports the following [authentication methods](#supported-authentication-types): * Password * Key pair * Single sign-on (SSO) * Password with MFA note [Snowflake is deprecating single-access password login](https://docs.snowflake.com/en/user-guide/security-mfa-rollout). Individual developers should use MFA or SSO instead of password authentication. Password-based login remains supported for service users (Snowflake user type: `LEGACY_SERVICE`). #### Warehouse permissions The Snowflake user or service account that dbt v2 connects as must be able to run dbt workloads (queries, metadata, and typical materializations). Grant privileges through a Snowflake role assigned to that user. ##### Required Snowflake objects Before connecting, these objects must exist: | Object | Purpose | | ------------- | -------------------------------------------------- | | **Account** | Your Snowflake account identifier | | **User** | Service account or user for dbt v2 | | **Role** | Role assigned to the user with required privileges | | **Warehouse** | Virtual warehouse for compute | | **Database** | Target database or databases for dbt models | | **Schema** | Target schema or schemas within the database | ##### Core operations The following are required permissions for fundamental dbt operations: | Permission | Object | Purpose | | -------------- | --------------- | ------------------------------------------ | | `USAGE` | Warehouse | Execute queries | | `USAGE` | Database | Access the database | | `USAGE` | Schema | Access schemas | | `SELECT` | Tables or views | Read existing data and sources | | `CREATE TABLE` | Schema | Create models materialized as tables | | `CREATE VIEW` | Schema | Create models materialized as views | | `INSERT` | Tables | Load data into tables | | `UPDATE` | Tables | Incremental model updates | | `DELETE` | Tables | Incremental models using delete and insert | | `TRUNCATE` | Tables | Full refresh of incremental models | | `DROP` | Tables or views | Replace existing objects | ##### Metadata operations The following are required permissions for dbt metadata operations: | Permission | Object | Purpose | | --------------------- | ------------------- | ------------------------------- | | `USAGE` | INFORMATION\_SCHEMA | Query table and column metadata | | `DESCRIBE TABLE` | Tables | Get schema information | | `SHOW OBJECTS` | Schema | List relations in schema | | `SHOW USER FUNCTIONS` | Schema | Discover UDFs (if used) | ##### Schema and database management The following are conditional permissions for schema and database management: | Permission | Object | When required | | ----------------- | -------- | -------------------------------- | | `CREATE SCHEMA` | Database | dbt should auto-create schemas | | `CREATE DATABASE` | Account | dbt should auto-create databases | ##### Advanced features The following are optional permissions for advanced features: | Permission | Object | Feature | | -------------------- | ------ | ----------------------------------- | | `CREATE STAGE` | Schema | File staging for seeds or snapshots | | `CREATE FILE FORMAT` | Schema | Custom file formats | | `CREATE SEQUENCE` | Schema | Sequences | | `EXECUTE TASK` | Schema | Snowflake tasks | For role examples and SQL grants in Snowflake, you can also refer to [Snowflake permissions](https://docs.getdbt.com/reference/database-permissions/snowflake-permissions.md). #### Snowflake configuration details The information required for configuring the Snowflake adapter can be found conveniently in your Snowflake account menu: 1. Click on your name from the Snowflake sidebar. 2. Hover over the **Account** field. 3. In the field with your account name, click **View account details**. 4. Click **Config file** and select the appropriate **Warehouse** and **Database**. [![Sample config file in Snowflake.](/img/v2/connect-adapters/snowflake-account-details.png?v=2 "Sample config file in Snowflake.")](#)Sample config file in Snowflake. #### Configure dbt v2 Executing `dbt init` in your CLI will prompt for the following fields: * **Account:** Snowflake account number * **User:** Your Snowflake username * **Database:** The database within your Snowflake account to connect to your project * **Warehouse:** The compute warehouse that will handle the tasks for your project * **Schema:** The development/staging/deployment schema for the project * **Role (Optional):** The role dbt should assume when connecting to the warehouse Alternatively, you can manually create the `profiles.yml` file and configure the fields. See examples in [authentication](#supported-authentication-types) section for formatting. If there is an existing `profiles.yml` file, you are given the option to retain the existing fields or overwrite them. Next, select your authentication method. Follow the on-screen prompts to provide the required information. #### Supported authentication types ##### Password Password authentication prompts for your Snowflake account password. This is becoming an increasingly less common option as organizations opt for more secure authentication. Selecting **Password with MFA** redirects you to the Snowflake account login to provide your passkey or authenticator password. ###### Example password configuration profiles.yml ```yml default: target: dev outputs: dev: type: snowflake threads: 16 account: ABC123 user: JANE.SMITH@YOURCOMPANY.COM database: JAFFLE_SHOP warehouse: TRANSFORM schema: JANE_SMITH password: THISISMYPASSWORD ``` ###### Example password with MFA configuration profiles.yml ```yml default: target: dev outputs: dev: type: snowflake threads: 16 authenticator: username_password_mfa account: ABC123 user: JANE.SMITH@YOURCOMPANY.COM database: JAFFLE_SHOP warehouse: TRANSFORM schema: JANE_SMITH ``` ##### Key pair Key pair authentication gives you the option to: * Define the path to the key. * Provide the plain-text PEM format key inline. We recommend using PKCS#8 format with AES-256 encryption for key pair authentication with dbt v2. dbt v2 doesn't support legacy 3DES encryption or headerless key formats. Using older key formats may cause authentication failures. If you encounter the `Key is PKCS#1 (RSA private key). Snowflake requires PKCS#8` error, then your private key is in the wrong format. You have two options: * (Recommended fix) Re-export your key with modern encryption: ```bash # Convert to PKCS#8 with AES-256 encryption openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 aes-256-cbc -inform PEM -out rsa_key.p8 ``` * (Temporary workaround) Add the `BEGIN` header and `END` footer to your PEM body: ```text -----BEGIN ENCRYPTED PRIVATE KEY----- < Your existing encrypted private key contents > -----END ENCRYPTED PRIVATE KEY----- ``` Once the key is configured, you will be given the option to provide a passphrase, if required. ###### Example key pair configuration profiles.yml ```yml default: target: dev outputs: dev: type: snowflake threads: 16 account: ABC123 user: JANE.SMITH@YOURCOMPANY.COM database: JAFFLE_SHOP warehouse: TRANSFORM schema: JANE_SMITH private_key: '<Your existing encrypted private key contents>' private_key_passphrase: YOURPASSPHRASEHERE ``` ##### Single sign-on Single sign-on will leverage your browser to authenticate the Snowflake session. By default, every connection that dbt opens will require you to re-authenticate in a browser. The Snowflake connector package supports caching your session token, but it [currently only supports Windows and Mac OS](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#optional-using-connection-caching-to-minimize-the-number-of-prompts-for-authentication). Refer to the [Snowflake docs](https://docs.snowflake.com/en/sql-reference/parameters.html#label-allow-id-token) for information on enabling this feature in your account. ###### Example SSO configuration profiles.yml ```yml default: target: dev outputs: dev: type: snowflake threads: 16 authenticator: externalbrowser account: ABC123 user: JANE.SMITH@YOURCOMPANY.COM database: JAFFLE_SHOP warehouse: TRANSFORM schema: JANE_SMITH ``` #### More information Find Snowflake-specific configuration information in the [Snowflake adapter reference guide](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md). --- ##### Connect SQLite to dbt v1 Local development Community plugin Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below. * **Maintained by**: Community * **Authors**: Jeff Chiu (https://github.com/codeforkjeff) * **GitHub repo**: [codeforkjeff/dbt-sqlite](https://github.com/codeforkjeff/dbt-sqlite) [![](https://img.shields.io/github/stars/codeforkjeff/dbt-sqlite?style=for-the-badge)](https://github.com/codeforkjeff/dbt-sqlite) * **PyPI package**: `dbt-sqlite` [![](https://badge.fury.io/py/dbt-sqlite.svg)](https://badge.fury.io/py/dbt-sqlite) * **Slack channel**: [n/a](https://www.getdbt.com/community) * **Supported dbt version**: v1.1.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: SQlite Version 3.0 #### Installing dbt-sqlite Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-sqlite` #### Configuring dbt-sqlite For SQLite-specific configuration, please refer to [SQLite configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) Starting with the release of dbt 1.0.0, versions of dbt-sqlite are aligned to the same major+minor [version](https://semver.org/) of dbt v1. * versions 1.1.x of this adapter work with dbt 1.1.x * versions 1.0.x of this adapter work with dbt 1.0.x #### Connecting to SQLite with dbt-sqlite SQLite targets should be set up using the following configuration in your `profiles.yml` file. Example: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sqlite threads: 1 database: 'database' schema: 'main' schemas_and_paths: main: 'file_path/database_name.db' schema_directory: 'file_path' #optional fields extensions: - "/path/to/sqlean/crypto.so" ``` ###### Description of SQLite Profile Fields | Field | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Required. Must be set to `sqlite`. | | `threads` | Required. Must be set to `1`. SQLite locks the whole db on writes so anything > 1 won't help. | | `database` | Required but the value is arbitrary because there is no 'database' portion of relation names in SQLite so it gets stripped from the output of ref() and from SQL everywhere. It still needs to be set in the configuration and is used by dbt internally. | | `schema` | Value of 'schema' must be defined in schema\_paths below. in most cases, this should be main. | | `schemas_and_paths` | Connect schemas to paths: at least one of these must be 'main' | | `schema_directory` | Directory where all \*.db files are attached as schema, using base filename as schema name, and where new schemas are created. This can overlap with the dirs of files in schemas\_and\_paths as long as there's no conflicts. | | `extensions` | Optional. List of file paths of SQLite extensions to load. crypto.so is needed for snapshots to work; see SQLite Extensions below. | #### Caveats * Schemas are implemented as attached database files. (SQLite conflates databases and schemas.) * SQLite automatically assigns 'main' to the file you initially connect to, so this must be defined in your profile. Other schemas defined in your profile get attached when database connection is created. * If dbt needs to create a new schema, it will be created in `schema_directory` as `schema_name.db`. Dropping a schema results in dropping all its relations and detaching the database file from the session. * Schema names are stored in view definitions, so when you access a non-'main' database file outside dbt, you'll need to attach it using the same name, or the views won't work. * SQLite does not allow views in one schema (i.e. database file) to reference objects in another schema. You'll get this error from SQLite: "view \[someview] cannot reference objects in database \[somedatabase]". You must set `materialized='table'` in models that reference other schemas. * Materializations are simplified: they drop and re-create the model, instead of doing the backup-and-swap-in new model that the other dbt database adapters support. This choice was made because SQLite doesn't support `DROP ... CASCADE` or `ALTER VIEW` or provide information about relation dependencies in something information\_schema-like. These limitations make it really difficult to make the backup-and-swap-in functionality work properly. Given how SQLite aggressively [locks](https://sqlite.org/lockingv3.html) the database anyway, it's probably not worth the effort. #### SQLite Extensions For snapshots to work, you'll need the `crypto` module from SQLean to get an `md5()` function. It's recommended that you install all the SQLean modules, as they provide many common SQL functions missing from SQLite. Precompiled binaries are available for download from the [SQLean github repository page](https://github.com/nalgeon/sqlean). You can also compile them yourself if you want. Point to these module files in your profile config as shown in the example above. Mac OS seems to ship with [SQLite libraries that do not have support for loading extensions compiled in](https://docs.python.org/3/library/sqlite3.html#f1), so this won't work "out of the box." Accordingly, snapshots won't work. If you need snapshot functionality, you'll need to compile SQLite/python or find a python distribution for Mac OS with this support. --- ##### Connect Starburst/Trino to dbt v1 Local development `profiles.yml` file is for dbt v1 and dbt v2 only If you're using dbt platform, you don't need to create a `profiles.yml` file. This file is only necessary when you use v1 or v2 locally. To learn more about v2 prerequisites, refer to [Supported features](https://docs.getdbt.com/docs/dbt/supported-features.md). To connect your data platform to dbt, refer to [About data platforms](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * **Maintained by**: Starburst Data, Inc. * **Authors**: Marius Grama, Przemek Denkiewicz, Michiel de Smet, Damian Owsianny * **GitHub repo**: [starburstdata/dbt-trino](https://github.com/starburstdata/dbt-trino) [![](https://img.shields.io/github/stars/starburstdata/dbt-trino?style=for-the-badge)](https://github.com/starburstdata/dbt-trino) * **PyPI package**: `dbt-trino` [![](https://badge.fury.io/py/dbt-trino.svg)](https://badge.fury.io/py/dbt-trino) * **Slack channel**: [#db-starburst-and-trino](https://getdbt.slack.com/archives/CNNPBQ24R) * **Supported dbt version**: v0.20.0 and newer * **dbt support**: Supported * **Minimum data platform version**: n/a #### Installing dbt-trino Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-trino` #### Configuring dbt-trino For Starburst/Trino-specific configuration, please refer to [Starburst/Trino configs.](https://docs.getdbt.com/reference/resource-configs/trino-configs.md) #### Connecting to Starburst/Trino To connect to a data platform with dbt v1, create appropriate *profile* and *target* YAML keys/values in the `profiles.yml` configuration file for your Starburst/Trino clusters. This dbt YAML file lives in the `.dbt/` directory of your user/home directory. For more information, refer to [Connection profiles](https://docs.getdbt.com/docs/local/profiles.yml.md) and [profiles.yml](https://docs.getdbt.com/docs/local/profiles.yml.md). The parameters for setting up a connection are for Starburst Enterprise, Starburst Galaxy, and Trino clusters. Unless specified, "cluster" will mean any of these products' clusters. #### Host parameters The following profile fields are always required except for `user`, which is also required unless you're using the `oauth`, `oauth_console`, `cert`, or `jwt` authentication methods. | Field | Example | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `host` | `mycluster.mydomain.com`<br /><br />Format for Starburst Galaxy:<br />- `mygalaxyaccountname-myclustername.trino.galaxy.starburst.io` | The hostname of your cluster.<br /><br />Don't include the `http://` or `https://` prefix. | | `database` | `my_postgres_catalog` | The name of a catalog in your cluster. | | `schema` | `my_schema` | The name of a schema within your cluster's catalog.<br /><br />It's *not recommended* to use schema names that have upper case or mixed case letters. | | `port` | `443` | The port to connect to your cluster. By default, it's 443 for TLS enabled clusters. | | `user` | Format for Starburst Enterprise or Trino:<br />- `user.name`<br />- `user.name@mydomain.com`<br />Format for Starburst Galaxy:<br />- `user.name@mydomain.com/role` | The username (of the account) to log in to your cluster. When connecting to Starburst Galaxy clusters, you must include the role of the user as a suffix to the username. | ##### Roles in Starburst Enterprise If connecting to a Starburst Enterprise cluster with built-in access controls enabled, you must specify a role using the format detailed in [Additional parameters](#additional-parameters). If a role is not specified, the default role for the provided username is used. ##### Schemas and databases When selecting the catalog and the schema, make sure the user has read and write access to both. This selection does not limit your ability to query the catalog. Instead, they serve as the default location for where tables and views are materialized. In addition, the Trino connector used in the catalog must support creating tables. This *default* can be changed later from within your dbt project. #### Additional parameters The following profile fields are optional to set up. They let you configure your cluster's session and dbt for your connection. | Profile field | Example | Description | | ------------------------------------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `threads` | `8` | How many threads dbt should use (default is `1`) | | `roles` | `system: analyst` | Catalog roles can be set under the optional `roles` parameter using the following format: `catalog: role`. | | `session_properties` | `query_max_run_time: 4h` | Sets Trino session properties used in the connection. Execute `SHOW SESSION` to see available options | | `prepared_statements_enabled` | `true` or `false` | Enable usage of Trino prepared statements (used in `dbt seed` commands) (default: `true`) | | `retries` | `10` | Configure how many times all database operation is retried when connection issues arise (default: `3`) | | `timezone` | `Europe/Brussels` | The time zone for the Trino session (default: client-side local timezone) | | `client_tags` | `['analytics', 'fault-tolerant']` | A list of client tags sent to Trino in the `X-Trino-Client-Tags` header. For configuration details, refer to [Query routing](https://docs.getdbt.com/reference/resource-configs/trino-configs.md#query-routing). | | `http_headers` | `X-Trino-Client-Info: dbt-trino` | HTTP headers to send alongside requests to Trino, specified as a YAML dictionary of header names and values. For configuration details, refer to [Query routing](https://docs.getdbt.com/reference/resource-configs/trino-configs.md#query-routing). | | `http_scheme` | `https` or `http` | The HTTP scheme to use for requests to Trino (default: `http`, or `https` if `kerberos`, `ldap` or `jwt`) | | `starburst_url` | `https://my-tenant.galaxy.starburst.io` | The base URL of your Starburst Data Discovery REST API. Must use `https`. Setting this opts in to syncing the model and column descriptions written by [`persist_docs`](https://docs.getdbt.com/reference/resource-configs/trino-configs.md#persist_docs) to Starburst's Data Discovery catalog. Available in `dbt-trino` v1.10.3 and later. | | `starburst_client_id` | `{{ env_var('DBT_ENV_SECRET_STARBURST_CLIENT_ID') }}` | The client ID used to authenticate with the Starburst Data Discovery API. Required when `starburst_url` is set. Available in `dbt-trino` v1.10.3 and later. | | `starburst_secret_key` | `{{ env_var('DBT_ENV_SECRET_STARBURST_SECRET_KEY') }}` | The secret key used to authenticate with the Starburst Data Discovery API. Required when `starburst_url` is set. To avoid storing this secret in `profiles.yml`, source it from an environment variable. Available in `dbt-trino` v1.10.3 and later. | | `starburst_metadata_failure_strategy` | `continue_on_error` (default) or `fail_fast` | The approach for handling errors from the Data Discovery API when syncing `persist_docs` metadata: `continue_on_error` logs a warning and continues, while `fail_fast` aborts the run. Available in `dbt-trino` v1.10.3 and later. | | `starburst_max_column_batch_size` | `100` (default) | The maximum number of column descriptions sent per batch request when syncing `persist_docs` metadata to Data Discovery. Available in `dbt-trino` v1.10.3 and later. | #### Authentication parameters The authentication methods that dbt v1 supports are: * `ldap` — LDAP (username and password) * `kerberos` — Kerberos * `jwt` — JSON Web Token (JWT) * `certificate` — Certificate-based authentication * `oauth` — Open Authentication (OAuth) * `oauth_console` — Open Authentication (OAuth) with authentication URL printed to the console * `none` — None, no authentication Set the `method` field to the authentication method you intend to use for the connection. For a high-level introduction to authentication in Trino, see [Trino Security: Authentication types](https://trino.io/docs/current/security/authentication-types.html). Click on one of these authentication methods for further details on how to configure your connection profile. Each tab also includes an example `profiles.yml` configuration file for you to review. ##### LDAP The following table lists the authentication parameters to set for LDAP. For more information, refer to [LDAP authentication](https://trino.io/docs/current/security/ldap.html) in the Trino docs. | Profile field | Example | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `method` | `ldap` | Set LDAP as the authentication method. | | `user` | Format for Starburst Enterprise or Trino:<br />- `user.name`<br />- `user.name@mydomain.com`<br />Format for Starburst Galaxy:<br />- `user.name@mydomain.com/role` | The username (of the account) to log in to your cluster. When connecting to Starburst Galaxy clusters, you must include the role of the user as a suffix to the username. | | `password` | `abc123` | Password for authentication. | | `impersonation_user` (optional) | `impersonated_tom` | Override the provided username. This lets you impersonate another user. | <br /> ###### Example profiles.yml for LDAP \~/.dbt/profiles.yml ```yaml trino: target: dev outputs: dev: type: trino method: ldap user: [user] password: [password] host: [hostname] database: [database name] schema: [your dbt schema] port: [port number] threads: [1 or more] ``` ##### Kerberos The following table lists the authentication parameters to set for Kerberos. For more information, refer to [Kerberos authentication](https://trino.io/docs/current/security/kerberos.html) in the Trino docs. | Profile field | Example | Description | | ------------------------------------------- | ------------------- | --------------------------------------------------------------- | | `method` | `kerberos` | Set Kerberos as the authentication method. | | `user` | `commander` | Username for authentication | | `keytab` | `/tmp/trino.keytab` | Path to keytab | | `krb5_config` | `/tmp/krb5.conf` | Path to config | | `principal` | `trino@EXAMPLE.COM` | Principal | | `service_name` (optional) | `abc123` | Service name (default is `trino`) | | `hostname_override` (optional) | `EXAMPLE.COM` | Kerberos hostname for a host whose DNS name doesn't match | | `mutual_authentication` (optional) | `false` | Boolean flag for mutual authentication | | `force_preemptive` (optional) | `false` | Boolean flag to preemptively initiate the Kerberos GSS exchange | | `sanitize_mutual_error_response` (optional) | `true` | Boolean flag to strip content and headers from error responses | | `delegate` (optional) | `false` | Boolean flag for credential delegation (`GSS_C_DELEG_FLAG`) | <br /> ###### Example profiles.yml for Kerberos \~/.dbt/profiles.yml ```yaml trino: target: dev outputs: dev: type: trino method: kerberos user: commander keytab: /tmp/trino.keytab krb5_config: /tmp/krb5.conf principal: trino@EXAMPLE.COM host: trino.example.com port: 443 database: analytics schema: public ``` ##### JWT The following table lists the authentication parameters to set for JSON Web Token. For more information, refer to [JWT authentication](https://trino.io/docs/current/security/jwt.html) in the Trino docs. | Profile field | Example | Description | | ------------- | ------------------- | ------------------------------------- | | `method` | `jwt` | Set JWT as the authentication method. | | `jwt_token` | `aaaaa.bbbbb.ccccc` | The JWT string. | <br /> ###### Example profiles.yml for JWT \~/.dbt/profiles.yml ```yaml trino: target: dev outputs: dev: type: trino method: jwt jwt_token: [my_long_jwt_token_string] host: [hostname] database: [database name] schema: [your dbt schema] port: [port number] threads: [1 or more] ``` ##### Certificate The following table lists the authentication parameters to set for certificates. For more information, refer to [Certificate authentication](https://trino.io/docs/current/security/certificate.html) in the Trino docs. | Profile field | Example | Description | | -------------------- | -------------- | -------------------------------------------------- | | `method` | `certificate` | Set certificate-based authentication as the method | | `client_certificate` | `/tmp/tls.crt` | Path to client certificate | | `client_private_key` | `/tmp/tls.key` | Path to client private key | | `cert` | | The full path to a certificate file | <br /> ###### Example profiles.yml for certificate \~/.dbt/profiles.yml ```yaml trino: target: dev outputs: dev: type: trino method: certificate cert: [path/to/cert_file] client_certificate: [path/to/client/cert] client_private_key: [path to client key] database: [database name] schema: [your dbt schema] port: [port number] threads: [1 or more] ``` ##### OAuth The only authentication parameter to set for OAuth 2.0 is `method: oauth`. If you're using Starburst Enterprise or Starburst Galaxy, you must enable OAuth 2.0 in Starburst before you can use this authentication method. For more information, refer to both [OAuth 2.0 authentication](https://trino.io/docs/current/security/oauth2.html) in the Trino docs and the [README](https://github.com/trinodb/trino-python-client#oauth2-authentication) for the Trino Python client. It's recommended that you install `keyring` to cache the OAuth 2.0 token over multiple dbt invocations by running `python -m pip install 'trino[external-authentication-token-cache]'`. The `keyring` package is not installed by default. ###### Example profiles.yml for OAuth ```yaml sandbox-galaxy: target: oauth outputs: oauth: type: trino method: oauth host: bunbundersders.trino.galaxy-dev.io catalog: dbt_target schema: dataders port: 443 ``` ##### OAuth (console) The only authentication parameter to set for OAuth 2.0 is `method: oauth_console`. If you're using Starburst Enterprise or Starburst Galaxy, you must enable OAuth 2.0 in Starburst before you can use this authentication method. For more information, refer to both [OAuth 2.0 authentication](https://trino.io/docs/current/security/oauth2.html) in the Trino docs and the [README](https://github.com/trinodb/trino-python-client#oauth2-authentication) for the Trino Python client. The only difference between `oauth_console` and `oauth` is: * `oauth` — An authentication URL automatically opens in a browser. * `oauth_console` — A URL is printed to the console. It's recommended that you install `keyring` to cache the OAuth 2.0 token over multiple dbt invocations by running `python -m pip install 'trino[external-authentication-token-cache]'`. The `keyring` package is not installed by default. ###### Example profiles.yml for OAuth ```yaml sandbox-galaxy: target: oauth_console outputs: oauth: type: trino method: oauth_console host: bunbundersders.trino.galaxy-dev.io catalog: dbt_target schema: dataders port: 443 ``` ##### None You don't need to set up authentication (`method: none`), however, dbt Labs strongly discourages people from using it in any real application. Its use case is only for toy purposes (as in to play around with it), like local examples such as running Trino and dbt entirely within a single Docker container. ###### Example profiles.yml for no authentication \~/.dbt/profiles.yml ```yaml trino: target: dev outputs: dev: type: trino method: none user: commander host: trino.example.com port: 443 database: analytics schema: public ``` --- ##### Connect StarRocks to dbt v1 Local development #### Overview of dbt-starrocks * **Maintained by**: Starrocks * **Authors**: Astralidea * **GitHub repo**: [StarRocks/dbt-starrocks](https://github.com/StarRocks/dbt-starrocks)[![](https://img.shields.io/github/stars/StarRocks/dbt-starrocks?style=for-the-badge)](https://github.com/StarRocks/dbt-starrocks) * **PyPI package**: `dbt-starrocks` [![](https://badge.fury.io/py/dbt-starrocks.svg)](https://badge.fury.io/py/dbt-starrocks) * **Slack channel**: [#db-starrocks](https://www.getdbt.com/community) * **Supported dbt v1 version**: v1.6.2 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Starrocks 2.5 #### Installing dbt-starrocks pip is the easiest way to install the adapter: `python -m pip install dbt-starrocks` Installing `dbt-starrocks` will also install `dbt-core` and any other dependencies. #### Configuring dbt-starrocks For Starrocks-specifc configuration please refer to [Starrocks Configuration](https://docs.getdbt.com/reference/resource-configs/starrocks-configs.md) For further info, refer to the GitHub repository: [StarRocks/dbt-starrocks](https://github.com/StarRocks/dbt-starrocks) #### Authentication Methods ##### User / Password Authentication Starrocks can be configured using basic user/password authentication as shown below. \~/.dbt/profiles.yml ```yaml my-starrocks-db: target: dev outputs: dev: type: starrocks host: localhost port: 9030 schema: analytics # User/password auth username: your_starrocks_username password: your_starrocks_password ``` ###### Description of Profile Fields | Option | Description | Required? | Example | | -------- | ------------------------------------------------------ | --------- | ------------------------------ | | type | The specific adapter to use | Required | `starrocks` | | host | The hostname to connect to | Required | `192.168.100.28` | | port | The port to use | Required | `9030` | | schema | Specify the schema (database) to build models into | Required | `analytics` | | username | The username to use to connect to the server | Required | `dbt_admin` | | password | The password to use for authenticating to the server | Required | `correct-horse-battery-staple` | | version | Let Plugin try to go to a compatible starrocks version | Optional | `3.1.0` | #### Supported features | Starrocks <= 2.5 | Starrocks 2.5 ~ 3.1 | Starrocks >= 3.1 | Feature | | ---------------- | -------------------- | ---------------- | --------------------------------- | | ✅ | ✅ | ✅ | Table materialization | | ✅ | ✅ | ✅ | View materialization | | ❌ | ❌ | ✅ | Materialized View materialization | | ❌ | ✅ | ✅ | Incremental materialization | | ❌ | ✅ | ✅ | Primary Key Model | | ✅ | ✅ | ✅ | Sources | | ✅ | ✅ | ✅ | Custom data tests | | ✅ | ✅ | ✅ | Docs generate | | ❌ | ❌ | ❌ | Kafka | ##### Notice 1. When StarRocks Version < 2.5, `Create table as` can only set engine='OLAP' and table\_type='DUPLICATE' 2. When StarRocks Version >= 2.5, `Create table as` supports table\_type='PRIMARY' 3. When StarRocks Version < 3.1 distributed\_by is required It is recommended to use the latest starrocks version and dbt-starrocks version for the best experience. --- ##### Connect Teradata to dbt v1 Local development Some core functionality may be limited. If you're interested in contributing, check out the source code in the repository listed in the next section. `profiles.yml` file is for dbt v1 and dbt v2 only If you're using dbt platform, you don't need to create a `profiles.yml` file. This file is only necessary when you use v1 or v2 locally. To learn more about v2 prerequisites, refer to [Supported features](https://docs.getdbt.com/docs/dbt/supported-features.md). To connect your data platform to dbt, refer to [About data platforms](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * **Maintained by**: Teradata * **Authors**: Teradata * **GitHub repo**: [Teradata/dbt-teradata](https://github.com/Teradata/dbt-teradata) [![](https://img.shields.io/github/stars/Teradata/dbt-teradata?style=for-the-badge)](https://github.com/Teradata/dbt-teradata) * **PyPI package**: `dbt-teradata` [![](https://badge.fury.io/py/dbt-teradata.svg)](https://badge.fury.io/py/dbt-teradata) * **Slack channel**: [#db-teradata](https://getdbt.slack.com/archives/C027B6BHMT3) * **Supported dbt version**: v0.21.0 and newer * **dbt support**: Supported * **Minimum data platform version**: n/a #### Installing dbt-teradata Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-teradata` #### Configuring dbt-teradata For Teradata-specific configuration, please refer to [Teradata configs.](https://docs.getdbt.com/reference/resource-configs/teradata-configs.md) #### Python compatibility | Plugin version | Python 3.9 | Python 3.10 | Python 3.11 | Python 3.12 | Python 3.13 | | -------------- | ---------- | ----------- | ----------- | ----------- | ----------- | | 1.0.0.x | ✅ | ❌ | ❌ | ❌ | ❌ | | 1.1.x.x | ✅ | ✅ | ❌ | ❌ | ❌ | | 1.2.x.x | ✅ | ✅ | ❌ | ❌ | ❌ | | 1.3.x.x | ✅ | ✅ | ❌ | ❌ | ❌ | | 1.4.x.x | ✅ | ✅ | ✅ | ❌ | ❌ | | 1.5.x | ✅ | ✅ | ✅ | ❌ | ❌ | | 1.6.x | ✅ | ✅ | ✅ | ❌ | ❌ | | 1.7.x | ✅ | ✅ | ✅ | ❌ | ❌ | | 1.8.x | ✅ | ✅ | ✅ | ✅ | ❌ | | 1.9.x | ✅ | ✅ | ✅ | ✅ | ❌ | | 1.10.x | ✅ | ✅ | ✅ | ✅ | ✅ | #### dbt dependent packages version compatibility | dbt-teradata | dbt v1 | dbt-teradata-util | dbt-util | | ------------ | ------ | ----------------- | -------------- | | 1.2.x | 1.2.x | 0.1.0 | 0.9.x or below | | 1.6.7 | 1.6.7 | 1.1.1 | 1.1.1 | | 1.7.x | 1.7.x | 1.1.1 | 1.1.1 | | 1.8.x | 1.8.x | 1.1.1 | 1.1.1 | | 1.8.x | 1.8.x | 1.2.0 | 1.2.0 | | 1.8.x | 1.8.x | 1.3.0 | 1.3.0 | | 1.9.x | 1.9.x | 1.3.0 | 1.3.0 | | 1.10.x | 1.10.x | 1.3.0 | 1.3.0 | ##### Connecting to Teradata To connect to Teradata Vantage from dbt, you'll need to add a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) to your `profiles.yml` file. A Teradata profile conforms to the following syntax: profiles.yml ```yaml <profile-name>: target: <target-name> outputs: <target-name>: type: teradata user: <username> password: <password> schema: <database-name> tmode: ANSI threads: [optional, 1 or more] #optional fields <field-name: <field-value> ``` ###### Description of Teradata Profile Fields The following fields are required: | Parameter | Default | Type | Description | | ---------- | -------- | ------ | -------------------------------------------------------------------------------------------------------- | | `user` | | string | Specifies the database username. Equivalent to the Teradata JDBC Driver `USER` connection parameter. | | `password` | | string | Specifies the database password. Equivalent to the Teradata JDBC Driver `PASSWORD` connection parameter. | | `schema` | | string | Specifies the initial database to use after logon, instead of the user's default database. | | `tmode` | `"ANSI"` | string | Specifies the transaction mode. Only `ANSI` mode is currently supported. | The plugin also supports the following optional connection parameters: | Parameter | Default | Type | Description | | --------------------- | -------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `account` | | string | Specifies the database account. Equivalent to the Teradata JDBC Driver `ACCOUNT` connection parameter. | | `browser` | | string | Specifies the command to open the browser for Browser Authentication, when logmech is BROWSER. Browser Authentication is supported for Windows and macOS. Equivalent to the Teradata JDBC Driver BROWSER connection parameter. | | `browser_tab_timeout` | `"5"` | quoted integer | Specifies the number of seconds to wait before closing the browser tab after Browser Authentication is completed. The default is 5 seconds. The behavior is under the browser's control, and not all browsers support automatic closing of browser tabs. | | `browser_timeout` | `"180"` | quoted integer | Specifies the number of seconds that the driver will wait for Browser Authentication to complete. The default is 180 seconds (3 minutes). | | `column_name` | `"false"` | quoted boolean | Controls the behavior of cursor `.description` sequence `name` items. Equivalent to the Teradata JDBC Driver `COLUMN_NAME` connection parameter. False specifies that a cursor `.description` sequence `name` item provides the AS-clause name if available, or the column name if available, or the column title. True specifies that a cursor `.description` sequence `name` item provides the column name if available, but has no effect when StatementInfo parcel support is unavailable. | | `connect_timeout` | `"10000"` | quoted integer | Specifies the timeout in milliseconds for establishing a TCP socket connection. Specify 0 for no timeout. The default is 10 seconds (10000 milliseconds). | | `cop` | `"true"` | quoted boolean | Specifies whether COP Discovery is performed. Equivalent to the Teradata JDBC Driver `COP` connection parameter. | | `coplast` | `"false"` | quoted boolean | Specifies how COP Discovery determines the last COP hostname. Equivalent to the Teradata JDBC Driver `COPLAST` connection parameter. When `coplast` is `false` or omitted, or COP Discovery is turned off, then no DNS lookup occurs for the coplast hostname. When `coplast` is `true`, and COP Discovery is turned on, then a DNS lookup occurs for a coplast hostname. | | `port` | `"1025"` | quoted integer | Specifies the database port number. Equivalent to the Teradata JDBC Driver `DBS_PORT` connection parameter. | | `encryptdata` | `"false"` | quoted boolean | Controls encryption of data exchanged between the driver and the database. Equivalent to the Teradata JDBC Driver `ENCRYPTDATA` connection parameter. | | `fake_result_sets` | `"false"` | quoted boolean | Controls whether a fake result set containing statement metadata precedes each real result set. | | `field_quote` | `"\""` | string | Specifies a single character string used to quote fields in a CSV file. | | `field_sep` | `","` | string | Specifies a single character string used to separate fields in a CSV file. Equivalent to the Teradata JDBC Driver `FIELD_SEP` connection parameter. | | `host` | | string | Specifies the database hostname. | | `https_port` | `"443"` | quoted integer | Specifies the database port number for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `HTTPS_PORT` connection parameter. | | `lob_support` | `"true"` | quoted boolean | Controls LOB support. Equivalent to the Teradata JDBC Driver `LOB_SUPPORT` connection parameter. | | `log` | `"0"` | quoted integer | Controls debug logging. Somewhat equivalent to the Teradata JDBC Driver `LOG` connection parameter. This parameter's behavior is subject to change in the future. This parameter's value is currently defined as an integer in which the 1-bit governs function and method tracing, the 2-bit governs debug logging, the 4-bit governs transmit and receive message hex dumps, and the 8-bit governs timing. Compose the value by adding together 1, 2, 4, and/or 8. | | `logdata` | | string | Specifies extra data for the chosen logon authentication method. Equivalent to the Teradata JDBC Driver `LOGDATA` connection parameter. | | `logon_timeout` | `"0"` | quoted integer | Specifies the logon timeout in seconds. Zero means no timeout. | | `logmech` | `"TD2"` | string | Specifies the logon authentication method. Equivalent to the Teradata JDBC Driver `LOGMECH` connection parameter. Possible values are `TD2` (the default), `JWT`, `LDAP`, `BROWSER`, `KRB5` for Kerberos, or `TDNEGO`. | | `max_message_body` | `"2097000"` | quoted integer | Specifies the maximum Response Message size in bytes. Equivalent to the Teradata JDBC Driver `MAX_MESSAGE_BODY` connection parameter. | | `partition` | `"DBC/SQL"` | string | Specifies the database partition. Equivalent to the Teradata JDBC Driver `PARTITION` connection parameter. | | `request_timeout` | `"0"` | quoted integer | Specifies the timeout for executing each SQL request. Zero means no timeout. | | `retries` | `0` | integer | Allows an adapter to automatically try again when the attempt to open a new connection on the database has a transient, infrequent error. This option can be set using the retries configuration. Default value is 0. The default wait period between connection attempts is one second. retry\_timeout (seconds) option allows us to adjust this waiting period. | | `runstartup` | "false" | quoted boolean | Controls whether the user's STARTUP SQL request is executed after logon. For more information, refer to User STARTUP SQL Request. Equivalent to the Teradata JDBC Driver RUNSTARTUP connection parameter. If retries is set to 3, the adapter will try to establish a new connection three times if an error occurs. | | `sessions` | | quoted integer | Specifies the number of data transfer connections for FastLoad or FastExport. The default (recommended) lets the database choose the appropriate number of connections. Equivalent to the Teradata JDBC Driver SESSIONS connection parameter. | | `sip_support` | `"true"` | quoted boolean | Controls whether StatementInfo parcel is used. Equivalent to the Teradata JDBC Driver `SIP_SUPPORT` connection parameter. | | `sp_spl` | `"true"` | quoted boolean | Controls whether stored procedure source code is saved in the database when a SQL stored procedure is created. Equivalent to the Teradata JDBC Driver SP\_SPL connection parameter. | | `sslca` | | string | Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Equivalent to the Teradata JDBC Driver `SSLCA` connection parameter. | | `sslcrc` | `"ALLOW"` | string | Equivalent to the Teradata JDBC Driver SSLCRC connection parameter. Values are case-insensitive.<br />• ALLOW provides "soft fail" behavior such that communication failures are ignored during certificate revocation checking.<br />• REQUIRE mandates that certificate revocation checking must succeed. | | `sslcapath` | | string | Specifies a directory of PEM files that contain Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Only files with an extension of `.pem` are used. Other files in the specified directory are not used. Equivalent to the Teradata JDBC Driver `SSLCAPATH` connection parameter. | | `sslcipher` | | string | Specifies the TLS cipher for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `SSLCIPHER` connection parameter. | | `sslmode` | `"PREFER"` | string | Specifies the mode for connections to the database. Equivalent to the Teradata JDBC Driver `SSLMODE` connection parameter.<br />• `DISABLE` disables HTTPS/TLS connections and uses only non-TLS connections.<br />• `ALLOW` uses non-TLS connections unless the database requires HTTPS/TLS connections.<br />• `PREFER` uses HTTPS/TLS connections unless the database does not offer HTTPS/TLS connections.<br />• `REQUIRE` uses only HTTPS/TLS connections.<br />• `VERIFY-CA` uses only HTTPS/TLS connections and verifies that the server certificate is valid and trusted.<br />• `VERIFY-FULL` uses only HTTPS/TLS connections, verifies that the server certificate is valid and trusted, and verifies that the server certificate matches the database hostname. | | `sslprotocol` | `"TLSv1.2"` | string | Specifies the TLS protocol for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `SSLPROTOCOL` connection parameter. | | `teradata_values` | `"true"` | quoted boolean | Controls whether `str` or a more specific Python data type is used for certain result set column value types. | | `query_band` | `"org=teradata-internal-telem;appname=dbt;"` | string | Specifies the Query Band string to be set for each SQL request. | Refer to [connection parameters](https://github.com/Teradata/python-driver#connection-parameters) for the full description of the connection parameters. #### Supported features ##### Materializations * `view` * `table` * `ephemeral` * `incremental` ###### Incremental Materialization The following incremental materialization strategies are supported: * `append` (default) * `delete+insert` * `merge` * `valid_history` * `microbatch` info * To learn more about dbt incremental strategies, refer to [the dbt incremental strategy documentation](https://docs.getdbt.com/docs/build/incremental-strategy.md). * To learn more about `valid_history` incremental strategy, refer to [Teradata configs](https://docs.getdbt.com/reference/resource-configs/teradata-configs.md). ##### Commands All dbt commands are supported. #### Support for model contracts Model contracts are supported with dbt-teradata v1.7.1 and onwards. Constraint support and enforcement in dbt-teradata: | Constraint type | Support Platform | enforcement | | --------------- | ---------------- | ----------- | | not\_null | ✅ Supported | ✅ Enforced | | primary\_key | ✅ Supported | ✅ Enforced | | foreign\_key | ✅ Supported | ✅ Enforced | | unique | ✅ Supported | ✅ Enforced | | check | ✅ Supported | ✅ Enforced | Refer to [Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) for more info. #### Support for `dbt-utils` package `dbt-utils` package is supported through `teradata/teradata_utils` dbt package. The package provides a compatibility layer between `dbt_utils` and `dbt-teradata`. See [teradata\_utils](https://hub.getdbt.com/teradata/teradata_utils/latest/) package for install instructions. ##### Cross DB macros Starting with release 1.3, some macros were migrated from [teradata-dbt-utils](https://github.com/Teradata/dbt-teradata-utils) dbt package to the connector. Refer the following table for the macros supported by the connector. For using cross-DB macros, teradata-utils as a macro namespace will not be used, as cross-DB macros have been migrated from teradata-utils to Dbt-Teradata. ###### Compatibility | Macro Group | Macro Name | Status | Comment | | --------------------- | ------------------ | ------ | ------------------------------------------------------------ | | Cross-database macros | current\_timestamp | ✅ | custom macro provided | | Cross-database macros | dateadd | ✅ | custom macro provided | | Cross-database macros | datediff | ✅ | custom macro provided, see [compatibility note](#datediff) | | Cross-database macros | split\_part | ✅ | custom macro provided | | Cross-database macros | date\_trunc | ✅ | custom macro provided | | Cross-database macros | hash | ✅ | custom macro provided, see [compatibility note](#hash) | | Cross-database macros | replace | ✅ | custom macro provided | | Cross-database macros | type\_string | ✅ | custom macro provided | | Cross-database macros | last\_day | ✅ | no customization needed, see [compatibility note](#last_day) | | Cross-database macros | width\_bucket | ✅ | no customization | | Cross-database macros | generate\_series | ✅ | custom macro provided | | Cross-database macros | date\_spine | ✅ | no customization | ###### examples for cross DB macros ###### []()replace {{ dbt.replace("string\_text\_column", "old\_chars", "new\_chars") }} {{ replace('abcgef', 'g', 'd') }} ###### []()date\_trunc {{ dbt.date\_trunc("date\_part", "date") }} {{ dbt.date\_trunc("DD", "'2018-01-05 12:00:00'") }} ###### []()datediff `datediff` macro in teradata supports difference between dates. Difference between timestamps is not supported. ###### []()hash `Hash` macro needs an `md5` function implementation. Teradata doesn't support `md5` natively. You need to install a User Defined Function (UDF) and optionally specify `md5_udf` [variable](https://docs.getdbt.com/docs/build/project-variables.md). If not specified the code defaults to using `GLOBAL_FUNCTIONS.hash_md5`. See the following instructions on how to install the custom UDF: 1. Download the md5 UDF implementation from Teradata (registration required): <https://downloads.teradata.com/download/extensibility/md5-message-digest-udf>. 2. Unzip the package and go to `src` directory. 3. Start up `bteq` and connect to your database. 4. Create database `GLOBAL_FUNCTIONS` that will host the UDF. You can't change the database name as it's hardcoded in the macro: ```sql CREATE DATABASE GLOBAL_FUNCTIONS AS PERMANENT = 60e6, SPOOL = 120e6; ``` 5. Create the UDF. Replace `<CURRENT_USER>` with your current database user: ```sql GRANT CREATE FUNCTION ON GLOBAL_FUNCTIONS TO <CURRENT_USER>; DATABASE GLOBAL_FUNCTIONS; .run file = hash_md5.btq ``` 6. Grant permissions to run the UDF with grant option. ```sql GRANT EXECUTE FUNCTION ON GLOBAL_FUNCTIONS TO PUBLIC WITH GRANT OPTION; ``` Instruction on how to add md5\_udf variable in dbt\_project.yml for custom hash function: ```yaml vars: md5_udf: Custom_database_name.hash_method_function ``` ###### []()last\_day `last_day` in `teradata_utils`, unlike the corresponding macro in `dbt_utils`, doesn't support `quarter` datepart. dbt-teradata 1.8.0 and later versions support unit tests, enabling you to validate SQL models and logic with a small set of static inputs before going to production. This feature enhances test-driven development and boosts developer efficiency and code reliability. Learn more about dbt unit tests [here](https://docs.getdbt.com/docs/build/unit-tests.md). #### Limitations ##### Browser authentication * When running a dbt job with logmech set to "browser", the initial authentication opens a browser window where you must enter your username and password. * After authentication, this window remains open, requiring you to manually switch back to the dbt console. * For every subsequent connection, a new browser tab briefly opens, displaying the message "TERADATA BROWSER AUTHENTICATION COMPLETED," and silently reuses the existing session. * However, the focus stays on the browser window, so you’ll need to manually switch back to the dbt console each time. * This behavior is the default functionality of the teradatasql driver and cannot be avoided at this time. * To prevent session expiration and the need to re-enter credentials, ensure the authentication browser window stays open until the job is complete. ##### Transaction mode Both ANSI and TERA modes are now supported in dbt-teradata. TERA mode's support is introduced with dbt-teradata 1.7.1, it is an initial implementation. TERA transaction mode This is an initial implementation of the TERA transaction mode and may not support some use cases. We highly recommend validating all records or transformations using this mode to avoid unexpected issues or errors. #### Credits The adapter was originally created by [Doug Beatty](https://github.com/dbeatty10). Teradata took over the adapter in January 2022. We are grateful to Doug for founding the project and accelerating the integration of dbt + Teradata. #### License The adapter is published using Apache-2.0 License. Refer to the [terms and conditions](https://github.com/dbt-labs/dbt/blob/main/LICENSE) to understand items such as creating derivative work and the support model. --- ##### Connect TiDB to dbt v1 Local development Vendor-supported plugin Some [core functionality](https://github.com/pingcap/dbt-tidb/blob/main/README.md#supported-features) may be limited. If you're interested in contributing, check out the source code repository listed below. * **Maintained by**: PingCAP * **Authors**: Xiang Zhang, Qiang Wu, Yuhang Shi * **GitHub repo**: [pingcap/dbt-tidb](https://github.com/pingcap/dbt-tidb) [![](https://img.shields.io/github/stars/pingcap/dbt-tidb?style=for-the-badge)](https://github.com/pingcap/dbt-tidb) * **PyPI package**: `dbt-tidb` [![](https://badge.fury.io/py/dbt-tidb.svg)](https://badge.fury.io/py/dbt-tidb) * **Slack channel**: [#db-tidb](https://getdbt.slack.com/archives/C03CC86R1NY) * **Supported dbt version**: v1.0.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-tidb Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-tidb` #### Configuring dbt-tidb For TiDB-specific configuration, please refer to [TiDB configs.](https://docs.getdbt.com/reference/resource-configs/no-configs.md) #### Connecting to TiDB with **dbt-tidb** ##### User / Password Authentication Configure your dbt profile for using TiDB: ###### TiDB connection profile profiles.yml ```yaml dbt-tidb: target: dev outputs: dev: type: tidb server: 127.0.0.1 port: 4000 schema: database_name username: tidb_username password: tidb_password # optional retries: 3 # default 1 ``` ###### Description of Profile Fields | Option | Description | Required? | Example | | -------- | ---------------------------------------------------- | --------- | ------------------ | | type | The specific adapter to use | Required | `tidb` | | server | The server (hostname) to connect to | Required | `yourorg.tidb.com` | | port | The port to use | Required | `4000` | | schema | Specify the schema (database) to build models into | Required | `analytics` | | username | The username to use to connect to the server | Required | `dbt_admin` | | password | The password to use for authenticating to the server | Required | `awesome_password` | | retries | The retry times after an unsuccessful connection | Optional | `default 1` | #### Database User Privileges Your database user would be able to have some abilities to read or write, such as `SELECT`, `CREATE`, and so on. You can find some help [here](https://docs.pingcap.com/tidb/v4.0/privilege-management) with TiDB privileges management. | Required Privilege | | ---------------------- | | SELECT | | CREATE | | CREATE TEMPORARY TABLE | | CREATE VIEW | | INSERT | | DROP | | SHOW DATABASE | | SHOW VIEW | | SUPER | #### Supported features | TiDB 4.X | TiDB 5.0 ~ 5.2 | TiDB >= 5.3 | Feature | | -------- | --------------- | ----------- | --------------------------- | | ✅ | ✅ | ✅ | Table materialization | | ✅ | ✅ | ✅ | View materialization | | ❌ | ❌ | ✅ | Incremental materialization | | ❌ | ✅ | ✅ | Ephemeral materialization | | ✅ | ✅ | ✅ | Seeds | | ✅ | ✅ | ✅ | Sources | | ✅ | ✅ | ✅ | Custom data tests | | ✅ | ✅ | ✅ | Docs generate | | ❌ | ❌ | ✅ | Snapshots | | ✅ | ✅ | ✅ | Grant | | ✅ | ✅ | ✅ | Connection retry | **Note:** * TiDB 4.0 ~ 5.0 does not support [CTE](https://docs.pingcap.com/tidb/dev/sql-statement-with), you should avoid using `WITH` in your SQL code. * TiDB 4.0 ~ 5.2 does not support creating a [temporary table or view](https://docs.pingcap.com/tidb/v5.2/sql-statement-create-table#:~:text=sec\)-,MySQL%20compatibility,-TiDB%20does%20not). * TiDB 4.X does not support using SQL func in `CREATE VIEW`, avoid it in your SQL code. You can find more detail [here](https://github.com/pingcap/tidb/pull/27252). --- ##### Connect Upsolver to dbt v1 Local development #### Overview of dbt-upsolver * **Maintained by**: Upsolver Team * **Authors**: Upsolver Team * **GitHub repo**: [Upsolver/dbt-upsolver](https://github.com/Upsolver/dbt-upsolver)[![](https://img.shields.io/github/stars/Upsolver/dbt-upsolver?style=for-the-badge)](https://github.com/Upsolver/dbt-upsolver) * **PyPI package**: `dbt-upsolver` [![](https://badge.fury.io/py/dbt-upsolver.svg)](https://badge.fury.io/py/dbt-upsolver) * **Slack channel**: [Upsolver Community](https://join.slack.com/t/upsolvercommunity/shared_invite/zt-1zo1dbyys-hj28WfaZvMh4Z4Id3OkkhA) * **Supported dbt v1 version**: v1.5.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-upsolver pip is the easiest way to install the adapter: `python -m pip install dbt-upsolver` Installing `dbt-upsolver` will also install `dbt-core` and any other dependencies. #### Configuring dbt-upsolver For Upsolver-specifc configuration please refer to [Upsolver Configuration](https://docs.getdbt.com/reference/resource-configs/upsolver-configs.md) For further info, refer to the GitHub repository: [Upsolver/dbt-upsolver](https://github.com/Upsolver/dbt-upsolver) #### Authentication Methods ##### User / Token authentication Upsolver can be configured using basic user/token authentication as shown below. \~/.dbt/profiles.yml ```yaml my-upsolver-db: target: dev outputs: dev: type: upsolver api_url: https://mt-api-prod.upsolver.com user: [username] token: [token] database: [database name] schema: [schema name] threads: [1 or more] ``` #### Configurations The configs for Upsolver targets are shown below. ##### All configurations | Config | Required? | Description | | -------- | --------- | ---------------------------------------------------------------------------------------------------------- | | token | Yes | The token to connect Upsolver [Upsolver's documentation](https://docs.upsolver.com/sqlake/api-integration) | | user | Yes | The user to log in as | | database | Yes | The database that dbt should create models in | | schema | Yes | The schema to build models into by default | | api\_url | Yes | The API url to connect. Common value `https://mt-api-prod.upsolver.com` | --- ##### Connect Vertica to dbt v1 Local development VENDOR-SUPPORTED PLUGIN If you're interested in contributing, check out the source code for each repository listed below. * **Maintained by**: Vertica * **Authors**: Vertica (Former authors: Matthew Carter, Andy Regan, Andrew Hedengren) * **GitHub repo**: [vertica/dbt-vertica](https://github.com/vertica/dbt-vertica) [![](https://img.shields.io/github/stars/vertica/dbt-vertica?style=for-the-badge)](https://github.com/vertica/dbt-vertica) * **PyPI package**: `dbt-vertica` [![](https://badge.fury.io/py/dbt-vertica.svg)](https://badge.fury.io/py/dbt-vertica) * **Slack channel**: [n/a](https://www.getdbt.com/community/) * **Supported dbt version**: v1.8.5 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Vertica 24.3.0 #### Installing dbt-vertica Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-vertica` #### Configuring dbt-vertica For Vertica-specific configuration, please refer to [Vertica configs.](https://docs.getdbt.com/reference/resource-configs/vertica-configs.md) ##### Connecting to Vertica with dbt-vertica ###### Username / password authentication Configure your dbt profile for using Vertica: ###### Vertica connection information profiles.yml ```yaml your-profile: outputs: dev: type: vertica # Don't change this! host: [hostname] port: [port] # or your custom port (optional) username: [your username] password: [your password] database: [database name] oauth_access_token: [access token] schema: [dbt schema] connection_load_balance: True backup_server_node: [list of backup hostnames or IPs] retries: [1 or more] autocommit: False threads: [1 or more] target: dev ``` ###### Description of Profile Fields: | Property | Description | Required? | Default Value | Example | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | type | The specific adapter to use. | Yes | None | vertica | | host | The host name or IP address of any active node in the Vertica Server. | Yes | None | 127.0.0.1 | | port | The port to use, default or custom. | Yes | 5433 | 5433 | | username | The username to use to connect to the server. | Yes | None | dbadmin | | password | The password to use for authenticating to the server. | Yes | None | my\_password | | database | The name of the database running on the server. | Yes | None | my\_db | | oauth\_access\_token | To authenticate via OAuth, provide an OAuth Access Token that authorizes a user to the database. | No | "" | Default: "" | | schema | The schema to build models into. | No | None | VMart | | connection\_load\_balance | A Boolean value that indicates whether the connection can be redirected to a host in the database other than host. | No | True | True | | backup\_server\_node | List of hosts to connect to if the primary host specified in the connection (host, port) is unreachable. Each item in the list should be either a host string (using default port 5433) or a (host, port) tuple. A host can be a host name or an IP address. | No | None | \['123.123.123.123','[www.abc.com',('123.123.123.124',5433)](http://www.abc.com',\('123.123.123.124',5433\))] | | retries | The retry times after an unsuccessful connection. | No | 2 | 3 | | threads | The number of threads the dbt project will run on. | No | 1 | 3 | | label | A session label to identify the connection. | No | An auto-generated label with format of: dbt\_username | dbt\_dbadmin | | autocommit | Boolean value that indicates if the connection can enable or disable auto-commit. | No | True | False | For more information on Vertica’s connection properties please refer to [Vertica-Python](https://github.com/vertica/vertica-python#create-a-connection) Connection Properties. --- ##### Connect YDB to dbt v1 Local development #### Overview of dbt-ydb * **Maintained by**: YDB Team * **Authors**: YDB Team * **GitHub repo**: [ydb-platform/dbt-ydb](https://github.com/ydb-platform/dbt-ydb)[![](https://img.shields.io/github/stars/ydb-platform/dbt-ydb?style=for-the-badge)](https://github.com/ydb-platform/dbt-ydb) * **PyPI package**: `dbt-ydb` [![](https://badge.fury.io/py/dbt-ydb.svg)](https://badge.fury.io/py/dbt-ydb) * **Slack channel**: [n/a]() * **Supported dbt v1 version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-ydb pip is the easiest way to install the adapter: `python -m pip install dbt-ydb` Installing `dbt-ydb` will also install `dbt-core` and any other dependencies. #### Configuring dbt-ydb For YDB-specifc configuration please refer to [YDB Configuration](https://docs.getdbt.com/reference/resource-configs/no-configs.md) For further info, refer to the GitHub repository: [ydb-platform/dbt-ydb](https://github.com/ydb-platform/dbt-ydb) #### Connecting to YDB To connect to YDB from dbt, you'll need to add a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) to your `profiles.yml` file. A YDB profile conforms to the following syntax: profiles.yml ```yaml profile-name: target: dev outputs: dev: type: ydb host: localhost port: 2136 database: /local schema: empty_string secure: False root_certificates_path: empty_string # Static credentials username: empty_string password: empty_string # Access token credentials token: empty_string # Service account credentials service_account_credentials_file: empty_string ``` ##### All configurations | Config | Required? | Default | Description | | ----------------------------------- | --------- | -------------- | ----------------------------------------------------------------------------- | | host | Yes | | YDB host | | port | Yes | | YDB port | | database | Yes | | YDB database | | schema | No | `empty_string` | Optional subfolder for dbt models. Use empty string or `/` to use root folder | | secure | No | False | If enabled, `grpcs` protocol will be used | | root\_certificates\_path | No | `empty_string` | Optional path to root certificates file | | username | No | `empty_string` | YDB username to use static Ccredentials | | password | No | `empty_string` | YDB password to use static credentials | | token | No | `empty_string` | YDB token to use Access Token credentials | | service\_account\_credentials\_file | No | `empty_string` | Path to service account credentials file to use service account credentials | --- ##### Connect Yellowbrick to dbt v1 Local development Community plugin Some core functionality may be limited. * **Maintained by**: Community * **Authors**: InfoCapital team * **GitHub repo**: [InfoCapital-AU/dbt-yellowbrick](https://github.com/InfoCapital-AU/dbt-yellowbrick) [![](https://img.shields.io/github/stars/InfoCapital-AU/dbt-yellowbrick?style=for-the-badge)](https://github.com/InfoCapital-AU/dbt-yellowbrick) * **PyPI package**: `dbt-yellowbrick` [![](https://badge.fury.io/py/dbt-yellowbrick.svg)](https://badge.fury.io/py/dbt-yellowbrick) * **Slack channel**: [n/a](https://www.getdbt.com/community) * **Supported dbt version**: v1.7.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: Yellowbrick 5.2 #### Installing dbt-yellowbrick Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-yellowbrick` #### Configuring dbt-yellowbrick For Yellowbrick Data-specific configuration, please refer to [Yellowbrick Data configs.](https://docs.getdbt.com/reference/resource-configs/yellowbrick-configs.md) #### Profile configuration Yellowbrick targets should be set up using the following configuration in your `profiles.yml` file. \~/.dbt/profiles.yml ```yaml company-name: target: dev outputs: dev: type: yellowbrick host: [hostname] user: [username] password: [password] port: [port] dbname: [database name] schema: [dbt schema] role: [optional, set the role dbt assumes when executing queries] sslmode: [optional, set the sslmode used to connect to the database] sslrootcert: [optional, set the sslrootcert config value to a new file path to customize the file location that contains root certificates] ``` ##### Configuration notes This adapter is based on the dbt-postgres adapter documented here [Postgres profile setup](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md) ###### role The `role` config controls the user role that dbt assumes when opening new connections to the database. ###### sslmode / sslrootcert The ssl config parameters control how dbt connects to Yellowbrick using SSL. Refer to the [Yellowbrick documentation](https://docs.yellowbrick.com/5.2.27/client_tools/config_ssl_for_clients_intro.html) for details. --- ##### Sail setup * **Maintained by**: LakeSail * **Authors**: LakeSail * **GitHub repo**: [lakehq/dbt-sail](https://github.com/lakehq/dbt-sail) [![](https://img.shields.io/github/stars/lakehq/dbt-sail?style=for-the-badge)](https://github.com/lakehq/dbt-sail) * **PyPI package**: `dbt-sail` [![](https://badge.fury.io/py/dbt-sail.svg)](https://badge.fury.io/py/dbt-sail) * **Slack channel**: [LakeSail Community Slack](https://www.launchpass.com/lakesail-community/free) * **Supported dbt version**: v1.8.0 and newer * **dbt support**: Not Supported * **Minimum data platform version**: n/a #### Installing dbt-sail Use \`pip\` to install the adapter. Use the following command for installation: `python -m pip install dbt-sail` #### Configuring dbt-sail For Sail-specific configuration, please refer to [Sail configs.](https://docs.getdbt.com/reference/resource-configs/spark-configs.md) #### Connecting to Sail with **dbt-sail** [Sail](https://github.com/lakehq/sail) is a drop-in replacement for Apache Spark, built and maintained by [LakeSail](https://lakesail.com/). Sail implements the Spark Connect protocol, so `dbt-sail` is a thin wrapper around `dbt-spark` that connects to Sail instead of JVM Spark. `dbt-sail` supports the same feature set as `dbt-spark`. `dbt-sail` supports two connection modes: * `embedded` - starts a Sail server in-process via [`pysail`](https://pypi.org/project/pysail/). No external server required. * `remote` - connects to an already-running Sail server via Spark Connect. #### Profile configuration \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: sail mode: embedded # or 'remote' host: 127.0.0.1 # required for 'remote'; defaults to 127.0.0.1 for 'embedded' port: 50051 # used by 'remote'; ignored by 'embedded' schema: SCHEMA_NAME threads: 1 ``` | Field | Description | Required? | Default | | ------------------------ | ----------------------------------------------------------------- | --------------------- | ---------------------- | | `type` | Must be `sail`. | Required | - | | `mode` | Connection mode. Either `embedded` or `remote`. | Required | - | | `host` | Hostname of the Sail server. Required for `remote` mode. | Required for `remote` | `127.0.0.1` (embedded) | | `port` | Spark Connect port on the Sail server. Used by `remote` mode. | Optional | `50051` | | `schema` | Default schema where dbt builds objects. | Required | - | | `threads` | Number of threads for dbt to use. | Optional | `1` | | `server_side_parameters` | Map of parameters passed through to the underlying Spark session. | Optional | `{}` | Because `dbt-sail` extends `dbt-spark`, additional `dbt-spark` profile fields are also accepted. See the [Apache Spark setup](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md) page for the full list. ##### Example: embedded mode \~/.dbt/profiles.yml ```yaml my_sail_project: target: dev outputs: dev: type: sail mode: embedded schema: analytics threads: 4 ``` ##### Example: remote mode \~/.dbt/profiles.yml ```yaml my_sail_project: target: dev outputs: dev: type: sail mode: remote host: sail.internal port: 50051 schema: analytics threads: 4 ``` #### Resource configuration `dbt-sail` accepts the same resource configurations as `dbt-spark`. See the [Apache Spark configurations](https://docs.getdbt.com/reference/resource-configs/spark-configs.md) page for materialization options, incremental strategies, and other model-level configs. --- ##### Salesforce Data 360 setup Beta ### Salesforce Data 360 setup [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development This `dbt-salesforce` adapter is available via the dbt v2 CLI. To access the adapter, [install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md). We recommend using the [VS Code Extension](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) as the development interface. dbt platform support coming soon. #### Prerequisites Before you can connect dbt to Salesforce Data 360, you need the following: * A Data 360 instance. * The `server.key` private key file. For more information, refer to [Generating a private key and certificate](#generating-a-private-key-and-certificate). * An external client app configured for JSON Web Token (JWT) Bearer token flow. For more information, refer to [Setting up the external client app](#setting-up-the-external-client-app). * `Data Cloud Architect` and `Data Cloud User` permissions. ##### Generating a private key and certificate Before creating the external client app, generate a private key (`server.key`) and a self-signed certificate (`server.crt`) in Salesforce. Salesforce uses the certificate to verify the JWT Bearer token that dbt sends. Refer to [Create a Private Key and Self-Signed Digital Certificate](https://developer.salesforce.com/docs/atlas.en-us.252.0.sfdx_dev.meta/sfdx_dev/sfdx_dev_auth_key_and_cert.htm) for instructions. note It is recommended to generate these credentials under a **service user account** rather than an individual user account to simplify credential management. ##### Creating the external client app To authenticate dbt using JWT Bearer token flow, you must create and configure an external client app in Salesforce Data 360. Follow the steps in [Create a Local External Client App](https://help.salesforce.com/s/articleView?id=xcloud.create_a_local_external_client_app.htm\&type=5) to create an external client app. For Step 6, make sure to select **Packaged**. Then, configure the OAuth settings for the app: 1. Under **API (Enable OAuth Settings)**, select **Enable OAuth**. 2. Set the **Callback URL** to `https://login.salesforce.com/services/oauth2/callback`. 3. Under **OAuth Scopes**, add the following to the **Selected OAuth Scopes** list: * `api` - To manage user data via APIs * `refresh_token`, `offline_access` - To perform requests at any time, even when the user is offline or tokens have expired * `cdp_query_api` - To perform SQL queries on Data 360 data 4. Under **Flow Enablement**, select **Enable JWT Bearer Flow**. 5. Click **Upload Files** and upload the `server.crt` file created in [Generating a private key and certificate](#generating-a-private-key-and-certificate). 6. Click **Create**. 7. On the **Policies** tab, select **Edit**. 8. Under **OAuth Policies** > **Plugin Policies** > **Permitted Users**, select **Admin approved users are pre-authorized**. A list of **Select Profiles and Select Permission Sets** appears. 9. Selct the profiles and permission sets that should have access to the external client app. 10. Under **App Authorization**, set **IP Relaxation** to **Relax IP Restrictions**. 11. Click **Save**. #### Configure dbt v2 To connect dbt to Salesforce Data 360, set up your `profiles.yml`. Refer to the following configuration: \~/.dbt/profiles.yml ```yaml company-name: target: dev outputs: dev: type: salesforce method: jwt_bearer client_id: [Consumer Key of your Data 360 app] private_key_path: [local file path of your server key] login_url: "https://login.salesforce.com" username: [username on the Data 360 Instance] ``` | Profile field | Required | Description | Example | | ------------------ | -------- | -------------------------------------------------------------- | ------------------------------------------------------------- | | `method` | Yes | Authentication Method. Currently, only `jwt_bearer` supported. | `jwt_bearer` | | `client_id` | Yes | This is the `Consumer Key` from your connected app secrets. | | | `private_key_path` | Yes | File path of the `server.key` file in your computer. | `/Users/dbt_user/Documents/server.key` | | `login_url` | Yes | Login URL of the Salesforce instance. | [https://login.salesforce.com](https://login.salesforce.com/) | | `username` | Yes | Username on the Data 360 Instance. | <dbt_user@dbtlabs.com> | #### More information Find Salesforce-specific configuration information in the [Salesforce adapter reference guide](https://docs.getdbt.com/reference/resource-configs/data-cloud-configs.md). --- #### Install dbt Fusion engine ##### About self-hosted dbt v2 installation important dbt v2 is available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our dbt Community Slack channel [`#dbt-v2`](https://getdbt.slack.com/archives/C088YCAB6GH). Learn more about installing dbt v2 locally, along with important prerequisites, step-by-step installation instructions, troubleshooting common issues, and configuration guidance. #### Prerequisites Before installing dbt v2, ensure that you: * Have administrative privileges to install software on your local machine. * Are comfortable using a command-line interface (Terminal on macOS/Linux, PowerShell on Windows). * Use a supported data warehouse and authentication method and configure permissions as needed:  BigQuery * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks * Service Account / User Token * Native OAuth  Redshift * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  ClickHouse[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password  Apache Spark (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Thrift * Simple Authentication and Security Layer (SASL) PLAIN * No SASL (NOSASL) * Livy * Basic authentication (username and password) * When deployed on Amazon Web Services (AWS): AWS Signature Version 4 * Supports authentication using single sign-on, service accounts, or user tokens  DuckDB (CLI only) DuckDB does not require authentication — it runs locally on your machine. *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* * Use a supported operating system: * **macOS:** Supported on both Intel (x86-64) and Apple Silicon (ARM) * **Linux:** Supported on both x86-64 and ARM * **Windows:** Supported on x86-64; ARM support coming soon #### Getting started If you're ready to get started, choose one of the following options. To learn more about which tool is best for you, see the [v2 availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md) page. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) ###### [dbt VS Code Extension](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [Learn how to connect to a data platform, integrate with secure authentication methods, and configure a sync with a git repo.](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) ###### [dbt v2 from the CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [Learn how to install dbt v2 on the command line interface (CLI).](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) ###### [dbt v2 upgrade](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) [Learn how you can upgrade and leverage the speed and scale of dbt v2](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) --- ### dbt platform #### About dbt setup dbt platform dbt platform is the fastest and most reliable way to deploy your dbt jobs. It contains a myriad of settings that can be configured by admins, from the necessities (data platform integration) to security enhancements (SSO) and quality-of-life features (RBAC). This portion of our documentation will take you through the various settings in the dbt UI, including: * [Connecting to a data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) * Configuring access to [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md), [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md), or your own [git repo URL](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md). * [Managing users and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) * [Configuring secure access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) For steps on installing dbt development tools, refer to the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or the [Studio IDE (browser-based)](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). These settings are intended for dbt administrators. If you need a more detailed first-time setup guide for specific data platforms, read our [quickstart guides](https://docs.getdbt.com/guides.md) or follow the [dbt platform configuration checklist](https://docs.getdbt.com/docs/configuration-checklist.md). If you want a more in-depth learning experience, we recommend taking the dbt Fundamentals on our [dbt Learn site](https://learn.getdbt.com/). #### Prerequisites * To set up dbt, you'll need to have a dbt account with administrator access. If you still need to create a dbt account, [sign up today](https://getdbt.com) on our North American servers or [contact us](https://getdbt.com/contact) for international options. * To have the best experience using dbt, we recommend you use modern and up-to-date web browsers like Chrome, Safari, Edge, and Firefox. --- #### Account integrations in dbt dbt platform The following sections describe the different **Account integrations** available from your dbt account under the account **Settings** section. [![Example of Account integrations from the sidebar](/img/docs/dbt-platform/account-integrations.png?v=2 "Example of Account integrations from the sidebar")](#)Example of Account integrations from the sidebar #### Git integrations Connect your dbt account to your Git provider to enable dbt users to authenticate your personal accounts. dbt will perform Git actions on behalf of your authenticated self, against repositories to which you have access according to your Git provider permissions. To configure a Git account integration: 1. Navigate to **Account settings** in the side menu. 2. Under the **Settings** section, click on **Integrations**. 3. Click on the Git provider from the list and select the **Pencil** icon to the right of the provider. 4. dbt [natively connects](https://docs.getdbt.com/docs/platform/git/configure-git.md) to the following Git providers: * [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md) * Enterprise and Enterprise+ accounts can use their [own GitHub application](https://docs.getdbt.com/docs/platform/git/connect-github.md#custom-github-application) instead of the shared one. * [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) * [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") You can connect your dbt account to additional Git providers by importing a git repository from any valid git URL. Refer to [Import a git repository](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) for more information. [![Example of the Git integration page](/img/docs/dbt-platform/account-integration-git.png?v=2 "Example of the Git integration page")](#)Example of the Git integration page #### OAuth integrations Connect your dbt account to an OAuth provider that are integrated with dbt. To configure an OAuth account integration: 1. Navigate to **Account settings** in the side menu. 2. Under the **Settings** section, click on **Integrations**. 3. Under **OAuth**, click on **Link** to [connect your Slack account](https://docs.getdbt.com/docs/deploy/job-notifications.md#set-up-the-slack-integration). 4. For custom OAuth providers, under **Custom OAuth integrations**, click on **Add integration** and select the [OAuth provider](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) from the list. Fill in the required fields and click **Save**. [![Example of the OAuth integration page](/img/docs/dbt-platform/account-integration-oauth.png?v=2 "Example of the OAuth integration page")](#)Example of the OAuth integration page --- #### Account settings in dbt dbt platform The following sections describe the different **Account settings** available from your dbt account in the sidebar (under your account name on the lower left-hand side). [![Example of Account settings from the sidebar](/img/docs/dbt-platform/example-sidebar-account-settings.png?v=2 "Example of Account settings from the sidebar")](#)Example of Account settings from the sidebar #### Git repository caching [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") repo caching enabled by default Git repository caching is enabled by default for all new Enterprise and Enterprise+ accounts, improving reliability by allowing dbt to use a cached copy of your repo if cloning fails. See the next section for more details on repo caching, retention, and more. At the start of every [job](https://docs.getdbt.com/docs/deploy/jobs.md) run, dbt clones the project's Git repository so it has the latest versions of your project's code and runs `dbt deps` to install your dependencies. For improved reliability and performance on your job runs, you can enable dbt to keep a cache of the project's Git repository. So, if there's a third-party outage that causes the cloning operation to fail, dbt will instead use the cached copy of the repo so your jobs can continue running as scheduled. dbt caches your project's Git repo after each successful run and retains it for 8 days if there are no repo updates. It caches all packages regardless of installation method and does not fetch code outside of the job runs. dbt will use the cached copy of your project's Git repo under these circumstances: * Outages from third-party services (for example, the [dbt package hub](https://hub.getdbt.com/)). * Git authentication fails. * There are syntax errors in the `packages.yml` file. You can set up and use [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) to find these errors sooner. * If a package doesn't work with the current dbt version. You can set up and use [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) to identify this issue sooner. Repository caching is a single account-level setting, which means you don't need to configure it per job or environment. dbt automatically skips the cache for CI jobs, so they always test the latest code in your pull request. To use, select the **Enable repository caching** option from your account settings. [![Example of the Enable repository caching option](/img/docs/deploy/account-settings-repository-caching.png?v=2 "Example of the Enable repository caching option")](#)Example of the Enable repository caching option #### Partial parsing At the start of every dbt invocation, dbt reads all the files in your project, extracts information, and constructs an internal manifest containing every object (model, source, macro, and so on). Among other things, it uses the `ref()`, `source()`, and `config()` macro calls within models to set properties, infer dependencies, and construct your project's DAG. When dbt finishes parsing your project, it stores the internal manifest in a file called `partial_parse.msgpack`. Parsing projects can be time-consuming, especially for large projects with hundreds of models and thousands of files. To reduce the time it takes dbt to parse your project, use the partial parsing feature in dbt for your environment. When enabled, dbt uses the `partial_parse.msgpack` file to determine which files have changed (if any) since the project was last parsed, and then it parses *only* the changed files and the files related to those changes. Partial parsing in dbt requires dbt version 1.4 or newer. The feature does have some known limitations. Refer to [Known limitations](https://docs.getdbt.com/reference/parsing.md#known-limitations) to learn more about them. To use, select the **Enable partial parsing between deployment runs** option from your account settings. [![Example of the Enable partial parsing between deployment runs option](/img/docs/deploy/account-settings-partial-parsing.png?v=2 "Example of the Enable partial parsing between deployment runs option")](#)Example of the Enable partial parsing between deployment runs option #### Account access and enablement Enable access to features in your account by selecting the appropriate option from your account settings. ##### Enabling AI features Admins can enable access to both dbt Wizard (dbt Labs’ AI agent layer, available on the dbt platform and CLI) and dbt Copilot for authorized users across the dbt platform. ##### Enabling Advanced CI features [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md) features, such as [compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes), allow dbt account members to view details about the changes between what's in the production environment and the pull request. To use Advanced CI features, your dbt account must have access to them. Ask your dbt administrator to enable Advanced CI features on your account, which they can do by choosing the **Enable account access to Advanced CI** option from the account settings. Once enabled, the **dbt compare** option becomes available in the CI job settings for you to select. [![The Enable account access to Advanced CI option](/img/docs/deploy/account-settings-advanced-ci.png?v=2 "The Enable account access to Advanced CI option")](#)The Enable account access to Advanced CI option ##### Enable global account discovery When **Enable global account discovery** is on, users can discover all accounts associated with their email address at login. Users still access accounts using their credentials or the account's designated auth method (for example, SSO). Refer to [Log in to dbt platform](https://docs.getdbt.com/docs/platform/about-platform/login.md) for more info. info `login.dbt.com` is currently available for multi-tenant accounts with an account-specific domain (for example, `abc123.us1.dbt.com`). Support for single-tenant accounts is coming soon. In the meantime, single-tenant users can sign in directly using their account **Access URL** (like `MY_COMPANY.us1.dbt.com`). OAuth clients such as [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md?version=2.0), and [dbt MCP](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) have not yet been updated to use `login.dbt.com` and continue to authenticate through their account [**Access URL**](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Disabling this setting means users must know their [account URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md#accessing-your-account) to log in; they will not see a list of accounts at login. To change this setting, select or clear the **Enable global account discovery** option in your account settings. If you disable it, a confirmation pop-up box explains that users will need the account URL to log in and access the account. #### Project settings history You can view historical project settings changes over the last 90 days. To view the change history: 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Click **Projects**. 3. Click a **project name**. 4. Click **History**. [![Example of the project history option. ](/img/docs/deploy/project-history.png?v=2 "Example of the project history option. ")](#)Example of the project history option. --- #### dbt environments dbt platform An environment determines how dbt will execute your project in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) (for development) and scheduled jobs (for deployment). Critically, in order to execute dbt, environments define three variables: 1. The version of dbt that will be used to run your project 2. The warehouse connection information (including the target database/schema settings) 3. The version of your code to execute Each dbt project can have only one [development environment](#create-a-development-environment), but there is no limit to the number of [deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md), providing you the flexibility and customization to tailor the execution of scheduled jobs. Use environments to customize settings for different stages of your project and streamline the execution process by using software engineering principles. [![dbt environment hierarchy showing projects, environments, connections, and orchestration jobs.](/img/dbt-env.png?v=2 "dbt environment hierarchy showing projects, environments, connections, and orchestration jobs.")](#)dbt environment hierarchy showing projects, environments, connections, and orchestration jobs. The following sections detail the different types of environments and how to intuitively configure your development environment in dbt. #### Types of environments In dbt, there are two types of environments: * **Deployment environment** — Determines the settings used when jobs created within that environment are executed. <br /> Types of deployment environments: * General * Staging * Production * **Development environment** — Determines the settings used in the Studio IDE or dbt platform CLI, for that particular project. Each dbt project can only have a single development environment, but can have any number of General deployment environments, one Production deployment environment and one Staging deployment environment. | | Development | General | Production | Staging | | -------------------------------------- | ------------------------------ | ------------ | ------------ | ------------ | | **Determines settings for** | Studio IDE or dbt platform CLI | dbt Job runs | dbt Job runs | dbt Job runs | | **How many can I have in my project?** | 1 | Any number | 1 | 1 | note For users familiar with development on dbt v1, each environment is roughly analogous to an entry in your `profiles.yml` file, with some additional information about your repository to ensure the proper version of code is executed. More info on v1 environments [here](https://docs.getdbt.com/docs/local/dbt-environments.md). #### Common environment settings Both development and deployment environments have a section called **General Settings**, which has some basic settings that all environments will define: | Setting | Example Value | Definition | Accepted Values | | --------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | Environment name | Production | The environment name | Any string! | | Environment type | Deployment | The type of environment | Deployment, Development | | Set deployment type | PROD | Designates the deployment environment type. | Production, Staging, General | | dbt version | **v1 Latest** | dbt automatically upgrades the dbt version running in this environment, based on the [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) you select. | **v2 Stable**, **v2 Nightly**, **v2 Extended**, **v1 Latest**, **v1 Compatible**, **v1 Extended** | | Only run on a custom branch | ☑️ | Determines whether to use a branch other than the repository’s default | See below | | Custom branch | dev | Custom Branch name | See below | About dbt version dbt allows users to select a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive ongoing dbt version upgrades at the cadence that makes sense for their team. ##### Custom branch behavior By default, all environments will use the default branch in your repository (usually the `main` branch) when accessing your dbt code. This is overridable within each dbt Environment using the **Default to a custom branch** option. This setting will have slightly different behavior depending on the environment type: * **Development**: determines which branch in the Studio IDE or dbt platform CLI developers create branches from and open PRs against. * **Deployment:** determines the branch is cloned during job executions for each environment. For more info, check out this [FAQ page on this topic](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md)! ##### Extended attributes note Extended attributes are currently *not* supported for SSH tunneling Extended attributes allow you to set a [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) snippet that changes how dbt connects to your data platform. You can set any YAML attributes that a dbt adapter accepts in its `profiles.yml`. Extended attributes partially override connection or environment credentials, including custom environment variables. Where you set them depends on the environment: **Deployment:** On the [connection profile](https://docs.getdbt.com/docs/platform/about-profiles.md) assigned to the environment: 1. Go to **Orchestration** > **Environments**. 2. Open the deployment environment and click **Settings**. 3. Click **Edit**. 4. Under **Connection profiles**, click the profile name. 5. Add **Extended attributes** and save your changes. A profile can be assigned to more than one deployment environment, so these attributes apply to every environment that uses it. **Development:** [Profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) apply to deployment environments only. Set **Extended attributes** on the development environment **Settings** page: 1. Go to **Orchestration** > **Environments**. 2. Open the development environment and click **Settings**. 3. Click **Edit profile**, then set **Extended attributes** and save your changes. [![The Extended attributes field on a connection profile.](/img/docs/dbt-platform/using-dbt-platform/extended-attributes.png?v=2 "The Extended attributes field on a connection profile.")](#)The Extended attributes field on a connection profile. <br /> The following code is an example of the types of attributes you can add in the **Extended attributes** text box: ```yaml dbname: jaffle_shop schema: dbt_alice threads: 4 username: alice password: '{{ env_var(''DBT_ENV_SECRET_PASSWORD'') }}' ``` You can also use arrays as values for keys. For example, to pass a list of database groups: ```yaml db_groups: - db_editor - db_viewer ``` ###### Extended attributes don't mask secret values * We recommend you avoid setting secret values to prevent visibility in the text box and logs. A common workaround is to wrap extended attributes in [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md). In the earlier example, `password: '{{ env_var(''DBT_ENV_SECRET_PASSWORD'') }}'` will get a value from the `DBT_ENV_SECRET_PASSWORD` environment variable at runtime. * If you're using [profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) for deployment environments, any `env_var` references in Extended attributes must use *project-scoped* environment variables. Since profiles are environment-agnostic, environment-scoped variables aren't available during connection tests. Jobs run normally since they have a real environment, but connection tests will fail if the referenced variable is only set at the environment level. To set a project-scoped variable, go to **Orchestration** > **Environments** > **Environment variables**, and set a value in the **Project default** column. This value applies across all environments in the project, making it available to profiles during connection tests. See [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md?version=2.0#setting-environment-variables) for more information. ###### How extended attributes work If you're developing in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), or [orchestrating job runs](https://docs.getdbt.com/docs/deploy/deployments.md), extended attributes parses through the provided YAML and extracts the `profiles.yml` attributes. For each individual attribute: * If the attribute exists in another source (such as your project settings), it will replace its value (like environment-level values) in the profile. It also overrides any custom environment variables (if not itself wired using the syntax described for secrets above) * If the attribute doesn't exist, it will add the attribute or value pair to the profile. ###### Only the **top-level keys** are accepted in extended attributes This means that if you want to change a specific sub-key value, you must provide the entire top-level key as a JSON block in your resulting YAML. For example, if you want to customize a particular field within a [service account JSON](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#service-account-json) for your BigQuery connection (like 'project\_id' or 'client\_email'), you need to provide an override for the entire top-level `keyfile_json` main key/attribute using extended attributes. Include the sub-fields as a nested JSON block. #### Create a development environment To create a new dbt development environment: 1. Navigate to **Orchestration** > **Environments**. 2. Click **+ Create Environment**. 3. Select **Development** as the environment type. You can only create one Development environment for a project. 4. Fill in the fields under **Environment settings** and **User credentials**. 5. (Optional) If dbt State is enabled on your account, the **Enable dbt State** option appears. Enable it to reduce unnecessary node rebuilds. For details, refer to [Enabling dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md#enabling-dbt-state-on-a-development-environment). 6. Click **Save** to create the environment. [![Creating a development environment](/img/docs/dbt-platform/refresh-ide/new-development-environment-fields.png?v=2 "Creating a development environment")](#)Creating a development environment ##### Set user credentials To use the dbt Studio IDE or dbt platform CLI, each developer will need to set up [personal user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) to your warehouse connection. Click your account name in the bottom left, select **Your profile**, then go to **Credentials**. This allows you to set separate target information and maintain individual credentials to connect to your warehouse. #### Deployment environment Deployment environments in dbt are necessary to execute scheduled jobs and use other features (like different workspaces for different tasks). You can have many environments in a single dbt project, enabling you to set up each space in a way that suits different needs (such as experimenting or testing). Even though you can have many environments, only one of them can be the "main" deployment environment. This would be considered your "production" environment and represents your project's "source of truth", meaning it's where your most reliable and final data transformations live. To learn more about dbt deployment environments and how to configure them, refer to the [Deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md) page. For our best practices guide, read [dbt environment best practices](https://docs.getdbt.com/guides/set-up-ci.md) for more info. #### Change environment settings To change an environment's settings (such as its name, credentials, or dbt version): 1. Navigate to **Orchestration** > **Environments**. 2. Click the **environment name** you want to change. 3. Click **Settings** and then **Edit**. 4. Update the fields you want to change. For example, you can edit the environment name, dbt version (for eligible account tiers and supported adapters, **v2 Stable** is the default for new deployment environments). 5. Click **Save** to apply your changes. For details on available settings, see [Common environment settings](https://docs.getdbt.com/docs/dbt-platform-environments.md#common-environment-settings) in the environment info above. #### Delete an environment Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. Follow these steps to delete an environment in dbt: 1. Navigate to **Orchestration** > **Environments**. 2. Select the environment you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete** to delete the environment. [![Delete an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-environment.png?v=2 "Delete an environment")](#)Delete an environment 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. 6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Job monitoring On the **Environments** page, there are two sections that provide an overview of the jobs for that environment: * **In progress** — Lists the currently in progress jobs with information on when the run started * **Top jobs by models built** — Ranks jobs by the number of models built over a specific time [![In progress jobs and Top jobs by models built](/img/docs/deploy/in-progress-top-jobs.png?v=2 "In progress jobs and Top jobs by models built")](#)In progress jobs and Top jobs by models built #### Environment settings history You can view historical environment settings changes over the last 90 days. To view the change history: 1. Navigate to **Orchestration** from the main menu and click **Environments**. 2. Click an **environment name**. 3. Click **Settings**. 4. Click **History**. [![Example of the environment history option.](/img/docs/deploy/environment-history.png?v=2 "Example of the environment history option.")](#)Example of the environment history option. --- #### Multi-cell migration checklist dbt platform dbt Labs is in the process of rolling out a new cell-based architecture for dbt. This architecture provides the foundation of dbt for years to come, and brings improved reliability, performance, and consistency to users of dbt. We're scheduling migrations by account. When we're ready to migrate your account, you will receive a banner or email communication with your migration date. If you have not received this communication, then you don't need to take action at this time. dbt Labs will share information about your migration with you, with appropriate advance notice, when applicable to your account. Your account will be automatically migrated on or after its scheduled date. However, if you use certain features, you must take action before that date to avoid service disruptions. #### Recommended actions Rescheduling your migration If you're on the dbt Enterprise tier, you can postpone your account migration by up to 45 days. To reschedule your migration, navigate to **Account Settings** → **Migration guide**. For help, contact the dbt Support Team at <support@getdbt.com>. We highly recommended you take these actions: * Ensure pending user invitations are accepted or note outstanding invitations. Pending user invitations might be voided during the migration. You can resend user invitations after the migration is complete. * Commit unsaved changes in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). Unsaved changes might be lost during migration. * Export and download [audit logs](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md) older than 90 days, as they will be unavailable from dbt after the migration is complete. Logs older than 90 days while within the data retention period are not deleted, but you will have to work with the dbt Labs Customer Support team to recover. #### Required actions These actions are required to prevent users from losing access dbt: * If you still need to, complete [Auth0 migration for SSO](https://docs.getdbt.com/docs/platform/manage-access/auth0-migration.md) before your scheduled migration date to avoid service disruptions. If you've completed the Auth0 migration, your account SSO configurations will be transferred automatically. * Update your IP allow lists. dbt will be using new IPs to access your warehouse post-migration. Allow inbound traffic from all of the following new IPs in your firewall and include them in any database grants: * `52.3.77.232` * `3.214.191.130` * `34.233.79.135` Keep the old dbt IPs listed until the migration is complete. #### Run related data retention All runs available in dbt will be migrated with your account. This includes metadata about the run, like its status, execution start time, and duration. However, the individual steps associated with a run will not be migrated with your account. Therefore, the dbt commands executed during a run, along with their logs and artifact files, will not be available in dbt after your migration. The [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) will contain a subset of data after your account has been migrated. Metadata generated in the past 7 days will be migrated with your account. A maximum of 20 runs will be available when querying the [job object](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md) after migration. #### Post-migration​ Complete all of these items to ensure your dbt resources and jobs will continue working without interruption. Use one of these URL login options: * <https://login.dbt.com>: Universal login shows the accounts you can access across instances. Account administrators can enable or disable this path with **Enable global account discovery** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). * `us1.dbt.com`: If you previously signed in with a username and password at `login.dbt.com`, plan to sign in at `us1.dbt.com` instead. The original URL still works, but you need to click through to continue after sign-in. If SSO is configured, use the unique URL listed in your SSO account settings, for example, `ACCOUNT_PREFIX.us1.dbt.com`. * `ACCOUNT_PREFIX.us1.dbt.com`: A unique URL specifically for your account. If you belong to multiple accounts, each will have a unique URL available as long as they have been migrated to multi-cell. Check out [access, regions, and IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for more information. Remove the following old IP addresses from your firewall and database grants: * `52.45.144.63` * `54.81.134.249` * `52.22.161.231` --- #### Connect data platform ##### About data platform connections dbt platform The dbt platform can connect with a variety of data platform providers. (Applies to dbt v2.0 and later) For most supported data platforms, we recommend the v2 experience, which supports the following data platforms: | Connection | Available on v2 | | -------------------------------------------------------------------------------------------------- | --------------- | | [Snowflake](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md) | ✅ | | [Amazon Redshift](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md) | ✅ | | [Databricks](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-databricks.md) | ✅ | | [Google BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md) | ✅ | | [ClickHouse](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-clickhouse.md) | Private beta | *Adapter lifecycle can differ between the dbt platform and local development — an adapter can reach GA in the dbt platform before it reaches GA for local use.* <br /> <br /> If your data platform isn’t listed yet, support for more data platforms will roll out over time. You can continue using v1 for that project in the meantime. #### Connect your data platform in dbt To connect to your database in dbt: 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Select **Connections** from the top left, and from there, click **New connection**. [![Choose a connection](/img/docs/connect-data-platform/choose-a-connection.png?v=2 "Choose a connection")](#)Choose a connection These connection instructions provide the basic fields required for configuring a data platform connection in dbt. For more detailed guides, which include demo project data, read our [Quickstart guides](https://docs.getdbt.com/guides.md). ##### Supported authentication methods The following tables show which authentication types are supported for each connection available on the dbt platform: (Applies to dbt v2.0 and later) | Integration | User credentials/token | Service account credentials | Warehouse OAuth for users | External OAuth for users | Service-to-service OAuth | Key/Pair | MFA | SSH | Private connectivity support\*\* | | ----------- | ---------------------- | --------------------------- | ------------------------- | ------------------------ | ------------------------ | -------- | --- | --- | -------------------------------- | | Snowflake | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | | BigQuery | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | | Databricks | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | Redshift | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | ClickHouse | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | \*\* Private connectivity is only supported for certain cloud providers and deployment types. See [Private connectivity documentation](https://docs.getdbt.com/docs/platform/secure/private-connectivity/private-connectivity.md) for details. #### Connection management Warehouse connections are an account-level resource. You can find them under **Accounts settings** > **Connections**. Warehouse connections can be re-used across projects. If multiple projects all connect to the same warehouse, you should re-use the same connection to streamline your management operations. Connections are assigned to a project via an [environment](https://docs.getdbt.com/docs/dbt-platform-environments.md). [![Connection model](/img/docs/dbt-platform/platform-configuring-dbt-platform/connections-new-model.png?v=2 "Connection model")](#)Connection model As shown in the image, a project with 2 environments can target between 1 and 2 different connections. If you want to separate your production environment from your non-production environment, assign multiple connections to a single project. ##### Migration from project-level connections to account-level connections Rolling out account-level connections will not require any interruption of service in your current usage (Studio IDE, CLI, jobs, and so on.). Why am I prompted to configure a development environment? If your project did not previously have a development environment, you may be redirected to the project setup page. Your project is still intact. Choose a connection for your new development environment, and you can view all your environments again. However, to fully utilize the value of account-level connections, you may have to rethink how you assign and use connections across projects and environments. [![Typical connection setup post rollout](/img/docs/dbt-platform/platform-configuring-dbt-platform/connections-post-rollout.png?v=2 "Typical connection setup post rollout")](#)Typical connection setup post rollout Please consider the following actions, as the steps you take will depend on the desired outcome. * The initial clean-up of your connection list * Delete unused connections with 0 environments. * Rename connections with a temporary, descriptive naming scheme to better understand where each is used [![Post initial clean-up](/img/docs/dbt-platform/platform-configuring-dbt-platform/connections-post-rollout-2.png?v=2 "Post initial clean-up")](#)Post initial clean-up * Get granular with your connections * Define an intent for each connection, usually a combination of warehouse/database instance, intended use (dev, prod, etc), and administrative surface (which teams/projects will need to collaborate on the connection) * Aim to minimize the need for local overrides (like extended attributes) * Come to a consensus on a naming convention. We recommend you name connections after the server hostname and distinct intent/domain/configuration. It will be easier to reuse connections across projects this way [![Granularity determined](/img/docs/dbt-platform/platform-configuring-dbt-platform/connections-post-rollout-3.png?v=2 "Granularity determined")](#)Granularity determined * Deduplication (connection list + environment details — not touching extended attributes for now) * Based of the granularity of your connection details, determine which connections should remain among groups of duplicates, and update every relevant environment to leverage that connection * Delete unused connections with 0 environments as you go * Deduplicate thoughtfully. If you want connections to be maintained by two different groups of users, you may want to preserve two identical connections to the same warehouse so each can evolve as each group sees fit without impacting the other group * Do not update extended attributes at this stage [![Connections de-duplicated](/img/docs/dbt-platform/platform-configuring-dbt-platform/connections-post-rollout-4.png?v=2 "Connections de-duplicated")](#)Connections de-duplicated * Normalization * Understand how new connections should be created to avoid local overrides. If you currently use extended attributes to override the warehouse instance in your production environment - you should instead create a new connection for that instance, and wire your production environment to it, removing the need for the local overrides * Create new connections, update relevant environments to target these connections, removing now unnecessary local overrides (which may not be all of them!) * Test the new wiring by triggering jobs or starting Studio IDE sessions [![Connections normalized](/img/docs/dbt-platform/platform-configuring-dbt-platform/connections-post-rollout-5.png?v=2 "Connections normalized")](#)Connections normalized #### IP Restrictions dbt will always connect to your data platform from the IP addresses specified in the [Regions & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) page. Be sure to allow traffic from these IPs in your firewall, and include them in any database grants. Allowing these IP addresses only enables the connection to your data warehouse. However, you might want to send API requests from your restricted network to the dbt API. Using the dbt API requires allowing the `cloud.getdbt.com` subdomain. For more on the dbt architecture, see [Deployment architecture](https://docs.getdbt.com/docs/platform/about-platform/architecture.md). --- ##### Connect Amazon Athena dbt platform Your environment(s) must be on a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to use the Amazon Athena connection. Connect dbt to Amazon's Athena interactive query service to build your dbt project. The following are the required and optional fields for configuring the Athena connection: | Field | Option | Description | Type | Required? | Example | | ----------------------------- | --------------------- | ------------------------------------------------------------------------------------------------ | ------- | --------- | --------------------- | | AWS region name | region\_name | AWS region of your Athena instance | String | Required | eu-west-1 | | Database (catalog) | database | Specify the database (Data catalog) to build models into (lowercase only) | String | Required | awsdatacatalog | | AWS S3 staging directory | s3\_staging\_dir | S3 location to store Athena query results and metadata | String | Required | s3://bucket/dbt/ | | Athena workgroup | work\_group | Identifier of Athena workgroup | String | Optional | my-custom-workgroup | | Athena Spark workgroup | spark\_work\_group | Identifier of Athena Spark workgroup for running Python models | String | Optional | my-spark-workgroup | | AWS S3 data directory | s3\_data\_dir | Prefix for storing tables, if different from the connection's s3\_staging\_dir | String | Optional | s3://bucket2/dbt/ | | AWS S3 data naming convention | s3\_data\_naming | How to generate table paths in s3\_data\_dir | String | Optional | schema\_table\_unique | | AWS S3 temp tables prefix | s3\_tmp\_table\_dir | Prefix for storing temporary tables, if different from the connection's s3\_data\_dir | String | Optional | s3://bucket3/dbt/ | | Poll interval | poll\_interval | Interval in seconds to use for polling the status of query results in Athena | Integer | Optional | 5 | | Query retries | num\_retries | Number of times to retry a failing query | Integer | Optional | 3 | | Boto3 retries | num\_boto3\_retries | Number of times to retry boto3 requests (for example, deleting S3 files for materialized tables) | Integer | Optional | 5 | | Iceberg retries | num\_iceberg\_retries | Number of times to retry iceberg commit queries to fix ICEBERG\_COMMIT\_ERROR | Integer | Optional | 0 | ##### Development credentials Enter your *development* (not deployment) credentials with the following fields: | Field | Option | Description | Type | Required | Example | | --------------------- | ------------------------ | -------------------------------------------------------------------------- | ------- | -------- | ---------------------------------------- | | AWS Access Key ID | aws\_access\_key\_id | Access key ID of the user performing requests | String | Required | AKIAIOSFODNN7EXAMPLE | | AWS Secret Access Key | aws\_secret\_access\_key | Secret access key of the user performing requests | String | Required | wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY | | Schema | schema | Specify the schema (Athena database) to build models into (lowercase only) | String | Required | dbt | | Threads | threads | | Integer | Optional | 3 | ##### Temporary credentials If you prefer to not store long-lived IAM user AWS Access Key ID and AWS Secret Access Key in the dbt platform, you can use `aws_session_token`, which is part of [temporary AWS Security Token Service (STS) credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html), instead. This approach mirrors a user or role’s long-term permissions. To use temporary credentials: 1. Enter the **AWS Access Key ID** and **AWS Secret Access Key** in the dbt platform UI: * For a deployment environment, set them on the [connection profile](https://docs.getdbt.com/docs/platform/about-profiles.md). * For a development environment, set them in the [development environment settings](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment). 2. Since the `aws_session_token` isn't available as a UI field in dbt platform, add it with [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes), which explains where to configure them for deployment and development environments. When you set the `aws_session_token` in **Extended attributes**, the key is case-sensitive and must be exactly `aws_session_token`. For example, you can use an [environment variable](https://docs.getdbt.com/docs/build/environment-variables.md) so the token isn't stored in the text box UI: ```yaml aws_session_token: '{{ env_var(''DBT_ENV_AWS_SESSION_TOKEN'') }}' ``` 3. [Assign](https://docs.getdbt.com/docs/build/environment-variables.md#setting-environment-variables) `DBT_ENV_AWS_SESSION_TOKEN`: * For a deployment connection profile, set a value in the **Project default** column. Profile-backed `env_var()` references need a project-scoped value for connection tests. * For a development environment, you can assign the variable for that environment. Since temporary credentials expire, you need to refresh your STS credentials and update the variable value before expiration. --- ##### Connect Apache Spark dbt platform If you're using Databricks, use `dbt-databricks` If you're using Databricks, the `dbt-databricks` adapter is recommended over `dbt-spark`. If you're still using dbt-spark with Databricks consider [migrating from the dbt-spark adapter to the dbt-databricks adapter](https://docs.getdbt.com/guides/migrate-from-spark-to-databricks.md). For the Databricks version of this page, refer to [Databricks setup](#databricks-setup). note See [Connect Databricks](#connect-databricks) for the Databricks version of this page. dbt supports connecting to an Apache Spark cluster using the HTTP method or the Thrift method. Note: While the HTTP method can be used to connect to an all-purpose Databricks cluster, the ODBC method is recommended for all Databricks connections. For further details on configuring these connection parameters, please see the [dbt-spark documentation](https://github.com/dbt-labs/dbt-spark#configuring-your-profile). To learn how to optimize performance with data platform-specific configurations in dbt, refer to [Apache Spark-specific configuration](https://docs.getdbt.com/reference/resource-configs/spark-configs.md). The following fields are available when creating an Apache Spark connection using the HTTP and Thrift connection methods: | Field | Description | Examples | | --------------------- | --------------------------------------------------------------- | ----------------------- | | Host Name | The hostname of the Spark cluster to connect to | `yourorg.sparkhost.com` | | Port | The port to connect to Spark on | 443 | | Organization | Optional (default: 0) | 0123456789 | | Cluster | The ID of the cluster to connect to | 1234-567890-abc12345 | | Connection Timeout | Number of seconds after which to timeout a connection | 10 | | Connection Retries | Number of times to attempt connecting to cluster before failing | 10 | | User | Optional | dbt\_cloud\_user | | Auth | Optional, supply if using Kerberos | `KERBEROS` | | Kerberos Service Name | Optional, supply if using Kerberos | `hive` | [![Configuring a Spark connection](/img/docs/dbt-platform/platform-configuring-dbt-platform/spark-connection.png?v=2 "Configuring a Spark connection")](#)Configuring a Spark connection --- ##### Connect Azure Synapse Analytics dbt platform #### Supported authentication methods The supported authentication methods are: * Microsoft Entra ID service principal * Active Directory password * SQL server authentication ##### Microsoft Entra ID service principal The following are the required fields for setting up a connection with Azure Synapse Analytics using Microsoft Entra ID service principal authentication. | Field | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | **Server** | The service principal's **Synapse host name** value (without the trailing string `, 1433`) for the Synapse test endpoint. | | **Port** | The port to connect to Azure Synapse Analytics. You can use `1433` (the default), which is the standard SQL server port number. | | **Database** | The service principal's **database** value for the Synapse test endpoint. | | **Authentication** | Choose **Service Principal** from the dropdown. | | **Tenant ID** | The service principal's **Directory (tenant) ID**. | | **Client ID** | The service principal's **application (client) ID**. | | **Client secret** | The service principal's **client secret** (not the **client secret id**). | ##### Active Directory password The following are the required fields for setting up a connection with Azure Synapse Analytics using Active Directory password authentication. | Field | Description | | ------------------ | ------------------------------------------------------------------------------------------------ | | **Server** | The server hostname to connect to Azure Synapse Analytics. | | **Port** | The server port. You can use `1433` (the default), which is the standard SQL server port number. | | **Database** | The database name. | | **Authentication** | Choose **Active Directory Password** from the dropdown. | | **User** | The AD username. | | **Password** | The AD username's password. | ##### SQL server authentication The following are the required fields for setting up a connection with Azure Synapse Analytics using SQL server authentication. | Field | Description | | ------------------ | ------------------------------------------------------------------------------------------------ | | **Server** | The server hostname or IP to connect to Azure Synapse Analytics. | | **Port** | The server port. You can use `1433` (the default), which is the standard SQL server port number. | | **Database** | The database name. | | **Authentication** | Choose **SQL** from the dropdown. | | **User** | The username. | | **Password** | The username's password. | #### Configuration To learn how to optimize performance with data platform-specific configurations in dbt, refer to [Microsoft Azure Synapse DWH configurations](https://docs.getdbt.com/reference/resource-configs/azuresynapse-configs.md). --- ##### Connect BigQuery dbt v2 compatible dbt platform To see which BigQuery functions are supported in dbt v2 in `strict` mode, refer to [BigQuery function support](https://docs.getdbt.com/reference/resource-configs/bigquery-function-support.md). #### Required permissions dbt user accounts need the following permissions to read from and create tables and views in a BigQuery project: * BigQuery Data Editor * BigQuery User (Applies to dbt v1.12 and later) dbt lists and looks up BigQuery routines (functions and stored procedures) when it scans datasets. If you use a custom IAM role instead of a predefined role, also grant: * `bigquery.routines.list`: list routines in a dataset * `bigquery.routines.get`: read routine metadata Without these, dbt can miss existing tables or fail on some lookups. Predefined roles such as BigQuery Data Editor typically already include them. For BigQuery with dbt v2, users also need: * BigQuery Read Session User (for Storage Read API access) To use the [Query History](https://docs.getdbt.com/docs/explore/model-query-history.md#bigquery-model-query-history) feature, add: * BigQuery Resource Viewer For BigQuery DataFrames, users need these additional permissions: * BigQuery Job User * BigQuery Read Session User * Notebook Runtime User * Code Creator * colabEnterpriseUser #### Warehouse permissions for dbt v2 The Google Cloud identity (service account or user) that dbt v2 uses must have IAM permissions to run jobs, read and write table data, and read metadata dbt v2 uses for introspection and source freshness. ##### Required Google Cloud objects Before connecting, these objects must exist or be accessible: | Object | BigQuery term | Purpose | | --------------------------- | ------------- | --------------------------------------------------- | | **Project** | GCP project | Your Google Cloud project ID | | **Dataset** | Dataset | Target dataset (equivalent to a schema) | | **Service account or user** | IAM identity | Identity for authentication | | **Location or region** | Location | Data location (for example, `US`, `EU`, `us-east1`) | ##### IAM permissions The following permissions are required for fundamental dbt features:: | Permission | Purpose | | ---------------------------- | ------------------------------ | | `bigquery.datasets.get` | Access dataset metadata | | `bigquery.tables.create` | Create tables | | `bigquery.tables.delete` | Drop or replace tables | | `bigquery.tables.get` | Read table metadata | | `bigquery.tables.getData` | Read table data | | `bigquery.tables.list` | List tables in dataset | | `bigquery.tables.update` | Update table schema | | `bigquery.tables.updateData` | Insert, update, or delete rows | | `bigquery.routines.get` | Read routine metadata | | `bigquery.routines.list` | List routines in a dataset | | `bigquery.jobs.create` | Run queries | The following are optional permissions for additional dbt features: | Permission | When required | | -------------------------- | -------------------------------- | | `bigquery.datasets.create` | Auto-create datasets | | `bigquery.routines.create` | Create UDFs or stored procedures | | `bigquery.jobs.get` | Monitor job status | | `storage.objects.create` | Python models (GCS staging) | | `dataproc.*` | Python models on Dataproc | ###### Predefined IAM roles The following roles represent the typical starting point for dbt access: | Role | Description | Use case | | --------------------------- | --------------------------------- | ----------------------- | | `roles/bigquery.dataEditor` | Read and write tables in datasets | Standard dbt operations | | `roles/bigquery.user` | Run jobs, create datasets | Job execution | | `roles/bigquery.jobUser` | Run jobs only | Minimal query execution | For Storage Read API access with dbt v2, also grant **BigQuery Read Session User** (`roles/bigquery.readSessionUser`) on the project, as noted in [Connect BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md#required-permissions). ##### Metadata operations The following are required for fundamental dbt features: | Query type | SQL or API used | Required permission | | ---------------- | --------------------------- | ------------------------------------------------ | | Get table schema | `get_table_schema` API | `bigquery.tables.get` | | List relations | Query against dataset | `bigquery.tables.list`, `bigquery.routines.list` | | Source freshness | Query `__TABLES__` metadata | `bigquery.tables.get` | | Get table stats | Query `INFORMATION_SCHEMA` | `bigquery.tables.get` | | Create dataset | `CREATE SCHEMA` | `bigquery.datasets.create` | ##### INFORMATION\_SCHEMA and metadata views dbt v2 queries these BigQuery system views: | View | Purpose | Scope | | ------------------------------- | ------------------------ | ----------------- | | `INFORMATION_SCHEMA.TABLES` | List tables | Dataset or region | | `INFORMATION_SCHEMA.COLUMNS` | Column metadata | Dataset or region | | `INFORMATION_SCHEMA.VIEWS` | View definitions | Dataset or region | | `INFORMATION_SCHEMA.PARTITIONS` | Partition information | Dataset only | | `__TABLES__` (deprecated) | Table modification times | Dataset | ##### BigQuery DataFrames (optional) For BigQuery DataFrames workflows, users typically need additional roles such as `BigQuery Job User`, `BigQuery Read Session User`, `Notebook Runtime User`, `Code Creator`, and `colabEnterpriseUser`. See your Google Cloud admin for exact role names in your organization. #### Authentication dbt supports different authentication methods depending on your environment and plan type: * Development environments support: * Service JSON * BigQuery OAuth [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Deployment environments support: * Service JSON * BigQuery Workload Identity Federation (WIF) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") These authentication methods are set up in the [global connections account settings](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md), rather than single sign-on or integration settings. When you create a new BigQuery connection, you will be presented with two schema options for the connection (both use the same adapter): * **BigQuery:** Supports all connection types (Use this option) * **BigQuery (Legacy):**  Supports all connection types except for WIF (Deprecated feature. Do not use.) All new connections should use the **BigQuery** option as **BigQuery (Legacy)** will be deprecated. To update existing connections and credentials in an environment to use the new BigQuery option, first, use the [APIs](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to remove the configurations. ##### JSON keyfile Uploading a service account JSON keyfile While the fields in a BigQuery connection can be entered manually, we recommend uploading a service account JSON keyfile to quickly and accurately configure a connection to BigQuery. You can provide the JSON keyfile in one of two formats: * JSON keyfile upload — Upload the keyfile directly using its normal JSON format. * Base64-encoded string — Provide the keyfile as a base64-encoded string. When you provide a base64-encoded string, dbt decodes it automatically and populates the necessary fields. The JSON keyfile option is available for configuring both **development** and **deployment** environments. Uploading a valid JSON keyfile will populate the following fields: * Project ID * Private key ID * Private key * Client email * Client ID * Auth URI * Token URI * Auth provider x509 cert url * Client x509 cert url In addition to these fields, two other optional fields can be configured in a BigQuery connection: | Field | Description | Examples | | -------- | ------------------------------------------------------------------------------------------------------------ | ---------- | | Timeout | Deprecated; exists for backwards compatibility with older versions of dbt and will be removed in the future. | `300` | | Location | The [location](https://cloud.google.com/bigquery/docs/locations) where dbt should create datasets. | `US`, `EU` | [![A valid BigQuery connection](/img/docs/dbt-platform/platform-configuring-dbt-platform/bigquery-connection.png?v=2 "A valid BigQuery connection")](#)A valid BigQuery connection ##### BigQuery OAuth **Available in:** Development environments, Enterprise-tier plans only The OAuth auth method permits dbt to run queries on behalf of a BigQuery user or workload without storing the BigQuery service account keyfile in dbt. However, the JSON must still be provided, or fields must be manually filled out to complete the configuration in dbt Cloud. Those values do not have to be real for this bypass to work (for example, they can be `N/A`). For more information on the initial configuration of a BigQuery OAuth connection in dbt>, please see [the docs on setting up BigQuery OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md). As an end user, if your organization has set up BigQuery OAuth, you can link a project with your personal BigQuery account in your Profile in dbt. ##### BigQuery Workload Identity Federation [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") note If you're using BigQuery WIF, we recommend using it with BigQuery OAuth. Otherwise, you must create two connections - one with service JSON and one with WIF to use service JSON for development environments. **Available in:** Deployment environments The BigQuery WIF auth method permits dbt to run deployment queries as a service account without configuring a BigQuery service account keyfile in dbt. For more information on the initial configuration of a BigQuery WIF connection in dbt, refer to [Set up BigQuery Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation). #### Configuration To learn how to optimize performance with data platform-specific configurations in dbt, refer to [BigQuery-specific configuration](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md). ##### Optional configurations In BigQuery, optional configurations let you tailor settings for tasks such as query priority, dataset location, job timeout, and more. These options give you greater control over how BigQuery functions behind the scenes to meet your requirements. To customize your optional configurations in dbt: 1. Click your account name at the bottom left-hand menu and go to **Account settings** > **Projects**. 2. Select your BigQuery project. 3. Go to **Development connection** and select **BigQuery**. 4. Click **Edit** and then scroll down to **Optional settings**. [![BigQuery optional configuration](/img/bigquery/bigquery-optional-config.png?v=2 "BigQuery optional configuration")](#)BigQuery optional configuration The following are the optional configurations you can set in dbt: | Configuration | Information | Type | Example | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------- | | [Priority](#priority) | Sets the priority for BigQuery jobs (either `interactive` or queued for `batch` processing) | String | `batch` or `interactive` | | [Retries](#retries) | Specifies the number of retries for failed jobs due to temporary issues | Integer | `3` | | [Location](#location) | Location for creating new datasets | String | `US`, `EU`, `us-west2` | | [Maximum bytes billed](#maximum-bytes-billed) | Limits the maximum number of bytes that can be billed for a query | Integer | `1000000000` | | [Execution project](#execution-project) | Specifies the project ID to bill for query execution | String | `my-project-id` | | [Impersonate service account](#impersonate-service-account) | Allows users authenticated locally to access BigQuery resources under a specified service account | String | `service-account@project.iam.gserviceaccount.com` | | [Job retry deadline seconds](#job-retry-deadline-seconds) | Sets the total number of seconds BigQuery will attempt to retry a job if it fails | Integer | `600` | | [Job creation timeout seconds](#job-creation-timeout-seconds) | Specifies the maximum timeout for the job creation step | Integer | `120` | | [Google cloud storage-bucket](#google-cloud-storage-bucket) | Location for storing objects in Google Cloud Storage | String | `my-bucket` | | [Dataproc region](#dataproc-region) | Specifies the cloud region for running data processing jobs | String | `US`, `EU`, `asia-northeast1` | | [Dataproc cluster name](#dataproc-cluster-name) | Assigns a unique identifier to a group of virtual machines in Dataproc | String | `my-cluster` | | [Notebook Template ID](#notebook-template-id) | Unique identifier to a Colab Enterprise notebook runtime | Integer | `7018811640745295872` | | [Compute Region](#compute-region) | Assigns a unique identifier to a group of virtual machines in Dataproc | String | `US`, `EU`, `asia-northeast1` |  Priority The `priority` for the BigQuery jobs that dbt executes can be configured with the `priority` configuration in your BigQuery profile. The priority field can be set to one of `batch` or `interactive`. For more information on query priority, consult the [BigQuery documentation](https://cloud.google.com/bigquery/docs/running-queries).  Retries Retries in BigQuery help to ensure that jobs complete successfully by trying again after temporary failures, making your operations more robust and reliable.  Location The `location` of BigQuery datasets can be set using the `location` setting in a BigQuery profile. As per the [BigQuery documentation](https://cloud.google.com/bigquery/docs/locations), `location` may be either a multi-regional location (for example, `EU`, `US`), or a regional location (like `us-west2`).  Maximum bytes billed When a `maximum_bytes_billed` value is configured for a BigQuery profile, that allows you to limit how much data your query can process. It’s a safeguard to prevent your query from accidentally processing more data than you expect, which could lead to higher costs. Queries executed by dbt will fail if they exceed the configured maximum bytes threshold. This configuration should be supplied as an integer number of bytes. If your `maximum_bytes_billed` is 1000000000, you would enter that value in the `maximum_bytes_billed` field in dbt.  Execution project By default, dbt will use the specified `project`/`database` as both: 1. The location to materialize resources (models, seeds, snapshots, and so on), unless they specify a custom project/database config 2. The GCP project that receives the bill for query costs or slot usage Optionally, you may specify an execution project to bill for query execution, instead of the project/database where you materialize most resources.  Impersonate service account This feature allows users authenticating using local OAuth to access BigQuery resources based on the permissions of a service account. For a general overview of this process, see the official docs for [Creating Short-lived Service Account Credentials](https://cloud.google.com/iam/docs/create-short-lived-credentials-direct).  Job retry deadline seconds Job retry deadline seconds is the maximum amount of time BigQuery will spend retrying a job before it gives up.  Job creation timeout seconds Job creation timeout seconds is the maximum time BigQuery will wait to start the job. If the job doesn’t start within that time, it times out. From dbt v1.10, the BigQuery adapter cancels BigQuery jobs that exceed their configured timeout by sending a cancellation request. If the request succeeds, dbt stops the job. If the request fails, the BigQuery job may keep running in the background until it finishes or you cancel it manually. ###### Run dbt python models on Google Cloud Platform To run dbt Python models on GCP, dbt uses companion services, Dataproc and Cloud Storage, that offer tight integrations with BigQuery. You may use an existing Dataproc cluster and Cloud Storage bucket, or create new ones: * <https://cloud.google.com/dataproc/docs/guides/create-cluster> * <https://cloud.google.com/storage/docs/creating-buckets>  Google cloud storage bucket Everything you store in Cloud Storage must be placed inside a [bucket](https://cloud.google.com/storage/docs/buckets). Buckets help you organize your data and manage access to it.  Dataproc region A designated location in the cloud where you can run your data processing jobs efficiently. This region must match the location of your BigQuery dataset if you want to use Dataproc with BigQuery to ensure data doesn't move across regions, which can be inefficient and costly. For more information on [Dataproc regions](https://cloud.google.com/bigquery/docs/locations), refer to the BigQuery documentation.  Dataproc cluster name A unique label you give to your group of virtual machines to help you identify and manage your data processing tasks in the cloud. When you integrate Dataproc with BigQuery, you need to provide the cluster name so BigQuery knows which specific set of resources (the cluster) to use for running the data jobs. Have a look at [Dataproc's document on Create a cluster](https://cloud.google.com/dataproc/docs/guides/create-cluster) for an overview on how clusters work.  Notebook Template ID The unique identifier associated with a specific Colab notebook, which acts are the python runtime for BigQuery DataFrames.  Compute Region This region must match the location of your BigQuery dataset if you want to use BigQuery DataFrames, ensure the Colab runtime is also within the same region. ##### Account level connections and credential management You can re-use connections across multiple projects with [global connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#migration-from-project-level-connections-to-account-level-connections). Connections are attached at the environment level (formerly project level), so you can use multiple connections inside of a single project (to handle dev, staging, production, and more). BigQuery connections in dbt currently expect the credentials to be handled at the connection level (and only BigQuery connections). This was originally designed to facilitate creating a new connection by uploading a service account keyfile. This describes how to override those credentials with [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes), *so project administrators can manage credentials independently* of the account-level connection details. Refer to [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) for where to configure them for deployment and development environments. For a project, you will first create an environment variable to store the secret `private_key` value. Then, you will use extended attributes to override the entire service account JSON (you can't only override the secret key due to a constraint of extended attributes). 1. **New environment variable** * Create a new *secret* [environment variable](https://docs.getdbt.com/docs/build/environment-variables.md#handling-secrets) to handle the private key: `DBT_ENV_SECRET_PROJECTXXX_PRIVATE_KEY` * Fill in the private key value according the environment To automate your deployment, use the following [admin API request](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Create%20Projects%20Environment%20Variables%20Bulk), with `XXXXX` your account number, `YYYYY` your project number, `ZZZZZ` your [API token](https://docs.getdbt.com/docs/dbt-apis/authentication.md): ```shell curl --request POST \ --url https://cloud.getdbt.com/api/v3/accounts/XXXXX/projects/YYYYY/environment-variables/bulk/ \ --header 'Accept: application/json' \ --header 'Authorization: Bearer ZZZZZ' \ --header 'Content-Type: application/json' \ --data '{ "env_var": [ { "new_name": "DBT_ENV_SECRET_PROJECTXXX_PRIVATE_KEY", "project": "Value by default for the entire project", "ENVIRONMENT_NAME_1": "Optional, if wanted, value for environment name 1", "ENVIRONMENT_NAME_2": "Optional, if wanted, value for environment name 2" } ] }' ``` 2. **Extended attributes** Complete the [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) block with the following payload (replacing `XXX` with your corresponding information). That page explains where to configure them for deployment and development environments: ```yaml keyfile_json: type: service_account project_id: xxx private_key_id: xxx private_key: '{{ env_var(''DBT_ENV_SECRET_PROJECTXXX_PRIVATE_KEY'') }}' client_email: xxx client_id: xxx auth_uri: xxx token_uri: xxx auth_provider_x509_cert_url: xxx client_x509_cert_url: xxx ``` If you require [other fields](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#service-account-json) to be overridden at the environment level via extended attributes, please respect the [expected indentation](https://docs.getdbt.com/docs/dbt-platform-environments.md#only-the-top-level-keys-are-accepted-in-extended-attributes) (ordering doesn't matter): ```yaml priority: interactive keyfile_json: type: xxx project_id: xxx private_key_id: xxx private_key: '{{ env_var(''DBT_ENV_SECRET_PROJECTXXX_PRIVATE_KEY'') }}' client_email: xxx client_id: xxx auth_uri: xxx token_uri: xxx auth_provider_x509_cert_url: xxx client_x509_cert_url: xxx execution_project: buck-stops-here-456 ``` To automate your deployment, you first need to [create the extended attributes payload](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Create%20Extended%20Attributes) for a given project, and then [assign it](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Update%20Environment) to a specific environment. With `XXXXX` as your account number, `YYYYY` as your project number, and `ZZZZZ` as your [API token](https://docs.getdbt.com/docs/dbt-apis/authentication.md): ```shell curl --request POST \ --url https://cloud.getdbt.com/api/v3/accounts/XXXXX/projects/YYYYY/extended-attributes/ \ --header 'Accept: application/json' \ --header 'Authorization: Bearer ZZZZZ' \ --header 'Content-Type: application/json' \ --data '{ "id": null, "extended_attributes": {"type":"service_account","project_id":"xxx","private_key_id":"xxx","private_key":"{{ env_var('DBT_ENV_SECRET_PROJECTXXX_PRIVATE_KEY') }}","client_email":"xxx","client_id":xxx,"auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"xxx"}, "state": 1 }' ``` *Make a note of the `id` returned in the message.* It will be used in the following call. With `EEEEE` the environment id, `FFFFF` the extended attributes id: ```shell curl --request POST \ --url https://cloud.getdbt.com/api/v3/accounts/XXXXX/projects/YYYYY/environments/EEEEE/ \ --header 'Accept: application/json' \ --header 'Authorization: Bearer ZZZZZZ' \ --header 'Content-Type: application/json' \ --data '{ "extended_attributes_id": FFFFF }' ``` --- ##### Connect ClickHouse Private beta dbt v2 compatible ### Connect ClickHouse [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2 compatible Available in v2 | dbt platform The dbt v2 in dbt platform supports connecting to [ClickHouse Cloud](https://clickhouse.com/cloud) and to self-managed single-node ClickHouse. Use a dbt v2 release track for the environment that uses this connection. ClickHouse private beta ClickHouse connections on v2 are in private beta and not production-ready. To request access, contact your account representative. Expect some minor bugs, and avoid using them in production environments for now. Refer to [Limitations](#limitations) before you connect. #### Warehouse permissions for dbt v2 The ClickHouse user that the dbt v2 connects as must be able to run dbt workloads in the target database and read the system tables used for introspection. ##### Required ClickHouse objects Before connecting, these objects must exist or be accessible: | Object | Purpose | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Service (ClickHouse Cloud) or server (self-managed, single node) | Compute resource | | Database | Target database. ClickHouse has no separate schema level, so the dbt `schema` maps to a database | | User | Database user for authentication | ##### Core permissions The following permissions are required for fundamental dbt features: | Permission | Object | Purpose | | -------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SELECT` | Tables and views | Read data | | `INSERT` | Tables | Load models, seeds, and snapshots | | `ALTER` | Tables | Schema changes (`on_schema_change`), indexes and projections, comments (`persist_docs`), `REPLACE PARTITION` (`insert_overwrite`), and lightweight deletes (`delete+insert`) | | `TRUNCATE` | Tables | Full-refresh seeds | | `CREATE TABLE`, `CREATE VIEW`, `CREATE DICTIONARY` | Database | Create materializations, including the intermediate and backup relations used for atomic rebuilds (`EXCHANGE TABLES`, `RENAME TABLE`) | | `DROP TABLE`, `DROP VIEW`, `DROP DICTIONARY` | Database | Drop or replace objects | ##### Metadata operations dbt v2 reads these ClickHouse system tables: | System table | Purpose | | ----------------------- | -------------------------------------------------------------------------------- | | `system.tables` | List relations, build the catalog, detect materialized views pointing at a table | | `system.columns` | Column metadata for the catalog and for schema-change detection | | `system.databases` | Check whether the target database exists | | `system.settings` | Capability probes (lightweight deletes, `insert_distributed_sync`) | | `system.view_refreshes` | Validate refreshable materialized view dependencies | ClickHouse filters system tables to the objects the user can access, so no separate grant is needed for them. ##### Database management Conditional permissions for database management: | Permission | Object | When required | | ----------------- | ------ | --------------------------------------------------------- | | `CREATE DATABASE` | Server | Auto-create the target database when it doesn't exist yet | For the full privilege model, refer to [access control in the ClickHouse documentation](https://clickhouse.com/docs/operations/access-rights). #### Connection fields Configure the following fields when you create a ClickHouse connection. | Field | Description | Type | Required? | Example | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | --------- | --------------------------------------- | | Server Hostname | The ClickHouse Cloud endpoint URL. Do not include `https://` or the port. Copy this from **Connect** in the [ClickHouse Cloud console](https://clickhouse.com/docs/get-started/quickstarts/obtain-your-cloud-connection-details#find-your-connection-details). | String | Required | `abc123.us-east-1.aws.clickhouse.cloud` | | Port | The port to connect to. The dbt ClickHouse adapter connects over HTTPS. Use port `8443`. Port `9440` (native protocol) is not supported by the adapter. | String | Optional | `8443` | | Database | The name of the database to connect to. | String | Optional | `default` | [![Example of the ClickHouse connection fields.](/img/docs/dbt-platform/clickhouse-connection.png?v=2 "Example of the ClickHouse connection fields.")](#)Example of the ClickHouse connection fields. After you save the connection, set up your development environment: 1. Create a new project or open an existing one. 2. In your project settings, select **Environments** from the left menu and open your development environment. 3. Under **Connection**, select the ClickHouse connection you just created. 4. Save the environment. [![Select the ClickHouse connection for the development environment.](/img/docs/dbt-platform/clickhouse-environment.png?v=2 "Select the ClickHouse connection for the development environment.")](#)Select the ClickHouse connection for the development environment. #### Development and deployment credentials Each developer enters personal development credentials in **Your profile** → **Credentials**. For ClickHouse Cloud, copy the username and password from the **Connect** dialog in the [ClickHouse Cloud console](https://clickhouse.com/docs/get-started/quickstarts/obtain-your-cloud-connection-details#find-your-connection-details). The username is typically `default`. | Field | Description | Type | Required? | Example | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------- | --------- | ------------------- | | Username | The database username. | String | Required | `default` | | Password | The database password. | String | Optional | DatabasePassword123 | | Schema | In development, dbt builds your models into a schema with this name. Use a schema unique to your personal development environment. | String | Required | dbtlabsdocstest | | Target name | The target name for this credential. | String | Optional | `default` | | Threads | The number of threads to use for dbt operations. | Integer | Optional | `4` | [![Example of the ClickHouse user credential fields.](/img/docs/dbt-platform/clickhouse-credentials.png?v=2 "Example of the ClickHouse user credential fields.")](#)Example of the ClickHouse user credential fields. #### Configuration To learn how to optimize performance with data platform-specific configurations in dbt, refer to [ClickHouse configurations](https://docs.getdbt.com/reference/resource-configs/clickhouse-configs.md). For a description of the ClickHouse profile fields that the connection maps to, refer to [ClickHouse setup](https://docs.getdbt.com/docs/local/connect-data-platform/clickhouse-setup.md). #### Limitations The ClickHouse connection is in private beta. On the dbt platform specifically: * The dbt Semantic Layer isn't supported for ClickHouse connections yet. * Only username and password authentication is available. OAuth, key pair authentication, SSH tunneling, and private connectivity aren't supported for ClickHouse yet. The dbt v2 ClickHouse adapter limitations apply here too, including the gaps in clusters, grants, `dbt clone`, `dbt source freshness`, and SQL comprehension. For the complete picture, refer to [ClickHouse limitations](https://docs.getdbt.com/docs/local/connect-data-platform/clickhouse-setup.md#limitations). --- ##### Connect Databricks dbt v2 compatible dbt platform The dbt-databricks adapter is maintained by the Databricks team. The Databricks team is committed to supporting and improving the adapter over time, so you can be sure the integrated experience will provide the best of dbt and the best of Databricks. Connecting to Databricks via dbt-spark has been deprecated. #### About the dbt-databricks adapter dbt-databricks is compatible with the following versions of dbt v1 in dbt with varying degrees of functionality. | Feature | dbt Versions | | -------------- | -------------------------------------- | | dbt-databricks | Available starting with dbt 1.0 in dbt | | Unity Catalog | Available starting with dbt 1.1 | | Python models | Available starting with dbt 1.3 | The dbt-databricks adapter offers: * **Easier set up** * **Better defaults:** The dbt-databricks adapter is more opinionated, guiding users to an improved experience with less effort. Design choices of this adapter include defaulting to Delta format, using merge for incremental models, and running expensive queries with Photon. * **Support for Unity Catalog:** Unity Catalog allows Databricks users to centrally manage all data assets, simplifying access management and improving search and query performance. Databricks users can now get three-part data hierarchies – catalog, schema, model name – which solves a longstanding friction point in data organization and governance. To learn how to optimize performance with data platform-specific configurations in dbt, refer to [Databricks-specific configuration](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md). To grant users or roles database permissions (access rights and privileges), refer to the [example permissions](https://docs.getdbt.com/reference/database-permissions/databricks-permissions.md) page. #### Warehouse permissions for dbt v2 The Databricks user or service principal that dbt v2 uses must have privileges on the catalog and schemas where models run, plus access required for metadata queries. Requirements depend on whether you use Unity Catalog or the legacy Hive Metastore. ##### Required Databricks objects Before connecting, these objects must exist or be accessible: | Object | Purpose | | ----------------------------- | --------------------------------------- | | **Workspace** | Your Databricks workspace URL (host) | | **SQL warehouse or cluster** | Compute resource (using `http_path`) | | **Catalog** | Unity Catalog or Hive Metastore catalog | | **Schema** | Target schema within the catalog | | **User or service principal** | Identity for authentication | ##### Unity Catalog Required access for the Unity Catalog: | Permission | Object | Purpose | | --------------- | --------------- | ------------------------------------ | | `USE CATALOG` | Catalog | Access the catalog | | `USE SCHEMA` | Schema | Access schemas | | `SELECT` | Tables or views | Read existing data and sources | | `CREATE TABLE` | Schema | Create models materialized as tables | | `CREATE VIEW` | Schema | Create models materialized as views | | `MODIFY` | Tables | Insert, update, and delete data | | `CREATE SCHEMA` | Catalog | Auto-create schemas (if needed) | ##### Hive Metastore Required access for the legacy Hive Metastore: | Permission | Object | Purpose | | ---------- | -------- | ----------------------- | | `USAGE` | Database | Access the database | | `SELECT` | Tables | Read data | | `CREATE` | Database | Create tables and views | | `MODIFY` | Tables | Modify data | ##### Metadata operations The following are required for fundamental dbt features: | Query type | SQL used | Required permission | | ---------------------------- | ---------------------------------------------------- | -------------------------- | | Get table schema | `DESCRIBE TABLE` or `DESCRIBE TABLE EXTENDED` | SELECT on table | | Get table schema (DBR 16.2+) | `DESCRIBE TABLE EXTENDED ... AS JSON` | SELECT on table | | List relations | Query `INFORMATION_SCHEMA.TABLES` | USE CATALOG and USE SCHEMA | | Source freshness | Query `INFORMATION_SCHEMA.TABLES` for `last_altered` | USE CATALOG | | Get view definition | Query `SYSTEM.INFORMATION_SCHEMA.VIEWS` | Access to system catalog | | Create schemas | `CREATE SCHEMA IF NOT EXISTS` | CREATE SCHEMA on catalog | ##### Python models Optional permissions for environments using Python models | Permission | Object | Purpose | | -------------------- | ---------------------- | --------------------------- | | Workspace API access | `/api/2.0/workspace/*` | Create notebook directories | | Notebook import | Workspace | Import Python notebooks | | Job execution | Cluster or warehouse | Run Python models | #### Connection fields To set up the Databricks connection, supply the following fields: | Field | Description | Examples | | --------------- | -------------------------------------------------------- | -------------------------------------- | | Server Hostname | The hostname of the Databricks account to connect to | dbc-a2c61234-1234.cloud.databricks.com | | HTTP Path | The HTTP path of the Databricks cluster or SQL warehouse | /sql/1.0/warehouses/1a23b4596cd7e8fg | | Catalog | Name of Databricks Catalog (optional) | Production | [![Configuring a Databricks connection using the dbt-databricks adapter](/img/docs/dbt-platform/platform-configuring-dbt-platform/dbt-databricks.png?v=2 "Configuring a Databricks connection using the dbt-databricks adapter")](#)Configuring a Databricks connection using the dbt-databricks adapter --- ##### Connect Microsoft Fabric dbt platform #### Supported authentication methods The supported authentication methods are: * Microsoft Entra service principal * Microsoft Entra password SQL password (LDAP) is not supported in Microsoft Fabric Data Warehouse so you must use Microsoft Entra ID. This means that to use [Microsoft Fabric](https://www.microsoft.com/en-us/microsoft-fabric) in dbt, you will need at least one Microsoft Entra service principal to connect dbt to Fabric, ideally one service principal for each user. ##### Microsoft Entra service principal The following are the required fields for setting up a connection with a Microsoft Fabric using Microsoft Entra service principal authentication. | Field | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------------ | | **Server** | The service principal's **host** value for the Fabric test endpoint. | | **Port** | The port to connect to Microsoft Fabric. You can use `1433` (the default), which is the standard SQL server port number. | | **Database** | The service principal's **database** value for the Fabric test endpoint. | | **Authentication** | Choose **Service Principal** from the dropdown. | | **Tenant ID** | The service principal's **Directory (tenant) ID**. | | **Client ID** | The service principal's **application (client) ID**. | | **Client secret** | The service principal's **client secret** (not the **client secret id**). | ##### Microsoft Entra password The following are the required fields for setting up a connection with a Microsoft Fabric using Microsoft Entra password authentication. | Field | Description | | ------------------ | ------------------------------------------------------------------------------------------------ | | **Server** | The server hostname to connect to Microsoft Fabric. | | **Port** | The server port. You can use `1433` (the default), which is the standard SQL server port number. | | **Database** | The database name. | | **Authentication** | Choose **Active Directory Password** from the dropdown. | | **User** | The Microsoft Entra username. | | **Password** | The Microsoft Entra password. | #### Configuration To learn how to optimize performance with data platform-specific configurations in dbt, refer to [Microsoft Fabric Data Warehouse configurations](https://docs.getdbt.com/reference/resource-configs/fabric-configs.md). --- ##### Connect Onehouse dbt platform dbt supports connecting to [Onehouse SQL](https://www.onehouse.ai/product/quanton) using the Apache Spark Connector with the Thrift method. note Connect to a Onehouse SQL Cluster with the [dbt-spark](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-apache-spark.md) adapter.\*\* #### Requirements * For dbt, ensure your Onehouse SQL endpoint is accessible via external DNS/IP, whitelisting dbt IPs. #### What works * All dbt Commands, including: `dbt clean`, `dbt compile`, `dbt debug`, `dbt seed`, and `dbt run`. * dbt materializations: `table` and `incremental` * Apache Hudi table types of Merge on Read (MoR) and Copy on Write (CoW). It is recommended to use MoR for mutable workloads. #### Limitations * Views are not supported * `dbt seed` has row / record limits. * `dbt seed` only supports Copy on Write tables. #### dbt connection Fill in the following fields when creating an **Apache Spark** warehouse connection using the Thrift connection method: | Field | Description | Examples | | --------------------- | --------------------------------------------------------------- | --------------------------- | | Method | The method for connecting to Spark | Thrift | | Hostname | The hostname of your Onehouse SQL Cluster endpoint | `yourProject.sparkHost.com` | | Port | The port to connect to Spark on | 10000 | | Cluster | Onehouse does not use this field | | | Connection Timeout | Number of seconds after which to timeout a connection | 10 | | Connection Retries | Number of times to attempt connecting to cluster before failing | 0 | | Organization | Onehouse does not use this field | | | User | Optional. Not enabled by default. | dbt\_cloud\_user | | Auth | Optional, supply if using Kerberos. Not enabled by default. | `KERBEROS` | | Kerberos Service Name | Optional, supply if using Kerberos. Not enabled by default. | `hive` | [![Onehouse configuration](/img/onehouse/onehouse-dbt.png?v=2 "Onehouse configuration")](#)Onehouse configuration #### dbt project We recommend that you set default configurations on the dbt\_project.yml to ensure that the adapter executes with Onehouse compatible sql | Field | Description | Required | Default | Recommended | | ----------------- | ----------------------------------------------------- | -------- | --------------------- | ------------------------------ | | materialized | materialization the project/directory will default to | Yes | without input, `view` | `table` | | file\_format | table format the project will default to | Yes | N/A | hudi | | location\_root | Location of the database in DFS | Yes | N/A | `<your_database_location_dfs>` | | hoodie.table.type | Merge on Read or Copy on Write | No | cow | mor | dbt\_project.yml template ```yml +materialized: table | incremental +file_format: hudi +location_root: <storage_uri> +tblproperties: hoodie.table.type: mor | cow ``` A dbt\_project.yml example if using jaffle shop would be ```sql models: jaffle_shop: +file_format: hudi +location_root: s3://lakehouse/demolake/dbt_ecomm/ +tblproperties: hoodie.table.type: mor staging: +materialized: incremental marts: +materialized: table ``` --- ##### Connect PostgreSQL, Lakebase and AlloyDB dbt platform dbt platform supports connecting to PostgreSQL and Postgres-compatible databases (AlloyDB, Lakebase). The following fields are required when creating a connection: | Field | Description | Examples | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | | Host Name | The hostname of the database to connect to. This can either be a hostname or an IP address. Refer to the [set up pages](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md) for adapter-specific details. | Postgres: `xxx.us-east-1.amazonaws.com` | | Port | Usually 5432 | `5439` | | Database | The logical database to connect to and run queries against. | `analytics` | **Note**: When you set up a Postgres connection in dbt, SSL-related parameters aren't available as inputs. [![Configuring a Postgres connection](/img/docs/dbt-platform/platform-configuring-dbt-platform/postgres-redshift-connection.png?v=2 "Configuring a Postgres connection")](#)Configuring a Postgres connection ##### Authentication Parameters For authentication, dbt users can use **Database username and password** for Postgres and Postgres-compatible databases. For more information on what is supported, check out the database specific setup page for limitations and helpful tips. The following table contains the parameters for the database (password-based) connection method. | Field | Description | Examples | | ---------- | ----------------------------------------- | ------------ | | `user` | Account username to log into your cluster | myuser | | `password` | Password for authentication | \_password1! | ##### Connecting using an SSH Tunnel Use an SSH tunnel when your Postgres or AlloyDB instance is not publicly accessible and must be reached through a [bastion server](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md#about-the-bastion-server-in-aws). When enabled, dbt platform connects to your database by first establishing a secure connection to the bastion host, which then forwards traffic to your database. To configure a connection using an SSH tunnel: 1. Navigate to **Account settings** (by clicking on your account name in the left side menu) and select **Connections**. 2. Select an existing connection to edit it, or click **+ New connection**. 3. In **Connection settings**, ensure **SSH Tunnel Enabled** is checked. 4. Enter the hostname, username, and port for the bastion server. [![A public key is generated after saving](/img/docs/dbt-platform/platform-configuring-dbt-platform/postgres-redshift-ssh-tunnel.png?v=2 "A public key is generated after saving")](#)A public key is generated after saving 5. Click **Save**. dbt platform generates and displays a public key. 6. Copy the newly generated public key to the bastion server and add it to the server’s `authorized_keys` file to authorize dbt platform to connect through the bastion host. If the new key is not added, the SSH tunnel connection will fail. important Each time you create and save a new SSH tunnel connection, dbt platform generates a unique SSH key pair, even when the connection details are identical to an existing connection. ###### About the Bastion server in AWS What is a bastion server? A bastion server in [Amazon Web Services (AWS)](https://aws.amazon.com/blogs/security/how-to-record-ssh-sessions-established-through-a-bastion-host/) is a host that allows dbt to open an SSH connection. <br /> dbt only sends queries and doesn't transmit large data volumes. This means the bastion server can run on an AWS instance of any size, like a t2.small instance or t2.micro.<br /><br /> Make sure the location of the instance is the same Virtual Private Cloud (VPC) as the Postgres instance, and configure the security group for the bastion server to ensure that it's able to connect to the warehouse port. ###### Configuring the Bastion Server in AWS To configure the SSH tunnel in dbt, you'll need to provide the hostname/IP of your bastion server, username, and port, of your choosing, that dbt will connect to. Review the following steps: 1. Verify the bastion server has its network security rules set up to accept connections from the [dbt IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) on whatever port you configured. 2. Set up the user account by using the bastion servers instance's CLI, The following example uses the username `dbtcloud`: ```shell sudo groupadd dbtcloud sudo useradd -m -g dbtcloud dbtcloud sudo su - dbtcloud mkdir ~/.ssh chmod 700 ~/.ssh touch ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys ``` 3. Copy and paste the dbt generated public key, into the authorized\_keys file. The bastion server should now be ready for dbt to use as a tunnel into the Postgres environment. #### Configuration To grant users or roles database permissions (access rights and privileges), refer to the [Postgres permissions](https://docs.getdbt.com/reference/database-permissions/postgres-permissions.md) page. #### FAQs  Database Error - could not connect to server: Connection timed out When setting up a database connection using an SSH tunnel, you need the following components: * A load balancer (like ELB or NLB) to manage traffic. * A bastion host (or jump server) that runs the SSH protocol, acting as a secure entry point. * The database itself (such as a Postgres cluster). dbt uses an SSH tunnel to connect through the load balancer to the database. This connection is established at the start of any dbt job run. If the tunnel connection drops, the job fails. Tunnel failures usually happen because: * The SSH daemon times out if it's idle for too long. * The load balancer cuts off the connection if it's idle. * dbt tries to keep the connection alive by checking in every 30 seconds, and the system will end the connection if there's no response from the SSH service after 300 seconds. This helps avoid drops due to inactivity unless the Load Balancer's timeout is less than 30 seconds. Bastion hosts might have additional SSH settings to disconnect inactive clients after several checks without a response. By default, it checks three times. To prevent premature disconnections, you can adjust the settings on the bastion host: * `ClientAliveCountMax `— Configures the number of checks before deciding the client is inactive. For example, `ClientAliveCountMax 10` checks 10 times. * `ClientAliveInterval` — Configures when to check for client activity. For example, `ClientAliveInterval 30` checks every 30 seconds. The example adjustments ensure that inactive SSH clients are disconnected after about 300 seconds, reducing the chance of tunnel failures. --- ##### Connect Redshift dbt v2 compatible dbt platform dbt platform supports connecting to Redshift. #### Warehouse permissions for dbt v2 The Redshift database user that dbt v2 uses must be able to run dbt workloads and read catalog metadata used for introspection. ##### Required Redshift objects Before connecting, these objects must exist or be accessible: | Object | Purpose | | ------------------------------------------------------- | --------------------------------- | | **Cluster** (provisioned) or **workgroup** (serverless) | Compute resource | | **Database** | Target database | | **Schema** | Target schema within the database | | **User** | Database user for authentication | | **IAM role or profile** (optional) | For IAM-based authentication | ##### Core permissions The following permissions are required for fundamental dbt features: | Permission | Object | Purpose | | ---------- | --------------- | ------------------------------------- | | `USAGE` | Schema | Access the schema | | `CREATE` | Schema | Create tables and views in the schema | | `SELECT` | Tables or views | Read data | | `INSERT` | Tables | Insert data | | `UPDATE` | Tables | Update data | | `DELETE` | Tables | Delete data | | `DROP` | Tables or views | Drop or replace objects | | `TRUNCATE` | Tables | Truncate tables | ##### Metadata operations dbt v2 queries these Redshift system relations: | System relation | Purpose | Permission required | | ------------------ | ------------------------------------------------------ | ------------------------- | | `SVV_ALL_COLUMNS` | Column metadata | SELECT on the system view | | `pg_class` | List relations | Access to system catalog | | `pg_namespace` | Schema information | Access to system catalog | | `sys_query_detail` | Source freshness (last insert time) | SELECT on the system view | | `svv_table_info` | List materialized views when the project includes them | SELECT on the system view | | `svv_mv_info` | List materialized views when the project includes them | SELECT on the system view | ##### Schema management Conditional permissions for schema management | Permission | Object | When required | | --------------- | -------- | ------------------- | | `CREATE SCHEMA` | Database | Auto-create schemas | For example SQL grants in Redshift, refer to [Redshift permissions](https://docs.getdbt.com/reference/database-permissions/redshift-permissions.md). #### Connection fields Use the following fields when creating a connection: | Field | Required? | Description | Examples | | --------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | Server Hostname | Required | The hostname of the database to connect to. This can either be a hostname or an IP address. Refer to [Find your cluster](https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-connection-string.html) to find the hostname. | Redshift: `hostname.region.redshift.amazonaws.com` | | Port | Required | Usually 5439 (Redshift) | `5439` | | Database | Optional | The logical database to connect to and run queries against. | `analytics` | | Retries | Optional | Number of retries (on each statement). Defaults to 3 if not set. Max value is 10. | `4` | **Note**: When you set up a Redshift connection in dbt, SSL-related parameters aren't available as inputs. [![Configuring a Redshift connection](/img/docs/dbt-platform/platform-configuring-dbt-platform/postgres-redshift-connection.png?v=2 "Configuring a Redshift connection")](#)Configuring a Redshift connection ##### Authentication Parameters See the following supported authentication methods for Redshift: * Username and password * SSH tunneling * AWS IAM Identity Center via [external OAuth](https://docs.getdbt.com/docs/platform/manage-access/redshift-external-oauth.md) (Okta or Entra ID) for development connections * IAM User authentication via [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) On the dbt platform, the IAM user authentication is currently only supported via [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes). Once the project is created, development and deployment environments can be updated to use extended attributes to pass the fields described below, as some are not supported via textbox. You will need to create an IAM User, generate an [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey), and either: * on a cluster, a database user is expected in the `user` field. The IAM user is only leveraged for authentication, the database user for authorization * on Serverless, grant permission to the IAM user in Redshift. The `user` field is ignored (but still required) * For both, the `password` field will be ignored. | Profile field | Example | Description | | ------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `method` | IAM | use IAM to authenticate via IAM User authentication | | `cluster_id` | CLUSTER\_ID | Required for IAM authentication only for provisioned cluster, not for Serverless | | `user` | username | User querying the database, ignored for Serverless (but still required) | | `region` | us-east-1 | Region of your Redshift instance | | `access_key_id` | ACCESS\_KEY\_ID | IAM user [access key id](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) | | `secret_access_key` | SECRET\_ACCESS\_KEY | IAM user secret access key | <br /> ###### Example Extended Attributes for IAM User on Redshift Serverless To avoid pasting secrets in extended attributes, leverage [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md#handling-secrets): \~/.dbt/profiles.yml ```yaml host: my-production-instance.myregion.redshift-serverless.amazonaws.com method: iam region: us-east-2 access_key_id: '{{ env_var(''DBT_ENV_ACCESS_KEY_ID'') }}' secret_access_key: '{{ env_var(''DBT_ENV_SECRET_ACCESS_KEY'') }}' ``` Both `DBT_ENV_ACCESS_KEY_ID` and `DBT_ENV_SECRET_ACCESS_KEY` will need [to be assigned](https://docs.getdbt.com/docs/build/environment-variables.md) for every environment leveraging extended attributes as such. ##### Connecting using an SSH Tunnel Use an SSH tunnel when your Redshift instance is not publicly accessible and must be reached through a [bastion server](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md#about-the-bastion-server-in-aws). When enabled, dbt platform connects to your database by first establishing a secure connection to the bastion host, which then forwards traffic to your database. To configure a connection using an SSH tunnel: 1. Navigate to **Account settings** (by clicking on your account name in the left side menu) and select **Connections**. 2. Select an existing connection to edit it, or click **+ New connection**. 3. In **Connection settings**, ensure **SSH Tunnel Enabled** is checked. 4. Enter the hostname, username, and port for the bastion server. [![A public key is generated after saving](/img/docs/dbt-platform/platform-configuring-dbt-platform/postgres-redshift-ssh-tunnel.png?v=2 "A public key is generated after saving")](#)A public key is generated after saving 5. Click **Save**. dbt platform generates and displays a public key. 6. Copy the newly generated public key to the bastion server and add it to the server’s `authorized_keys` file to authorize dbt platform to connect through the bastion host. If the new key is not added, the SSH tunnel connection will fail. important Each time you create and save a new SSH tunnel connection, dbt platform generates a unique SSH key pair, even when the connection details are identical to an existing connection. ###### About the Bastion server in AWS What is a bastion server? A bastion server in [Amazon Web Services (AWS)](https://aws.amazon.com/blogs/security/how-to-record-ssh-sessions-established-through-a-bastion-host/) is a host that allows dbt to open an SSH connection. <br /> dbt only sends queries and doesn't transmit large data volumes. This means the bastion server can run on an AWS instance of any size, like a t2.small instance or t2.micro.<br /><br /> Make sure the location of the instance is the same Virtual Private Cloud (VPC) as the Redshift instance, and configure the security group for the bastion server to ensure that it's able to connect to the warehouse port. ###### Configuring the Bastion Server in AWS To configure the SSH tunnel in dbt, you'll need to provide the hostname/IP of your bastion server, username, and port, of your choosing, that dbt will connect to. Review the following steps: 1. Verify the bastion server has its network security rules set up to accept connections from the [dbt IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) on whatever port you configured. 2. Set up the user account by using the bastion servers instance's CLI, The following example uses the username `dbtcloud`: ```shell sudo groupadd dbtcloud sudo useradd -m -g dbtcloud dbtcloud sudo su - dbtcloud mkdir ~/.ssh chmod 700 ~/.ssh touch ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys ``` 3. Copy and paste the dbt generated public key, into the authorized\_keys file. The bastion server should now be ready for dbt to use as a tunnel into the Redshift environment. #### Configuration To optimize performance with data platform-specific configurations in dbt, refer to [Redshift-specific configuration](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md). To grant users or roles database permissions (access rights and privileges), refer to the [Redshift permissions](https://docs.getdbt.com/reference/database-permissions/redshift-permissions.md) page. #### FAQs  Database Error - could not connect to server: Connection timed out When setting up a database connection using an SSH tunnel, you need the following components: * A load balancer (like ELB or NLB) to manage traffic. * A bastion host (or jump server) that runs the SSH protocol, acting as a secure entry point. * The database itself (such as a Redshift cluster). dbt uses an SSH tunnel to connect through the load balancer to the database. This connection is established at the start of any dbt job run. If the tunnel connection drops, the job fails. Tunnel failures usually happen because: * The SSH daemon times out if it's idle for too long. * The load balancer cuts off the connection if it's idle. * dbt tries to keep the connection alive by checking in every 30 seconds, and the system will end the connection if there's no response from the SSH service after 300 seconds. This helps avoid drops due to inactivity unless the Load Balancer's timeout is less than 30 seconds. Bastion hosts might have additional SSH settings to disconnect inactive clients after several checks without a response. By default, it checks three times. To prevent premature disconnections, you can adjust the settings on the bastion host: * `ClientAliveCountMax `— Configures the number of checks before deciding the client is inactive. For example, `ClientAliveCountMax 10` checks 10 times. * `ClientAliveInterval` — Configures when to check for client activity. For example, `ClientAliveInterval 30` checks every 30 seconds. The example adjustments ensure that inactive SSH clients are disconnected after about 300 seconds, reducing the chance of tunnel failures. --- ##### Connect Salesforce Data 360 Beta dbt v2 compatible ### Connect Salesforce Data 360 [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2 compatible dbt platform dbt v2 in dbt platform supports connecting to Salesforce Data 360. #### Prerequisites Before you can connect dbt to Salesforce Data 360, you need the following: * A Data 360 instance. * The `server.key` private key file. For more information, refer to [Generating a private key and certificate](#generating-a-private-key-and-certificate). * An external client app configured for JSON Web Token (JWT) Bearer token flow. For more information, refer to [Setting up the external client app](#setting-up-the-external-client-app). * `Data Cloud Architect` and `Data Cloud User` permissions. ##### Generating a private key and certificate Before creating the external client app, generate a private key (`server.key`) and a self-signed certificate (`server.crt`) in Salesforce. Salesforce uses the certificate to verify the JWT Bearer token that dbt sends. Refer to [Create a Private Key and Self-Signed Digital Certificate](https://developer.salesforce.com/docs/atlas.en-us.252.0.sfdx_dev.meta/sfdx_dev/sfdx_dev_auth_key_and_cert.htm) for instructions. note It is recommended to generate these credentials under a **service user account** rather than an individual user account to simplify credential management. ##### Creating the external client app To authenticate dbt using JWT Bearer token flow, you must create and configure an external client app in Salesforce Data 360. Follow the steps in [Create a Local External Client App](https://help.salesforce.com/s/articleView?id=xcloud.create_a_local_external_client_app.htm\&type=5) to create an external client app. For Step 6, make sure to select **Packaged**. Then, configure the OAuth settings for the app: 1. Under **API (Enable OAuth Settings)**, select **Enable OAuth**. 2. Set the **Callback URL** to `https://login.salesforce.com/services/oauth2/callback`. 3. Under **OAuth Scopes**, add the following to the **Selected OAuth Scopes** list: * `api` - To manage user data via APIs * `refresh_token`, `offline_access` - To perform requests at any time, even when the user is offline or tokens have expired * `cdp_query_api` - To perform SQL queries on Data 360 data 4. Under **Flow Enablement**, select **Enable JWT Bearer Flow**. 5. Click **Upload Files** and upload the `server.crt` file created in [Generating a private key and certificate](#generating-a-private-key-and-certificate). 6. Click **Create**. 7. On the **Policies** tab, select **Edit**. 8. Under **OAuth Policies** > **Plugin Policies** > **Permitted Users**, select **Admin approved users are pre-authorized**. A list of **Select Profiles and Select Permission Sets** appears. 9. Selct the profiles and permission sets that should have access to the external client app. 10. Under **App Authorization**, set **IP Relaxation** to **Relax IP Restrictions**. 11. Click **Save**. #### Connection fields To connect to Salesforce Data 360, configure the connection settings and provide your credentials. ##### Connection Configure the following fields to set up your Salesforce Data 360 connection: | Field | Description | Example | | -------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------- | | Connection name | A name for your Salesforce Data 360 connection. | | | Auth method | The authentication method used to connect to Salesforce Data 360. JWT is the only supported method. | JWT Bearer Flow (default) | | Login URL | The Salesforce instance URL. | `https://login.salesforce.com` (default) | | Database | (Optional) The Salesforce Data 360 database to connect to. | | | Data Transform Run Timeout | (Optional) The timeout duration (in milliseconds) for data transformation runs. | | ##### Credentials Enter the following credentials to authenticate with Salesforce Data 360: | Field | Description | Example | | ----------- | --------------------------------------------------- | ----------------------- | | Username | Your Salesforce Data 360 username. | <user.name@example.com> | | Client ID | The `Consumer Key` of the Salesforce Data 360 app. | | | Private Key | The private key for JWT bearer flow authentication. | | * For **Username**, use your Salesforce account username. Authentication will only succeed if that username has one of the profiles or permission sets added to the external client app in [step 9](#creating-the-external-client-app). * For **Client ID**, get the Consumer Key from your external client app in Salesforce. Go to **Settings** > **OAuth Settings** > **Consumer Key and Secret**. Copy the **Consumer Key** and paste to the **Client ID** field. * The **Private Key** is the `server.key` file created in [Generating a private key and certificate](#generating-a-private-key-and-certificate). #### Authentication method Salesforce Data 360 supports JWT bearer authentication only. JWT bearer flow is a machine-to-machine authentication method that uses a private key file and Consumer Key (Client ID) to authenticate without requiring user interaction. #### Configuration To learn how to optimize performance with data platform-specific configurations in dbt platform, refer to [Salesforce Data 360 configurations](https://docs.getdbt.com/reference/resource-configs/data-cloud-configs.md). #### Limitations The following dbt features are not yet supported for Salesforce Data 360 connections in dbt platform: * [Seeds](https://docs.getdbt.com/docs/build/seeds.md) * As a workaround, log in to Salesforce and upload the CSV file manually. * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) * [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) --- ##### Connect Snowflake Generally available (GA) dbt v2 compatible ### Connect Snowflake [Generally available (GA)](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2 compatible dbt platform Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. dbt platform connections and credentials inherit the permissions of the accounts configured. You can customize roles and associated permissions in Snowflake to fit your company's requirements and fine-tune access to database objects in your account. Refer to [Snowflake permissions](https://docs.getdbt.com/reference/database-permissions/snowflake-permissions.md) for more information about customizing roles in Snowflake. To see which Snowflake functions are supported in dbt v2 in `strict` mode, refer to [Snowflake function support](https://docs.getdbt.com/reference/resource-configs/snowflake-function-support.md). #### Warehouse permissions for dbt v2 The Snowflake user or service account that dbt v2 connects as must be able to run dbt workloads (queries, metadata, and typical materializations). Grant privileges through a Snowflake role assigned to that user. ##### Required Snowflake objects Before connecting, these objects must exist: | Object | Purpose | | ------------- | -------------------------------------------------- | | **Account** | Your Snowflake account identifier | | **User** | Service account or user for dbt v2 | | **Role** | Role assigned to the user with required privileges | | **Warehouse** | Virtual warehouse for compute | | **Database** | Target database or databases for dbt models | | **Schema** | Target schema or schemas within the database | ##### Core operations The following are required permissions for fundamental dbt operations: | Permission | Object | Purpose | | -------------- | --------------- | ------------------------------------------ | | `USAGE` | Warehouse | Execute queries | | `USAGE` | Database | Access the database | | `USAGE` | Schema | Access schemas | | `SELECT` | Tables or views | Read existing data and sources | | `CREATE TABLE` | Schema | Create models materialized as tables | | `CREATE VIEW` | Schema | Create models materialized as views | | `INSERT` | Tables | Load data into tables | | `UPDATE` | Tables | Incremental model updates | | `DELETE` | Tables | Incremental models using delete and insert | | `TRUNCATE` | Tables | Full refresh of incremental models | | `DROP` | Tables or views | Replace existing objects | ##### Metadata operations The following are required permissions for dbt metadata operations: | Permission | Object | Purpose | | --------------------- | ------------------- | ------------------------------- | | `USAGE` | INFORMATION\_SCHEMA | Query table and column metadata | | `DESCRIBE TABLE` | Tables | Get schema information | | `SHOW OBJECTS` | Schema | List relations in schema | | `SHOW USER FUNCTIONS` | Schema | Discover UDFs (if used) | ##### Schema and database management The following are conditional permissions for schema and database management: | Permission | Object | When required | | ----------------- | -------- | -------------------------------- | | `CREATE SCHEMA` | Database | dbt should auto-create schemas | | `CREATE DATABASE` | Account | dbt should auto-create databases | ##### Advanced features The following are optional permissions for advanced features: | Permission | Object | Feature | | -------------------- | ------ | ----------------------------------- | | `CREATE STAGE` | Schema | File staging for seeds or snapshots | | `CREATE FILE FORMAT` | Schema | Custom file formats | | `CREATE SEQUENCE` | Schema | Sequences | | `EXECUTE TASK` | Schema | Snowflake tasks | For role examples in Snowflake, see [Snowflake permissions](https://docs.getdbt.com/reference/database-permissions/snowflake-permissions.md). #### Connection fields The following fields are required when creating a Snowflake connection: | Field | Description | Examples | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Account | The Snowflake account to connect to. Take a look [here](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md#account) to determine what the account field should look like based on your region. |   ✅ `db5261993` or `db5261993.east-us-2.azure`<br />  ❌ `db5261993.eu-central-1.snowflakecomputing.com` | | Role | A mandatory field indicating what role should be assumed after connecting to Snowflake | `transformer` | | Database | The logical database to connect to and run queries against. | `analytics` | | Warehouse | The virtual warehouse to use for running queries. | `transforming` | #### Authentication methods This section describes the different authentication methods for connecting dbt to Snowflake. Configure deployment environment (Production, Staging, General) credentials globally in the [**Connections**](https://docs.getdbt.com/docs/deploy/deploy-environments.md#deployment-connection) area of **Account settings**. Individual users configure their user credentials in the [**Credentials**](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) area of their user profile. Snowflake authentication in the dbt platform You cannot create new Snowflake credentials with username and password in dbt platform. New development and deployment credentials default to [key pair](#key-pair) authentication. For user credentials on Enterprise-tier plans, [Snowflake OAuth](#snowflake-oauth) is also available when configured on the connection. To update existing password credentials, refer to [Username and password with MFA](#username-and-password-with-mfa). ##### Key pair **Available in:** Development environments, Deployment environments If you are creating Snowflake credentials for the first time in dbt platform, key pair is the default authentication method. Use it for both development and deployment credentials. The `Keypair` auth method uses Snowflake's [Key Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth) to authenticate development or deployment credentials for a dbt project. 1. After [generating an encrypted key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), be sure to set the `rsa_public_key` for the Snowflake user to authenticate in dbt: ```sql alter user jsmith set rsa_public_key='MIIBIjANBgkqh...'; ``` 2. Finally, set the **Private Key** and **Private Key Passphrase** fields in the **Credentials** page to finish configuring dbt to authenticate with Snowflake using a key pair. * **Note:** Unencrypted private keys are permitted. Use a passphrase only if needed. dbt can specify a `private_key` directly as a string instead of a `private_key_path`. This `private_key` string can be in either Base64-encoded DER format, representing the key bytes, or in plain-text PEM format. Refer to [Snowflake documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth) for more info on how they generate the key. * Specifying a private key using an [environment variable](https://docs.getdbt.com/docs/build/environment-variables.md) (for example, `{{ env_var('DBT_PRIVATE_KEY') }}`) is not supported. 3. To successfully fill in the Private Key field, you *must* include commented lines. If you receive a `Could not deserialize key data` or `JWT token` error, refer to [Troubleshooting](#troubleshooting) for more info. **Example:** ```sql -----BEGIN ENCRYPTED PRIVATE KEY----- < encrypted private key contents here - line 1 > < encrypted private key contents here - line 2 > < ... > -----END ENCRYPTED PRIVATE KEY----- ``` [![Snowflake keypair authentication](/img/docs/dbt-platform/snowflake-keypair-auth.png?v=2 "Snowflake keypair authentication")](#)Snowflake keypair authentication ###### dbt v2 key pair We recommend using PKCS#8 format with AES-256 encryption for key pair authentication with dbt v2. dbt v2 doesn't support legacy 3DES encryption or headerless key formats. Using older key formats may cause authentication failures. If you encounter the `Key is PKCS#1 (RSA private key). Snowflake requires PKCS#8` error, then your private key is in the wrong format. You have two options: * (Recommended fix) Re-export your key with modern encryption: ```bash # Convert to PKCS#8 with AES-256 encryption openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 aes-256-cbc -inform PEM -out rsa_key.p8 ``` * (Temporary workaround) Add the `BEGIN` header and `END` footer to your PEM body: ```text -----BEGIN ENCRYPTED PRIVATE KEY----- < Your existing encrypted private key contents > -----END ENCRYPTED PRIVATE KEY----- ``` ##### Snowflake OAuth **Available in:** Development environments, Enterprise-tier plans only The OAuth auth method permits dbt to run development queries on behalf of a Snowflake user without the configuration of Snowflake password in dbt. For more information on configuring a Snowflake OAuth connection in dbt, please see [the docs on setting up Snowflake OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md). [![Configuring Snowflake OAuth connection](/img/docs/dbt-platform/dbt-platform-enterprise/database-connection-snowflake-oauth.png?v=2 "Configuring Snowflake OAuth connection")](#)Configuring Snowflake OAuth connection Migrate from username and password If your credentials still use username and password, you can view the existing configuration in **Credentials**, connection profiles, or deployment credential forms, but you cannot save changes until you switch **Auth method** to key pair or OAuth. A warning banner may also appear at the top of your account when password credentials are still in use. To migrate off of username and password, follow the steps in [Key pair](#key-pair) or [Snowflake OAuth](#snowflake-oauth). ##### Username and password with MFA **Available in:** Existing development and deployment credentials only If you are still on username and password while you plan your migration, [MFA](https://docs.snowflake.com/en/user-guide/security-mfa) is required by Snowflake for all password logins. Snowflake's MFA support is powered by the Duo Security service. * In dbt, set the following [extended attribute](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) in the development environment **General settings** page, under the **Extended attributes** section: ```yaml authenticator: username_password_mfa ``` * To reduce the number of user prompts when connecting to Snowflake with MFA, [enable token caching](https://docs.snowflake.com/en/user-guide/security-mfa#using-mfa-token-caching-to-minimize-the-number-of-prompts-during-authentication-optional) in Snowflake. * Optionally, if users miss prompts and their Snowflake accounts get locked, you can prevent automatic retries by adding the following in the same **Extended attributes** section: ```yaml connect_retries: 0 ``` [![Configure the MFA username and password, and connect\_retries in the development environment settings.](/img/docs/dbt-platform/platform-configuring-dbt-platform/extended-attributes-mfa.png?v=2 "Configure the MFA username and password, and connect_retries in the development environment settings.")](#)Configure the MFA username and password, and connect\_retries in the development environment settings. #### Configuration To learn how to optimize performance with data platform-specific configurations in dbt, refer to [Snowflake-specific configuration](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md). ##### Custom domain URL To connect to Snowflake through a custom domain (vanity URL) instead of the account locator, use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) to configure the `host` parameter with the custom domain: ```yaml host: https://custom_domain_to_snowflake.com ``` This configuration may conflict with Snowflake OAuth when used with PrivateLink. IF users can't reach Snowflake authentication servers from a networking standpoint, please [contact dbt Support](mailto:support@getdbt.com) to find a workaround with this architecture. #### Troubleshooting If you're receiving a `Could not deserialize key data` or `JWT token` error, refer to the following causes and solutions:  Error: \`Could not deserialize key data\` Possible cause and solution for the error "Could not deserialize key data" in dbt. * This could be because of mistakes like not copying correctly, missing dashes, or leaving out commented lines. **Solution**: * You can copy the key from its source and paste it into a text editor to verify it before using it in dbt.  Error: \`JWT token\` Possible cause and solution for the error "JWT token" in dbt. * This could be a transient issue between Snowflake and dbt. When connecting to Snowflake, dbt gets a JWT token valid for only 60 seconds. If there's no response from Snowflake within this time, you might see a `JWT token is invalid` error in dbt. * The public key was not entered correctly in Snowflake. **Solutions** * dbt needs to retry connections to Snowflake. * Confirm and enter Snowflake's public key correctly. Additionally, you can reach out to Snowflake for help or refer to this Snowflake doc for more info: [Key-Based Authentication Failed with JWT token is invalid Error](https://community.snowflake.com/s/article/Key-Based-Authentication-Failed-with-JWT-token-is-invalid-Error). --- ##### Connect Starburst or Trino dbt platform The following are the required fields for setting up a connection with a [Starburst Enterprise](https://docs.starburst.io/starburst-enterprise/index.html), [Starburst Galaxy](https://docs.starburst.io/starburst-galaxy/index.html), or [Trino](https://trino.io/) cluster. Unless specified, "cluster" means any of these products' clusters. | Field | Description | Examples | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Host** | The hostname of your cluster. Don't include the HTTP protocol prefix. | `mycluster.mydomain.com` | | **Port** | The port to connect to your cluster. By default, it's 443 for TLS enabled clusters. | `443` | | **User** | The username (of the account) to log in to your cluster. When connecting to Starburst Galaxy clusters, you must include the role of the user as a suffix to the username.<br /><br /> | Format for Starburst Enterprise or Trino depends on your configured authentication method.<br />Format for Starburst Galaxy:<br />- `user.name@mydomain.com/role` | | **Password** | The user's password. | - | | **Database** | The name of a catalog in your cluster. | `example_catalog` | | **Schema** | The name of a schema that exists within the specified catalog.  | `example_schema` | #### Roles in Starburst Enterprise If connecting to a Starburst Enterprise cluster with built-in access controls enabled, you must specify a role using the format detailed in [Additional parameters](#additional-parameters). If a role is not specified, the default role for the provided username is used. #### Catalogs and schemas When selecting the catalog and the schema, make sure the user has read and write access to both. This selection does not limit your ability to query the catalog. Instead, they serve as the default location for where tables and views are materialized. In addition, the Trino connector used in the catalog must support creating tables. This *default* can be changed later from within your dbt project. #### Configuration To learn how to optimize performance with data platform-specific configurations in dbt, refer to [Starburst/Trino-specific configuration](https://docs.getdbt.com/reference/resource-configs/trino-configs.md). --- ##### Connect Teradata dbt platform Your environment(s) must be on a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to use the Teradata connection. | Field | Description | Type | Required? | Example | | --------------- | ------------------------------------------------------------------------------------------------ | -------------- | --------- | ------------------------------------- | | Host | Host name of your Teradata environment. | String | Required | host-name.env.clearscape.teradata.com | | Port | The database port number. Equivalent to the Teradata JDBC Driver DBS\_PORT connection parameter. | Quoted integer | Optional | 1025 | | Retries | Number of times to retry to connect to database upon error. | Integer | optional | 10 | | Request timeout | The waiting period between connections attempts in seconds. Default is "1" second. | Quoted integer | Optional | 3 | [![Example of the Teradata connection fields.](/img/docs/dbt-platform/teradata-connection.png?v=2 "Example of the Teradata connection fields.")](#)Example of the Teradata connection fields. ##### Development and deployment credentials | Field | Description | Type | Required? | Example | | -------- | -------------------------------------------------------------------------------------------- | ------ | --------- | ------------------- | | Username | The database username. Equivalent to the Teradata JDBC Driver USER connection parameter. | String | Required | database\_username | | Password | The database password. Equivalent to the Teradata JDBC Driver PASSWORD connection parameter. | String | Required | DatabasePassword123 | | Schema | Specifies the initial database to use after login, rather than the user's default database. | String | Required | dbtlabsdocstest | [![Example of the user credential fields.](/img/docs/dbt-platform/teradata-deployment.png?v=2 "Example of the user credential fields.")](#)Example of the user credential fields. --- #### Git ##### About git dbt platform A [version control](https://en.wikipedia.org/wiki/Version_control) system allows you and your teammates to work collaboratively, safely, and simultaneously on a single project. Version control helps you track all the code changes made in your dbt project. In a distributed version control system, every developer has a full copy of the project and project history. Git is one of the most popular distributed version control systems and is commonly used for both open source and commercial software development, with great benefits for individuals, teams and businesses. [![Git overview](/img/docs/dbt-platform/platform-ide/git-overview.png?v=2 "Git overview")](#)Git overview Git allows developers see the entire timeline of their changes, decisions, and progression of any project in one place. From the moment they access the history of a project, the developer has all the context they need to understand it and start contributing. When you develop in the command line interface (CLI) or Cloud integrated development environment (Studio IDE), you can leverage Git directly to version control your code. To use version control, make sure you are connected to a Git repository in the CLI or Cloud Studio IDE. #### Related docs * [Version control basics](https://docs.getdbt.com/docs/platform/git/version-control-basics.md) * [Merge conflicts](https://docs.getdbt.com/docs/platform/git/merge-conflicts.md) * [Connect to GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md) * [Connect to GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) * [Connect to Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) --- ##### Authenticate with Azure DevOps dbt platform | Enterprise, Enterprise+ If you use the Studio IDE or dbt CLI to collaborate on your team's Azure DevOps dbt repo, you need to [link your dbt profile to Azure DevOps](#link-your-dbt-profile-to-azure-devops), which provides an extra layer of authentication. #### Link your dbt profile to Azure DevOps Connect your dbt profile to Azure DevOps using OAuth: 1. Click your account name at the bottom of the left-side menu and click **Account settings** 2. Scroll down to **Your profile** and select **Personal profile**. 3. Go to the **Linked accounts** section in the middle of the page. [![Azure DevOps Authorization Screen](/img/docs/dbt-platform/connecting-azure-devops/LinktoAzure.png?v=2 "Azure DevOps Authorization Screen")](#)Azure DevOps Authorization Screen 4. Once you're redirected to Azure DevOps, sign into your account. 5. When you see the permission request screen from Azure DevOps App, click **Accept**. [![Azure DevOps Authorization Screen](</img/docs/dbt-platform/connecting-azure-devops/OAuth Acceptance.png?v=2> "Azure DevOps Authorization Screen")](#)Azure DevOps Authorization Screen You will be directed back to dbt, and your profile should be linked. You are now ready to develop in dbt! #### Troubleshooting Use these troubleshooting steps for common issues when authenticating with Azure DevOps.  Missing Azure user error when starting Studio If the Studio IDE fails to start and displays the following error, your personal Azure DevOps account isn't linked to dbt: `Missing Azure user; link your Azure DevOps account in your personal profile` To resolve it, [link your dbt profile to Azure DevOps](#link-your-dbt-profile-to-azure-devops).  Azure Service User not found error when starting Studio If the Studio IDE fails to start and displays the following error, the account-level service user isn't connected: `Failed to start server. The project is not fully set up: Azure Service User not found` This applies to accounts that use the legacy service user authentication. An account admin needs to reconnect the service user, which is also required when an account has had no scheduled, API, or CI runs for over 90 days. Refer to [Connect a service user](https://docs.getdbt.com/docs/platform/git/setup-service-user.md#connect-a-service-user) for more details. #### FAQs How can I fix my .gitignore file? A `.gitignore` file specifies which files git should intentionally ignore or 'untrack'. dbt indicates untracked files in the project file explorer pane by putting the file or folder name in *italics*. If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the Studio IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt to work correctly. The following sections describe how to fix the `.gitignore` file in:  Fix in the Studio IDE To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. 1. Launch the Studio IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. 3. Open the new or existing `gitignore` file, and add the following: ```bash # ✅ Correct target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` * **Note** — You can place these lines anywhere in the file, as long as they're on separate lines. The lines shown are wildcards that will include all nested files and folders. Avoid adding a trailing `'*'` to the lines, such as `target/*`. For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com/docs/gitignore). 4. Save the changes but *don't commit*. 5. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right corner of the IDE screen and select **Restart IDE**. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 6. Once the Studio IDE restarts, go to the **File Catalog** to delete the following files or folders (if they exist). No data will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. **Save** and then **Commit and sync** the changes. 8. Restart the Studio IDE again using the same procedure as step 5. 9. Once the Studio IDE restarts, use the **Create a pull request** (PR) button under the **Version Control** menu to start the process of integrating the changes. 10. When the git provider's website opens to a page with the new PR, follow the necessary steps to complete and merge the PR into the main branch of that repository. * **Note** — The 'main' branch might also be called 'master', 'dev', 'qa', 'prod', or something else depending on the organizational naming conventions. The goal is to merge these changes into the root branch that all other development branches are created from. 11. Return to the Studio IDE and use the **Change Branch** button, to switch to the main branch of the project. 12. Once the branch has changed, click the **Pull from remote** button to pull in all the changes. 13. Verify the changes by making sure the files/folders in the `.gitignore` file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).  Fix in the Git provider Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. There are two options for this approach: editing the main branch directly if allowed, or creating a pull request to implement the changes if required: ##### Edit in main branch When permissions allow it, it's possible to edit the `.gitignore` directly on the main branch of your repo. Here are the following steps: 1. Go to your repository's web interface. 2. Switch to the main branch and the root directory of your dbt project. 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deletions to the main branch. 8. Switch to the Studio IDE , and open the project that you're fixing. 9. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 10. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 11. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 12. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! ##### Unable to edit main branch If you can't edit the `.gitignore` directly on the main branch of your repo, follow these steps: 1. Go to your repository's web interface. 2. Switch to an existing development branch, or create a new branch just for these changes (This is often faster and cleaner). 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deleted folders. 8. Open a merge request using the git provider web interface. The merge request should attempt to merge the changes into the 'main' branch that all development branches are created from. 9. Follow the necessary procedures to get the branch approved and merged into the 'main' branch. You can delete the branch after the merge is complete. 10. Once the merge is complete, go back to the Studio IDE, and open the project that you're fixing. 11. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the Studio IDE by clicking on the three dots next to the **Studio IDE Status** button on the lower right corner of the Studio IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 12. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 13. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 14. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. How to migrate git providers To migrate from one git provider to another, refer to the following steps to avoid minimal disruption: 1. Outside of dbt, you'll need to import your existing repository into your new provider. By default, connecting your repository in one account won't automatically disconnected it from another account. As an example, if you're migrating from GitHub to Azure DevOps, you'll need to import your existing repository (GitHub) into your new Git provider (Azure DevOps). For detailed steps on how to do this, refer to your Git provider's documentation (Such as [GitHub](https://docs.github.com/en/migrations/importing-source-code/using-github-importer/importing-a-repository-with-github-importer), [GitLab](https://docs.gitlab.com/ee/user/project/import/repo_by_url.html), [Azure DevOps](https://learn.microsoft.com/en-us/azure/devops/repos/git/import-git-repository?view=azure-devops)) 2. Go back to dbt and set up your [integration for the new Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md), if needed. 3. Disconnect the old repository in dbt by going to **Account Settings** and then **Projects**. 4. Click on the **Repository** link, then click **Edit** and **Disconnect**. [![Disconnect and reconnect your Git repository in your dbt Account settings page.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your Git repository in your dbt Account settings page.")](#)Disconnect and reconnect your Git repository in your dbt Account settings page. 5. Click **Confirm Disconnect**. 6. On the same page, connect to the new Git provider repository by clicking **Configure Repository** * If you're using the native integration, you may need to OAuth to it. 7. That's it, you should now be connected to the new Git provider! 🎉 Note — As a tip, we recommend you refresh your page and Studio IDE before performing any actions. --- ##### Configure Git in dbt dbt platform [Version control](https://docs.getdbt.com/docs/platform/git/version-control-basics.md) — a system that allows you and your teammates to work safely and simultaneously on a single project — is an essential part of the dbt workflow. It enables teams to collaborate effectively and maintain a history of changes to their dbt projects. In dbt, you can configure Git integrations to manage your dbt project code with ease. dbt offers multiple ways to integrate with you Git provider, catering to diverse team needs and preferences. Whether you use a Git integration that natively connects with dbt or prefer to work with a managed or cloned repository, dbt supports flexible options to streamline your workflow. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/git/managed-repository.md) ###### [Managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) [Learn how to quickly set up a project with a managed repository.](https://docs.getdbt.com/docs/platform/git/managed-repository.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) ###### [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) [Learn how to connect to a git repository using a git URL and deploy keys.](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/git/connect-github.md) ###### [Connect to GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md) [Learn how to connect to GitHub using dbt's native integration.](https://docs.getdbt.com/docs/platform/git/connect-github.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) ###### [Connect to GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) [Learn how to connect to GitLab using dbt's native integration.](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) ###### [Connect to Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) [Learn how to connect to Azure DevOps using dbt's native integration.](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) <br /> <br /> [Available on dbt Enterprise or Enterprise+ plans.](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-integration.md#git-providers-who-support-ci) ###### [Availability of CI features by Git provider](https://docs.getdbt.com/docs/deploy/continuous-integration.md#git-providers-who-support-ci) [Learn which Git providers have native support for Continuous Integration workflows](https://docs.getdbt.com/docs/deploy/continuous-integration.md#git-providers-who-support-ci) --- ##### Connect to Azure DevOps dbt platform | Enterprise, Enterprise+ Available for dbt Enterprise and Enterprise+ Connecting an Azure DevOps cloud account is available for organizations using the dbt Enterprise or Enterprise+ plans. dbt's native Azure DevOps integration does not support Azure DevOps Server (on-premise). Instead, you can [import a project by git URL](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to connect to an Azure DevOps Server. #### About Azure DevOps and dbt Connect your Azure DevOps cloud account in dbt to unlock new product experiences: * Import new Azure DevOps repos with a couple clicks during dbt project setup. * Clone repos using HTTPS rather than SSH * Enforce user authorization with OAuth 2.0. * Carry Azure DevOps user repository permissions (read / write access) through to Studio IDE or dbt CLI's git actions. * Trigger Continuous integration (CI) builds when pull requests are opened in Azure DevOps. Currently, there are multiple methods for integrating Azure DevOps with dbt. The following methods are available to all accounts: * [**Service principal (recommended)**](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md) * [**Service user (legacy)**](https://docs.getdbt.com/docs/platform/git/setup-service-user.md) * [**Service user to service principal migration**](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md#migrate-to-service-principal) No matter which approach you take, you will need admins for dbt, Azure Entra ID, and Azure DevOps to complete the integration. For more information, follow the setup guide that's right for you. #### Troubleshooting  CI jobs don't trigger on pull requests If continuous integration (CI) jobs aren't triggered when you open pull requests in Azure DevOps, and you don't see an error in dbt or Azure DevOps, try these checks: * Confirm the Azure DevOps service hooks for the repository still exist and point to your dbt account. * Confirm your service principal or service user still has access to the repository. If those look fine, disconnect and reconnect the repository: 1. From **Account settings**, select **Projects**, then select the project. 2. Select the **Repository** link, then **Edit**, then **Disconnect**. 3. Select **Confirm Disconnect**. 4. Select **Configure Repository** and connect the same Azure DevOps repository. --- ##### Connect to GitHub dbt platform Connecting your GitHub account to dbt provides convenience and another layer of security to dbt: * Import new GitHub repositories with a couple clicks during dbt project setup. * Clone repos using HTTPS rather than SSH. * Trigger [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md)(CI) builds when pull requests are opened in GitHub. GitHub Enterprise Cloud (`ghe.com`) domains If your organization uses GitHub Enterprise Cloud, hosted on a `ghe.com` domain, the shared dbt GitHub App can't reach it. Note that `ghe.com` accounts are cloud-managed (not on-premises), but this limitation still applies. Enterprise and Enterprise+ accounts can connect by registering a [custom GitHub application](#custom-github-application) instead. On other plans, use [importing a project by git URL](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) with SSH/deploy keys. Your organization's SSH URL configuration may require additional steps, and some native integration features are unavailable with the git URL method. For additional help with your specific setup, contact [dbt Support](mailto:support@getdbt.com) or your dbt account team. #### Prerequisites * **GitHub permissions**: You must be a GitHub organization owner to [install the application](https://docs.getdbt.com/docs/platform/git/connect-github.md#installing-dbt-in-your-github-account). Learn more about [GitHub organization](https://docs.github.com/en/organizations/managing-peoples-access-to-your-organization-with-roles/roles-in-an-organization) roles. * **dbt permissions**: Your GitHub organization owner also needs [*Owner*](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md) or [*Account Admin*](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permissions in dbt platform. If needed, create a temporary account with these [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for the installation. * **On-premises GitHub or EU-hosted GitHub**: On Enterprise and Enterprise+ plans, you can use the native integration by registering a [custom GitHub application](#custom-github-application). On other plans, [connect using a Git URL](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md), which supports [fewer](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#limited-integration) Git features. Case-sensitive repository names When specifying a GitHub repository in the dbt platform using the UI, API, or Terraform provider, the repository name must exactly match the case used in the GitHub URL to avoid cloning errors or job failures. For example, if the URL of your repository is `github.com/my-org/MyRepo`, enter the name as `MyRepo`, not `myrepo`. #### Choose your GitHub application dbt connects to GitHub through a GitHub App. You have two options: * [Shared dbt application (default)](#installing-dbt-in-your-github-account), available to any org connecting to `github.com`. * Your own [custom GitHub application](#custom-github-application), available in private beta to Enterprise and Enterprise+ accounts on MT or ST using GitHub Enterprise Server or GitHub with EU data residency. Reach out to your account manager to join the beta. [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") You can use one GitHub application per dbt account, not one per project. #### Installing dbt in your GitHub account You can connect your dbt account to GitHub by installing the dbt application in your GitHub organization and providing access to the right repos: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Select **Personal profile** under the **Your profile** section. 3. Scroll down to **Linked accounts**. [![Navigated to Linked Accounts under your profile](/img/docs/dbt-platform/platform-configuring-dbt-platform/connecting-github/github-connect-1.png?v=2 "Navigated to Linked Accounts under your profile")](#)Navigated to Linked Accounts under your profile 4. In the **Linked accounts** section, set up your GitHub account connection to dbt by clicking **Link** to the right of GitHub. This redirects you to your account on GitHub where you will be asked to install and configure the dbt application. 5. Select the GitHub organization and repos dbt should access. [![Installing the dbt application into a GitHub organization](/img/docs/dbt-platform/platform-configuring-dbt-platform/connecting-github/github-app-install.png?v=2 "Installing the dbt application into a GitHub organization")](#)Installing the dbt application into a GitHub organization 6. Assign the dbt GitHub App the following permissions: * Read access to metadata * Read and write access to Checks * Read and write access to Commit statuses * Read and write access to Contents (Code) * Read and write access to Pull requests * Read and write access to Webhooks * Read and write access to Workflows 7. Once you grant access to the app, you will be redirected back to dbt and shown a linked account success state. You are now personally authenticated. 8. Ask your team members to individually authenticate by connecting their [personal GitHub profiles](#authenticate-your-personal-github-account). #### Limiting repository access in GitHub If you are your GitHub organization owner, you can also configure the dbt GitHub application to have access to only select repositories. This configuration must be done in GitHub, but we provide an easy link in dbt to start this process. [![Configuring the dbt app](/img/docs/dbt-platform/platform-configuring-dbt-platform/connecting-github/configure-github.png?v=2 "Configuring the dbt app")](#)Configuring the dbt app #### Custom GitHub application [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Available in private beta Configuring a custom GitHub app is available in private beta. To join, reach out to your account manager. The shared dbt GitHub App is hosted per [region](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md), so it can only reach repositories on `github.com`. If your organization runs [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.14/admin/overview/about-github-enterprise-server) or [GitHub with EU data residency](https://github.com/enterprise/data-residency), register your own GitHub application and point dbt at it. An account admin (or security admin) can set this up once: 1. [Register a GitHub application](#register-a-github-application) in your GitHub organization 2. [Add your GitHub application to dbt](#add-your-github-application-to-dbt) Then each developer on the account needs to [relink their GitHub profile](#relink-your-github-profile). ##### Register a GitHub application Repo availability dbt platform doesn’t support concurrent integrations with public and private GitHub repositories. To register a GitHub App, you need a private repository. Disconnect your configured public repository before you follow the steps in this section. We recommend creating the app in your GitHub organization instead of a personal account to ensure the connection doesn't break should a person leave. For the full walkthrough on the GitHub side, check out GitHub's guide to [registering a GitHub App](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app). 1. In dbt, go to **Account settings** > **Integrations** > **Git** and expand the **GitHub** section. Select **Copy** next to **Redirect URL** — you need it in the next step. 2. In your GitHub organization, go to **Settings** > **Developer settings** > **GitHub Apps** and select **New GitHub App**. 3. Enter the following: | Field | Value | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **GitHub App name** | Something recognizable, such as `dbt platform` | | **Homepage URL** | Your dbt [access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) | | **Redirect URL** | The **Redirect URL** you copied from dbt | | **Webhook** | Leave **Active** selected and enter any placeholder URL (such as <https://example.com>) in the **Webhook URL** field. dbt doesn't use it but GitHub requires it | 4. Grant the app these repository permissions: * Read access to metadata * Read and write access to Checks * Read and write access to Commit statuses * Read and write access to Contents (Code) * Read and write access to Pull requests * Read and write access to Webhooks * Read and write access to Workflows 5. Select **Create GitHub App**. GitHub generates an **App ID** and a **Client ID**. 6. Generate a **Client secret** and copy it somewhere safe as GitHub only shows it once. 7. Under **Private keys**, select **Generate a private key**. GitHub downloads a `.pem` file. You'll then paste its contents into dbt. 8. Install the app into your organization and note its **install URL**. Refer to GitHub's docs on [sharing your app via an install link](https://docs.github.com/en/apps/sharing-github-apps/sharing-your-github-app#sharing-your-github-app-via-an-install-link). If you're a Business Critical customer using [IP restrictions](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md), make sure your GitHub instance's CIDRs are allowed, or the connection will fail. ##### Add your GitHub application to dbt Existing GitHub connections break If your account already uses the shared dbt GitHub App, switching to your own application invalidates every existing GitHub profile link. Everyone on the account has to [relink their GitHub profile](#relink-your-github-profile) before they can develop or run CI. Plan the switch with your team. 1. In dbt, go back to **Account settings** > **Integrations** > **Git** and expand the **GitHub** section. 2. Enter the following: | Field | Value | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **GitHub base URL** | Your GitHub hostname. Use `https://github.com` for GitHub.com, or your own hostname for GitHub Enterprise Server, such as `https://github.yourgreatcompany.com` | | **App ID** | The **App ID** from your GitHub app | | **Client ID** | The **Client ID** from your GitHub app | | **Client secret** | The client secret you generated | | **Private key** | The full contents of the `.pem` file you downloaded, including the `BEGIN` and `END` lines | | **Install URL** | Optional. Your app's install link | [![Adding your own GitHub application under Account settings, Integrations, Git](/img/docs/dbt-platform/platform-configuring-dbt-platform/connecting-github/github-application-platform.png?v=2 "Adding your own GitHub application under Account settings, Integrations, Git")](#)Adding your own GitHub application under Account settings, Integrations, Git 3. Select **Save**. dbt validates the credentials against GitHub. If something isn't right, dbt shows an error and doesn't save the configuration. dbt now routes all GitHub authorization for this account through your application. ##### Relink your GitHub profile Every developer authenticates against the account's GitHub application, so repo access matches what your GitHub admin actually granted them. After an admin adds or changes the application, relink your profile: 1. From dbt, click your account name in the left side menu and select **Account settings**. 2. Select **Personal profile** under the **Your profile** section. 3. Scroll to **Linked accounts** and select **Unlink** next to GitHub if an old connection is still listed. 4. Select **Link** and authorize your organization's GitHub application. For the full walkthrough, refer to [Authenticate your personal GitHub account](#authenticate-your-personal-github-account). ##### Remove a GitHub integration Account admins and security admins can delete a GitHub integration from **Account settings** > **Integrations** > **Git**. Removing an integration disconnects everyone Deleting the integration breaks repo access and CI for the *whole* account until you configure a new one. You can't stage a replacement first, because an account supports only one GitHub integration at a time. ##### Limitations Note that the following limitations apply: * One GitHub application per dbt account. You can't set a different application per project. * You can't configure multiple GitHub integrations on the same account. * Only account admins and security admins can create, update, or delete a GitHub integration. #### Authenticate your personal GitHub account After the dbt administrator [sets up a connection](https://docs.getdbt.com/docs/platform/git/connect-github.md#installing-dbt-in-your-github-account) to your organization's GitHub account, you need to authenticate using your personal account. You must connect your personal GitHub profile to dbt to use the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) and [CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) and verify your read and write access to the repository. GitHub profile connection * dbt developers on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/) must each connect their GitHub profiles to dbt. This is because the Studio IDE verifies every developer's read / write access for the dbt repo. * dbt developers on the [Starter plan](https://www.getdbt.com/pricing/) don't need to each connect their profiles to GitHub, however, it's still recommended to do so. If your account uses a [custom GitHub application](#custom-github-application), you authorize that application instead of the shared dbt app, and your repository access mirrors your GitHub App installation. To connect a personal GitHub account: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Select **Personal profile** under the **Your profile** section. 3. Scroll down to **Linked accounts**. If your GitHub account is not connected, you’ll see "No connected account". 4. Select **Link** to begin the setup process. You’ll be redirected to GitHub, and asked to authorize dbt in a grant screen. [![Authorizing the dbt app for developers](/img/docs/dbt-platform/platform-configuring-dbt-platform/connecting-github/github-auth.png?v=2 "Authorizing the dbt app for developers")](#)Authorizing the dbt app for developers 5. Once you approve authorization, you will be redirected to dbt, and you should now see your connected account. You can now use the Studio IDE or dbt CLI. #### FAQs How can I fix my .gitignore file? A `.gitignore` file specifies which files git should intentionally ignore or 'untrack'. dbt indicates untracked files in the project file explorer pane by putting the file or folder name in *italics*. If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the Studio IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt to work correctly. The following sections describe how to fix the `.gitignore` file in:  Fix in the Studio IDE To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. 1. Launch the Studio IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. 3. Open the new or existing `gitignore` file, and add the following: ```bash # ✅ Correct target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` * **Note** — You can place these lines anywhere in the file, as long as they're on separate lines. The lines shown are wildcards that will include all nested files and folders. Avoid adding a trailing `'*'` to the lines, such as `target/*`. For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com/docs/gitignore). 4. Save the changes but *don't commit*. 5. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right corner of the IDE screen and select **Restart IDE**. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 6. Once the Studio IDE restarts, go to the **File Catalog** to delete the following files or folders (if they exist). No data will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. **Save** and then **Commit and sync** the changes. 8. Restart the Studio IDE again using the same procedure as step 5. 9. Once the Studio IDE restarts, use the **Create a pull request** (PR) button under the **Version Control** menu to start the process of integrating the changes. 10. When the git provider's website opens to a page with the new PR, follow the necessary steps to complete and merge the PR into the main branch of that repository. * **Note** — The 'main' branch might also be called 'master', 'dev', 'qa', 'prod', or something else depending on the organizational naming conventions. The goal is to merge these changes into the root branch that all other development branches are created from. 11. Return to the Studio IDE and use the **Change Branch** button, to switch to the main branch of the project. 12. Once the branch has changed, click the **Pull from remote** button to pull in all the changes. 13. Verify the changes by making sure the files/folders in the `.gitignore` file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).  Fix in the Git provider Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. There are two options for this approach: editing the main branch directly if allowed, or creating a pull request to implement the changes if required: ##### Edit in main branch When permissions allow it, it's possible to edit the `.gitignore` directly on the main branch of your repo. Here are the following steps: 1. Go to your repository's web interface. 2. Switch to the main branch and the root directory of your dbt project. 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deletions to the main branch. 8. Switch to the Studio IDE , and open the project that you're fixing. 9. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 10. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 11. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 12. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! ##### Unable to edit main branch If you can't edit the `.gitignore` directly on the main branch of your repo, follow these steps: 1. Go to your repository's web interface. 2. Switch to an existing development branch, or create a new branch just for these changes (This is often faster and cleaner). 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deleted folders. 8. Open a merge request using the git provider web interface. The merge request should attempt to merge the changes into the 'main' branch that all development branches are created from. 9. Follow the necessary procedures to get the branch approved and merged into the 'main' branch. You can delete the branch after the merge is complete. 10. Once the merge is complete, go back to the Studio IDE, and open the project that you're fixing. 11. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the Studio IDE by clicking on the three dots next to the **Studio IDE Status** button on the lower right corner of the Studio IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 12. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 13. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 14. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. How to migrate git providers To migrate from one git provider to another, refer to the following steps to avoid minimal disruption: 1. Outside of dbt, you'll need to import your existing repository into your new provider. By default, connecting your repository in one account won't automatically disconnected it from another account. As an example, if you're migrating from GitHub to Azure DevOps, you'll need to import your existing repository (GitHub) into your new Git provider (Azure DevOps). For detailed steps on how to do this, refer to your Git provider's documentation (Such as [GitHub](https://docs.github.com/en/migrations/importing-source-code/using-github-importer/importing-a-repository-with-github-importer), [GitLab](https://docs.gitlab.com/ee/user/project/import/repo_by_url.html), [Azure DevOps](https://learn.microsoft.com/en-us/azure/devops/repos/git/import-git-repository?view=azure-devops)) 2. Go back to dbt and set up your [integration for the new Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md), if needed. 3. Disconnect the old repository in dbt by going to **Account Settings** and then **Projects**. 4. Click on the **Repository** link, then click **Edit** and **Disconnect**. [![Disconnect and reconnect your Git repository in your dbt Account settings page.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your Git repository in your dbt Account settings page.")](#)Disconnect and reconnect your Git repository in your dbt Account settings page. 5. Click **Confirm Disconnect**. 6. On the same page, connect to the new Git provider repository by clicking **Configure Repository** * If you're using the native integration, you may need to OAuth to it. 7. That's it, you should now be connected to the new Git provider! 🎉 Note — As a tip, we recommend you refresh your page and Studio IDE before performing any actions. Why does dbt reject my custom GitHub application details? dbt validates your GitHub application against GitHub before saving it, so an invalid configuration is rejected instead of quietly breaking your team's connection. If you see an error when you save, check these in order: * **App ID**: Use the **App ID** from your GitHub app's settings page, not the client ID or the installation ID. * **Private key**: Paste the entire contents of the `.pem` file, including the `-----BEGIN RSA PRIVATE KEY-----` and `-----END RSA PRIVATE KEY-----` lines. If you lost the file, generate a new private key in GitHub. * **Client ID**: Paste the **Client ID** from your GitHub app. * **GitHub base URL**: Use the hostname of your GitHub instance with no trailing slash and no `/api/v3` path, for example `https://github.yourgreatcompany.com`. * **Client secret**: GitHub only shows this once. If you didn't copy it, generate a new one. * **Network access**: Your GitHub instance must be reachable from dbt. If you use [IP restrictions](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md), confirm the relevant CIDRs are allowed. For setup steps, refer to [Custom GitHub application](https://docs.getdbt.com/docs/platform/git/connect-github.md#custom-github-application), available in private beta. --- ##### Connect to GitLab dbt platform Connecting your GitLab account to dbt provides convenience and another layer of security to dbt: * Import new GitLab repos with a couple of clicks during dbt project setup. * Clone repos using HTTPS rather than SSH. * Carry GitLab user permissions through to dbt and dbt platform CLI git actions. * Trigger [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) builds when merge requests are opened in GitLab. info When configuring the repository in dbt, GitLab automatically: * Registers a webhook that triggers pipeline jobs in dbt. * Creates a [project access token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) in your GitLab repository, which sends the job run status back to GitLab using the dbt API for CI jobs. dbt automatically refreshes this token for you. You must have a [GitLab Premium or Ultimate account](https://about.gitlab.com/pricing/). Depending on your plan, use these steps to integrate GitLab with dbt: * For the Developer or Starter plans, follow these [instructions](#for-dbt-developer-and-starter-plans). * For the Enterprise or Enterprise+ plans, follow these [instructions](#for-the-dbt-enterprise-plans). #### For dbt Developer and Starter plans Before you can work with GitLab repositories in dbt, you’ll need to connect your GitLab account to your user profile. This allows dbt to authenticate your actions when interacting with Git repositories. Make sure to read the [requirements and limitations](#requirements-and-limitations) of the Starter and Developer plans before you connect your account. To connect your GitLab account: 1. From dbt, click on your account name in the left-side menu and select **Account settings**. 2. Select **Personal profile** under the **Your profile** section. 3. Go to **Linked accounts**. 4. Click **Link** to the right of your GitLab account. 5. In GitLab, sign in if prompted. When an authorization prompt appears, select the option to authorize dbt. After you authorize the request, you’ll be redirected back to dbt, and you'll see that your account has been linked to your profile. ##### Requirements and limitations dbt Starter and Developer plans use a single GitLab deploy token created by the first user who connects the repository, which means: * All repositories that users access from the dbt platform must belong to a [GitLab group](https://docs.gitlab.com/user/group/). * All Git operations (like commits and pushes) from the Studio IDE appear as coming from the same deploy token. * GitLab push rules may reject pushes made through dbt, particularly when multiple users are committing via the same deploy token. To support advanced Git workflows and multi-user commit behavior, upgrade to the Enterprise plan, which provides more flexible Git authentication strategies. #### For the dbt Enterprise plans dbt Enterprise and Enterprise+ customers have the added benefit of bringing their own GitLab OAuth application to dbt. This tier benefits from extra security, as dbt will: * Enforce user authorization with OAuth. * Carry GitLab's user repository permissions (read/write access) through to dbt or dbt platform CLI git actions. In order to connect GitLab in dbt, a GitLab account admin must: 1. [Set up a GitLab OAuth application](#setting-up-a-gitlab-oauth-application). 2. [Add the GitLab app to dbt](#adding-the-gitlab-oauth-application-to-dbt). Once the admin completes those steps, dbt developers need to: 1. [Personally authenticate with GitLab](#personally-authenticating-with-gitlab) from dbt. ##### Setting up a GitLab OAuth application We recommend that before you set up a project in dbt, a GitLab account admin set up an OAuth application in GitLab for use in dbt. For more detail, GitLab has a [guide for creating a Group Application](https://docs.gitlab.com/ee/integration/oauth_provider.html#group-owned-applications). To create a group-owned OAuth application in GitLab: 1. In GitLab, navigate to your group settings and select **Applications**. [![GitLab application navigation](</img/docs/dbt-platform/connecting-gitlab/gitlab nav.gif?v=2> "GitLab application navigation")](#)GitLab application navigation 2. When creating your group application, enter the following: | Field | Value | | ---------------- | ----------------------------------------- | | **Name** | dbt | | **Redirect URI** | `https://YOUR_ACCESS_URL/complete/gitlab` | | **Confidential** | ✅ | | **Scopes** | ✅ api | For the **Redirect URI** field, replace `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. 3. Click **Save application**. GitLab will generate an **Application ID** and **Secret**. These values remain available even if you close the app screen, so you can return to save them later. If you're a Business Critical customer using [IP restrictions](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md), ensure you've added the appropriate GitLab CIDRs to your IP restriction rules, or else the GitLab connection will fail. ##### Adding the GitLab OAuth application to dbt After you've created your GitLab application, add it to dbt: 1. In dbt, go to **Account settings** > **Integrations**, then expand the GitLab section. 2. Enter the following values: | Field | Value | | ------------------- | ---------------------------- | | **GitLab Instance** | `https://gitlab.com` | | **Application ID** | *copy value from GitLab app* | | **Secret** | *copy value from GitLab app* | If you use a self-hosted GitLab instance, set **GitLab Instance** to your organization’s GitLab hostname (for example, `https://gitlab.yourgreatcompany.com`). 3. Click **Save**. 4. You will then be redirected to GitLab and prompted to sign in. 5. When an authorization prompt appears, select the option to authorize dbt. After you authorize the request, you’ll be redirected back to dbt. Your integration is now ready for developers on your team to [personally authenticate with GitLab](#personally-authenticating-with-gitlab). ##### Personally authenticating with GitLab dbt developers on the Enterprise or Enterprise+ plan must each connect their GitLab profiles to dbt, as every developer's read/write access for the dbt repo is checked in the Studio IDE or dbt platform CLI. To connect a personal GitLab account: 1. From dbt, click on your account name in the left-side menu and select **Account settings**. 2. Select **Personal profile** under the **Your profile** section. 3. Go to **Linked accounts**. 4. If your GitLab account is not connected, you’ll see "No connected account". Select **Link** to begin the setup process. 5. In GitLab, sign in if prompted. When an authorization prompt appears, select the option to authorize dbt. Once you approve authorization, you will be redirected to dbt, and you should see your connected account. You're now ready to start developing in the Studio IDE or dbt CLI. #### Troubleshooting Unable to trigger a CI job with GitLab When you connect dbt to a GitLab repository, GitLab automatically registers a webhook in the background, viewable under the repository settings. This webhook is also used to trigger [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) when you push to the repository. If you're unable to trigger a CI job, this usually indicates that the webhook registration is missing or incorrect. To resolve this issue, navigate to the repository settings in GitLab and view the webhook registrations by navigating to GitLab --> **Settings** --> **Webhooks**. Some things to check: * The webhook registration is enabled in GitLab. * The webhook registration is configured with the correct URL and secret. If you're still experiencing this issue, reach out to the Support team at <support@getdbt.com> and we'll be happy to help! Errors importing a repository on dbt project set up If you don't see your repository listed, double-check that: * Your repository is in a Gitlab group you have access to. dbt will not read repos associated with a user. If you do see your repository listed, but are unable to import the repository successfully, double-check that: * You are a maintainer of that repository. Only users with maintainer permissions can set up repository connections. If you imported a repository using the dbt native integration with GitLab, you should be able to see if the clone strategy is using a `deploy_token`. If it's relying on an SSH key, this means the repository was not set up using the native GitLab integration, but rather using the generic git clone option. The repository must be reconnected in order to get the benefits described above. How can I fix my .gitignore file? A `.gitignore` file specifies which files git should intentionally ignore or 'untrack'. dbt indicates untracked files in the project file explorer pane by putting the file or folder name in *italics*. If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the Studio IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt to work correctly. The following sections describe how to fix the `.gitignore` file in:  Fix in the Studio IDE To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. 1. Launch the Studio IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. 3. Open the new or existing `gitignore` file, and add the following: ```bash # ✅ Correct target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` * **Note** — You can place these lines anywhere in the file, as long as they're on separate lines. The lines shown are wildcards that will include all nested files and folders. Avoid adding a trailing `'*'` to the lines, such as `target/*`. For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com/docs/gitignore). 4. Save the changes but *don't commit*. 5. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right corner of the IDE screen and select **Restart IDE**. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 6. Once the Studio IDE restarts, go to the **File Catalog** to delete the following files or folders (if they exist). No data will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. **Save** and then **Commit and sync** the changes. 8. Restart the Studio IDE again using the same procedure as step 5. 9. Once the Studio IDE restarts, use the **Create a pull request** (PR) button under the **Version Control** menu to start the process of integrating the changes. 10. When the git provider's website opens to a page with the new PR, follow the necessary steps to complete and merge the PR into the main branch of that repository. * **Note** — The 'main' branch might also be called 'master', 'dev', 'qa', 'prod', or something else depending on the organizational naming conventions. The goal is to merge these changes into the root branch that all other development branches are created from. 11. Return to the Studio IDE and use the **Change Branch** button, to switch to the main branch of the project. 12. Once the branch has changed, click the **Pull from remote** button to pull in all the changes. 13. Verify the changes by making sure the files/folders in the `.gitignore` file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).  Fix in the Git provider Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. There are two options for this approach: editing the main branch directly if allowed, or creating a pull request to implement the changes if required: ##### Edit in main branch When permissions allow it, it's possible to edit the `.gitignore` directly on the main branch of your repo. Here are the following steps: 1. Go to your repository's web interface. 2. Switch to the main branch and the root directory of your dbt project. 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deletions to the main branch. 8. Switch to the Studio IDE , and open the project that you're fixing. 9. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 10. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 11. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 12. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! ##### Unable to edit main branch If you can't edit the `.gitignore` directly on the main branch of your repo, follow these steps: 1. Go to your repository's web interface. 2. Switch to an existing development branch, or create a new branch just for these changes (This is often faster and cleaner). 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deleted folders. 8. Open a merge request using the git provider web interface. The merge request should attempt to merge the changes into the 'main' branch that all development branches are created from. 9. Follow the necessary procedures to get the branch approved and merged into the 'main' branch. You can delete the branch after the merge is complete. 10. Once the merge is complete, go back to the Studio IDE, and open the project that you're fixing. 11. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the Studio IDE by clicking on the three dots next to the **Studio IDE Status** button on the lower right corner of the Studio IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 12. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 13. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 14. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. I'm seeing a Gitlab authentication out of date error loop If you're seeing a 'GitLab Authentication is out of date' 500 server error page - this usually occurs when the deploy key in the repository settings in both dbt and GitLab do not match. No worries - this is a current issue the dbt Labs team is working on and we have a few workarounds for you to try: ###### First workaround 1. Disconnect repo from project in dbt. 2. Go to Gitlab and click on Settings > Repository. 3. Under Repository Settings, remove/revoke active dbt deploy tokens and deploy keys. 4. Attempt to reconnect your repository via dbt. 5. You would then need to check Gitlab to make sure that the new deploy key is added. 6. Once confirmed that it's added, refresh dbt and try developing once again. ###### Second workaround 1. Keep repo in project as is -- don't disconnect. 2. Copy the deploy key generated in dbt. 3. Go to Gitlab and click on Settings > Repository. 4. Under Repository Settings, manually add to your Gitlab project deploy key repo (with `Grant write permissions` box checked). 5. Go back to dbt, refresh your page and try developing again. If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! Can self-hosted GitLab instances only be connected via dbt Enterprise plans? Presently yes, this is only available to Enterprise users. This is because of the way you have to set up the GitLab app redirect URL for auth, which can only be customized if you're a user on an Enterprise plan. Check out our [pricing page](https://www.getdbt.com/pricing/) for more information or feel free to [contact us](https://www.getdbt.com/contact) to build your Enterprise pricing. How to migrate git providers To migrate from one git provider to another, refer to the following steps to avoid minimal disruption: 1. Outside of dbt, you'll need to import your existing repository into your new provider. By default, connecting your repository in one account won't automatically disconnected it from another account. As an example, if you're migrating from GitHub to Azure DevOps, you'll need to import your existing repository (GitHub) into your new Git provider (Azure DevOps). For detailed steps on how to do this, refer to your Git provider's documentation (Such as [GitHub](https://docs.github.com/en/migrations/importing-source-code/using-github-importer/importing-a-repository-with-github-importer), [GitLab](https://docs.gitlab.com/ee/user/project/import/repo_by_url.html), [Azure DevOps](https://learn.microsoft.com/en-us/azure/devops/repos/git/import-git-repository?view=azure-devops)) 2. Go back to dbt and set up your [integration for the new Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md), if needed. 3. Disconnect the old repository in dbt by going to **Account Settings** and then **Projects**. 4. Click on the **Repository** link, then click **Edit** and **Disconnect**. [![Disconnect and reconnect your Git repository in your dbt Account settings page.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your Git repository in your dbt Account settings page.")](#)Disconnect and reconnect your Git repository in your dbt Account settings page. 5. Click **Confirm Disconnect**. 6. On the same page, connect to the new Git provider repository by clicking **Configure Repository** * If you're using the native integration, you may need to OAuth to it. 7. That's it, you should now be connected to the new Git provider! 🎉 Note — As a tip, we recommend you refresh your page and Studio IDE before performing any actions. GitLab token refresh message When you connect dbt to a GitLab repository, GitLab automatically creates a [project access token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) in your GitLab repository in the background. This sends the job run status back to Gitlab using the dbt API for CI jobs. By default, the project access token follows a naming pattern: `dbt token for GitLab project: <project_id>`. If you have multiple tokens in your repository, look for one that follows this pattern to identify the correct token used by dbt. If you're receiving a "Refresh token" message, don't worry — dbt automatically refreshes this project access token for you, which means you never have to manually rotate it. If you still experience any token refresh errors, please try disconnecting and reconnecting the repository in your dbt project to refresh the token. For any issues, please reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ##### Connect with Git clone dbt platform In dbt, you can import a git repository from any valid git URL that points to a dbt project. There are some important considerations to keep in mind when doing this. #### Git protocols Use an SSH clone URL for your repo. dbt supports both common SSH formats: * `git@github.com:organization/repo-name.git` * `ssh://git@github.com/organization/repo-name.git` Don't use `https://...` or other HTTP URLs. dbt clones repos over SSH and can't clone repos that use the HTTP protocol. ##### Converting SSH URLs to ssh:// Git providers often show SSH clone URLs in SCP-style format, with a colon between the host and repo path (usually found under **Code** -> **Clone** -> **SSH**). Example of SCP-style with a colon: ```text git@github.com:organization/repo-name.git user@custom-host.example.com:organization/repo-name.git ``` If you prefer a `ssh://` URL instead, you can rewrite it as follows. The equivalent ssh:// format uses a slash after the host: ```text ssh://git@github.com/organization/repo-name.git ssh://user@custom-host.example.com/organization/repo-name.git ``` #### Availability of features by Git provider * If your git provider has a [native dbt integration](https://docs.getdbt.com/docs/platform/git/configure-git.md), you can seamlessly set up [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/ci-jobs.md) jobs directly within dbt. * For providers without native integration, you can still use the [Git clone method](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to import your git URL and leverage the [dbt Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to trigger a CI job to run. The following table outlines the available integration options and their corresponding capabilities. | **Git provider** | **Native dbt integration** | **Automated CI job** | **Git clone** | **Information** | **Supported plans** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | -------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md)<br /> | ✅ | ✅ | ✅ | Organizations on the Starter and Developer plans can connect to Azure DevOps using a deploy key. Note, you won’t be able to configure automated CI jobs but you can still develop. | Enterprise, Enterprise+ | | [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | All other git providers using [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) ([BitBucket](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#bitbucket), [AWS CodeCommit](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#aws-codecommit), and others) | ❌ | ❌ | ✅ | Refer to the [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=1) guide to set up continuous integration and continuous deployment (CI/CD). | | #### Managing deploy keys After importing a project by Git URL, dbt will generate a Deploy Key for your repository. To find the deploy key in dbt: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Go to **Projects** and select a project. 3. Click the **Repository** link to the repository details page. 4. Copy the key under the **Deploy Key** section. You must provide this Deploy Key in the Repository configuration of your Git host. Configure this Deploy Key to allow *read and write access* to the specified repositories. **Note**: Each dbt project will generate a different deploy key when connected to a repo, even if two projects are connected to the same repo. You will need to supply both deploy keys to your Git provider. #### Troubleshooting clone errors If dbt can't clone your repository after you connect it using a git URL, the issue is usually one of the following: * **The URL is invalid**: Confirm the repo path is correct, the repo exists, and you are using an SSH clone URL (`git@...` or `ssh://...`). Do not use `https://...` or other HTTP URLs. * **Your git provider is unavailable**: Check your provider's status page (for example, GitHub, GitLab, Bitbucket, Azure DevOps) and retry after service is restored. * **SSH access isn't configured correctly**: Make sure the deploy key generated by dbt is added to your repository with the required access (including write access where applicable). If your organization restricts SSH keys, coordinate with your admins to allow deploy keys for the repository. If you still can't connect, [contact dbt Support](mailto:support@getdbt.com) and include the repository URL you're using and the time the error occurred. #### GitHub Use GitHub? If you use GitHub, you can import your repo directly using [dbt's GitHub Application](https://docs.getdbt.com/docs/platform/git/connect-github.md). Connecting your repo via the GitHub Application [enables Continuous Integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). ##### Locate your SSH URL 1. Navigate to your repository on GitHub. 2. Click the **Code** button. 3. Select the **SSH** tab to view the SSH clone URL. 4. Copy the URL from the **SSH** tab. SCP-style (`git@...`) and `ssh://...` URLs are both supported. If you want the `ssh://` form, refer to [Converting SSH URLs to ssh://](#converting-ssh-urls-to-ssh). ##### Add a deploy key To add a deploy key to a GitHub account, navigate to the Deploy keys tab of the settings page in your GitHub repository. * After supplying a name for the deploy key and pasting in your deploy key (generated by dbt), be sure to check the **Allow write access** checkbox. * After adding this key, dbt will be able to read and write files in your dbt project. * Refer to [Adding a deploy key in GitHub](https://github.blog/2015-06-16-read-only-deploy-keys/) [![Configuring a GitHub Deploy Key](/img/docs/dbt-platform/platform-configuring-dbt-platform/cd7351c-Screen_Shot_2019-10-16_at_1.09.41_PM.png?v=2 "Configuring a GitHub Deploy Key")](#)Configuring a GitHub Deploy Key #### GitLab Use GitLab? If you use GitLab, you can import your repo directly using [dbt's GitLab Application](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md). Connecting your repo via the GitLab Application [enables Continuous Integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). * To add a deploy key to a GitLab account, navigate to the [SSH keys](https://gitlab.com/profile/keys) tab in the User Settings page of your GitLab account. * Next, paste in the deploy key generated by dbt for your repository. * After saving this SSH key, dbt will be able to read and write files in your GitLab repository. * Refer to [Adding a read only deploy key in GitLab](https://docs.gitlab.com/ee/user/project/deploy_keys/) [![Configuring a GitLab SSH Key](/img/docs/dbt-platform/platform-configuring-dbt-platform/f3ea88d-Screen_Shot_2019-10-16_at_4.45.50_PM.png?v=2 "Configuring a GitLab SSH Key")](#)Configuring a GitLab SSH Key #### BitBucket Use a deploy key to import your BitBucket repository into dbt. To preserve account security, use a service account to add the BitBucket deploy key and maintain the connection between your BitBucket repository and dbt. BitBucket links every repository commit and other Git actions (such as opening a pull request) to the email associated with the user's BitBucket account. To add a deploy key to a BitBucket account: * Navigate to **SSH keys** tab in the Personal Settings page of your BitBucket account. * Next, click the **Add key** button and paste in the deploy key generated by dbt for your repository. * After saving this SSH key, dbt will be able to read and write files in your BitBucket repository. [![Configuring a BitBucket SSH Key](/img/docs/dbt-platform/platform-configuring-dbt-platform/bitbucket-ssh-key.png?v=2 "Configuring a BitBucket SSH Key")](#)Configuring a BitBucket SSH Key #### AWS CodeCommit dbt can work with dbt projects hosted on AWS CodeCommit, but there are some extra steps needed compared to GitHub or other git providers. This guide will help you connect your CodeCommit-hosted dbt project to dbt. ###### Step 1: Create an AWS User for dbt * To give dbt access to your repository, first you'll need to create an AWS IAM user for dbt. * Log into the AWS Console and navigate to the IAM section. * Click **Add User**, and create a new user by entering a unique and meaningful user name. * The user will need clone access to your repository. You can do this by adding the **AWSCodeCommitPowerUser** permission during setup. ###### Step 2: Import your repository by name * Open the AWS CodeCommit console and choose your repository. * Copy the SSH URL from that page. * Next, navigate to the **New Repository** page in dbt. * Choose the **Git Clone** tab, and paste in the SSH URL you copied from the console. * In the newly created Repository details page, you'll see a **Deploy Key** field. * Copy the contents of this field as you'll need it for [Step 3](#step-3-grant-dbt-cloud-aws-user-access). **Note:** The dbt-generated public key is the only key that will work in the next step. Any other key that has been generated outside of dbt will not work. ###### Step 3: Grant dbt AWS User access * Open up the newly created dbt user in the AWS IAM Console. * Choose the **Security Credentials** tab and then click **Upload SSH public key**. * Paste in the contents of the **Public Key** field from the dbt Repository page. * Once you've created the key, you'll see an **SSH key ID** for it. * **[Contact dbt Support](mailto:support@getdbt.com)** and share that field so that dbt Support team can complete the setup process for you. ###### Step 4: Specify a custom branch in dbt CodeCommit uses `master` as its default branch, and to initialize your project, you'll need to specify the `master` branch as a [custom branch](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md#development) in dbt. * Go to **Deploy** -> **Environments** -> **Development**. * Select **Settings** -> **Edit** and under **General Settings**, check the **Default to a custom branch** checkbox. * Specify the custom branch as `master` and click **Save**. ###### Step 5: Configure pull request template URLs (Optional) To prevent users from directly merging code changes into the default branch, configure the [PR Template URL](https://docs.getdbt.com/docs/platform/git/pr-template.md) in the **Repository details** page for your project. Once configured, dbt will prompt users to open a new PR after committing and syncing code changes on the branch in the Studio IDE, before merging any changes into the default branch. * Go to **Account Settings** -> **Projects** -> Select the project. * Click the repository link under **Repository**. * In the **Repository details** page, click **Edit** in the lower right. [![Configure PR template in the 'Repository details' page.](/img/docs/collaborate/repo-details.jpg?v=2 "Configure PR template in the 'Repository details' page.")](#)Configure PR template in the 'Repository details' page. * In the **Pull request URL** field, set the URL based on the suggested [PR template format](https://docs.getdbt.com/docs/platform/git/pr-template.md#aws-codecommit). * Replace `<repo>` with the name of your repository (Note that it is case sensitive). In the following example, the repository name is `New_Repo`. [![In the Pull request URL field example, the repository name is 'New\_Repo'.](/img/docs/collaborate/pr-template-example.jpg?v=2 "In the Pull request URL field example, the repository name is 'New_Repo'.")](#)In the Pull request URL field example, the repository name is 'New\_Repo'. * After filling the **Pull request URL** field, click **Save**. 🎉 **You're all set!** Once dbt Support handles your request and you've set your custom branch, your project is ready to execute dbt runs on dbt. #### Azure DevOps Use Azure DevOps? If you use Azure DevOps and you are on the dbt Enterprise or Enterprise+ plan, you can import your repo directly using [dbt's Azure DevOps Integration](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md). Connecting your repo via the Azure DevOps Application [enables Continuous Integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). 1. To add a deploy key to an Azure DevOps account, navigate to the **SSH public keys** page in the User Settings of your user's Azure DevOps account or a service user's account. 2. We recommend using a dedicated service user for the integration to ensure that dbt's connection to Azure DevOps is not interrupted by changes to user permissions. [![Navigate to the 'SSH public keys' settings page](/img/docs/dbt-platform/platform-configuring-dbt-platform/52bfdaa-Screen_Shot_2020-03-09_at_4.13.20_PM.png?v=2 "Navigate to the 'SSH public keys' settings page")](#)Navigate to the 'SSH public keys' settings page 3. Next, click the **+ New Key** button to create a new SSH key for the repository. [![Click the '+ New Key' button to create a new SSH key for the repository.](/img/docs/dbt-platform/platform-configuring-dbt-platform/6d8e980-Screen_Shot_2020-03-09_at_4.13.27_PM.png?v=2 "Click the '+ New Key' button to create a new SSH key for the repository.")](#)Click the '+ New Key' button to create a new SSH key for the repository. 4. Select a descriptive name for the key and then paste in the deploy key generated by dbt for your repository. 5. After saving this SSH key, dbt will be able to read and write files in your Azure DevOps repository. [![Enter and save the public key generated for your repository by dbt](/img/docs/dbt-platform/platform-configuring-dbt-platform/d19f199-Screen_Shot_2020-03-09_at_4.13.50_PM.png?v=2 "Enter and save the public key generated for your repository by dbt")](#)Enter and save the public key generated for your repository by dbt #### Other git providers Don't see your git provider here? Please [contact dbt Support](mailto:support@getdbt.com). We're happy to help you set up dbt with any supported git provider. #### Limited integration Some features of dbt require a tight integration with your git host, for example, updating GitHub pull requests with dbt run statuses. Importing your project by a URL prevents you from using these features. Once you give dbt access to your repository, you can continue to set up your project by adding a connection and creating and running your first dbt job. #### FAQs How can I fix my .gitignore file? A `.gitignore` file specifies which files git should intentionally ignore or 'untrack'. dbt indicates untracked files in the project file explorer pane by putting the file or folder name in *italics*. If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the Studio IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt to work correctly. The following sections describe how to fix the `.gitignore` file in:  Fix in the Studio IDE To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. 1. Launch the Studio IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. 3. Open the new or existing `gitignore` file, and add the following: ```bash # ✅ Correct target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` * **Note** — You can place these lines anywhere in the file, as long as they're on separate lines. The lines shown are wildcards that will include all nested files and folders. Avoid adding a trailing `'*'` to the lines, such as `target/*`. For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com/docs/gitignore). 4. Save the changes but *don't commit*. 5. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right corner of the IDE screen and select **Restart IDE**. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 6. Once the Studio IDE restarts, go to the **File Catalog** to delete the following files or folders (if they exist). No data will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. **Save** and then **Commit and sync** the changes. 8. Restart the Studio IDE again using the same procedure as step 5. 9. Once the Studio IDE restarts, use the **Create a pull request** (PR) button under the **Version Control** menu to start the process of integrating the changes. 10. When the git provider's website opens to a page with the new PR, follow the necessary steps to complete and merge the PR into the main branch of that repository. * **Note** — The 'main' branch might also be called 'master', 'dev', 'qa', 'prod', or something else depending on the organizational naming conventions. The goal is to merge these changes into the root branch that all other development branches are created from. 11. Return to the Studio IDE and use the **Change Branch** button, to switch to the main branch of the project. 12. Once the branch has changed, click the **Pull from remote** button to pull in all the changes. 13. Verify the changes by making sure the files/folders in the `.gitignore` file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).  Fix in the Git provider Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. There are two options for this approach: editing the main branch directly if allowed, or creating a pull request to implement the changes if required: ##### Edit in main branch When permissions allow it, it's possible to edit the `.gitignore` directly on the main branch of your repo. Here are the following steps: 1. Go to your repository's web interface. 2. Switch to the main branch and the root directory of your dbt project. 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deletions to the main branch. 8. Switch to the Studio IDE , and open the project that you're fixing. 9. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 10. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 11. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 12. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! ##### Unable to edit main branch If you can't edit the `.gitignore` directly on the main branch of your repo, follow these steps: 1. Go to your repository's web interface. 2. Switch to an existing development branch, or create a new branch just for these changes (This is often faster and cleaner). 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deleted folders. 8. Open a merge request using the git provider web interface. The merge request should attempt to merge the changes into the 'main' branch that all development branches are created from. 9. Follow the necessary procedures to get the branch approved and merged into the 'main' branch. You can delete the branch after the merge is complete. 10. Once the merge is complete, go back to the Studio IDE, and open the project that you're fixing. 11. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the Studio IDE by clicking on the three dots next to the **Studio IDE Status** button on the lower right corner of the Studio IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 12. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 13. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 14. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. How to migrate git providers To migrate from one git provider to another, refer to the following steps to avoid minimal disruption: 1. Outside of dbt, you'll need to import your existing repository into your new provider. By default, connecting your repository in one account won't automatically disconnected it from another account. As an example, if you're migrating from GitHub to Azure DevOps, you'll need to import your existing repository (GitHub) into your new Git provider (Azure DevOps). For detailed steps on how to do this, refer to your Git provider's documentation (Such as [GitHub](https://docs.github.com/en/migrations/importing-source-code/using-github-importer/importing-a-repository-with-github-importer), [GitLab](https://docs.gitlab.com/ee/user/project/import/repo_by_url.html), [Azure DevOps](https://learn.microsoft.com/en-us/azure/devops/repos/git/import-git-repository?view=azure-devops)) 2. Go back to dbt and set up your [integration for the new Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md), if needed. 3. Disconnect the old repository in dbt by going to **Account Settings** and then **Projects**. 4. Click on the **Repository** link, then click **Edit** and **Disconnect**. [![Disconnect and reconnect your Git repository in your dbt Account settings page.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your Git repository in your dbt Account settings page.")](#)Disconnect and reconnect your Git repository in your dbt Account settings page. 5. Click **Confirm Disconnect**. 6. On the same page, connect to the new Git provider repository by clicking **Configure Repository** * If you're using the native integration, you may need to OAuth to it. 7. That's it, you should now be connected to the new Git provider! 🎉 Note — As a tip, we recommend you refresh your page and Studio IDE before performing any actions. --- ##### Connect with managed repository dbt platform Managed repositories are a great way to trial dbt without needing to create a new repository. If you don't already have a Git repository for your dbt project, you can let dbt host and manage a repository for you. If in the future you choose to host this repository elsewhere, you can export the information from dbt at any time. Refer to [Move from a managed repository to a self-hosted repository](https://docs.getdbt.com/faqs/Git/managed-repo.md) for more information on how to do that. info dbt Labs recommends against using a managed repository in a production environment. You can't use Git features like pull requests, which are part of our recommended version control best practices. To set up a project with a managed repository: 1. From your **Account settings** in dbt, select the project you want to set up with a managed repository. If the project already has a repository set up, you need to edit the repository settings and disconnect the existing repository. 2. Click **Edit** for the project. 3. Under Repository, click **Configure repository**. 4. Select **Managed**. 5. Enter a name for the repository. For example, "analytics" or "dbt-models." 6. Click **Create**. [![Adding a managed repository](/img/docs/dbt-platform/platform-configuring-dbt-platform/managed-repo.png?v=2 "Adding a managed repository")](#)Adding a managed repository #### Download managed repository To download a copy of your managed repository from dbt to your local machine: 1. Use the **Project** selector on the main left-side menu to navigate to a project that's using a managed repository. 2. Click **Dashboard** from the main left-side menu. 3. From the dashboard, click **Settings**. 4. Locate the **Repository** field and click the hyperlink for the repo. 5. Below the **Deploy key** you will find the **Download repository** option. Click the button to download. If you don't see this option, you're either not assigned a [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-permissions) with `write` access to Git repositories, or you don't have a managed repo for your project. [![The download button for a managed repo.](/img/docs/dbt-platform/platform-configuring-dbt-platform/download-managed-repo.png?v=2 "The download button for a managed repo.")](#)The download button for a managed repo. --- ##### Merge conflicts dbt platform [Merge conflicts](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/about-merge-conflicts) in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) often occur when multiple users are simultaneously making edits to the same section in the same file. This makes it difficult for Git to decide what changes to incorporate in the final merge. The merge conflict process provides users the ability to choose which lines of code they'd like to preserve and commit. This document will show you how to resolve merge conflicts in the Studio IDE. #### Identify merge conflicts You can experience a merge conflict in two possible ways: * Pulling changes from your main branch when someone else has merged a conflicting change. * Committing your changes to the same branch when someone else has already committed their change first. The way to [resolve](#resolve-merge-conflicts) either scenario will be exactly the same. For example, if you and a teammate make changes to the same file and commit, you will encounter a merge conflict as soon as you **Commit and sync**. The Studio IDE will display: * **Commit and resolve** git action bar under **Version Control** instead of **Commit** — This indicates that the Cloud Studio IDE has detected some conflicts that you need to address. * A 2-split editor view — The left view includes your code changes and is read-only. The right view includes the additional changes, allows you to edit and marks the conflict with some flags: ```text <<<<<< HEAD your current code ====== conflicting code >>>>>> (some branch identifier) ``` * The file and path colored in red in the **File Catalog**, with a warning icon to highlight files that you need to resolve. * The file name colored in red in the **Changes** section, with a warning icon. * If you press commit without resolving the conflict, the Studio IDE will prompt a pop up box with a list which files need to be resolved. [![Conflicting section that needs resolution will be highlighted](/img/docs/dbt-platform/platform-ide/merge-conflict.png?v=2 "Conflicting section that needs resolution will be highlighted")](#)Conflicting section that needs resolution will be highlighted [![Pop up box when you commit without resolving the conflict](/img/docs/dbt-platform/platform-ide/commit-without-resolve.png?v=2 "Pop up box when you commit without resolving the conflict")](#)Pop up box when you commit without resolving the conflict #### Resolve merge conflicts You can seamlessly resolve merge conflicts that involve competing line changes in the Cloud Studio IDE. 1. In the Studio IDE, you can edit the right-side of the conflict file, choose which lines of code you'd like to preserve, and delete the rest. * Note: The left view editor is read-only and you cannot make changes. 2. Delete the special flags or conflict markers `<<<<<<<`, `=======`, `>>>>>>>` that highlight the merge conflict and also choose which lines of code to preserve. 3. If you have more than one merge conflict in your file, scroll down to the next set of conflict markers and repeat steps one and two to resolve your merge conflict. 4. Press **Save**. You will notice the line highlights disappear and return to a plain background. This means that you've resolved the conflict successfully. 5. Repeat this process for every file that has a merge conflict. [![Choosing lines of code to preserve](/img/docs/dbt-platform/platform-ide/resolve-conflict.png?v=2 "Choosing lines of code to preserve")](#)Choosing lines of code to preserve Edit conflict files * If you open the conflict file under **Changes**, the file name will display something like `model.sql (last commit)` and is fully read-only and cannot be edited. <br /> * If you open the conflict file under **File Catalog**, you can edit the file in the right view. #### Commit changes When you've resolved all the merge conflicts, the last step would be to commit the changes you've made. 1. Click the git action bar **Commit and resolve**. 2. The **Commit Changes** pop up box will confirm that all conflicts have been resolved. Write your commit message and click **Commit Changes**. 3. The Studio IDE will return to its normal state and you can continue developing! [![Conflict has been resolved](/img/docs/dbt-platform/platform-ide/commit-resolve.png?v=2 "Conflict has been resolved")](#)Conflict has been resolved [![Commit Changes pop up box to commit your changes](/img/docs/dbt-platform/platform-ide/commit-changes.png?v=2 "Commit Changes pop up box to commit your changes")](#)Commit Changes pop up box to commit your changes --- ##### PR template dbt platform #### Configure pull request (PR) template URLs When you commit changes to a branch in the Studio IDE, dbt can prompt users to open a new Pull Request for the code changes. To enable this functionality, ensure that a PR Template URL is configured in the **Repository details** page in your **Account Settings**. If this setting is blank, the Studio IDE will prompt users to merge the changes directly into their default branch. [![Configure a PR template in the 'Repository details' page.](/img/docs/collaborate/repo-details.jpg?v=2 "Configure a PR template in the 'Repository details' page.")](#)Configure a PR template in the 'Repository details' page. ##### PR Template URL by git provider The PR Template URL setting will be automatically set for most repositories, depending on the connection method. * If you connect to your repository via in-app integrations with your git provider or the "Git Clone" method via SSH, this URL setting will be auto-populated and editable. * For AWS CodeCommit, this URL setting isn't auto-populated and must be [manually configured](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#step-5-configure-pull-request-template-urls-optional). * If you connect via a dbt [Managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md), this URL will not be set, and the Studio IDE will prompt users to merge the changes directly into their default branch. The PR template URL supports two variables that can be used to build a URL string. These variables, `{{source}}` and `{{destination}}` return branch names based on the state of the configured Environment and active branch open in the IDE. The `{{source}}` variable represents the active development branch, and the `{{destination}}` variable represents the configured base branch for the environment, eg. `master`. A typical PR build URL looks like: ##### Template ```text https://github.com/dbt-labs/jaffle_shop/compare/{{destination}}..{{source}} ``` ##### Rendered ```text https://github.com/dbt-labs/jaffle_shop/compare/master..my-branch ``` #### Example templates Some common URL templates are provided below, but please note that the exact value may vary depending on your configured git provider. ##### GitHub ```text https://github.com/<org>/<repo>/compare/{{destination}}..{{source}} ``` If you're using Github Enterprise your template may look something like: ```text https://git.<mycompany>.com/<org>/<repo>/compare/{{destination}}..{{source}} ``` ##### GitLab ```text https://gitlab.com/<org>/<repo>/-/merge_requests/new?merge_request[source_branch]={{source}}&merge_request[target_branch]={{destination}} ``` ##### BitBucket ```text https://bitbucket.org/<org>/<repo>/pull-requests/new?source={{source}}&dest={{destination}} ``` If you're using BitBucket Server or Data Center your template may look something like: ```text https://<bitbucket-server>/projects/<proj>/repos/<repo>/pull-requests?create&sourceBranch={{source}}&targetBranch={{destination}} ``` ##### AWS CodeCommit ```text https://console.aws.amazon.com/codesuite/codecommit/repositories/<repo>/pull-requests/new/refs/heads/{{destination}}/.../refs/heads/{{source}} ``` ##### Azure DevOps ```text https://dev.azure.com/<org>/<project>/_git/<repo>/pullrequestcreate?sourceRef={{source}}&targetRef={{destination}} ``` --- ##### Set up Azure DevOps dbt platform | Enterprise, Enterprise+ #### Service principal overview note If this is your first time setting up an Entra app as a service principal, refer to the [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal) for any prerequisite steps you may need to take to prepare. To use dbt's native integration with Azure DevOps, an account admin needs to set up a Microsoft Entra ID app as a service principal. We recommend setting up a separate [Entra ID application than used for SSO](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-microsoft-entra-id.md). The application's service principal represents the Entra ID application object. While a "service user" represents a real user in Azure with an Entra ID (and an applicable license), the "service principal" is a secure identity used by an application to access Azure resources unattended. The service principal authenticates with a client ID and secret rather than a username and password (or any other form of user auth). Service principals are the [Microsoft recommended method](https://learn.microsoft.com/en-us/entra/architecture/secure-service-accounts#types-of-microsoft-entra-service-accounts) for authenticating apps. 1. [Register an Entra ID app](#register-a-microsoft-entra-id-app). 2. [Connect Azure DevOps to your new app](#connect-azure-devops-to-your-new-app). 3. [Add your Entra ID app to dbt](#connect-your-microsoft-entra-id-app-to-dbt). Once the Microsoft Entra ID app is added to dbt, it will act as a [service principal](https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals?tabs=browser), which will be used to power headless actions in dbt such as deployment runs and CI. The dbt developers can then personally authenticate in dbt from Azure DevOps. For more, see [Authenticate with Azure DevOps](https://docs.getdbt.com/docs/platform/git/authenticate-azure.md). The following personas are required to complete the steps on this page: * Microsoft Entra ID admin * Azure DevOps admin * dbt account admin * Azure admin (if your Entra ID and Azure DevOps environments are not connected) #### Register a Microsoft Entra ID app Admin consent If your tenant is configured to disallow user self-consent, an Entra ID admin must grant tenant-wide admin consent to this app after you add the API permissions (Enterprise Applications -> your app -> Permissions -> Grant admin consent). A Microsoft Entra ID admin needs to perform the following steps: 1. Sign into your Azure portal and click **Microsoft Entra ID**. 2. Select **App registrations** in the left panel. 3. Select **New registration**. The form for creating a new Entra ID app opens. 4. Provide a name for your app. We recommend using, "dbt Labs Azure DevOps app". 5. Select **Accounts in any organizational directory (Any Entra ID directory - Multitenant)** as the Supported Account Types. Many customers ask why they need to select Multitenant instead of Single Tenant, and they frequently get this step wrong. Microsoft considers Azure DevOps (formerly called Visual Studio) and Microsoft Entra ID separate tenants, and for the Entra ID application to work properly, you must select Multitenant. 6. Set **Redirect URI** to **Web**. Copy and paste the Redirect URI from dbt into the next field. To find the Redirect URI in dbt: 1. In dbt, navigate to **Account Settings** -> **Integrations**. 2. Click the **edit icon** next to **Azure DevOps**. 3. Copy the first **Redirect URIs** value which looks like `https://<YOUR_ACCESS_URL>/complete/azure_active_directory` and does NOT end with `service_user`. 7. Click **Register**. Here's what your app should look like before registering it: [![Registering a Microsoft Entra ID app](</img/docs/dbt-platform/connecting-azure-devops/AD app.png?v=2> "Registering a Microsoft Entra ID app")](#)Registering a Microsoft Entra ID app #### Create a client secret A Microsoft Entra ID admin needs to complete the following steps: 1. Navigate to **Microsoft Entra ID**, click **App registrations**, and click on your app. 2. Select **Certificates and Secrets** from the left navigation panel. 3. Select **Client secrets** and click **New client secret** 4. Give the secret a description and select the expiration time. Click **Add**. 5. Copy the **Value** field and securely share it with the dbt account admin, who will complete the setup. #### Create the app's service principal After you've created the app, you need to verify whether it has a service principal. In many cases, if this has been configured before, new apps will get one assigned upon creation. 1. Navigate to **Microsoft Entra ID**. 2. Under **Manage** on the left-side menu, click **App registrations**. 3. Click the app for the dbt and Azure DevOps integration. 4. Locate the **Managed application in local directory** field and, if it has the option, click **Create Service Principal**. If the field is already populated, a service principal has already been assigned. [![Example of the 'Create Service Principal' option highlighted .](/img/docs/platform-integrations/create-service-principal.png?v=2 "Example of the 'Create Service Principal' option highlighted .")](#)Example of the 'Create Service Principal' option highlighted . #### Add permissions to your service principal An Entra ID admin needs to provide your new app access to Azure DevOps: 1. Select **API permissions** in the left navigation panel. 2. Remove the **Microsoft Graph / User Read** permission. 3. Click **Add a permission**. 4. Select **Azure DevOps**. 5. Select the **user\_impersonation** permission. This is the only permission available for Azure DevOps. #### Connect Azure DevOps to your new app An Azure admin will need one of the following permissions in both the Microsoft Entra ID and Azure DevOps environments: * Azure Service Administrator * Azure Co-administrator note You can only add a managed identity or service principal for the tenant to which your organization is connected. You need to add a directory to your organization so that it can access all the service principals and other identities. Navigate to **Organization settings** --> **Microsoft Entra** --> **Connect Directory** to connect. 1. From your Azure DevOps account organization screen, click **Organization settings** in the bottom left. 2. Under **General** settings, click **Users**. 3. Click **Add users**, and in the resulting panel, enter the service principal's name in the first field. Then, click the name when it appears below the field. 4. In the **Add to projects** field, click the boxes for any projects you want to include (or select all). 5. Set the **Azure DevOps Groups** to **Project Administrator**. [![Example setup with the service principal added as a user.](/img/docs/dbt-platform/connecting-azure-devops/add-service-principal.png?v=2 "Example setup with the service principal added as a user.")](#)Example setup with the service principal added as a user. #### Connect your Microsoft Entra ID app to dbt A dbt account admin must take the following actions. Once you connect your Microsoft Entra ID app and Azure DevOps, you must provide dbt information about the app. If this is a first-time setup, you will create a new configuration. If you are [migrating from a service user](#migrate-to-service-principal), you can edit an existing configuration and change it to **Service principal**. To create the configuration: 1. Navigate to your account settings in dbt. 2. Select **Integrations**. 3. Scroll to the Azure DevOps section and click the **Edit icon**. 4. Select the **Service principal** option (service user configurations will auto-complete the fields, if applicable). 5. Complete/edit the form (if you are migrating, the existing configurations carry over): * **Azure DevOps Organization:** Must match the name of your Azure DevOps organization exactly. Do not include the `dev.azure.com/` prefix in this field. ✅ Use `my-DevOps-org` ❌ Avoid `dev.azure.com/my-DevOps-org` * **Application (client) ID:** Found in the Microsoft Entra ID app. * **Client Secret**: Copy the **Value** field in the Microsoft Entra ID app client secrets and paste it into the **Client Secret** field in dbt. Entra ID admins are responsible for the expiration of the app secret, and dbt Admins should note the expiration date for rotation. * **Directory(tenant) ID:** Found in the Microsoft Entra ID app. [![Fields for adding Entra ID app to dbt.](/img/docs/platform-integrations/service-principal-fields.png?v=2 "Fields for adding Entra ID app to dbt.")](#)Fields for adding Entra ID app to dbt. Your Microsoft Entra ID app should now be added to your dbt Account. People on your team who want to develop in the Studio IDE or dbt CLI can now personally [authorize Azure DevOps from their profiles](https://docs.getdbt.com/docs/platform/git/authenticate-azure.md). #### Migrate to service principal Migrate from a service user to a service principal using the existing app. It will only take a few steps, and you won't experience any service disruptions. * Verify whether or not your app has a service principal * If not, create the app service principal * Update the application's configuration * Update the configuration in dbt ##### Verify the service principal You will need an Entra ID admin to complete these steps. To confirm whether your existing app already has a service principal: 1. In the Azure account, navigate to **Microsoft Entra ID** -> **Manage** -> **App registrations**. 2. Click on the application for the service user integration with dbt. 3. Verify whether a name populates the **Managed application in local directory** field. * If a name exists: The service principal has been created. Move on to step 4. * If no name exists: Go to the next section, [Create the service principal](#create-the-service-principal). 4. Follow the instructions to [add permissions](#add-permissions-to-your-service-principal) to your service principal. 5. Follow the instructions to [connect DevOps to your app](#connect-azure-devops-to-your-new-app). 6. In your dbt account: 1. Navigate to **Account settings** and click **Integrations** 2. Click the **edit icon** to the right of the **Azure DevOps** settings. 3. Change **Service user** to **Service principal** and click **Save**. You do not need to edit any existing fields. ##### Create the service principal If there is no name populating that field, a Service Principal does not exist. To configure a Service Principal, please review the instructions here. If your dbt app does not have a service principal, take the following actions in your Azure account: 1. Navigate to **Microsoft Entra ID**. 2. Under **Manage** on the left-side menu, click **App registrations**. 3. Click the app for the dbt and Azure DevOps integration. 4. Locate the **Managed application in local directory** field and click **Create Service Principal**. [![Example of the 'Create Service Principal' option highlighted .](/img/docs/platform-integrations/create-service-principal.png?v=2 "Example of the 'Create Service Principal' option highlighted .")](#)Example of the 'Create Service Principal' option highlighted . 5. Follow the instructions to [add permissions](#add-permissions-to-your-service-principal) to your service principal. 6. Follow the instructions to [connect DevOps to your app](#connect-azure-devops-to-your-new-app). 7. In your dbt account: 1. Navigate to **Account settings** and click **Integrations** 2. Click the **edit icon** to the right of the **Azure DevOps** settings. 3. Change **Service user** to **Service principal** and click **Save**. You do not need to edit any existing fields. --- ##### Set up Azure DevOps with Service User dbt platform | Enterprise, Enterprise+ #### Service user overview important Service users are no longer a recommended method for authentication and dbt is rolling out a new [Entra ID service principal](https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals) option. Once the option is available in your account settings, you should plan to [migrate from service user to service principal](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md#migrate-to-service-principal). Service principals are the [Microsoft recommended service account type](https://learn.microsoft.com/en-us/entra/architecture/secure-service-accounts#types-of-microsoft-entra-service-accounts) for app authentication. To use our native integration with Azure DevOps in dbt, an account admin needs to set up an Microsoft Entra ID app. We recommend setting up a separate [Entra ID application than used for SSO](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-microsoft-entra-id.md). 1. [Register an Entra ID app](#register-a-microsoft-entra-id-app). 2. [Add permissions to your new app](#add-permissions-to-your-new-app). 3. [Add another redirect URI](#add-another-redirect-uri). 4. [Connect Azure DevOps to your new app](#connect-azure-devops-to-your-new-app). 5. [Add your Entra ID app to dbt](#add-your-azure-ad-app-to-dbt-cloud). Once the Microsoft Entra ID app is added to dbt, an account admin must also [connect a service user](#connect-a-service-user) using OAuth, which will be used to power headless actions in dbt such as deployment runs and CI. Once the Microsoft Entra ID app is added to dbt and the service user is connected, then dbt developers can personally authenticate in dbt from Azure DevOps. For more on this, see [Authenticate with Azure DevOps](https://docs.getdbt.com/docs/platform/git/authenticate-azure.md). The following personas are required to complete the steps on this page: * Microsoft Entra ID admin * Azure DevOps admin * dbt account admin * Azure admin (if your Entra ID and Azure DevOps environments are not connected) #### Register a Microsoft Entra ID app A Microsoft Entra ID admin needs to perform the following steps: 1. Sign into your Azure portal and click **Microsoft Entra ID**. 2. Select **App registrations** in the left panel. 3. Select **New registration**. The form for creating a new Entra ID app opens. 4. Provide a name for your app. We recommend using, "dbt Labs Azure DevOps app". 5. Select **Accounts in any organizational directory (Any Entra ID directory - Multitenant)** as the Supported Account Types. Many customers ask why they need to select Multitenant instead of Single tenant, and they frequently get this step wrong. Microsoft considers Azure DevOps (formerly called Visual Studio) and Microsoft Entra ID as separate tenants, and in order for this Entra ID application to work properly, you must select Multitenant. 6. Add a redirect URI. 1. Select **Web** as the platform. 2. In the field, enter `https://YOUR_ACCESS_URL/complete/azure_active_directory`. Make sure to replace `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. 7. Click **Register**. [![Navigating to the Entra ID app registrations](/img/docs/dbt-platform/connecting-azure-devops/ADnavigation.gif?v=2 "Navigating to the Entra ID app registrations")](#)Navigating to the Entra ID app registrations Here's what your app should look like before registering it: [![Registering a Microsoft Entra ID app](</img/docs/dbt-platform/connecting-azure-devops/AD app.png?v=2> "Registering a Microsoft Entra ID app")](#)Registering a Microsoft Entra ID app #### Add permissions to your new app An Entra ID admin needs to provide your new app access to Azure DevOps: 1. Select **API permissions** in the left navigation panel. 2. Remove the **Microsoft Graph / User Read** permission. 3. Click **Add a permission**. 4. Select **Azure DevOps**. 5. Select the **user\_impersonation** permission. This is the only permission available for Azure DevOps. [![Adding permissions to the app](/img/docs/dbt-platform/connecting-azure-devops/user-impersonation.gif?v=2 "Adding permissions to the app")](#)Adding permissions to the app #### Add another redirect URI A Microsoft Entra ID admin needs to add another redirect URI to your Entra ID application. This redirect URI will be used to authenticate the service user for headless actions in deployment environments. Before adding another redirect URI, make sure you selected **Web** as the platform when you [registered the Microsoft Entra ID app](#register-a-microsoft-entra-id-app). 1. Navigate to your Microsoft Entra ID application. 2. Select the link next to **Redirect URIs**. 3. Click **Add URI** and add the URI, replacing `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/complete/azure_active_directory_service_user` 4. Click **Save**. [![Adding the Service User redirect URI](/img/docs/dbt-platform/connecting-azure-devops/redirect-uri.gif?v=2 "Adding the Service User redirect URI")](#)Adding the Service User redirect URI #### Create a client secret A Microsoft Entra ID admin needs to complete the following steps: 1. Navigate to your Microsoft Entra ID application. 2. Select **Certificates and Secrets** from the left navigation panel. 3. Select **Client secrets** and click **New client secret** 4. Give the secret a description and select the expiration time. Click **Add**. 5. Copy the **Value** field and securely share it with the dbt account admin who will complete the setup. #### Connect Azure DevOps to your new app An Azure admin will need one of the following permissions in both the Microsoft Entra ID and Azure DevOps environments: * Azure Service Administrator * Azure Co-administrator If your Azure DevOps account is connected to Entra ID, then you can proceed to [Set up a service user](#set-up-a-service-user). However, if you're just getting set up, connect Azure DevOps to the Microsoft Entra ID app you just created: 1. From your Azure DevOps account, select **Organization settings** in the bottom left. 2. Navigate to Microsoft Entra ID. 3. Click **Connect directory**. 4. Select the directory you want to connect. 5. Click **Connect**. [![Connecting Azure DevOps and Microsoft Entra ID](</img/docs/dbt-platform/connecting-azure-devops/connect AD to Azure DevOps.gif?v=2> "Connecting Azure DevOps and Microsoft Entra ID")](#)Connecting Azure DevOps and Microsoft Entra ID #### Add your Microsoft Entra ID app to dbt A dbt account admin needs to perform the following steps. Once you connect your Microsoft Entra ID app and Azure DevOps, you need to provide dbt information about the app: 1. Navigate to your account settings in dbt. 2. Select **Integrations**. 3. Scroll to the Azure DevOps section and click on the pencil icon to edit the integration. 4. Complete the form: * **Azure DevOps Organization:** Must match the name of your Azure DevOps organization exactly. Do not include the `dev.azure.com/` prefix in this field. ✅ Use `my-devops-org` ❌ Avoid `dev.azure.com/my-devops-org` * **Application (client) ID:** Found in the Microsoft Entra ID app. * **Client Secrets:** Copy the **Value** field in the Microsoft Entra ID app client secrets and paste it in the **Client Secret** field in dbt. Entra ID admins are responsible for the Entra ID app secret expiration and dbt Admins should note the expiration date for rotation. * **Directory(tenant) ID:** Found in the Microsoft Entra ID app. [![Adding a Microsoft Entra ID app to dbt](/img/docs/dbt-platform/connecting-azure-devops/AzureDevopsAppdbtCloud.gif?v=2 "Adding a Microsoft Entra ID app to dbt")](#)Adding a Microsoft Entra ID app to dbt Your Microsoft Entra ID app should now be added to your dbt Account. People on your team who want to develop in the Studio IDE or dbt CLI can now personally [authorize Azure DevOps from their profiles](https://docs.getdbt.com/docs/platform/git/authenticate-azure.md). #### Set up a service user A service user is a pseudo user set up in the same way an admin would set up a real user, but it's given permissions specifically scoped for service to service interactions. You should avoid linking authentication to a real Azure DevOps user because if this person leaves your organization, dbt will lose privileges to the dbt Azure DevOps repositories, causing production runs to fail. Service user authentication expiration dbt will refresh the authentication for the service user on each run triggered by the scheduler, API, or CI. If your account does not have any active runs for over 90 days, an admin will need to manually refresh the authentication of the service user by disconnecting and reconnecting the service user's profile via the OAuth flow described above in order to resume headless interactions like project set up, deployment runs, and CI. If the Studio IDE fails to start and displays `Failed to start server. The project is not fully set up: Azure Service User not found`, the account-level service user isn't connected. Refer to [Connect a service user](#connect-a-service-user) for more details. If it displays `Missing Azure user` instead, a developer's own Azure DevOps account isn't linked to dbt. Refer to [Authenticate with Azure DevOps](https://docs.getdbt.com/docs/platform/git/authenticate-azure.md#troubleshooting) for more details. ##### Service users permissions A service user account must have the following Azure DevOps permissions for all Azure DevOps projects and repos you want accessible in dbt. Read more about how dbt uses each permission in the following paragraphs. * **Project Reader** * **ViewSubscriptions** * **EditSubscriptions** * **DeleteSubscriptions** \* * **PullRequestContribute** * **GenericContribute** \* Note: **DeleteSubscriptions** permission might be included in **EditSubscriptions** depending on your version of Azure. Some of these permissions are only accessible via the [Azure DevOps API](https://docs.microsoft.com/en-us/azure/devops/organizations/security/namespace-reference?view=azure-devops) or [CLI](https://learn.microsoft.com/en-us/cli/azure/devops?view=azure-cli-latest). We’ve also detailed more information on Azure DevOps API usage below to help accelerate the setup. Alternatively, you can use the Azure DevOps UI to enable permissions, but you cannot get the least permissioned set. ##### Required permissions for service users The service user's permissions will also power which repositories a team can select from during dbt project set up, so an Azure DevOps admin must grant at minimum Project Reader access to the service user *before* creating a new project in dbt. If you are migrating an existing dbt project to use the native Azure DevOps integration, the dbt account's service user must have proper permissions on the repository before migration. ##### Turn off MFA for service user While it's common to enforce multi-factor authentication (MFA) for normal user accounts, service user authentication must not need an extra factor. If you enable a second factor for the service user, this can interrupt production runs and cause a failure to clone the repository. In order for the OAuth access token to work, the best practice is to remove any more burden of proof of identity for service users. As a result, MFA must be explicitly disabled in the Office 365 or Microsoft Entra ID administration panel for the service user. Just having it "un-connected" will not be sufficient, as dbt will be prompted to set up MFA instead of allowing the credentials to be used as intended. **To disable MFA for a single user using the Office 365 Administration console:** * Go to Microsoft 365 admin center -> Users -> Active users -> Select the user -> Manage multifactor authentication -> Select the user -> Disable multi-factor authentication. **To use the Microsoft Entra ID interface:** Note, this procedure involves disabling Security Defaults in your Entra ID environment. 1. Go to the Azure Admin Center. Open Microsoft Entra ID and under the **Manage** section of the left navigation, click **Properties**, scroll down to **Manage Security defaults**, and then select **No** in "Enable Security Defaults" and click **Save**. 2. Go to **Microsoft Entra ID** -> Manage -> Users ->click on the ellipsis (...) and then the **Multi-Factor Authentication** link. If the link is grayed out, you need to make sure you disable **Security Defaults** from the previous step. 3. If MFA is enabled for users, select the user(s) and select **Disable** under **Quick steps**. 4. Select **Yes** to confirm your changes. To re-enable MFA for the user, select them again and click **Enable**. Note, you may have to go through MFA setup for that user after enabling it. **ViewSubscriptions** <br /> **Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 **Namespace:** ServiceHooks **Permission:** ```json { "bit": 1, "displayName": "View Subscriptions", "name": "ViewSubscriptions" } ``` **Uses:** To view existing Azure DevOps service hooks subscriptions **Token (where applicable - API only):** * PublisherSecurity for access to all projects * PublisherSecurity/\<azure\_devops\_project\_object\_id> for per project access **UI/API/CLI:** API/CLI only **Sample CLI code snippet** ```bash az devops security permission update --organization https://dev.azure.com/<org_name> --namespace-id cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 --subject <service_account>@xxxxxx.onmicrosoft.com --token PublisherSecurity/<azure_devops_project_object_id> --allow-bit 1 ``` **EditSubscriptions** <br /> **Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 **Namespace:** ServiceHooks **Permission:** ```json { "bit": 2, "displayName": "Edit Subscription", "name": "EditSubscriptions" } ``` **Uses:** To add or update existing Azure DevOps service hooks subscriptions **Token (where applicable - API only):** * PublisherSecurity for access to all projects * PublisherSecurity/\<azure\_devops\_project\_object\_id> for per project access **UI/API/CLI:** API/CLI only **Sample CLI code snippet** ```bash az devops security permission update --organization https://dev.azure.com/<org_name> --namespace-id cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 --subject <service_account>@xxxxxx.onmicrosoft.com --token PublisherSecurity/<azure_devops_project_object_id> --allow-bit 2 ``` **DeleteSubscriptions** <br /> **Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 **Namespace:** ServiceHooks **Permission:** ```json { "bit": 4, "displayName": "Delete Subscriptions", "name": "DeleteSubscriptions" } ``` **Uses:** To delete any redundant Azure DevOps service hooks subscriptions **Token (where applicable - API only):** * PublisherSecurity for access to all projects * PublisherSecurity/\<azure\_devops\_project\_object\_id> for per project access **UI/API/CLI:** API/CLI only **Sample CLI code snippet** ```bash az devops security permission update --organization https://dev.azure.com/<org_name> --namespace-id cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 --subject <service_account>@xxxxxx.onmicrosoft.com --token PublisherSecurity/<azure_devops_project_object_id> --allow-bit 4 ``` **Additional Notes:** This permission has been deprecated in recent Azure DevOps versions. Edit Subscriptions (bit 2) has Delete permissions. **PullRequestContribute** <br /> **Security Namespace ID:** 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 **Namespace:** Git Repositories **Permission:** ```json { "bit": 16384, "displayName": "Contribute to pull requests", "name": "PullRequestContribute" } ``` **Uses:** To post Pull Request statuses to Azure DevOps **Token (where applicable - API only):** * repoV2 for access to all projects * repoV2/\<azure\_devops\_project\_object\_id> for per project access * repoV2/\<azure\_devops\_project\_object\_id>/\<azure\_devops\_repository\_object\_id> for per repo access **UI/API/CLI:** UI, API, and CLI **Sample CLI code snippet** ```bash az devops security permission update --organization https://dev.azure.com/<org_name> --namespace-id 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 --subject <service_account>@xxxxxx.onmicrosoft.com --token repoV2/<azure_devops_project_object_id>/<azure_devops_repository_object_id> --allow-bit 16384 ``` **Additional Notes:** This permission is automatically inherited if Project Reader/Contributor/Administrator is set in the UI. **GenericContribute** <br /> **Security Namespace ID:** 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 **Namespace:** Git Repositories **Permission:** ```json { "bit": 4, "displayName": "Contribute", "name": "GenericContribute" } ``` **Uses:** To post commit statuses to Azure DevOps **Token (where applicable - API only):** * repoV2 for access to all projects * repoV2/\<azure\_devops\_project\_object\_id> for access to a single project at a time * repoV2/\<azure\_devops\_project\_object\_id>/\<azure\_devops\_repository\_object\_id> for access to a single repo at a time **UI/API/CLI:** UI, API, and CLI **Sample CLI code snippet** ```bash az devops security permission update --organization https://dev.azure.com/<org_name> --namespace-id 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 --subject <service_account>@xxxxxx.onmicrosoft.com --token repoV2/<azure_devops_project_object_id>/<azure_devops_repository_object_id> --allow-bit 4 ``` **Additional Notes:** This permission is automatically inherited if Project Contributor/Administrator is set in the UI. ##### Connect a service user You must connect your service user before setting up a dbt project, as the service user's permissions determine which projects dbt can import. A dbt account admin with access to the service user's Azure DevOps account must complete the following to connect the service user: 1. Sign in to the service user's Azure DevOps account. 2. In dbt, go to **Account settings** > **Integrations**. 3. Go to the **Azure DevOps** section and select **Service User**. 4. Enter values for the required fields. 5. Click **Save**. 6. Click **Link Azure service user**. 7. You will be directed to Azure DevOps and you must accept the Microsoft Entra ID app's permissions. 8. Finally, you will be redirected to dbt, and the service user will be connected. [![Connecting an Azure Service User](/img/docs/dbt-platform/connecting-azure-devops/azure-service-user.png?v=2 "Connecting an Azure Service User")](#)Connecting an Azure Service User Once connected, dbt displays the email address of the service user so you know which user's permissions are enabling headless actions in deployment environments. To change which account is connected, disconnect the profile in dbt, sign into the alternative Azure DevOps service account, and re-link the account in dbt. Personal Access Tokens (PATs) dbt leverages the service user to generate temporary access tokens called [PATs](https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?toc=%2Fazure%2Fdevops%2Fmarketplace-extensibility%2Ftoc.json\&view=azure-devops\&tabs=Windows). These tokens are limited in scope, are only valid for 5 minutes, and become invalid after a single API call. These tokens are limited to the following [scopes](https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/oauth?view=azure-devops): * `vso.code_full`: Grants full access to source code and version control metadata (commits, branches, and so on). Also grants the ability to create and manage code repositories, create and manage pull requests and code reviews, and receive notifications about version control events with service hooks. Also includes limited support for Client OM APIs. * `vso.project`: Grants the ability to read projects and teams. * `vso.build_execute`: Grants the ability to access build artifacts, including build results, definitions, and requests, and the ability to queue a build, update build properties, and the ability to receive notifications about build events with service hooks. --- ##### Version control basics dbt platform When you develop in the command line interface (CLI) or Cloud integrated development environment (Studio IDE), you can leverage Git directly to version control your code. To use version control, make sure you are connected to a Git repository in the CLI or Studio IDE. You can create a separate branch to develop and make changes. The changes you make aren’t merged into the default branch in your connected repository (typically named the `main` branch) unless it successfully passes tests. This helps keep the code organized and improves productivity by making the development process smooth. You can read more about git terminology below and also check out [GitHub Docs](https://docs.github.com/en) as well. #### Git overview Check out some common git terms below that you might encounter when developing: | Name | Definition | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Repository or repo | A repository is a directory that stores all the files, folders, and content needed for your project. You can think of this as an object database of the project, storing everything from the files themselves to the versions of those files, commits, and deletions. Repositories are not limited by user and can be shared and copied. | | Branch | A branch is a parallel version of a repository. It is contained within the repository but does not affect the primary or main branch allowing you to work freely without disrupting the live version. When you've made the changes you want to make, you can merge your branch back into the main branch to publish your changes | | Checkout | The `checkout` command is used to create a new branch, change your current working branch to a different branch, or switch to a different version of a file from a different branch. | | Commit | A commit is a user’s change to a file (or set of files). When you make a commit to save your work, Git creates a unique ID that allows you to keep a record of the specific changes committed along with who made them and when. Commits usually contain a commit message which is a brief description of what changes were made. | | main | The primary, base branch of all repositories. All committed and accepted changes should be on the main branch.<br /><br />In the Studio IDE, the main branch is protected. This means you can't directly edit, format, or lint files, and execute dbt commands in your protected primary git branch when using the dbt platform user interface. Keep in mind that all Studio IDE Git activity is subject to the permissions of your configured credentials, and the rules configured at the remote Git provider (for example, GitHub or GitLab branch protection). Since the Studio IDE prevents commits to the protected branch, you can commit those changes to a new branch. | | Merge | Merge takes the changes from one branch and adds them into another (usually main) branch. These commits are usually first requested via pull request before being merged by a maintainer. | | Pull Request | If someone has changed code on a separate branch of a project and wants it to be reviewed to add to the main branch, they can submit a pull request. Pull requests ask the repo maintainers to review the commits made, and then, if acceptable, merge the changes upstream. A pull happens when adding the changes to the main branch. | | Push | A `push` updates a remote branch with the commits made to the current branch. You are literally *pushing* your changes into the remote. | | Remote | This is the version of a repository or branch that is hosted on a server. Remote versions can be connected to local clones so that changes can be synced. | #### The git button in the Cloud IDE You can perform git tasks with the git button in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). The following are descriptions of each git button command and what they do: | Name | Actions | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Abort merge | This option allows you to cancel a merge that had conflicts. Be careful with this action because all changes will be reset and this operation can't be reverted, so make sure to commit or save all your changes before you start a merge. | | Change branch | This option allows you to change between branches (checkout). | | Commit | A commit is an individual change to a file (or set of files). When you make a commit to save your work, Git creates a unique ID (a.k.a. the "SHA" or "hash") that allows you to keep record of the specific changes committed along with who made them and when. Commits usually contain a commit message which is a brief description of what changes were made. When you make changes to your code in the future, you'll need to commit them as well. | | Create new branch | This allows you to branch off of your base branch and edit your project. You’ll notice after initializing your project that the main branch is protected.<br /><br />This means you can directly edit, format, or lint files and execute dbt commands in your protected primary git branch. When ready, you can commit those changes to a new branch. | | Initialize your project | This is done when first setting up your project. Initializing a project creates all required directories and files within an empty repository by using the dbt starter project.<br />Note: This option will not display if your repo isn't completely empty (i.e. includes a README file).<br />Once you click **Initialize your project**, click **Commit** to finish setting up your project. | | Open pull request | This allows you to open a pull request in Git for peers to review changes before merging into the base branch. | | Pull changes from main | This option is available if you are on any local branch that is behind the remote version of the base branch or the remote version of the branch that you're currently on. | | Pull from remote | This option is available if you’re on the local base branch and changes have recently been pushed to the remote version of the branch. Pulling in changes from the remote repo allows you to pull in the most recent version of the base branch. | | Rollback to remote | Reset changes to your repository directly from the Studio IDE. You can rollback your repository back to an earlier clone from your remote. To do this, click on the three dot ellipsis in the bottom right-hand side of the Studio IDE and select **Rollback to remote**. | | Refresh git state | This enables you to pull new branches from a different remote branch to your local branch with just one command. | #### Merge conflicts Merge conflicts often occur when multiple users are concurrently making edits to the same section in the same file. This makes it difficult for Git to determine which change should be kept. Refer to [merge conflicts](https://docs.getdbt.com/docs/platform/git/merge-conflicts.md) to learn how to resolve merge conflicts. #### The .gitignore file dbt implements a global [`.gitignore file`](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) that automatically excludes the following sub-folders from your git repository to ensure smooth operation: ```text dbt_packages/ logs/ target/ ``` This inclusion uses a trailing slash, making these lines in the `.gitignore` file act as 'folder wildcards' that prevent any files or folders within them from being tracked by git. You can also specify additional exclusions as needed for your project. However, this global `.gitignore` *does not* apply to dbt v1 and dbt platform CLI users directly. Therefore, if you're working with dbt v1 or dbt platform CLI, you need to manually add the three lines mentioned previously to your project's `.gitignore` file. It's worth noting that while some git providers generate a basic `.gitignore` file when the repository is created, these often lack the necessary exclusions for dbt. This means it's important to ensure you add the three lines mentioned previously in your `.gitignore` to ensure dbt operates smoothly. note * **dbt projects created after Dec 1, 2022** — If you use the **Initialize dbt Project** button in the Studio IDE to set up a new and empty dbt project, dbt will automatically add a `.gitignore` file with the required entries. If a `.gitignore` file already exists, the necessary folders will be appended to the existing file. * **Migrating project from dbt v1 to dbt** — Make sure you check the `.gitignore` file contains the necessary entries. dbt v1 doesn't interact with git so dbt doesn't automatically add or verify entries in the `.gitignore` file. Additionally, if the repository already contains dbt code and doesn't require initialization, dbt won't add any missing entries to the .gitignore file. For additional info or troubleshooting tips please refer to the [detailed FAQ](https://docs.getdbt.com/faqs/Git/gitignore.md). --- #### Manage access ##### About environment-level permissions dbt platform Environment-level permissions give dbt admins the ability to grant write permission to groups and service tokens for specific [environment types](https://docs.getdbt.com/docs/dbt-platform-environments.md) within a project. Granting access to an environment gives users access to all environment-level write actions and resources associated with their assigned roles. For example, users with a Developer role can create and run jobs within the environment(s) they have access to. For all other environments, those same users will have read-only access. You can configure multiple rows for the same permission set within a group when each row has different project scope and/or environment write access. For configuration instructions, check out the [setup page](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md). #### Current limitations Environment-level permissions give dbt admins more flexibility to protect their environments, but it's important to understand that there are some limitations to this feature, so those admins can make informed decisions about granting access. * Environment-level permissions do not allow you to create custom roles and permissions for each resource type in dbt. * You can only select environment types, and can’t specify a particular environment within a project. * You can't select specific resources within environments. dbt jobs and runs are environment resources. * For example, you can't specify that a user only has access to jobs but not runs. Access to a given environment gives the user access to everything within that environment. #### Environments and roles dbt has four different environment types per project: * **Production** — Primary deployment environment. Only one unique Production env per project. * **Development** — Developer testing environment. Only one unique Development env per project. * **Staging** — Pre-prod environment that sits between development and production. Only one unique Staging env per project. * **General** — Mixed use environments. No limit on the number per project. ##### Environment write permissions Environment write permissions grant access to create, edit, and delete runs and jobs within an environment. However, they don't grant users access to create or delete environments themselves. See [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information about elevated permission sets. Environment write permissions can be specified for the following roles: * Analyst * Database admin * Developer\* * Git admin * Job creator\*\* * Team admin \* In the [enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) table, the `Developer` role has read-only access to environment settings, but when added to a group, the **Environment write access** field defaults to `All environments`, which grants broader environment permissions than the table implies. The `Analyst`, `Database admin`, `Git admin`, and `Team admin` roles get **Read-only environment access** by default unless you specify different access under **Environment write access** for that group. \*\* [Job creator](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-creator) has read-only access to environments in the enterprise permissions table. Users can create, edit, and run jobs only in environment types where the group has **Environment write access**. Refer to [Set up environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md). Depending on your current group mappings, you may have to update roles to ensure users have the correct access level to environments. Determine what personas need updated environment access and the roles they should be mapped to. The personas below highlight a few scenarios for environment permissions: * **Developer** — Write access to create/run jobs in non-production environments * **Testing/QA** — Write access to staging and development environments to test * **Production deployment** — Write access to all environments, including production, for deploying * **Analyst** — Doesn't need environmental write access but read-only access for discovery and troubleshooting * **Other admins** — These admins may need write access to create/run jobs or configure integrations for any number of environments #### Projects and environments Environment-level permissions can be enforced over one or multiple projects with mixed access to the environments themselves. ##### Single project environments If you’re working with a single project, we recommend restricting access to the Production environment and ensuring groups have access to Development, Staging, or General environments where they can safely create and run jobs. The following is an example of how the personas could be mapped to roles: * **Developer:** Developer role with write access to Development and General environments * **Testing/QA:** Developer role with write access to Development, Staging, and General environments * **Production Deployment:** Developer role with write access to all environments or Job Admin which has access to all environments by default. * **Analyst:** Analyst role with no write access and read-only access to environments. * **Other Admins:** Depends on the admin needs. For example, if they are managing the production deployment grant access to all environments. ##### Multiple projects Let's say Acme corp has 12 projects and 3 of them belong to Finance, 3 belong to Marketing, 4 belong to Manufacturing, and 2 belong to Technology. With mixed access across projects: * **Developer:** If the user has the Developer role and has access to Projects A, B, C, then they only need access to Dev and General environments. * **Testing/QA:** If they have the Developer role and they have access to Projects A, B, C, then they only need access to Development, Staging, and General environments. * **Production Deployment:** If the user has the Admin *or* Developer role *and* they have access to Projects A, B, C, then they need access to all Environments. * **Analyst:** If the user has the Analyst role, then the need *no* write access to *any environment*. * **Other Admins:** A user (non-Admin) can have access to multiple projects depending on the requirements. If the user has the same roles across projects, you can apply environment access across all projects. #### Related docs * [Environment-level permissions setup](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md) --- ##### About user access "User access" is not "Model access" This page covers user groups and access, including: * User licenses, permissions, and group memberships * Role-based access controls for projects and environments * Single sign-on, and secure authentication For model-specific access and their availability across projects, refer to [Model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md). dbt platform You can regulate access to dbt by various measures, including licenses, groups, permissions, and role-based access control (RBAC). To understand the possible approaches to user access to dbt features and functionality, you should first know how we approach users and groups. #### Users Individual users in dbt can be people you [manually invite](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) or grant access via an external identity provider (IdP), such as Microsoft Entra ID, Okta, or Google Workspace. In either scenario, when you add a user to dbt, they are assigned a [license](#licenses). You assign licenses at the individual user or group levels. When you manually invite a user, you will assign the license in the invitation window. [![Example of the license dropdown in the user invitation window.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/license-dropdown.png?v=2 "Example of the license dropdown in the user invitation window.")](#)Example of the license dropdown in the user invitation window. You can edit an existing user's license by navigating to the **Users** section of the **Account settings**, clicking on a user, and clicking **Edit** on the user pane. Delete users from this same window to free up licenses for new users. [![Example of the user information window in the user directory](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/edit-user.png?v=2 "Example of the user information window in the user directory")](#)Example of the user information window in the user directory ##### User passwords By default, new users will be prompted to set a password for their account. All plan tiers support and enforce [multi-factor authentication](https://docs.getdbt.com/docs/platform/manage-access/mfa.md) for users with password logins. However, they will still need to configure their password before configuring MFA. Enterprise tier accounts can configure [SSO](#sso-mappings) and advanced authentication measures. Developer and Starter plans only support user passwords with MFA. User passwords must meet the following criteria: * Be at least nine characters in length * Contain at least one uppercase and one lowercase letter * Contain at least one number 0-9 * Contain at least one special character #### Groups Groups in dbt serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignments of permissions easier. The permissions available depends on whether you're on an [Enterprise-tier](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) or [self-service Starter](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md) plan. * Admins use groups in dbt to assign [licenses](#licenses) and [permissions](#permissions). * The permissions are more granular than licenses, and you only assign them at the group level; *you can’t assign permissions at the user level.* * Every user in dbt must be assigned to at least one group. There are three default groups available as soon as you create your dbt account (the person who created the account is added to all three automatically): * **Owner:** This group is for individuals responsible for the entire account and will give them elevated account admin privileges. You cannot change the permissions. * **Member:** This group is for the general members of your organization. Default permissions are broad, restricting only access to features that can alter billing or security. By default, dbt adds new users to this group. * **Everyone:** A general group for all members of your organization. Customize the permissions to fit your organizational needs. By default, dbt adds new users to this group. default group permissions The `Owner` and `Member` groups have default permission sets: * **Starter plan:** The `Owner` and `Member` groups use the `Owner` and `Member` [permission sets](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md#table-of-groups-licenses-and-permissions), respectively. * **Enterprise plans:** By default, dbt assigns the `Owner` group an [`Account Admin`](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) permission set, and the `Member` group a `Member` permission set, which doesn't appear in the settings, but has the same privileges as the [`Admin`](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#admin) permission set. Default groups are automatically provisioned for all accounts to simplify the initial set up. We recommend creating your own organizational groups so you can customize the permissions. Once you create your own groups, you can delete the default groups. ##### Create new groups [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Create new groups from the **Groups & Licenses** section of the **Account settings**. * If you use an external IdP for SSO, you can sync those SSO groups to dbt from the **Group details** pane when creating or editing existing groups. [![Example the new group pane in the account settings.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/new-group.png?v=2 "Example the new group pane in the account settings.")](#)Example the new group pane in the account settings. important If a user is assigned licenses and permissions from multiple groups, the group that grants the most access will take precedence. You must assign a permission set to any groups created beyond the three defaults, or users assigned will not have access to features beyond their user profile. ##### Group access and permissions The **Access & Permissions** section of a group is where you can assign users the right level of access based on their role or responsibilities. You decide: * Projects the group can access * Roles that the group members are assigned for each * Environments the group can edit This setup provides you with the flexibility to determine the level of access users in any given group will have. For example, you might allow one group of analysts to edit jobs in their project, but only let them view related projects, or you could grant admin-level access to a team that owns a specific project while keeping others restricted to read-only. [![Assign a variety of roles and access permissions to user groups.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/sample-access-policy.png?v=2 "Assign a variety of roles and access permissions to user groups.")](#)Assign a variety of roles and access permissions to user groups. ###### Environment write access Some permission sets grant users read-only access to environment settings that can be overridden if you assign them to a group with **Environment write access**. They will then be able to create, edit, and delete environment settings such as jobs and runs, bypassing the read-only restriction. This elevated access doesn't grant users the ability to create or delete environments. In the following example, the `analyst` permission set, which by default has read-only access to jobs, is assigned to the group across all projects; however, the **Environment write access** is set to `All Environments`. This grants all users in this group the ability to create, edit, and delete jobs across all environments and projects. [![Users assigned environment write access will be able to edit environment settings.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/environment-write.png?v=2 "Users assigned environment write access will be able to edit environment settings.")](#)Users assigned environment write access will be able to edit environment settings. Only use **Environment write access** settings when you intend to grant users the ability to edit environments. To grant users only the permissions inherent to their set, leave this setting blank (all boxes unchecked). ##### SSO mappings [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") SSO Mappings connect an identity provider (IdP) group membership to a dbt group. When users log into dbt via a supported identity provider, their IdP group memberships sync with dbt. Upon logging in successfully, the user's group memberships (and permissions) will automatically adjust within dbt. Creating SSO Mappings While dbt supports mapping multiple IdP groups to a single dbt group, we recommend using a 1:1 mapping to make administration as simple as possible. Use the same names for your dbt groups and your IdP groups. Create an SSO mapping in the group view: 1. Open an existing group to edit or create a new group. 2. In the **SSO** portion of the group screen, enter the name of the SSO group exactly as it appears in the IdP. If the name is not the same, the users will not be properly placed into the group. 3. In the **Users** section, ensure the **Add all users by default** option is disabled. 4. Save the group configuration. New SSO users will be added to the group upon login, and existing users will be added to the group upon their next login. [![Example of an SSO group mapped to a dbt group.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/sso-mapping.png?v=2 "Example of an SSO group mapped to a dbt group.")](#)Example of an SSO group mapped to a dbt group. Refer to [role-based access control](#role-based-access-control) for more information about mapping SSO groups for user assignment to dbt groups. #### Grant access dbt users have both a license (assigned to an individual user or by group membership) and permissions (by group membership only) that determine what actions they can take. Licenses are account-wide, and permissions provide more granular access or restrictions to specific features. ##### Licenses Every user in dbt will have a license assigned. Licenses consume "seats" which impact how your account is [billed](https://docs.getdbt.com/docs/platform/billing.md), depending on your [service plan](https://www.getdbt.com/pricing). There are four license types in dbt: * **Analyst**\* — Available on [Enterprise and Enterprise+ plans only](https://www.getdbt.com/pricing). * User can be granted *any* permission sets. * \* The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. * **Developer** — User can be granted *any* permission sets. * **IT** — Available on [Starter, Enterprise, and Enterprise+ plans only](https://www.getdbt.com/pricing). User has Security Admin and Billing Admin [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#permission-sets) applied, as well as permissions to edit **Connections** in the **Account settings** page. * Can manage users, groups, connections, and licenses, among other permissions. * *IT licensed users do not inherit rights from any permission sets*. * Every IT licensed user has the same access across the account, regardless of the group permissions assigned. * **Read-Only** — Available on [Starter, Enterprise, and Enterprise+ plans only](https://www.getdbt.com/pricing). * User has read-only permissions applied to all dbt resources. * Intended to view the [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) and the [deploy](https://docs.getdbt.com/docs/deploy/deployments.md) section (jobs, runs, schedules) in a dbt account, but can’t make changes. * *Read-only licensed users do not inherit rights from any permission sets*. * Every read-only licensed user has the same access across the account, regardless of the group permissions assigned. \* The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. Developer licenses will make up a majority of the users in your environment and have the highest impact on billing, so it's important to monitor how many you have at any given time. For more information on these license types, see [Seats & Users](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) License types override group permissions **User license types always override their assigned group permission sets.** For example, a user with a Read-Only license cannot perform administrative actions, even if they belong to an Account Admin group. This ensures that license restrictions are always enforced, regardless of group membership. ###### Enable granular permissions for Read-Only users [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Granular permissions let you restrict which projects Read-Only users can access, allowing you to set permissions by group rather than applying the default permissions for Read-Only licenses. Note that new accounts don't need this setting as Read-Only users on new accounts get granular permissions by default. Key things to know before enabling: * This setting is only available on Enterprise and Enterprise+ plans. * Access stays Read-Only: This setting controls access to projects, not permissions (users remain Read-Only). * It's permanent: Enabling this setting is a one-time, irreversible change. * Prepare first: Read-Only users keep their project access only if they're in a group with a Read-Only permission set that covers all projects. If a user isn't in a group, or their group's permission set only covers some projects, they'll lose access to the projects that aren't covered. To turn on granular permissions: 1. Go to **Account settings** → **Groups & Licenses**. 2. Click **Enable granular permissions**. 3. Select the checkbox acknowledging that this setting can't be reversed once enabled. 4. Click **Enable** to confirm, or **Cancel** to go back without making changes. ##### Permissions Permissions determine what users can do in your dbt account. By default, members of the `Owner` and `Member` groups have full access to all areas and features. When you want to restrict access to features, assign users to groups with stricter permission sets. Keep in mind that if a user belongs to multiple groups, the most permissive group will take precedence. The permissions available depend on whether you're on an [Enterprise, Enterprise+](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md), or [self-service Starter](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md) plan. Developer accounts only have a single user, so permissions aren't applicable. Some access to user settings (for example, **Credentials** settings in **Your profile**) can be granted with additional permissions (such as `user_credential_write`). Refer to [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information. [![Example permissions dropdown while editing an existing group.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/assign-group-permissions.png?v=2 "Example permissions dropdown while editing an existing group.")](#)Example permissions dropdown while editing an existing group. Some permissions (those that don't grant full access, like admins) allow groups to be "assigned" to specific projects and environments only. Read about [environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md) for more information on restricting environment access. [![Example environment access control for a group with Git admin assigned.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/environment-access-control.png?v=2 "Example environment access control for a group with Git admin assigned.")](#)Example environment access control for a group with Git admin assigned. ##### Set up read-only user access To give users read-only access to analyze dbt models and project resources, assign them the [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set through a group. Users won't have access until they're added to a group that's assigned the permission set. Availability The OAuth integration that lets read-only users connect to analysis features (such as the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md)) is available to use. **1. Create a group with the Analyst read permission set** You can also add the Analyst read permission set to an existing group. 1. Go to **Account settings** → **Groups & Licenses**. 2. Give the group a descriptive name. 3. Click **Add permission** and select the **Analyst read** permission set. 4. Select the projects the permission set should apply to. **All projects** is the default option. 5. Click **Save**. **2. Assign users the read-only license and add them to the group** You can skip or automate this step if license mapping and group mapping are enabled through SSO or SCIM. Use this flow to test with a single user. 1. Go to **Account settings** → **Users**. 2. Select the user you want to add to the group. 3. Select the group you want to add the user to and click **Save**. [![Assign a user the read-only license and add them to the group](/img/docs/dbt-platform/access-control/analyst-read-permission.png?v=2 "Assign a user the read-only license and add them to the group")](#)Assign a user the read-only license and add them to the group For more information, refer to [Grant access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#grant-access). #### Role-based access control [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Role-based access control (RBAC) allows you to grant users access to features and functionality based on their group membership. With this method, you can grant users varying access levels to different projects and environments. You can take access and security to the next level by integrating dbt with a third-party identity provider (IdP) to grant users access when they authenticate with your SSO or OAuth service. There are a few things you need to know before you configure RBAC for SSO users: * New SSO users join any groups with the **Add all new users by default** option enabled. By default, the `Everyone` and `Member` groups have this option enabled. Disable this option across all groups for the best RBAC experience. * You must have the appropriate SSO groups configured in the group details SSO section. If the SSO group name does not match *exactly*, users will not be placed in the group correctly. [![The Group details SSO section with a group configured.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/sso-window-details.png?v=2 "The Group details SSO section with a group configured.")](#)The Group details SSO section with a group configured. * dbt Labs recommends that your dbt group names match the IdP group names. Let's say you have a new employee being onboarded into your organization using [Okta](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-okta.md) as the IdP and dbt groups with SSO mappings. In this scenario, users are working on `The Big Project` and a new analyst named `Euclid Ean` is joining the group. Check out the following example configurations for an idea of how you can implement RBAC for your organization (these examples assume you have already configured [SSO](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md)):  Okta configuration You and your IdP team add `Euclid Ean` to your Okta environment and assign them to the `dbt` SSO app via a group called `The Big Project`. [![The user in the group in Okta.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/okta-group-config.png?v=2 "The user in the group in Okta.")](#)The user in the group in Okta. Configure the group attribute statements the `dbt` application in Okta. The group statements in the following example are set to the group name exactly (`The Big Project`), but yours will likely be a much broader configuration. Companies often use the same prefix across all dbt groups in their IdP. For example `DBT_GROUP_` [![Group attributes set in the dbt SAML 2.0 app in Okta.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/group-attributes.png?v=2 "Group attributes set in the dbt SAML 2.0 app in Okta.")](#)Group attributes set in the dbt SAML 2.0 app in Okta.  dbt configuration You and your dbt admin team configure the groups in your account's settings: 1. Navigate to the **Account settings** and click **Groups & Licenses** on the left-side menu. 2. Click **Create group** or select an existing group and click **Edit**. 3. Enter the group name in the **SSO** field. 4. Configure the **Access and permissions** fields to your needs. Select a [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md), the project they can access, and [environment-level access](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md). [![The group configuration with SSO field filled out in dbt.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/dbt-cloud-group-config.png?v=2 "The group configuration with SSO field filled out in dbt.")](#)The group configuration with SSO field filled out in dbt. Euclid is limited to the `Analyst` role, the `Jaffle Shop` project, and the `Development`, `Staging`, and `General` environments of that project. Euclid has no access to the `Production` environment in their role.  The user journey Euclid takes the following steps to log in: 1. Access the SSO URL or the dbt app in their Okta account. The URL can be found on the **SSO & SCIM** configuration page in the **Account settings**. [![The SSO login URL in the account settings.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/sso-login-url.png?v=2 "The SSO login URL in the account settings.")](#)The SSO login URL in the account settings. 2. Log in with their Okta credentials. [![The SSO login screen when using Okta as the identity provider.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/sso-login.png?v=2 "The SSO login screen when using Okta as the identity provider.")](#)The SSO login screen when using Okta as the identity provider. 3. Since it's their first time logging in with SSO, Euclid Ean is presented with a message and no option to move forward until they check the email address associated with their Okta account. [![The screen users see after their first SSO login.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/post-login-screen.png?v=2 "The screen users see after their first SSO login.")](#)The screen users see after their first SSO login. 4. They now open their email and click the link to join dbt Labs. [![The email the user receives on first SSO login.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/sample-email.png?v=2 "The email the user receives on first SSO login.")](#)The email the user receives on first SSO login. 5. Their email address is now verified. They click **Authenticate with your enterprise login**, which completes the process. [![The confirmation that the email address is verified.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/email-verified.png?v=2 "The confirmation that the email address is verified.")](#)The confirmation that the email address is verified. Euclid is now logged in to their account. They only have access to the `Jaffle Shop` project. Under **Orchestration**, they can configure user credentials. [![The Orchestration page with the environments.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/orchestration-environments.png?v=2 "The Orchestration page with the environments.")](#)The Orchestration page with the environments. The `Production` environment is visible, but it is `read-only`, and they have full access in the `Staging` environment. [![The Production environment landing page with read-only access.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/production-restricted.png?v=2 "The Production environment landing page with read-only access.")](#)The Production environment landing page with read-only access. [![The Staging environment landing page with full access.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/staging-access.png?v=2 "The Staging environment landing page with full access.")](#)The Staging environment landing page with full access. With RBAC configured, you now have granular control over user access to features across dbt. ##### SCIM license management As part of the SSO configuration for supported IdPs, you can also configure the [System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim.md) settings to add a layer of security to your user lifecycle management. As part of this process, you can integrate user license distribution into the user provisioning process through your IdP. See the [SCIM license management instructions](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md) for more information. #### FAQs  When are IdP group memberships updated for SSO Mapped groups? Group memberships are updated whenever a user logs into dbt via a supported SSO provider. If you've changed group memberships in your identity provider or dbt, ask your users to log back into dbt to synchronize these group memberships.  Can I set up SSO without RBAC? Yes. If a group has no [SSO mappings](#sso-mappings), it stays unmanaged — dbt won't change its membership when users log in through your IdP. Admins add and remove those users manually from the **Groups & Licenses** section of **Account settings**. Add an SSO mapping to an unmanaged group and it becomes managed: from then on, dbt adjusts membership at sign-in based on the user's IdP groups.  Can I configure a user's license type based on IdP attributes? Yes, see the docs on [managing license types](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#managing-license-types) for more information.  Why can't I edit a user's group membership? Don't try to edit your own user, as this isn't allowed for security reasons. You'll need a different user to make changes to your own user's group membership.  How do I add or remove users? Each dbt plan has a base number of Developer and Read-Only licenses. You can add or remove licenses by modifying the number of users in your account settings. * If you're on an Enterprise or Enterprise+ plan and have the correct [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md), you can add or remove developers by adjusting your developer user seat count in **Account settings** -> **Users**. * If you're on a Starter plan and have the correct [permissions](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md), you can add or remove developers by making two changes: adjust your developer user seat count AND your developer billing seat count in **Account settings** -> **Users** and then in **Account settings** -> **Billing**. For detailed steps, refer to [Users and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses). #### Learn more --- ##### Connect apps with OAuth Beta ### Connect apps with OAuth [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Starter, Enterprise, Enterprise+ The **App integrations** section in dbt platform lets admins manage OAuth 2.0 client registrations — a standard that lets external apps connect to dbt securely without sharing API tokens. Use it for: * AI tools that connect to the remote dbt MCP server. * Your own API integrations built against dbt platform APIs, when you want end users to sign in with their dbt identity instead of distributing tokens. * Third-party apps and IDE extensions that support OAuth for connecting to dbt. This feature is available to Account admins on Starter, Enterprise or Enterprise+ plans. To access this section, go to **Account settings** → **Integrations** → **App integrations**. [![App integrations page showing manually and dynamically registered OAuth clients](/img/docs/dbt-cloud/app-integrations-oauth.png?v=2 "App integrations page showing manually and dynamically registered OAuth clients")](#)App integrations page showing manually and dynamically registered OAuth clients #### Registration methods There are two ways you can register an app as an OAuth client: * [Dynamic registration](#dynamic-registration): Apps that support [Dynamic Client Registration (RFC 7591)](https://datatracker.ietf.org/doc/html/rfc7591) self-register automatically when a user connects them to dbt. No admin action is required. * [Manual registration](#manual-registration): For apps that don't support dynamic registration (for example, custom-built integrations), an admin can manually register an OAuth client. ##### Dynamic registration Apps that support [Dynamic Client Registration (RFC 7591)](https://datatracker.ietf.org/doc/html/rfc7591) self-register automatically when a user connects them to dbt. No admin action is required. Dynamically registered apps appear in the **Dynamically registered** table, which shows: | Column | Description | | ------------ | -------------------------------------------------- | | Name | The app name | | Redirect URL | The OAuth callback URL the app registered | | Created on | When the client was first registered | | Last used on | When the client last made an authenticated request | ##### Manual registration For apps that don't support dynamic registration (for example, Salesforce agent or custom-built integrations), an admin can manually register an OAuth client. To add a manually registered integration: 1. Go to **Account settings** → **Integrations** → **App integrations**. 2. Click **+ Add integration**. 3. Complete the registration form. Each manually registered client receives a unique client ID. Note that no client secret is issued, and the client must instead support the [PKCE extension (RFC 7636)](https://datatracker.ietf.org/doc/html/rfc7636). Manually registered clients appear in the **Manually registered** section. When configuring your client, use the following OAuth endpoints (replace `YOUR_PREFIX` with your dbt platform subdomain): | Endpoint | URL | | ------------- | --------------------------------------------- | | Authorization | `https://YOUR_PREFIX.dbt.com/oauth/authorize` | | Token | `https://YOUR_PREFIX.dbt.com/oauth/token` | #### Use with remote MCP When you connect an MCP client to the [remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp), it authenticates using OAuth. The client needs just your dbt platform MCP URL, which you can find under **Account settings** → **Access URLs** → **MCP Endpoint URL**. Clients that support dynamic registration complete the registration step automatically — you'll see them appear in the **Dynamically registered** table after first use. The following steps show how to connect dbt as a custom connector in Claude Desktop. The exact UI varies by tool, but the flow is the same: add a custom connector with your MCP URL, complete the OAuth consent flow, then connect. 1. In your AI tool, go to its connector settings and choose to add a custom connector (in Claude Desktop, go to **Chat → Customize → Connectors**, then click **Add custom connector**). 2. Enter a name (for example, `dbt`) and paste your dbt platform MCP URL (for example, `https://abc123.us1.dbt.com/api/ai/v1/mcp`), then click **Add**. [![Custom connector dialog showing the dbt MCP URL](/img/docs/dbt-cloud/oauth-add-custom-connector.png?v=2 "Custom connector dialog showing the dbt MCP URL")](#)Custom connector dialog showing the dbt MCP URL 3. Click **Connect**. The tool redirects you to dbt to complete the OAuth consent flow, where you can approve or deny individual [scopes](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent). [![OAuth consent screen showing requested scopes and project access](/img/docs/dbt-cloud/oauth-consent-screen.png?v=2 "OAuth consent screen showing requested scopes and project access")](#)OAuth consent screen showing requested scopes and project access 4. After you approve, the connector is added to the **Custom connectors** table and shows as connected. [![Adding a custom dbt connector in an AI tool's connector settings](/img/docs/dbt-cloud/oauth-connectors-page.png?v=2 "Adding a custom dbt connector in an AI tool's connector settings")](#)Adding a custom dbt connector in an AI tool's connector settings 5. That's it 🎉! Ask your tool a data question like *"What is the total revenue for the last 30 days?"* to confirm the connection. For more information on remote MCP OAuth setup, see [Use the remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md). #### Scopes and consent The first time a user connects a tool to dbt through OAuth, dbt shows a consent screen listing the scopes the client is requesting. Scopes act as a **filter on the user's existing permissions** — they don't grant any new access. A user can only consent to scopes for permissions they already have in dbt. From the consent screen, the user can: * Approve or deny individual scopes. * Choose whether to grant access to **all projects** or only **selected projects**. * Toggle **Keep session alive** to control whether dbt automatically refreshes their session (see [Sessions and refresh tokens](#sessions-and-refresh-tokens)). [![OAuth consent screen showing requested scopes and project access](/img/docs/dbt-cloud/oauth-consent-screen.png?v=2 "OAuth consent screen showing requested scopes and project access")](#)OAuth consent screen showing requested scopes and project access The exact scopes shown depend on what the connecting tool requests. The full list of available scopes is: | Scope | Description | | ------------------ | -------------------------------------------------------------------------------------- | | `offline_access` | Allows the client to request a refresh token to maintain long-lived sessions | | `account:read` | Read-only access to account details such as project and environment settings | | `projects:query` | Query project data using Semantic Layer metrics | | `catalog:read` | Access model descriptions, column lineage, and other project metadata from dbt Catalog | | `projects:develop` | Use development tooling, including the CLI, to make updates to dbt models | | `jobs:run` | View, edit, and re-trigger dbt jobs | ##### For developers: handling partial scope grants When a user unchecks one or more scopes on the consent screen, the access token is issued with only the approved subset. Your client should: * Request only the scopes your tool actually needs — don't request scopes as a precaution. * Handle `403` responses gracefully and surface a clear message if a required scope was not granted. * Avoid assuming the full set of requested scopes was approved; always check the `scope` field in the token response. To select the scopes your client should request, choose from the list above. Request scopes during the authorization request (`/oauth/authorize`). Scopes are not set during client registration. #### Sessions and refresh tokens OAuth sessions use refresh tokens with a 7-day inactivity window. As long as the client requests a new access token at least once every 7 days, the session stays alive. After 7 days of inactivity, the refresh token expires and the user has to sign in again. Users can opt out of automatic refresh at consent time by turning off **Keep session alive**. #### Revoke access To revoke an OAuth grant, the user disconnects dbt from their MCP client — for example, by removing the dbt connector in the client's settings. We'll soon add the ability to revoke access in dbt platform. [![MCP client connectors page with the option to remove the dbt connector](/img/docs/dbt-cloud/oauth-disconnect-client.png?v=2 "MCP client connectors page with the option to remove the dbt connector")](#)MCP client connectors page with the option to remove the dbt connector #### Audit logging OAuth-related events appear in the dbt [Audit log](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md) (**Account settings** → **Audit log**), so admins can see which clients were registered, which users authorized them, and what those users did through OAuth. Audited OAuth events include: * **Registering a client** (manual or dynamic). * **Authorizing a client** (a user completing the consent flow). Actions a user performs through an OAuth-connected client — for example, creating a job or running a model — appear in the audit log with the **user as the actor**, the same as actions taken through the dbt UI or with a personal access token. [![Audit log entry for an action taken through an OAuth-connected client](/img/docs/dbt-cloud/oauth-audit-log.png?v=2 "Audit log entry for an action taken through an OAuth-connected client")](#)Audit log entry for an action taken through an OAuth-connected client --- ##### dbt audit log dbt platform | Enterprise, Enterprise+ To review actions performed by people in your account, dbt provides logs of audited user and system events in real time. The audit log appears as events happen and includes details such as who performed the action, what the action was, and when it was performed. You can use these details to troubleshoot access issues, perform security audits, or analyze specific events. You can also [query and export audit log events through the API](#using-the-api). You must be an **Account admin**, **Account viewer**, or **Security admin** to access the audit log, and this feature is only available on Enterprise plans. The dbt audit log stores all the events that occurred in your account in real time, including: * For events within 90 days, the dbt audit log has a selectable date range that lists events triggered. * For events beyond 90 days, **Account admins**, **Account viewers**, and **Security admins** can [export all events](#exporting-logs) by using **Export All**. Note that the retention period for events in the audit log is at least 12 months. #### Accessing the audit log To access the audit log, click on your account name in the left-side menu and select **Account settings**. Click **Audit log** in the left sidebar. #### Understanding the audit log On the audit log page, you will see a list of various events and their associated event data. Each of these events shows the following information in dbt: * **Event name**: Action that was triggered * **Agent**: User who triggered that action/event * **Timestamp**: Local timestamp of when the event occurred ##### Event details Click the event card to see the details about the activity that triggered the event. This view provides important details, including when it happened and what type of event was triggered. For example, if someone changes the settings for a job, you can use the event details to see which job was changed (type of event: `job_definition.changed`), by whom (person who triggered the event: `actor`), and when (time it was triggered: `created_at`). For types of events and their descriptions, refer to [Events in audit log](#audit-log-events). The event details provide the key factors of an event: | Name | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | account\_id | Account ID of where the event occurred | | actor | Actor that carried out the event - User or Service | | actor\_id | Unique ID of the actor | | actor\_ip | IP address of the actor | | actor\_name | Identifying name of the actor | | actor\_type | Whether the action was done by a user or an API request | | created\_at | UTC timestamp of when the event occurred | | event\_type | Unique key identifying the event | | event\_context | This key will be different for each event and will match the event\_type. This data will include all the details about the object(s) that was changed. | | id | Unique ID of the event | | service | Service that carried out the action | | source | Source of the event - dbt UI or API | #### Audit log events The audit log supports various events for different objects in dbt. You will find events for authentication, environment, jobs, service tokens, groups, user, project, permissions, license, connection, repository, and credentials. ##### Authentication | Event name | Event type | Description | | -------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------- | | Access Token Issued | access\_token.issued | dbt issued an access token after OAuth sign-in (for example, VS Code extension or Model Context Protocol (MCP) server). | | Auth Provider Changed | auth\_provider.changed | Authentication provider settings changed | | Credential Login Succeeded | login.password.succeeded | User successfully logged in with username and password | | Refresh Token Issued | refresh\_token.issued | dbt issued a refresh token after OAuth sign-in (for example, VS Code extension or MCP server). | | SSO Login Failed | login.sso.failed | User sign-in through SSO failed | | SSO Login Succeeded | login.sso.succeeded | User successfully signed in through SSO | ##### OAuth clients These events cover the lifecycle of OAuth clients registered in **Account settings** → **Integrations** → **App integrations** (refer to [Connect apps with OAuth](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md)). Actions a user performs through an OAuth-connected client (for example, creating a job) are logged under the relevant event (such as `job_definition.added`) with the user as the actor. | Event name | Event type | Description | | ----------------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Client Authorization Approved | oauth\_client.authorization\_approved | A user approved an OAuth consent request for a client. | | Client Authorization Denied | oauth\_client.authorization\_denied | A user denied an OAuth consent request for a client. | | Client Authorized | oauth\_client.authorized | A user completed the OAuth consent flow and authorized a client to access dbt on their behalf. | | Client Registered | oauth\_client.registered | An OAuth client was registered, either manually by an admin or dynamically through [RFC 7591](https://datatracker.ietf.org/doc/html/rfc7591). | | Client Removed | oauth\_client.removed | An OAuth client was removed from **Account settings** → **Integrations** → **App integrations**. | ##### Environment | Event name | Event type | Description | | ------------------- | ------------------- | ------------------------------------ | | Environment Added | environment.added | New environment successfully created | | Environment Changed | environment.changed | Environment settings changed | | Environment Removed | environment.removed | Environment successfully removed | ##### Jobs | Event name | Event type | Description | | ----------- | ----------------------- | ---------------------------- | | Job Added | job\_definition.added | New Job successfully created | | Job Changed | job\_definition.changed | Job settings changed | | Job Removed | job\_definition.removed | Job definition removed | ##### Service Token | Event name | Event type | Description | | --------------------- | ---------------------- | ------------------------------------------ | | Service Token Created | service\_token.created | New Service Token was successfully created | | Service Token Revoked | service\_token.revoked | Service Token was revoked | ##### Group | Event name | Event type | Description | | ------------- | ------------- | ------------------------------ | | Group Added | group.added | New Group successfully created | | Group Changed | group.changed | Group settings changed | | Group Removed | group.removed | Group successfully removed | ##### User | Event name | Event type | Description | | ---------------------------- | ------------------------ | --------------------------------------------------- | | Invite Added | user.invite.added | User invitation added and sent to the user | | Invite Redeemed | user.invite.redeemed | User redeemed invitation | | User Added to Account | user.added | New user added to the account | | User Added to Group | group.user.added | An existing user was added to a group | | User Removed from Account | user.removed | User removed from the account | | User Removed from Group | group.user.removed | An existing user was removed from a group | | User License Created | user\_license.added | A new user license was consumed | | User License Removed | user\_license.removed | A user license was removed from the seat count | | Verification Email Confirmed | user.jit.email.confirmed | Email verification confirmed by user | | Verification Email Sent | user.jit.email.sent | Email verification sent to user created through JIT | ##### Project | Event name | Event type | Description | | --------------- | --------------- | ------------------------ | | Project Added | project.added | New project added | | Project Changed | project.changed | Project settings changed | | Project Removed | project.removed | Project is removed | ##### Permissions | Event name | Event type | Description | | ----------------------- | ------------------ | ------------------------------ | | User Permission Added | permission.added | New user permissions are added | | User Permission Removed | permission.removed | User permissions are removed | ##### License | Event name | Event type | Description | | ----------------------- | -------------------- | ----------------------------------------- | | License Mapping Added | license\_map.added | New user license mapping is added | | License Mapping Changed | license\_map.changed | User license mapping settings are changed | | License Mapping Removed | license\_map.removed | User license mapping is removed | ##### Connection | Event name | Event type | Description | | ------------------ | ------------------ | ------------------------------------------ | | Connection Added | connection.added | New Data Warehouse connection added | | Connection Changed | connection.changed | Data Warehouse Connection settings changed | | Connection Removed | connection.removed | Data Warehouse connection removed | ##### Repository | Event name | Event type | Description | | ------------------ | ------------------ | --------------------------- | | Repository Added | repository.added | New repository added | | Repository Changed | repository.changed | Repository settings changed | | Repository Removed | repository.removed | Repository removed | ##### Credentials | Event name | Event type | Description | | -------------------------------- | ------------------- | -------------------------------- | | Credentials Added to Project | credentials.added | Project credentials added | | Credentials Changed in Project | credentials.changed | Credentials changed in project | | Credentials Removed from Project | credentials.removed | Credentials removed from project | ##### Git integration | Event name | Event type | Description | | -------------------------- | --------------------------- | ----------------------------------- | | GitLab Application Changed | gitlab\_application.changed | GitLab configuration in dbt changed | ##### Webhooks | Event name | Event type | Description | | ----------------------------- | ----------------------------- | -------------------------------------- | | Webhook Subscriptions Added | webhook\_subscription.added | New webhook configured in settings | | Webhook Subscriptions Changed | webhook\_subscription.changed | Existing webhook configuration altered | | Webhook Subscriptions Removed | webhook\_subscription.removed | Existing webhook deleted | ##### Semantic Layer | Event name | Event type | Description | | ---------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------ | | Semantic Layer Config Added | semantic\_layer\_config.added | Semantic Layer config added | | Semantic Layer Config Changed | semantic\_layer\_config.changed | Semantic Layer config (not related to credentials) changed | | Semantic Layer Config Removed | semantic\_layer\_config.removed | Semantic Layer config removed | | Semantic Layer Credentials Added | semantic\_layer\_credentials.added | Semantic Layer credentials added | | Semantic Layer Credentials Changed | semantic\_layer\_credentials.changed | Semantic Layer credentials changed. Does not trigger `semantic_layer_config.changed` | | Semantic Layer Credentials Removed | semantic\_layer\_credentials.removed | Semantic Layer credentials removed | ##### Extended attributes | Event name | Event type | Description | | -------------------------- | ---------------------------- | ------------------------------------- | | Extended Attribute Added | extended\_attributes.added | Extended attribute added to a project | | Extended Attribute Changed | extended\_attributes.changed | Extended attribute changed or removed | ##### Account-scoped personal access token | Event name | Event type | Description | | -------------------------------------------- | ---------------------------- | --------------------------------- | | Account Scoped Personal Access Token Created | account\_scoped\_pat.created | An account-scoped PAT was created | | Account Scoped Personal Access Token Deleted | account\_scoped\_pat.deleted | An account-scoped PAT was deleted | ##### IP restrictions | Event name | Event type | Description | | ---------------------------- | ----------------------------- | ------------------------------------------- | | IP Restrictions Toggled | ip\_restrictions.toggled | IP restrictions feature enabled or disabled | | IP Restrictions Rule Added | ip\_restrictions.rule.added | IP restriction rule created | | IP Restrictions Rule Changed | ip\_restrictions.rule.changed | IP restriction rule edited | | IP Restrictions Rule Removed | ip\_restrictions.rule.removed | IP restriction rule deleted | ##### SCIM | Event name | Event type | Description | | -------------- | ----------------------------------- | -------------------------------------------- | | User Creation | v1.events.account.UserAdded | New user created by SCIM service | | User Update | v1.events.account.UserUpdated | User record updated by SCIM service | | User Removal | v1.events.account.UserRemoved | User deleted by the SCIM service | | Group Creation | v1.events.user\_group.Added | New group created by SCIM service | | Group Update | v1.events.user\_group\_user.Changed | Group membership was updated by SCIM service | | Group Removal | v1.events.user\_group.Removed | Group removed by SCIM service | #### Searching the audit log You can search the audit log to find a specific event or actor, which is limited to the ones listed in [Events in audit log](#audit-log-events). The audit log lists historical events from the last 90 days. You can search for an actor or event using the search bar, and then narrow your results using the time window. [![Use search bar to find content in the audit log](/img/docs/dbt-platform/dbt-platform-enterprise/audit-log-search.png?v=2 "Use search bar to find content in the audit log")](#)Use search bar to find content in the audit log #### Exporting logs You can use the audit log to export all historical audit results for security, compliance, and analysis purposes. Events in the audit log are retained for at least 12 months. * **For events within 90 days:** dbt will automatically display the 90-day selectable date range. Select **Export Selection** to download a CSV file of all the events that occurred in your account within 90 days. * **For events beyond 90 days:** Select **Export All**. An **Account admin**, **Account viewer**, or **Security admin** receives an email link to download a CSV file of all the events that occurred in your account. [![View audit log export options](/img/docs/dbt-platform/dbt-platform-enterprise/audit-log-section.png?v=2 "View audit log export options")](#)View audit log export options #### Using the API You can also query and export audit log events with the [Administrative API v3](https://docs.getdbt.com/dbt-cloud/api-v3). Use the API when you want to pull audit data into a script, a scheduled job, or security monitoring or log aggregation tooling. The same access requirements at the top of this page apply to the API. For common automation approaches, refer to [Common automation patterns](#common-automation-patterns). | If you want to… | Refer to | | ---------------------------------------- | --------------------------------- | | Pull recent events into a script or SIEM | [Query events](#query-events) | | Export a date range as CSV | [Download a CSV](#download-a-csv) | | Export full history | [Bulk export](#bulk-export) | Authenticate with a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md), passed as `Authorization: Bearer YOUR_TOKEN` or `Token YOUR_TOKEN`. Bulk exports require a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) as service tokens aren't supported for that step. Without date parameters, list and CSV download requests default to the last 90 days. In the following examples, replace `YOUR_ACCESS_URL` with your dbt access URL, `ACCOUNT_ID` with your account ID, and `YOUR_TOKEN` with your token. You can find your account ID in **Account settings**, or refer to [Finding your user and account IDs](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for more information. ##### Query events To fetch audit log events as JSON, send a `GET` request to `/api/v3/accounts/ACCOUNT_ID/audit-logs/`: ```shell curl --request GET \ --url 'https://YOUR_ACCESS_URL/api/v3/accounts/ACCOUNT_ID/audit-logs/?logged_at_start=2024-01-01&logged_at_end=2024-01-31&limit=10&offset=0' \ --header 'Accept: application/json' \ --header 'Authorization: Bearer YOUR_TOKEN' ``` The response includes a `data` array of events with details such as `event_type`, `event_label`, `actor`, `created_at`, and `event_context`. Use `logged_at_start` and `logged_at_end` to filter by date range, and `limit` and `offset` to paginate through results. For more options, refer to [List Recent Audit Log Events](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/List%20Recent%20Audit%20Log%20Events) in the API reference. ##### Download a CSV To download audit log events as a CSV file, send a `GET` request to `/api/v3/accounts/ACCOUNT_ID/audit-logs/download/`. This is the API equivalent of **Export Selection** in the UI: ```shell curl --request GET \ --url 'https://YOUR_ACCESS_URL/api/v3/accounts/ACCOUNT_ID/audit-logs/download/?logged_at_start=2024-01-01&logged_at_end=2024-01-31' \ --header 'Authorization: Bearer YOUR_TOKEN' \ --output audit-log.csv ``` You can use the same `logged_at_start` and `logged_at_end` query parameters as the list endpoint. ##### Bulk export To export your full audit log history through the API, use the same workflow as **Export All** in the UI: 1. Submit an export request with `POST` to `/api/v3/accounts/ACCOUNT_ID/audit-logs/export/`. The response returns a `job_id` in the `data` object (`data.job_id`). Use a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) for this step. ```shell curl --request POST \ --url 'https://YOUR_ACCESS_URL/api/v3/accounts/ACCOUNT_ID/audit-logs/export/' \ --header 'Authorization: Bearer YOUR_TOKEN' ``` 2. Check the export status with `GET` on `/api/v3/accounts/ACCOUNT_ID/audit-logs/export/`. Repeat this request until `is_running` in the `data` object is `false`. ```shell curl --request GET \ --url 'https://YOUR_ACCESS_URL/api/v3/accounts/ACCOUNT_ID/audit-logs/export/' \ --header 'Accept: application/json' \ --header 'Authorization: Bearer YOUR_TOKEN' ``` 3. Get a download URL with `GET` on `/api/v3/accounts/ACCOUNT_ID/audit-logs/export/JOB_ID/download/`. The response includes a `download_url` in the `data` object (`data.download_url`). Use the `job_id` from step 1 as `JOB_ID` in the URL. ```shell curl --request GET \ --url 'https://YOUR_ACCESS_URL/api/v3/accounts/ACCOUNT_ID/audit-logs/export/JOB_ID/download/' \ --header 'Accept: application/json' \ --header 'Authorization: Bearer YOUR_TOKEN' ``` 4. Download the CSV file from the `download_url` returned in step 3 (`data.download_url`): ```shell curl --request GET \ --url 'DOWNLOAD_URL' \ --output audit-log-full.csv ``` ##### Common automation patterns Teams often automate audit log access in a few ways: * **Scheduled incremental pulls:** Call the list endpoint on a schedule with `logged_at_start` and `logged_at_end` to fetch new events since the last run, then load the JSON into your security monitoring or log aggregation tooling. * **Periodic CSV exports:** Use the download endpoint to export a date range as CSV for compliance archival or analysis. * **Full history exports:** Use the bulk export workflow when you need your complete audit log history programmatically, similar to **Export All** in the UI. For endpoint details, refer to the [Administrative API v3 reference](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/List%20Recent%20Audit%20Log%20Events). --- ##### Enterprise permissions dbt platform | Enterprise, Enterprise+ Available to Enterprise-tier plans This feature is available to the dbt Enterprise and Enterprise+ plans. If you're interested in learning more, contact us at <sales@getdbt.com>. The dbt Enterprise and Enterprise+ plans support a number of pre-built permission sets to help manage access controls within a dbt account. See the docs on [access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) for more information on Role-Based access control (RBAC). #### Permission sets The following permission sets are available for assignment in all dbt Enterprise-tier accounts. They can be granted to dbt groups and then to users. A dbt group can be associated with more than one permission set. Permission assignments with more access take precedence. Access to dbt features and functionality is split into `account-level` and `project-level` permission sets. Account-level permissions are primarily for account administration (inviting users, configuring SSO, and creating groups). Project-level permissions are for the configuration and maintenance of the projects themselves (configuring environments, accessing IDE, and running jobs). Account permission sets may have access to project features, and project permission sets may have access to account features. Check out the [permissions tables](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-permissions) to compare sets and their access. Read-Only users If you have users with a Read-Only license, you can [enable granular permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#enable-granular-permissions-for-read-only-users) for your account. This is a one-time, irreversible setting. After you enable it, Read-Only users keep their project access only if they're in a group with a Read-Only permission set that covers all projects.  Account admin The Account admin permission set is the highest level of access and control over your dbt account and projects. We recommend limiting the number of users and groups assigned the account admin permission set. Notable features: * Account admin is an account-level set. * Unrestricted access to every feature. * The default permissions for every user who creates a new dbt account. * The default permissions assigned to the `Owner` group.  Admin The Admin permission set is intended for project administration but with limited account-level access to invite and assign users. Notable features: * Admin is a project-level set. * Unrestricted access to existing projects, but can't create new projects. * Can invite new members and assign access but can't create groups. * Can access Catalog. * The default permissions assigned to the `Member` group.  Account Viewer The Account Viewer permissions set provides read-only access to the dbt account. Useful for any persona who needs insights into your dbt account without access to create or change configurations. The Account Viewer permission set is frequently paired with the [Read-only license-type](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). Notable features: * Account Viewer is an account-level set. * Read-only access to all settings, projects, environments, and runs. * Read-only access to audit logs, including sensitive account-level information. * No access to the IDE. * Can access Catalog  Analyst The Analyst permission set is designed for users who need to run and analyze dbt models in the IDE but can't create or edit anything outside the IDE. Notable features: * Analyst is a project-level set. * Full access to the IDE and the ability to configure personal credentials for adapters and Git. * Read-only access to environment configs. * Can view jobs but can't edit them. * Can access Catalog.  Analyst read The Analyst read permission set is a project-level set designed for users who need read-only access to analyze dbt models and project resources without the ability to develop or make changes. It does not change the user's license. On Enterprise-tier plans, you can assign Analyst read to Read-Only users. That lets those users manage their own warehouse credentials and use the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md). They can connect with OAuth or a personal access token. For MCP authentication options, refer to [Connect dbt MCP server to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md). Notable features: * Analyst read is a project-level set. * Read-only access to **Connections** (account and project), **Projects**, repositories (Git repository settings), Semantic Layer configuration, **Environments**, custom environment variables, and Catalog metadata (Metadata GraphQL API). * No read access to jobs or runs. * Includes `user_credential_write`, so users can view and edit their own user credentials on **Your profile** > **Credentials** without access to the Studio IDE or dbt platform CLI. Read-only users need personal user credentials on this page to run warehouse queries in analysis features such as the Semantic Layer. In dbt Wizard [Explore mode](https://docs.getdbt.com/docs/platform/wizard-home.md#ask-questions-in-explore-mode), users without personal credentials fall back to the project's [analytics credential](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md#set-up-analytics-credentials) instead. * No write access and no access to develop in the Studio IDE or dbt platform CLI. To access the capabilities of this permission set, you *must* add users to a group that's assigned the Analyst read permission set. Users won't have access until they're added to the group. For the setup steps, refer to [Set up read-only user access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#set-up-read-only-user-access).  Billing admin The Billing admin permission set can review product usage information that impacts the final billing of dbt (for example, models run). Notable features: * Billing admin is an account-level set. * Unrestricted access to the **Billing** section of your **Account settings**. * Read access to public models. * No other access.  Cost Insights Admin The Cost Insights Admin permission set provides the minimum permissions needed to configure and manage [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) settings and view cost data. Notable features: * Cost Insights Admin is both an account-level and project-level set. * Can configure platform metadata credentials and Cost Insights settings in connection settings. * Can view cost and savings data across projects, models, and jobs. * Read-only access to connections, projects, jobs, and metadata. * Can access dbt Catalog.  Cost Insights Viewer The Cost Insights Viewer permission set provides read-only access to [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) data with the minimum permissions needed to view estimated cost and reduction information. A Read-Only license is not enough to view Cost Insights. Assign this permission set through a group. Notable features: * Cost Insights Viewer is both an account-level and project-level set. * Read-only access to cost and savings data across projects, models, and jobs. * Read-only access to connections, platform metadata credentials, projects, jobs, and metadata. * Cannot configure or edit Cost Insights settings. * Can access dbt Catalog.  Database admin Database admins manage configurations between dbt and the underlying databases. Notable features: * Database admin is a project-level set. * Can set up and maintain Semantic Layer configs. * Write access to data platform configurations within environments (credentials, warehouse, schema per environment), including: * Editing [profile](https://docs.getdbt.com/docs/platform/about-profiles.md) configs like profile name, deployment credentials, extended attributes, and connection overrides such as `schema`, `role`, `database`, and so on (fields vary by data platform). * Creating new profiles for projects they have access to, including setting which connection the profile is associated with * Helpful for scenarios where your data warehouse admins only need access to dbt to configure data platform settings within environments. * Read-only access to account-level connections, Git repo, job, and run settings. * Can access Catalog. * Has limited access to environment variables. Use the following table to see what's allowed at each level: | Level | Access | | ---------------- | -------------------------------------------------------------- | | Job | Can override an environment variable's value on a job | | Environment | Read-only. Can't create, edit, or delete environment variables | | User credentials | No access, because this set doesn't include Studio IDE access | Granting [environment write access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#environment-write-access) doesn't change this. It elevates job and run permissions to write, but leaves environment variables read-only.  Developer The Developer permission set is intended for users who build and maintain dbt models under development and manage production behavior. This is the primary permission set for users working in the IDE and should not be conflated with the [Developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses). Notable features: * Developer is a project-level set. * Can create, edit, and test dbt code in the IDE. * Read-only access to the underlying configs for environments, jobs, runs, and Git. * Users manage their credentials to data warehouses and Git. * Can access Catalog.  v2 Migration Admin This permission set enables users to interact with dbt v2 upgrade workflows. We recommend limiting this permission to users who are actively [working on migrating](https://docs.getdbt.com/guides/upgrade-to-v2.md?step=1) a project to dbt v2. By default, all users can access the dbt v2 upgrade experience. When the upgrade permissions setting is enabled, only users with the **v2 Migration Admin** or **Account admin** permission set can perform upgrades. If the setting is disabled (no check mark), upgrades are not restricted. * **v2 Migration Admin** — Assign to user accounts only. Cannot be assigned to service tokens. * **Account admin** — Assign to user accounts or service tokens. Allows both users and service tokens to perform upgrades. For more information, refer to [Upgrade to dbt v2](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2).  Git admin Git admins manage Git repository integrations and cloning. Notable features: * Git admin is a project-level set. * Can create new Git integrations and environment variables. * Can edit project settings. * Read-only access to account settings (including users and groups). * No access to the IDE. * Can access Catalog.  Job admin Job admin is an administrative permission set for users who create, run, and manage jobs in dbt. Notable features: * Job admin is a project-level set. * Job admins can create and edit jobs, runs, environment variables, and data warehouse configs. * Job admins can set up project integrations, including [Tableau lineage](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md). * Read-only access to project configs. * Read-only access to connections and public models. * Can access Catalog.  Job creator Job creator is a scoped project-level permission set for users who need to manage jobs within assigned projects without broader administrative access. Notable features: * Job creator is a project-level set. * Can create, edit, and run jobs in environment types where the group has [environment write access](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions-setup.md). Without write access, job and run actions are read-only. * Cannot create or delete environments or edit environment variables. * More limited than [Job admin](#job-admin), which can edit environments, environment variables, and data platform configurations.  Job runner Job runner is a specialized permission set for users who need access to run jobs and view the outcomes. Notable features: * Job runner is a project-level set. * Can run jobs. * Has read-only access to jobs, including status and results. * No other access to dbt features.  Job viewer Job viewer enables users to monitor and review job executions within dbt. Users with this role can see jobs’ status, logs, and outcomes but cannot initiate or modify them. Notable features: * Job viewer is a project-level set. * Read-only access to job results, status, and logs. * No other access to dbt features. * Can access Catalog.  Manage marketplace apps Manage marketplace apps is a specialized permission set associated with dbt marketplace apps. Usually implemented for the Snowflake Native App. Notable features: * Manage marketplace apps is an account-level set. * Used exclusively for marketplace app integrations. * Not intended for general user/group assignment.  Metadata (Discovery API only) Metadata is intended to be a read-only [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) integration permission set. Notable features: * Metadata is a project-level set. * Grants read-only access to metadata related to dbt models, runs, sources, and tests. * No access to modify, execute, or manage dbt jobs, repositories, or users. * No other access to dbt features.  Notification Manager The Notification Manager permission set allows users to configure and manage Slack, Microsoft Teams, and email [job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) without requiring full Account Admin access. This is ideal for team leads or operations users who need to manage who receives job alerts. Notable features: * Notification Manager is an account-level set. * Can create, edit, and delete job notification rules for Slack, Microsoft Teams, and email recipients **across all projects** on the account. * Read-only access to members (to populate the recipient picker). * Read-only access to Slack and Microsoft Teams OAuth configuration status (for viewing the Slack and Microsoft Teams integration connection). * Connecting or disconnecting the Slack or Microsoft Teams integration remains Account Admin-only. * No access to other account settings, jobs, runs, or environments.  Project creator The Project creator permission set can create, configure, and set up new projects. It is recommended for the admin of teams that will own a project. Notable features: * Project creator is an account-level set. * Only permission set other than Account admin that can create projects. * Limited account settings access. The project creator can create and edit connections, invite users, create groups, and assign licenses. * Unrestricted access to project configurations. * Can access Catalog  Security admin Security admins have limited access to the security settings and policies for the dbt account. This is intended for members of a security team who need to ensure compliance with security standards and oversee the implementation of best security practices across the account. The [IT license-type](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses) includes this permission set by default. Notable features: * Security admin is an account-level set. * Can create and edit users and groups and assign licenses. * Can create and edit authentication and SSO settings. * Can create and edit IP restrictions, view service tokens, and manage user access controls. * No access to jobs, runs, environments, or the IDE.  Semantic Layer A specialized permission set with strict access to only query the Semantic Layer using a service token. Notable features: * Semantic Layer is a project-level set. * Can only query Semantic Layer. * No other access to dbt features.  Stakeholder and Read-Only The Stakeholder and Read-Only are identical permission sets that are similar to Account Viewer, but without access to sensitive content such as account settings, billing information, or audit logs. Useful for personas who need to monitor projects and their configurations. Notable features: * Stakeholder is a project-level set. * Read-only access to projects, environments, jobs, and runs. * Read-only access to user and group information. * Can access Catalog. * No access to the IDE. * Limited access to audit log content that excludes sensitive information, such as user settings and account-level changes.  Team admin Team admin is an administrative permission set intended for team leaders or similar personas. The permissions grants the ability to manage projects for the team. Notable features: * Team admin is a project-level set. * Access to manage the project(s) for a team of users. Limited scope and access can be extended via environment permissions. * Read-only access to many account settings (excluding sensitive content like billing and auth providers). * Can access Catalog. License types override group permissions **User license types always override their assigned group permission sets.** For example, a user with a Read-Only license cannot perform administrative actions, even if they belong to an Account Admin group. This ensures that license restrictions are always enforced, regardless of group membership. Permissions: * **Account-level permissions** — Permissions related to the management of the dbt account. For example, billing and account settings. * **Project-level permissions** — Permissions related to the projects in dbt. For example, repos and access to the Studio IDE or dbt platform CLI. note Some permissions sets have read-only access to environment settings that can be overridden with more privileged access if the user is assigned to a group with [Environment write access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#environment-write-access) configured. ##### Account permissions Account permission sets enable you to manage the dbt account and manage the account settings (for example, generating service tokens, inviting users, and configuring SSO). They also provide project-level permissions. The **Account Admin** permission set is the highest level of access you can assign. Key: * **(W)rite** — Create new or modify existing. Includes `send`, `create`, `delete`, `allocate`, `modify`, and `develop`. * **(R)ead** — Can view but cannot create or change any fields. ###### Account access for account permissions | Account-level permission | Account Admin | Billing admin | Cost Insights Admin | Cost Insights Viewer | Manage marketplace apps | Notification Manager | Project creator | Security admin | Account Viewer | | ------------------------ | ------------- | ------------- | ------------------- | -------------------- | ----------------------- | -------------------- | --------------- | -------------- | -------------- | | Account settings\* | W | - | - | - | - | - | R | R | R | | Audit logs | R | - | - | - | - | - | - | R | R | | Auth provider | W | - | - | - | - | - | - | W | R | | Billing | W | W | - | - | - | - | - | - | R | | Connections | W | - | R\*\* | R | - | R | W | - | - | | Cost Insights | R | - | R | R | - | - | R | - | R | | Groups | W | - | - | - | - | - | R | W | R | | Invitations | W | - | - | - | - | - | W | W | R | | IP restrictions | W | - | - | - | - | - | - | W | R | | Job notifications | W | - | - | - | - | W | - | - | R | | Licenses | W | - | - | - | - | - | W | W | R | | Marketplace app | - | - | - | - | W | - | - | - | - | | Members | W | - | - | - | - | R | W | W | R | | Project (create) | W | - | - | - | - | - | W | - | - | | Public models | R | R | - | - | - | - | R | R | R | | Service tokens | W | - | - | - | - | - | - | R | R | | Webhooks | W | - | - | - | - | - | - | - | - | \*Permission sets with write (**W**) access to Account settings can modify account-level settings. The **Notification Manager** permission set has dedicated write access to **Job notifications** (Slack, Microsoft Teams, and email) without requiring full Account settings access. \*\***Cost Insights Admin** can edit [platform metadata credentials](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials) and [Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) settings in **Connection settings**, even though **Connections** is read-only (**R**) for this permission set. Credentials access Users can access the **Credentials** page under **Your profile** when they have `develop_access` or `user_credential_write` on at least one project. An admin can grant `user_credential_write` to any group, regardless of which permission set the group is assigned. Users with only `user_credential_write` can configure warehouse and Git credentials on that page; environment variable and dbt version overrides still require `develop_access`. ###### Project access for account permissions | Project-level permission | Account Admin | Billing admin | Cost Insights Admin | Cost Insights Viewer | Notification Manager | Project creator | Security admin | Account Viewer | | ---------------------------- | ------------- | ------------- | ------------------- | -------------------- | -------------------- | --------------- | -------------- | -------------- | | Environment credentials | W | - | - | - | - | W | - | R | | Custom env. variables | W | - | - | - | - | W | - | R | | Cost Insights | R | - | R | R | - | R | - | R | | Data platform configurations | W | - | - | - | - | W | - | R | | Develop (IDE or CLI) | W | - | - | - | - | W | - | - | | Environments | W | - | - | - | - | W | - | R | | Jobs | W | - | R | R | - | W | - | R | | Metadata GraphQL API access | R | - | R | R | - | R | - | R | | Permissions | W | - | - | - | - | W | W | R | | Projects | W | - | R | R | - | W | R | R | | Repositories | W | - | - | - | - | W | - | R | | Runs | W | - | - | - | - | W | - | R | | Semantic Layer config | W | - | - | - | - | W | - | R | ##### Project permissions The project permission sets enable you to work within the projects in various capacities. They primarily provide access to project-level permissions such as repos and the Studio IDE or dbt platform CLI, but may also provide some account-level permissions. Key: * **(W)rite** — Create new or modify existing. Includes `send`, `create`, `delete`, `allocate`, `modify`, and `develop`. * **(R)ead** — Can view but cannot create or change any fields. ###### Account access for project permissions | Account-level permission | Admin | Analyst | Analyst read | Cost Insights Admin | Cost Insights Viewer | Database admin | Developer | Git Admin | Job admin | Job creator | Job runner | Job viewer | Metadata (Discovery API only) | Semantic Layer | Stakeholder/Read-Only | Team admin | | ------------------------ | ----- | ------- | ------------ | ------------------- | -------------------- | -------------- | --------- | --------- | --------- | ----------- | ---------- | ---------- | ----------------------------- | -------------- | --------------------- | ---------- | | Account settings | R | - | R | - | - | R | - | R | - | - | - | - | - | - | R | R | | Auth provider | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | Billing | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | Connections | R | R | R | R\* | R | R | R | R | R | R | - | - | - | R | R | R | | Cost Insights | - | - | R | R | R | R | - | R | R | - | - | - | - | - | R | R | | Groups | R | - | R | - | - | R | R | R | - | - | - | - | - | R | R | R | | Invitations | W | R | R | - | - | R | R | R | R | - | R | - | - | R | R | R | | Licenses | W | R | R | - | - | R | R | R | R | - | R | - | - | - | R | R | | Members | W | - | R | - | - | R | R | R | - | - | - | - | - | R | R | R | | Project (create) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | Public models | R | R | R | - | - | R | R | R | R | R | R | R | R | R | R | R | | Service tokens | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | Webhooks | W | - | - | - | - | - | W | - | - | - | - | - | - | - | - | - | \***Cost Insights Admin** can edit [platform metadata credentials](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials) and [Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) settings in **Connection settings**, even though **Connections** is read-only (**R**) for this permission set. ###### Project access for project permissions | Project-level permission | Admin | Analyst | Analyst read\*\*\* | Cost Insights Admin | Cost Insights Viewer | Database admin | Developer | v2 Migration Admin | Git Admin | Job admin | Job creator | Job runner | Job viewer | Metadata (Discovery API only) | Semantic Layer | Stakeholder/Read-Only | Team admin | | --------------------------- | ----- | ------- | ------------------ | ------------------- | -------------------- | -------------- | --------- | ------------------ | --------- | --------- | ----------- | ---------- | ---------- | ----------------------------- | -------------- | --------------------- | ---------- | | Environment credentials | W | R | R | - | - | W | R | - | R | W | R | - | - | - | - | R | R | | Custom env. variables | W | W\*\* | R | - | - | R\*\*\*\* | W\*\* | - | W | W | R | - | R | - | - | R | W | | Cost Insights | - | - | - | R | R | R | - | - | R | R | - | - | - | - | - | - | R | | Data platform configs | W | W | R | - | - | W | W | - | R | W | R | - | - | - | - | R | R | | Develop (IDE or CLI) | W | W | - | - | - | - | W | - | - | - | - | - | - | - | - | - | - | | Environments | W | R | R | - | - | R | R | - | R | W | R | - | R | - | - | R | R | | dbt v2 upgrade | - | - | - | - | - | - | - | W | - | - | - | - | - | - | - | - | - | | Jobs | W | R\* | - | R | R | R\* | R\* | - | R\* | W | R\* | R | R | - | - | R | R\* | | Metadata GraphQL API access | R | R | R | R | R | R | R | - | R | R | R | - | R | R | - | R | R | | Permissions | W | - | - | - | - | R | R | - | R | - | - | - | - | - | - | - | R | | Projects | W | R | R | R | R | W | R | - | W | R | R | - | R | - | - | R | W | | Repositories | W | R | R | - | - | R | R | - | W | - | R | - | - | - | - | R | R | | Runs | W | R\* | - | - | - | R\* | R\* | - | R\* | W | R\* | W | R | - | - | R | R\* | | Semantic Layer config | W | R | R | - | - | W | R | - | R | R | R | - | - | - | W | R | R | \*These permissions are `R`ead-only by default, but may be changed to `W`rite with [environment permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md#environments-and-roles). \*\*Custom env. variables for the `Developer` and `Analyst` roles are set in the **Credentials** section of **Account settings**. \*\*\*The **Analyst read** permission set also includes `user_credential_write`, letting users manage their own credentials on the **Credentials** page (under **Your profile**). \*\*\*\***Database admin** can override an environment variable's value on a job, but can't create, edit, or delete environment variables at the environment level. Enabling [environment write access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#environment-write-access) doesn't change this. For more detail, refer to [Database admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#database-admin). #### Additional resources * [Grant users access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#grant-access) * [Role-based access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-) * [Environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md) --- ##### Invite users to dbt dbt platform | Starter, Enterprise, Enterprise+ dbt makes it easy to invite new users to your environment out of the box. This feature is available to all dbt customers on Starter, Enterprise, and Enterprise+ plans. #### Prerequisites You must have proper permissions to invite new users: * [**Starter accounts**](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md) — must have `member` or `owner` permissions. * [**Enterprise-tier accounts**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) — must have `admin`, `account admin`, `project creator`, or `security admin` permissions. * The admin inviting the users must have a `developer` or `IT` license. #### Invite new users 1. In your dbt account, select your account name in the bottom left corner. Then select **Account settings**. 2. Under **Settings**, select **Users**. 3. Click on **Invite users**. [![The invite users pane](/img/docs/dbt-platform/access-control/invite-users.png?v=2 "The invite users pane")](#)The invite users pane 4. In the **Email Addresses** field, enter the email addresses of the users you want to invite separated by a comma, semicolon, or a new line. 5. Select the license type for the batch of users from the **License** dropdown. 6. Select the group(s) you want the invitees to belong to. 7. Click **Send invitations**. * If the list of invitees exceeds the number of licenses your account has available, you will receive a warning when you click **Send Invitations** and the invitations will not be sent. #### User experience Email verification Email verification is mandatory for all new users in dbt, including using Single Sign-On (SSO)⁠⁠. Automatic provisioning without email verification is not allowed. This is a security requirement that cannot be bypassed. dbt generates and sends emails from `support@getdbt.com` to the specified addresses. Make sure that traffic from the `support@getdbt.com` email is allowed in your settings to avoid emails from going to spam or being blocked. This is the originating email address for all [instances worldwide](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). The email contains a link to create an account. When the user clicks on this link, they will be brought to one of two screens depending on whether SSO is configured or not. [![Example or an email invitation](/img/docs/dbt-platform/access-control/email-invite.png?v=2 "Example or an email invitation")](#)Example or an email invitation ##### Local user The default settings send the email, the user clicks the link, and is prompted to create their account: [![Default user invitation](/img/docs/dbt-platform/access-control/default-user-invite.png?v=2 "Default user invitation")](#)Default user invitation ##### SSO user If SSO is configured for the environment, the user must: 1. Click the link in the verification email. 2. Click the option to join the account. 3. A confirmation screen appears, with a link to authenticate against the company's identity provider. Click **Authenticate with your enterprise login**. Complete the SSO flow Accepting the invite doesn't fully complete the process. The user *must* log in using SSO to redeem the invite and access the account. [![User invitation with SSO configured](/img/docs/dbt-platform/access-control/sso-user-invite.png?v=2 "User invitation with SSO configured")](#)User invitation with SSO configured Once the user completes this process, their email and user information will populate in the **Users** screen in dbt. #### FAQ  Is there a limit to the number of users I can invite? Your ability to invite users is limited to the number of licenses you have available.  What happens if I reinstate a deleted user? If you invite a previously deleted user back to your account with the same email address, their personal profile information, including linked accounts, will persist (unless the user deleted the connection from the source account, like GitHub). Any previously assigned permissions and access settings are reset, and you will have to reassign them.  Why are users clicking the invitation link and getting an 'Invalid Invitation Code' error? We have seen scenarios where embedded secure link technology (such as enterprise Outlook's [Safe Link](https://learn.microsoft.com/en-us/microsoft-365/security/office-365-security/safe-links-about?view=o365-worldwide) feature) can result in errors when clicking on the email link. Be sure to include the `getdbt.com` URL in the allowlists for these services.  Can I have a mixture of users with SSO and username/password authentication? Once SSO is enabled, you will no longer be able to add local users. If you have contractors or similar contingent workers, we recommend you add them to your SSO service.  What happens if I need to resend the invitation? From the **Users** page, click on the invite record, and you will be presented with the option to resend the invitation.  What can I do if I entered an email address incorrectly? From the **Users** page, click on the invite record, and you will be presented with the option to revoke it. Once revoked, generate a new invitation to the correct email address. [![Resend or revoke the users invitation](/img/docs/dbt-platform/access-control/resend-invite.png?v=2 "Resend or revoke the users invitation")](#)Resend or revoke the users invitation --- ##### Manage user licenses with SCIM dbt platform | Enterprise, Enterprise+ You can manage user license assignments using System for Cross-Domain Identity Management (SCIM) and a user attribute in Okta, so the license type is set as users are provisioned and onboarded. License mapping with SCIM dbt platform supports automatic license assignment with SCIM, with these different identity provider options:  Toggle options per identity provider * **Okta:** Enable the **Ignore dbt license mapping** toggle in **Account settings > SSO & SCIM** and follow the [Okta license management doc](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md). * **Entra ID:** Use [SSO-based Active Directory group → license mapping](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration). It works alongside an active Entra ID SCIM setup. Keep the **Ignore dbt license mapping** toggle *disabled* as enabling it removes license mapping for Entra ID users. SCIM license mapping with [custom attributes](https://learn.microsoft.com/en-us/entra/fundamentals/custom-security-attributes-overview) may be possible depending on your Entra ID configuration, but it's not a supported setup — **do it at your own risk**. For more details, refer to the [Does SCIM support automatic license assignment?](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md#does-scim-support-automatic-license-assignment) FAQ. **Account Admin group required:** Before enabling SCIM on your dbt account, you must create a dbt Account Admin group in your IdP, and a group with a matching name in your dbt account. Assign the dbt group the [Account Admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) permission set. Without this group, your users could be removed from the default dbt 'Owner' group once SCIM provisioning is enabled and would lose admin access to your dbt account. The names of these two groups can match your IdP group naming scheme, but the names for both the dbt and IdP group must be the same value. #### Prerequisites Before you enable SCIM license mapping: * **Default license**: New users are assigned a Developer license unless you change it manually using [SSO license mappings](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration), or using SCIM. * **Best practice**: Use one source of truth for license assignment (either dbt platform or SCIM). Don't mix SCIM license management with manual or single sign-on (SSO) mapping changes. * **Analyst license**: Only available on [select plans](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). Assigning this license using SCIM will return an error if that license type isn't available for your account. The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. #### Enable SCIM license mapping To manage user licenses with SCIM, go to **Account settings** > **SSO & SCIM**. Under the **SCIM** section, enable the **Ignore dbt license mapping** toggle. This setting enforces license type for a user based on their SCIM attribute and disables the license mapping and manual configuration set up in dbt. [![Enable SCIM managed user license distribution.](/img/docs/dbt-platform/access-control/scim-managed-licenses.png?v=2 "Enable SCIM managed user license distribution.")](#)Enable SCIM managed user license distribution. We recommend that you complete the setup instructions for your identity provider (IdP) prior to enabling this toggle in your dbt account. Once enabled, any existing license mappings in dbt will be ignored. The recommended steps for migrating to SCIM license mapping are as follows: 1. Set up SCIM but keep the toggle disabled so existing license mappings continue to work as expected. 2. Configure license attributes in your IdP. 3. Test that SCIM attributes are being used to set license type in dbt. 4. Enable the toggle to ignore existing license mappings so that SCIM is the source of truth for assigning licenses to users. #### Enterprise default groups On the Enterprise licensing page, the following default groups are available. These are often used for initial setup but should be managed or removed once IdP groups are successfully synced via SCIM. | Group | Access permissions | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Owner** | Full access to account features and billing. | | **Member** | Robust access to the account with restrictions on billing and security settings. Users in the Member group are assigned a Developer license by default. | | **Everyone** | A catch-all group for all users. This group does not have permission assignments beyond the user's profile. Users must be assigned to the Member or Owner group to work in dbt. | Best practice After creating and syncing your specific IdP groups, remove users from these default groups to ensure that SCIM remains the single source of truth for permissions and licensing. Once all users have been removed, you can also delete the groups altogether. #### Automated license mapping Automating license assignments is available for Okta only. It's a common strategy to reduce administrative overhead. For SSO-based license mapping (for example, Entra ID), see [Mapped configuration](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration). ###### Define IdP groups A common strategy involves defining two primary groups in your IdP, for example: * `dbt_developers` * `dbt_read_only` ###### Fundamental licensing rules Understanding these rules will help you plan your group structure in Okta: * **Default assignment:** When someone new is added to your account, they automatically receive a Developer license unless you configure otherwise. * **Mapping basis:** The license someone receives is determined by which groups they belong to in your identity provider (Okta) — not by groups you create in dbt platform. In other words, your Okta groups are the source of truth. When you add or remove someone from a group in Okta, their license in dbt platform updates automatically to match. ###### Mapping logic and precedence With SSO license mapping, the Developer license takes precedence over all other licenses. With SCIM license mapping (Okta), precedence depends on your configuration — whether you assign the license attribute directly to the user or derive it from group membership through the expression in your Okta Profile Editor. When you use groups like `dbt_developers` and `dbt_read_only`, a user might be in one group, both groups, or neither. The following table shows one common scenario (group-based mapping) and can be useful when you're deciding how to structure your groups or troubleshooting why someone received a particular license. Users in the Enterprise default group **Member** are assigned a **Developer** license by default. Until you remove users from **Member** (per the [best practice](#enterprise-default-groups) earlier), that default applies when they're not in any IdP license-mapping groups. Once you enable SCIM license mapping, the IdP group mapping overrides the Member default. **SSO license mapping (IdP group-based):** | In `dbt_developers` group? | In `dbt_read_only` group? | License assigned | | -------------------------- | ------------------------- | --------------------------------------------------- | | No | No | Developer (Member default or default for new users) | | No | Yes | Read-Only | | Yes | No | Developer | | Yes | Yes | Developer (Developer takes precedence) | #### Add license type attribute for Okta To add the attribute for license types to your Okta environment: 1. From your Okta application, navigate to the **Provisioning** tab, scroll down to **Attribute Mappings**, and click **Go to Profile Editor**. 2. Click **Add Attribute**. 3. Configure the attribute fields as follows (the casing should match for each value): * **Data type:** `string` * **Display name:** `License Type` * **Variable name:** `licenseType` * **External name:** `licenseType` * **External namespace:** `urn:ietf:params:scim:schemas:extension:dbtLabs:2.0:User` * **Description:** An arbitrary string of your choosing. * **Enum:** Select the box for **Define enumerated list of values**. * **Attribute members:** Add the initial attribute and then click **Add another** until each license type is defined. We recommend adding all of the values even if you don't use them today, so they'll be available in the future. Refer to the following table for the values you can use. | Display name | Value | | ------------- | ----------- | | **IT** | `it` | | **Analyst** | `analyst` | | **Developer** | `developer` | | **Read Only** | `read_only` | The **Analyst** license is only available on [select plans](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). It is not available for new purchase. * **Attribute type:** Group [![Enter the fields as they appear in the image. Ensure the cases match.](/img/docs/dbt-platform/access-control/scim-license-attributes.png?v=2 "Enter the fields as they appear in the image. Ensure the cases match.")](#)Enter the fields as they appear in the image. Ensure the cases match. 4. **Save** the attribute mapping. Users can now have license types set in their profiles and when they are being provisioned. [![Set the license type for the user in their Okta profile.](/img/docs/dbt-platform/access-control/scim-license-provisioning.png?v=2 "Set the license type for the user in their Okta profile.")](#)Set the license type for the user in their Okta profile. #### Automate license assignments with Okta groups To automate seat assignments in Okta, use the Profile Editor to map Okta group memberships to dbt license types. 1. Create groups in Okta, for example: * `dbt_developers` * `dbt_read_only` 2. Within the dbt app Profile Editor in Okta, create mapping rules for Okta users to dbt app users: * **Attribute:** `licenseType` * **Logic (Expression):** `(isMemberOfGroupName("dbt_developers") ? 'developer' : 'read_only')` * **Default behavior:** Users not in the `dbt_developers` group will default to Read-Only. Adding or removing users from these Okta groups automatically updates their dbt app profile and triggers a SCIM update to synchronize the `licenseType` in dbt. Admins also have the option of using **Manual Push** in the Okta app to synchronize the changes. --- ##### Migrating to Auth0 for SSO dbt platform | Enterprise, Enterprise+ dbt Labs is partnering with Auth0 to bring enhanced features to dbt's single sign-on (SSO) capabilities. Auth0 is an identity and access management (IAM) platform with advanced security features, and it will be leveraged by dbt. These changes will require some action from customers with SSO configured in dbt today, and this guide will outline the necessary changes for each environment. If you have not yet configured SSO in dbt, refer instead to our setup guides for [SAML](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-saml-2.0.md), [Okta](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-okta.md), [Google Workspace](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-google-workspace.md), or [Microsoft Entra ID (formerly Azure AD)](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-microsoft-entra-id.md) single sign-on services. #### Start the migration The Auth0 migration feature is being rolled out incrementally to customers who have SSO features already enabled. When the migration option has been enabled on your account, you will see **SSO Update Required** on the right side of the menu bar, near the settings icon. Alternatively, you can start the process by clicking your account name at the bottom left-hand menu and going to **Account settings** > **SSO & SCIM**. Click the **Begin Migration** button to start. vanity urls Don't use vanity URLs when configuring the SSO settings. You need to use the generic URL provided in the SSO settings for your environment. For example, if your vanity URL is `cloud.MY_COMPANY.getdbt.com`, configure `auth.cloud.getdbt.com` as `<YOUR_AUTH0_URI>`. There are two fields in the SSO settings that you need for the migration: * **Single sign-on URL:** This will be in the format of your login URL `https://<YOUR_AUTH0_URI>/login/callback?connection=<SLUG>` * **Audience URI (SP Entity ID):** This will be in the format `urn:auth0:<YOUR_AUTH0_ENTITYID>:<SLUG>` Once you have opted to begin the migration process, the following steps will vary depending on the configured identity provider. You can just skip to the section that's right for your environment. These steps only apply to customers going through the migration; new setups will use the existing [setup instructions](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md). #### SAML 2.0 SAML 2.0 users must update a few fields in the SSO app configuration to match the new Auth0 URL and URI. You can approach this by editing the existing SSO app settings or creating a new one to accommodate the Auth0 settings. One approach isn't inherently better, so you can choose whichever works best for your organization. ##### SAML 2.0 and Okta The Okta fields that will be updated are: * Single sign-on URL — `https://<YOUR_AUTH0_URI>/login/callback?connection=<SLUG>` * Audience URI (SP Entity ID) — `urn:auth0:<YOUR_AUTH0_ENTITYID>:<SLUG>` Below are sample steps to update. You must complete all of them to ensure uninterrupted access to dbt and you should coordinate with your identity provider admin when making these changes. 1. Replace `<SLUG>` with your account’s login URL slug. Here is an example of an updated SAML 2.0 setup in Okta. [![Okta configuration with new URL](/img/docs/dbt-platform/access-control/new-okta-config.png?v=2 "Okta configuration with new URL")](#)Okta configuration with new URL 2. Save the configuration, and your SAML settings will look something like this: [![New Okta configuration completed](/img/docs/dbt-platform/access-control/new-okta-completed.png?v=2 "New Okta configuration completed")](#)New Okta configuration completed 3. Toggle the `Enable new SSO authentication` option to ensure the traffic is routed correctly. *The new SSO migration action is final and cannot be undone.* 4. Save the settings and test the new configuration using the SSO login URL provided on the settings page. ##### SAML 2.0 and Entra ID The Entra ID fields that will be updated are: * Single sign-on URL — `https://<YOUR_AUTH0_URI>/login/callback?connection=<SLUG>` * Audience URI (SP Entity ID) — `urn:auth0:<YOUR_AUTH0_ENTITYID>:<SLUG>` The new values for these fields can be found in dbt by navigating to **Account settings** > **SSO & SCIM**. 1. Replace `<SLUG>` with your organization’s login URL slug. 2. Locate your dbt SAML2.0 app in the **Enterprise applications** section of Azure. Click **Single sign-on** on the left side menu. 3. Edit the **Basic SAML configuration** tile and enter the values from your account: * Entra ID **Identifier (Entity ID)** = dbt **Audience URI (SP Entity ID)** * Entra ID **Reply URL (Assertion Consumer Service URL)** = dbt **Single sign-on URL** [![Editing the SAML configuration window in Entra ID](/img/docs/dbt-platform/access-control/edit-entra-saml.png?v=2 "Editing the SAML configuration window in Entra ID")](#)Editing the SAML configuration window in Entra ID 4. Save the fields and the completed configuration will look something like this: [![Completed configuration of the SAML fields in Entra ID](/img/docs/dbt-platform/access-control/entra-id-saml.png?v=2 "Completed configuration of the SAML fields in Entra ID")](#)Completed configuration of the SAML fields in Entra ID 5. Toggle the `Enable new SSO authentication` option to ensure the traffic is routed correctly. *The new SSO migration action is final and cannot be undone.* 6. Save the settings and test the new configuration using the SSO login URL provided on the settings page. #### Microsoft Entra ID Microsoft Entra ID admins using OpenID Connect (ODIC) will need to make a slight adjustment to the existing authentication app in the Azure portal. This migration does not require that the entire app be deleted or recreated; you can edit the existing app. Start by opening the Azure portal and navigating to the Microsoft Entra ID overview. Below are steps to update. You must complete all of them to ensure uninterrupted access to dbt and you should coordinate with your identity provider admin when making these changes. 1. Click **App Registrations** on the left side menu. [![Select App Registrations](/img/docs/dbt-platform/access-control/aad-app-registration.png?v=2 "Select App Registrations")](#)Select App Registrations 2. Select the proper **dbt** app (name may vary) from the list. From the app overview, click on the hyperlink next to **Redirect URI** [![Click the Redirect URI hyperlink](/img/docs/dbt-platform/access-control/app-overview.png?v=2 "Click the Redirect URI hyperlink")](#)Click the Redirect URI hyperlink 3. In the **Web** pane with **Redirect URIs**, click **Add URI** and enter the appropriate `https://<YOUR_AUTH0_URI>/login/callback`. Save the settings and verify it is counted in the updated app overview. [![Enter new redirect URI](/img/docs/dbt-platform/access-control/redirect-URI.png?v=2 "Enter new redirect URI")](#)Enter new redirect URI 4. Navigate to the dbt environment and open the **Account settings**. Click the **SSO & SCIM** option from the left-side menu and click the **Edit** option from the right side of the SSO pane. The **domain** field is the domain your organization uses to login to Microsoft Entra ID. Toggle the **Enable New SSO Authentication** option and **Save**. *Once this option is enabled, it cannot be undone.* Domain authorization You must complete the domain authorization before you toggle `Enable New SSO Authentication`, or the migration will not complete successfully. #### Google Workspace Google Workspace admins updating their SSO APIs with the Auth0 URL won't have to do much if it is an existing setup. This can be done as a new project or by editing an existing SSO setup. No additional scopes are needed since this is migrating from an existing setup. All scopes were defined during the initial configuration. Below are steps to update. You must complete all of them to ensure uninterrupted access to dbt and you should coordinate with your identity provider admin when making these changes. 1. Open the [Google Cloud console](https://console.cloud.google.com/) and select the project with your dbt single sign-on settings. From the project page **Quick Access**, select **APIs and Services** [![Google Cloud Console](/img/docs/dbt-platform/access-control/google-cloud-sso.png?v=2 "Google Cloud Console")](#)Google Cloud Console 2. Click **Credentials** from the left side pane and click the appropriate name from **OAuth 2.0 Client IDs** [![Select the OAuth 2.0 Client ID](/img/docs/dbt-platform/access-control/sso-project.png?v=2 "Select the OAuth 2.0 Client ID")](#)Select the OAuth 2.0 Client ID 3. In the **Client ID for Web application** window, find the **Authorized Redirect URIs** field and click **Add URI** and enter `https://<YOUR_AUTH0_URI>/login/callback`. Click **Save** once you are done. 4. *You will need a person with Google Workspace admin privileges to complete these steps in dbt*. In dbt, navigate to the **Account settings**, click on **SSO & SCIM**, and then click **Edit** on the right side of the **Single sign-on** pane. Toggle the **Enable New SSO Authentication** option and select **Save**. This will trigger an authorization window from Google that will require admin credentials. *The migration action is final and cannot be undone*. Once the authentication has gone through, test the new configuration using the SSO login URL provided on the settings page. Domain authorization You must complete the domain authorization before you toggle `Enable New SSO Authentication`, or the migration will not complete successfully. --- ##### Multi-factor authentication dbt platform important dbt enforces multi-factor authentication (MFA) for all users with username and password credentials. If MFA is not set up, you will see a notification bar prompting you to configure one of the supported methods when you log in. If you do not, you will have to configure MFA upon subsequent logins, or you will be unable to access dbt. dbt provides multiple options for multi-factor authentication (MFA), which adds an extra layer of security to username and password logins. MFA is available across dbt plans for users with username and password logins only. The available MFA methods are: * SMS verification code * Currently, only phone numbers with the North American Numbering Plan (NANP) +1 country code are supported. * Authenticator app * Webauthn-compliant security key #### Configuration You can only have one of the three MFA methods configured per user. These are enabled at the user level, not the account level. 1. Navigate to the **Account settings** and under **Your profile** click on **Password & Security**. Click **Enroll** next to the preferred method. [![List of available MFA enrollment methods in dbt.](/img/docs/dbt-platform/mfa-enrollment.png?v=2 "List of available MFA enrollment methods in dbt.")](#)List of available MFA enrollment methods in dbt. Choose the next steps based on your preferred enrollment selection:  SMS verification code 2. Select the +1 country code, enter your phone number in the field, and click **Continue**. [![The phone number selection, including a dropdown for country code.](/img/docs/dbt-platform/sms-enter-phone.png?v=2 "The phone number selection, including a dropdown for country code.")](#)The phone number selection, including a dropdown for country code. 3. You will receive an SMS message with a six digit code. Enter the code in dbt. [![Enter the 6-digit code.](/img/docs/dbt-platform/enter-code.png?v=2 "Enter the 6-digit code.")](#)Enter the 6-digit code.  Authenticator app 2. Open your preferred authentication app (like Google Authenticator) and scan the QR code. [![Example of the user generated QR code.](/img/docs/dbt-platform/scan-qr.png?v=2 "Example of the user generated QR code.")](#)Example of the user generated QR code. 3. Enter the code provided for "dbt Labs: YOUR\_EMAIL\_ADDRESS" from your authenticator app into the field in dbt.  Webauthn-compliant security key 2. Follow the instructions in the modal window and click **Use security key**. [![Example of the Security Key activation window.](/img/docs/dbt-platform/create-security-key.png?v=2 "Example of the Security Key activation window.")](#)Example of the Security Key activation window. 3. Scan the QR code or insert and touch activate your USB key to begin the process. Follow the on-screen prompts. 4) You will be given a backup passcode, store it in a secure location. This key will be useful if the MFA method fails (like a lost or broken phone). #### Account Recovery When setting up MFA, ensure that you store your recovery codes in a secure location, in case your MFA method fails. If you are unable to access your account, reach out to <support@getdbt.com> for further support. You may need to create a new account if your account cannot be recovered. If possible, it's recommended to configure multiple MFA methods so that if one fails, there is a backup option. #### Disclaimer The terms below apply to dbt’s MFA via SMS program, that dbt Labs (“dbt Labs”, “we”, or “us”) uses to facilitate auto sending of authorization codes to users via SMS for dbt log-in requests. Any clients of dbt Labs that use dbt Labs 2FA via SMS program (after password is input) are subject to the dbt Labs privacy policy, the client warranty in TOU Section 5.1 second paragraph that Client's use will comply with the Documentation (or similar language in the negotiated service agreement between the parties) and these terms: (1) The message frequency is a maximum of 1 message per user login; (2) Message and data rates may apply; (3) Carriers are not liable for delayed or undelivered messages; (4) For help, please reply HELP to the SMS number from which you receive the log-in authorization code(s); (5) To opt-out of future SMS messages, please reply STOP to the SMS number from which you receive the log-in authorization code(s). We encourage you to enable an alternate 2FA method before opting-out of SMS messages or you might not be able to log into your account. Further questions can be submitted to <support@getdbt.com>. --- ##### OAuth overview dbt platform | Enterprise, Enterprise+ OAuth in dbt platform lets developers authorize their user credentials with a data platform using that platform’s login or its single sign-on (SSO) instead of storing usernames and passwords in dbt platform. This improves security and aligns with how your team already signs in to the warehouse. **Who it’s for:** Account admins and developers on Enterprise or Enterprise+ plans who connect to supported data platforms. Admins configure the OAuth integration; developers complete a one-time authorization in their profile. #### Available integrations Pick the best documentation for your platform to configure OAuth and have developers authorize their credentials. | Platform | Doc | Description | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | **Snowflake** | [Set up Snowflake OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md) | Authorize user credentials with Snowflake (or Snowflake SSO). | | **Databricks** | [Set up Databricks OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-databricks-oauth.md) | Authorize user credentials with Databricks. | | **BigQuery** | [Set up BigQuery OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md) | Authorize user credentials with Google. | | **Snowflake (external)** | [Set up external OAuth with Snowflake](https://docs.getdbt.com/docs/platform/manage-access/snowflake-external-oauth.md) | Use an external identity provider (IdP) for Snowflake OAuth. | | **Redshift (external)** | [Set up external OAuth with Redshift](https://docs.getdbt.com/docs/platform/manage-access/redshift-external-oauth.md) | Use an external identity provider (IdP) for Redshift OAuth. | --- ##### SCIM FAQs and troubleshooting dbt platform | Enterprise, Enterprise+ Find answers to common questions about configuring and using SCIM provisioning in dbt platform, plus guidance for resolving common issues. #### FAQs  Do I need an Account Admin group before enabling SCIM? **Account Admin group required:** Before enabling SCIM on your dbt account, you must create a dbt Account Admin group in your IdP, and a group with a matching name in your dbt account. Assign the dbt group the [Account Admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) permission set. Without this group, your users could be removed from the default dbt 'Owner' group once SCIM provisioning is enabled and would lose admin access to your dbt account. The names of these two groups can match your IdP group naming scheme, but the names for both the dbt and IdP group must be the same value.  Do the userName and email.value fields have to be the same value for SCIM to work? Yes. Both must match the email address the user uses to sign in to dbt platform (email value on the user object in your IdP that's used to sign in). If they don't match, a validation error will occur during provisioning.  What is the recommended workflow for onboarding new users through SCIM? ###### Entra ID Once SCIM provisioning is configured in Entra ID according to the schema requirements, provisioning begins automatically. Entra ID will sync users and groups assigned to the Entra app with dbt platform. **For users:** * Matching is based on the `userName` (and `email.value`) field, which should be set to the user's email — the same email used in dbt platform and expected at sign-in. * If a user with that email already exists in dbt platform, they will be linked and become SCIM-managed. * If no matching user exists, a new user will be provisioned and invited through email. **For groups:** * Matching is based on the group name. * If a group with the same name exists in dbt platform, it will be linked and become SCIM-managed. * If no match exists, a new group will be created through SCIM. **Important considerations:** * Entra ID does not support importing existing users or groups into the app for SCIM — users and groups must already exist in Entra ID and be assigned to the app. * To ensure users and groups become SCIM-managed, they must exist in both Entra and dbt platform with matching identifiers (email for users, name for groups). * After setup, ongoing syncs automatically provision and manage any newly assigned users or groups in Entra. ###### Okta Unlike Entra ID, Okta does not run continuous full syncs. You trigger initial provisioning manually, then Okta continues provisioning incrementally as changes are made. When you enable SCIM on an existing Okta SSO app, trigger the initial sync (if there are existing users in the Okta app) using the **Provision Users** button in the Okta Assignments tab. This runs a one-time sync of users to dbt platform. Okta doesn't provide a native way to re-run this full sync — which means reprocessing typically requires removing and re-adding users. **For users:** User matching is based on the `userName` (email) field, which must match the email used in dbt platform: * If a match is found, Okta links the user and marks them as SCIM-managed. * If no match is found, Okta provisions a new user. * If users exist in dbt platform but not in Okta, you can import them into Okta to keep both systems aligned. **For groups:** * After initial user sync, assign new users to the Okta app to provision them automatically into dbt platform. * Manage group memberships with Okta's Push Groups feature, which pushes groups and memberships into dbt platform. For new Okta SSO applications with no assigned users, either manually assign users to the app to trigger provisioning, or import users from dbt platform into Okta first. Users must exist in both systems with matching emails to link correctly and avoid duplicates. For larger rollouts, consider working with your IdP admin to plan based on your setup and [SCIM license mapping](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md) to reduce manual steps.  Do SSO group mappings still apply when SCIM is enabled? No. SSO group mappings do not apply to SCIM-managed users. For SCIM-managed users, group membership is controlled by your IdP. SSO group mappings only apply to users who authenticate with SSO and are not SCIM-managed. This means that if you have a dbt group with SSO mappings, those mappings will not be applied to users who have been provisioned through SCIM.  What does "Allow manual updates" mean? The **Allow manual updates** toggle determines whether admins can manually modify SCIM-managed users and groups in dbt platform, including sending invites. * **Disabled (default)**: Your IdP remains the source of truth for SCIM-managed users and groups. Manual changes to SCIM-managed users in dbt platform are blocked. This is the recommended setting because any manual changes made while enabled can be overwritten by later SCIM updates. * **Enabled:** Admins can make manual changes to users in dbt platform alongside SCIM. This can be useful during initial setup and testing, but manual changes do not prevent SCIM from overriding them.  What happens to existing users and groups when I enable SCIM? Enabling SCIM does not automatically convert existing users and groups to SCIM-managed status. Your IdP only manages users who have been explicitly assigned to the app and provisioned through SCIM. To bring existing users and groups under SCIM management, assign them to the dbt platform app in your IdP and trigger a sync. Until a user is provisioned through SCIM, they remain unmanaged and are unaffected by SCIM sync operations.  What happens when a user's email address changes in my IdP when SCIM is enabled? When a SCIM-managed user's email is updated in the IdP, dbt platform receives a SCIM request to update it. A confirmation email is sent to the new address, and once accepted, the change takes effect in dbt platform.  Does SCIM support automatic license assignment? SCIM-native license mapping (using a SCIM attribute) is supported for Okta only. For Okta license mapping setup, refer to [Manage user licenses with SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md). SCIM license mapping with Entra ID *may* be possible using [custom attributes](https://learn.microsoft.com/en-us/entra/fundamentals/custom-security-attributes-overview), depending on your Entra ID configuration. Because this setup varies by environment, we don't provide a standard configuration or troubleshooting guidance for it. **Use this approach at your own risk.** For a simpler, supported approach, use SSO-based Active Directory group → license mapping alongside your Entra ID SCIM setup. This works so long as the **Ignore dbt license mapping** toggle (found in **Account settings > SSO & SCIM**) stays disabled. * **Disabled (recommended)**: dbt platform honors your [SSO license mappings](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration) based on Entra ID group membership. * **Enabled:** dbt platform ignores your SSO license mappings. Because SCIM-native license mapping isn't a supported configuration for Entra ID, enabling this toggle effectively removes license mapping entirely for Entra ID users.  Can I use Okta for SSO and Entra ID for SCIM (or vice versa)? SSO and SCIM should be configured using the same IdP. Using different providers — for example, Okta for SSO and Entra ID for SCIM — can cause discrepancies in user state and unintended behavior. If your organization uses separate IdPs for authentication and directory management, contact your account team to discuss your options. *** #### Troubleshooting  There is a "All users must have licenses on the account" error This error occurs when a SCIM group push includes a user who hasn't been licensed in dbt platform yet — typically because they haven't accepted their invitation. **Steps to resolve:** 1. Identify the user(s) causing the error from your IdP's provisioning logs. 2. Check whether those users have accepted their dbt platform invitation. Users are not licensed until they complete this step. 3. Once the user accepts and signs in, retry the group push from your IdP. 4. If the invitation has expired, remove the user from the push group temporarily, re-invite them using dbt platform, have them accept, then re-add them to the group.  Existing users and groups are not becoming SCIM-managed after enabling SCIM After enabling SCIM and completing the initial sync, pre-existing dbt platform users and groups do not show as SCIM-managed. **Why this happens:** SCIM provisioning links an IdP identity to a dbt platform user record. Users created before SCIM was enabled don't have this link unless the IdP explicitly provisions them through SCIM. **Steps to resolve:** 1. In your IdP, assign existing users to the dbt platform SCIM application. 2. Trigger a provisioning sync. The IdP will attempt to match users by their `userName` (typically their email address) and establish the SCIM-managed link. Entra ID note For Entra ID, note that the provisioning sync is one-way (push only) — there is no import option to pull existing dbt users into Entra as a managed identity. 3. If users aren't being matched correctly after a sync, confirm that the `userName` and `email.value` attributes in your IdP exactly match the email on the user's dbt platform account, including case.  Entra SCIM provisioning fails due to IP allowlisting If your dbt platform account has **IP restrictions** enabled, Entra's SCIM provisioning requests may be blocked. Azure's provisioning service IPs rotate approximately every two weeks and can't be statically allowlisted. **Recommended approach:** 1. Filter to the `AzureActiveDirectory` service tag in [Azure's published IP ranges JSON](https://www.microsoft.com/en-us/download/details.aspx?id=56519) rather than allowlisting all Azure IPs. 2. Use the dbt platform Admin API with a service token to update your IP allowlist on a schedule — for example, a weekly script that pulls the current `AzureActiveDirectory` ranges and updates your allowlist through the API. Contact <support@getdbt.com> for guidance on using the Admin API for allowlist management. --- ##### Self-service Starter account permissions dbt platform | Starter Self-service Starter accounts are a quick and easy way to get dbt up and running for a small team. For teams looking to scale and access advanced features like SSO, group management, and support for larger user bases, upgrading to an [Enterprise-tier](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) account unlocks these capabilities — if you're interested in upgrading, contact [dbt Labs today](https://www.getdbt.com/contact). #### Groups and permissions Groups determine a user's permission and there are three groups are available for the Starter plan dbt accounts: Owner, Member, and Everyone. The first Owner user is the person who created the dbt account. New users are added to the Member and Everyone groups when they onboard but this can be changed when the invitation is created. These groups only affect users with a [Developer license](#licenses) assigned. The group access permissions are as follows: * **Owner** — Full access to account features. * **Member** — Robust access to the account with restrictions on features that can alter billing or security. * **Everyone** — A catch-all group for all users in the account. This group does not have any permission assignments beyond the user's profile. Users must be assigned to either the Member or Owner group to work in dbt. #### Licenses You assign licenses to every user onboarded into dbt. You only assign Developer-licensed users to the Owner and Member groups. The groups have no impact on Read-only or IT licensed users. There are three license types: * **Developer** — The default license. Developer licenses don't restrict access to any features, so users with this license should be assigned to either the Owner or Member group. You're allotted up to 5 developer licenses per account. * **Read-Only** — Read-only access to your project, including environments and Catalog. Doesn't have access to account settings at all. Functions the same regardless of group assignments. Non-developer [plans](https://www.getdbt.com/pricing) may provision an unlimited number of Read-Only licenses at no additional charge. * **IT** — Partial access to the account settings including users, integrations, billing, and API settings. Cannot create or edit connects or access the project at all. Functions the same regardless of group assignments. You're allocated 1 seat per account. See [Seats and Users](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) for more information on the impact of licenses on these permissions. License types override group permissions **User license types always override their assigned group permission sets.** For example, a user with a Read-Only license cannot perform administrative actions, even if they belong to an Account Admin group. This ensures that license restrictions are always enforced, regardless of group membership. #### Table of groups, licenses, and permissions Key: * (W)rite — Create new or modify existing. Includes `send`, `create`, `delete`, `allocate`, `modify`, and `read`. * (R)ead — Can view but can not create or change any fields. * No value — No access to the feature. Permissions: * [Account-level permissions](#account-permissions-for-account-roles) — Permissions related to management of the dbt account. For example, billing and account settings. * [Project-level permissions](#project-permissions-for-account-roles) — Permissions related to the projects in dbt. For example, Catalog and the Studio IDE. The following tables outline the access that users have if they are assigned a Developer license and the Owner or Member group, Read-only license, or IT license. ###### Account permissions for account roles | Account-level permission | Owner | Member | Read-only license | IT license | | ------------------------ | ----- | ------ | ----------------- | ---------- | | Account settings | W | W | - | W | | Billing | W | - | - | W | | Invitations | W | W | - | W | | Licenses | W | R | - | W | | Users | W | R | - | W | | Project (create) | W | W | - | W | | Connections | W | W | - | W | | Service tokens | W | - | - | W | | Webhooks | W | W | - | - | ###### Project permissions for account roles | Project-level permission | Owner | Member | Read-only | IT license | | ------------------------------- | ----- | ------ | --------- | ---------- | | Adapters | W | W | R | - | | Connections | W | W | R | W | | Credentials | W | W | R | - | | Custom env. variables | W | W | R | - | | Develop (Studio IDE or dbt CLI) | W | W | - | - | | Environments | W | W | R | - | | Jobs | W | W | R | - | | Catalog | W | W | R | - | | Permissions | W | R | - | - | | Profile | W | W | R | - | | Projects | W | W | R | - | | Repositories | W | W | R | - | | Runs | W | W | R | - | | Semantic Layer Config | W | W | R | - | --- ##### Set up BigQuery OAuth dbt platform | Enterprise, Enterprise+ Enterprise-tier feature This guide describes a feature available in the dbt platform Enterprise and Enterprise+ plans. If you’re interested in learning more about our Enterprise-tier plans, contact us at <sales@getdbt.com>. The dbt platform supports [OAuth](https://cloud.google.com/bigquery/docs/authentication) with BigQuery, providing an additional layer of security for dbt enterprise users. #### Set up BigQuery native OAuth When BigQuery OAuth is enabled for a dbt platform project, all dbt platform developers must authenticate with BigQuery to access development tools, such as the Studio IDE. To set up BigQuery OAuth in the dbt platform, a BigQuery admin must: 1. [Locate the redirect URI value](#locate-the-redirect-uri-value) in the dbt platform. 2. [Create a BigQuery OAuth 2.0 client ID and secret](#creating-a-bigquery-oauth-20-client-id-and-secret) in BigQuery. 3. [Configure the connection](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#configure-the-connection-in-dbt) in the dbt platform. To use BigQuery in the Studio IDE, all developers must: 1. [Authenticate to BigQuery](#authenticating-to-bigquery) in their profile credentials. ##### Locate the redirect URI value To get started, locate the connection's redirect URI for configuring BigQuery OAuth. To do so: 1. Navigate to your account name, above your profile icon on the left side panel. 2. Select **Account settings** from the menu. 3. From the left sidebar, select **Connections**. 4. Click the BigQuery connection. 5. Locate the **Redirect URI** field under the **Development OAuth** section. Copy this value to your clipboard to use later on. [![Accessing the BigQuery OAuth configuration in dbt](/img/docs/dbt-platform/using-dbt-platform/dbt-platform-enterprise/BQ-auth/dbt-cloud-bq-id-secret-02.png?v=2 "Accessing the BigQuery OAuth configuration in dbt")](#)Accessing the BigQuery OAuth configuration in dbt ##### Creating a BigQuery OAuth 2.0 client ID and secret note The dbt platform will request the required BigQuery scopes automatically during the OAuth flow. [Configuring scopes in the BigQuery OAuth consent screen](https://developers.google.com/workspace/guides/configure-oauth-consent) is optional and not required for dbt platform to connect to BigQuery. Required scopes are requested and approved when users authenticate with OAuth. To get started, you need to create a client ID and secret for [authentication](https://cloud.google.com/bigquery/docs/authentication) with BigQuery. This client ID and secret will be stored in the dbt platform to manage the OAuth connection between the dbt platform users and BigQuery. 1. In the BigQuery console, navigate to **APIs & Services** and select **Credentials**: [![BigQuery navigation to credentials](/img/docs/dbt-platform/using-dbt-platform/dbt-platform-enterprise/BQ-auth/BQ-nav.gif?v=2 "BigQuery navigation to credentials")](#)BigQuery navigation to credentials On the **Credentials** page, you can see your existing keys, client IDs, and service accounts. 2. Set up an [OAuth consent screen](https://support.google.com/cloud/answer/6158849) if you haven't already. 3. Click **+ Create Credentials** at the top of the page and select **OAuth client ID**. 4. Fill in the client ID configuration. **Authorized JavaScript Origins** are not applicable. 5. Add an item to **Authorized redirect URIs** and replace `REDIRECT_URI` with the value you copied to your clipboard earlier from the connection's **OAuth 2.0 Settings** section in the dbt platform: | Config | Value | | ---------------------------- | --------------- | | **Application type** | Web application | | **Name** | dbt platform | | **Authorized redirect URIs** | `REDIRECT_URI` | <br /> 6. Click **Create** to create the BigQuery OAuth app and see the app client ID and secret values. These values are available even if you close the app screen, so this isn't the only chance you have to save them. [![Create an OAuth app in BigQuery](/img/docs/dbt-platform/using-dbt-platform/dbt-platform-enterprise/BQ-auth/bq-oauth-app-02.png?v=2 "Create an OAuth app in BigQuery")](#)Create an OAuth app in BigQuery ##### Configure the Connection in dbt Now that you have an OAuth app set up in BigQuery, you'll need to add the client ID and secret to the dbt platform. To do so: 1. Navigate back to the Connection details page, as described in [Locate the redirect URI value](#locate-the-redirect-uri-value). 2. Add the client ID and secret from the BigQuery OAuth app under the **OAuth 2.0 Settings** section. 3. Enter the BigQuery token URI. The default value is `https://oauth2.googleapis.com/token`. ##### Authenticating to BigQuery Once the BigQuery OAuth app is set up for a dbt platform project, each dbt platform user will need to authenticate with BigQuery in order to use the Studio IDE. To do so: 1. Navigate to your account name, above your profile icon on the left side panel. 2. Select **Account settings** from the menu. 3. From the left sidebar, select **Credentials**. 4. Choose the project from the list. 5. Ensure the **Authentication Method** is set to **Native OAuth**. 6. Select **Connect to BigQuery**. [![Navigate to account settings in dbt platform.](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/1-settings-gear-icon.png?v=2 "Navigate to account settings in dbt platform.")](#)Navigate to account settings in dbt platform. [![Connect BigQuery account.](/img/docs/dbt-platform/connect-to-bigquery.png?v=2 "Connect BigQuery account.")](#)Connect BigQuery account. You will then be redirected to BigQuery and asked to approve the drive, cloud platform, and BigQuery scopes, unless the connection is less privileged. [![BigQuery access request](/img/docs/dbt-platform/using-dbt-platform/dbt-platform-enterprise/BQ-auth/BQ-access.png?v=2 "BigQuery access request")](#)BigQuery access request Select **Allow**. This redirects you back to the dbt platform. You are now an authenticated BigQuery user and can begin accessing dbt development tools. #### Set up BigQuery Workload Identity Federation [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Workload Identity Federation (WIF) allows application workloads, running outside the dbt platform, to act as a service account without the need to manage service accounts or other keys for deployment environments. The following instructions will enable you to authenticate your BigQuery connection in the dbt platform using WIF. Currently, Microsoft Entra ID is the only supported identity provider (IdP). If you need additional IdP support, please contact your account team. ##### 1. Set up Entra ID Create an app in Entra where dbt will request access tokens when authenticating to BigQuery via the workload identity pool: 1. From the **app registrations** screen, click **New registration**. 2. Give the app a name that makes it easily identifiable. 3. Ensure **Supported account types** are set to “Accounts in this organizational directory only (Org name - Single Tenant).” 4. Click **Register** to see the application’s overview screen. 5. From the **app overview**, click **Expose an API** in the left menu. 6. Click **Add** next to Application ID URI. The field will automatically populate. 7. Click **Save**. 8. (Optional) To include the `sub` claim in tokens issued by this application, configure [optional claims in Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims?tabs=appui).<br />The `sub` (subject) claim uniquely identifies the user or service principal for whom the token is issued.<br />When you configure service account impersonation in GCP, the Workload Identity Federation mapping uses this `sub` value to verify the identity of the calling Entra application. 9. (Optional but recommended) Test your Entra ID configuration by requesting a token: Run the following command, replacing `<client-id>`, `<client-secret>`, `<application-ID-URI>`, and `<tenant-id>` with your actual values: ```bash curl -X POST -H "Content-Type: application/x-www-form-urlencoded" \ -d 'client_id=<client-id> \ &scope=<application-ID-URI>/.default \ &client_secret=<client-secret> \ &grant_type=client_credentials' \ 'https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token' ``` The response will include an `access_token`. You can decode this token using [jwt.io](https://jwt.io) to view the `sub` claim value. Workload Identity Federation utilizes a machine-to-machine OAuth flow that is unattended by the user; as such, a redirect URI won't need to be set for the application. Step 3 in this section is crucial because it determines the audience for tokens issued from the app and informs the workpool in GCP whether the calling application has permission to access the resources guarded by the workpool. * **Related documentation:** [GCP — Prepare your external identity provider](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#create) ##### 2. Create a Workpool and Workpool Provider in GCP 1. In your GCP account's main menu, navigate to **IAM & Admin** and click the **Workload Identity Federation** option (not to be confused with the **Work\_force\_ Identity Federation** option directly adjacent). 2. If you haven’t created a workpool yet, click **Get started** or create a new workpool (use button near the top of the page). 3. Give the workpool a name and description. Per the [GCP documentation](https://cloud.google.com/iam/docs/workload-identity-federation#pools), a new pool should be created for each non-Google Cloud environment that needs to access Google Cloud resources, such as development, staging, or production environments. The workpool should be named accordingly to make it easily identifiable in the future. 4. When creating your provider: * Set the type of the provider to **OpenID Connect (OIDC)**. * Name the provider something identifiable, like `Entra ID`. * Set the URL to <https://sts.windows.net/YOUR_TENANT_ID/>. This can be found in the token itself, if you decode it via jwt.io. You can also see a reference to the expected issuer URL for Entra in the [GCP documentation for WIF](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#create_the_workload_identity_pool_and_provider). * Replace `YOUR_TENANT_ID` with your tenant ID. * The tenant (provider) ID can be found in the app registration created in [section 1 of these instructions](#1-set-up-entra-id); it's called **Directory (tenant) ID** and can be found in the overview section for the application. * For **Audiences**, select **Allowed Audiences** and set the value to the **Application ID URI** that was defined for your Entra ID app. 5. Click **Continue**. 6. Under **Configure provider attributes**, set the mapping for `google.subject` to `assertion.sub`. 7. Click **Save**. ##### 3. Service Account Impersonation A workpool either uses a service account or is granted direct resource access to determine which resources a caller can access. The [GCP documentation](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#access) provides more detailed information on configuring both for your workpool. We chose the service account approach in our implementation because it offered greater flexibility. If you haven’t already, create a new service account: 1. From the main menu, select **IAM & Admin** 2. Click **Service Accounts**. 3. Click **Create service account**. Google recommends creating a service account per workload. 4. Assign the relevant roles you would like this service account to have. In our experience, `BigQuery Admin` is the default role with required access. Once you've created the service account, navigate back to the workpool you created in the previous step: 1. Click the **Grant Access** option at the top of the page. 2. Select **Grant access using Service Account Impersonation**. 3. Select the service account you just created. 4. Under **Select Principals**, set `subject` as the **Attribute Name**. For the **Attribute Value**, set it to the `sub` (subject) claim value from the Entra ID access token.  Obtain the sub value To obtain the `sub` value, request an access token from Entra ID. The `sub` claim is consistent across all tokens issued by this application: Run the following command, replacing `<client-id>`, `<client-secret>`, `<application-ID-URI>`, and `<tenant-id>` with your actual values: ```bash curl -X POST -H "Content-Type: application/x-www-form-urlencoded" \ -d 'client_id=<client-id> \ &scope=<application-ID-URI>/.default \ &client_secret=<client-secret> \ &grant_type=client_credentials' \ 'https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token' ``` The response will include an `access_token`. You can decode this token using [jwt.io](https://jwt.io) to view the `sub` claim value. ##### 4. Set up dbt To configure a BigQuery connection to use WIF authentication in the dbt platform, you must set up a custom OAuth integration configured with details from the Entra application used as your workpool provider in GCP. dbt platform: 1. Navigate to **Account settings** --> **Integrations**. 2. Scroll down to the section for **Custom OAuth Integrations** and create a new integration. 3. Fill out all fields with the appropriate information from your IdP environment. * The Application ID URI should be set to the expected audience claim on tokens issued from the Entra application. It will be the same URI your workpool provider has been configured to expect. * You do not have to add the Redirect URI to your Entra application. ##### 5. Create connections in dbt To get started, create a new connection in the dbt platform: 1. Navigate to **Account settings** --> **Connections**. 2. Click **New connection** and select **BigQuery** as the connection type. You will then see the option to select **BigQuery** or **BigQuery (Legacy)**. Select **BigQuery**. 3. For the **Deployment Environment Authentication Method**, select **Workload Identity Federation**. 4. Fill out the **Google Cloud Project ID** and any optional settings you need. 5. Select the OAuth Configuration you created in the previous section from the drop-down. 6. Configure your development connection: * [BigQuery OAuth](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md#bigquery-oauth) (recommended). * Set this up in the same connection as the one you're using for WIF under **`OAuth2.0 settings`** * Service JSON. * You must create a separate connection with the Service JSON configuration. ##### 6. Set up project To connect a new project to your WIF configuration: 1. Navigate to **Account settings** --> **Projects**. 2. Click **New project**. 3. Give your project a name and (optional) subdirectory path and click **Continue**. 4. Select the **Connection** with the WIF configuration. 5. Configure the remainder of the project with the appropriate fields. ##### 7. Set up deployment environment Create a new or updated environment to use the WIF connection. When you set your environment connection to the WIF configuration, you will then see two fields appear under the Deployment credentials section: * **Workload pool provider path:** This field is required for all WIF configurations. For example: * `//iam.googleapis.com/projects/<numeric_project_id>/locations/global/workloadIdentityPools/<workpool_name>/providers/<workpool_providername>` * **Service account impersonation URL:** Used only if you’ve configured your workpool to use a service account impersonation for accessing your BigQuery resources (as opposed to granting the workpool direct resource access to the BigQuery resources). For example: * `https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<serviceaccountemail>:generateAccessToken` To avoid errors from incorrect casing or typos, we recommend retrieving these values directly from GCP rather than constructing them manually: 1. In the GCP console, go to **IAM & Admin > Workload Identity Federation**. 2. Click your workload identity pool, then click the provider you created for this configuration. 3. Click **Download configuration** at the top of the provider detail page. 4. Open the downloaded JSON file and copy the following values: * The `audience` field — paste into the **Workload pool provider path** field in dbt. * The `service_account_impersonation_url` field — paste into the **Service account impersonation URL** field in dbt. If you don't already have a job based on the deployment environment with a connection set up for WIF, you should create one now. Once you've configured it with the preferred settings, run the job. #### FAQs Why does the BigQuery OAuth application require scopes to Google Drive? BigQuery supports external tables over both personal Google Drive files and shared files. For more information, refer to [Create Google Drive external tables](https://cloud.google.com/bigquery/docs/external-data-drive). #### Troubleshooting The following section provides troubleshooting steps for common issues with BigQuery OAuth connections. ###### Connection fails after granting Google permissions When connecting a BigQuery account, you may successfully sign in to Google and approve the requested permissions, but then see an error page (like `403 Account restricted`) instead of returning to the dbt platform. This typically appears as a server error message rather than the credentials page you started from. This can happen when your Google Workspace organization restricts which third-party applications can access Google services. Even though Google allowed you to sign in and view the consent screen, the organization's security policy blocked the final step of issuing credentials to the dbt platform. This troubleshooting section will explain Why this happens, how to resolve it, and what to do if it keeps happening.  1. Why this affects BigQuery OAuth Google classifies OAuth scopes into three tiers: non-sensitive, sensitive, and restricted. Applications that only request non-sensitive scopes (such as basic sign-in) typically work without additional admin approval. The BigQuery OAuth connection requires scopes that Google classifies as [restricted](https://support.google.com/cloud/answer/9110914), which always require explicit admin trust regardless of your organization's default policy for third-party apps. The BigQuery OAuth connection requests these restricted scopes: * `bigquery` — Required to run queries and access BigQuery resources * `cloud-platform` — Required for cross-project BigQuery access and service interoperability * `drive` — Required to support [BigQuery external tables over Google Drive](https://cloud.google.com/bigquery/docs/external-data-drive) Other applications you use may connect without this approval step because they likely request only non-sensitive or sensitive scopes that don't trigger the same restriction.  2. How to resolve it Your Google Workspace administrator must trust the OAuth client ID that your dbt platform BigQuery connection uses. Find this client ID in the dbt platform under **Account settings** > **Connections** > your BigQuery connection > **OAuth 2.0 Settings**. Once you have the client ID, ask your Google Workspace administrator to: 1. Sign in to the [Google Workspace Admin Console](https://admin.google.com). 2. Navigate to **Security** > **Access and data control** > **API controls** > **Manage third-party app access**. 3. Click **Add app** > **OAuth App Name Or Client ID**. 4. Enter the client ID from your dbt platform BigQuery connection. 5. Set the access level to **Trusted**. 6. Apply the setting to the relevant organizational unit or the entire organization. After the administrator completes these steps, retry the BigQuery connection from your credentials page in the dbt platform. For more details, see Google's documentation on [controlling third-party app access to Google Workspace data](https://support.google.com/a/answer/7281227).  3. If the issue persists If the connection still fails after your administrator trusts the OAuth client in Google Workspace, ask your GCP project administrator to check whether a Google Cloud [Organization Policy](https://cloud.google.com/resource-manager/docs/organization-policy/overview) restricts external OAuth clients. The relevant constraint is `constraints/iam.allowedExternalOAuthClients`. If neither of these steps resolves the issue, contact [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support) with the approximate time of the failed connection attempt and the email address you used to authenticate with Google. --- ##### Set up Databricks OAuth dbt platform | Enterprise, Enterprise+ dbt platform accounts on Enterprise and Enterprise+ plans support developer OAuth ([OAuth for partner solutions](https://docs.databricks.com/en/integrations/manage-oauth.html)) with Databricks. When you enable Databricks OAuth for a dbt project, all developers must authenticate with Databricks in order to use the Studio IDE. ##### Development and deployment authentication Databricks OAuth uses user-to-machine (U2M) authentication for interactive development in the Studio IDE. It applies to development credentials only. Deployment environments use separate credentials configured at the environment level. They do not use developer OAuth. For deployment and batch jobs, configure token-based credentials in a [deployment connection profile](https://docs.getdbt.com/docs/platform/about-profiles.md). A [service principal](https://docs.databricks.com/en/admin/users-groups/service-principals.html) token is recommended for production and non-production jobs. For setup steps, refer to [Defining your dbt deployment environment](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md#defining-your-dbt-deployment-environment). For more information on deployment environments, refer to [Deploy environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md). ##### Configure Databricks OAuth (Databricks admin) To get started, add dbt as a custom OAuth application in Databricks. The following steps use the Databricks UI. Alternatively, use the [Databricks CLI](https://docs.databricks.com/en/integrations/enable-disable-oauth.html#enable-custom-oauth-applications-using-the-cli). Use the [redirect URLs for your region](#redirect-urls-by-region) for either method. 1. Log in to the [account console](https://accounts.cloud.databricks.com/?_ga=2.255771976.118201544.1712797799-1002575874.1704693634) and click the **Settings** icon in the sidebar. 2. On the **App connections** tab, click **Add connection**. 3. Enter the following details: * A name for your connection. * The redirect URLs for your OAuth connection. Use both URLs listed for your account region. * For Access scopes, the APIs the application should have access to: * For BI applications, the SQL scope is required to allow the connected app to access Databricks SQL APIs (this is required for SQL models). * For applications that need to access Databricks APIs for purposes other than querying, the ALL APIs scope is required (this is required if running Python models). * The access token time-to-live (TTL) in minutes. Default: 60. * The refresh token time-to-live (TTL) in minutes. Default: 10080. 4. Select **Generate a client secret**. Copy and securely store the client secret. The client secret will not be available later. ###### Redirect URLs by region Use the redirect URLs that correspond to your dbt account region when configuring Databricks OAuth. info As part of the account-specific URL migration, some customers should update OAuth redirect URLs and other integration settings. Existing getdbt.com region URLs continue to support integrations until November 1st, 2026, unless otherwise specified. For more migration timing and account-specific URL guidance, refer to the [account URL migration documentation](https://docs.getdbt.com/docs/platform/about-platform/account-url-migration.md). | Region | Redirect URLs | | ------------------- | ------------------------------------------------------------------------------------------------------- | | **US multi-tenant** | <https://cloud.getdbt.com/callback><br /><https://cloud.getdbt.com/complete/databricks> | | **US cell 1** | <https://us1.dbt.com/callback><br /><https://us1.dbt.com/complete/databricks> | | **EMEA** | <https://emea.dbt.com/callback><br /><https://emea.dbt.com/complete/databricks> | | **APAC** | <https://au.dbt.com/callback><br /><https://au.dbt.com/complete/databricks> | | **Single tenant** | <https://INSTANCE_NAME.getdbt.com/callback><br /><https://INSTANCE_NAME.getdbt.com/complete/databricks> | ##### Configure the Connection in dbt (dbt project admin) Now that you have an OAuth app set up in Databricks, you'll need to add the client ID and secret to dbt. To do so: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Select **Projects** from the menu. 3. Choose your project from the list. 4. Click **Connections** and select the Databricks connection. 5. Click **Edit**. 6. Under the **Optional settings** section, add the **OAuth Client ID** and **OAuth Client Secret** from the Databricks OAuth app. [![Add Databricks OAuth credentials to the connection](/img/docs/dbt-platform/using-dbt-platform/dbt-platform-enterprise/DBX-auth/dbt-databricks-oauth.png?v=2 "Add Databricks OAuth credentials to the connection")](#)Add Databricks OAuth credentials to the connection ##### Authenticating to Databricks (Studio IDE developer) Once the Databricks OAuth connection is set up for a dbt project, each dbt user will need to authenticate with Databricks in order to use the Studio IDE. To do so: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Under **Your profile**, select **Credentials**. 3. Choose your project from the list and click **Edit**. 4. Select **OAuth** as the authentication method, and click **Save**. 5. Finalize by clicking the **Connect Databricks Account** button. [![Connect your Databricks account from development credentials](/img/docs/dbt-platform/using-dbt-platform/dbt-platform-enterprise/DBX-auth/dbt-databricks-oauth-user.png?v=2 "Connect your Databricks account from development credentials")](#)Connect your Databricks account from development credentials You will then be redirected to Databricks and asked to approve the connection. This redirects you back to dbt. You should now be an authenticated Databricks user, ready to use the Studio IDE. ##### Studio OAuth flow Once a user has authorized dbt with Databricks, Databricks returns a refresh token to the dbt application. At the start of each Studio IDE session, dbt exchanges the refresh token for an access token, which it uses to connect to Databricks and execute queries on the user's behalf. dbt does not fetch new access tokens during an active session. When the access token expires (default: 60 minutes), the user must restart the Studio IDE to continue development. The refresh token lifetime is set when you configure the OAuth app in Databricks (default refresh token TTL: 10080 minutes). When a user's refresh token expires, they must click **Connect Databricks Account** again to re-authenticate. --- ##### Set up environment-level permissions dbt platform To set up and configure environment-level permissions, you must have write permissions to the **Groups & Licenses** settings of your dbt account. For more information about roles and permissions, check out [User permissions and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). Environment-level permissions are not the same as account-level [role-based access control (RBAC)](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control) and are configured separately from those workflows. #### Setup instructions In your dbt account: 1. Click your account name, above your profile icon on the left side panel. Then select **Account settings**. 2. Select **Groups & Licenses**. We recommend deleting the default `Owner`, `Member`, and `Everyone` groups and, instead, assigning users to your organizational groups to avoid granting them unnecessary elevated privileges. However, before deleting these groups, ensure that any existing users — including yourself — are reassigned to their appropriate organizational groups. You won’t be able to delete the `Owner` group until *at least* one user is added to another group with the account admin permission set or if there is a user with an IT license. This safety ensures that an account admin is always available to manage group changes. [![Groups & Licenses page in dbt with the default groups highlighted.](/img/docs/dbt-platform/groups-and-licenses.png?v=2 "Groups & Licenses page in dbt with the default groups highlighted.")](#)Groups & Licenses page in dbt with the default groups highlighted. 3. Create a new or open an existing group. If it's a new group, give it a name, then scroll down to **Access & permissions**. Click **Add permission**. [![The Access & permissions section with the Add button highlighted.](/img/docs/dbt-platform/add-permissions.png?v=2 "The Access & permissions section with the Add button highlighted.")](#)The Access & permissions section with the Add button highlighted. 4. Select the **Permission set** for the group. Only the following permissions sets can have environment-level permissions configured: * Database admin * Git admin * Team admin * Analyst * Developer * Job creator Other permission sets are restricted because they have access to everything (for example, Account admin), or limitations prevent them from having write access to environments (for example, Account viewer). If you select a permission set that is not supported, the environment permission option will not appear. [![The view of the permissions box if there is no option for environment permissions.](/img/docs/dbt-platform/no-option.png?v=2 "The view of the permissions box if there is no option for environment permissions.")](#)The view of the permissions box if there is no option for environment permissions. 5. Select the **Environment** for group access. The default is **All environments**, but you can select multiple. If none are selected, the group will have read-only access. [![A list of available environments with the Staging and General boxes selected.](/img/docs/dbt-platform/environment-options.png?v=2 "A list of available environments with the Staging and General boxes selected.")](#)A list of available environments with the Staging and General boxes selected. 6. If you need the same permission set with different project scope or environment write access, click **Add permission** again and configure another row. Each row can use the same permission set with different project and environment selections. 7. Save the Group settings. You're now setup and ready to assign users! #### User experience Users with permissions to the environment will see all capabilities assigned to their role. The environment-level permissions are `write` or `read-only` access. This feature does not currently support determining which features in the environment are accessible. For more details on what can and can not be done with environment-level permissions, refer to [About environment-permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md). For example, here is an overview of the **Jobs** section of the environment page if a user has been granted access: [![The jobs page with write access and the 'Create job' button visible .](/img/docs/dbt-platform/write-access.png?v=2 "The jobs page with write access and the 'Create job' button visible .")](#)The jobs page with write access and the 'Create job' button visible . The same page if the user has not been granted environment-level permissions: [![The jobs page with read-only access and the 'Create job' button is not visible .](/img/docs/dbt-platform/read-only-access.png?v=2 "The jobs page with read-only access and the 'Create job' button is not visible .")](#)The jobs page with read-only access and the 'Create job' button is not visible . --- ##### Set up external OAuth with Redshift dbt platform | Enterprise, Enterprise+ note This feature is currently only available for Okta and Entra ID identity providers. dbt Enterprise and Enterprise+ plans support OAuth authentication with external providers. When **External OAuth** is enabled, users can authorize their user credentials using single sign-on (SSO) via the identity provider (IdP). External OAuth authorizes users to access multiple applications, including dbt, without sharing their static credentials with the service. This makes the process of authenticating for development environments easier for the user and provides an additional layer of security to your dbt account. #### Getting started The process of setting up external OAuth will require a little bit of back-and-forth between your dbt, IdP, and data warehouse accounts, and having them open in multiple browser tabs will help speed up the configuration process: * **dbt:** You’ll primarily be working in the **Account settings** —> **Integrations** page. You will need [proper permission](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to set up the integration and create the connections. * **Identity providers:** * **Okta:** You’ll be working in multiple areas of the Okta account, but you can start in the **Applications** section. You will need permissions to [create an application](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Application_permissions) and an [authorization server](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Authorization_server_permissions). * **Entra ID** An admin with access to create [Entra ID apps](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/custom-available-permissions) who is also a user in the data warehouse is required. * **Data warehouse:** * **Redshift:** Create and manage the [Identity Center integration](https://aws.amazon.com/blogs/big-data/integrate-identity-provider-idp-with-amazon-redshift-query-editor-v2-and-sql-client-using-aws-iam-identity-center-for-seamless-single-sign-on/) with your identity provider. If the admins that handle these products are all different people, it’s better to have them coordinating simultaneously to reduce friction. Ensure your Amazon admins have completed the [Amazon Identity Center integration](https://aws.amazon.com/blogs/big-data/integrate-identity-provider-idp-with-amazon-redshift-query-editor-v2-and-sql-client-using-aws-iam-identity-center-for-seamless-single-sign-on/) with Okta or Entra ID. #### Identity provider configuration Select a supported identity provider (IdP) for instructions on configuring external OAuth in their environment and completing the integration in dbt: ##### Okta ##### 1. Initialize the dbt settings 1. In your dbt account, navigate to **Account settings** —> **Integrations**. 2. Scroll down to **Custom integrations** and click **Add integrations** 3. Leave this window open. You can set the **Integration type** to Okta and note the **Redirect URI** at the bottom of the page. Copy this to your clipboard for use in the next steps. [![Copy the callback URI at the bottom of the integration page in dbt.](/img/docs/dbt-platform/callback-uri.png?v=2 "Copy the callback URI at the bottom of the integration page in dbt.")](#)Copy the callback URI at the bottom of the integration page in dbt. ##### 2. Create the Okta app 1. Expand the **Applications** section from the Okta dashboard and click **Applications.** Click the **Create app integration** button. 2. Select **OIDC** as the sign-in method and **Web applications** as the application type. Click **Next**. [![The Okta app creation window with OIDC and Web Application selected.](/img/docs/dbt-platform/create-okta-app.png?v=2 "The Okta app creation window with OIDC and Web Application selected.")](#)The Okta app creation window with OIDC and Web Application selected. 3. Give the application an appropriate name, something like “External OAuth app for dbt,” that will make it easily identifiable. 4. In the **Grant type** section, enable the **Refresh token** option. 5. Scroll down to the **Sign-in redirect URIs** option. You’ll need to paste the redirect URI you gathered from dbt in step 1.3. [![The Okta app configuration window with the sign-in redirect URI configured to the dbt value.](/img/docs/dbt-platform/configure-okta-app.png?v=2 "The Okta app configuration window with the sign-in redirect URI configured to the dbt value.")](#)The Okta app configuration window with the sign-in redirect URI configured to the dbt value. 6. Save the app configuration. You’ll come back to it, but move on to the next steps for now. ##### 3. Create the Okta API 1. Expand the **Security** section and click **API** from the Okta sidebar menu. 2. On the API screen, click **Add authorization server**. Give the authorization server a name (a nickname for your data warehouse account would be appropriate). For the **Audience** field, copy and paste your data warehouse login URL. Give the server an appropriate description and click **Save**. [![The Okta API window with the Audience value set.](/img/docs/dbt-platform/create-okta-api.png?v=2 "The Okta API window with the Audience value set.")](#)The Okta API window with the Audience value set. 3. On the authorization server config screen, open the **Metadata URI** in a new tab. You’ll need information from this screen in later steps. [![The Okta API settings page with the metadata URI highlighted.](/img/docs/dbt-platform/metadata-uri.png?v=2 "The Okta API settings page with the metadata URI highlighted.")](#)The Okta API settings page with the metadata URI highlighted. [![Sample output of the metadata URI.](/img/docs/dbt-platform/metadata-example.png?v=2 "Sample output of the metadata URI.")](#)Sample output of the metadata URI. 4. Click on the **Scopes** tab and **Add scope**. In the **Name** field, add `session:role-any`. (Optional) Configure **Display phrase** and **Description** and click **Create**. [![API scope configured in the Add Scope window.](/img/docs/dbt-platform/add-api-scope.png?v=2 "API scope configured in the Add Scope window.")](#)API scope configured in the Add Scope window. 5. Open the **Access policies** tab and click **Add policy**. Give the policy a **Name** and **Description** and set **Assign to** as **The following clients**. Start typing the name of the app you created in step 2.3, and you’ll see it autofill. Select the app and click **Create Policy**. [![Assignment field autofilling the value.](/img/docs/dbt-platform/add-api-assignment.png?v=2 "Assignment field autofilling the value.")](#)Assignment field autofilling the value. 6. On the **access policy** screen, click **Add rule**. [![API Add rule button highlighted.](/img/docs/dbt-platform/add-api-rule.png?v=2 "API Add rule button highlighted.")](#)API Add rule button highlighted. 7. Give the rule a descriptive name and scroll down to **token lifetimes**. Configure the **Access token lifetime is**, **Refresh token lifetime is**, and **but will expire if not used every** settings according to your organizational policies. We recommend the defaults of 1 hour and 90 days. Stricter rules increase the odds of your users having to re-authenticate. [![Token lifetime settings in the API rule window.](/img/docs/dbt-platform/configure-token-lifetime.png?v=2 "Token lifetime settings in the API rule window.")](#)Token lifetime settings in the API rule window. 8. Navigate back to the **Settings** tab and leave it open in your browser. You’ll need some of the information in later steps. ##### 4. Create the OAuth settings in the data warehouse Ensure your Amazon admins have completed the Identity Center integration with Okta. Configure the Okta application and APIs in accordance with your Amazon configs. ##### 5. Configuring the integration in dbt 1. Navigate back to the dbt **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. 1. `Integration name`: Give the integration a descriptive name that includes identifying information about the Okta environment so future users won’t have to guess where it belongs. 2. `Client ID` and `Client secrets`: Retrieve these from the Okta application page. [![The client ID and secret highlighted in the Okta app.](/img/docs/dbt-platform/gather-clientid-secret.png?v=2 "The client ID and secret highlighted in the Okta app.")](#)The client ID and secret highlighted in the Okta app. 3. Authorize URL and Token URL: Found in the metadata URI. [![The authorize and token URLs highlighted in the metadata URI.](/img/docs/dbt-platform/gather-authorization-token-endpoints.png?v=2 "The authorize and token URLs highlighted in the metadata URI.")](#)The authorize and token URLs highlighted in the metadata URI. 2. **Save** the configuration ##### 6. Create a new connection in dbt 1. Navigate to **Account settings** and click **Connections** from the menu. Click **New connection**. 2. Configure the `Account`, `Database`, and `Warehouse` as you normally would, and for the `OAuth method`, select the external OAuth you just created. [![The new configuration window in dbt with the External OAuth showing as an option.](/img/docs/dbt-platform/configure-new-connection.png?v=2 "The new configuration window in dbt with the External OAuth showing as an option.")](#)The new configuration window in dbt with the External OAuth showing as an option. 3. Scroll down to the **External OAuth** configurations box and select the config from the list. [![The new connection displayed in the External OAuth Configurations box.](/img/docs/dbt-platform/select-oauth-config.png?v=2 "The new connection displayed in the External OAuth Configurations box.")](#)The new connection displayed in the External OAuth Configurations box. 4. **Save** the connection, and you have now configured External OAuth with Okta! ##### Entra ID ##### 1. Initialize the dbt settings 1. In your dbt account, navigate to **Account settings** —> **Integrations**. 2. Scroll down to **Custom integrations** and click **Add integrations**. 3. Leave this window open. You can set the **Integration type** to Entra ID and note the **Redirect URI** at the bottom of the page. Copy this to your clipboard for use in the next steps. ##### 2. Create the Entra ID apps You’ll create two apps in the Azure portal: A resource server and a client app. ###### Create a resource server In your Entra ID account: 1. From the app registrations screen, click **New registration**. 1. Give the app a name. 2. Ensure **Supported account types** are set to “Accounts in this organizational directory only (`Org name` - Single Tenant).” 3. Click **Register** to see the application’s overview. 2. From the app overview page left menu, click **Expose an API**. 3. Click **Add** next to **Application ID URI**. The field will automatically populate. 4. Click **Save**. [![Create the Entra ID resource server.](/img/docs/dbt-platform/create-resource-server.png?v=2 "Create the Entra ID resource server.")](#)Create the Entra ID resource server. 5. Record the `value` field for use in a future step. 6. From the same screen, click **Add scope**: 1. Name the scope `dbt-redshift`. 2. Set **Who can consent?** to **Admins and users**. 3. Set **Admin consent display name** to `dbt-redshift` and give it a description. 4. Ensure **State** is set to **Enabled**. 5. Click **Add scope**. ###### Create a client app 1. From the **App registration page**, click **New registration**. 1. Give the app a name that uniquely identifies it as the client app. 2. Ensure **Supported account types** are set to “Accounts in this organizational directory only (`Org name` - Single Tenant).” 3. Set the **Redirect URI** to **Web** and copy/paste the **Redirect URI** from dbt into the field. 4. Click **Register**. 2. From the app overview page, click **API permissions** from the left menu, and click **Add permission**. [![Add permissions to the Entra ID app.](/img/docs/dbt-platform/add-permission-entra.png?v=2 "Add permissions to the Entra ID app.")](#)Add permissions to the Entra ID app. 3. From the pop-out screen, click **APIs my organization uses**, search for the resource server name from the previous steps, and click it. 4. Ensure the box for the **Permissions** `dbt-redshift` is enabled and click **Add permissions**. 5. Click **Grant admin consent** and from the popup modal click **Yes**. 6. From the left menu, click **Certificates and secrets** and click **New client secret**. Name the secret, set an expiration, and click **Add**.  * **Note**: Microsoft does not allow “forever” as an expiration date. The maximum time is two years. Documenting the expiration date so you can refresh the secret before the expiration or user authorization fails is essential. 7. Record the `value` for use in a future step and record it immediately.  * **Note**: Entra ID will not display this value again once you navigate away from this screen. ##### 3. Configuring the integration in dbt 1. Navigate back to the dbt **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. There will be some back-and-forth between the Entra ID account and dbt. 2. `Integration name`: Give the integration a descriptive name that includes identifying information about the Entra ID environment so future users won’t have to guess where it belongs. 3. `Client secrets`: Found in the Client ID from the **Certificates and secrets** page. `Value` is the `Client secret`. Note that it only appears when created; *Microsoft hides the secret if you return later, and you must recreate it.* 4. `Client ID`: Copy the’ Application (client) ID’ on the overview page for the client ID app. 5. `Authorization URL` and `Token URL`: From the client ID app, open the `Endpoints` tab. These URLs map to the `OAuth 2.0 authorization endpoint (v2)` and `OAuth 2.0 token endpoint (v2)` fields. *You must use v2 of the `OAuth 2.0 authorization endpoint`. Do not use V1.* You can use either version of the `OAuth 2.0 token endpoint`. 6. `Application ID URI`: Copy the `Application ID URI` field from the resource server’s Overview screen. #### Configure the Trusted Token Issuer in IAM IdC A *trusted token issuer* generates an access token that is used to identify a user, and then authenticates that user. This essentially lets services outside of the AWS ecosystem, such as the dbt platform, connect to IAM IdC (and Redshift) with access tokens they have generated or retrieved from an external IdP (Entra ID or Okta). The following steps are outlined per [this blog post](https://aws.amazon.com/blogs/big-data/integrate-tableau-and-microsoft-entra-id-with-amazon-redshift-using-aws-iam-identity-center/): 1. Open the AWS Management Console and navigate to [IAM Identity Center](https://console.aws.amazon.com/singlesignon), and then to the **Settings**. 2. Select the **Authentication** tab and under **Trusted token issuers**, choose **Create trusted token issuer**. 3. On the **Set up an external IdP to issue trusted tokens** page, under **Trusted token issuer details**, do the following: 1. For **Issuer URL**, enter the OIDC discovery URL of the external IdP that will issue tokens for trusted identity propagation. *Include the forward slash at the end of the URL*. 2. For **Trusted token issuer name**, enter a name to identify this TTI in IAM Identity Center and the application console. 3. Under Map attributes, do the following: 1. For **Identity provider attribute**, select an attribute from the list to map to an attribute in the Identity Center identity store. You can choose: * Email  * Object Identifier * Subject * Other — When using this options with UPN, it's been our experience that `upn` matched up with `Email`. #### Configure Redshift IdC application to utilize TTI To start, select **IAM Identity Center connection** from the Amazon Redshift console menu. [![The AWS Redshift console.](/img/docs/dbt-platform/redshift-idc.png?v=2 "The AWS Redshift console.")](#)The AWS Redshift console. 1. Select the Amazon Redshift application that you created as part of the setup. 2. Select the **Client connections tab** and choose **Edit**. 3. Choose **Yes** under **Configure client connections that use third-party IdPs**. 4. Select the checkbox for **Trusted token issuer** that you created in the previous section. 5. Enter the aud claim value under **Configure selected trusted token issuers**. **This should be the application ID URI you set for the integration in the dbt platform.** #### Finalizing the dbt configuration If you have an existing connection, make sure that the OAuth method is set to **External OAuth** and select the integration you created in an earlier step. Otherwise, create a new Redshift connection, being sure to set values for: * **Server Hostname** * **OAuth Method** * **Database name** (this field can be found under the **Optional Settings**) This connection should be set as the connection for a development environment in an existing or new project. Once the connection has been assigned to a development environment, you can configure your user credentials for that development environment under `Account Settings > Your Profile > Credentials > <Your Project Name>`. Set the authentication method to `External OAuth`, set the `schema` and other fields if desired, and save the credentials. You can then click the `Connect to Redshift` button. ##### Verify connection in Studio Once your development session has initialized, you can test that you’re able to connect to Redshift using external OAuth by running `dbt debug`. --- ##### Set up external OAuth with Snowflake dbt platform | Enterprise, Enterprise+ note This feature is currently only available for Okta and Entra ID identity providers. dbt Enterprise and Enterprise+ plans support OAuth authentication with external providers. When **External OAuth** is enabled, users can authorize their user credentials using single sign-on (SSO) via the identity provider (IdP). External OAuth authorizes users to access multiple applications, including dbt, without sharing their static credentials with the service. This makes the process of authenticating for development environments easier for the user and provides an additional layer of security to your dbt account. #### Getting started The process of setting up external OAuth will require a little bit of back-and-forth between your dbt, IdP, and data warehouse accounts, and having them open in multiple browser tabs will help speed up the configuration process: * **dbt:** You’ll primarily be working in the **Account settings** —> **Integrations** page. You will need [proper permission](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to set up the integration and create the connections. **Identity providers:** * **Okta:** You’ll be working in multiple areas of the Okta account, but you can start in the **Applications** section. You will need permissions to [create an application](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Application_permissions) and an [authorization server](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Authorization_server_permissions). * **Entra ID** An admin with access to create [Entra ID apps](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/custom-available-permissions) who is also a user in the data warehouse is required. **Data warehouse:** * **Snowflake:** Open a worksheet in an account that has permissions to [create a security integration](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration). If the admins that handle these products are all different people, it’s better to have them coordinating simultaneously to reduce friction. Snowflake and IdP username matching required Ensure that the username/email address entered by the IdP admin matches the Snowflake credentials username. If the email address used in the dbt setup is different from the Snowflake email address, the connection will fail or you may run into issues. #### Data warehouse configurations The following is a template for creating the OAuth configurations in the Snowflake environment: ```sql create security integration your_integration_name type = external_oauth enabled = true external_oauth_type = okta external_oauth_issuer = '' external_oauth_jws_keys_url = '' external_oauth_audience_list = ('') external_oauth_token_user_mapping_claim = 'sub' external_oauth_snowflake_user_mapping_attribute = 'email_address' external_oauth_any_role_mode = 'ENABLE' ``` The `external_oauth_token_user_mapping_claim` and `external_oauth_snowflake_user_mapping_attribute` can be modified based on the your organizations needs. These values point to the claim in the users’ token. In the example, Snowflake will look up the Snowflake user whose `email` matches the value in the `sub` claim. **Notes:** * The Snowflake default roles ACCOUNTADMIN, ORGADMIN, or SECURITYADMIN, are blocked from external OAuth by default and they will likely fail to authenticate. See the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-external) for more information. * The value for `external_oauth_snowflake_user_mapping_attribute` must map correctly to the Snowflake username. For example, if `email_address` is used, the email in the token from the IdP must match the Snowflake username exactly. #### Identity provider configuration Select a supported identity provider (IdP) for instructions on configuring external OAuth in their environment and completing the integration in dbt: ##### Okta ##### 1. Initialize the dbt settings 1. In your dbt account, navigate to **Account settings** —> **Integrations**. 2. Scroll down to **Custom integrations** and click **Add integrations** 3. Leave this window open. You can set the **Integration type** to Okta and note the **Redirect URI** at the bottom of the page. Copy this to your clipboard for use in the next steps. [![Copy the callback URI at the bottom of the integration page in dbt.](/img/docs/dbt-platform/callback-uri.png?v=2 "Copy the callback URI at the bottom of the integration page in dbt.")](#)Copy the callback URI at the bottom of the integration page in dbt. ##### 2. Create the Okta app 1. Expand the **Applications** section from the Okta dashboard and click **Applications.** Click the **Create app integration** button. 2. Select **OIDC** as the sign-in method and **Web applications** as the application type. Click **Next**. [![The Okta app creation window with OIDC and Web Application selected.](/img/docs/dbt-platform/create-okta-app.png?v=2 "The Okta app creation window with OIDC and Web Application selected.")](#)The Okta app creation window with OIDC and Web Application selected. 3. Give the application an appropriate name, something like “External OAuth app for dbt,” that will make it easily identifiable. 4. In the **Grant type** section, enable the **Refresh token** option. 5. Scroll down to the **Sign-in redirect URIs** option. You’ll need to paste the redirect URI you gathered from dbt in step 1.3. [![The Okta app configuration window with the sign-in redirect URI configured to the dbt value.](/img/docs/dbt-platform/configure-okta-app.png?v=2 "The Okta app configuration window with the sign-in redirect URI configured to the dbt value.")](#)The Okta app configuration window with the sign-in redirect URI configured to the dbt value. 6. Save the app configuration. You’ll come back to it, but move on to the next steps for now. ##### 3. Create the Okta API 1. Expand the **Security** section and click **API** from the Okta sidebar menu. 2. On the API screen, click **Add authorization server**. Give the authorization server a name (a nickname for your data warehouse account would be appropriate). For the **Audience** field, copy and paste your data warehouse login URL (for example, <https://abdc-ef1234.snowflakecomputing.com>). Give the server an appropriate description and click **Save**. [![The Okta API window with the Audience value set.](/img/docs/dbt-platform/create-okta-api.png?v=2 "The Okta API window with the Audience value set.")](#)The Okta API window with the Audience value set. 3. On the authorization server config screen, open the **Metadata URI** in a new tab. You’ll need information from this screen in later steps. [![The Okta API settings page with the metadata URI highlighted.](/img/docs/dbt-platform/metadata-uri.png?v=2 "The Okta API settings page with the metadata URI highlighted.")](#)The Okta API settings page with the metadata URI highlighted. [![Sample output of the metadata URI.](/img/docs/dbt-platform/metadata-example.png?v=2 "Sample output of the metadata URI.")](#)Sample output of the metadata URI. 4. Click on the **Scopes** tab and **Add scope**. In the **Name** field, add `session:role-any`. (Optional) Configure **Display phrase** and **Description** and click **Create**. [![API scope configured in the Add Scope window.](/img/docs/dbt-platform/add-api-scope.png?v=2 "API scope configured in the Add Scope window.")](#)API scope configured in the Add Scope window. 5. Open the **Access policies** tab and click **Add policy**. Give the policy a **Name** and **Description** and set **Assign to** as **The following clients**. Start typing the name of the app you created in step 2.3, and you’ll see it autofill. Select the app and click **Create Policy**. [![Assignment field autofilling the value.](/img/docs/dbt-platform/add-api-assignment.png?v=2 "Assignment field autofilling the value.")](#)Assignment field autofilling the value. 6. On the **access policy** screen, click **Add rule**. [![API Add rule button highlighted.](/img/docs/dbt-platform/add-api-rule.png?v=2 "API Add rule button highlighted.")](#)API Add rule button highlighted. 7. Give the rule a descriptive name and scroll down to **token lifetimes**. Configure the **Access token lifetime is**, **Refresh token lifetime is**, and **but will expire if not used every** settings according to your organizational policies. We recommend the defaults of 1 hour and 90 days. Stricter rules increase the odds of your users having to re-authenticate. [![Token lifetime settings in the API rule window.](/img/docs/dbt-platform/configure-token-lifetime.png?v=2 "Token lifetime settings in the API rule window.")](#)Token lifetime settings in the API rule window. 8. Navigate back to the **Settings** tab and leave it open in your browser. You’ll need some of the information in later steps. ##### 4. Create the OAuth settings in the data warehouse 1. Open up a Snowflake SQL file and copy/paste the following: ```sql create security integration your_integration_name type = external_oauth enabled = true external_oauth_type = okta external_oauth_issuer = '' external_oauth_jws_keys_url = '' external_oauth_audience_list = ('') external_oauth_token_user_mapping_claim = 'sub' external_oauth_snowflake_user_mapping_attribute = 'email_address' external_oauth_any_role_mode = 'ENABLE' ``` 2. Change `your_integration_name` to something appropriately descriptive. For example, `dev_OktaAccountNumber_okta`. Copy the `external_oauth_issuer` and `external_oauth_jws_keys_url` from the metadata URI in step 3.3. Use the same Snowflake URL you entered in step 3.2 as the `external_oauth_audience_list`. Adjust the other settings as needed to meet your organization's configurations in Okta and Snowflake. [![The issuer and jws keys URIs in the metadata URL](/img/docs/dbt-platform/gather-uris.png?v=2 "The issuer and jws keys URIs in the metadata URL")](#)The issuer and jws keys URIs in the metadata URL 3. Run the steps to create the integration in Snowflake. Username consistency Ensure that the username (for example, email address) entered in the IdP matches the Snowflake credentials for all users. Mismatched usernames will result in authentication failures. ##### 5. Configuring the integration in dbt 1. Navigate back to the dbt **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. 1. `Integration name`: Give the integration a descriptive name that includes identifying information about the Okta environment so future users won’t have to guess where it belongs. 2. `Client ID` and `Client secrets`: Retrieve these from the Okta application page. [![The client ID and secret highlighted in the Okta app.](/img/docs/dbt-platform/gather-clientid-secret.png?v=2 "The client ID and secret highlighted in the Okta app.")](#)The client ID and secret highlighted in the Okta app. 3. Authorize URL and Token URL: Found in the metadata URI. [![The authorize and token URLs highlighted in the metadata URI.](/img/docs/dbt-platform/gather-authorization-token-endpoints.png?v=2 "The authorize and token URLs highlighted in the metadata URI.")](#)The authorize and token URLs highlighted in the metadata URI. 2. **Save** the configuration ##### 6. Create a new connection in dbt 1. Navigate to **Account settings** and click **Connections** from the menu. Click **New connection**. 2. Configure the `Account`, `Database`, and `Warehouse` as you normally would, and for the `OAuth method`, select the external OAuth you just created. [![The new configuration window in dbt with the External OAuth showing as an option.](/img/docs/dbt-platform/configure-new-connection.png?v=2 "The new configuration window in dbt with the External OAuth showing as an option.")](#)The new configuration window in dbt with the External OAuth showing as an option. 3. Scroll down to the **External OAuth** configurations box and select the config from the list. [![The new connection displayed in the External OAuth Configurations box.](/img/docs/dbt-platform/select-oauth-config.png?v=2 "The new connection displayed in the External OAuth Configurations box.")](#)The new connection displayed in the External OAuth Configurations box. 4. **Save** the connection, and you have now configured External OAuth with Okta! ##### Entra ID ##### 1. Initialize the dbt settings 1. In your dbt account, navigate to **Account settings** —> **Integrations**. 2. Scroll down to **Custom integrations** and click **Add integrations**. 3. Leave this window open. You can set the **Integration type** to Entra ID and note the **Redirect URI** at the bottom of the page. Copy this to your clipboard for use in the next steps. ##### 2. Create the Entra ID apps * You’ll create two apps in the Azure portal: A resource server and a client app. * In your Azure portal, open the **Entra ID** and click **App registrations** from the left menu. important * You need both an Entra ID admin and a data warehouse admin to complete the setup. These roles don’t need to be the same person — as long as they collaborate, everything should work smoothly. * Typically, the Entra ID admin handles app registration and permissions, while the data warehouse admin manages roles, grants, and integrations on the warehouse side. * The `value` field gathered in these steps is only displayed once. When created, record it immediately. * Ensure that the username (for example, email address) entered in the IdP matches the data warehouse credentials for all users. Mismatched usernames will result in authentication failures. ##### 3. Create a resource server 1. From the app registrations screen, click **New registration**. 1. Give the app a name. 2. Ensure **Supported account types** are set to “Accounts in this organizational directory only (`Org name` - Single Tenant).” 3. Click **Register**to see the application’s overview. 2. From the app overview page, click **Expose an API** from the left menu. 3. Click **Add** next to **Application ID URI**. The field will automatically populate. Click **Save**. 4. Record the `value` field for use in a future step. *This is only displayed once. Be sure to record it immediately. Microsoft hides the field when you leave the page and come back.* 5. From the same screen, click **Add scope**. 1. Name the scope `session:role-any`. 2. Set “Who can consent?” to **Admins and users**. 3. Set **Admin consent display name** to `session:role-any` and give it a description. 4. Ensure **State** is set to **Enabled**. 5. Click **Add scope**. ##### 4. Create a client app 1. From the **App registration page**, click **New registration**. 1. Give the app a name that uniquely identifies it as the client app. 2. Ensure **Supported account types** are set to “Accounts in this organizational directory only (`Org name` - Single Tenant).” 3. Set the **Redirect URI** to **Web** and copy/paste the **Redirect URI** from dbt into the field. 4. Click **Register**. 2. From the app overview page, click **API permissions** from the left menu, and click **Add permission**. 3. From the pop-out screen, click **APIs my organization uses**, search for the resource server name from the previous steps, and click it. 4. Ensure the box for the **Permissions** `session:role-any` is enabled and click **Add permissions**. 5. Click **Grant admin consent** and from the popup modal click **Yes**. 6. From the left menu, click **Certificates and secrets** and click **New client secret**. Name the secret, set an expiration, and click **Add**. **Note**: Microsoft does not allow “forever” as an expiration date. The maximum time is two years. Documenting the expiration date so you can refresh the secret before the expiration or user authorization fails is essential. 7. Record the `value` for use in a future step and record it immediately. **Note**: Entra ID will not display this value again once you navigate away from this screen. ##### 5. Data warehouse configuration You'll be switching between the Entra ID site and Snowflake. Keep your Entra ID account open for this process. Copy and paste the following as a template in a Snowflake SQL file: ```sql create or replace security integration <whatever you want to name it> type = external_oauth enabled = true external_oauth_type = azure external_oauth_issuer = '<AZURE_AD_ISSUER>' external_oauth_jws_keys_url = '<AZURE_AD_JWS_KEY_ENDPOINT>' external_oauth_audience_list = ('<SNOWFLAKE_APPLICATION_ID_URI>') external_oauth_token_user_mapping_claim = 'upn' external_oauth_any_role_mode = 'ENABLE' external_oauth_snowflake_user_mapping_attribute = 'login_name'; ``` On the Entra ID site: 1. From the Client ID app in Entra ID, click **Endpoints** and open the **Federation metadata document** in a new tab. * The **entity ID** on this page maps to the `external_oauth_issuer` field in the Snowflake config. 2. Back on the list of endpoints, open the **OpenID Connect metadata document** in a new tab. * The **jwks\_uri** field maps to the `external_oauth_jws_keys_url` field in Snowflake. 3. Navigate to the resource server in previous steps. * The **Application ID URI** maps to the `external_oauth_audience_list` field in Snowflake. 4. Run the configurations. You need both an Entra ID admin and a data warehouse admin to complete the setup. If these admins are not the same person, they should work together to complete the configuration. ##### 6. Configuring the integration in dbt 1. Navigate back to the dbt **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. There will be some back-and-forth between the Entra ID account and dbt. 2. `Integration name`: Give the integration a descriptive name that includes identifying information about the Entra ID environment so future users won’t have to guess where it belongs. 3. `Client secrets`: Found in the Client ID from the **Certificates and secrets** page. `Value` is the `Client secret`. Note that it only appears when created; *Microsoft hides the secret if you return later, and you must recreate it.* 4. `Client ID`: Copy the’ Application (client) ID’ on the overview page for the client ID app. 5. `Authorization URL` and `Token URL`: From the client ID app, open the `Endpoints` tab. These URLs map to the `OAuth 2.0 authorization endpoint (v2)` and `OAuth 2.0 token endpoint (v2)` fields. *You must use v2 of the `OAuth 2.0 authorization endpoint`. Do not use V1.* You can use either version of the `OAuth 2.0 token endpoint`. 6. `Application ID URI`: Copy the `Application ID URI` field from the resource server’s Overview screen. #### FAQs Receiving a \`Failed to connect to DB\` error when connecting to Snowflake 1. If you see the following error: ```text Failed to connect to DB: xxxxxxx.snowflakecomputing.com:443. The role requested in the connection, or the default role if none was requested in the connection ('xxxxx'), is not listed in the Access Token or was filtered. Please specify another role, or contact your OAuth Authorization server administrator. ``` 2. Edit your OAuth Security integration and explicitly specify this scope mapping attribute: ```sql ALTER INTEGRATION <my_int_name> SET EXTERNAL_OAUTH_SCOPE_MAPPING_ATTRIBUTE = 'scp'; ``` You can read more about this error in [Snowflake's documentation](https://community.snowflake.com/s/article/external-custom-oauth-error-the-role-requested-in-the-connection-is-not-listed-in-the-access-token). *** 1. If you see the following error: ```text Failed to connect to DB: xxxxxxx.snowflakecomputing.com:443. Incorrect username or password was specified. ``` * **Unique email addresses** — Each user in Snowflake must have a unique email address. You can't have multiple users (for example, a human user and a service account) using the same email, such as `alice@acme.com`, to authenticate to Snowflake. * **Match email addresses with identity provider** — The email address of your Snowflake user must exactly match the email address you use to authenticate with your Identity Provider (IdP). For example, if your Snowflake user's email is `alice@acme.com` but you log in to Entra or Okta with `alice_adm@acme.com`, this mismatch can cause an error. --- ##### Set up SCIM dbt platform | Enterprise, Enterprise+ The System for Cross-Domain Identity Management (SCIM) makes user data more secure and simplifies the admin and end-user lifecycle experience by automating user identities and groups. You can create or disable user identities in your Identity Provider (IdP), and SCIM will automatically make those changes in near real-time downstream in dbt. When configuring your IdP, review [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md) for SCIM provisioning quotas and `429` retry behavior. #### Prerequisites To configure SCIM in your dbt environment: * You must be on an [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing). * You must use Okta or Entra ID as your SSO provider and have it connected in the dbt platform. * You must have [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to configure the account settings in dbt platform and change application settings in [Okta](https://help.okta.com/en-us/content/topics/security/administrators-admin-comparison.htm). * If you have IP restrictions enabled, you must add [Okta's IPs](https://help.okta.com/en-us/content/topics/security/ip-address-allow-listing.htm) to your allowlist. If you're using Entra ID with IP restrictions enabled, refer to [Azure SCIM provisioning fails due to IP allowlisting](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md) in the SCIM FAQ. * **Account Admin group required:** Before enabling SCIM on your dbt account, you must create a dbt Account Admin group in your IdP, and a group with a matching name in your dbt account. Assign the dbt group the [Account Admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) permission set. Without this group, your users could be removed from the default dbt 'Owner' group once SCIM provisioning is enabled and would lose admin access to your dbt account. The names of these two groups can match your IdP group naming scheme, but the names for both the dbt and IdP group must be the same value. ##### Supported features The currently available supported features for SCIM are: * User provisioning and de-provisioning * User profile updates * Group creation and management * Importing groups and users When SCIM is enabled, the following functionality will change: * Users are not automatically added to default groups * Manual actions such as inviting users, updating user information and updating group memberships are disabled by default * SSO group mappings are disabled in favor of SCIM group management. Refer to [Do SSO group mappings still apply when SCIM is enabled?](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md) for details. To overwrite these updates to functionality with SCIM enabled, enable manual updates as part of the SCIM configuration (not recommended). When users are provisioned, the following attributes are supported * Username * Family name * Given name The following IdPs are supported in the dbt user interface: * [Set up SCIM with Okta](https://docs.getdbt.com/docs/platform/manage-access/scim-okta.md) (includes [license management](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md)) * [Set up SCIM with Entra ID](https://docs.getdbt.com/docs/platform/manage-access/scim-entra-id.md) If your IdP isn't on the list, it can be supported using dbt [APIs](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Retrieve%20SCIM%20configuration). License mapping with SCIM dbt platform supports automatic license assignment with SCIM, with these different identity provider options:  Toggle options per identity provider * **Okta:** Enable the **Ignore dbt license mapping** toggle in **Account settings > SSO & SCIM** and follow the [Okta license management doc](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md). * **Entra ID:** Use [SSO-based Active Directory group → license mapping](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration). It works alongside an active Entra ID SCIM setup. Keep the **Ignore dbt license mapping** toggle *disabled* as enabling it removes license mapping for Entra ID users. SCIM license mapping with [custom attributes](https://learn.microsoft.com/en-us/entra/fundamentals/custom-security-attributes-overview) may be possible depending on your Entra ID configuration, but it's not a supported setup — **do it at your own risk**. For more details, refer to the [Does SCIM support automatic license assignment?](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md#does-scim-support-automatic-license-assignment) FAQ. #### Set up dbt To retrieve the necessary dbt configurations for use in Okta or Entra ID: 1. Navigate to your dbt **Account settings**. 2. Under **Settings**, click **SSO & SCIM**. 3. Scroll to the bottom of your SSO configuration settings and click **Enable SCIM**. [![SCIM enabled in the configuration settings.](/img/docs/dbt-platform/access-control/enable-scim.png?v=2 "SCIM enabled in the configuration settings.")](#)SCIM enabled in the configuration settings. 4. Record the **SCIM base URL** field for use in a later step. 5. Click **Create SCIM token**. note To follow best practices, you should regularly rotate your SCIM tokens. To do so, follow these same instructions you did to create a new one. To avoid service disruptions, remember to replace your token in your IdP before deleting the old token in dbt. 6. In the pop-up window, give the token a name that will make it easily identifiable. Click **Save**. [![Give your token and identifier.](/img/docs/dbt-platform/access-control/name-scim-token.png?v=2 "Give your token and identifier.")](#)Give your token and identifier. 7. Copy the token and record it securely, as *it will not be available again after you close the window*. You must create a new token if you lose the current one. [![Give your token and identifier.](/img/docs/dbt-platform/access-control/copy-scim-token.png?v=2 "Give your token and identifier.")](#)Give your token and identifier. 8. (Optional) Manual updates are turned off by default for all SCIM-managed entities, including the ability to invite new users manually. This ensures SCIM-managed entities stay in sync with the IdP, and we recommend keeping this setting disabled. * However, if you need to make manual updates (like update group membership for a SCIM-managed group), you can enable this setting by clicking **Allow manual updates** and confirming the **Allow manual updates** pop up. For more details on this setting, refer to [What does "Allow manual updates" mean?](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md?version=1.12#what-does-allow-manual-updates-mean) in the SCIM FAQ. [![Enabling manual updates in SCIM settings.](/img/docs/dbt-platform/access-control/scim-manual-updates.png?v=2 "Enabling manual updates in SCIM settings.")](#)Enabling manual updates in SCIM settings. #### Next steps Configure SCIM for your identity provider and optionally manage licenses: * **[Set up SCIM with Okta](https://docs.getdbt.com/docs/platform/manage-access/scim-okta.md)** — User and group provisioning, profile updates, and [license management](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md) (Okta only). * **[Set up SCIM with Entra ID](https://docs.getdbt.com/docs/platform/manage-access/scim-entra-id.md)** — User and group provisioning and profile updates, plus assigning users to the SCIM app. * **[SCIM FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md)** — Common questions and troubleshooting for SCIM provisioning. --- ##### Set up SCIM with Entra ID dbt platform | Enterprise, Enterprise+ dbt platform supports System for Cross-Domain Identity Management (SCIM) with Microsoft Entra ID for user and group provisioning and profile updates. License mapping with SCIM dbt platform supports automatic license assignment with SCIM, with these different identity provider options:  Toggle options per identity provider * **Okta:** Enable the **Ignore dbt license mapping** toggle in **Account settings > SSO & SCIM** and follow the [Okta license management doc](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md). * **Entra ID:** Use [SSO-based Active Directory group → license mapping](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration). It works alongside an active Entra ID SCIM setup. Keep the **Ignore dbt license mapping** toggle *disabled* as enabling it removes license mapping for Entra ID users. SCIM license mapping with [custom attributes](https://learn.microsoft.com/en-us/entra/fundamentals/custom-security-attributes-overview) may be possible depending on your Entra ID configuration, but it's not a supported setup — **do it at your own risk**. For more details, refer to the [Does SCIM support automatic license assignment?](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md#does-scim-support-automatic-license-assignment) FAQ. #### Prerequisites * Available on [Enterprise or Enterprise+ plans](https://www.getdbt.com/pricing). * You must use Entra ID as your single sign-on (SSO) provider and have it connected in the dbt platform. * You must have [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to configure the account settings in dbt platform. * Complete [setup SSO with Entra ID](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-microsoft-entra-id.md) before configuring SCIM settings. * Complete the [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md#set-up-dbt) to get your SCIM base URL and token. * **Account Admin group required:** Before enabling SCIM on your dbt account, you must create a dbt Account Admin group in your IdP, and a group with a matching name in your dbt account. Assign the dbt group the [Account Admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) permission set. Without this group, your users could be removed from the default dbt 'Owner' group once SCIM provisioning is enabled and would lose admin access to your dbt account. The names of these two groups can match your IdP group naming scheme, but the names for both the dbt and IdP group must be the same value. #### Set up Entra ID 1. Log in to your Azure account and open the **Entra ID** configurations. 2. From the sidebar, under **Manage**, click **Enterprise Applications**. 3. Click **New Application** and select the option to **Create your own application**. [![Create your own application.](/img/docs/dbt-platform/access-control/create-your-own.png?v=2 "Create your own application.")](#)Create your own application. 4. Give your app a unique name and ensure the **Integrate any other application you don't find in the gallery (Non-gallery)** field is selected. Ignore any prompts for existing apps. Click **Create**. [![Give your app a unique name.](/img/docs/dbt-platform/access-control/create-application.png?v=2 "Give your app a unique name.")](#)Give your app a unique name. 5. From the application **Overview** screen, click **Provision User Accounts**. [![The 'Provision user accounts' option.](/img/docs/dbt-platform/access-control/provision-user-accounts.png?v=2 "The 'Provision user accounts' option.")](#)The 'Provision user accounts' option. 6. From the **Create configuration** section, click **Connect your application**. 7. Fill out the form with the information from your dbt account: * The **Tenant URL** in Entra ID is your **SCIM base URL** from dbt. * The **Secret token** in Entra ID is your **SCIM token** from dbt. 8. Click **Test connection** and click **Create** once complete. [![Configure the app and test the connection.](/img/docs/dbt-platform/access-control/provisioning-config.png?v=2 "Configure the app and test the connection.")](#)Configure the app and test the connection. #### Attribute mapping To map the attributes that will sync with dbt: 1. From the enterprise app **Overview** screen sidebar menu, click **Provisioning**. [![The Provisioning option on the sidebar.](/img/docs/dbt-platform/access-control/provisioning.png?v=2 "The Provisioning option on the sidebar.")](#)The Provisioning option on the sidebar. 2. Under **Manage**, click **Provisioning** again. 3. Expand the **Mappings** section and click **Provision Microsoft Entra ID users**. [![Provision the Entra ID users.](/img/docs/dbt-platform/access-control/provision-entra-users.png?v=2 "Provision the Entra ID users.")](#)Provision the Entra ID users. 4. Select the box for **Show advanced options** and then click **Edit attribute list for customappsso**. [![Click to edit the customappsso attributes.](/img/docs/dbt-platform/access-control/customappsso-attributes.png?v=2 "Click to edit the customappsso attributes.")](#)Click to edit the customappsso attributes. 5. Scroll to the bottom of the **Edit Attribute List** window and find an empty field where you can add a new entry with the following fields: * **Name:** `emails[type eq "work"].primary` * **Type:** `Boolean` * **Required:** True [![Add the new field to the entry list.](/img/docs/dbt-platform/access-control/customappsso-entry.png?v=2 "Add the new field to the entry list.")](#)Add the new field to the entry list. 6. Mark all of the fields listed in Step 10 below as `Required`. [![Mark the fields as required.](/img/docs/dbt-platform/access-control/mark-as-required.png?v=2 "Mark the fields as required.")](#)Mark the fields as required. 7. Click **Save**. 8. Back on the **Attribute mapping** window, click **Add new mapping** and complete fields with the following: * **Mapping type:** `none` * **Default value if null (optional):** `True` * **Target attribute:** `emails[type eq "work"].primary` * **Match objects using this attribute:** `No` * **Matching precedence:** *Leave blank* * **Apply this mapping:** `Always` [![Complete the fields as shown.](/img/docs/dbt-platform/access-control/edit-attribute.png?v=2 "Complete the fields as shown.")](#)Complete the fields as shown. 9. Click **Ok**. 10. Make sure the following mappings are in place and delete any others: * **UserName:** `userPrincipalName` or the value you want users to leverage to log in to dbt. * **active:** `Switch([IsSoftDeleted], , "False", "True", "True", "False")` * **emails\[type eq "work"].value:** `userPrincipalName` is the most common, but this value needs to be the same set for **UserName**. * **name.givenName:** `givenName` * **name.familyName:** `surname` * **externalid:** `mailNickname` * **emails\[type eq "work"].primary** [![Edit the attributes so they match the list as shown.](/img/docs/dbt-platform/access-control/attribute-list.png?v=2 "Edit the attributes so they match the list as shown.")](#)Edit the attributes so they match the list as shown. SSO and SCIM username When you use both SSO and SCIM with Entra ID, map SCIM **UserName** to **userPrincipalName** (or the same Entra attribute users use to sign in to dbt platform). Set **emails\[type eq "work"].value** to that same source. SCIM needs the values to match so provisioning aligns with SSO; if **UserName** and **emails\[type eq "work"].value** don't match, users may not sync or match correctly. 11. Click **Save**. You can now begin assigning users to your SCIM app in Entra ID! #### Assign users to SCIM app The following steps go over how to assign users/groups to the SCIM app. Refer to Microsoft's [official instructions for assigning users or groups to an Enterprise App in Entra ID](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/scim/aad#step-3-assign-users-and-groups-to-the-application) to learn more. Although the article is written for Databricks, the steps are identical. 1. Navigate to Enterprise applications and select the SCIM app. 2. Go to **Manage** > **Provisioning**. 3. To synchronize Microsoft Entra ID users and groups to dbt, click **Start provisioning**. [![Start provisioning to synchronize users and groups.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/scim-entraid-start-provision.png?v=2 "Start provisioning to synchronize users and groups.")](#)Start provisioning to synchronize users and groups. 4. Navigate back to the SCIM app's overview page and go to **Manage** > **Users and groups**. 5. Click **Add user/group** and select the users and groups. [![Add user/group.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/scim-entraid-add-users.png?v=2 "Add user/group.")](#)Add user/group. 6. Click the **Assign** button. 7. Wait a few minutes. In the dbt platform, confirm the users and groups exist in your dbt account. * Users and groups that you add and assign will automatically be provisioned to your dbt account when Microsoft Entra ID schedules the next sync. * By enabling provisioning, you immediately trigger the initial Microsoft Entra ID sync. Subsequent syncs are triggered every 20-40 minutes, depending on the number of users and groups in the application. Refer to Microsoft Entra ID's [Provisioning tips](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/scim/aad#provisioning-tips) documentation for more information. * You can also prompt a manual provisioning outside of the cycle by clicking **Restart provisioning**. [![Prompt manual provisioning.](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/scim-entraid-manual.png?v=2 "Prompt manual provisioning.")](#)Prompt manual provisioning. #### Assign permission sets to SCIM groups SCIM syncs groups and memberships into dbt platform, but it does not assign [permission sets](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md). Without a permission set, group members will not have access to features beyond their user profile. 1. After the sync, go to **Account settings** → **Groups & Licenses**, where the SCIM group appears automatically. 2. Open the SCIM-synced group. Don't create another group for the same IdP group, as this creates a duplicate. 3. Under **Access & permissions**, click **Add permission**. 4. Select a [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md), the projects it should apply to, and [environment-level access](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md) if applicable. 5. Click **Save**. Repeat for each SCIM-synced group that needs access. #### FAQ and troubleshooting For common questions about SCIM provisioning with Entra ID — including onboarding workflows, attribute matching, IP allowlisting issues, and troubleshooting — refer to [SCIM FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md). --- ##### Set up SCIM with Okta dbt platform | Enterprise, Enterprise+ SCIM available for Okta System for Cross-Domain Identity Management (SCIM) [license mapping](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md) is currently only supported for Okta. For other providers, license types must be [managed](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration) within the dbt platform user interface. #### Prerequisites * Available on [Enterprise or Enterprise+ plans](https://www.getdbt.com/pricing). * You must use Okta as your single sign-on (SSO) provider and have it connected in the dbt platform. * You must have [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to configure the account settings in dbt platform. * Complete [setup SSO with Okta](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-okta.md) before configuring SCIM settings. * Complete the [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md#set-up-dbt) to get your SCIM base URL and token. * **Account Admin group required:** Before enabling SCIM on your dbt account, you must create a dbt Account Admin group in your IdP, and a group with a matching name in your dbt account. Assign the dbt group the [Account Admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) permission set. Without this group, your users could be removed from the default dbt 'Owner' group once SCIM provisioning is enabled and would lose admin access to your dbt account. The names of these two groups can match your IdP group naming scheme, but the names for both the dbt and IdP group must be the same value. #### Set up Okta 1. Log in to your Okta account and select **Applications** from the sidebar. Open the application you configured for SSO. 2. Click on the app and navigate to the **General** tab. 3. In **App Settings**, click **Edit** and make sure you check the **SCIM** checkbox on the **Provisioning** row to enable SCIM provisioning. 4. Click **Save** and the **Provisioning** tab will now be visible. [![Enable SCIM provisioning in Okta.](/img/docs/dbt-platform/access-control/scim-provisioned.png?v=2 "Enable SCIM provisioning in Okta.")](#)Enable SCIM provisioning in Okta. 5. Open the **Provisioning** tab and select **Integration**. 6. Click **Edit** and enter the **SCIM base URL** from [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md#set-up-dbt) in the first field. For **Unique identifier field for users**, we recommend `userName`. 7. Under **Supported provisioning actions**, select: * **Push New Users** * **Push Profile Updates** * **Push Groups** * **Import New Users and Profile Updates** (Optional for users created before SSO/SCIM setup) * **Import Groups** (Optional) 8. For **Authentication mode** dropdown, select **HTTP Header**. 9. In the **Authorization** section, enter the token from dbt into the **Bearer** field. [![The completed SCIM configuration in the Okta app.](/img/docs/dbt-platform/access-control/scim-okta-config.png?v=2 "The completed SCIM configuration in the Okta app.")](#)The completed SCIM configuration in the Okta app. 10. Click **Save** to test the connection and you will be taken to the **Provisioning** tab. 11. Click **Edit** and ensure the following provisioning actions are selected: * **Create Users** * **Update User Attributes** * **Deactivate Users** [![Ensure the users are properly provisioned with these settings.](/img/docs/dbt-platform/access-control/provisioning-actions.png?v=2 "Ensure the users are properly provisioned with these settings.")](#)Ensure the users are properly provisioned with these settings. 12. Click **Save** to complete the provisioning configuration. 13. To complete your group setup, go to **Push Groups** and push your Okta groups to dbt platform. This makes the groups available in dbt platform. #### Assign permission sets to SCIM groups SCIM syncs groups and memberships into dbt platform, but it does not assign [permission sets](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md). Without a permission set, group members will not have access to features beyond their user profile. 1. After the sync, go to **Account settings** → **Groups & Licenses**, where the SCIM group appears automatically. 2. Open the SCIM-synced group. Don't create another group for the same IdP group, as this creates a duplicate. 3. Under **Access & permissions**, click **Add permission**. 4. Select a [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md), the projects it should apply to, and [environment-level access](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md) if applicable. 5. Click **Save**. Repeat for each SCIM-synced group that needs access. You've now configured SCIM for the Okta SSO integration in dbt platform. You can [manage user licenses with SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md) to set license type for users as they are provisioned. #### SCIM username format For dbt platform SCIM with Okta, `userName` **must be the email address format**. dbt platform uses `userName` to look up existing users during SCIM sync. If Okta sends another format (such as an Okta internal ID like `00u...` or an employee ID), dbt platform cannot match the existing user, and provisioning will fail. If your Okta configurations map the `Username` field to a different attribute, set your Okta app config to `Email`: 1. Open the SAML app created for the dbt integration. 2. In the **Sign on** tab, click **Edit** in the **Settings** pane. 3. Set the **Application username format** field to **Email**. 4. Click **Save**. SSO and SCIM username When you use both SSO and SCIM with Okta, the SAML **Application username** format must be **Email**. SCIM requires `userName` in email address format, and it must be the same value as the email attribute so users match between SSO and provisioning. #### SCIM license mapping To automate seat assignments in Okta for users as they are provisioned, see [Manage user licenses with SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md). #### Existing Okta integrations If you are adding SCIM to an existing Okta integration in dbt platform (as opposed to setting up SCIM and SSO concurrently for the first time), be aware of the following behavior. Refer to [SCIM FAQ](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md) for more details on what happens to pre-existing users: * Users and groups already synced to dbt will become SCIM-managed once you complete the SCIM configuration. * (Recommended) Import and manage existing dbt groups and users with Okta's **Import Groups** and **Import Users** features. Update the groups in your IdP with the same naming convention used for dbt groups. New users, groups, and changes to existing profiles will be automatically imported into dbt. * Ensure the **Import users and profile updates** and **Import Groups** boxes are selected under the **Provisioning settings** tab in the Okta SCIM configuration. * Use **Import Users** to sync all users from dbt, including previously deleted users, if you need to re-provision those users. * Read more about this feature in the [Okta documentation](https://help.okta.com/en-us/content/topics/users-groups-profiles/usgp-import-groups-app-provisioning.htm). To set license type for users as they are provisioned, see [Manage user licenses with SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md). #### FAQ and troubleshooting For common questions about SCIM provisioning with Okta — including onboarding workflows, SSO group mapping behavior, and troubleshooting, refer to [SCIM FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md). --- ##### Set up Snowflake OAuth dbt platform | Enterprise, Enterprise+ Subdomain migration We're migrating dbt platform [multi-tenant accounts worldwide](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) to static subdomains. After the migration, you’ll be automatically redirected from your original URL (for example, `cloud.getdbt.com`) to the new URL static subdomain (for example, `abc123.us1.dbt.com`), which you can find in your account settings. If your organization has network allow listing, add the `us1.dbt.com` domain to your allow list. The migration may require additional actions in your Snowflake account. See [subdomain migration](#subdomain-migration) for more information. dbt Enterprise and Enterprise+ supports [OAuth authentication](https://docs.snowflake.net/manuals/user-guide/oauth-intro.html) with Snowflake. When Snowflake OAuth is enabled, users can authorize their user credentials using Single Sign On (SSO) via Snowflake rather than submitting a username and password to dbt. If Snowflake is set up with SSO through a third-party identity provider, developers can use this method to log into Snowflake and authorize the dbt user credentials without any additional setup. Snowflake OAuth with PrivateLink Users connecting to Snowflake using [Snowflake OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md) over an AWS PrivateLink connection from dbt will also require access to a PrivateLink endpoint from their local workstation. Where possible, use [Snowflake External OAuth](https://docs.getdbt.com/docs/platform/manage-access/snowflake-external-oauth.md) instead to bypass this limitation. From the [Snowflake](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-overview#label-sso-private-connectivity) docs: > Currently, for any given Snowflake account, SSO works with only one account URL at a time: either the public account URL or the URL associated with the private connectivity service To set up Snowflake OAuth in dbt, admins from both are required for the following steps: 1. [Locate the redirect URI value](#locate-the-redirect-uri-value) in dbt. 2. [Create a security integration](#create-a-security-integration) in Snowflake. 3. [Configure a connection](#configure-a-connection-in-dbt-cloud) in dbt. To use Snowflake in the Studio IDE, all developers must [authenticate with Snowflake](#authorize-developer-credentials) in their profile credentials. ##### Locate the redirect URI value To get started, copy the connection's redirect URI from dbt: 1. Navigate to **Account settings**. 2. Select **Projects** and choose a project from the list. 3. Click the **Development connection** field to view its details and set the **OAuth method** to "Snowflake SSO". 4. Copy the **Redirect URI** to use in the later steps. [![The OAuth method and Redirect URI inputs for a Snowflake connection in dbt.](/img/docs/dbt-platform/dbt-platform-enterprise/snowflake-oauth-redirect-uri.png?v=2 "Locate the Snowflake OAuth redirect URI")](#)Locate the Snowflake OAuth redirect URI ##### Create a security integration In Snowflake, execute a query to create a security integration. Please find the complete documentation on creating a security integration for custom clients [here](https://docs.snowflake.net/manuals/sql-reference/sql/create-security-integration.html#syntax). In the following `CREATE OR REPLACE SECURITY INTEGRATION` example query, replace `<REDIRECT_URI>` value with the Redirect URI (also referred to as the [access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md)) copied in dbt. To locate the Redirect URI, refer to the previous [locate the redirect URI value](#locate-the-redirect-uri-value) section. Important: If you’re using secondary roles, you must include `OAUTH_USE_SECONDARY_ROLES = 'IMPLICIT';` in the statement. ```sql CREATE OR REPLACE SECURITY INTEGRATION DBT_CLOUD TYPE = OAUTH ENABLED = TRUE OAUTH_CLIENT = CUSTOM OAUTH_CLIENT_TYPE = 'CONFIDENTIAL' OAUTH_REDIRECT_URI = '<REDIRECT_URI>' OAUTH_ISSUE_REFRESH_TOKENS = TRUE OAUTH_REFRESH_TOKEN_VALIDITY = 7776000 OAUTH_USE_SECONDARY_ROLES = 'IMPLICIT'; -- Required for secondary roles ``` Permissions Note: Only Snowflake account administrators (users with the `ACCOUNTADMIN` role) or a role with the global `CREATE INTEGRATION` privilege can execute this SQL command. | Field | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | TYPE | Required | | ENABLED | Required | | OAUTH\_CLIENT | Required | | OAUTH\_CLIENT\_TYPE | Required | | OAUTH\_REDIRECT\_URI | Required. Use the value in the [dbt account settings](#locate-the-redirect-uri-value). | | OAUTH\_ISSUE\_REFRESH\_TOKENS | Required | | OAUTH\_REFRESH\_TOKEN\_VALIDITY | Required. This configuration dictates the number of seconds that a refresh token is valid for. Use a smaller value to force users to re-authenticate with Snowflake more frequently. | Additional configuration options may be specified for the security integration as needed. ##### Configure a Connection in dbt The Database Admin is responsible for creating a Snowflake Connection in dbt. This Connection is configured using a Snowflake Client ID and Client Secret. These values can be determined by running the following query in Snowflake: ```sql with integration_secrets as ( select parse_json(system$show_oauth_client_secrets('DBT_CLOUD')) as secrets ) select secrets:"OAUTH_CLIENT_ID"::string as client_id, secrets:"OAUTH_CLIENT_SECRET"::string as client_secret from integration_secrets; ``` To complete the creation of your connection in dbt: 1. Navigate to your **Account Settings**, click **Connections**, and select a connection. 2. Edit the connection and enter the Client ID and Client Secret. 3. Click **Save**. [![Configuring Snowflake OAuth credentials in dbt](/img/docs/dbt-platform/dbt-platform-enterprise/database-connection-snowflake-oauth.png?v=2 "Configuring Snowflake OAuth credentials in dbt")](#)Configuring Snowflake OAuth credentials in dbt ##### Authorize developer credentials Once Snowflake SSO is enabled, users on the project will be able to configure their credentials in their Profiles. By clicking the "Connect to Snowflake Account" button, users will be redirected to Snowflake to authorize with the configured SSO provider, then back to dbt to complete the setup process. At this point, users should now be able to access the Studio IDE with their user credentials. ##### SSO OAuth flow diagram [![SSO OAuth flow diagram](/img/docs/dbt-platform/dbt-platform-enterprise/84427818-841b3680-abf3-11ea-8faf-693d4a39cffb.png?v=2 "SSO OAuth flow diagram")](#)SSO OAuth flow diagram Once a user has authorized dbt with Snowflake via their identity provider, Snowflake will return a Refresh Token to the dbt application. dbt is then able to exchange this refresh token for an Access Token which can then be used to open a Snowflake connection and execute queries in the Studio IDE on behalf of users. **NOTE**: The lifetime of the refresh token is dictated by the OAUTH\_REFRESH\_TOKEN\_VALIDITY parameter supplied in the “create security integration” statement. When a user’s refresh token expires, the user will need to re-authorize with Snowflake to continue development in dbt. ##### Setting up multiple dbt projects with Snowflake 0Auth If you are planning to set up the same Snowflake account to different dbt projects, you can use the same security integration for all of the projects. #### Subdomain migration If your account uses a static subdomain for the [access URL migration](https://docs.getdbt.com/docs/platform/about-platform/account-url-migration.md), you might need to update Snowflake security integrations shared across projects or connections to prevent service disruptions. Connections created after migration require an additional OAuth redirect URI. Find this redirect URI under the [connection's OAuth details](#locate-the-redirect-uri-value) or in [API access URLs](https://docs.getdbt.com/docs/dbt-apis/overview.md), then add it to your existing Snowflake security integration. ```sql ALTER SECURITY INTEGRATION IF EXISTS <INTEGRATION_NAME> SET OAUTH_ALTERNATE_REDIRECT_URIS = ('<ACCESS_URL>/complete/snowflake') ``` Values for `OAUTH_REDIRECT_URI` and `OAUTH_ALTERNATE_REDIRECT_URIS` are interchangeable. After you add the new redirect URI, make sure to update the existing connection in dbt platform to use it. Refer to [Update redirect URIs in dbt platform](https://docs.getdbt.com/docs/platform/about-platform/account-url-migration.md#update-redirect-uris-in-dbt-platform), for more infromation. ##### Troubleshooting  Invalid consent request When clicking on the `Connect Snowflake Account` successfully redirects you to the Snowflake login page, but you receive an `Invalid consent request` error. This could mean: * Your user might not have access to the Snowflake role defined on the user credentials in the dbt **Account settings**. Double-check that you have access to that role and if the role name has been correctly entered in as Snowflake is case sensitive. * You're trying to use a role that is in the [BLOCKED\_ROLES\_LIST](https://docs.snowflake.com/en/user-guide/oauth-partner.html#blocking-specific-roles-from-using-the-integration), such as `ACCOUNTADMIN`.  The requested scope is invalid When you select the `Connect Snowflake Account` button to try to connect to your Snowflake account, you might get an error that says `The requested scope is invalid` even though you were redirected to the Snowflake login page successfully. This error might be because of a configuration issue in the Snowflake OAuth flow, where the `role` in the profile config is mandatory for each user and doesn't inherit it from the project connection page. This means each user needs to supply their role information, regardless of whether it's provided on the project connection page. * In the Snowflake OAuth flow, `role` in the profile config is not optional, as it does not inherit from the project connection config. So each user must supply their role, regardless of whether it is provided in the project connection.  Server error 500 If you experience a 500 server error when redirected from Snowflake to dbt, double-check that you have allow-listed [dbt's IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md), or [VPC Endpoint ID (for PrivateLink connections)](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-snowflake.md#configuring-network-policies), on a Snowflake account level. Enterprise customers who have single-tenant deployments will have a different range of IP addresses (network CIDR ranges) to allow list. Depending on how you've configured your Snowflake network policies or IP allow listing, you may have to explicitly add the network policy that includes the allow listed dbt IPs to the security integration you just made. ```sql ALTER SECURITY INTEGRATION <security_integration_name> SET NETWORK_POLICY = <network_policy_name> ; ```  Secondary role not working. Error: USE ROLE not allowed If you want to use secondary roles but experience `Current sessions is restricted. USE ROLE not allowed` error when setting up Snowflake OAuth, double-check you added the following statement to the query: ```sql OAUTH_USE_SECONDARY_ROLES = 'IMPLICIT'; ``` For the full query example, see [Create a security integration](#create-a-security-integration). #### Learn more --- ##### Set up SSO with Google Workspace dbt platform | Enterprise, Enterprise+ dbt Enterprise-tier plans support Single-Sign On (SSO) via Google GSuite. You will need permissions to create and manage a new Google OAuth2 application, as well as access to enable the Google Admin SDK. Gsuite is a component within Google Cloud Platform (GCP), so you will also need access to a login with permissions to manage the GSuite application within a GCP account. Some customers choose to use different cloud providers for User and Group permission setup than for hosting infrastructure. For example, it's certainly possible to use GSuite to manage login information and Multifactor Authentication (MFA) configuration while hosting data workloads on AWS. Currently supported features include: * SP-initiated SSO * Just-in-time provisioning This guide outlines the setup process for authenticating to dbt with Google GSuite. #### Configuration of the GSuite organization within GCP dbt uses a Client ID and Client Secret to authenticate users of a GSuite organization. The steps below outline how to create a Client ID and Client Secret for use in dbt. ##### Creating credentials 1. Navigate to the GCP [API Manager](https://console.developers.google.com/projectselector/apis/credentials) 2. Select an existing project, or create a new project for your API Credentials 3. Click on **Create Credentials** and select **OAuth Client ID** in the resulting popup 4. Google requires that you configure an OAuth consent screen for OAuth credentials. Click the **Configure consent screen** button to create a new consent screen if prompted. 5. On the OAuth consent screen page, configure the following settings ([Google docs](https://support.google.com/cloud/answer/6158849?hl=en#userconsent)): | Configuration | Value | notes | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | **Application type** | internal | required | | **Application name** | dbt | required | | **Application logo** | Download the logo [here](https://cdn.sanity.io/images/wl0ndo6t/main/333fef4fc72db6f1ce4d1bc0789f355b4f0bbaa2-1280x1280.png) | optional | | **Authorized domains** | `getdbt.com` (US multi-tenant) `getdbt.com` and `dbt.com`(US Cell 1) `dbt.com` (EMEA or AU) | If deploying into a VPC, use the domain for your deployment | | **Scopes** | `email, profile, openid` | The default scopes are sufficient | [![GSuite Consent Screen configuration](/img/docs/dbt-platform/dbt-platform-enterprise/gsuite/gsuite-sso-consent-top.png?v=2 "GSuite Consent Screen configuration")](#)GSuite Consent Screen configuration 6. Save the **Consent screen** settings to navigate back to the **Create OAuth client id** page. 7. Use the following configuration values when creating your Credentials, replacing `YOUR_ACCESS_URL` and `YOUR_AUTH0_URI`, which need to be replaced with the appropriate Access URL and Auth0 URI from your [account settings](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris). | Config | Value | | --------------------------------- | --------------------------------------- | | **Application type** | Web application | | **Name** | dbt | | **Authorized Javascript origins** | `https://YOUR_ACCESS_URL` | | **Authorized Redirect URIs** | `https://YOUR_AUTH0_URI/login/callback` | [![GSuite Credentials configuration](/img/docs/dbt-platform/dbt-platform-enterprise/gsuite/gsuite-sso-credentials.png?v=2 "GSuite Credentials configuration")](#)GSuite Credentials configuration 8. Press "Create" to create your new credentials. A popup will appear with a **Client ID** and **Client Secret**. Write these down as you will need them later! ##### Enabling the Admin SDK dbt requires that the Admin SDK is enabled in this application to request Group Membership information from the GSuite API. To enable the Admin SDK for this project, navigate to the [Admin SDK Settings page](https://console.developers.google.com/apis/api/admin.googleapis.com/overview) and ensure that the API is enabled. [![The 'Admin SDK' page](/img/docs/dbt-platform/dbt-platform-enterprise/7f36f50-Screen_Shot_2019-12-03_at_10.15.01_AM.png?v=2 "The 'Admin SDK' page")](#)The 'Admin SDK' page #### Configuration in dbt To complete setup, follow the steps below in the dbt application. ##### Supply your OAuth Client ID and Client Secret 1. Navigate to **Account settings** > **SSO & SCIM**. 2. Click **Get started** if SSO has not been configured, or **Edit** if it has already been set up. 3. Supply the following SSO details: * **Log in with**: GSuite * **Client ID**: Paste the Client ID generated in the steps above * **Client Secret**: Paste the Client Secret generated in the steps above * **Domain in GSuite**: Enter the domain name for your GSuite account (eg. `dbtlabs.com`). Only users with an email address from this domain will be able to log into your dbt account using GSuite auth. Optionally, you may specify a CSV of domains which are *all* authorized to access your dbt account (eg. `dbtlabs.com, fishtowndata.com`) [![GSuite SSO Configuration](/img/docs/dbt-platform/dbt-platform-enterprise/gsuite/gsuite-sso-cloud-config.png?v=2 "GSuite SSO Configuration")](#)GSuite SSO Configuration 4. Click **Save & Authorize** to authorize your credentials. You should be dropped into the GSuite OAuth flow and prompted to log into dbt with your work email address. If authentication is successful, you will be redirected back to the dbt application. 5. On the **Credentials** page, verify that a `groups` entry is present, and that it reflects the groups you are a member of in GSuite. If you do not see a `groups` entry in the IdP attribute list, consult the following Troubleshooting steps. [![GSuite verify groups](/img/docs/dbt-platform/dbt-platform-enterprise/gsuite/gsuite-sso-cloud-verify.png?v=2 "GSuite verify groups")](#)GSuite verify groups If the verification information looks appropriate, then you have completed the configuration of GSuite SSO. Logging in Users can sign in at <https://login.dbt.com> to view accounts they have access to across instances and choose where to open dbt platform. This is the recommended entry point for most users. For SSO through your identity provider, you can also use the following URL format with your account login URL slug. Replace `LOGIN_SLUG` with the value from the previous steps and `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/enterprise-login/LOGIN-SLUG` Account administrators can turn account discovery on or off with **Enable global account discovery** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). #### Setting up RBAC Now you have completed setting up SSO with GSuite, the next steps will be to set up [RBAC groups](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-) to complete your access control configuration. #### Troubleshooting ##### Invalid client error If you experience an `Error 401: invalid_client` when authorizing with GSuite, double check that: * The Client ID provided matches the value generated in the GCP API Credentials page. * Ensure the Domain Name(s) provided matches the one(s) for your GSuite account. ##### OAuth errors If OAuth verification does not complete successfully, double check that: * The Admin SDK is enabled in your GCP project * The Client ID and Client Secret provided match the values generated in the GCP Credentials page * An Authorized Domain was provided in the OAuth Consent Screen configuration If authentication with the GSuite API succeeds but you do not see a `groups` entry on the **Credentials** page, then you may not have permissions to access Groups in your GSuite account. Either request that your GSuite user is granted the ability to request groups from an administrator, or have an administrator log into dbt and authorize the GSuite integration. For additional troubleshooting — including email verification issues, MFA double prompts, and SSO enforcement behavior — refer to [SSO FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/sso-faq.md). #### Learn more --- ##### Set up SSO with Microsoft Entra ID dbt platform | Enterprise, Enterprise+ dbt Enterprise-tier plans support single-sign on via Microsoft Entra ID (formerly Azure AD). SCIM available for Entra ID After setting up single sign-on (SSO), you can [set up System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim-entra-id.md) with Entra ID to automate user and group provisioning. Currently supported SSO features include: * IdP-initiated SSO * SP-initiated SSO * Just-in-time provisioning #### Configuration dbt supports both single tenant and multi-tenant Microsoft Entra ID (formerly Azure AD) SSO Connections. For most Enterprise purposes, you will want to use the single-tenant flow when creating a Microsoft Entra ID Application. ##### Creating an application Log into the Azure portal for your organization. Using the [**Microsoft Entra ID**](https://portal.azure.com/#home) page, you will need to select the appropriate directory and then register a new application. 1. Under **Manage**, select **App registrations**. 2. Click **+ New Registration** to begin creating a new application registration. [![Creating a new app registration](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-app-registration-empty.png?v=2 "Creating a new app registration")](#)Creating a new app registration 3. Supply configurations for the **Name** and **Supported account types** fields. Choose the **Supported account types** value based on which tenants you want to allow, and note the matching **Microsoft Entra ID Domain** value you'll need later when [supplying credentials](#supplying-credentials) in dbt: | Customer need | Supported account types (Azure) | Value to enter in dbt's Microsoft Entra ID Domain field | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------- | | One tenant only *(default, recommended for most enterprise use-cases)* | Accounts in this organizational directory only | The primary domain name for your Azure directory | | Multiple specific Entra ID tenants | Accounts in organizational directories set by Entra admin | `organizations` | | Any org tenant plus personal Microsoft accounts | Accounts in any organizational directory and personal Microsoft accounts | `common` | | Personal Microsoft accounts only | Personal Microsoft accounts only | `consumers` | 4. (Optional) To ensure your multi-tenant setup works correctly, you’ll need to make two key adjustments beyond just selecting “Multi-tenant” in your Azure account settings: * Update the Microsoft Entra ID Domain: In the dbt “Microsoft Entra ID Domain:” field, enter the specific authority string (`organizations`, `common`, or `consumers`) rather than the domain name for your Azure directory. For more details, see the [Supplying credentials](#supplying-credentials) * Grant Admin Consent for Each Tenant: Because this is an Entra (formerly Azure AD) requirement, each separate tenant will need its own administrator to grant consent. If users from other tenants attempt to log in before this is done, they will see an “admin approval required” screen. An admin can resolve this by visiting the specific consent URL provided by Microsoft for their tenant (for example,`https://login.microsoftonline.com/{TENANT_ID}/adminconsent?client_id={CLIENT_ID}`) 5. Configure the **Redirect URI**. Set the type to **Web** and reference the table below for the appropriate Redirect URI values for single-tenant and multi-tenant Entra ID app deployments. For most enterprise use-cases, you will want to use the single-tenant Redirect URI. Replace `YOUR_AUTH0_URI` with the [appropriate Auth0 URI](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris) for your region and plan. **Note:** Your dbt platform tenancy has no bearing on this setting. This Entra ID app setting controls app access: * **Single-tenant:** Only users from your Entra ID tenant can access the app. * **Multi-tenant:** Users from *any* Entra ID tenant can access the app. | Application Type | Redirect URI | | ----------------------------- | --------------------------------------- | | Single-tenant *(recommended)* | `https://YOUR_AUTH0_URI/login/callback` | | Multi-tenant | `https://YOUR_AUTH0_URI/login/callback` | [![Configuring a new app registration](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-new-application-alternative.png?v=2 "Configuring a new app registration")](#)Configuring a new app registration 6. Save the App registration to continue setting up Microsoft Entra ID SSO. Configuration with the new Microsoft Entra ID interface (optional) Depending on your Microsoft Entra ID settings, your App Registration page might look different than the screenshots shown earlier. If you are *not* prompted to configure a Redirect URI on the **New Registration** page, then follow steps 7 - 8 below after creating your App Registration. If you were able to set up the Redirect URI in the steps above, then skip ahead to [step 8](#adding-users-to-an-enterprise-application). 7. After registering the new application without specifying a Redirect URI, click on **App registration** and then navigate to the **Authentication** tab for the new application. 8. Click **+ Add platform** and enter a Redirect URI for your application. See step 4 above for more information on the correct Redirect URI value for your dbt application. Platform type When selecting the platform type, choose **Web**, not **Single-page application (SPA)**. The dbt SSO integration redeems the authorization code from the server using a client secret. So, if you add the **Redirect URI** under **SPA**, Entra ID enforces PKCE and rejects the server-side token exchange, causing sign-in to fail with the error `AADSTS9002325: Proof Key for Code Exchange is required for cross-origin authorization code redemption.` [![Configuring a Redirect URI](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-redirect-uri.png?v=2 "Configuring a Redirect URI")](#)Configuring a Redirect URI ##### Azure <-> dbt User and Group mapping info There is a [limitation](https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/how-to-connect-fed-group-claims#important-caveats-for-this-functionality) on the number of groups Azure will emit (capped at 150) via the SSO token, meaning if a user belongs to more than 150 groups, it will appear as though they belong to none. To prevent this, configure [group assignments](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/assign-user-or-group-access-portal?pivots=portal) with the dbt app in Azure and set a [group claim](https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/how-to-connect-fed-group-claims#add-group-claims-to-tokens-for-saml-applications-using-sso-configuration) so Azure emits only the relevant groups. The Azure users and groups you will create in the following steps are mapped to groups created in dbt based on the group name. Reference the docs on [enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for additional information on how users, groups, and permission sets are configured in dbt. The dbt platform uses the **User principal name** (UPN) in Microsoft Entra ID to identify and match users logging in to dbt through SSO. The UPN is usually formatted as an email address. ##### Adding users to an Enterprise application Once you've registered the application, the next step is to assign users to it. Add the users you want to be viewable to dbt with the following steps: 8. Navigate back to the [**Default Directory**](https://portal.azure.com/#home) (or **Home**) and click **Enterprise Applications**. 9. Click the name of the application you created earlier. 10. Click **Assign Users and Groups**. 11. Click **Add User/Group**. 12. Assign additional users and groups as needed. [![Adding Users to an Enterprise Application a Redirect URI](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-enterprise-app-users.png?v=2 "Adding Users to an Enterprise Application a Redirect URI")](#)Adding Users to an Enterprise Application a Redirect URI User assignment required? Under **Properties** check the toggle setting for **User assignment required?** and confirm it aligns to your requirements. Most customers will want this toggled to **Yes** so that only users/groups explicitly assigned to dbt will be able to sign in. If this setting is toggled to **No** any user will be able to access the application if they have a direct link to the application per [Microsoft Entra ID Documentation](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/assign-user-or-group-access-portal#configure-an-application-to-require-user-assignment) ##### Configuring permissions 13. Navigate back to [**Default Directory**](https://portal.azure.com/#home) (or **Home**) and then **App registration**. 14. Select your application and then select **API permissions**. 15. Click **+Add a permission** and add the permissions shown in the following table: | API Name | Type | Permission | Required? | | --------------- | --------- | ---------------------- | ---------------------------------------------------------------- | | Microsoft Graph | Delegated | `User.Read` | Yes | | Microsoft Graph | Delegated | `GroupMember.Read.All` | Yes | | Microsoft Graph | Delegated | `Directory.Read.All` | Optional — may be required if users are assigned to > 200 groups | The default scope only requires `User.Read` and `GroupMember.Read.All`. If you assign a user to more than 200 groups, you may need to grant additional permissions such as `Directory.Read.All`. SSO before December 2025 If you set up SSO before December 2025, your existing configuration may request `Directory.Read.All` instead of `GroupMember.Read.All`. To use the updated scopes, delete and re-create your SSO [configuration](#configuring-permissions). 16. Save these permissions, then click **Grant admin consent** to grant admin consent for this directory on behalf of all of your users. [![Configuring application permissions](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-permissions-overview.png?v=2 "Configuring application permissions")](#)Configuring application permissions ##### Creating a client secret 17. Under **Manage**, click **Certificates & secrets**. 18. Click **+New client secret**. 19. Name the client secret "dbt" (or similar) to identify the secret. 20. Select **730 days (24 months)** as the expiration value for this secret (recommended). 21. Click **Add** to finish creating the client secret value (not the client secret ID). 22. Record the generated client secret somewhere safe. Later in the setup process, we'll use this client secret in dbt to finish configuring the integration. [![Configuring certificates & secrets](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-secret-config.png?v=2 "Configuring certificates & secrets")](#)Configuring certificates & secrets [![Recording the client secret](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-secret-saved.png?v=2 "Recording the client secret")](#)Recording the client secret ##### Collect client credentials 23. Navigate to the **Overview** page for the app registration. 24. Note the **Application (client) ID** and **Directory (tenant) ID** shown in this form and record them along with your client secret. We'll use these keys in the steps below to finish configuring the integration in dbt. [![Collecting credentials. Store these somewhere safe](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-overview.png?v=2 "Collecting credentials. Store these somewhere safe")](#)Collecting credentials. Store these somewhere safe #### Configuring dbt To complete setup, follow the steps below in the dbt application. ##### Supplying credentials To complete this section, you will need your login URL slug. This slug controls the URL where users on your account can log into your application. dbt automatically generates login URL slugs, which can't be altered. It will contain only letters, numbers, and dashes. For example, the login URL slug for dbt Labs would look something like `dbt-labs-afk123`. Login URL slugs are unique across all dbt accounts. Users can also sign in at <https://login.dbt.com> to see accounts they have access to across instances. The enterprise login URL that includes your slug remains the URL used for IdP-initiated SSO flows with your identity provider. 25. From dbt, click on your account name in the left side menu and select **Account settings**. 26. Click **SSO & SCIM** from the menu. 27. Click **Get started** if SSO has not been configured, or **Edit** if it has already been set up. 28. Supply the following SSO details: | Field | Value | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Log in with** | Microsoft Entra ID Single Tenant, or Microsoft Entra ID Multi Tenant if you configured a multi-tenant **Supported account types** value | | **Client ID** | Paste the **Application (client) ID** recorded in the steps above | | **Client Secret** | Paste the **Client Secret** (remember to use the Secret Value instead of the Secret ID) from the steps above;<br />**Note:** When the client secret expires, an Entra ID admin will have to generate a new one to be pasted into dbt for uninterrupted application access. | | **Tenant ID** | Paste the **Directory (tenant) ID** recorded in the steps above. (This field only appears when you select **Microsoft Entra ID Single Tenant**; it is not needed for multi-tenant). | | **Microsoft Entra ID Domain** | For single tenant, enter the domain name for your Azure directory (such as `fishtownanalytics.com`). Only use the primary domain; this won't block access for other domains. For multi-tenant, enter the matching authority string (`organizations`, `common`, or `consumers`) instead. Refer to [Supported account types table](#creating-an-application). | [![Configuring Entra ID AD SSO in dbt](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-cloud-sso.png?v=2 "Configuring Entra ID AD SSO in dbt")](#)Configuring Entra ID AD SSO in dbt 29. Click **Save** to complete setup for the Microsoft Entra ID SSO integration. From here, you can navigate to the login URL generated for your account's *slug* to test logging in with Entra ID. Logging in Users can sign in at <https://login.dbt.com> to view accounts they have access to across instances and choose where to open dbt platform. This is the recommended entry point for most users. For SSO through your identity provider, you can also use the following URL format with your account login URL slug. Replace `LOGIN_SLUG` with the value from the previous steps and `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/enterprise-login/LOGIN-SLUG` Account administrators can turn account discovery on or off with **Enable global account discovery** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). ##### Additional configuration options The **Single sign-on** section also contains additional configuration options which are located after the credentials fields. * **Include all groups:** Retrieve all groups to which a user belongs from your identity provider. If a user is a member of nested groups, it will also include the parent groups. When this option is disabled, only groups where the user has direct membership will be supplied. This option is enabled by default. * **Maximum number of groups to retrieve:** Provides a configurable limit to the number of groups to retrieve for users. By default, this is set to 250 groups, but this number can be increased if users' group memberships exceed that amount. #### Setting up RBAC Now you have completed setting up SSO with Entra ID, the next steps will be to set up [RBAC groups](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to complete your access control configuration. Set up SCIM Now that you've set up SSO with Entra ID, you can [set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim-entra-id.md) to automate user and group provisioning. #### Troubleshooting tips  Receiving a 'AADSTS90094: Admin consent is required' error If you set up SSO before December 2025, your existing configuration may request `Directory.Read.All` instead of `GroupMember.Read.All`. To use the updated scopes, delete and re-create your SSO [configuration](#configuring-permissions).  Domain name mismatch Ensure that the domain name under which user accounts exist in Azure matches the domain you supplied in [Supplying credentials](#supplying-credentials) when you configured SSO. [![Obtaining the user domain from Azure](/img/docs/dbt-platform/dbt-platform-enterprise/azure/azure-get-domain.png?v=2 "Obtaining the user domain from Azure")](#)Obtaining the user domain from Azure  Receiving a 'Server error' message after signing in After completing the Entra ID login flow, you're redirected back to the dbt platform login page and see the following message: ```text Server error! There was a server error. Please try again, or contact support@getdbt.com if this persists. ``` This typically happens when Microsoft Entra ID can't find a SAML signing certificate configured for the application. The underlying error (`AADSTS500031: Cannot find signing certificate configured`) isn't surfaced to the user. To confirm this is the cause: 1. Check your Entra ID **Enterprise Applications → YOUR\_APP → Sign-in logs** 2. Filter by **Failure** status. 3. If you see the `AADSTS500031` error there, resolve it by going to **Enterprise Applications → YOUR\_APP → Single sign-on → SAML Signing Certificate**, and add the missing certificate. 4. If a certificate exists but is corrupted, create a new certificate, set an expiration date, mark it **Active** to override the existing one, and then remove the unused certificate. For additional troubleshooting — including "Admin consent required" prompts for new users, "Access Denied" after SAML authentication, and issues with Entity ID or ACS URL changes — refer to [SSO FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/sso-faq.md). #### Learn more by video The following video explains how to set up SSO with Microsoft Entra ID: [](/img/docs/dbt-platform/dbt-platform-enterprise/access-control/sso-dbt-entra-id.mp4) --- ##### Set up SSO with Okta dbt platform | Enterprise, Enterprise+ dbt Enterprise-tier plans support single-sign on via Okta (using SAML). SCIM available for Okta After setting up single sign-on (SSO), you can [set up System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim-okta.md) with Okta to automate user and group provisioning, and license assignment. Currently supported SSO features include: * IdP-initiated SSO * SP-initiated SSO * Just-in-time provisioning This guide outlines the setup process for authenticating to dbt with Okta. #### Configuration in Okta ##### Create a new application Note: You'll need administrator access to your Okta organization to follow this guide. First, log into your Okta account. Using the Admin dashboard, create a new app. [![Create a new app](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-1-new-app.png?v=2 "Create a new app")](#)Create a new app On the following screen, select the following configurations: * **Platform**: Web * **Sign on method**: SAML 2.0 Click **Create** to continue the setup process. [![Configure a new app](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-1-new-app-create.png?v=2 "Configure a new app")](#)Configure a new app ##### Configure the Okta application On the **General Settings** page, enter the following details:: * **App name**: dbt * **App logo** (optional): You can optionally [download the dbt logo](https://cdn.sanity.io/images/wl0ndo6t/main/333fef4fc72db6f1ce4d1bc0789f355b4f0bbaa2-1280x1280.png), and upload it to Okta to use as the logo for this app. Click **Next** to continue. [![Configure the app's General Settings](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-2-general-settings.png?v=2 "Configure the app's General Settings")](#)Configure the app's General Settings ##### Configure SAML Settings The SAML Settings page configures how Okta and dbt communicate. You will want to use an [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. To complete this section, you will need your login URL slug. This slug controls the URL where users on your account can log into your application. dbt automatically generates login URL slugs, which can't be altered. It will contain only letters, numbers, and dashes. For example, the login URL slug for dbt Labs would look something like `dbt-labs-afk123`. Login URL slugs are unique across all dbt accounts. Users can also sign in at <https://login.dbt.com> to see accounts they have access to across instances. The enterprise login URL that includes your slug remains the URL used for IdP-initiated SSO flows with your identity provider. The following steps use `YOUR_AUTH0_URI` and `YOUR_AUTH0_ENTITYID`. Replace these placeholders with the [appropriate Auth0 URI and Auth0 Entity ID](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris) for your region. You can find these values in **Account settings** > **SSO & SCIM** > **Edit** or **Get started** after selecting your identity provider. * **Single sign on URL**: `https://YOUR_AUTH0_URI/login/callback?connection=<login URL slug>` * **Audience URI (SP Entity ID)**: `urn:auth0:<YOUR_AUTH0_ENTITYID>:{login URL slug}` * **Relay State**: `<login URL slug>` * **Name ID format**: `Unspecified` * **Application username**: `Custom` / `user.getInternalProperty("id")` * **Update Application username on**: `Create and update` [![Configure the app's SAML Settings](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-3-saml-settings-top.png?v=2 "Configure the app's SAML Settings")](#)Configure the app's SAML Settings Application username configuration The **Application username** setting depends on whether you plan to use SCIM or not: * **SSO only:** Use a unique value such as `Custom` / `user.getInternalProperty("id")` (recommended in earlier steps). * **SSO and SCIM:** Use **Email** format. SCIM requires the username to be in email address format and to be the same value as the email attribute. Use the **Attribute Statements** and **Group Attribute Statements** forms to map your organization's Okta User and Group Attributes to the format that dbt expects. Expected **User Attribute Statements**: | Name | Name format | Value | Description | | ------------ | ----------- | ---------------- | -------------------------- | | `email` | Unspecified | `user.email` | *The user's email address* | | `first_name` | Unspecified | `user.firstName` | *The user's first name* | | `last_name` | Unspecified | `user.lastName` | *The user's last name* | Expected **Group Attribute Statements**: | Name | Name format | Filter | Value | Description | | -------- | ----------- | ------------- | ----- | ------------------------------------- | | `groups` | Unspecified | Matches regex | `.*` | *The groups that the user belongs to* | **Note:** You may use a more restrictive Group Attribute Statement than the example shown above. For example, if all of your dbt groups start with `DBT_CLOUD_`, you may use a filter like `Starts With: DBT_CLOUD_`. **Okta only returns 100 groups for each user, so if your users belong to more than 100 IdP groups, you will need to use a more restrictive filter**. Please contact support if you have any questions. [![Configure the app's User and Group Attribute Statements](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-3-saml-settings-bottom.png?v=2 "Configure the app's User and Group Attribute Statements")](#)Configure the app's User and Group Attribute Statements Click **Next** to continue. ##### Finish Okta setup Select *I'm an Okta customer adding an internal app*, and select *This is an internal app that we have created*. Click **Finish** to finish setting up the app. [![Finishing setup in Okta](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-4-feedback.png?v=2 "Finishing setup in Okta")](#)Finishing setup in Okta ##### View setup instructions On the next page, click **View Setup Instructions**. In the steps below, you'll supply these values in your dbt Account Settings to complete the integration between Okta and dbt. [![Viewing the configured application](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-5-view-instructions.png?v=2 "Viewing the configured application")](#)Viewing the configured application [![Application setup instructions](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-5-instructions.png?v=2 "Application setup instructions")](#)Application setup instructions #### Configuration in dbt To complete setup, follow the steps below in dbt. ##### Supplying credentials 1. Navigate to **Account settings** > **SSO & SCIM**. 2. Click **Get started** if SSO has not been configured, or **Edit** if it has already been set up. 3. Supply the following SSO details: | Field | Value | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Log in with** | Okta | | **Identity Provider SSO Url** | Paste the **Identity Provider Single Sign-On URL** shown in the Okta setup instructions | | **Identity Provider Issuer** | Paste the **Identity Provider Issuer** shown in the Okta setup instructions | | **X.509 Certificate** | Paste the **X.509 Certificate** shown in the Okta setup instructions;<br />**Note:** When the certificate expires, an Okta admin will have to generate a new one to be pasted into dbt for uninterrupted application access. | [![Configuring the application in dbt](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-6-setup-integration.png?v=2 "Configuring the application in dbt")](#)Configuring the application in dbt 4. Click **Save** to complete setup for the Okta integration. From here, you can navigate to the URL generated for your account's *slug* to test logging in with Okta. Additionally, users added the Okta app will be able to log in to dbt from Okta directly. Logging in Users can sign in at <https://login.dbt.com> to view accounts they have access to across instances and choose where to open dbt platform. This is the recommended entry point for most users. For SSO through your identity provider, you can also use the following URL format with your account login URL slug. Replace `LOGIN_SLUG` with the value from the previous steps and `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/enterprise-login/LOGIN-SLUG` Account administrators can turn account discovery on or off with **Enable global account discovery** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). #### Setting up RBAC Now you have completed setting up SSO with Okta, the next steps will be to set up [RBAC groups](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-) to complete your access control configuration. Set up SCIM Now that you've set up SSO with Okta, you can [set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim-okta.md) to automate user and group provisioning (and license assignment for Okta). #### FAQ and troubleshooting For common questions and troubleshooting guidance — including issues with email verification, MFA prompts, and SSO group mapping behavior — refer to [SSO FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/sso-faq.md). #### Learn more --- ##### Set up SSO with SAML 2.0 dbt platform | Enterprise, Enterprise+ dbt Enterprise-tier plans support single-sign on (SSO) for any SAML 2.0-compliant identity provider (IdP). Currently supported features include: * IdP-initiated SSO * SP-initiated SSO * Just-in-time provisioning This document details the steps to integrate dbt with an identity provider in order to configure Single Sign On and [role-based access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control). #### Auth0 URIs The URI used for SSO connections will vary based on your dbt hosted region. To find the Auth0 URI (also called the **Single sign-on URL**, **Authorization URL**, or **Callback URI**) for your environment: 1. Navigate to your **Account settings** and click **SSO & SCIM** in the left menu. 2. In the **Single sign-on** pane, click **Get started** (if SSO has not been configured) or **Edit** (if it has already been set up). 3. Select the appropriate **Identity provider** from the **Provider type** dropdown. 4. The Auth0 URI is displayed under the **Identity provider values** section. The field label depends on the provider you selected: | Identity provider | Field label | Example URI | | ------------------ | --------------------------- | --------------------------------------------------------------- | | SAML 2.0 | **Single sign-on URL** | `https://YOUR_AUTH0_URI/login/callback` | | Okta | **Single sign-on URL** | `https://YOUR_AUTH0_URI/login/callback?connection=ACCOUNT_NAME` | | Google Workspace | **Authorized Redirect URI** | `https://YOUR_AUTH0_URI/login/callback` | | Microsoft Entra ID | **Callback URI** | `https://YOUR_AUTH0_URI/login/callback` | *Replace `YOUR_AUTH0_URI` and `ACCOUNT_NAME` with your account values.* [![Example of the identity provider values for a SAML 2.0 provider](/img/docs/dbt-platform/access-control/sso-uri.png?v=2 "Example of the identity provider values for a SAML 2.0 provider")](#)Example of the identity provider values for a SAML 2.0 provider Auth0 URI The Auth0 URI always contains YOUR\_AUTH0\_URI (for example, auth.cloud.getdbt.com), not your account-specific prefix URL (such as ks123.us1.dbt.com). This is because dbt uses Auth0 as a centralized authentication service across all regions and accounts. You don't need to replace this value with your cell-specific URL. #### Generic SAML 2.0 integrations If your SAML identity provider is one of Okta, Google, Azure or OneLogin, navigate to the relevant section further down this page. For all other SAML compliant identity providers, you can use the instructions in this section to configure that identity provider. ##### Configure your identity provider You'll need administrator access to your SAML 2.0 compliant identity provider to configure the identity provider. You can use the following instructions with any SAML 2.0 compliant identity provider. ##### Creating the application 1. Log into your SAML 2.0 identity provider and create a new application. 2. When promoted, configure the application with the following details: * **Platform:** Web * **Sign on method:** SAML 2.0 * **App name:** dbt * **App logo (optional):** You can optionally [download the dbt logo](https://drive.google.com/file/d/1fnsWHRu2a_UkJBJgkZtqt99x5bSyf3Aw/view?usp=sharing), and use as the logo for this app. ###### Configuring the application The following steps use `YOUR_AUTH0_URI` and `YOUR_AUTH0_ENTITYID`. Replace these placeholders with the [appropriate Auth0 URI and Auth0 Entity ID](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris) for your region. You can find these values in **Account settings** > **SSO & SCIM** > **Edit** or **Get started** after selecting your identity provider. To complete this section, you will need your login URL slug. This slug controls the URL where users on your account can log into your application. dbt automatically generates login URL slugs, which can't be altered. It will contain only letters, numbers, and dashes. For example, the login URL slug for dbt Labs would look something like `dbt-labs-afk123`. Login URL slugs are unique across all dbt accounts. Users can also sign in at <https://login.dbt.com> to see accounts they have access to across instances. The enterprise login URL that includes your slug remains the URL used for IdP-initiated SSO flows with your identity provider. When prompted for the SAML 2.0 application configurations, supply the following values: * Single sign on URL: `https://YOUR_AUTH0_URI/login/callback?connection=<login URL slug>` * Audience URI (SP Entity ID): `urn:auth0:<YOUR_AUTH0_ENTITYID>:{login URL slug}` - Relay State: `<login URL slug>` (Note: Relay state may be shown as optional in the IdP settings; it is *required* for the dbt SSO configuration.) Additionally, you may configure the IdP attributes passed from your identity provider into dbt. [SCIM configuration](https://docs.getdbt.com/docs/platform/manage-access/scim.md) requires `NameID` and `email` to associate logins with the correct user. If you're using license mapping for groups, you need to additionally configure the `groups` attribute. We recommend using the following values: | name | name format | value | description | | ----------- | ----------- | ---------------- | ------------------------ | | email | Unspecified | user.email | The user's email address | | first\_name | Unspecified | user.first\_name | The user's first name | | last\_name | Unspecified | user.last\_name | The user's last name | | NameID | Unspecified | ID | The user's unchanging ID | `NameID` values can be persistent (`urn:oasis:names:tc:SAML:2.0:nameid-format:persistent`) rather than unspecified if your IdP supports these values. Using an email address for `NameID` will work, but dbt creates an entirely new user if that email address changes. Configuring a value that will not change, even if the user's email address does, is a best practice. dbt's [role-based access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control) relies on group mappings from the IdP to assign dbt users to dbt groups. To use role-based access control in dbt, also configure your identity provider to provide group membership information in user attribute called `groups`: | name | name format | value | description | | ------ | ----------- | ---------------- | --------------------------------------- | | groups | Unspecified | `<IdP-specific>` | The groups a user belongs to in the IdP | Note You may use a restricted group attribute statement to limit the groups set to dbt for each authenticated user. For example, if all of your dbt groups start with `DBT_CLOUD_...`, you may optionally apply a filter like `Starts With: DBT_CLOUD_`. ##### Collect integration secrets After confirming your details, the IdP should show you the following values for the new SAML 2.0 integration. Keep these values somewhere safe, as you will need them to complete setup in dbt. * Identity Provider Issuer * Identity Provider SSO Url * X.509 Certificate (PEM format required) *  Example of PEM format ```text -----BEGIN CERTIFICATE----- MIIC8DCCAdigAwIBAgIQSANTIKwxA1221kqhkiG9w0dbtLabsBAQsFADA0MTIwMAYDVQQD EylNaWNyb3NvZnQgQXp1cmUgRmVkZXJhdGVkIFNTTyBDZXJ0aWZpY2F0ZTAeFw0yMzEyMjIwMDU1 MDNaFw0yNjEyMjIwMDU1MDNaMDQxMjAwBgNVBAMTKU1pY3Jvc29mdCBBenVyZSBGZWRlcmF0ZWQg U1NPIENlcnRpZmljYXRlMIIBIjANBgkqhkiG9w0BAEFAAFRANKIEMIIBCgKCAQEAqfXQGc/D8ofK aXbPXftPotqYLEQtvqMymgvhFuUm+bQ9YSpS1zwNQ9D9hWVmcqis6gO/VFw61e0lFnsOuyx+XMKL rJjAIsuWORavFqzKFnAz7hsPrDw5lkNZaO4T7tKs+E8N/Qm4kUp5omZv/UjRxN0XaD+o5iJJKPSZ PBUDo22m+306DE6ZE8wqxT4jTq4g0uXEitD2ZyKaD6WoPRETZELSl5oiCB47Pgn/mpqae9o0Q2aQ LP9zosNZ07IjKkIfyFKMP7xHwzrl5a60y0rSIYS/edqwEhkpzaz0f8QW5pws668CpZ1AVgfP9TtD Y1EuxBSDQoY5TLR8++2eH4te0QIDAQABMA0GCSqGSIb3DmAKINgAA4IBAQCEts9ujwaokRGfdtgH 76kGrRHiFVWTyWdcpl1dNDvGhUtCRsTC76qwvCcPnDEFBebVimE0ik4oSwwQJALExriSvxtcNW1b qvnY52duXeZ1CSfwHkHkQLyWBANv8ZCkgtcSWnoHELLOWORLD4aSrAAY2s5hP3ukWdV9zQscUw2b GwN0/bTxxQgA2NLZzFuHSnkuRX5dbtrun21USPTHMGmFFYBqZqwePZXTcyxp64f3Mtj3g327r/qZ squyPSq5BrF4ivguYoTcGg4SCP7qfiNRFyBUTTERFLYU0n46MuPmVC7vXTsPRQtNRTpJj/b2gGLk 1RcPb1JosS1ct5Mtjs41 -----END CERTIFICATE----- ``` ##### Finish setup After creating the application, follow the instructions in the [dbt setup](#dbt-setup) section to complete the integration. #### Okta integration You can use the instructions in this section to configure Okta as your identity provider. 1. Log into your Okta account. Using the Admin dashboard, create a new app. [![Create a new app](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-1-new-app.png?v=2 "Create a new app")](#)Create a new app 2. Select the following configurations: * **Platform**: Web * **Sign on method**: SAML 2.0 3. Click **Create** to continue the setup process. [![Configure a new app](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-1-new-app-create.png?v=2 "Configure a new app")](#)Configure a new app ##### Configure the Okta application The following steps use `YOUR_AUTH0_URI` and `YOUR_AUTH0_ENTITYID`. Replace these placeholders with the [appropriate Auth0 URI and Auth0 Entity ID](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris) for your region. You can find these values in **Account settings** > **SSO & SCIM** > **Edit** or **Get started** after selecting your identity provider. To complete this section, you will need your login URL slug. This slug controls the URL where users on your account can log into your application. dbt automatically generates login URL slugs, which can't be altered. It will contain only letters, numbers, and dashes. For example, the login URL slug for dbt Labs would look something like `dbt-labs-afk123`. Login URL slugs are unique across all dbt accounts. Users can also sign in at <https://login.dbt.com> to see accounts they have access to across instances. The enterprise login URL that includes your slug remains the URL used for IdP-initiated SSO flows with your identity provider. 1. On the **General Settings** page, enter the following details: * **App name**: dbt * **App logo** (optional): You can optionally [download the dbt logo](https://drive.google.com/file/d/1fnsWHRu2a_UkJBJgkZtqt99x5bSyf3Aw/view?usp=sharing), and upload it to Okta to use as the logo for this app. 2. Click **Next** to continue. [![Configure the app's General Settings](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-2-general-settings.png?v=2 "Configure the app's General Settings")](#)Configure the app's General Settings ##### Configure SAML Settings 1. On the **SAML Settings** page, enter the following values: * **Single sign on URL**: `https://YOUR_AUTH0_URI/login/callback?connection=<login URL slug>` * **Audience URI (SP Entity ID)**: `urn:auth0:<YOUR_AUTH0_ENTITYID>:<login URL slug>` * **Relay State**: `<login URL slug>` * **Name ID format**: `Unspecified` * **Application username**: `Custom` / `user.getInternalProperty("id")` * **Update Application username on**: `Create and update` [![Configure the app's SAML Settings](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-3-saml-settings-top.png?v=2 "Configure the app's SAML Settings")](#)Configure the app's SAML Settings 2. Map your organization's Okta User and Group Attributes to the format that dbt expects by using the Attribute Statements and Group Attribute Statements forms. [SCIM configuration](https://docs.getdbt.com/docs/platform/manage-access/scim.md) requires `email` to associate logins with the correct user. If you're using license mapping for groups, you need to additionally configure the `groups` attribute. 3. The following table illustrates expected User Attribute Statements: | Name | Name format | Value | Description | | ------------ | ----------- | ---------------- | -------------------------- | | `email` | Unspecified | `user.email` | *The user's email address* | | `first_name` | Unspecified | `user.firstName` | *The user's first name* | | `last_name` | Unspecified | `user.lastName` | *The user's last name* | 4. The following table illustrates expected **Group Attribute Statements**: | Name | Name format | Filter | Value | Description | | -------- | ----------- | ------------- | ----- | ------------------------------------- | | `groups` | Unspecified | Matches regex | `.*` | *The groups that the user belongs to* | You can instead use a more restrictive Group Attribute Statement than the example shown in the previous steps. For example, if all of your dbt groups start with `DBT_CLOUD_`, you may use a filter like `Starts With: DBT_CLOUD_`. **Okta only returns 100 groups for each user, so if your users belong to more than 100 IdP groups, you will need to use a more restrictive filter**. Please contact support if you have any questions. [![Configure the app's User and Group Attribute Statements](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-3-saml-settings-bottom.png?v=2 "Configure the app's User and Group Attribute Statements")](#)Configure the app's User and Group Attribute Statements 5. Click **Next** to continue. ##### Finish Okta setup 1. Select *I'm an Okta customer adding an internal app*. 2. Select *This is an internal app that we have created*. 3. Click **Finish** to finish setting up the app. [![Finishing setup in Okta](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-4-feedback.png?v=2 "Finishing setup in Okta")](#)Finishing setup in Okta ##### View setup instructions 1. On the next page, click **View Setup Instructions**. 2. In the steps below, you'll supply these values in your dbt Account Settings to complete the integration between Okta and dbt. [![Viewing the configured application](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-5-view-instructions.png?v=2 "Viewing the configured application")](#)Viewing the configured application [![Application setup instructions](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-5-instructions.png?v=2 "Application setup instructions")](#)Application setup instructions 3. After creating the Okta application, follow the instructions in the [dbt setup](#dbt-setup) section to complete the integration. #### Google integration Use this section if you are configuring Google as your identity provider. ##### Configure the Google application The following steps use `YOUR_AUTH0_URI` and `YOUR_AUTH0_ENTITYID`. Replace these placeholders with the [appropriate Auth0 URI and Auth0 Entity ID](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris) for your region. You can find these values in **Account settings** > **SSO & SCIM** > **Edit** or **Get started** after selecting your identity provider. To complete this section, you will need your login URL slug. This slug controls the URL where users on your account can log into your application. dbt automatically generates login URL slugs, which can't be altered. It will contain only letters, numbers, and dashes. For example, the login URL slug for dbt Labs would look something like `dbt-labs-afk123`. Login URL slugs are unique across all dbt accounts. Users can also sign in at <https://login.dbt.com> to see accounts they have access to across instances. The enterprise login URL that includes your slug remains the URL used for IdP-initiated SSO flows with your identity provider. 1. Sign into your **Google Admin Console** via an account with super administrator privileges. 2. From the Admin console Home page, go to **Apps** and then click **Web and mobile apps**. 3. Click **Add**, then click **Add custom SAML app**. 4. Click **Next** to continue. 5. Make these changes on the App Details page: * Name the custom app * Upload an app logo (optional) * Click **Continue**. ##### Configure SAML Settings 1. Go to the **Google Identity Provider details** page. 2. Download the **IDP metadata**. 3. Copy the **SSO URL** and **Entity ID** and download the **Certificate** (or **SHA-256 fingerprint**, if needed). 4. Enter the following values on the **Service Provider Details** window: * **ACS URL**: `https://YOUR_AUTH0_URI/login/callback?connection=<login URL slug>` * **Audience URI (SP Entity ID)**: `urn:auth0:<YOUR_AUTH0_ENTITYID>:<login URL slug>` - **Start URL**: `<login URL slug>` 5. Select the **Signed response** checkbox. 6. The default **Name ID** is the primary email. Multi-value input is not supported. If your user profile has a unique, stable value that will persist across email address changes, it's best to use that; otherwise, email will work. 7. Use the **Attribute mapping** page to map your organization's Google Directory Attributes to the format that dbt expects. 8. Click **Add another mapping** to map additional attributes. Expected **Attributes**: | Name | Name format | Value | Description | | --------------- | ----------- | ------------ | ------------------------- | | `First name` | Unspecified | `first_name` | The user's first name. | | `Last name` | Unspecified | `last_name` | The user's last name. | | `Primary email` | Unspecified | `email` | The user's email address. | 9. To use [role-based access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control) in dbt, enter the groups in the **Group membership** field during configuration: | Google groups | App attributes | | -------------- | -------------- | | Name of groups | `groups` | 10. Click **Finish** to continue. ##### Finish Google setup 1. From the Admin console Home page, go to **Apps** and then click **Web and mobile apps**. 2. Select your SAML app. 3. Click **User access**. 4. To turn on or off a service for everyone in your organization, click **On for everyone** or **Off for everyone**, and then click **Save**. 5. Ensure that the email addresses your users use to sign in to the SAML app match the email addresses they use to sign in to your Google domain. **Note:** Changes typically take effect in minutes, but can take up to 24 hours. ##### Finish setup After creating the Google application, follow the instructions in the [dbt setup](#dbt-setup) #### Microsoft Entra ID (formerly Azure AD) integration If you're using Microsoft Entra ID (formerly Azure AD), the instructions below will help you configure it as your identity provider. ##### Create a Microsoft Entra ID Enterprise application The following steps use `YOUR_AUTH0_URI` and `YOUR_AUTH0_ENTITYID`. Replace these placeholders with the [appropriate Auth0 URI and Auth0 Entity ID](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris) for your region. You can find these values in **Account settings** > **SSO & SCIM** > **Edit** or **Get started** after selecting your identity provider. To complete this section, you will need your login URL slug. This slug controls the URL where users on your account can log into your application. dbt automatically generates login URL slugs, which can't be altered. It will contain only letters, numbers, and dashes. For example, the login URL slug for dbt Labs would look something like `dbt-labs-afk123`. Login URL slugs are unique across all dbt accounts. Users can also sign in at <https://login.dbt.com> to see accounts they have access to across instances. The enterprise login URL that includes your slug remains the URL used for IdP-initiated SSO flows with your identity provider. Follow these steps to set up single sign-on (SSO) with dbt: 1. Log into your Azure account. 2. In the Entra ID portal, select **Enterprise applications** and click **+ New application**. 3. Select **Create your own application**. 4. Name the application "dbt" or another descriptive name. 5. Select **Integrate any other application you don't find in the gallery (Non-gallery)** as the application type. 6. Click **Create**. 7. You can find the new application by clicking **Enterprise applications** and selecting **All applications**. 8. Click the application you just created. 9. Select **Single sign-on** under Manage in the left navigation. 10. Click **Set up single sign on** under Getting Started. [![In your Overview page, select 'Set up single sign on](/img/docs/dbt-platform/access-control/single-sign-on-overview.jpg?v=2 "In your Overview page, select 'Set up single sign on")](#)In your Overview page, select 'Set up single sign on 11. Click **SAML** in "Select a single sign-on method" section. [![Select the 'SAML' card in the 'Seelct a single sign-on method' section. ](/img/docs/dbt-platform/access-control/saml.jpg?v=2 "Select the 'SAML' card in the 'Seelct a single sign-on method' section. ")](#)Select the 'SAML' card in the 'Seelct a single sign-on method' section. 12. Click **Edit** in the Basic SAML Configuration section. [![In the 'Set up Single Sign-On with SAML' page, click 'Edit' in the 'Basic SAML Configuration' card](/img/docs/dbt-platform/access-control/basic-saml.jpg?v=2 "In the 'Set up Single Sign-On with SAML' page, click 'Edit' in the 'Basic SAML Configuration' card")](#)In the 'Set up Single Sign-On with SAML' page, click 'Edit' in the 'Basic SAML Configuration' card 13. Use the following table to complete the required fields and connect to dbt: | Field | Value | | ---------------------------------------------- | ------------------------------------------------------------------------ | | **Identifier (Entity ID)** | Use `urn:auth0:<YOUR_AUTH0_ENTITYID>:<login URL slug>`. | | **Reply URL (Assertion Consumer Service URL)** | Use `https://YOUR_AUTH0_URI/login/callback?connection=<login URL slug>`. | | **Relay State** | `<login URL slug>` | 14. Click **Save** at the top of the form. ##### Creating SAML settings From the Set up Single Sign-On with SAML page: 1. Click **Edit** in the User Attributes & Claims section. 2. Click **Unique User Identifier (Name ID)** under **Required claim.** 3. Set **Name identifier format** to **Unspecified**. 4. Set **Source attribute** to **user.objectid**. 5. Delete all claims under **Additional claims.** 6. Click **Add new claim** and add the following new claims: | Name | Source attribute | | --------------- | ---------------- | | **email** | user.mail | | **first\_name** | user.givenname | | **last\_name** | user.surname | 7. Click **Add a group claim** from **User Attributes and Claims.** 8. If you assign users directly to the enterprise application, select **Security Groups**. If not, select **Groups assigned to the application**. 9. Set **Source attribute** to **Group ID**. 10. Under **Advanced options**, check **Customize the name of the group claim** and specify **Name** to **groups**. **Note:** Keep in mind that the Group ID in Entra ID maps to that group's GUID. It should be specified in lowercase for the mappings to work as expected. The Source Attribute field alternatively can be set to a different value of your preference. ##### Finish setup 9. After creating the Azure application, follow the instructions in the [dbt setup](#dbt-setup) section to complete the integration. The names for fields in dbt vary from those in the Entra ID app. They're mapped as follows: | dbt field | Corresponding Entra ID field | | ----------------------------- | ---------------------------- | | **Identity Provider SSO URL** | Login URL | | **Identity Provider Issuer** | Microsoft Entra Identifier | #### OneLogin integration Use this section if you are configuring OneLogin as your identity provider. To configure OneLogin, you will need **Administrator** access. ##### Configure the OneLogin application The following steps use `YOUR_AUTH0_URI` and `YOUR_AUTH0_ENTITYID`. Replace these placeholders with the [appropriate Auth0 URI and Auth0 Entity ID](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md#auth0-uris) for your region. You can find these values in **Account settings** > **SSO & SCIM** > **Edit** or **Get started** after selecting your identity provider. To complete this section, you will need your login URL slug. This slug controls the URL where users on your account can log into your application. dbt automatically generates login URL slugs, which can't be altered. It will contain only letters, numbers, and dashes. For example, the login URL slug for dbt Labs would look something like `dbt-labs-afk123`. Login URL slugs are unique across all dbt accounts. Users can also sign in at <https://login.dbt.com> to see accounts they have access to across instances. The enterprise login URL that includes your slug remains the URL used for IdP-initiated SSO flows with your identity provider. 1. Log into OneLogin, and add a new SAML 2.0 Application. 2. Configure the application with the following details: * **Platform:** Web * **Sign on method:** SAML 2.0 * **App name:** dbt * **App logo (optional):** You can optionally [download the dbt logo](https://drive.google.com/file/d/1fnsWHRu2a_UkJBJgkZtqt99x5bSyf3Aw/view?usp=sharing), and use as the logo for this app. ##### Configure SAML settings 3. Under the **Configuration tab**, input the following values: * **RelayState:** `<login URL slug>` * **Audience (EntityID):** `urn:auth0:<YOUR_AUTH0_ENTITYID>:<login URL slug>` * **ACS (Consumer) URL Validator:** `https://YOUR_AUTH0_URI/login/callback?connection=<login URL slug>` * **ACS (Consumer) URL:** `https://YOUR_AUTH0_URI/login/callback?connection=<login URL slug>` 4. Next, go to the **Parameters tab**. You must have a parameter for the Email, First Name, and Last Name attributes and include all parameters in the SAML assertions. When you add the custom parameters, make sure you select the **Include in SAML assertion** checkbox, and ensure that the casing and format matches exactly as seen in the following table. We recommend using the following values: | name | name format | value | | ----------- | ----------- | ----------- | | NameID | Unspecified | OneLogin ID | | email | Unspecified | Email | | first\_name | Unspecified | First Name | | last\_name | Unspecified | Last Name | dbt's [role-based access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control) relies on group mappings from the IdP to assign dbt users to dbt groups. To use role-based access control in dbt, also configure OneLogin to provide group membership information in user attribute called `groups`: | name | name format | value | description | | ------ | ----------- | ------------------------------------------------- | --------------------------------------- | | groups | Unspecified | Series of groups to be used for your organization | The groups a user belongs to in the IdP | ##### Collect integration secrets 5. After confirming your details, go to the **SSO tab**. OneLogin should show you the following values for the new integration. Keep these values somewhere safe, as you will need them to complete setup in dbt. * Issuer URL * SAML 2.0 Endpoint (HTTP) * X.509 Certificate (PEM format required) *  Example of PEM format ```text -----BEGIN CERTIFICATE----- MIIC8DCCAdigAwIBAgIQSANTIKwxA1221kqhkiG9w0dbtLabsBAQsFADA0MTIwMAYDVQQD EylNaWNyb3NvZnQgQXp1cmUgRmVkZXJhdGVkIFNTTyBDZXJ0aWZpY2F0ZTAeFw0yMzEyMjIwMDU1 MDNaFw0yNjEyMjIwMDU1MDNaMDQxMjAwBgNVBAMTKU1pY3Jvc29mdCBBenVyZSBGZWRlcmF0ZWQg U1NPIENlcnRpZmljYXRlMIIBIjANBgkqhkiG9w0BAEFAAFRANKIEMIIBCgKCAQEAqfXQGc/D8ofK aXbPXftPotqYLEQtvqMymgvhFuUm+bQ9YSpS1zwNQ9D9hWVmcqis6gO/VFw61e0lFnsOuyx+XMKL rJjAIsuWORavFqzKFnAz7hsPrDw5lkNZaO4T7tKs+E8N/Qm4kUp5omZv/UjRxN0XaD+o5iJJKPSZ PBUDo22m+306DE6ZE8wqxT4jTq4g0uXEitD2ZyKaD6WoPRETZELSl5oiCB47Pgn/mpqae9o0Q2aQ LP9zosNZ07IjKkIfyFKMP7xHwzrl5a60y0rSIYS/edqwEhkpzaz0f8QW5pws668CpZ1AVgfP9TtD Y1EuxBSDQoY5TLR8++2eH4te0QIDAQABMA0GCSqGSIb3DmAKINgAA4IBAQCEts9ujwaokRGfdtgH 76kGrRHiFVWTyWdcpl1dNDvGhUtCRsTC76qwvCcPnDEFBebVimE0ik4oSwwQJALExriSvxtcNW1b qvnY52duXeZ1CSfwHkHkQLyWBANv8ZCkgtcSWnoHELLOWORLD4aSrAAY2s5hP3ukWdV9zQscUw2b GwN0/bTxxQgA2NLZzFuHSnkuRX5dbtrun21USPTHMGmFFYBqZqwePZXTcyxp64f3Mtj3g327r/qZ squyPSq5BrF4ivguYoTcGg4SCP7qfiNRFyBUTTERFLYU0n46MuPmVC7vXTsPRQtNRTpJj/b2gGLk 1RcPb1JosS1ct5Mtjs41 -----END CERTIFICATE----- ``` ##### Finish setup 6. After creating the OneLogin application, follow the instructions in the [dbt setup](#dbt-setup) section to complete the integration. #### dbt setup ##### Providing IdP values to dbt To complete setup, follow the steps below in dbt: 1. Navigate to **Account settings** > **SSO & SCIM**. 2. Click **Get started** if SSO has not been configured, or **Edit** if it has already been set up. 3. Provide the following SSO details: | Field | Value | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Log in with | SAML 2.0 | | Identity Provider SSO Url | Paste the **Identity Provider Single Sign-On URL** shown in the IdP setup instructions | | Identity Provider Issuer | Paste the **Identity Provider Issuer** shown in the IdP setup instructions | | X.509 Certificate | Paste the **X.509 Certificate** shown in the IdP setup instructions;<br />**Note:** When the certificate expires, an Idp admin will have to generate a new one to be pasted into dbt for uninterrupted application access. | [![Configuring the application in dbt](/img/docs/dbt-platform/dbt-platform-enterprise/okta/okta-6-setup-integration.png?v=2 "Configuring the application in dbt")](#)Configuring the application in dbt 4. Click **Save** to complete setup for the SAML 2.0 integration. 5. After completing the setup, you can navigate to the URL generated for your account's *slug* to test logging in with your identity provider. Additionally, users added the SAML 2.0 app will be able to log in to dbt from the IdP directly. ##### Additional configuration options The **Single sign-on** section also contains additional configuration options which are located after the credentials fields. * **Sign SAML Auth Request:** dbt will sign SAML requests sent to your identity provider when users attempt to log in. Metadata for configuring this in your identity provider can be downloaded from the value shown in **SAML Metadata URL**. We recommend leaving this disabled for most situations. * **Attribute Mappings:** Associate SAML attributes that dbt needs with attributes your identity provider includes in SAML assertions. The value must be a valid JSON object with the `email`, `first_name`, `last_name`, or `groups` keys and values that are strings or lists of strings. For example, if your identity provider is unable to include an `email` attribute in assertions, but does include one called `EmailAddress`, then **Attribute Mappings** should be set to `{ "email": "EmailAddress" }`. The mappings are only needed if you cannot configure attributes as specified in the instructions on this page. If you can, the default value of `{}` is acceptable. Logging in Users can sign in at <https://login.dbt.com> to view accounts they have access to across instances and choose where to open dbt platform. This is the recommended entry point for most users. For SSO through your identity provider, you can also use the following URL format with your account login URL slug. Replace `LOGIN_SLUG` with the value from the previous steps and `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/enterprise-login/LOGIN-SLUG` Account administrators can turn account discovery on or off with **Enable global account discovery** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). ##### Setting up RBAC After configuring an identity provider, you will be able to set up [role-based access control](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for your account. For common questions and troubleshooting — including "Access Denied" after SAML authentication, ACS URL or Entity ID changes, and email verification issues — refer to [SSO FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/sso-faq.md). --- ##### Single sign-on (SSO) overview dbt platform | Enterprise, Enterprise+ This overview explains how users are provisioned in dbt using single sign-on (SSO). dbt supports JIT (Just-in-Time) provisioning and IdP-initiated login. * Use the <https://login.dbt.com> URL as the recommended way to sign in to dbt platform. After you enter your email, dbt sends a verification code to your email to verify it. Once you verify your email, you can view the accounts you have access to across instances and select an account to open. For a full explanation of the login process, refer to [Log in to dbt platform](https://docs.getdbt.com/docs/platform/about-platform/login.md). * You can also open your account using your **Access URL** from **Account settings** if you already know it. Refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for more information. * Account administrators can turn account discovery on or off with **Enable global account discovery** in [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). When account discovery is disabled, users sign in using the account **Access URL** and their existing [SSO](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) configuration, if applicable. For SSO, users also use your account **Access URL** or **dbt Enterprise Login URL**, as described in this page. In the dbt platform, dbt assigns an SSO slug (an identifier used in your SSO login URL) to your SSO configuration. The slug is system-generated and read-only. Existing SSO configurations remain valid, but you can’t edit the slug. If you delete and recreate your SSO configuration, the new configuration uses a new, system-generated slug. To further automate your workflow, you can use [System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim.md) to provision users, manage group memberships, and automate license assignments directly from your identity provider (IdP) (Okta or Microsoft Entra ID). #### Prerequisites * You have a dbt Enterprise or Enterprise+ plan. [Contact us](mailto:sales@getdbt.com) to learn more, book a demo, or enroll. For plan details, refer to [Pricing](https://www.getdbt.com/pricing/). #### Auth0 URIs The URI used for SSO connections will vary based on your dbt hosted region. To find the Auth0 URI (also called the **Single sign-on URL**, **Authorization URL**, or **Callback URI**) for your environment: 1. Navigate to your **Account settings** and click **SSO & SCIM** in the left menu. 2. In the **Single sign-on** pane, click **Get started** (if SSO has not been configured) or **Edit** (if it has already been set up). 3. Select the appropriate **Identity provider** from the **Provider type** dropdown. 4. The Auth0 URI is displayed under the **Identity provider values** section. The field label depends on the provider you selected: | Identity provider | Field label | Example URI | | ------------------ | --------------------------- | --------------------------------------------------------------- | | SAML 2.0 | **Single sign-on URL** | `https://YOUR_AUTH0_URI/login/callback` | | Okta | **Single sign-on URL** | `https://YOUR_AUTH0_URI/login/callback?connection=ACCOUNT_NAME` | | Google Workspace | **Authorized Redirect URI** | `https://YOUR_AUTH0_URI/login/callback` | | Microsoft Entra ID | **Callback URI** | `https://YOUR_AUTH0_URI/login/callback` | *Replace `YOUR_AUTH0_URI` and `ACCOUNT_NAME` with your account values.* [![Example of the identity provider values for a SAML 2.0 provider](/img/docs/dbt-platform/access-control/sso-uri.png?v=2 "Example of the identity provider values for a SAML 2.0 provider")](#)Example of the identity provider values for a SAML 2.0 provider Auth0 URI The Auth0 URI always contains YOUR\_AUTH0\_URI (for example, auth.cloud.getdbt.com), not your account-specific prefix URL (such as ks123.us1.dbt.com). This is because dbt uses Auth0 as a centralized authentication service across all regions and accounts. You don't need to replace this value with your cell-specific URL. #### SSO process The diagram below explains the basic process by which users are provisioned in dbt upon logging in with SSO. [![SSO diagram](/img/sso_overview.png?v=2 "SSO diagram")](#)SSO diagram ###### Diagram Explanation * **Login Page**: The user accesses the dbt login experience (for example, <https://login.dbt.com>), their account **Access URL**, or their **dbt Enterprise Login URL**, and initiates the SSO flow. * **IdP-Initiated Login**: The user accesses the dbt login page within the Identity Provider by selecting the dbt application. This starts the IdP login flow. * **IdP Login Page**: The user is prompted to log in to the Identity Provider. This grants the dbt application access to the details of their account. * **Login?**: The user can choose to continue or to abort the login process. * **Yes**: The user logs in, grants the dbt application, and continues. * **No**: The user does not log in. They return to the IdP login page. * **User Exists?**: This step checks if the user already exists in dbt's user database. * **Yes**: If so, skip the user creation process * **No**: If so, create a new entry in the dbt database for the new user. * **Create dbt User**: This will create a new entry in the dbt database for the new user. This user record contains the user's email address, first and last name, and any IdP attributes (for example, groups) passed along from the Identity Provider. dbt will send a verification email, and the user must follow the steps in the [User experience section](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md#user-experience) to use SSO in dbt. * **Attach Matching Accounts**: dbt finds all of the accounts configured to match the SSO config used by this user to log in, and then creates a user license record mapping the user to the account. This step also deletes any licenses that the user should not have based on the current SSO config. * **Attach Matching Permissions (Groups)**: dbt iterates through the groups on the matching accounts and finds all that fit one of the following categories: * Have an SSO mapping group that is assigned to the user * Have the **Assign by Default** option selected. Then, assign all of these (and only these) to the user license. This step also removes any permissions that the user should not have based on the current SSO group mappings. * **dbt Application**: After these steps, the user is redirected into the dbt application, and they can begin to use the application normally. License and permission mappings use IdP groups License type mappings and SSO group mappings are based on **IdP group** membership (groups in your identity provider), not dbt platform group names. When configuring [license mappings](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#mapped-configuration) or group assignments, use the group names and memberships from your IdP. #### SSO enforcement * **SSO Enforcement:** If SSO is turned on in your organization, dbt will enforce SSO-only logins for all non-admin users. By default, if an Account Admin or Security Admin already has a password, they can continue logging in with a password. To restrict admins from using passwords, turn off **Allow password logins for account administrators** in the **SSO & SCIM** section of your organization's **Account settings**. * **SSO Re-Authentication:** dbt will prompt you to re-authenticate using your SSO provider every 24 hours to ensure high security. ##### How should non-admin users log in? Non-admin users that currently login with a password can no longer do so. They must sign in using <https://login.dbt.com>, the dbt Enterprise Login URL for your account, or an identity provider (IdP), for example, Okta or Microsoft Entra ID. ##### Security best practices There are a few scenarios that might require you to login with a password. We recommend these security best-practices for the two most common scenarios: * **Onboarding partners and contractors** — We highly recommend that you add partners and contractors to your Identity Provider. IdPs like Okta and Microsoft Entra ID offer capabilities explicitly for temporary employees. We highly recommend that you reach out to your IT team to provision an SSO license for these situations. Using an IdP highly secure, reduces any breach risk, and significantly increases the security posture of your dbt environment. * **Identity Provider is down** — Account admins will continue to be able to log in with a password which would allow them to work with your Identity Provider to troubleshoot the problem. * **Offboarding admins** — When offboarding admins, revoke access to dbt by deleting the user from your environment; otherwise, they can continue to use username/password credentials to log in. ##### Next steps for non-admin users currently logging in with passwords If you have any non-admin users logging into dbt with a password today: 1. Ensure that all users have a user account in your identity provider and are assigned dbt so they won’t lose access. 2. Alert all dbt users that they won’t be able to use a password for logging in anymore unless they are already an Admin with a password. 3. We **DO NOT** recommend promoting any users to Admins just to preserve password-based logins because you will reduce security of your dbt environment. #### FAQ and troubleshooting For common questions and troubleshooting guidance, refer to [SSO FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/sso-faq.md). --- ##### Single sign-on and OAuth dbt platform | Enterprise, Enterprise+ This section covers how to authenticate users and connect data platforms in dbt platform using: * [Single sign-on (SSO)](#sso) * [System for Cross-Domain Identity Management (SCIM)](#scim) * [Connection OAuth](#connection-oauth) These features are available on Enterprise and Enterprise+ plans and are typically configured by account admins or security teams. #### SSO Lets users log in to dbt with your identity provider (IdP) instead of a password. Supports Just-in-Time provisioning and IdP-initiated login. *For admins setting up Okta, Microsoft Entra ID, Google Workspace, or SAML 2.0.* * [Single sign-on (SSO) overview](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) — How SSO works and prerequisites * [Migrating to Auth0 for SSO](https://docs.getdbt.com/docs/platform/manage-access/auth0-migration.md) * [Set up SSO with SAML 2.0](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-saml-2.0.md) * [Set up SSO with Okta](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-okta.md) * [Set up SSO with Google Workspace](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-google-workspace.md) * [Set up SSO with Microsoft Entra ID](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-microsoft-entra-id.md) #### SCIM Automates user and group provisioning from your IdP into dbt (and, with Okta, license assignment). *For admins using Okta or Microsoft Entra ID who want to sync users and groups.* * [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md) — Prerequisites and enabling SCIM in dbt * [Set up SCIM with Okta](https://docs.getdbt.com/docs/platform/manage-access/scim-okta.md) (includes [license management](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md)) * [Set up SCIM with Entra ID](https://docs.getdbt.com/docs/platform/manage-access/scim-entra-id.md) #### Connection OAuth Connection OAuth is for authenticating to your data platform (like Snowflake, BigQuery), which is different from SSO, which handles user login to dbt platform. It lets developers authorize their user credentials with a data platform using that platform's login instead of storing passwords in dbt. *For admins and developers connecting to supported data platforms.* * [OAuth overview](https://docs.getdbt.com/docs/platform/manage-access/oauth-intro.md) — What's available by platform * [Set up Snowflake OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md) * [Set up Databricks OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-databricks-oauth.md) * [Set up BigQuery OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md) * [Set up external OAuth with Snowflake](https://docs.getdbt.com/docs/platform/manage-access/snowflake-external-oauth.md) * [Set up external OAuth with Redshift](https://docs.getdbt.com/docs/platform/manage-access/redshift-external-oauth.md) --- ##### SSO FAQs and troubleshooting dbt platform | Enterprise, Enterprise+ Find answers to common questions about configuring and using single sign-on (SSO) in dbt platform, plus guidance for resolving common issues. #### FAQs  Do SSO group mappings apply to SCIM-managed users? No. When SCIM is enabled, SSO group mappings do not apply to SCIM-managed users. Group membership is controlled entirely by your IdP. This means that if you have a dbt group with SSO group mappings configured, those mappings will only apply to users who authenticate with SSO and are not SCIM-managed  Where do I find my Auth0 URI for configuring SSO? Your Auth0 URI is displayed on the **Single Sign-On** settings page in dbt platform under **Account settings → Single Sign-On**. When you begin the SSO setup flow and select your provider (for example, SAML 2.0 or Microsoft Entra ID), the page will show your IdP values. These are the values you share with your identity provider (IdP) administrator to configure the enterprise application on their end. The Auth0 URI is not the same as your SCIM Base URL, though they share a similar domain pattern. Use the value shown on the SSO settings page directly — do not copy it from the SCIM section.  Can I disable password logins for SSO users? When SSO is configured, it enforces SSO-only logins for the account. However, if [**Allow password logins for account administrators**](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md?version=1.12#sso-enforcement) is enabled, account admins and IT-licensed users can still sign in with email and password. caution Before enabling SSO enforcement, ensure that at least one account admin can successfully sign in through SSO. If SSO is misconfigured and enforcement is enabled, admins may be locked out of the account. Contact <support@getdbt.com> if this occurs.  What Entra ID permissions does the dbt SSO integration require? The required permissions are `User.Read` and `GroupMember.Read.All`. However, depending on your Entra ID tenant configuration — particularly if your tenant has more than 200 groups — you may see an additional request for `Directory.Read.All`, which requires admin consent.  Can I use different IdPs for SSO and SCIM? SSO and SCIM are configured independently in dbt platform, but should use the same IdP. Using different providers — for example, Okta for SSO and Microsoft Entra ID for SCIM — is not supported and can cause unintended behavior. If your organization uses separate IdPs for authentication and directory management, contact your account team to discuss your options. *** #### Troubleshooting  User is stuck on the email verification screen — no verification email received When a user signs in to a dbt platform account through SSO for the first time, a verification email is sent to confirm their identity. If the email is not received, it's usually caused by one of the following: * **Email security tools (for example, Proofpoint and Outlook Safe Links)** may follow the verification link automatically before the user clicks it, consuming the one-time token and causing it to expire. Contact your IT or email security team to allowlist dbt verification link domains. * **The email is being suppressed** in our email delivery provider. Contact <support@getdbt.com> to check delivery logs for the affected address.  Users are prompted for MFA twice when signing in If a user signs in with a regional root URL (for example, `au.dbt.com` or `eu2.dbt.com`) instead of their account-specific URL (for example, `abc123.au.dbt.com`), they may be redirected to an account with an active license, which can result in a second MFA prompt if they haven't remembered their device. Ask users to bookmark and sign in with their account-specific URL. If MFA is only prompted once there, this is expected behavior. Enabling "Remember this device for 30 days" on the first MFA prompt also suppresses the second prompt when using a regional root URL.  "Access Denied" redirect after a successful SAML authentication The SAML assertion succeeds on the IdP side, but dbt platform is rejecting the response. This is almost always caused by a mismatch between the values configured in the IdP and what dbt platform expects. To resolve, verify all the SSO configuration values in dbt platform match what was configured in your IdP  New users see an "Admin consent required" prompt after SSO is set up This occurs when tenant-level admin consent hasn't been granted for the dbt SSO app's permissions. Users who have previously signed in are unaffected, they've already consented individually. **Steps to resolve:** 1. Have an Entra ID admin navigate to **Azure portal → Enterprise Applications → Your dbt application → Permissions**. 2. Click **Grant admin consent** for the organization. This grants consent on behalf of all users in the tenant and prevents the prompt from appearing for new users going forward. 3. Once granted, have the affected users retry signing in.  Receiving a 'AADSTS90094: Admin consent is required' error If you set up SSO before December 2025, your existing configuration may request `Directory.Read.All` instead of `GroupMember.Read.All` for new setups. To use the updated scopes, delete and re-create your SSO [configuration](https://docs.getdbt.com/docs/platform/manage-access/set-up-sso-microsoft-entra-id.md#configuring-permissions).  "This SSO endpoint is disabled. Please contact your administrator to have them update your IdP SSO settings to use Auth0 instead." This error message appears after a successful login when your IdP is misconfigured and the callback URL is pointing to dbt platform instead of Auth0. Your admin needs to update the IdP configuration to use the correct IdP values from **Account settings → Single Sign-On**. --- ##### Users and licenses dbt platform In dbt, *licenses* are used to allocate users to your account. There are four license types in dbt: * **Analyst**\* — Available on [Enterprise and Enterprise+ plans only](https://www.getdbt.com/pricing). * User can be granted *any* permission sets. * \* The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. * **Developer** — User can be granted *any* permission sets. * **IT** — Available on [Starter, Enterprise, and Enterprise+ plans only](https://www.getdbt.com/pricing). User has Security Admin and Billing Admin [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#permission-sets) applied, as well as permissions to edit **Connections** in the **Account settings** page. * Can manage users, groups, connections, and licenses, among other permissions. * *IT licensed users do not inherit rights from any permission sets*. * Every IT licensed user has the same access across the account, regardless of the group permissions assigned. * **Read-Only** — Available on [Starter, Enterprise, and Enterprise+ plans only](https://www.getdbt.com/pricing). * User has read-only permissions applied to all dbt resources. * Intended to view the [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) and the [deploy](https://docs.getdbt.com/docs/deploy/deployments.md) section (jobs, runs, schedules) in a dbt account, but can’t make changes. * *Read-only licensed users do not inherit rights from any permission sets*. * Every read-only licensed user has the same access across the account, regardless of the group permissions assigned. \* The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. The user's assigned license determines the specific capabilities they can access in dbt. | Functionality | Developer or Analyst license\*\* \* | Read-Only users | IT license \* | | ------------------------------------------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | Use the Studio IDE | ✅ | ❌ | ❌ | | Use the dbt CLI | ✅ | ❌ | ❌ | | Use [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) | ✅ | ❌ | ❌ | | Use [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) | ✅ | ❌ | ❌ | | Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) | ✅ | ✅ ([Explore mode](https://docs.getdbt.com/docs/platform/wizard-home.md#ask-questions-in-explore-mode) in the home tab only) | ❌ | | Use Jobs | ✅ | ❌ | ❌ | | Manage Account | ✅ | ❌ | ✅ | | API access<br />(create personal access tokens) | ✅ | ✅ | ✅ | | API access<br />(create service tokens) | ✅ | ❌ | ❌ | | Use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) | ✅ | ✅ | ❌ | | Use [Source Freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) | ✅ | ✅ | ❌ | | Use [Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) | ✅ | ✅ | ❌ | | Receive [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) | ✅ | ✅ | ✅ | | Use the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) | ✅ | ✅ (with [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set) | ❌ | | Manage own warehouse credentials | ✅ | ✅ (with [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set) | ❌ | | Use [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) | ✅ | ✅ (with [Cost Insights Viewer](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#cost-insights-viewer) permission set) | ❌ | <br /> \*The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. \*\*Available on Starter, Enterprise, and Enterprise+ plans only. IT seats are limited to 1 seat per Starter or Enterprise-tier account and don't count toward developer seat usage. #### Licenses License types override group permissions **User license types always override their assigned group permission sets.** For example, a user with a Read-Only license cannot perform administrative actions, even if they belong to an Account Admin group. This ensures that license restrictions are always enforced, regardless of group membership. dbt has different [license types](https://www.getdbt.com/pricing) and you can add or remove them by modifying the number of users in your account settings. If you have a Developer plan account and want to add more people to your team, you'll need to upgrade to the Starter plan. Refer to [dbt Pricing Plans](https://www.getdbt.com/pricing/) for more information about licenses available with each plan. The following tabs detail steps on how to modify your user license count: ##### Enterprise-tier plans If you're on an Enterprise-tier plan and have the correct [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md), you can add or remove licenses by adjusting your user seat count. Note, an IT license does not count toward seat usage. * To remove a user, click on your account name in the left side menu, click **Account settings** and select **Users**. * Select the user you want to remove, click **Edit**, and then **Delete**. * This action cannot be undone. However, you can re-invite the user with the same info if you deleted the user in error. <br /> * To add a user, go to **Account Settings** and select **Users**. * Click the [**Invite Users**](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) button. * For fine-grained permission configuration, refer to [Role based access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-). ##### Starter plans If you're on a Starter plan and have the correct [permissions](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md), you can add or remove developers. Refer to [Self-service Starter account permissions](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md#licenses) for more information on the number of each license type included in the Starter plan. You'll need to make two changes: * Adjust your developer user seat count, which manages the users invited to your dbt project. * Adjust your developer billing seat count, which manages the number of billable seats. You can add or remove developers by increasing or decreasing the number of users and billable seats in your account settings: ##### Adding users To add a user in dbt, you must be an account owner or have admin privileges. 1. From dbt, click on your account name in the left side menu and select **Account settings**. [![Navigate to Account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to Account settings")](#)Navigate to Account settings 2. In **Account Settings**, select **Billing**. 3. Under **Billing details**, enter the number of developer seats you want and make sure you fill in all the payment details, including the **Billing address** section. Leaving these blank won't allow you to save your changes. 4. Press **Update Payment Information** to save your changes. [![Navigate to Account settings -> Billing to modify billing seat count](/img/docs/dbt-platform/faq-account-settings-billing.png?v=2 "Navigate to Account settings -> Billing to modify billing seat count")](#)Navigate to Account settings -> Billing to modify billing seat count Now that you've updated your billing, you can now [invite users](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) to join your dbt account: Great work! After completing those these steps, your dbt user count and billing count should now be the same. ##### Deleting users To delete a user in dbt, you must be an account owner or have admin privileges. If the user has a `developer` license type, this will open up their seat for another user or allow the admins to lower the total number of seats. 1. From dbt, click on your account name in the left side menu and select **Account settings**. [![Navigate to Account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to Account settings")](#)Navigate to Account settings 2. In **Account settings**, select **Users**. 3. Select the user you want to delete, then click **Edit**. 4. Click **Delete** in the bottom left. Click **Confirm Delete** to immediately delete the user without additional password prompts. This action cannot be undone. However, you can re-invite the user with the same information if the deletion was made in error. [![Deleting a user](/img/docs/dbt-platform/delete_user_20221023.gif?v=2 "Deleting a user")](#)Deleting a user If you are on a **Starter** plan and you're deleting users to reduce the number of billable seats, follow these steps to lower the license count to avoid being overcharged: 1. In **Account Settings**, select **Billing**. 2. Under **Billing details**, enter the number of developer seats you want and make sure you fill in all the payment details, including the **Billing address** section. If you leave any field blank, you won't be able to save your changes. 3. Click **Update Payment Information** to save your changes. [![The Billing\*\* page in your \*\*Account settings](/img/docs/dbt-platform/faq-account-settings-billing.png?v=2 "The Billing** page in your **Account settings")](#)The Billing\*\* page in your \*\*Account settings Great work! After completing these steps, your dbt user count and billing count should now be the same. #### Managing license types You can assign licenses to users individually or through group membership. Add users to groups when inviting them or after they join dbt. On Enterprise plans, you can also automatically assign users to groups using [SSO config](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) and [role-based access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-). New users receive your account's default license. Admins can check the default and update it as needed. Refer to [set a different default license](#default-license-for-new-users) for more info. ##### Manual configuration To manually assign a specific type of license to a user on your team: 1. Click on your account name in the left side menu and select **Account settings**. 2. Navigate to the **Users** page in your **Account settings**. 3. Select the user you want to manage and click the **Edit** button. 4. On the **User details** page, you can select the license type and relevant groups for the user. **Note:** You will need to have an available license ready to allocate for the user. If your account does not have an available license to allocate, you will need to add more licenses to your plan to complete the license change. [![Manually assigning licenses](/img/docs/dbt-platform/access-control/license-manual.png?v=2 "Manually assigning licenses")](#)Manually assigning licenses ##### Default license for new users [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Users who aren't assigned a license through SSO, SCIM, or license mapping receive your account's default license. Admins should check the current default and change it to **Read-Only**, **Developer**, or **Analyst** (if available) as needed. To set the default license: 1. Click your account name in the left side menu and select **Account settings**. 2. Go to **Groups & Licenses** and scroll to the **Default license type** section. 3. Click **Edit**. 4. Select **Developer**, **Read-Only**, or **Analyst**. 5. Click **Save**. The **IT** license can't be set as the default. [![Set the default license for new users](/img/docs/dbt-platform/access-control/default-license.png?v=2 "Set the default license for new users")](#)Set the default license for new users ##### Mapped configuration [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") If your account is connected to an Identity Provider (IdP) for [Single Sign On](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md), you can automatically map IdP user groups to specific license types in dbt. For SCIM-based license mapping with Okta, see [Automated license mapping](https://docs.getdbt.com/docs/platform/manage-access/scim-manage-user-licenses.md#automated-license-mapping). ###### Configure license mappings 1. Click on your account name in the left side menu and select **Account settings**. 2. Navigate to **Groups & Licenses** and scroll to the **License mappings** section. 3. Create or edit SSO mappings for both Read-Only and Developer license types. 4. Enter a comma-separated list of **IdP group names** that should receive each license type. [![Configuring IdP group license mapping](/img/docs/dbt-platform/access-control/license-mapping.png?v=2 "Configuring IdP group license mapping")](#)Configuring IdP group license mapping ###### Fundamental licensing rules * **Default assignment**: New members without a mapped license receive your account's [default license](#default-license-for-new-users), unless an admin changes it. * **Mapping basis**: License type mappings are based on *IdP groups* (groups in your identity provider), not *dbt groups*. Check group memberships in your IdP when configuring or troubleshooting. * **When changes take effect**: License types are adjusted when users sign into dbt using single sign-on. Changes to license type mappings take effect the next time users sign in. ###### Mapping logic and precedence When a user belongs to multiple IdP groups, the Developer license takes precedence. The following table shows how group membership determines the assigned license: | In a Developer license-mapped group? | In a Read-Only license-mapped group? | License assigned | | ------------------------------------ | ------------------------------------ | -------------------------------------- | | No | No | Account default license | | No | Yes | Read-Only | | Yes | No | Developer | | Yes | Yes | Developer (Developer takes precedence) | note If a user's IdP groups do not match *any* license type mappings, dbt assigns your account's [default license](#default-license-for-new-users). Admins can check the default and update as needed. #### Granular permissioning [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") dbt Enterprise-tier plans support role-based access controls for configuring granular in-app permissions. See [access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) for more information on Enterprise permissioning. ###### Read-Only granular permissions dbt Enterprise-tier plans support granular permissions to help you control which projects your Read-Only users can access by assigning them to specific groups, rather than granting access to the entire account. Enabling granular permissions is a one-time, permanent change. Before you enable this, make sure your Read-Only users are in a group with a Read-Only permission set that covers all projects. If a user isn't in a group, or their group's permission set only covers some projects, they'll lose access to the projects that aren't covered. For complete setup instructions, refer to [How to enable granular permissions for read-only users](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#enable-granular-permissions-for-read-only-users) for the full setup steps. --- #### Secure ##### About network security dbt platform Network security in dbt gives you control over how traffic flows between dbt and your infrastructure. Choose the approach that best fits your security requirements. #### Choose your connectivity approach [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md) ###### [Over the public internet](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md) [Use IP restrictions to limit which IP addresses can access dbt Cloud or your data platform.](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/secure/private-connectivity/private-connectivity.md) ###### [Over a private network](https://docs.getdbt.com/docs/platform/secure/private-connectivity/private-connectivity.md) [Use your cloud provider's private connectivity technology to keep traffic off the public internet entirely.](https://docs.getdbt.com/docs/platform/secure/private-connectivity/private-connectivity.md) --- ##### About private connectivity dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. Private connections enables secure communication from any dbt environment to your data platform hosted on a cloud provider, such as [AWS](https://aws.amazon.com/privatelink/), [Azure](https://azure.microsoft.com/en-us/products/private-link), or [GCP](https://cloud.google.com/vpc/docs/private-service-connect), using that provider's private connection technology. Private connections allow dbt customers to meet security and compliance controls as it allows connectivity between dbt and your data platform without traversing the public internet. This feature is supported in most regions across North America, Europe, and Asia, but [contact us](https://www.getdbt.com/contact/) if you have questions about availability. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Snowflake) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Private endpoint configuration When managing private connectivity, consider the following: * Using [Environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) when configuring private connection endpoints isn't supported in dbt. Use [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) instead, which explains where to configure them for deployment and development environments. * The [Administrative API v3](https://docs.getdbt.com/dbt-cloud/api-v3) supports private endpoint operations — [`list`](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/List%20Private%20Endpoints), [`create`](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Create%20Private%20Endpoints%20List%20Alias%20View), [`retrieve`](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Retrieve%20Private%20Endpoint), [`update`](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Update%20Private%20Endpoints%20Detail%20View), and [`delete`](https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Delete%20Private%20Endpoints%20Detail%20View). You can use these endpoints to manage private connectivity programmatically. #### Available platforms Select your cloud platform to view private connectivity options, support matrix, and configuration guides. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-overview.md) ###### [AWS](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-overview.md) [Amazon Web Services PrivateLink](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-overview.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-overview.md) ###### [Azure](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-overview.md) [Microsoft Azure Private Link](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-overview.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform/secure/private-connectivity/gcp/gcp-overview.md) ###### [GCP](https://docs.getdbt.com/docs/platform/secure/private-connectivity/gcp/gcp-overview.md) [Google Cloud Platform Private Service Connect](https://docs.getdbt.com/docs/platform/secure/private-connectivity/gcp/gcp-overview.md) --- ##### AWS private connectivity dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. AWS PrivateLink enables secure, private connectivity between dbt and your AWS-hosted services. With PrivateLink, traffic between dbt and your data platforms or self-hosted services stays within the AWS network and does not traverse the public internet. For more details, refer to the [AWS PrivateLink documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/). #### AWS private connectivity matrix The following charts outline private connectivity options for AWS deployments of dbt ([multi-tenant and single-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md)). **Legend:** * ✅ = Available * ❌ = Not currently available *Tenancy:* MT (multi-tenant) and ST (single-tenant) — [learn more about tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). About the following matrix tables These tables indicate whether private connectivity can be established to specific services, considering major factors such as the network and basic auth layers. dbt has validated these configurations using common deployment patterns and typical use cases. However, individual configurations may vary. If you encounter issues or have questions about your environment, [contact dbt Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support) for guidance. *** ##### Connecting to the dbt platform (Ingress) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Your services can connect to dbt over private connectivity using the dbt-provisioned model. In this case, dbt is the service producer and you are the consumer. | Connectivity type | MT | ST | Setup guide | | ------------------------------ | -- | -- | -------------------------------------------------------------------------------------------- | | Private dbt access | ❌ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-ingress.md) | | Dual access (public + private) | ❌ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-ingress.md) | *** ##### Connecting the dbt platform to managed services (Egress) dbt can establish private connections to managed data platforms and cloud-native services. | Service | MT | ST | Setup guide | | -------------------------- | -- | -- | ----------------------------------------------------------------------------------------------- | | Snowflake | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-snowflake.md) | |   Snowflake Internal Stage | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-snowflake.md) | | Databricks | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-databricks.md) | | Redshift | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-redshift.md) | | Redshift Serverless | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-redshift.md) | | Amazon Athena w/ AWS Glue | ❌ | ✅ | | | AWS CodeCommit | ❌ | ✅ | | | Teradata VantageCloud | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-teradata.md) | *** ##### Connecting the dbt platform to self-hosted services (Egress) All of the services below share a common PrivateLink setup guide — backend configuration varies by service. Self-hosted connections use the customer-provisioned model — you are the service producer and dbt is the consumer. **Setup guide:** [Configuring AWS PrivateLink for self-hosted services](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-self-hosted.md) | Service | MT | ST | | ------------------------ | -- | -- | | GitHub Enterprise Server | ✅ | ✅ | | GitLab Self-Managed | ✅ | ✅ | | Bitbucket Data Center | ✅ | ✅ | | Azure DevOps Server | ✅ | ✅ | | Postgres | ✅ | ✅ | | Spark | ✅ | ✅ | | Starburst / Trino | ✅ | ✅ | | Teradata (self-hosted) | ✅ | ✅ | If you have questions about whether your specific architecture is supported, [contact dbt Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support). #### Cross-region private connections dbt Labs has globally connected private networks specifically used to host private endpoints, which are connected to dbt instance environments. This connectivity allows dbt environments to connect to any supported region from any dbt instance within the same cloud provider network. To ensure security, access to these endpoints is protected by security groups, network policies, and application connection safeguards, in addition to the authentication and authorization mechanisms provided by each of the connected platforms. --- ##### Azure private connectivity dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. Azure Private Link enables secure, private connectivity between dbt and your Azure-hosted services. With Private Link, traffic between dbt and your data platforms or self-hosted services stays within the Azure network and does not traverse the public internet. For more details, refer to the [Azure Private Link documentation](https://learn.microsoft.com/en-us/azure/private-link/). #### Azure private connectivity matrix The following charts outline private connectivity options for Azure deployments of dbt ([multi-tenant and single-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md)). **Legend:** * ✅ = Available * ❌ = Not currently available *Tenancy:* MT (multi-tenant) and ST (single-tenant) — [learn more about tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). About the following matrix tables These tables indicate whether private connectivity can be established to specific services, considering major factors such as the network and basic auth layers. dbt has validated these configurations using common deployment patterns and typical use cases. However, individual configurations may vary. If you encounter issues or have questions about your environment, [contact dbt Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support) for guidance. *** ##### Connecting to the dbt platform (Ingress) Your services can connect to dbt over private connectivity using the dbt-provisioned model. In this case, dbt is the service producer and you are the consumer. | Connectivity type | MT | ST | | ------------------------------ | -- | -- | | Private dbt access | ❌ | ✅ | | Dual access (public + private) | ❌ | ✅ | *** ##### Connecting the dbt platform to managed services (Egress) dbt can establish private connections to managed data platforms and cloud-native services. | Service | MT | ST | Setup guide | | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -- | -- | --------------------------------------------------------------------------------------------------- | | Snowflake | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-snowflake.md) | |   Snowflake Internal Stage | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-snowflake.md) | | Databricks | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-databricks.md) | | Azure Database for PostgreSQL Flexible Server | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-postgres.md) | | Azure Synapse | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-synapse.md) | | Azure Fabric [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-fabric.md) | | Teradata VantageCloud | ✅ | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-teradata.md) | *** ##### Connecting the dbt platform to self-hosted services (Egress) All of the services below share a common Private Link setup guide — backend configuration varies by service. Self-hosted connections use the customer-provisioned model — you are the service producer and dbt is the consumer. **Setup guide:** [Configuring Azure Private Link for self-hosted services](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-self-hosted.md) | Service | MT | ST | | ------------------------ | -- | -- | | GitHub Enterprise Server | ✅ | ✅ | | GitLab Self-Managed | ✅ | ✅ | | Bitbucket Data Center | ✅ | ✅ | | Azure DevOps Server | ✅ | ✅ | | Postgres | ✅ | ✅ | | Starburst / Trino | ✅ | ✅ | | Teradata (self-hosted) | ✅ | ✅ | If you have questions about whether your specific architecture is supported, [contact dbt Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support). #### Cross-region private connections dbt Labs has globally connected private networks specifically used to host private endpoints, which are connected to dbt instance environments. This connectivity allows dbt environments to connect to any supported region from any dbt instance within the same cloud provider network. To ensure security, access to these endpoints is protected by security groups, network policies, and application connection safeguards, in addition to the authentication and authorization mechanisms provided by each of the connected platforms. --- ##### Configure AWS PrivateLink for Postgres dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. A Postgres database, hosted either in AWS or in a properly connected on-prem data center, can be accessed through a private network connection using AWS Interface-type PrivateLink. The type of Target Group connected to the Network Load Balancer (NLB) may vary based on the location and type of Postgres instance being connected, as explained in the following steps. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Postgres) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Configuring Postgres interface-type PrivateLink ##### 1. Provision AWS resources Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in the account containing, or connected to, the Postgres instance: * **Security Group (AWS hosted only)** — If you are connecting to an existing Postgres instance, this likely already exists, however, you may need to add or modify Security Group rules to accept traffic from the Network Load Balancer (NLB) created for this Endpoint Service. * **Target Group** — The Target Group will be attached to the NLB to tell it where to route requests. There are various target types available for NLB Target Groups, so choose the one appropriate for your Postgres setup. * Target Type: * *[Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/)* - **IP** * Find the IP address of your RDS instance using a command line tool such as `nslookup <endpoint>` or `dig +short <endpoint>` with your RDS DNS endpoint * *Note*: With RDS Multi-AZ failover capabilities the IP address of your RDS instance can change, at which point your Target Group would need to be updated. See [this AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for more details and a possible solution. * *On-prem Postgres server* - **IP** * Use the IP address of the on-prem Postgres server linked to AWS through AWS Direct Connect or a Site-to-Site VPN connection * *Postgres on EC2* - **Instance/ASG** (or **IP**) * If your Postgres instance is hosted on EC2 the *instance* Target Group type (or ideally [using the instance type to connect to an auto-scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/attach-load-balancer-asg.html)) can be used to attach the instance without needing a static IP address * The IP type can also be used, with the understanding that the IP of the EC2 instance can change if the instance is relaunched for any reason * Target Group protocol: **TCP** * **Network Load Balancer (NLB)** — Requires creating a Listener that attaches to the newly created Target Group for port `5432` * **Scheme:** Internal * **IP address type:** IPv4 * **Network mapping:** Choose the VPC that the VPC Endpoint Service and NLB are being deployed in, and choose subnets from at least two Availability Zones. * **Security Groups:** The Network Load Balancer (NLB) associated with the VPC endpoint service must either not have an associated security group, or the security group must have a rule that allows requests from the appropriate dbt **private CIDR(s)**. Note that *this is different* than the static public IPs listed on the dbt [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) page. dbt Support can provide the correct private CIDR(s) upon request. If necessary, until you can refine the rule to the smaller CIDR provided by dbt, allow connectivity by temporarily adding an allow rule of `10.0.0.0/8`. * **Listeners:** Create one listener per target group that maps the appropriate incoming port to the corresponding target group ([details](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html)). * **VPC Endpoint Service** — Attach to the newly created NLB. * Acceptance required (optional) — Requires you to [accept our connection request](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests) after dbt creates the endpoint. Cross-Zone Load Balancing We highly recommend cross-zone load balancing for your NLB or Target Group; some connections may require it. Cross-zone load balancing may also [improve routing distribution and connection resiliency](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#cross-zone-load-balancing). Note that cross-zone connectivity may incur additional data transfer charges, though this should be minimal for requests from dbt. * [Enabling cross-zone load balancing for a load balancer or target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/edit-target-group-attributes.html#target-group-cross-zone) ##### 2. Grant dbt AWS account access to the VPC endpoint service On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the root user in the appropriate production AWS account and save your changes. * Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` [![Enter ARN](/img/docs/dbt-platform/privatelink-allow-principals.png?v=2 "Enter ARN")](#)Enter ARN ##### 3. Obtain VPC endpoint service name Once the VPC Endpoint Service is provisioned, you can find the service name in the AWS console by navigating to **VPC** → **Endpoint Services** and selecting the appropriate endpoint service. You can copy the service name field value and include it in your communication to dbt support. [![Get service name field value](/img/docs/dbt-platform/privatelink-endpoint-service-name.png?v=2 "Get service name field value")](#)Get service name field value ##### 4. Submit your request to dbt Support Add the required information to the template below and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Multi-Tenant PrivateLink Request - Type: Postgres Interface-type - dbt platform account URL: - VPC Endpoint Service Name: - Postgres server AWS Region (for example, us-east-1, eu-west-2): - dbt AWS multi-tenant environment (US, EMEA, AU, JP): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. ##### 5. Accept the connection request When you receive notification that the resources are provisioned within the dbt environment, you must accept the endpoint connection (unless the VPC endpoint service is set to auto-accept connection requests). You can accept requests through the AWS console, as shown below, or through the AWS CLI. [![Accept the connection request](/img/docs/dbt-platform/platform-configuring-dbt-platform/accept-request.png?v=2 "Accept the connection request")](#)Accept the connection request #### Create connection in dbt Once dbt Support completes the configuration, you can start creating new connections using PrivateLink. 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **PostgreSQL**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. #### Troubleshooting If the PrivateLink endpoint has been provisioned and configured in dbt but connectivity is still failing, check the following in your networking setup to ensure requests and responses can be successfully routed between dbt and the backing service. ##### Configuration Start with the configuration:  1. NLB Security Group The Network Load Balancer (NLB) associated with the VPC Endpoint Service must either not have an associated Security Group or the Security Group must have a rule that allows requests from the appropriate dbt *private CIDR(s)*. Note that this differs from the static public IPs listed on the dbt Connection page. dbt Support can provide the correct private CIDR(s) upon request. * **Note**\*: To test if this is the issue, temporarily adding an allow rule of `10.0.0.0/8` should allow connectivity until the rule can be refined to a smaller CIDR  2. NLB Listener and Target Group Check that there is a Listener connected to the NLB that matches the port that dbt is trying to connect to. This Listener must have a configured action to forward to a Target Group with targets that point to your backing service. At least one (but preferably all) of these targets must be **Healthy**. Unhealthy targets could suggest that the backing service is, in fact, unhealthy or that the service is protected by a Security Group that doesn't allow requests from the NLB.  3. Cross-zone Load Balancing Check that *Cross-zone load balancing* is enabled for your NLB (check the **Attributes** tab of the NLB in the AWS console). If this is disabled, and the zones that dbt is connected to are misaligned with the zones where the service is running, requests may not be able to be routed correctly. Enabling cross-zone load balancing will also make the connection more resilient in the case of a failover in a zone outage scenario. Cross-zone connectivity may incur additional data transfer charges, though this should be minimal for requests from dbt.  4. Routing tables and ACLs If all the above check out, it may be possible that requests are not routing correctly within the private network. This could be due to a misconfiguration in the VPCs routing tables or access control lists. Review these settings with your network administrator to ensure that requests can be routed from the VPC Endpoint Service to the backing service and that the response can be returned to the VPC Endpoint Service. One way to test this is to create a VPC endpoint in another VPC in your network to test that connectivity is working independent of dbt's connection. ##### Monitoring To help isolate connection issues over a PrivateLink connection from dbt, there are a few monitoring sources that can be used to verify request activity. Requests must first be sent to the endpoint to see anything in the monitoring. Contact dbt Support to understand when connection testing occurred or request new connection attempts. Use these times to correlate with activity in the following monitoring sources.  VPC Endpoint Service Monitoring In the AWS Console, navigate to VPC -> Endpoint Services. Select the Endpoint Service being tested and click the **Monitoring** tab. Update the time selection to include when test connection attempts were sent. If there is activity in the *New connections* and *Bytes processed* graphs, then requests have been received by the Endpoint Service, suggesting that the dbt endpoint is routing properly.  NLB Monitoring In the AWS Console, navigate to EC2 -> Load Balancers. Select the Network Load Balancer (NLB) being tested and click the **Monitoring** tab. Update the time selection to include when test connection attempts were sent. If there is activity in the *New flow count* and *Processed bytes* graphs, then requests have been received by the NLB from the Endpoint Service, suggesting the NLB Listener, Target Group, and Security Group are correctly configured.  VPC Flow Logs VPC Flow Logs can provide various helpful information for requests being routed through your VPCs, though they can sometimes be challenging to locate and interpret. Flow logs can be written to either S3 or CloudWatch Logs, so determine the availability of these logs for your VPC and query them accordingly. Flow logs record the Elastic Network Interface (ENI) ID, source and destination IP and port, and whether the request was accepted or rejected by the security group and/or network ACL. This can be useful in understanding if a request arrived at a certain network interface and whether that request was accepted, potentially illuminating overly restrictive rules. For more information on accessing and interpreting VPC Flow Logs, see the related [AWS documentation](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html). --- ##### Configure AWS PrivateLink for Redshift dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. AWS provides two different ways to create a PrivateLink VPC endpoint for a Redshift cluster that is running in another VPC: * [Redshift-managed PrivateLink Endpoints](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-cross-vpc.html) * [Redshift Interface-type PrivateLink Endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) dbt supports both types of endpoints, but there are several [considerations](https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-cross-vpc.html#managing-cluster-cross-vpc-considerations) to take into account when deciding which endpoint type to use. Redshift-managed provides a simpler setup with no additional cost, which might make it the preferred option for many, but may not be an option in all environments. Based on these criteria, determine which type is right for your system. Follow the instructions from the section below that corresponds to your chosen endpoint type. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Redshift) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. You can set up a Redshift AWS PrivateLink endpoint in two ways: * [Self-serve private endpoints](#self-serve-private-endpoints): Create and manage Redshift PrivateLink endpoints directly in the dbt platform user interface. Currently in beta. * [Support-led setup](#support-led-setup): Contact dbt Support to configure your Redshift PrivateLink endpoint. The AWS-side setup is the same for both paths. You provision your AWS resources (grant access for Redshift-managed, or create the VPC endpoint service for interface-type), then either submit the request in the UI (self-serve) or email dbt Support (support-led). #### Self-serve private endpoints [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") *Self-serve private endpoints are currently in beta for Redshift on AWS, and available to all eligible customers. If you don't see **Private endpoints** in your account settings, use the [Support-led setup](#support-led-setup) instead.* With self-serve, you request a Redshift PrivateLink endpoint in dbt platform without opening a support ticket. If a request fails, you can edit the request and resubmit, or delete the endpoint and retry on your own. dbt platform supports all three Redshift types through self-serve: * **Redshift Managed** — provisioned Redshift clusters (select **Redshift Managed** in the UI). * **Redshift Managed Serverless** — Redshift Serverless (select **Redshift Managed** in the UI, then choose **Serverless**). * **Redshift interface-type** (select **Redshift** in the UI). ##### Prerequisites * [Account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md?version=2.0#account-admin) or [Project creator](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md?version=2.0#project-creator) permission sets in dbt platform. Users with an IT license can also create private endpoints. * Completed the AWS-side setup for your endpoint type. Follow the same steps as the [support-led setup](#support-led-setup) — grant access for [Redshift-managed](#configuring-redshift-managed-privatelink), or provision the VPC endpoint service for [interface-type](#configuring-redshift-interface-type-privatelink) — but stop before the **Submit your request to dbt Support** step and use the UI below instead. ##### Request a new private endpoint 1. In dbt platform, go to **Account settings → Private endpoints**. 2. In the **Private endpoints** table, review your existing endpoints. The table shows all private endpoints in your account (including non-Redshift ones) with details like **Name**, **Connection type**, **URL**, **Connectivity status**, and the number of **Connections** using the endpoint. You can search by **Name** or **URL**. 3. To request a new endpoint, click **Request new**. 4. Under **Provider type**, select the option that matches your endpoint type, then fill in the fields for that type: ##### Redshift Managed / Serverless Select **Redshift Managed** for both provisioned Redshift clusters and Redshift Serverless. 1. In **Step 1: Choose your deployment type**, select **Provisioned** or **Serverless**. If you choose **Serverless** and have multiple workgroups, make sure you grant access under **Granted accounts** for each workgroup during the AWS-side setup. Refer to [Per-workgroup authorization required](#per-workgroup-authorization). 2. In **Step 2: Enter your cluster details**, enter your **Cluster identifier** and **Resource owner AWS account ID**. 3. In **Step 3: Select your AWS region**, choose the AWS region where your Redshift cluster or workgroup is hosted. 4. Click **Submit request**. [![Redshift Managed endpoint request form showing deployment type, cluster details, and AWS region fields](/img/docs/dbt-platform/redshift-managed-private-endpoint-request.png?v=2 "Redshift Managed endpoint request form showing deployment type, cluster details, and AWS region fields")](#)Redshift Managed endpoint request form showing deployment type, cluster details, and AWS region fields ##### Redshift (interface-type) Select **Redshift** for an interface-type endpoint. 1. In **Step 1: Enter your AWS PrivateLink service name**, enter the name of the AWS VPC endpoint service you configured for your Redshift cluster (for example, `com.amazonaws.vpce.us-east-1.vpce-svc-xxxxxxxxxxxxxxxxx`). 2. In **Step 2: Name your Redshift endpoint**, choose a name to identify the endpoint. This is used as the hostname prefix when it's registered. 3. Click **Submit request**. [![Redshift interface-type endpoint request form showing AWS PrivateLink service name and endpoint name fields](/img/docs/dbt-platform/redshift-interface-private-endpoint-request.png?v=2 "Redshift interface-type endpoint request form showing AWS PrivateLink service name and endpoint name fields")](#)Redshift interface-type endpoint request form showing AWS PrivateLink service name and endpoint name fields 5. After submission, you'll see the request and its status in the **Private endpoints** table. Once approved, you'll be notified. 6. Proceed to the **Connections** page and follow the steps in the [Create connection in dbt](#create-connection-in-dbt) section to configure PrivateLink. Once configured, the new endpoint appears under **Private endpoints → Associated connections**. DNS propagation If the connection test fails immediately after setup, this is expected — it doesn't mean something is wrong. DNS changes can take a few minutes to propagate. Wait a few minutes, then test again before contacting support. ##### Troubleshooting and errors If an endpoint request fails, dbt platform displays the error in a banner on the endpoint details page, along with details that are safe to share externally. Review the message, correct the underlying issue (for example, confirm dbt's AWS account has been granted access under the cluster's **Granted accounts** section), then click **Retry**. [![Provisioning failed banner on a Redshift endpoint details page with a Retry button](/img/docs/dbt-platform/redshift-private-endpoint-error.png?v=2 "Provisioning failed banner on a Redshift endpoint details page with a Retry button")](#)Provisioning failed banner on a Redshift endpoint details page with a Retry button If you see a failure state without clear next steps, collect the request details (endpoint name, creation time, and status) and contact [dbt Support](mailto:support@getdbt.com). #### Support-led setup If **Private endpoints** isn't available in your account settings, configure Redshift PrivateLink by following the steps below and submitting a request to dbt Support. #### Configuring Redshift-managed PrivateLink 1. Locate the **Granted accounts** section of the Redshift configuration * **Standard Redshift** * On the running Redshift cluster, select the **Properties** tab. [![Redshift Properties tab](/img/docs/dbt-platform/redshiftprivatelink1.png?v=2 "Redshift Properties tab")](#)Redshift Properties tab * **Redshift Serverless** * On the Redshift Serverless **Workgroup configuration** page. If you have multiple workgroups, make sure you grant access under **Granted accounts** for each workgroup during the AWS-side setup. Refer to [Per-workgroup authorization required](#per-workgroup-authorization). 2. In the **Granted accounts** section, click **Grant access**. [![Redshift granted accounts](/img/docs/dbt-platform/redshiftprivatelink2.png?v=2 "Redshift granted accounts")](#)Redshift granted accounts 3. Enter the AWS account ID: `346425330055` - *NOTE: This account ID only applies to dbt Multi-Tenant environments. For Virtual Private/Single-Tenant account IDs please contact [Support](mailto:support@getdbt.com).* 4. Choose **Grant access to all VPCs** —or— (optional) contact [Support](mailto:support@getdbt.com) for the appropriate regional VPC ID to designate in the **Grant access to specific VPCs** field. [![Redshift grant access](/img/docs/dbt-platform/redshiftprivatelink3.png?v=2 "Redshift grant access")](#)Redshift grant access 5. Add the required information to the following template, and submit your request to [dbt Support](mailto:support@getdbt.com): * **Standard Redshift**  Support request email template ```text Subject: New Multi-Tenant PrivateLink Request - Type: Redshift-managed - dbt platform account URL: - Redshift cluster name: - Redshift cluster AWS account ID: - Redshift cluster AWS Region (for example, us-east-1, eu-west-2): - dbt multi-tenant environment (US, EMEA, AU, JP): ``` * **Redshift Serverless**  Support request email template ```text Subject: New Multi-Tenant PrivateLink Request - Type: Redshift-managed - Serverless - dbt platform account URL: - Redshift workgroup name: - Redshift workgroup AWS account ID: - Redshift workgroup AWS Region (for example, us-east-1, eu-west-2): - dbt multi-tenant environment (US, EMEA, AU, JP): ``` ##### Per-workgroup authorization required For Redshift Serverless, **Granted accounts** is scoped to a single workgroup. If your environment has multiple workgroups, you must grant access separately for each workgroup you want to connect to dbt. * Authorizing one workgroup (for example, *sales*) doesn't extend to any other workgroup (for example, *finance* or *product*). * Repeat steps 1 to 4 in. [Configuring Redshift-managed](#configuring-redshift-managed-privatelink) section for each workgroup individually. This applies whether you set up the endpoint through [self-serve](#self-serve-private-endpoints) or support-led. dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### Configuring Redshift interface-type PrivateLink ##### 1. Provision AWS resources Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in the account containing the Redshift cluster: * **Security Group** — If you are connecting to an existing Redshift cluster, this likely already exists, however, you may need to add or modify Security Group rules to accept traffic from the Network Load Balancer (NLB) created for this Endpoint Service. * **Target Group** — The Target Group will be attached to the NLB to tell it where to route requests. There are various target types available for NLB Target Groups, but you will use the IP address type. * Target Type: **IP** * **Standard Redshift** * Use IP addresses from the Redshift cluster’s **Network Interfaces** whenever possible. While IPs listed in the **Node IP addresses** section will work, they are also more likely to change. [![Target type: IP address](/img/docs/dbt-platform/redshiftprivatelink4.png?v=2 "Target type: IP address")](#)Target type: IP address * There will likely be only one Network Interface (NI) to start, but if the cluster fails over to another availability zone (AZ), a new NI will also be created for that AZ. The NI IP from the original AZ will still work, but the new NI IP can also be added to the Target Group. If adding additional IPs, note that the NLB will also need to add the corresponding AZ. Once created, the NI(s) should stay the same (This is our observation from testing, but AWS does not officially document it). * **Redshift Serverless** * To find the IP addresses for Redshift Serverless instance locate and copy the endpoint (only the URL listed before the port) in the Workgroup configuration section of the AWS console for the instance. [![Redshift Serverless endpoint](/img/docs/dbt-platform/redshiftserverless.png?v=2 "Redshift Serverless endpoint")](#)Redshift Serverless endpoint * From a command line run the command `nslookup <endpoint>` using the endpoint found in the previous step and use the associated IP(s) for the Target Group. * Target Group protocol: **TCP** * **Network Load Balancer (NLB)** — Requires creating a Listener that attaches to the newly created Target Group (port `5439`is the default) * **Scheme:** Internal * **IP address type:** IPv4 * **Network mapping:** Choose the VPC that the VPC Endpoint Service and NLB are being deployed in, and choose subnets from at least two Availability Zones. * **Security Groups:** The Network Load Balancer (NLB) associated with the VPC endpoint service must either not have an associated security group, or the security group must have a rule that allows requests from the appropriate dbt **private CIDR(s)**. Note that *this is different* than the static public IPs listed on the dbt [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) page. dbt Support can provide the correct private CIDR(s) upon request. If necessary, until you can refine the rule to the smaller CIDR provided by dbt, allow connectivity by temporarily adding an allow rule of `10.0.0.0/8`. * **Listeners:** Create one listener per target group that maps the appropriate incoming port to the corresponding target group ([details](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html)). * **VPC Endpoint Service** — Attach to the newly created NLB. * Acceptance required (optional) — Requires you to [accept our connection request](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests) after dbt creates the endpoint. Cross-Zone Load Balancing We highly recommend cross-zone load balancing for your NLB or Target Group; some connections may require it. Cross-zone load balancing may also [improve routing distribution and connection resiliency](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#cross-zone-load-balancing). Note that cross-zone connectivity may incur additional data transfer charges, though this should be minimal for requests from dbt. * [Enabling cross-zone load balancing for a load balancer or target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/edit-target-group-attributes.html#target-group-cross-zone) ##### 2. Grant dbt AWS account access to the VPC endpoint service On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the root user in the appropriate production AWS account and save your changes. * Principal: `arn:aws:iam::346425330055:role/MTPL_Admin` [![Enter ARN](/img/docs/dbt-platform/privatelink-allow-principals.png?v=2 "Enter ARN")](#)Enter ARN ##### 3. Obtain VPC endpoint service name Once the VPC Endpoint Service is provisioned, you can find the service name in the AWS console by navigating to **VPC** → **Endpoint Services** and selecting the appropriate endpoint service. You can copy the service name field value and include it in your communication to dbt support. [![Get service name field value](/img/docs/dbt-platform/privatelink-endpoint-service-name.png?v=2 "Get service name field value")](#)Get service name field value ##### 4. Submit your request to dbt Support Add the required information to the template below and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Multi-Tenant PrivateLink Request - Type: Redshift Interface-type - dbt platform account URL: - VPC Endpoint Service Name: - Redshift cluster AWS Region (for example, us-east-1, eu-west-2): - dbt AWS multi-tenant environment (US, EMEA, AU, JP): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### Create connection in dbt Once dbt Support completes the configuration, you can start creating new connections using PrivateLink. 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Redshift**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. #### Troubleshooting If the PrivateLink endpoint has been provisioned and configured in dbt but connectivity is still failing, check the following in your networking setup to ensure requests and responses can be successfully routed between dbt and the backing service. ##### Configuration Start with the configuration:  1. NLB Security Group The Network Load Balancer (NLB) associated with the VPC Endpoint Service must either not have an associated Security Group or the Security Group must have a rule that allows requests from the appropriate dbt *private CIDR(s)*. Note that this differs from the static public IPs listed on the dbt Connection page. dbt Support can provide the correct private CIDR(s) upon request. * **Note**\*: To test if this is the issue, temporarily adding an allow rule of `10.0.0.0/8` should allow connectivity until the rule can be refined to a smaller CIDR  2. NLB Listener and Target Group Check that there is a Listener connected to the NLB that matches the port that dbt is trying to connect to. This Listener must have a configured action to forward to a Target Group with targets that point to your backing service. At least one (but preferably all) of these targets must be **Healthy**. Unhealthy targets could suggest that the backing service is, in fact, unhealthy or that the service is protected by a Security Group that doesn't allow requests from the NLB.  3. Cross-zone Load Balancing Check that *Cross-zone load balancing* is enabled for your NLB (check the **Attributes** tab of the NLB in the AWS console). If this is disabled, and the zones that dbt is connected to are misaligned with the zones where the service is running, requests may not be able to be routed correctly. Enabling cross-zone load balancing will also make the connection more resilient in the case of a failover in a zone outage scenario. Cross-zone connectivity may incur additional data transfer charges, though this should be minimal for requests from dbt.  4. Routing tables and ACLs If all the above check out, it may be possible that requests are not routing correctly within the private network. This could be due to a misconfiguration in the VPCs routing tables or access control lists. Review these settings with your network administrator to ensure that requests can be routed from the VPC Endpoint Service to the backing service and that the response can be returned to the VPC Endpoint Service. One way to test this is to create a VPC endpoint in another VPC in your network to test that connectivity is working independent of dbt's connection. ##### Monitoring To help isolate connection issues over a PrivateLink connection from dbt, there are a few monitoring sources that can be used to verify request activity. Requests must first be sent to the endpoint to see anything in the monitoring. Contact dbt Support to understand when connection testing occurred or request new connection attempts. Use these times to correlate with activity in the following monitoring sources.  VPC Endpoint Service Monitoring In the AWS Console, navigate to VPC -> Endpoint Services. Select the Endpoint Service being tested and click the **Monitoring** tab. Update the time selection to include when test connection attempts were sent. If there is activity in the *New connections* and *Bytes processed* graphs, then requests have been received by the Endpoint Service, suggesting that the dbt endpoint is routing properly.  NLB Monitoring In the AWS Console, navigate to EC2 -> Load Balancers. Select the Network Load Balancer (NLB) being tested and click the **Monitoring** tab. Update the time selection to include when test connection attempts were sent. If there is activity in the *New flow count* and *Processed bytes* graphs, then requests have been received by the NLB from the Endpoint Service, suggesting the NLB Listener, Target Group, and Security Group are correctly configured.  VPC Flow Logs VPC Flow Logs can provide various helpful information for requests being routed through your VPCs, though they can sometimes be challenging to locate and interpret. Flow logs can be written to either S3 or CloudWatch Logs, so determine the availability of these logs for your VPC and query them accordingly. Flow logs record the Elastic Network Interface (ENI) ID, source and destination IP and port, and whether the request was accepted or rejected by the security group and/or network ACL. This can be useful in understanding if a request arrived at a certain network interface and whether that request was accepted, potentially illuminating overly restrictive rules. For more information on accessing and interpreting VPC Flow Logs, see the related [AWS documentation](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html). --- ##### Configuring AWS PrivateLink for a self-hosted service dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. AWS PrivateLink enables secure, private connectivity between dbt and your self-hosted services. These services may include version control systems (VCS), data warehouses, or any other applications you manage. With PrivateLink, you do not need to expose your service to the public internet. All communication occurs over a private network, significantly enhancing security. For more details, refer to the [AWS PrivateLink documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/). #### What this guide covers The focus of this guide is not on any particular service or backend architecture, but on the [Endpoint Service](#terminology) that interconnects dbt with your self-hosted service. This process should be standard across most use cases. [![The scope of this guide](/img/docs/dbt-platform/aws-self-hosted-privatelink/scope-of-guide.png?v=2 "The scope of this guide")](#)The scope of this guide Out of scope This guide does not cover the configuration or troubleshooting of your self-hosted service, load balancer, or target group health, due to the virtually limitless ways these environments can be configured. While dbt Support may assist with such issues on a best-effort basis, we recommend engaging [AWS Support](https://aws.amazon.com/support/) to expedite resolution. #### Audience This guide is intended for cloud network administrators or engineers responsible for configuring and maintaining secure network communications within your organization's AWS environment. #### Terminology This guide uses several important terms related to AWS PrivateLink. Understanding these definitions will help ensure successful implementation. For a more detailed explanation of these concepts, refer to the [AWS PrivateLink documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html). * **Consumer:** In this context, the Consumer is dbt, which creates a VPC Endpoint to connect to your Endpoint Service. * **Service provider:** Your organization, which owns and operates the service behind the Network Load Balancer and creates the Endpoint Service. * **Endpoint Service:** The AWS resource that exposes your service to consumers, allowing them to create VPC Endpoints to access it. This is tied to a Network Load Balancer. * **Service Name:** A globally unique identifier for your Endpoint Service (format: `com.amazonaws.vpce.region.vpce-svc-xxx`). You share this with dbt Support to establish the connection. * **Network Load Balancer (NLB):** The required load balancer type (internal) that sits in front of your service. Your application must run behind an NLB to use PrivateLink. * **Target Group:** Routes traffic from the NLB to your service instances (EC2, IP addresses, or ALB). #### Prerequisites Before you begin, make sure to review the following requirements: 1. **Supported Load Balancer Types** dbt has officially validated PrivateLink functionality with the following load balancer type: * Network Load Balancer (Internal) > While other configurations may be compatible with AWS PrivateLink, this guide assumes your service is configured behind an Internal Network Load Balancer. For more details, see the [AWS Network Load Balancer documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html). 2. **Service Health** * Confirm that your service or application is operational and healthy behind the designated load balancer before proceeding. 3. **dbt AWS IAM Role ARN** * Contact [dbt Support](mailto:support@getdbt.com) to obtain the dbt AWS IAM role ARN. You will need this in order to allow dbt to connect to your Endpoint Service. #### Additional NLB configuration The following settings are optional but recommended when configuring your Network Load Balancer for PrivateLink connectivity with dbt. ##### Cross-zone load balancing Enable [cross-zone load balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#cross-zone-load-balancing) on your NLB to avoid availability zone mismatches between your service and dbt's VPC endpoint. This ensures traffic is distributed evenly across all healthy targets, regardless of which availability zone the request originates from. ##### Security group configuration If your NLB has an associated [security group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-security-groups.html), you need to ensure PrivateLink traffic from dbt is allowed. By default, when a security group is associated with an NLB, inbound rules are enforced on all traffic — including PrivateLink traffic. You have two options: | Option | Description | | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Disable enforcement** (recommended) | Turn off security group enforcement for PrivateLink traffic. This is the simplest approach and doesn't require knowledge of dbt's internal CIDRs. In the AWS Console: NLB → Security → Edit → Clear **Enforce inbound rules on PrivateLink traffic**. | | **Add dbt CIDRs to inbound rules** | If your use case requires security group enforcement on PrivateLink traffic, [contact dbt Support](mailto:support@getdbt.com) to obtain the internal CIDR ranges to add to your NLB's security group inbound rules. | For more details, see [Update the security groups for your Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-security-groups.html). #### Instructions 1. Log in to the [AWS Console](https://console.aws.amazon.com). 2. Navigate to the AWS Account and Region where your self-hosted service is located. ##### Create a VPC endpoint service 3. In the AWS Console, navigate to **VPC** → **Endpoint Services** → **Create Endpoint Service** 4. In the Create endpoint service page: a. **Load balancer type:** Select **Network** b. **Available load balancers:** Select the NLB in front of your service c. **Acceptance required:** Enable this option (recommended) to manually approve connection requests d. Click **Create** ##### Grant dbt access to the endpoint service 5. After the Endpoint Service is created, select it and go to the **Allow principals** tab 6. Click **Allow principals** and add the dbt AWS IAM role ARN that you obtained from support: * Principal: `arn:aws:iam::<dbt-account-id>:role/MTPL_Admin` ##### Obtain the endpoint service name 7. On the Endpoint Service details page, copy the **Service name** value (format: `com.amazonaws.vpce.region.vpce-svc-xxx`) [![Copy the Endpoint Service name](/img/docs/dbt-platform/aws-self-hosted-privatelink/obtain-endpoint-svc-name.png?v=2 "Copy the Endpoint Service name")](#)Copy the Endpoint Service name ##### Providing dbt Support with connection details 8. Add the required information to the template below, and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New AWS Self-hosted PrivateLink Request - Type: Self-hosted PrivateLink - Platform/Service (for example, Postgres, Starburst, Spark, GitLab, etc.): - **dbt platform account URL:** - VPC Endpoint Service Name: - Custom DNS (if HTTPS/TLS): - DNS record: - Service Region (for example, us-east-1, eu-west-2): - dbt AWS environment (US, EMEA, AU, JP): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### Troubleshooting If the PrivateLink endpoint has been provisioned and configured in dbt but connectivity is still failing, check the following in your networking setup to ensure requests and responses can be successfully routed between dbt and your service. ##### Configuration checklist 1. **NLB security group** The Network Load Balancer (NLB) associated with the VPC Endpoint Service must either not have an associated security group, or the security group must have a rule that allows requests from dbt's private CIDR(s). See [Security group configuration](#security-group-configuration) for details. Testing tip To test if this is the issue, temporarily adding an allow rule of `10.0.0.0/8` should allow connectivity until the rule can be refined to the dbt-provided CIDR. 2. **NLB listener and target group** Check that there is a Listener connected to the NLB that matches the port that dbt is trying to connect to. This Listener must have a configured action to forward to a Target Group with targets that point to your service. At least one (but preferably all) of these targets must be **Healthy**. Unhealthy targets could suggest that the service is down or that the service is protected by a security group that doesn't allow requests from the NLB. 3. **Cross-zone load balancing** Check that cross-zone load balancing is enabled for your NLB (check the **Attributes** tab of the NLB in the AWS console). If this is disabled, and the zones that dbt is connected to are misaligned with the zones where the service is running, requests may not be able to be routed correctly. See [Cross-zone load balancing](#cross-zone-load-balancing) for details. 4. **Routing tables and ACLs** If all the above check out, it may be possible that requests are not routing correctly within the private network. This could be due to a misconfiguration in the VPC's routing tables or access control lists. Review these settings with your network administrator to ensure that requests can be routed from the VPC Endpoint Service to the service and that the response can be returned to the VPC Endpoint Service. Testing tip One way to test this is to create a VPC endpoint in another VPC in your network to verify that connectivity is working independent of dbt's connection. ##### Monitoring To help isolate connection issues over a PrivateLink connection from dbt, there are a few monitoring sources that can be used to verify request activity. Requests must first be sent to the endpoint to see anything in the monitoring. [Contact dbt Support](mailto:support@getdbt.com) to understand when connection testing occurred or request new connection attempts. Use these times to correlate with activity in the following monitoring sources. ###### VPC Endpoint Service monitoring In the AWS Console, navigate to **VPC** → **Endpoint Services**. Select the Endpoint Service being tested and click the **Monitoring** tab. Update the time selection to include when test connection attempts were sent. If there is activity in the *New connections* and *Bytes processed* graphs, then requests have been received by the Endpoint Service, suggesting that the dbt endpoint is routing properly. ###### NLB monitoring In the AWS Console, navigate to **EC2** → **Load Balancers**. Select the Network Load Balancer (NLB) being tested and click the **Monitoring** tab. Update the time selection to include when test connection attempts were sent. If there is activity in the *New flow count* and *Processed bytes* graphs, then requests have been received by the NLB from the Endpoint Service, suggesting the NLB Listener, Target Group, and security group are correctly configured. ###### VPC Flow Logs VPC Flow Logs can provide various helpful information for requests being routed through your VPCs, though they can sometimes be challenging to locate and interpret. Flow logs can be written to either S3 or CloudWatch Logs, so determine the availability of these logs for your VPC and query them accordingly. Flow logs record the Elastic Network Interface (ENI) ID, source and destination IP and port, and whether the request was accepted or rejected by the security group and/or network ACL. This can be useful in understanding if a request arrived at a certain network interface and whether that request was accepted, potentially illuminating overly restrictive rules. For more information on accessing and interpreting VPC Flow Logs, see the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html). --- ##### Configuring AWS PrivateLink for single-tenant ingress Beta ### Configuring AWS PrivateLink for single-tenant ingress [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. Inbound AWS PrivateLink lets your services reach a single-tenant dbt platform instance privately over your access URL, without traffic traversing the public internet. You create an interface VPC endpoint in your AWS account that connects to the PrivateLink endpoint service dbt publishes in front of your instance. Single-tenant only, provisioned by dbt Labs Ingress PrivateLink is available only for single-tenant deployments; multi-tenant ingress isn't currently supported (refer to the [AWS private connectivity matrix](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-overview.md)). dbt Labs provisions it for your instance. To scope it and receive your endpoint service name, contact your **dbt Labs account team or Solutions Architect**. If you don't have a single-tenant deployment yet, [contact our sales team](mailto:sales@getdbt.com). #### Roles This guide uses two roles, matching the AWS PrivateLink model: * **Provider**: (dbt) Hosts the VPC endpoint service in front of your single-tenant instance. * **Customer**: (You) Creates the interface VPC endpoint in your own AWS account. #### Access URL naming convention Single-tenant ingress uses the domain convention `<customer_name>.private.dbt.com` for your access URL. When you enable private DNS on the endpoint (see [Step 3](#step-3-select-your-vpc-and-enable-private-dns)), AWS provisions a private hosted zone containing a wildcard record for this domain. The wildcard covers the product subdomains served by your instance, such as the semantic layer and metadata (Discovery) endpoints, so a single endpoint serves all of them. The provider must verify your custom domain first Private DNS on the endpoint only works once dbt (the provider) has verified the custom domain on the endpoint service. Confirm with your dbt Labs account team or Solutions Architect that your instance is ready for ingress PrivateLink before you begin. #### Prerequisites * A single-tenant dbt platform instance with ingress PrivateLink enabled by dbt. * The **VPC endpoint service name** for your instance, provided by dbt (for example, `com.amazonaws.vpce.us-east-1.vpce-svc-xxxxxxxxxxxxxxxxx`). * Your **access URL** for the instance (`<customer_name>.private.dbt.com`). * An AWS VPC in the same Region as the endpoint service, with **DNS hostnames** and **DNS resolution** enabled (both are required for the private DNS name feature). * Permissions in your AWS account to create interface VPC endpoints and manage the associated security group. #### Create the interface VPC endpoint ##### Step 1: Start creating the endpoint 1. In the AWS console, go to **VPC → Endpoints** and select **Create endpoint**. 2. (Optional) Add a **Name tag** to identify the endpoint. 3. Under **Type**, select **Endpoint services that use NLBs and GWLBs**. [![Create endpoint page with 'Endpoint services that use NLBs and GWLBs' selected as the type](/img/docs/dbt-platform/aws-ingress-privatelink/create-endpoint-select-type.png?v=2 "Create endpoint page with 'Endpoint services that use NLBs and GWLBs' selected as the type")](#)Create endpoint page with 'Endpoint services that use NLBs and GWLBs' selected as the type ##### Step 2: Enter and verify the service name 1. In **Service name**, paste the endpoint service name that dbt provided. 2. Leave **Enable Cross Region endpoint** unchecked — the endpoint is created in the same Region as the service. 3. Select **Verify service** and confirm you see **Service name verified**. [![Service name entered and verified, with Cross Region endpoint left disabled](/img/docs/dbt-platform/aws-ingress-privatelink/verify-service-name.png?v=2 "Service name entered and verified, with Cross Region endpoint left disabled")](#)Service name entered and verified, with Cross Region endpoint left disabled ##### Step 3: Select your VPC and enable private DNS 1. Under **Network settings**, select the **VPC** where your workloads run. 2. Expand **Additional settings** and select **Enable private DNS name**. [![Enable private DNS name selected under Additional settings](/img/docs/dbt-platform/aws-ingress-privatelink/enable-private-dns-name.png?v=2 "Enable private DNS name selected under Additional settings")](#)Enable private DNS name selected under Additional settings Always enable private DNS name Selecting **Enable private DNS name** automatically provisions the private hosted zone and record set for your `<customer_name>.private.dbt.com` access URL. This is the standard, recommended configuration. Configuring hostnames manually instead can lead to TLS errors and other issues, so keep this box checked. ##### Step 4: Choose subnets and a security group 1. Select the **subnets** for the endpoint. A single subnet is enough for testing; use subnets across multiple Availability Zones for production. 2. Choose a **security group** that allows inbound traffic on **port 443** from your workloads. 3. Select **Create endpoint**. ##### Step 5: Wait for the endpoint to become available The endpoint is created in a **Pending** state while dbt accepts the connection, then moves to **Available**. Once available, the **Private DNS names enabled** field shows **Yes** and the **Private DNS names** list includes your `*.<customer_name>.private.dbt.com` record. [![Endpoint details page showing status Available, private DNS names enabled, and the private.dbt.com record](/img/docs/dbt-platform/aws-ingress-privatelink/endpoint-available.png?v=2 "Endpoint details page showing status Available, private DNS names enabled, and the private.dbt.com record")](#)Endpoint details page showing status Available, private DNS names enabled, and the private.dbt.com record ##### Step 6: (Optional) Configure DNS for on-premises access Workloads inside the associated VPC resolve the access URL automatically. Clients connecting from **on-premises** networks over **AWS VPN** or **AWS Direct Connect** need additional DNS configuration so they resolve the access URL to the endpoint's private IPs. To forward DNS queries from your on-premises network into the VPC, refer to the AWS documentation on [Route 53 Resolver inbound endpoints](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-forwarding-inbound-queries.html). If all your users access the instance from inside the VPC, you can skip this step. #### Validate connectivity From an instance inside the selected VPC, confirm that: 1. Your access URL resolves to a *private* IP address from the endpoint (not a public address). 2. The TLS handshake succeeds and the returned certificate matches your access URL. Clients must send the correct TLS SNI The endpoint selects its certificate by TLS **Server Name Indication (SNI)**. Clients must send your access URL as the SNI. Some PrivateLink clients and libraries do not set SNI by default — without it, the endpoint returns a default certificate that will not match your access URL and the handshake fails. --- ##### Configuring AWS PrivateLink for Teradata VantageCloud dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a Teradata VantageCloud AWS PrivateLink endpoint in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Teradata VantageCloud) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Configure AWS PrivateLink 1. Contact your Teradata account team or Teradata Support to obtain: * The VPC endpoint service name for your Teradata PrivateLink Service * The AWS region of your Teradata environment The service name is generated by AWS and looks similar to: ```text com.amazonaws.vpce.us-east-1.vpce-svc-0abc123def456ghi7 ``` dbt AWS account If granting access to your Teradata endpoint service requires the dbt AWS account ID, use **AWS account ID**: `346425330055` — *Note: This account ID only applies to AWS dbt multi-tenant environments. For AWS Virtual Private/Single-Tenant account IDs, contact [dbt Support](mailto:support@getdbt.com).* 2. Add the required information to the following template and submit your AWS PrivateLink request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New AWS Multi-Tenant PrivateLink Request - Type: Teradata - dbt platform account URL: - Display name (appears in the dbt connection dropdown): - VPC Endpoint Service Name: - Port: 1025 (change if not default) - Teradata AWS Region (for example, us-east-1, eu-west-2): - dbt AWS multi-tenant environment (US, EMEA, AU, JP): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. 3. Once dbt Support confirms the endpoint has been provisioned and the connection has been tested, they will reply with the hostname to use when configuring your project. #### Create connection in dbt Once you've completed the setup with dbt Support, you can configure a private endpoint in dbt: 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Teradata**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. --- ##### Configuring Azure Private Link for a self-hosted service dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. Azure Private Link enables secure, private connectivity between dbt and your self-hosted services. These services may include version control systems (VCS), data warehouses, or any other applications you manage. With Private Link, you do not need to expose your service to the public internet. All communication occurs over a private network, significantly enhancing security. For more details, refer to the Azure [Private Link documentation](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview). #### What this guide covers The focus of this guide is not on any particular service or backend architecture, but on the [Private Link Service](#terminology) that interconnects dbt with your self-hosted service. This process should be standard across most use cases. [![The scope of this guide](/img/docs/dbt-platform/az-self-hosted-privatelink/scope-of-guide.png?v=2 "The scope of this guide")](#)The scope of this guide Out of scope This guide does not cover the configuration or troubleshooting of your self-hosted service, load balancer, or backend pool health, due to the virtually limitless ways these environments can be configured. While dbt Support may assist with such issues on a best-effort basis, we recommend engaging [Azure Support](https://azure.microsoft.com/en-us/support/) to expedite resolution. #### Audience This guide is intended for cloud network administrators or engineers responsible for configuring and maintaining secure network communications within your organization's Microsoft Azure environment. #### Terminology This guide uses several important terms related to Azure Private Link. Understanding these definitions will help ensure successful implementation. For a more detailed explanation of these concepts, refer to the [Azure Private Link Service documentation](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview). * **Consumer:** In this context, the Consumer is dbt, which creates a private endpoint to connect to your Private Link Service. * **Service provider:** Your organization, which owns and operates the service behind the Standard Load Balancer and creates the Private Link Service. * **Private Link Service:** The Azure resource that exposes your service to consumers, allowing them to create private endpoints to access it. This is tied to a Standard Load Balancer frontend IP configuration. * **Alias:** A globally unique name generated by Azure for your Private Link Service. You share this alias with dbt Support to establish the connection to your service as a consumer. * **Standard Load Balancer:** The required load balancer type that sits in front of your service. Your application must run behind a Standard Load Balancer to use Private Link Service. * **NAT subnet:** A dedicated subnet in your VNet used for Source Network Address Translation (SNAT) IP addresses for the Private Link Service. Consumer traffic appears to originate from this pool of private IP addresses. #### Prerequisites Before you begin, make sure to review the following requirements: 1. **Supported Load Balancer Types** dbt has officially validated Private Link functionality with the following load balancer type: * Standard Load Balancer (Internal) > While other configurations may be compatible with Azure Private Link Services, this guide assumes your service is configured behind a Standard Internal Load Balancer. For more details, see the [Azure Load Balancer documentation](https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-overview). 2. **Service Health** * Confirm that your service or application is operational and healthy behind the designated load balancer before proceeding. 3. **dbt Azure Subscription ID** * Contact [dbt Support](mailto:support@getdbt.com) to obtain the dbt Azure subscription ID. You will need this in order to allow dbt to connect to your Private Link Service. #### Instructions 1. Log in to the [Azure Portal](https://portal.azure.com). 2. Navigate to the Azure Subscription and Resource Group where your self-hosted service is located. ##### Create a NAT subnet for the Private Link service 3. Under the **Resources** section, search for **Virtual network** and go into the VNet where your self-hosted service is running. 4. Expand the **Settings** in the left side panel, and go into **Subnets**. Click the **+ Subnet** button to create a new subnet. 5. In the subnet creation panel: a. **Subnet purpose:** Leave as **Default** b. **Name:** Provide a descriptive name, such as **private-link-nat-subnet** c. **IPv4 address range:** Choose the appropriate CIDR block from your VNet that you want to create a NAT subnet from. In this example, the CIDR is 10.30.0.0/16, as seen in the screenshot below. d. **Starting address:** Your desired starting address of the new subnet e. **Size:** The smallest available size is recommended (for example, /28). f. Check the **Enable private subnet (no default outbound access)** checkbox. g. **NAT gateway:** Leave as **None** h. Leave **Network security group** and **Route table** fields as **None** unless your environment requires specific values here. i. Leave all remaining fields as their default values. j. Click **Add** to create the subnet. [![Screenshot of step 3: Search for VNet of self-hosted service](/img/docs/dbt-platform/az-self-hosted-privatelink/vnet-search.png?v=2 "Screenshot of step 3: Search for VNet of self-hosted service")](#)Screenshot of step 3: Search for VNet of self-hosted service [![Screenshot of steps 4-5: NAT Subnet creation for Private Link Service](/img/docs/dbt-platform/az-self-hosted-privatelink/nat-subnet-creation.png?v=2 "Screenshot of steps 4-5: NAT Subnet creation for Private Link Service")](#)Screenshot of steps 4-5: NAT Subnet creation for Private Link Service ##### Create a Private Link service 6. After the subnet creation has completed, in the search field at the top-middle of the portal, search for **Private link services**, and click on its page. 7. Click the **+ Create** button. 8. In the Create private link service page: **Under Basics** a. Select your **Subscription** and **Resource group** b. **Name:** Give a descriptive name, such as **pls-to-my-vcs** c. **Region:** Select the region where your self-hosted service is located **Under Outbound settings** d. **Load balancer:** In the dropdown, choose the Standard Internal Load Balancer that is in front of your self-hosted service e. **Load balancer frontend IP address:** Choose the frontend IP configuration for your load balancer f. **Source NAT subnet:** Select the NAT subnet you created in step 5 above g. **Source NAT Virtual network:** This will auto-populate based on your subnet selection h. **Enable TCP proxy V2:** Leave this disabled **Under Access security** i. Select **Restricted by subscription** j. Click **Add subscriptions** and add dbt's Azure subscription ID that you acquired from support k. Set **Request Auto-approve** selection to **Yes** for dbt's subscription l. Click **Next: Review + create**, then **Create** [![Screenshot of step 8: Creation of Azure Private Link Service](/img/docs/dbt-platform/az-self-hosted-privatelink/privatelink-service-creation.png?v=2 "Screenshot of step 8: Creation of Azure Private Link Service")](#)Screenshot of step 8: Creation of Azure Private Link Service 9. After the Private Link Service has been created, click on it to open its details page. 10. Copy the **Alias** value (this is the identifier you'll share with dbt Support). [![Screenshot of step 10: Copy the Private Link Service Alias](/img/docs/dbt-platform/az-self-hosted-privatelink/alias-info.png?v=2 "Screenshot of step 10: Copy the Private Link Service Alias")](#)Screenshot of step 10: Copy the Private Link Service Alias ##### Providing dbt Support with connection details 11. Add the required information to the template below, and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Azure Self-hosted Private Link Request - Type: Self-hosted Private Link - Platform/Service (for example, Postgres, Starburst, Spark, GitLab, etc.): - dbt platform account URL: - Private Link Service Alias: - Custom DNS (if HTTPS/TLS): - DNS record: - Service Region (for example, East US, West Europe): - dbt Azure environment (EMEA): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### Troubleshooting If the Private Link endpoint has been provisioned and configured in dbt but connectivity is still failing, check the following in your networking setup to ensure requests and responses can be successfully routed between dbt and your service. ##### Configuration checklist 1. **Private Link Service status** In the Azure Portal, navigate to your Private Link Service and verify the **Provisioning state** is **Succeeded**. Check the **Private endpoint connections** tab to confirm dbt's connection shows as **Approved**. 2. **Load balancer backend health** Navigate to your Standard Load Balancer and check the **Backend pool** health. At least one backend instance must be **Healthy**. Unhealthy backends could indicate the service is down or that a Network Security Group (NSG) is blocking traffic from the load balancer. 3. **NAT subnet configuration** Verify the NAT subnet has sufficient IP addresses available. Azure Private Link Service uses these IPs for SNAT. If the subnet is exhausted, new connections may fail. 4. **Network Security Groups** If you have NSGs applied to the NAT subnet or backend subnet, ensure they allow traffic appropriately: * NAT subnet: Recommended to leave NSG as **None** (as noted in the setup instructions) * Backend subnet: Must allow traffic from the load balancer's frontend IP ##### Monitoring To help isolate connection issues, use Azure's monitoring tools: ###### Private Link Service metrics In the Azure Portal, navigate to your Private Link Service and click **Metrics**. Monitor: * **Bytes In/Out** — Confirms traffic is flowing through the service * **NAT Port Usage** — High usage may indicate the NAT subnet needs more IPs ###### Load Balancer metrics Navigate to your Standard Load Balancer and click **Metrics**. Monitor: * **Health Probe Status** — Shows backend health over time * **Byte Count** — Confirms traffic is reaching the load balancer * **SNAT Connection Count** — Tracks outbound connections For more information, see [Azure Private Link monitoring](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#monitoring) and [Load Balancer monitoring](https://learn.microsoft.com/en-us/azure/load-balancer/monitor-load-balancer). --- ##### Configuring Azure Private Link for Teradata VantageCloud dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a Teradata VantageCloud Azure Private Link endpoint in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Teradata VantageCloud) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Configure Azure Private Link 1. Contact your Teradata account team or Teradata Support to obtain: * The Azure resource alias for your Teradata Private Link Service * The Azure region of your Teradata environment The alias is generated by Azure and looks similar to: ```text teradata-pls.abc12345-6789-0def-1234-56789abcdef0.eastus.azure.privatelinkservice ``` dbt Azure subscription ID If granting access to your Teradata endpoint service requires the dbt Azure subscription ID, [contact dbt Support](mailto:support@getdbt.com) to request it. 2. Add the required information to the following template and submit your Azure Private Link request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Azure Multi-Tenant Private Link Request - Type: Teradata - dbt platform account URL: - Display name (appears in the dbt connection dropdown): - Teradata Private Link Service alias: - Port: 1025 (change if not default) - Teradata Azure region: - dbt Azure multi-tenant environment (US or EMEA): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. 3. Once dbt Support confirms the endpoint has been provisioned and the connection has been tested, they will reply with the hostname to use when configuring your project. #### Create connection in dbt Once you've completed the setup with dbt Support, you can configure a private endpoint in dbt: 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Teradata**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. --- ##### Configuring BigQuery Private Service Connect dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a GCP BigQuery [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) (PSC) endpoint in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like BigQuery) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Enabling dbt for GCP Private Service Connect To enable dbt to privately connect to your BigQuery project via PSC, the regional PSC endpoint needs be enabled for your dbt account. Using the following template, submit a request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Multi-Tenant GCP PSC Request - Type: BigQuery - dbt platform account URL: - BigQuery project region: - dbt GCP multi-tenant environment: ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### (Optional) Generate BigQuery credentials You may already have credentials set up for your datasets. If not, you can follow the steps in our [BigQuery quickstart guide](https://docs.getdbt.com/guides/bigquery.md?step=4) to generate credentials. #### Create connection in dbt Once dbt Support completes the configuration, you can start creating new connections using PSC: 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **BigQuery**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. --- ##### Configuring Databricks and Azure Private Link dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a Databricks Azure Private Link endpoint in the dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Databricks) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. VNet injection required Azure supports private endpoints for Databricks workspaces only when the workspace uses VNet injection (a customer-managed VNet). Workspaces on the default Azure-managed VNet reject private endpoint connections with a `NonVNetInjectedWorkspaceNotSupported` error. Confirm your workspace uses VNet injection before you submit the request. If it doesn't, create a new workspace with VNet injection enabled first. For more information, refer to [Deploy Azure Databricks in your own virtual network](https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/vnet-inject). #### Configure Azure Private Link 1. Navigate to your Azure Databricks workspace. The path format is: `/subscriptions/<subscription_uuid>/resourceGroups/<resource_group_name>/providers/Microsoft.Databricks/workspaces/<workspace_name>`. 2. From the workspace overview, click **JSON view**. 3. Copy the value in the `resource_id` field. 4. Add the required information to the following template and submit your Azure Private Link request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Azure Multi-Tenant Private Link Request - Type: Databricks - dbt platform account URL: - Databricks instance name: - Azure Databricks Workspace URL (for example, adb-################.##.azuredatabricks.net) - Databricks Azure resource ID: /subscriptions/SUB_ID/resourceGroups/RG/providers/Microsoft.Databricks/workspaces/WORKSPACE_NAME - Use the full ARM resource ID, replacing SUB_ID, RG, and WORKSPACE_NAME with your values - dbt Azure multi-tenant environment (US or EMEA): - Azure Databricks workspace region (for example, WestEurope, NorthEurope) ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. 5. Once our Support team confirms the resources are available in the Azure portal, navigate to the Azure Databricks Workspace and browse to **Networking** > **Private Endpoint Connections**. Then, highlight the `dbt` named option and select **Approve**. DNS impact when approving the first private endpoint Approving the first private endpoint connection for an Azure Databricks workspace modifies the workspace's DNS resolution. Azure inserts a `privatelink.azuredatabricks.net` DNS `A` record that redirects the workspace URL to the private IP address instead of the public endpoint. Any clients — including other dbt platform projects, tools, or users — that connect to this workspace over the public internet **may lose connectivity** after this change. Coordinate this change with all teams that use the workspace before approving. For more information, see [Azure services DNS zone configuration](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns#azure-services-dns-zone-configuration). #### Create connection in dbt Once you've completed the setup in the Databricks environment, you can configure a private endpoint in dbt: 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Databricks**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. --- ##### Configuring Databricks PrivateLink dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a Databricks AWS PrivateLink endpoint in the dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Databricks) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Configure AWS PrivateLink You can set up a Databricks AWS PrivateLink endpoint in two ways: * [Self-serve private endpoints](#self-serve-private-endpoints): Create and manage Databricks PrivateLink endpoints directly in the dbt platform user interface. Currently in beta. * [Support-led setup](#support-led-setup): Contact dbt Support to configure your Databricks PrivateLink endpoint. ##### Self-serve private endpoints [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") *Self-serve private endpoints are currently in beta for Databricks on AWS, and available to all eligible customers. This feature isn't available for Azure or GCP. If you don't see **Private endpoints** in your account settings, use the [Support-led setup](#support-led-setup) instead.* With self-serve, you request a Databricks PrivateLink endpoint in dbt platform without opening a support ticket. If a request fails, you can delete the endpoint in dbt platform and retry on your own. ###### Prerequisites * [Account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md?version=2.0#account-admin) or [Project creator](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md?version=2.0#project-creator) permission sets in dbt platform. Users with an IT license can also create private endpoints. * Your Databricks workspace name and the AWS region where the workspace is hosted. ###### Request a new private endpoint 1. In dbt platform, go to **Account settings → Private endpoints**. 2. In the **Private endpoints** table, review your existing endpoints. The table shows all private endpoints in your account (including non-Databricks ones) with the following details: * **Name** * **Connection type** (for example, Databricks) * **URL** * **Connectivity status** (for example, **Success** or **Unknown**) * **Connections** — the number of dbt platform connections using the endpoint You can search by **Name** or **URL**. [![Private endpoints table showing existing endpoints, connectivity status, and the Request new button](/img/docs/dbt-platform/private-endpoint-page.png?v=2 "Private endpoints table showing existing endpoints, connectivity status, and the Request new button")](#)Private endpoints table showing existing endpoints, connectivity status, and the Request new button 3. To request a new endpoint, click **Request new**. 4. Under **Provider type**, select **Databricks**. 5. In **Step 1: Enter your workspace name**, paste the full workspace URL (for example, `my-workspace.cloud.databricks.com`) or just the workspace name (for example, `my-workspace`). 6. In **Step 2: Select your AWS region**, choose the AWS region where your Databricks workspace is hosted. Only regions that support Databricks private connectivity are listed. 7. Click **Submit request**. [![Databricks endpoint request form showing Provider type, workspace name, and AWS region fields](/img/docs/dbt-platform/databricks-private-endpoint-request.png?v=2 "Databricks endpoint request form showing Provider type, workspace name, and AWS region fields")](#)Databricks endpoint request form showing Provider type, workspace name, and AWS region fields Workspace name and region can't be changed The workspace name and AWS region are fixed once you create an endpoint. This means you can't edit them afterward. If either value is entered incorrectly, provisioning will fail. To fix it, delete the failed request and create a new one with the correct details. No support ticket is needed. 8. After submission, a confirmation popup appears. From the popup, you can request another endpoint or return to **Private endpoints** to track request status. 9. Proceed to the **Connections** page and follow the steps in the [Create connection in dbt](#create-connection-in-dbt) section to configure PrivateLink. Once you configure PrivateLink on the **Connections** page, the new endpoint appears under **Private endpoints → Associated connections**. DNS propagation If the connection test fails immediately after setup, this is expected — it doesn't mean something is wrong. DNS changes can take a few minutes to propagate. Wait a few minutes, then test again before contacting support. ###### Reuse an existing endpoint Databricks exposes one predefined endpoint service per AWS region. If you submit a request for a region that already has an interface endpoint, dbt platform displays an **Endpoint already exists** popup with two options: * **Re-use an existing interface endpoint** (recommended) — Creates only a new private endpoint, linked to the interface endpoint you select. In most cases this is the better choice, since Databricks shares one endpoint service per region and reusing it avoids duplicating infrastructure. * **Create a new interface endpoint** — Creates a new interface endpoint alongside the new private endpoint. Use this only if you need a separate interface endpoint. Select your preferred option and click **Confirm & Submit**. [![Endpoint already exists popup with options to create a new interface endpoint or re-use an existing one](/img/docs/dbt-platform/endpoint-exists.png?v=2 "Endpoint already exists popup with options to create a new interface endpoint or re-use an existing one")](#)Endpoint already exists popup with options to create a new interface endpoint or re-use an existing one ###### Edit or delete a private endpoint  Edit or delete a private endpoint If you don't see **Edit** or **Delete endpoint**, contact your account manager to enable private endpoint updates for your account. **Edit an endpoint** You can update the endpoint **Name** and **Port**. The workspace name and AWS region can't be changed after creation. 1. In the **Private endpoints** table, click the endpoint you want to update. 2. On the endpoint details page, click **Edit**. 3. Update **Name** and/or **Port** as needed. 4. Click **Save changes**. 5. In the **Save changes?** modal, click **Save changes** to apply your updates. **Delete an endpoint** An endpoint with associated connections can't be deleted. Remove those connections first. 1. In the **Private endpoints** table, click the endpoint you want to delete. 2. On the endpoint details page, click **Edit**. 3. Scroll to the bottom of the page and click **Delete endpoint**. 4. In the **Delete endpoint** modal, type `DELETE` to confirm, then click **Delete endpoint**. ###### Troubleshooting and errors If an endpoint request fails, dbt platform displays error details that are safe to share externally. Because the workspace name and region are locked after creation, a failed request usually means one of those values was incorrect — delete the request and submit a new one with the correct details. If you see a failure state without clear next steps, collect the request details (endpoint name, creation time, and status) and contact [dbt Support](mailto:support@getdbt.com). ##### Support-led setup If **Private endpoints** isn't available in your account settings, configure Databricks PrivateLink by following these steps and submitting a request to dbt Support. 1. Locate your [Databricks instance name](https://docs.databricks.com/en/workspace/workspace-details.html#workspace-instance-names-urls-and-ids). * Example: `cust-success.cloud.databricks.com` 2. Add the required information to the following template and submit your AWS PrivateLink request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New AWS Multi-Tenant PrivateLink Request - Type: Databricks - dbt platform account URL: - Databricks instance name: - Databricks cluster AWS Region (for example, us-east-1, eu-west-2): - dbt AWS multi-tenant environment (US, EMEA, AU, JP): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. 3. Once dbt Support notifies you that setup is complete, [register the VPC endpoint in Databricks](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html#step-3-register-privatelink-objects-and-attach-them-to-a-workspace) and attach it to the workspace: * [Register your VPC endpoint](https://docs.databricks.com/en/security/network/classic/vpc-endpoints.html) — Register the VPC endpoint using the VPC endpoint ID provided by dbt Support. * [Create a Private Access Settings object](https://docs.databricks.com/en/security/network/classic/private-access-settings.html) — Create a Private Access Settings (PAS) object with your desired public access settings, and setting Private Access Level to **Endpoint**. Choose the registered endpoint created in the previous step. * [Create or update your workspace](https://docs.databricks.com/en/security/network/classic/privatelink.html#step-3d-create-or-update-the-workspace-front-end-back-end-or-both) — Create a workspace, or update an existing workspace. Under **Advanced configurations → Private Link** choose the private access settings object created in the previous step. warning If using an existing Databricks workspace, all workloads running in the workspace need to be stopped to enable Private Link. Workloads also can't be started for another 20 minutes after making changes. From the [Databricks documentation](https://docs.databricks.com/en/security/network/classic/privatelink.html#step-3d-create-or-update-the-workspace-front-end-back-end-or-both): "After creating (or updating) a workspace, wait until it’s available for using or creating clusters. The workspace status stays at status RUNNING and the VPC change happens immediately. However, you cannot use or create clusters for another 20 minutes. If you create or use clusters before this time interval elapses, clusters do not launch successfully, fail, or could cause other unexpected behavior." #### Create connection in dbt Once you've completed the setup, you can configure a private endpoint in dbt: 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Databricks**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. --- ##### Configuring GCP Private Service Connect for a self-hosted service dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. GCP Private Service Connect (PSC) enables secure, private connectivity between dbt and your self-hosted services. These services may include version control systems (VCS), data warehouses, or any other applications you manage. With PSC, you do not need to expose your service to the public internet. All communication occurs over a private network, significantly enhancing security. For more details, refer to the GCP [Private Service Connect documentation](https://cloud.google.com/private-service-connect). #### What this guide covers The focus of this guide is not on any particular service or [Backend](#terminology) architecture, but on the [Service Attachment](#terminology) that interconnects dbt with your self-hosted service. This attachment process should be standard across most use cases. [![The scope of this guide](/img/docs/dbt-platform/gcp-self-hosted-psc/scope-of-guide.png?v=2 "The scope of this guide")](#)The scope of this guide Out of scope This guide does not cover the configuration or troubleshooting of your self-hosted service, load balancer, or backend health, due to the virtually limitless ways these environments can be configured. While dbt Support may assist with such issues on a best-effort basis, we recommend engaging [Google Cloud Support](https://cloud.google.com/support) to expedite resolution. #### Audience This guide is intended for cloud network administrators or engineers responsible for configuring and maintaining secure network communications within your organization's Google Cloud Platform (GCP) environment. #### Terminology This guide uses several important terms related to Private Service Connect. Understanding these definitions helps ensure successful implementation. For a more detailed explanation of these concepts, refer to the [GCP Private Service Connect documentation](https://cloud.google.com/vpc/docs/private-service-connect#managed-services). * **Consumer:** In this context, the Consumer is dbt, which establishes the PSC connection as the client. * **Published Service:** The service you are exposing via PSC to the dbt platform, such as your version control system (VCS), data warehouse, or another application. * **Service Attachment:** Refers to the resource that is shared with consumer(s) of your Published Service, so that they can establish endpoints to it. * **Backend:** Can also be referred to as Network Endpoint Groups (NEGs). This is the particular architecture that your service is running on. For example, this may be VMs, GKE Instance Groups, or even on-prem IPs. #### Prerequisites Before you begin, make sure to review the following requirements: 1. **Supported Load Balancer Types** dbt has officially validated Private Service Connect (PSC) functionality with the following load balancer types: * Regional Internal Proxy Load Balancer * Cross-Regional Internal Proxy Load Balancer > While other load balancer types can be compatible with PSC Service Attachments, this guide assumes your service is configured behind one of the officially supported Proxy Load Balancers. For more details, see the [Proxy Load Balancers documentation](https://docs.cloud.google.com/load-balancing/docs/tcp/internal-proxy). 2. **Service Health** * Confirm that your service or application is operational and healthy behind the designated load balancer before proceeding. 3. **dbt GCP Project ID** * Contact [dbt Support](mailto:support@getdbt.com) to obtain the dbt GCP project ID. You will need this in order to share your service attachment with the dbt platform. #### Instructions 1. Log in to the Google Cloud Platform [console](https://console.cloud.google.com) 2. Navigate to the GCP Organization and Project that your self-hosted service is in. ##### Create a dedicated service attachment subnet 3. In the search field at the top-middle of the console, search for **VPC networks** and navigate to its product page. 4. On the product page, click the VPC network link where your self-hosted service is located. 5. Select the **Subnets** tab on the next page, and click the **Add subnet** button. 6. In the subnet creation panel: a. **Name:** Provide a descriptive name, such as **service-attachment-subnet** b. **Description:** This subnet is dedicated to service attachment(s) c. **Region:** Pick the region of your self-hosted service d. **Purpose:** Choose **Private Service Connect** e. Click **Add** to create the subnet [![Screenshot of step 6: Subnet creation for PSC Service Attachment](/img/docs/dbt-platform/gcp-self-hosted-psc/service-attach-subnet-creation.png?v=2 "Screenshot of step 6: Subnet creation for PSC Service Attachment")](#)Screenshot of step 6: Subnet creation for PSC Service Attachment ##### Create a service attachment 7. After the subnet creation for the service attachment has completed, in the search field at the top-middle of the console, search for **Private Service Connect**, and click on its product page. 8. On the product page, select the **Published services** tab, and click the **Publish service** button. 9. In the Publish service page: **Under Target details** a. Choose **Load Balancer** b. The load balancer types that dbt has validated are the **Regional Internal Proxy Load Balancer** and the **Cross-Regional Internal Proxy Load Balancer**. However, the others may work as well, although not officially supported. c. In the **Load balancer** dropdown, choose the load balancer that is in front of your self-hosted service. d. Choose the relevant **Forwarding rule** from the dropdown for your load balancer. **Under Service details** e. Give a descriptive **Service Name**, such as **service-to-my-vcs** f. In the **Subnets** dropdown, choose the subnet that you created in step 6 above. **Under Connection Preference** g. Leave the selection on **Accept connections from selected projects** h. Click the **Add accepted project** button and add dbt's GCP project ID that you acquired from support. Note: This project ID may differ for each configuration. * Set connection limit to 1 i. Click **Add service** [![Screenshot of step 9: Creation of PSC Service Attachment](/img/docs/dbt-platform/gcp-self-hosted-psc/service-attach-creation.png?v=2 "Screenshot of step 9: Creation of PSC Service Attachment")](#)Screenshot of step 9: Creation of PSC Service Attachment 10. After the Published Service attachment has been created, click on it to open its details page. 11. Copy the **Service attachment** URI (*not* the Service attachment ID). [![Screenshot of step 11: Copy the Service attachment URI](/img/docs/dbt-platform/gcp-self-hosted-psc/service-attach-details.png?v=2 "Screenshot of step 11: Copy the Service attachment URI")](#)Screenshot of step 11: Copy the Service attachment URI ##### Providing dbt Support with connection details 12. Add the required information to the template below, and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New GCP Self-hosted Private Service Connect Request - Type: Self-hosted PSC - dbt platform account URL: - Platform/Service (for example, Postgres, Starburst, Spark, GitLab, etc.): - Service Attachment URI: - Custom DNS (if HTTPS/TLS): - DNS record: - Service Region (for example, us-east1, us-central1): - dbt GCP environment (US): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### Troubleshooting If the Private Service Connect endpoint has been provisioned and configured in dbt but connectivity is still failing, check the following in your networking setup to ensure requests and responses can be successfully routed between dbt and your service. ##### Configuration checklist 1. **Service Attachment status** In the Google Cloud Console, navigate to **Network services** → **Private Service Connect** → **Published services**. Select your Service Attachment and verify: * Status is **Active** * dbt's project appears in the **Connected projects** list with status **Accepted** 2. **Load balancer backend health** Navigate to **Network services** → **Load balancing** and select your load balancer. Check the **Backend services** tab to confirm at least one backend is **Healthy**. Unhealthy backends could indicate the service is down or that firewall rules are blocking health check probes. 3. **NAT subnet configuration** Verify the Private Service Connect subnet has sufficient IP addresses available. GCP uses these IPs for SNAT when routing consumer traffic to your backends. 4. **Firewall rules** Ensure your VPC firewall rules allow: * Health check traffic from Google's health check ranges (`35.191.0.0/16` and `130.211.0.0/22`) * Traffic from the proxy-only subnet to your backends (for Proxy Load Balancers) For more details, see [Firewall rules for health checks](https://cloud.google.com/load-balancing/docs/health-check-concepts#ip-ranges). ##### Monitoring To help isolate connection issues, use Google Cloud's monitoring tools: ###### Service Attachment metrics In the Google Cloud Console, navigate to **Monitoring** → **Metrics Explorer**. Search for Private Service Connect metrics: * `compute.googleapis.com/nat/nat_connections` — Tracks active NAT connections * `compute.googleapis.com/nat/sent_bytes_count` — Confirms traffic is flowing ###### Load Balancer logs Enable logging on your load balancer's backend service to capture request details. Navigate to your backend service, click **Edit**, and enable **Logging** with a sample rate of 1.0 for troubleshooting. For more information, see [Private Service Connect monitoring](https://cloud.google.com/vpc/docs/monitor-private-service-connect-connections) and [Load Balancer logging](https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring). --- ##### Configuring Private Link for Azure Database for Postgres Flexible Server dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a Private Link endpoint for Azure Database for Postgres Flexible Server in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Azure Database) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Configure Azure Private Link From your Azure portal: 1. Navigate to your Azure Database for Postgres Flexible Server. 2. From the server overview, click **JSON view**. 3. Copy the value in the **Resource ID** field at the top of the pane.<br />The path format is: `/subscriptions/<subscription_uuid>/resourceGroups/<resource_group_name>/providers/Microsoft.DBforPostgreSQL/flexibleServers/<server_name>`. 4. Add the required information to the following template and submit your Azure Private Link request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Azure Multi-Tenant Private Link Request - Type: Azure Database for Postgres Flexible Server - dbt platform account URL: - Postgres Flexible Server name: - Azure Database for Postgres Flexible Server resource ID: - dbt Azure multi-tenant environment (US or EMEA): - Azure Postgres server region (for example, WestEurope, NorthEurope): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. 5. Once our Support team confirms the endpoint has been created, navigate to the Azure Database for Postgres Flexible Server in the Azure Portal and browse to **Settings** > **Networking**. In the **Private Endpoints** section, highlight the `dbt` named option and select **Approve**. Confirm with dbt Support that the connection has been approved so they can validate the connection and make it available for use in dbt. #### Create connection in dbt Once you've completed the setup in the Azure environment, you can configure a private endpoint in dbt: 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Postgres**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. --- ##### Configuring Private Link for Azure Synapse dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a Private Link endpoint for Azure Synapse in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Azure Synapse) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. #### Configure Azure Private Link From your Azure portal: 1. Navigate to your Azure Synapse workspace. 2. From the workspace overview, click **JSON view**. 3. Copy the value in the **Resource ID** field at the top of the pane.<br />The path format is: `/subscriptions/<subscription_uuid>/resourceGroups/<resource_group_name>/providers/Microsoft.Synapse/workspaces/<workspace_name>`. 4. Add the required information to the following template and submit your Azure Private Link request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Azure Multi-Tenant Private Link Request - Type: Azure Synapse - dbt platform account URL: - Server name: - Azure Synapse workspace resource ID: - dbt Azure multi-tenant environment (US or EMEA): - Azure Synapse workspace region (for example, WestEurope, NorthEurope): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. 5. Once our Support team confirms the endpoint has been created, navigate to the Azure Synapse workspace in the Azure Portal and browse to **Security** > **Private endpoint connections**. In the **Private endpoint connections** table, highlight the `dbt` named option and select **Approve**. Confirm with dbt Support that the connection has been approved so they can validate the connection and make it available for use in dbt. #### Create connection in dbt Once you've completed the step above, you can configure a private endpoint in dbt: 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Synapse**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. --- ##### Configuring Private Link for Microsoft Fabric Beta ### Configuring Private Link for Microsoft Fabric [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a Private Link endpoint for Microsoft Fabric in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Microsoft Fabric) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. Workspace-level Private Link Microsoft Fabric offers two private connectivity models. Only **workspace-level Private Link** supports cross-tenant connections and is compatible with dbt. Tenant-level Private Link is designed for intra-organization access and does not support connections from external services like dbt. #### Prerequisites Before configuring Private Link, make sure you have the following: * You have Fabric Administrator access * You have an Azure subscription with permission to register resource providers and deploy templates * You have a Fabric workspace already created * You have a service principal set up with the right Fabric permissions (Member or Contributor on the workspace, and Fabric APIs enabled in the Admin portal) #### Configure Fabric Private Link Complete the following steps in your Microsoft Fabric and Azure environments before submitting a request to dbt. ##### 1. Enable the workspace-level Private Link tenant setting A Fabric Administrator must enable workspace-level private link for your tenant before it can be configured on individual workspaces. 1. Open the [Microsoft Fabric Admin portal](https://app.fabric.microsoft.com/admin-portal). 2. Navigate to **Tenant settings**. 3. Search for **Configure workspace-level inbound network rules** and expand the setting. 4. Toggle the setting to **Enabled**. 5. Click **Apply**. ##### 2. Configure your workspace network settings 1. Open your Fabric workspace in the [Microsoft Fabric portal](https://app.fabric.microsoft.com). 2. Navigate to **Workspace settings** → **Inbound networking**. 3. Select **Allow connections from selected networks and workspace level private links**. 4. Click **Save**. Restricting inbound access may block your own access When you select this option, only connections from approved IP addresses and workspace-level private links are allowed. If you or your team need to continue accessing this workspace from the Fabric portal, add your organization's public IP addresses before saving. Consult your network administrator if you're unsure which IP ranges to include. If you get locked out, a Fabric Administrator can restore access by navigating to the [Admin portal](https://app.fabric.microsoft.com/admin-portal) → **Tenant settings** → **Advanced networking** → **Configure workspace-level inbound network rules** and toggling the setting to **Disabled**. This reverts all workspaces to their previous network configuration. ##### 3. Register the Microsoft.Fabric resource provider The `Microsoft.Fabric` resource provider must be registered in your Azure subscription before you can create Fabric Private Link resources. 1. In the [Azure portal](https://portal.azure.com), navigate to your **Subscription** → **Settings** → **Resource providers**. 2. Search for `Microsoft.Fabric`. 3. Select it and click **Register** (or **Re-register** if already registered). Wait for the status to change to **Registered**. ##### 4. Create the Private Link Service resource Fabric workspace-level Private Link uses a dedicated Azure resource (`Microsoft.Fabric/privateLinkServicesForFabric`) to expose your workspace as a private endpoint target. This resource is not available in the Azure portal marketplace, so you must deploy it using a custom template. 1. Locate the two IDs you'll need: * **Workspace ID**: found in the workspace URL — `https://app.fabric.microsoft.com/groups/WORKSPACE_ID`. Replace `WORKSPACE_ID` with your value. * **Tenant ID**: found in the [Azure portal](https://portal.azure.com) under **Microsoft Entra ID** → **Overview** → **Tenant ID**. This is the same Directory (tenant) ID used for service principal authentication. 2. In the [Azure portal](https://portal.azure.com), search for **Deploy a custom template** and select it. 3. Click **Build your own template in the editor**. 4. Paste the following template into the editor. Replace `WORKSPACE_NAME`, `WORKSPACE_ID`, and `TENANT_ID` with your values, then click **Save**: ```json { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "resources": [ { "type": "Microsoft.Fabric/privateLinkServicesForFabric", "apiVersion": "2024-06-01", "name": "WORKSPACE_NAME-pls", "location": "global", "properties": { "tenantId": "TENANT_ID", "workspaceId": "WORKSPACE_ID" } } ] } ``` 5. Select your **Subscription** and **Resource group**, then click **Review + create** → **Create**. 6. Once the deployment completes, navigate to your **resource group** and select **Manage view** → **Show hidden types** to reveal the Private Link Service resource. Click the resource to open it and copy the **Resource ID** from the overview page. You will need the Resource ID to submit your Private Link request. The Resource ID path format is: `/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Fabric/privateLinkServicesForFabric/NAME` #### Submit a Private Link request to dbt Add the required information to the following template and submit your Azure Private Link request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Azure Multi-Tenant Private Link Request - Type: Microsoft Fabric - dbt platform account URL: - Fabric workspace name: - Fabric Private Link Service resource ID: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Fabric/privateLinkServicesForFabric/NAME - dbt Azure multi-tenant environment (EMEA): - Azure region of your Fabric workspace (for example, centralus, westeurope): ``` Make sure you replace `SUBSCRIPTION_ID`, `RESOURCE_GROUP`, and `NAME` with your values. #### Approve the Private Link connection Once dbt Support confirms the private endpoint has been created on our side: 1. In the [Azure portal](https://portal.azure.com), search for **Private Link** in the top search bar and select **Private Link Center**. 2. In the left sidebar, select **Pending connections**. 3. Verify the correct **Subscription** filter is selected (the subscription containing your Private Link Service resource). 4. Select the connection from dbt (the name will contain `dbt`). 5. Click **Approve** and confirm. 6. Confirm with dbt Support that the connection has been approved so they can validate it and make it available in dbt. Use Private Link Center, not the PLS resource Navigate to the **Private Link Center** service to approve the connection. The Fabric Private Link Service resource blade does not display pending connections. You must use **Private Link Center** → **Pending connections** to see and approve the connection from dbt. #### Create connection in dbt Once dbt Support confirms the endpoint is available, configure the connection in dbt: 1. Navigate to **Settings** → **Create new connection** → select **Microsoft Fabric**. 2. You will see two radio buttons: **Default Endpoint** and **PrivateLink Endpoint**. Select **PrivateLink Endpoint**. 3. Select the private endpoint from the dropdown. This populates the **Server** field with the private SQL hostname provided by dbt Support. 4. Configure the remaining connection details: | Field | Value | | ------------ | -------------------------- | | **Port** | Auto-populated with `1433` | | **Database** | Your Fabric warehouse name | 5. Save it and proceed to the next step. ##### Configure authentication The **Connection** page only includes **Server**, **Port**, and **Database**. Configure the following later when you [set up your development credentials](https://docs.getdbt.com/docs/dbt-platform-environments.md#set-user-credentials) or [create a deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md): | Field | Value | | ------------------ | ------------------------------------------------ | | **Authentication** | Your Service Principal | | **Tenant ID** | Your Microsoft Entra Directory (tenant) ID | | **Client ID** | Your service principal's application (client) ID | | **Client secret** | Your service principal's client secret | Add the service principal you use for authentication as a **Member** or **Contributor** on the Fabric workspace. In the Fabric Admin portal, you must also enable **Service principals can use Fabric APIs** under **Tenant settings** → **Developer settings**. --- ##### Configuring public IP restrictions dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. IP restrictions help control which IP addresses can connect to dbt. They allow dbt customers to meet security and compliance controls by only allowing approved IPs to connect to their dbt environment. This feature is supported in all regions across NA, Europe, and Asia-Pacific, but contact us if you have questions about availability. #### Configuring IP restrictions To configure IP restrictions, go to **Account Settings** → **IP Restrictions**. IP restrictions provide two methods for determining which IPs can access dbt: an allowlist and a blocklist. IPs in the allowlist can access dbt, and IPs in the blocklist are blocked from accessing dbt. You can use IP restrictions for a range of use cases, including: * Allow only corporate VPN traffic and deny all other traffic * Deny IPs flagged by the security team * Allow only VPN traffic but make an exception for contractors' IP addresses IP restrictions block all service tokens, user requests made through the API (using personal user tokens), and the UI if they come from blocked IP addresses. For any version control system integrations (GitHub, GitLab, ADO, and others) inbound into dbt, ensure you add their IP addresses to the allowed list. ##### Allowing IPs To add an IP to the allowlist, from the **IP Restrictions** page: 1. Click **Edit**. 2. Click **Add Rule**. 3. Add a name and description for the rule. * For example, Corporate VPN CIDR Range 4. Select **Allow**. 5. Add the ranges in CIDR notation. * For example, 1.1.1.1/8 * You can add multiple ranges in the same rule. 6. Click **Save**. Add multiple IP ranges by clicking the **Add IP range** button to create a new text field. Simply adding the IP ranges does not enforce IP restrictions. For more information, refer to the [Enabling restrictions](#enabling-restrictions) section. If you only want to allow the IP ranges added to this list and deny all other requests, you don't need to add a blocklist. By default, if you only add an allowlist, dbt only allows IPs in the allowable range and denies all other IPs. However, you can add a blocklist if you want to deny specific IP addresses within your allowlist CIDR range. ##### Blocking IPs (deny) If you have IPs defined in the allowlist that need to be denied, you can add those IP ranges to the blocklist: 1. Click **Edit**. 2. Click **Add Rule**. 3. Add a name and description for the rule. * For example, "Corporate VPN Deny Range" 4. Select **Deny**. 5. Add the ranges or individual IP addresses in CIDR notation. 6. Click **Save**. Duplicate IP addresses If identical IP addresses are in both the allow and block configurations, the second entry fails to save. You can put an IP range on one list and then a sub-range or IP address that is part of it on the other. Using USA (range) and NY (sub-range) as an example, the expected behavior is: * USA is on blocklist and NY is on allowlist — Traffic from the USA is blocked, but IPs from NY are allowed. * USA is on the allowlist and NY is on the blocklist — USA traffic is allowed, but IPs from NY are blocked. #### Enabling restrictions Once you finish adding all your ranges, you can enable IP restrictions by selecting **Enable IP restrictions** and clicking **Save**. If your IP address is in any of the blocklist ranges, you can't save or enable IP restrictions — this prevents accidental account lockouts. If you get locked out due to IP changes on your end, reach out to <support@getdbt.com>. Once enabled, when someone attempts to access dbt from a restricted IP, they encounter one of the following messages depending on whether they use email and password or SSO login: * For email logins: "Access denied! Please contact your admin for more details." * For SSO logins: "Access denied! Please contact your admin for more details." on a dbt login page --- ##### Configuring Snowflake and Azure Private Link dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of an Azure-hosted Snowflake Private Link endpoint in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Snowflake) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. Snowflake OAuth with Private Link Users connecting to Snowflake using [Snowflake OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md) over an Azure Private Link connection from dbt also require access to a Private Link endpoint from their local workstation. Where possible, use [Snowflake External OAuth](https://docs.getdbt.com/docs/platform/manage-access/snowflake-external-oauth.md) instead to bypass this limitation. Snowflake docs: > Currently, for any given Snowflake account, SSO works with only one account URL at a time: either the public account URL or the URL associated with the private connectivity service * [Snowflake SSO with Private Connectivity](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-overview#label-sso-private-connectivity) #### Configure Azure Private Link To configure Snowflake instances hosted on Azure for [Private Link](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview): 1. In your Snowflake account, run the following SQL statements and copy the output: ```sql USE ROLE ACCOUNTADMIN; SELECT SYSTEM$GET_PRIVATELINK_CONFIG(); ``` 2. Add the required information to the following template and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Azure Multi-Tenant Private Link Request - Type: Snowflake - dbt platform account URL: - The output from SYSTEM$GET_PRIVATELINK_CONFIG: - Include the privatelink-pls-id - Enable Internal Stage Private Link? Y/N (If Y, output must include privatelink-internal-stage) - dbt Azure multi-tenant environment (US or EMEA): ``` dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. 3. dbt Support will provide the `private endpoint resource_id` of our `private_endpoint` for you to complete the [PrivateLink configuration](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors) by contacting the Snowflake Support team. 4. (Optional) If enabling an [Azure private endpoint for an Internal Stage](https://docs.snowflake.com/en/user-guide/private-internal-stages-azure), it will also provide the `resource_id` for the Internal Stage endpoint. As the Snowflake administrator, call the `SYSTEM$AUTHORIZE_STAGE_PRIVATELINK_ACCESS` function using the resource ID value as the function argument. This authorizes access to the Snowflake internal stage through the private endpoint. ```sql USE ROLE ACCOUNTADMIN; -- Azure Private Link SELECT SYSTEM$AUTHORIZE_STAGE_PRIVATELINK_ACCESS ( 'AZURE_PRIVATE_ENDPOINT_RESOURCE_ID' ); ``` #### Create connection in dbt Once dbt Support completes the configuration, you can start creating new connections using Private Link. 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Snowflake**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. #### Configuring network policies If your organization uses [Snowflake Network Policies](https://docs.snowflake.com/en/user-guide/network-policies) to restrict access to your Snowflake account, you need to add a network rule for dbt. Use the Azure Link ID to identify the dbt private endpoint in the network rule. ##### Find the endpoint Azure Link ID Snowflake allows you to find the Azure Link ID of configured endpoints by running the `SYSTEM$GET_PRIVATELINK_AUTHORIZED_ENDPOINTS` command. Use the following to isolate the Link ID value and the associated endpoint resource name: ```sql select value:linkIdentifier, REGEXP_SUBSTR(value: endpointId, '([^\/]+$)') from table( flatten( input => parse_json(system$get_privatelink_authorized_endpoints()) ) ); ``` ##### Using the UI Open the Snowflake UI and take the following steps: 1. Go to the **Security** tab. 2. Click on **Network Rules**. 3. Click on **+ Network Rule**. 4. Give the rule a name. 5. Select a database and schema where the rule will be stored. These selections are for permission settings and organizational purposes; they do not affect the rule itself. 6. Set the type to `Azure Link ID` and the mode to `Ingress`. 7. In the identifier box, type the Azure Link ID obtained in the previous section and press **Enter**. 8. Click **Create Network Rule**. [![Create Network Rule](/img/docs/dbt-platform/snowflakeprivatelink2.png?v=2 "Create Network Rule")](#)Create Network Rule 9. In the **Network Policy** tab, edit the policy to which you want to add the rule. This could be your account-level policy or one specific to the users connecting from dbt. 10. Add the new rule to the allowed list and click **Update Network Policy**. [![Update Network Policy](/img/docs/dbt-platform/snowflakeprivatelink3.png?v=2 "Update Network Policy")](#)Update Network Policy ##### Using SQL For quick and automated setup of network rules via SQL in Snowflake, the following commands allow you to create and configure access rules for dbt. These SQL examples demonstrate how to add a network rule and update your network policy accordingly. 1. Create a new network rule with the following SQL: ```sql CREATE NETWORK RULE allow_dbt_cloud_access MODE = INGRESS TYPE = AZURELINKID VALUE_LIST = ('<Azure Link ID>'); -- Replace '<Azure Link ID>' with the actual ID obtained above ``` 2. Add the rule to a network policy with the following SQL: ```sql ALTER NETWORK POLICY <network_policy_name> ADD ALLOWED_NETWORK_RULE_LIST =('allow_dbt_cloud_access'); ``` --- ##### Configuring Snowflake Private Service Connect dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of a GCP Snowflake Private Service Connect (PSC) endpoint in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Snowflake) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. warning GCP Internal Stage PSC connections are not currently supported. #### Configure GCP Private Service Connect The dbt Labs GCP project has been pre-authorized for connections to Snowflake accounts. To configure Snowflake instances hosted on GCP for [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect): 1. Run the Snowflake system function [SYSTEM$GET\_PRIVATELINK\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) and copy the output. 2. Add the required information to the following template and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Multi-Tenant GCP PSC Request - Type: Snowflake - dbt platform account URL: - SYSTEM$GET_PRIVATELINK_CONFIG output: - *Use privatelink-account-url or regionless-privatelink-account-url?: - dbt GCP multi-tenant environment: ``` *\*By default, dbt will be configured to use `privatelink-account-url` from the provided [SYSTEM$GET\_PRIVATELINK\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) as the PrivateLink endpoint. Upon request, `regionless-privatelink-account-url` can be used instead.* dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### Create connection in dbt Once dbt Support completes the configuration, you can start creating new connections using PrivateLink. 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Snowflake**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. #### Configuring network policies If your organization uses [Snowflake Network Policies](https://docs.snowflake.com/en/user-guide/network-policies) to restrict access to your Snowflake account, you need to add a network rule for dbt. Request the **PSC connection ID** from [dbt Support](mailto:support@getdbt.com) to use in a network rule. Snowflake supports [`GCPPSCID` as a network rule identifier type](https://docs.snowflake.com/en/sql-reference/sql/create-network-rule), and this is the recommended approach. A PSC connection ID uniquely identifies your organization's connection endpoint, whereas IP-based rules rely on CIDR ranges that may be shared across multiple dbt customers. ##### Using the UI Open the Snowflake UI and take the following steps: 1. Go to the **Security** tab. 2. Click on **Network Rules**. 3. Click on **Add Rule**. 4. Give the rule a name. 5. Select a database and schema where the rule will be stored. These selections are for permission settings and organizational purposes; they do not affect the rule itself. 6. Set the type to `GCPPSCID` and the mode to `Ingress`. 7. Enter the PSC connection ID provided by dbt Support into the identifier box and press **Enter**. 8. Click **Create Network Rule**. 9. In the **Network Policy** tab, edit the policy you want to add the rule to. This could be your account-level policy or a policy specific to the users connecting from dbt. 10. Add the new rule to the allowed list and click **Update Network Policy**. ##### Using SQL For quick and automated setup of network rules via SQL in Snowflake, the following commands allow you to create and configure access rules for dbt. These SQL examples demonstrate how to add a network rule and update your network policy accordingly. 1. Create a new network rule with the following SQL: ```sql CREATE NETWORK RULE allow_dbt_cloud_access MODE = INGRESS TYPE = GCPPSCID VALUE_LIST = ('<PSC_CONNECTION_ID>'); -- Replace with the PSC connection ID from dbt Support ``` 2. Add the rule to a network policy with the following SQL: ```sql ALTER NETWORK POLICY <network_policy_name> ADD ALLOWED_NETWORK_RULE_LIST =('allow_dbt_cloud_access'); ``` --- ##### Configuring Snowflake PrivateLink dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. The following steps walk you through the setup of an AWS-hosted Snowflake PrivateLink endpoint in a dbt multi-tenant environment. Private connection endpoints can't connect across cloud providers (AWS, Azure, and GCP). For a private connection to work, both dbt and the server (like Snowflake) must be hosted on the same cloud provider. For example, dbt hosted on AWS cannot connect to services hosted on Azure, and dbt hosted on Azure can’t connect to services hosted on GCP. Snowflake OAuth with PrivateLink Users connecting to Snowflake using [Snowflake OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md) over an AWS PrivateLink connection from dbt will also require access to a PrivateLink endpoint from their local workstation. Where possible, use [Snowflake External OAuth](https://docs.getdbt.com/docs/platform/manage-access/snowflake-external-oauth.md) instead to bypass this limitation. From the [Snowflake](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-overview#label-sso-private-connectivity) docs: > Currently, for any given Snowflake account, SSO works with only one account URL at a time: either the public account URL or the URL associated with the private connectivity service #### Configure AWS PrivateLink This section walks you through the setup of an AWS-hosted Snowflake PrivateLink endpoint in a dbt platform. You can set up in two ways: * [Self-serve private endpoints](#self-serve-private-endpoints): Self-serve configuration of Snowflake PrivateLink endpoints directly in dbt platform user interface. Currently in beta. * [Support-led setup](#support-led-setup): Requires contacting dbt Support to configure Snowflake PrivateLink endpoints. Non-self service configuration of Snowflake PrivateLink endpoints. ##### Self-serve private endpoints [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") *Self-serve private endpoints are currently in beta for Snowflake on AWS, and available to all eligible customers. This feature isn't available for Azure or GCP. If you don't see **Private endpoints** in your account settings, use the [Support-led setup](#support-led-setup) instead.* This section walks you through the process of requesting a new Snowflake PrivateLink endpoint in dbt platform. ###### Prerequisites * [Account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md?version=2.0#account-admin) or [Project creator](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md?version=2.0#project-creator) permission sets in dbt platform. Users with an IT license can also create private endpoints. * [Snowflake `ACCOUNTADMIN` permissions](https://docs.snowflake.com/en/user-guide/security-access-control-overview#system-defined-roles). ###### Before you begin Before requesting a private endpoint, allowlist dbt Labs' AWS account in Snowflake. This is a one-time setup per Snowflake account and is handled through a Snowflake support case, which can take time — complete it before starting the request flow. 1. Open a support case with Snowflake to allow access from the dbt AWS account. * Snowflake prefers that the account owner opens the support case directly rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors). * Provide your dbt account ID along with any other information requested in the article. * **AWS account ID**: `346425330055` — *Note: This account ID only applies to AWS dbt multi-tenant environments. For AWS Virtual Private/Single-Tenant account IDs, contact [dbt Support](mailto:support@getdbt.com).* 2. Wait for Snowflake to confirm access has been granted before proceeding. [![Open snowflake case](/img/docs/dbt-platform/snowflakeprivatelink1.png?v=2 "Open snowflake case")](#)Open snowflake case ###### Request a new private endpoint After Snowflake confirms they've allowlisted dbt Labs' AWS account in Snowflake, you can request a new private endpoint. Follow these steps to do so: 1. In dbt platform, go to **Account settings → Private endpoints**. 2. In the **Private endpoints** table, review your existing endpoints. The table shows all private endpoints in your account (including non-Snowflake ones) with the following details: * **Name** * **Connection type** (for example, Snowflake) * **URL** * **Connectivity status** (for example, **Success** or **Unknown**) * **Connections** — the number of dbt platform connections using the endpoint You can search by **Name** or **URL**. The table lists all private endpoints in your account, but self-serve create, edit, and delete is only available for Snowflake on AWS at this time. [![Private endpoints table showing existing endpoints, connectivity status, and the Request new button](/img/docs/dbt-platform/private-endpoint-page.png?v=2 "Private endpoints table showing existing endpoints, connectivity status, and the Request new button")](#)Private endpoints table showing existing endpoints, connectivity status, and the Request new button 3. To request a new endpoint, click **Request new**. 4. Under **Provider type**, confirm **Snowflake** is selected. Currently other endpoint providers aren't supported, contact [dbt Support](mailto:support@getdbt.com) if you need to connect to a different service. 5. Copy the SQL command in the **SQL command snippet** section. 6. Go to Snowflake and run the SQL command snippet you copied from dbt platform: `SELECT SYSTEM$GET_PRIVATELINK_CONFIG();` 7. Copy the output from Snowflake and return to dbt platform to paste it into the **Snowflake output** field. If the output is correct, you'll see an inline **Output looks good** type message below the text box. If there's an error, review the message and make any updates as necessary. 8. Click **Submit request**. [![Endpoint request form showing Provider type, SQL command snippet, and Snowflake output fields](/img/docs/dbt-platform/private-endpoint-config.png?v=2 "Endpoint request form showing Provider type, SQL command snippet, and Snowflake output fields")](#)Endpoint request form showing Provider type, SQL command snippet, and Snowflake output fields 9. After submission, a confirmation popup appears (for example, **Endpoint request submitted**). From the popup, you can request another endpoint or return to **Private endpoints** to track request status. 10. Proceed to the **Connections** page and following the steps in the [Create connection in dbt](#create-connection-in-dbt) section to configure PrivateLink. Once you configure PrivateLink on the **Connections** page, you'll see the new endpoint appear under **Private endpoints → Associated connections**. DNS propagation If the connection test fails immediately after setup, this is expected — it doesn't mean something is wrong. DNS changes can take a few minutes to propagate. Wait a few minutes, then test again before contacting support.  Edit or delete a private endpoint If you don't see **Edit** or **Delete endpoint**, contact your account manager to enable private endpoint updates for your account. **Edit an endpoint** 1. In the **Private endpoints** table, click the endpoint you want to update. 2. On the endpoint details page, click **Edit**. 3. Update **Name** and/or **Port** as needed. 4. Click **Save changes**. 5. In the **Save changes?** modal, click **Save changes** to apply your updates. [![Private endpoint details page with the Edit button](/img/docs/dbt-platform/private-endpoint-details-edit.png?v=2 "Private endpoint details page with the Edit button")](#)Private endpoint details page with the Edit button **Delete an endpoint** An endpoint with associated connections can't be deleted. Remove those connections first. 1. In the **Private endpoints** table, click the endpoint you want to delete. 2. On the endpoint details page, click **Edit**. 3. Scroll to the bottom of the page and click **Delete endpoint**. 4. In the **Delete endpoint** modal, type `DELETE` to confirm, then click **Delete endpoint**. [![Private endpoint details page in edit mode with the Delete endpoint button](/img/docs/dbt-platform/private-endpoint-details-delete.png?v=2 "Private endpoint details page in edit mode with the Delete endpoint button")](#)Private endpoint details page in edit mode with the Delete endpoint button ###### Duplicate endpoint requests If you submit a request using a VPCE ID that matches an existing endpoint, dbt platform displays an **Endpoint already exists** popup with two options: * **Reuse existing interface endpoint** (default, recommended) — Links the new private endpoint to an already-approved interface endpoint. Use this option when your VPCE is already approved to avoid duplicating infrastructure. * **Create new interface endpoint** — Creates a new interface endpoint with its own network policy. Use this only if you need a distinct network policy configuration. Select your preferred option and click **Confirm & Submit**. [![Endpoint already exists popup with options to create a new interface endpoint or re-use an existing one](/img/docs/dbt-platform/endpoint-exists.png?v=2 "Endpoint already exists popup with options to create a new interface endpoint or re-use an existing one")](#)Endpoint already exists popup with options to create a new interface endpoint or re-use an existing one ###### Troubleshooting and errors If an endpoint request fails, dbt platform displays error details that are safe to share externally. If you see a failure state without clear next steps, collect the request details (endpoint name, creation time, status, and the Snowflake output you provided) and contact [dbt Support](mailto:support@getdbt.com). ##### Support-led setup If **Private endpoints** is not available in your account settings, configure Snowflake PrivateLink by following these steps and submitting a request to dbt Support. To configure Snowflake instances hosted on AWS for [PrivateLink](https://aws.amazon.com/privatelink): 1. Open a support case with Snowflake to allow access from the dbt AWS account. * Snowflake prefers that the account owner opens the support case directly rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors). * Provide them with your dbt account ID along with any other information requested in the article. * **AWS account ID**: `346425330055` — *Note: This account ID only applies to AWS dbt multi-tenant environments. For AWS Virtual Private/Single-Tenant account IDs, contact [dbt Support](mailto:support@getdbt.com).* * You need [Snowflake's `ACCOUNTADMIN` permissions](https://docs.snowflake.com/en/user-guide/security-access-control-overview#system-defined-roles). [![Open snowflake case](/img/docs/dbt-platform/snowflakeprivatelink1.png?v=2 "Open snowflake case")](#)Open snowflake case 2. After Snowflake has granted the requested access, run the Snowflake system function [SYSTEM$GET\_PRIVATELINK\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) and copy the output. 3. Add the required information to the following template and submit your request to [dbt Support](mailto:support@getdbt.com):  Support request email template ```text Subject: New Multi-Tenant (Azure or AWS) PrivateLink Request - Type: Snowflake - dbt platform account URL: - SYSTEM$GET_PRIVATELINK_CONFIG output: - *Use privatelink-account-url or regionless-privatelink-account-url?: - **Create Internal Stage PrivateLink endpoint? (Y/N): - dbt AWS multi-tenant environment (US, EMEA, AU, JP): ``` *\*By default, dbt will be configured to use `privatelink-account-url` from the provided [SYSTEM$GET\_PRIVATELINK\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) as the PrivateLink endpoint. Upon request, `regionless-privatelink-account-url` can be used instead.* *\*\* Internal Stage PrivateLink must be [enabled on the Snowflake account](https://docs.snowflake.com/en/user-guide/private-internal-stages-aws#prerequisites) to use this feature* dbt Labs will work on your behalf to complete the private connection setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available. #### Create connection in dbt Once dbt Support completes the configuration, you can start creating new connections using PrivateLink. 1. Navigate to **Account Settings** → **Projects** → **Create new project**. 2. Under the **Configure your development environment step** section, select **Add new connection**. 3. You'll be directed to the **Add new connection** page, select **Snowflake**. 4. Under the **Settings** section, select **PrivateLink Endpoint**. 5. Select the private endpoint from the dropdown (this automatically populates the hostname/account field). 6. Configure the remaining data platform details. 7. Test your connection and save it. #### Configuring internal stage PrivateLink in dbt If an Internal Stage PrivateLink endpoint has been provisioned, your dbt environments must be configured to use this endpoint instead of the account default set in Snowflake. 1. Obtain the Internal Stage PrivateLink endpoint DNS from dbt Support. For example, `*.vpce-012345678abcdefgh-4321dcba.s3.us-west-2.vpce.amazonaws.com`. 2. In the appropriate dbt project, navigate to **Orchestration** → **Environments**. 3. Open the environment that should use the dbt Internal Stage PrivateLink endpoint and set **Extended attributes**. Configuration steps differ for deployment and development; refer to [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes). Use a value similar to the following: ```text s3_stage_vpce_dns_name: '*.vpce-012345678abcdefgh-4321dcba.s3.us-west-2.vpce.amazonaws.com' ``` 4. Save the changes. [![Internal Stage DNS](/img/docs/dbt-platform/snowflake-internal-stage-dns.png?v=2 "Internal Stage DNS")](#)Internal Stage DNS #### Configuring network policies If your organization uses [Snowflake Network Policies](https://docs.snowflake.com/en/user-guide/network-policies) to restrict access to your Snowflake account, you need to add a network rule for dbt. Snowflake supports two network rule types: VPCE ID-based (recommended) and IP/CIDR-based. The following steps use VPCE ID. If your organization requires an IP-based network policy instead, the CIDR range isn't available in the dbt platform UI. Please contact [dbt Support](mailto:support@getdbt.com) to get it. You need a VPCE ID to create a network policy in Snowflake: 1. In dbt platform, go to **Account settings → Private endpoints** 2. Click the endpoint in the table, then locate its **Endpoint identifier** field on the endpoint details page. 3. If you configured PrivateLink through [Support-led setup](#support-led-setup), or **Private endpoints** is not available in your account settings, contact [dbt Support](mailto:support@getdbt.com) to obtain the VPCE ID. 4. If you're creating an endpoint for Internal Stage, the VPCE ID is different from the VPCE ID for the main service endpoint. Network Policy for Snowflake Internal Stage PrivateLink For guidance on protecting both the Snowflake service and Internal Stage consult the Snowflake [network policies](https://docs.snowflake.com/en/user-guide/network-policies#strategies-for-protecting-both-service-and-internal-stage) and [network rules](https://docs.snowflake.com/en/user-guide/network-rules#incoming-requests) docs. ##### Using the UI Open the Snowflake UI and take the following steps: 1. Go to the **Security** tab. 2. Click on **Network Rules**. 3. Click on **Add Rule**. 4. Give the rule a name. 5. Select a database and schema where the rule will be stored. These selections are for permission settings and organizational purposes; they do not affect the rule itself. 6. Set the type to `AWS VPCE ID` and the mode to `Ingress`. 7. Enter the VPCE ID from the endpoint details page in dbt platform (or from dbt Support if you used Support-led setup) into the identifier box. 8. Click **Create Network Rule**. [![Create Network Rule](/img/docs/dbt-platform/snowflakeprivatelink2.png?v=2 "Create Network Rule")](#)Create Network Rule 9. In the **Network Policy** tab, edit the policy you want to add the rule to. This could be your account-level policy or a policy specific to the users connecting from dbt. 10. Add the new rule to the allowed list and click **Update Network Policy**. [![Update Network Policy](/img/docs/dbt-platform/snowflakeprivatelink3.png?v=2 "Update Network Policy")](#)Update Network Policy ##### Using SQL For quick and automated setup of network rules via SQL in Snowflake, the following commands allow you to create and configure access rules for dbt. These SQL examples demonstrate how to add a network rule and update your network policy accordingly. 1. Create a new network rule with the following SQL: ```sql CREATE NETWORK RULE allow_dbt_cloud_access MODE = INGRESS TYPE = AWSVPCEID VALUE_LIST = ('<VPCE_ID>'); -- Replace '<VPCE_ID>' with the VPCE ID the actual value ``` 2. Add the rule to a network policy with the following SQL: ```sql ALTER NETWORK POLICY <network_policy_name> ADD ALLOWED_NETWORK_RULE_LIST =('allow_dbt_cloud_access'); ``` --- ##### GCP private connectivity dbt platform | Enterprise+ Available to certain Enterprise tiers This feature is available on the following dbt Enterprise tiers: * Enterprise+ * Virtual Private To learn more about these tiers, contact us at <sales@getdbt.com>. GCP Private Service Connect enables secure, private connectivity between dbt and your GCP-hosted services. With Private Service Connect, traffic between dbt and your data platforms or self-hosted services stays within the Google Cloud network and does not traverse the public internet. For more details, refer to the [GCP Private Service Connect documentation](https://cloud.google.com/vpc/docs/private-service-connect). #### GCP private connectivity matrix The following charts outline private connectivity options for GCP deployments of dbt ([multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md)). **Legend:** * ✅ = Available * ❌ = Not currently available * \* = Shared endpoint (all others are dedicated) *Tenancy:* MT (multi-tenant) — [learn more about tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). About the following matrix tables These tables indicate whether private connectivity can be established to specific services, considering major factors such as the network and basic auth layers. dbt has validated these configurations using common deployment patterns and typical use cases. However, individual configurations may vary. If you encounter issues or have questions about your environment, [contact dbt Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support) for guidance. **GCP regional considerations:** Some GCP services, such as BigQuery, may have regional restrictions for Private Service Connect endpoints. Refer to [Google's Private Service Connect documentation](https://cloud.google.com/vpc/docs/private-service-connect) for service-specific regional availability. *** ##### Connecting the dbt platform to managed services (Egress) dbt can establish private connections to managed data platforms and cloud-native services. | Service | MT | Setup guide | | --------------------- | ---- | ---------------------------------------------------------------------------------------------- | | Snowflake | ✅ | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/gcp/gcp-snowflake.md) | | Google BigQuery | ✅\* | [View](https://docs.getdbt.com/docs/platform/secure/private-connectivity/gcp/gcp-bigquery.md) | | Teradata VantageCloud | ✅ | | *** ##### Connecting the dbt platform to self-hosted services (Egress) All of the services below share a common Private Service Connect setup guide — backend configuration varies by service. Self-hosted connections use the customer-provisioned model — you are the service producer and dbt is the consumer. **Setup guide:** [Configuring GCP Private Service Connect for self-hosted services](https://docs.getdbt.com/docs/platform/secure/private-connectivity/gcp/gcp-self-hosted.md) | Service | MT | | ------------------------ | -- | | GitHub Enterprise Server | ✅ | | GitLab Self-Managed | ✅ | | Bitbucket Data Center | ✅ | | Azure DevOps Server | ✅ | | Postgres | ✅ | | Starburst / Trino | ✅ | | Teradata (self-hosted) | ✅ | If you have questions about whether your specific architecture is supported, [contact dbt Support](https://docs.getdbt.com/community/resources/getting-help.md#dbt-cloud-support). #### Cross-region private connections dbt Labs has globally connected private networks specifically used to host private endpoints, which are connected to dbt instance environments. This connectivity allows dbt environments to connect to any supported region from any dbt instance within the same cloud provider network. To ensure security, access to these endpoints is protected by security groups, network policies, and application connection safeguards, in addition to the authentication and authorization mechanisms provided by each of the connected platforms. --- ## Sql Reference ### SQL AND The AND operator returns results that meet all requirements passed into it; compared to the [OR operator](https://docs.getdbt.com/sql-reference/or.md) that only needs to have one true requirement. You’ll often see the AND operator used in a [WHERE clause](https://docs.getdbt.com/sql-reference/where.md) to filter query results or in a case statement to create multiple criteria for a result. Use this page to understand how to use the AND operator and why it might be helpful in analytics engineering work. #### How to use the AND operator It’s straightforward to use the AND operator, and you’ll typically see it appear in a WHERE clause to filter query results appropriately, in case statements, or joins that involve multiple fields. ```sql -- and in a where clause where <condition_1> and <condition_2> and… -- and in a case statement case when <condition_1> and <condition_2> then <result_1> … -- and in a join from <table_a> join <table_b> on <a_id_1> = <b_id_1> and <a_id_2> = <b_id_2> ``` Surrogate keys > joins with AND Using surrogate keys, hashed values of multiple columns, is a great way to avoid using AND operators in joins. Typically, having AND or [OR operators](https://docs.getdbt.com/sql-reference/or.md) in a join can cause the query or model to be potentially inefficient, especially at considerable data volume, so creating surrogate keys earlier in your upstream tables ([using the surrogate key macro](https://docs.getdbt.com/blog/sql-surrogate-keys)) can potentially improve performance in downstream models. ##### SQL AND operator example ```sql select order_id, status, round(amount) as amount from {{ ref('orders') }} where status = 'shipped' and amount > 20 limit 3 ``` This query using the sample dataset Jaffle Shop’s `orders` table will return results where the order status is shipped and the order amount is greater than $20: | **order\_id** | **status** | **amount** | | ------------- | ---------- | ---------- | | 74 | shipped | 30 | | 88 | shipped | 29 | | 78 | shipped | 26 | #### AND operator syntax in Snowflake, Databricks, BigQuery, and Redshift Snowflake, Databricks, Google BigQuery, and Amazon Redshift all support the AND operator with the same syntax for it across each platform. --- ### SQL ANY and ALL The SQL ANY and ALL operators are useful for evaluating conditions to limit query results; they are often passed in with [LIKE](https://docs.getdbt.com/sql-reference/like.md) and [ILIKE](https://docs.getdbt.com/sql-reference/ilike.md) operators. The ANY operator will return true if any of the conditions passed into evaluate to true, while ALL will only return true if *all* conditions passed into it are true. Use this page to better understand how to use ANY and ALL operators, use cases for these operators, and which data warehouses support them. #### How to use the SQL ANY and ALL operators The ANY and ALL operators have very simple syntax and are often passed in the LIKE/ILIKE operator or subquery: `where <field_name> like/ilike any/all (array_of_options)` `where <field_name> = any/all (subquery)` Some notes on this operator’s syntax and functionality: * You may pass in a subquery into the ANY or ALL operator instead of an array of options * Use the ILIKE operator with ANY or ALL to avoid case sensitivity Let’s dive into a practical example using the ANY operator now. ##### SQL ANY example ```sql select order_id, status from {{ ref('orders') }} where status like any ('return%', 'ship%') ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return orders whose status is like the patterns `start with 'return'` or `start with 'ship'`: | order\_id | status | | --------- | --------------- | | 18 | returned | | 23 | return\_pending | | 74 | shipped | Because LIKE is case-sensitive, it would not return results in this query for orders whose status were say `RETURNED` or `SHIPPED`. If you have a mix of uppercase and lowercase strings in your data, consider standardizing casing for strings using the [UPPER](https://docs.getdbt.com/sql-reference/upper.md) and [LOWER](https://docs.getdbt.com/sql-reference/lower.md) functions or use the more flexible ILIKE operator. #### ANY and ALL syntax in Snowflake, Databricks, BigQuery, and Redshift Snowflake and Databricks support the ability to use ANY in a LIKE operator. Amazon Redshift and Google BigQuery, however, do not support the use of ANY in a LIKE or ILIKE operator. Use the table below to read more on the documentation for the ANY operator in your data warehouse. | **Data warehouse** | **ANY support?** | **ALL support?** | | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | [Snowflake](https://docs.snowflake.com/en/sql-reference/functions/like_any.html) | ✅ | ✅ | | [Databricks](https://docs.databricks.com/sql/language-manual/functions/like.html) | ✅ | ✅ | | Amazon Redshift | ❌Not supported; consider utilizing multiple OR clauses or [IN operators](https://docs.getdbt.com/sql-reference/in.md). | ❌Not supported; consider utilizing multiple [AND clauses](https://docs.getdbt.com/sql-reference/and.md) | | Google BigQuery | ❌Not supported; consider utilizing [multiple OR clauses](https://stackoverflow.com/questions/54645666/how-to-implement-like-any-in-bigquery-standard-sql) or IN operators. | ❌Not supported; consider utilizing multiple AND clauses | --- ### SQL ARRAY_AGG In any typical programming language such as Python or Javascript, arrays are typically innate and bountiful; when you’re processing data in SQL, arrays are a little less common but are a handy way to provide more structure to your data. To create an array of multiple data values in SQL, you’ll likely leverage the ARRAY\_AGG function (short for *array aggregation*), which puts your input column values into an array. #### How to use SQL ARRAY\_AGG The ARRAY\_AGG function has the following syntax: `array_agg( [distinct] <field_name>) [within group (<order_by field>) over ([partition by <field>])` A few notes on the functionality of this function: * Most of the example syntax from above is optional, meaning the ARRAY\_AGG function can be as simple as `array_agg(<field_name>)` or used as a more complex as a window function * [DISTINCT](https://docs.getdbt.com/sql-reference/distinct.md) is an optional argument that can be passed in, so only distinct values are in the return array * If input column is empty, the returning array will also be empty * Since the ARRAY\_AGG is an aggregate function (gasp!), you’ll need a GROUP BY statement at the end of your query if you’re grouping by certain field * ARRAY\_AGG and similar aggregate functions can become inefficient or costly to compute on large datasets, so use ARRAY\_AGG wisely and truly understand your use cases for having arrays in your datasets Let’s dive into a practical example using the ARRAY\_AGG function. ##### SQL ARRAY\_AGG example ```sql select date_trunc('month', order_date) as order_month, array_agg(distinct status) as status_array from {{ ref('orders') }} group by 1 order by 1 ``` This simple query using the sample dataset [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table is returning a new column of distinct order statuses by order month: | order\_month | status\_array | | ------------ | ----------------------------------------------- | | 2018-01-01 | \[ "returned", "completed", "return\_pending" ] | | 2018-02-01 | \[ "completed", "return\_pending" ] | | 2018-03-01 | \[ "completed", "shipped", "placed" ] | | 2018-04-01 | \[ "placed" ] | Looking at the query results—this makes sense! We’d expect newer orders to likely not have any returns, and older orders to have completed returns. #### SQL ARRAY\_AGG syntax in Snowflake, Databricks, BigQuery, and Redshift [Snowflake](https://docs.snowflake.com/en/sql-reference/functions/array_agg.html), [Databricks](https://docs.databricks.com/sql/language-manual/functions/array_agg.html), and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#array_agg) all support the ARRAY\_AGG function. Redshift, however, supports an out-of-the-box [LISTAGG function](https://docs.aws.amazon.com/redshift/latest/dg/r_LISTAGG.html) that can perform similar functionality to ARRAY\_AGG. The primary difference is that LISTAGG allows you to explicitly choose a delimiter to separate a list whereas arrays are naturally delimited by commas. #### ARRAY\_AGG use cases There are definitely too many use cases to list out for using the ARRAY\_AGG function in your dbt models, but it’s very likely that ARRAY\_AGG is used pretty downstream in your DAG since you likely don’t want your data so bundled up earlier in your DAG to improve modularity and dryness. A few downstream use cases for ARRAY\_AGG: * In [`export_` models](https://www.getdbt.com/open-source-data-culture/reverse-etl-playbook) that are used to send data to platforms using a reverse ETL tool to pair down multiple rows into a single row. Some downstream platforms, for example, require certain values that we’d usually keep as separate rows to be one singular row per customer or user. ARRAY\_AGG is handy to bring multiple column values together by a singular id, such as creating an array of all items a user has ever purchased and sending that array downstream to an email platform to create a custom email campaign. * Similar to export models, you may see ARRAY\_AGG used in [mart tables](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md) to create final aggregate arrays per a singular dimension; performance concerns of ARRAY\_AGG in these likely larger tables can potentially be bypassed with use of [incremental models in dbt](https://docs.getdbt.com/docs/build/incremental-models.md). --- ### SQL AVG You’re a data person, so we assume you’re going to be calculating averages of some metrics \**waves hands airily*\* at some point in your career. And the way to calculate averages of a numeric column in SQL is by using the AVG function. #### How to use the AVG function The AVG function is a part of the group of mathematical or aggregate functions (ex. MIN, MAX, SUM) that are often used in SQL to summarize datasets. You’ll most likely see the AVG function used to straightforwardly calculate the average of a numeric column, but you may also see it used in a window function to calculate rolling averages. ##### AVG function example The following example is querying from a sample dataset created by dbt Labs called [jaffle\_shop](https://github.com/dbt-labs/jaffle_shop): ```sql select date_trunc('month', order_date) as order_month, round(avg(amount)) as avg_order_amount from {{ ref('orders') }} where status not in ('returned', 'return_pending') group by 1 ``` This query using the Jaffle Shop’s `orders` table will return the rounded order amount per each order month: | order\_month | avg\_order\_amount | | ------------ | ------------------ | | 2018-01-01 | 18 | | 2018-02-01 | 15 | | 2018-03-01 | 18 | | 2018-04-01 | 17 | The AVG function, like many other mathematical functions, is an aggregate function. Aggregate functions operate across all rows, or a group of rows, to return a singular value. When calculating the average of a column across a dimension (or group of dimensions)—in our example above, `order_month`—you need a GROUP BY statement; the query above would not successfully run without it. #### SQL AVG function syntax in Snowflake, Databricks, BigQuery, and Redshift Snowflake, Databricks, Google BigQuery, and Amazon Redshift all support the ability to take the average of a column value and the syntax for the AVG functions is the same across all of those data platforms. #### AVG function use cases We most commonly see the AVG function used in data work to calculate: * The average of key metrics (ex. Average CSAT, average lead time, average order amount) in downstream [fact or dim models](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md) * Rolling or moving averages (ex. 7-day, 30-day averages for key metrics) using window functions * Averages in [dbt metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) This isn’t an extensive list of where your team may be using the AVG function throughout your dbt models and BI tool logic, but contains some common scenarios analytics engineers face in their day-to-day. --- ### SQL BETWEEN The SQL BETWEEN condition allows you to specify a range of numerical, date-type, or text values to filter rows on in a query. It’s particularly useful during ad hoc analysis work to narrow query results on a specific data range. In this page, we’ll dive into how to use the SQL BETWEEN condition and elaborate on why it might be useful to you. #### How to use the SQL BETWEEN condition The BETWEEN condition has a simple syntax and should be passed in a WHERE clause: `where <field_name> between <beginning_value> and <end_value>` It’s important to note that the BETWEEN condition is inclusive of `beginning_value` and `end_value`. Let’s take a look at a practical example using the BETWEEN condition below. ##### SQL BETWEEN example ```sql select customer_id, order_id, order_date from {{ ref('orders') }} where order_date between '2018-01-01' and '2018-01-31' ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return all rows where the `order_date` falls during January 2018: | **customer\_id** | **order\_id** | **order\_date** | | ---------------- | ------------- | --------------- | | 1 | 1 | 2018-01-01 | | 3 | 2 | 2018-01-02 | | 94 | 3 | 2018-01-04 | | 50 | 4 | 2018-01-05 | | 64 | 5 | 2018-01-05 | | 54 | 6 | 2018-01-07 | Alternatively, you could build this same query using >/= operators (`where order_date >= 2018-01-01' and order_date <= '2018-01-31'` or `where order_date >= '2018-01-01' and order_date < '2018-02-01'`). You may additionally see the NOT clause used in front of BETWEEN to exclude rows that fall between specified ranges. #### BETWEEN syntax in Snowflake, Databricks, BigQuery, and Redshift Most, if not all, modern data warehouses support the BETWEEN condition; the syntax is also the same across them. If your data warehouse does not support the BETWEEN condition, consider using the >/= operators similar to the example outlined above. Use the table below to read more on the documentation for the BETWEEN operator in your data warehouse. | **Data warehouse** | **BETWEEN support?** | | ------------------ | -------------------- | | Snowflake | ✅ | | Databricks | ✅ | | Amazon Redshift | ✅ | | Google BigQuery | ✅ | #### SQL BETWEEN condition use cases You’ll most commonly see the BETWEEN condition used in data work to: * Filter query results to be in a specified date range * Create buckets for data using case statements, common for bucketing web session engagement or NPS score classification ```sql case when time_engaged between 0 and 9 then 'low_engagement' when time_engaged between 10 and 29 then 'medium_engagement' else 'high_engagement' end as engagement ``` This isn’t an extensive list of where your team may be using the BETWEEN condition throughout your dbt models or ad hoc analyses, but contains some common scenarios analytics engineers may encounter. --- ### SQL CASE WHEN SQL case statements are the backbone of analytics engineers and dbt projects. They help add context to data, make fields more readable or usable, and allow you to create specified buckets with your data. To informally formalize it, case statements are the SQL equivalent of an if-then statement in other programming languages. They allow you to cascade through multiple scenarios (or cases) in your data, evaluate them if they’re true, and output a corresponding value for each case. In this page, we’ll break down how to use SQL case statements and demonstrate why they’re valuable to modern data teams. #### How to use the SQL case statements Case when statements are created in [SELECT statements](https://docs.getdbt.com/sql-reference/select.md) along with other fields you choose to select. The general syntax for SQL case when statements is as follows: ```sql case when [scenario 1] then [result 1] when [scenario 2] then [result 2] -- …as many scenarios as you want when [scenario n] then [result n] else [fallback result] -- this else is optional end as <new_field_name> ``` Some notes on case statement functionality: * Scenarios in case statements are *evaluated in the order they’re listed*. What does this mean? It means that if multiple scenarios evaluate to true, the earliest listed true scenario is the one whose result is returned. * The results in each scenario need to be of the same data type; if scenario 1 results in a string, all other scenarios need to be [strings](https://docs.getdbt.com/sql-reference/strings.md). * Oftentimes data teams will omit a final `else` scenario since the `else [fallback result]`is optional and defaulted to `else null`. * In general, case statement performance in select statements is relatively efficient (compared to other SQL functionality like aggregates or clunky joins involving ANDs and ORs); this isn’t to say it’s efficient (or smart) to be comparing a ton of scenarios, but it likely won’t be the bottleneck in your data models. * Case when statement results can also be passed into aggregate functions, such as [MAX](https://docs.getdbt.com/sql-reference/max.md), [MIN](https://docs.getdbt.com/sql-reference/min.md), and [COUNT](https://docs.getdbt.com/sql-reference/count.md), or even date functions (ex. `date_trunc('month', <case when statement>`) Below, let’s take a look at a practical example using a case statement. ##### SQL CASE WHEN example ```sql select order_id, round(amount) as amount, case when amount between 0 and 10 then 'low' when amount between 11 and 20 then 'medium' else 'high' end as order_value_bucket from {{ ref('orders') }} ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return a new field that buckets order amount based on criteria: | **order\_id** | **amount** | **order\_value\_bucket** | | ------------- | ---------- | ------------------------ | | 1 | 10 | low | | 2 | 20 | medium | | 3 | 1 | low | | 4 | 25 | high | | 5 | 17 | medium | #### SQL CASE WHEN syntax in Snowflake, Databricks, BigQuery, and Redshift Since it’s a fundamental of SQL, most, if not all, modern data warehouses support the ability to add case when statements to their queries. Snowflake, Databricks, Google BigQuery, and Amazon Redshift all support case statements and have the same syntax for them. #### CASE WHEN use cases The use cases for case statements in dbt models and ad hoc queries is almost endless; as a result, we won’t (be able to) create an exhaustive list of where you might see case statements in the wild. Instead, it’s important to know *why* you’d want to use them in your data work and when you wouldn’t want to use them. Some example reasons you’d want to leverage case statements: * Create booleans from your existing data (ex. `case when cnt > 1 then true else false end as is_active`) * Establish mappings between raw data and more general buckets of data (see example earlier in the page); note that if you find yourself creating many case when scenarios for a mapping that doesn’t change over time, you’ll likely want to import that mapping either as its own dbt model or data source (a good use case for [seeds](https://docs.getdbt.com/docs/build/seeds.md)) * If you find yourself creating the same case when statement throughout your models, consider abstracting that case when into its own model or into a DRY [macro](https://docs.getdbt.com/docs/build/jinja-macros.md) * Generate more business-user friendly columns values that can be easily comprehended by business users --- ### SQL CAST Let’s set the scene: You are knee-deep in a new data model and cannot figure out why the join between `user_id` in` table a` is not successfully joining with the `user_id` in `table b`. You dig a little deeper and discover that `user_id` in `table a` is an integer and `user_id` in `table b` is a string. *Cue throwing hands in the air.* It *will* happen: You’ll find column types in your source data or upstream models that will likely need to be cast into different data types; perhaps to make joins easier, calculations more intuitive, or data more readable. Regardless of the reason, you’ll find yourself inevitably casting some data as an analytics engineer and using the SQL CAST function to help you out. #### How to use SQL CAST function The syntax for using the CAST function looks like the following: ```sql cast(<column_name> as <new_data_type>) ``` Executing this function in a SELECT statement will return the column you specified as the newly specified data type. Analytics engineers will typically be casting fields to more appropriate or useful numeric, strings, and date types. You may additionally use the CAST function in WHERE clauses and in joins. Below, we’ll walk through a practical example using the CAST function. ##### SQL CAST function example You can cast the `order_id` and `customer_id` fields of the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` model from number types to strings using the following code: ```sql select cast(order_id as string) as order_id, cast(customer_id as string) as customer_id, order_date, status from {{ ref('orders') }} ``` After running this query, the `orders` table will look a little something like this: | order\_id | customer\_id | order\_date | status | | --------- | ------------ | ----------- | --------- | | 1 | 1 | 2018-01-01 | returned | | 2 | 3 | 2018-01-02 | completed | | 3 | 94 | 2018-01-04 | completed | Let’s be clear: the resulting data from this query looks exactly the same as the upstream `orders` model. However, the `order_id` and `customer_id` fields are now strings, meaning you could easily concat different string variables to them. > Casting columns to their appropriate types typically happens in our dbt project’s [staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md). A few reasons for that: data cleanup and standardization, such as aliasing, casting, and lower or upper casing, should ideally happen in staging models to create downstream uniformity and improve downstream performance. #### SQL CAST function syntax in Snowflake, Databricks, BigQuery, and Redshift Google BigQuery, Amazon Redshift, Snowflake, Postgres, and Databricks all support the ability to cast columns and data to different types. In addition, the syntax to cast is the same across all of them using the CAST function. You may also see the CAST function replaced with a double colon (::), followed by the data type to convert to; `cast(order_id as string)` is the same thing as `order_id::string` in most data warehouses. #### CAST function use cases You know at one point you’re going to need to cast a column to a different data type. But what are the scenarios folks run into that call for these conversions? At their core, these conversions need to happen because raw source data doesn’t match the analytics or business use case. This typically happens for a few reasons: * Differences in needs or miscommunication from [backend developers](https://docs.getdbt.com/blog/when-backend-devs-spark-joy#signs-the-data-is-sparking-joy) * ETL tools [defaulting to certain data types](https://airbytehq.github.io/integrations/sources/google-sheets/) * BI tools require certain fields to be specific data types A key thing to remember when you’re casting data is the user experience in your end BI tool: are business users expecting `customer_id` to be filtered on 1 or '1'? What is more intuitive for them? If one `id` field is an integer, all `id` fields should be integers. Just like all data modeling, consistency and standardization is key when determining when and what to cast. --- ### SQL Comments SQL comments…a two-folded thing: Are we talking about comments *inline* in SQL? Or comments on a table or view in the database? Why not both!? In this page, we’ll unpack how to create both inline and database object-level comments, general best practices around SQL comments, and how dbt can help you improve (and version-control) your comments. #### How to create SQL comments Inline SQL comments will begin with two dashes (--) in front of them in a query or dbt model; any text following these dashes is therefore what you’d call “commented out.” For longer, multi-line comments, you’ll typically see this syntax `/* your multi-line comment here */` used. ##### SQL comment example ```sql /* these lines form a multi-line SQL comment; if it’s uncommented, it will make this query error out */ select customer_id, -- order_id, this row is commented out order_date from {{ ref ('orders') }} ``` In practice, you’ll likely see SQL comments at the beginning of complex code logic, to help future developers or even advanced business users understand what specific blocks of code are accomplishing. Other times, you’ll see comments like the code above, that are commenting out lines no longer needed (or in existence) for that query or model. We’ll dive more into best practices around inline comments later on this page. For comments *on* database objects, such as views and tables, there’s a different syntax to add these explicit comments: ```sql comment on [database object type] <database object name> is 'comment text here'; ``` These database object-level comments are more useful for adding additional context or metadata to these objects versus inline comments being useful for explaining code functionality. Alternatively, these table and view-level comments can be easily abstracted out and version-controlled using [model descriptions in dbt](https://docs.getdbt.com/reference/resource-properties/description.md) and persisted in the objects using the [persist\_docs config](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) in dbt. #### SQL comments in Snowflake, Databricks, BigQuery, and Redshift Google BigQuery, Amazon Redshift, Snowflake, and Databricks all support the ability to add inline SQL comments. With the exception of BigQuery, these data warehouses also support native database object-level comments; BigQuery does, however, support native field-level descriptions. #### SQL commenting best practices In general, inline SQL comments should be used thoughtfully; another analytics engineer should be able to pair your comments with your code to clearly understand model functionality. We recommend leveraging inline comments in the following situations: * Explain complex code logic that if you had to scratch your head at, someone else will have to scratch their head at * Explain niche, unique-to-your-business logic * Separate out field types (ex. Ids, booleans, strings, dates, numerics, and timestamps) in [staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) to create more readable, organized, and formulaic models * Clearly label tech debt (`-- [TODO]: TECH DEBT`) in queries or models If you find your inline SQL comments are getting out of control, less scannable and readable, that’s a sign to lean more heavily on dbt Docs and markdown files in your dbt project. dbt supports [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md), which allow you to add robust model (or macro, source, snapshot, seed, and source) and column descriptions that will populate in hosted dbt Docs. For models or columns that need more thorough or customizable documentation, leverage [doc blocks in markdown and YAML files](https://docs.getdbt.com/reference/resource-properties/description.md#use-a-docs-block-in-a-description) to create more detailed explanations and comments. --- ### SQL CONCAT There is no better or simpler way to join multiple string values in a query than by using the CONCAT function. Full stop. It’s a straightforward function with pretty straightforward use cases. Use this page to understand how to use the CONCAT function in your data warehouse and why analytics engineers use it throughout their dbt models. #### How to use the CONCAT function Using the CONCAT function is pretty straightforward: you’ll pass in the strings or binary values you want to join together in the correct order into the CONCAT function. You can pass in as many expressions into the CONCAT function as you would like. ##### CONCAT function example ```sql select user_id, first_name, last_name, concat(first_name, ' ', last_name) as full_name from {{ ref('customers') }} limit 3 ``` This query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `customers` table will return results like this with a new column of the combined `first_name` and `last_name` field with a space between them: | user\_id | first\_name | last\_name | full\_name | | -------- | ----------- | ---------- | ----------- | | 1 | Michael | P. | Michael P. | | 2 | Shawn | M. | Shawn M. | | 3 | Kathleen | P. | Kathleen P. | #### CONCAT function syntax in Snowflake, Databricks, BigQuery, and Redshift Snowflake, Databricks, Google BigQuery, and Amazon Redshift all support the CONCAT function with the syntax looking the same in each platform. You may additionally see the CONCAT function represented by the `||` operator (ex. `select first_name || last_name AS full_name from {{ ref('customers') }}`) which has the same functionality as the CONCAT function in these data platforms. #### CONCAT use cases We most commonly see concatenation in SQL for strings to: * Join together address/geo columns into one field * Add hard-coded string values to columns to create clearer column values * Create surrogate keys using a hashing method and multiple column values (ex. `md5(column_1 || column_2) as unique_id` This isn’t an extensive list of where your team may be using CONCAT throughout your data work, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL COUNT COUNT is a SQL function you need to know how to use. Whether it’s in an ad hoc query, a data model, or in a BI tool calculation, you’ll be using the SQL COUNT function countless times (pun intended) in your data work. To formalize it, COUNT is an aggregate function that is used to return the count of rows of a specified field (`count(<field_name>)`) or all rows in a dataset (`count(*)`). It is commonly used to get baseline statistical information of a dataset, help ensure primary keys are unique, and calculate business metrics. #### How to use SQL COUNT in a query Use the following syntax to generate the aggregate count of a field: `count(<field_name>)` Since COUNT is an aggregate function, you’ll need a GROUP BY statement in your query if you’re looking at counts broken out by dimension(s). If you’re calculating the standalone counts of fields without the need to break them down by another field, you don’t need a GROUP BY statement. Let’s take a look at a practical example using COUNT, DISTINCT, and GROUP BY below. ##### COUNT example The following example is querying from a sample dataset created by dbt Labs called [jaffle\_shop](https://github.com/dbt-labs/jaffle_shop): ```sql select date_part('month', order_date) as order_month, count(order_id) as count_all_orders, count(distinct(customer_id)) as count_distinct_customers from {{ ref('orders') }} group by 1 ``` This simple query is something you may do while doing initial exploration of your data; it will return the count of `order_ids` and count of distinct `customer_ids` per order month that appear in the Jaffle Shop’s `orders` table: | order\_month | count\_all\_orders | count\_distinct\_customers | | ------------ | ------------------ | -------------------------- | | 1 | 29 | 24 | | 2 | 27 | 25 | | 3 | 35 | 31 | | 4 | 8 | 8 | An analyst or analytics engineer may want to perform a query like this to understand the ratio of orders to customers and see how it changes seasonally. #### SQL COUNT syntax in Snowflake, Databricks, BigQuery, and Redshift All modern data warehouses support the ability to use the COUNT function (and follow the same syntax!). Some data warehouses, such as Snowflake and Google BigQuery, additionally support a COUNT\_IF/COUNTIF function that allows you to pass in a boolean expression to determine whether to count a row or not. #### COUNT use cases We most commonly see queries using COUNT to: * Perform initial data exploration on a dataset to understand dataset volume, primary key uniqueness, distribution of column values, and more. * Calculate the counts of key business metrics (daily orders, customers created, etc.) in your data models or BI tool. * Define [metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) to aggregate key metrics. This isn’t an extensive list of where your team may be using COUNT throughout your development work, dbt models, and BI tool logic, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL CROSS JOIN A truly rarely seen, but important join: the cross join. The majority of your analytics engineering work will require you to join tables together to create robust, wide tables that will eventually be exposed to end business users. These models will usually be created using mostly [left](https://docs.getdbt.com/sql-reference/left-join.md) (and some [inner](https://docs.getdbt.com/sql-reference/inner-join.md)) joins. A cross join, on the other hand, typically takes two columns between two database objects and creates a table forming a combination of all rows across joined tables, called a cartesian product. Use this page to understand how to use cross joins and where you might leverage them in your dbt project. #### How to create a cross join Unlike regular joins, cross joins don’t use keys to join database objects together: ```text select <fields> from <table_1> as t1 cross join <table_2> as t2 ``` Cross joins are one of those SQL concepts that is easier to understand with a tangible example, so let’s jump into it. ##### SQL cross join example Table A `date_spine` | date | | ---------- | | 2022-01-01 | | 2022-01-02 | | 2022-01-03 | Table B `users` | user\_id | | -------- | | 1 | | 3 | | 4 | ```sql select users.user_id as user_id, date.date as date from {{ ref('users') }} as users cross join {{ ref('date_spine') }} as date order by 1 ``` This simple query will return a cartesian cross of all users and dates, essentially creating a unique combination of user per date per row: | user\_id | type | | -------- | ---------- | | 1 | 2022-01-01 | | 1 | 2022-01-02 | | 1 | 2022-01-03 | | 2 | 2022-01-01 | | 2 | 2022-01-02 | | 2 | 2022-01-03 | | 3 | 2022-01-01 | | 3 | 2022-01-02 | | 3 | 2022-01-03 | Generate surrogate keys from cross joins In the generated table above, the unique key is a combination of the `user_id` and `date` per row. To add a primary key to this table, you could generate a surrogate key using an MD5 hash the `generate_surrogate_key` macro in dbt-utils (ex. `{{ dbt_utils.generate_surrogate_key(user_id, type) }}` that could eventually be joined onto other tables. #### SQL cross join use case When would the generated table above be useful? Cross joining unique dates and users can be an effective way to create a base table to join various event counts, such as key website, email, or product events, to. These report-type tables are useful to expose to end business users in BI tools to look at aggregate counts per day per user and other useful measures. --- ### SQL Data Types Below, we’ll unpack the different umbrellas of data types and the unique data types that fall under each category. #### Numeric data types There are many different numeric types in SQL and that makes sense because…we’re data people and numbers are important, bit length is important, decimal places are even more important, and numbers are ultimately what allow stakeholders to make certain decisions. There’s slight differentiation in which numeric data types are supported across each data warehouse, but fundamentally, it’s most important to understand the differences between integers, decimals, and floats. | **Type** | **Definition** | **Use cases** | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Integer | Integers are numbers without fractions. Think 1, 2, 72384191203—nice, clean numbers. | Though many column values may look like integers (and in theory, they are), they’re often reflected or cast as decimal/numeric types to offer future precision and scale if required. | | Decimal | Decimal, also known as the NUMERIC type, is a numeric data type that has a default precision of 38 and a scale of 0. | Typical numeric columns in datasets, such as lifetime value or user ids. Most likely the most common form of numeric data in your tables. | | Float | Floats are used to provide approximate numeric values of fractions, with a precision of up to 64 bits. Floats offer a larger range of values compared to decimals. | Columns that are percentages; longitude/latitude. | #### String data types Strings are everywhere in data—they allow folks to have descriptive text field columns, use regex in their data work, and honestly, they just make the data world go ‘round. To formalize it, a string type is a word, or the combination of characters that you’ll typically see encased in single quotes (ex. 'Jaffle Shop', '1234 Shire Lane', 'Plan A'). Snowflake, Databricks, Google BigQuery, and Amazon Redshift all support the string data type. They may have slightly varying sub-types for strings; some data warehouses such as Snowflake and Redshift support `text`, `char`, and `character` string types which typically differ in byte length in comparison to the generic string type. Again, since most string type columns are inherent in your data, you’ll likely be ok using generic varchar or strings for casting, but it never hurts to read up on the docs specific to your data warehouse string support! #### Date data types Dates, timestamps, timezones—all the fun (slightly painful) data things that make analytics engineers real data practitioners (people who occasionally want to yank their hair out). Below, we’ll unpack dates, datetimes, times, and timestamps, to help you better understand the core date data types. Working our way from simplest to most complex, dates, typically represented with the DATE type are what you typically associate with a calendar date (ex. 2022-12-16), and are limited to the range of 0001-01-01 to 9999-12-31. DATETIME values contain both calendar date and time (ex. 2022-12-16 02:33:24) and may additionally include the sub-seconds. TIME types are typically represented as the HH:MM:SS of a time and don’t contain a specified timezone. TIMESTAMP data types allow for the greatest specification and precision of a point in time and can be specified with or without a timezone. Most event-driven data fields (ex. Order completed time, account created time, user churned time) will be represented as timestamps in your data sources. Some data warehouses, such as [Amazon Redshift](https://docs.amazonaws.cn/en_us/redshift/latest/dg/r_Datetime_types.html) and [Snowflake](https://docs.snowflake.com/en/sql-reference/data-types-datetime.html#date-time-data-types), support different timestamp options that allow for explicit specification of a timezone (or lack thereof). In general, the two best practices when it comes to dates and times are: 1. Keep (or convert) timestamps to the same timezone. 2. Keep date types in the most specific date-type as possible: you can always zoom out of a timestamp to get a date, but can’t get a timestamp from a date. You’ll ultimately leverage handy date functions to zoom in and out of dates, convert dates, or add times to dates. #### Booleans A boolean is a column value that is either true, false, or null. In your datasets, you’ll use booleans to create `is_` or `has_` fields to create clear segments in your data; for example, you may use booleans to indicate whether a customer has churned (`has_churned`) or denote employee records (`is_employee`), or filter out records that have been removed from your source data (`is_deleted`). Typically, you’ll see `True` or `False` as the actual boolean values in a column, but may also choose to use numeric values, such as 1 and 0, to represent true and false values. The strings of `True` and `False`, however, tend to be a bit easier to read and interpret for end business users. #### Semi-structured data types Semi-structured data types are a great way to combine or aggregate data across multiple fields; you may also find yourself in the inverse situation where you need to unpack semi-structured data, such as a JSON object, and unnest it into its individual key-value pair. The two primary semi-structured data types you’ll see across data warehouses are JSON and arrays. Below, we’ll unpack what the difference is between the two and provide an example of each one. | **Type** | **Definition** | **Example** | **Use case** | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | JSON | When looking at data formatted in JSON, we say that the data is stored in JSON objects. These are composed of key-value pairs. JSON objects are enclosed in curly brackets ({ }) and each key-value pair is separated by a comma. Read more about using JSON here. | {"customer\_id":2947, "order\_id":4923, "order\_items":"cheesecake"} | One of the great things about JSON data is that it doesn't require schema definition—until you unnest it. Extract exactly what you need from your JSON object, and you can forget about the rest! JSON values will often come inherent in your data sources, so learn how to unnest them and your life will become easier. | | Array | Similar to arrays in other programming languages, an array contains multiple elements that are accessible via its position in that array. | \["cheesecake", "cupcake", "brownie"] | Arrays are a clear way to aggregate multiple values together to create a singular value. Many use cases here, but be cautious: using aggregate functions, such as `array_agg` , can become inefficient on large datasets. | --- ### SQL DATE_PART In this post, we’re going to give a deep dive into the DATE\_PART function, how it works, and why we use it. The DATE\_PART function allows you to extract a specified date part from a date/time. For example, if you were to extract the month from the date February 14, 2022, it would return 2 since February is the second month in the year. #### How to use the DATE\_PART function Like most other SQL functions, you need to pass in arguments; for the DATE\_PART function, you’ll pass in a date/timestamp/date field that you want to extract a date part from and specify the part you want removed. You can extract the numeric month, date, year, hour, seconds, etc. from a timestamp or date field) using the DATE\_PART function using the following syntax: `date_part(<date/time part>, <date field/date>)` Let’s take a look at a practical example below. ##### DATE\_PART function example ```sql select date_part('month', order_date) as order_month, round(avg(amount)) as avg_order_amount from {{ ref('orders') }} group by 1 ``` This query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return the rounded order amount per each order month (represented as a numeric value): | order\_month | avg\_order\_amount | | ------------ | ------------------ | | 1 | 17 | | 2 | 15 | | 3 | 18 | | 4 | 17 | Unlike the DATE\_TRUNC function that actually truncates a date to its first instance of a given date part (so it maintains a date structure), the DATE\_PART function returns a numeric value from a date field. You may commonly see the DATE\_PART function replaced with an EXTRACT function, which performs the same functionality. #### DATE\_PART function syntax in Snowflake, Databricks, BigQuery, and Redshift | Data warehouse | DATE\_PART support? | Notes | | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Snowflake | ✅ | | | Databricks | ✅ | | | Amazon Redshift | ✅ | | | Google BigQuery | ❌ | BigQuery supports the EXTRACT function which performs the same functionality as the DATE\_PART function | | Postgres | ✅ | This is overly pedantic and you’ll likely never encounter an issue with DATE\_PART and EXTRACT evaluating to differences in values that truly matter, but it’s worth noting. Postgres’ DATE\_PART and EXTRACT functions would previously evaluate to the same output. However, with Postgres 14, the EXTRACT function now returns a numeric type instead of an 8-byte float. | #### DATE\_PART function use cases We most commonly see the DATE\_PART or EXTRACT function used in data work to analyze: * Fiscal calendars: If your business uses fiscal years, or calendars that differ from the normal 12-month cycle, DATE\_PART functions can help create alignment between fiscal calendars and normal calendars * Ad hoc analysis: The DATE\_PART function are useful in ad hoc analyses and queries when you need to look at values grouped by date periods or for period comparisons This isn’t an extensive list of where your team may be using the DATE\_PART function throughout your dbt models and BI tool logic, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL DATE_TRUNC In general, data people prefer the more granular over the less granular. [Timestamps > dates](https://docs.getdbt.com/blog/when-backend-devs-spark-joy#signs-the-data-is-sparking-joy), daily data > weekly data, etc.; having data at a more granular level always allows you to zoom in. However, you’re likely looking at your data at a somewhat zoomed-out level—weekly, monthly, or even yearly. To do that, you’re going to need a handy dandy function that helps you round out date or time fields. The DATE\_TRUNC function will truncate a date or time to the first instance of a given date part. Wordy, wordy, wordy! What does this really mean? If you were to truncate `2021-12-13` out to its month, it would return `2021-12-01` (the first day of the month). Using the DATE\_TRUNC function, you can truncate to the weeks, months, years, or other date parts for a date or time field. This can make date/time fields easier to read, as well as help perform cleaner time-based analyses. Overall, it’s a great function to use to help you aggregate your data into specific date parts while keeping a date format. However, the DATE\_TRUNC function isn’t your swiss army knife–it’s not able to do magic or solve all of your problems (we’re looking at you [star](https://getdbt.com/sql-foundations/star-sql-love-letter/)). Instead, DATE\_TRUNC is your standard kitchen knife—it’s simple and efficient, and you almost never start cooking (data modeling) without it. #### How to use the DATE\_TRUNC function​ For the DATE\_TRUNC function, there are two arguments you must pass in: * The date part: This is the days/months/weeks/years (level) you want your field to be truncated out to * The date/time you want to be truncated The DATE\_TRUNC function can be used in [SELECT](https://docs.getdbt.com/sql-reference/select.md) statements and [WHERE](https://docs.getdbt.com/sql-reference/where.md) clauses. Most, if not all, modern cloud data warehouses support some type of the DATE\_TRUNC function. There may be some minor differences between the argument order for DATE\_TRUNC across data warehouses, but the functionality very much remains the same. Below, we’ll outline some of the slight differences in the implementation between some of the data warehouses. #### The DATE\_TRUNC function in Snowflake and Databricks​ In [Snowflake](https://docs.snowflake.com/en/sql-reference/functions/date_trunc.html) and [Databricks](https://docs.databricks.com/sql/language-manual/functions/date_trunc.html), you can use the DATE\_TRUNC function using the following syntax: ```sql date_trunc(<date_part>, <date/time field>) ``` In these platforms, the \<date\_part> is passed in as the first argument in the DATE\_TRUNC function. #### The DATE\_TRUNC function in Google BigQuery and Amazon Redshift​ In [Google BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date_trunc) and [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_DATE_TRUNC.html), the `<date_part>` is passed in as the first argument and the `<date/time field>` is the second argument: ```sql date_trunc(<date/time field>, <date part>) ``` A note on BigQuery: BigQuery’s DATE\_TRUNC function supports the truncation of date types, whereas Snowflake, Redshift, and Databricks’ `<date/time field>` can be a date or timestamp data type. BigQuery also supports DATETIME\_TRUNC and TIMESTAMP\_TRUNC functions to support truncation of more granular date/time types. #### A dbt macro to remember​ Why Snowflake, Amazon Redshift, Databricks, and Google BigQuery decided to use different implementations of essentially the same function is beyond us and it’s not worth the headache trying to figure that out. Instead of remembering if the `<date_part>` or the `<date/time field>` comes first, (which, let’s be honest, we can literally never remember) you can rely on a dbt macro to help you get away from finicky syntax. [Adapters](https://docs.getdbt.com/docs/supported-data-platforms.md) support [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md) to help you write certain functions, like DATE\_TRUNC and DATEDIFF, without having to memorize sticky function syntax. Using the [Jaffle Shop](https://github.com/dbt-labs/jaffle_shop/blob/main/models/orders.sql), a simple dataset and dbt project, you can truncate the `order_date` from the orders table using the [dbt DATE\_TRUNC macro](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md#date_trunc): ```sql select order_id, order_date, {{ date_trunc("week", "order_date") }} as order_week, {{ date_trunc("month", "order_date") }} as order_month, {{ date_trunc("year", "order_date") }} as order_year from {{ ref('orders') }} ``` Running the above would product the following sample results: | **order\_id** | **order\_date** | **order\_week** | **order\_month** | **order\_year** | | ------------- | --------------- | --------------- | ---------------- | --------------- | | 1 | 2018-01-01 | 2018-01-01 | 2018-01-01 | 2018-01-01 | | 70 | 2018-03-12 | 2018-03-12 | 2018-03-01 | 2018-01-01 | | 91 | 2018-03-31 | 2018-03-26 | 2018-03-01 | 2018-01-01 | The `order_week`, `order_month`, and `order_yea`r fields are the truncated values from the `order_date` field. --- ### SQL DATEADD If you’ve ever used the DATEADD SQL function across dialects (such as BigQuery, Postgres and Snowflake), you’ve probably had to google the syntax of the function every time. It's almost impossible to remember the argument order (or exact function name) of dateadd. This article will go over how the DATEADD function works, the nuances of using it across the major cloud warehouses, and how to standardize the syntax variances using dbt macro. #### What is the DATEADD SQL function? The DATEADD function in SQL adds a time/date interval to a date and then returns the date. This allows you to add or subtract a certain period of time from a given start date. Sounds simple enough, but this function lets you do some pretty useful things like calculating an estimated shipment date based on the ordered date. #### Differences in DATEADD syntax across data warehouse platforms All of them accept the same rough parameters, in slightly different syntax and order: * Start / from date * Datepart (day, week, month, year) * Interval (integer to increment by) The *functions themselves* are named slightly differently, which is common across SQL dialects. ##### For example, the DATEADD function in Snowflake… ```text dateadd( {{ datepart }}, {{ interval }}, {{ from_date }} ) ``` *Hour, minute and second are supported!* ##### The DATEADD function in Databricks ```sql date_add( {{ startDate }}, {{ numDays }} ) ``` ##### The DATEADD function in BigQuery… ```sql date_add( {{ from_date }}, INTERVAL {{ interval }} {{ datepart }} ) ``` *Dateparts of less than a day (hour / minute / second) are not supported.* ##### The DATEADD function in Postgres… Postgres doesn’t provide a dateadd function out of the box, so you’ve got to go it alone - but the syntax looks very similar to BigQuery’s function… ```sql {{ from_date }} + (interval '{{ interval }} {{ datepart }}') ``` Switching back and forth between those SQL syntaxes usually requires a quick scan through the warehouse’s docs to get back on the horse. #### Standardizing your DATEADD SQL syntax with a dbt macro But couldn’t we be doing something better with those keystrokes, like typing out and then deleting a tweet? dbt helps smooth out these wrinkles of writing [SQL across data warehouses](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md). Instead of looking up the syntax each time you use it, you can just write it the same way each time, and the macro compiles it to run on your chosen warehouse: ```text {{ dateadd(datepart, interval, from_date_or_timestamp) }} ``` Adding 1 month to a specific date would look like… ```text {{ dateadd(datepart="month", interval=1, from_date_or_timestamp="'2021-08-12'") }} ``` --- ### SQL DATEDIFF *“How long has it been since this customer last ordered with us?”*<br />*“What is the average number of days to conversion?”* Business users will have these questions, data people will have to answer these questions, and the only way to solve them is by calculating the time between two different dates. Luckily, there’s a handy DATEDIFF function that can do that for you. The DATEDIFF function will return the difference in specified units (ex. days, weeks, years) between a start date/time and an end date/time. It’s a simple and widely used function that you’ll find yourself using more often than you expect. DATEDIFF is a little bit like your favorite pair of socks; you’ll usually find the first one easily and feel like the day is going to be great. But for some reason, the matching sock requires a little digging in the drawer. DATEDIFF is this pair of socks—you’ll inevitably find yourself Googling the syntax almost every time you use it, but you can’t go through your day without using it. This page will go over how to use the DATEDIFF function across different data warehouses and how to write more standardized DATEDIFF functions using a dbt macro (or successfully find your socks as a pair in one go). #### How to use the DATEDIFF function​ For the DATEDIFF function, there are three elements, or arguments, passed in: * The date part: This is the days/months/weeks/years (unit) of the difference calculated * The first (start) date/time * The second (end) date/time The DATEDIFF function can be used in [SELECT](https://docs.getdbt.com/sql-reference/select.md) statements and WHERE clauses. Most, if not all, modern cloud data warehouses support some type of the DATEDIFF function. There may be some minor differences between the argument order and function name for DATEDIFF across data warehouses, but the functionality very much remains the same. Below, we’ll outline some of the slight differences in the implementation between some data warehouses. #### SQL DATEDIFF function syntax in Snowflake, Databricks, and Redshift The syntax for using the DATEDIFF function in Snowflake and Amazon Redshift, and Databricks looks like the following: ```sql datediff(<date part>, <start date/time>, <end date/time>) ``` A note on Databricks: Databricks additionally supports a separate [DATEDIFF function](https://docs.databricks.com/sql/language-manual/functions/datediff.html) that takes only two arguments: a start date and an end date. The function will always return the difference between two dates in days. ##### DATEDIFF in Google BigQuery​ The syntax for using the DATEDIFF function in [Google BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#datetime_diff) looks like the following: * Unlike in Snowflake, Amazon Redshift, and Databricks where the `<date part>` is passed as the first argument, the `<date part>` is passed in as the last argument in Google BigQuery. * Google BigQuery also calls the function DATETIME\_DIFF with an additional underscore separating the function name. This is on par with [Google BigQuery’s preference to have underscores in function names](https://cloud.google.com/bigquery/docs/reference/standard-sql/date_functions). * The DATETIME\_DIFF arguments are datetimes, not dates; Snowflake, Redshift, and Databricks’ DATEDIFF functions support multiple [date types](https://docs.getdbt.com/sql-reference/data-types.md#date-data-types) such as dates and timestamps. BigQuery also supports a separate [DATE\_DIFF function](https://cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#date_diff) that will return the difference between two date types, unlike the DATETIME\_DIFF that only supports the datetime type. #### A hero in the shadows: The DATEDIFF dbt macro!​ You may be able to memorize the syntax for the DATEDIFF function for the primary data warehouse you use. What happens when you switch to a different one for a new job or a new data stack? Remembering if there’s an underscore in the function name or which argument the `<date part>` is passed in as is… no fun and leads to the inevitable, countless “datediff in bigquery” Google searches. Luckily, [dbt](https://github.com/dbt-labs/dbt) has your back! dbt is the product that helps data folks write their [data transformations](https://www.getdbt.com/analytics-engineering/transformation/) following software engineering best practices. [Adapters](https://docs.getdbt.com/docs/supported-data-platforms.md) support [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md) to help you write certain functions, like DATE\_TRUNC and DATEDIFF, without having to memorize sticky function syntax. Using the DATEDIFF macro, you can calculate the difference between two dates without having to worry about finicky syntax. Specifically, this means you could successfully run the same code across multiple databases without having to worry about the finicky differences in syntax. Using the [jaffle shop](https://github.com/dbt-labs/jaffle_shop/blob/main/models/orders.sql), a simple dataset and dbt project, we can calculate the difference between two dates using the dbt DATEDIFF macro: ```sql select *, {{ datediff("order_date", "'2022-06-09'", "day") }} from {{ ref('orders') }} ``` This would return all fields from the orders table and the difference in days between order dates and June 9, 2022. Under the hood, this macro is taking your inputs and creating the appropriate SQL syntax for the DATEDIFF function *specific to your data warehouse*. --- ### SQL DISTINCT Let’s just put it out there: at one point in your data work, you’ll encounter duplicates in your data. They may be introduced from a faulty data source or created during the joining and transforming of data. You may need a more sophisticated or refactored solution for the latter scenario, but it never hurts to know how to use DISTINCT in a query. Using DISTINCT in a SELECT statement will force a query to only return non-duplicate rows. You may commonly see a DISTINCT clause in COUNT functions to get counts of distinct rows. #### How to use SQL DISTINCT in a query To remove duplicate rows from a query, you add DISTINCT immediately after SELECT followed by the rows you want to be selected: ```sql select distinct row_1, row_2 from my_data_source ``` Let’s take a look at a practical example using DISTINCT below. ##### SQL DISTINCT example ```sql select count(customer_id) as cnt_all_orders, count(distinct customer_id) as cnt_distinct_customers from {{ ref('orders') }} ``` This simple query is something you may do while doing initial exploration of your data; it will return the count of `customer_ids` and count of distinct `customer_ids` that appear in the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table: | cnt\_all\_orders | cnt\_distinct\_customers | | ---------------- | ------------------------ | | 99 | 62 | As you can see from the query results, there are 99 orders placed by customers, but only 62 distinct customers in the table. #### DISTINCT syntax in Snowflake, Databricks, BigQuery, and Redshift Since it’s a pillar of SQL, all modern data warehouses support the ability to use DISTINCT in a SELECT statement 😀 #### DISTINCT use cases You’ll most commonly see queries using a DISTINCT statement to: * Remove unnecessary duplicate rows from a data model; a word of caution on this: if you need to use DISTINCT in a downstream, non-source model that contains joins, there’s a chance that there could be faulty logic producing duplicates in the data, so always double-check that they are true duplicates. * Find the counts of distinct fields in a dataset, especially for primary or surrogate keys. This isn’t an extensive list of where your team may be using DISTINCT throughout your development work, dbt models, and BI tool logic, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL FROM What makes the analytics world go ‘round? Queries and bad graphs. (Since we’re here to keep it brief, we won’t go into the latter here 😉) The first thing someone learns in SQL: how to build a query using [SELECT](https://docs.getdbt.com/sql-reference/select.md) and FROM statements. The SQL FROM statement is the fundamental building block of any query: it allows you to identify the database schema object (table/view) you want to select data from in a query. In a dbt project, a SQL dbt model is technically a singular SELECT statement (often built leveraging CTEs or subqueries) using a [reference](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) to an upstream data model or table in a FROM statement. #### How to use SQL FROM statements Any query begins with a simple SELECT statement and wrapped up with a FROM statement: ```sql select order_id, --select your columns customer_id, order_date from {{ ref('orders') }} --the table/view/model you want to select from limit 3 ``` Woah woah woah! That is not the typical FROM statement you’re probably used to seeing! Most FROM statements in the non-dbt world, such as when you’re running ad-hoc queries directly in your data warehouse, will follow the `FROM database.schema.table_name` syntax. In dbt projects, analytics engineers leverage [the ref statement](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) to refer to other data models and sources to automatically build a dependency graph and avoid having to hard-code schema names. This flexibility is valuable as analytics engineers develop in their own development environments (schemas) without having to rename tables in their FROM statements. This basic query is selecting three columns from the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop/blob/main/models/orders.sql) `orders` table and returning three rows. If you execute this query in your data warehouse, it will return a result looking like this: | **order\_id** | **customer\_id** | **order\_date** | | ------------- | ---------------- | --------------- | | 1 | 1 | 2018-01-01 | | 2 | 3 | 2018-01-02 | | 3 | 95 | 2018-01-04 | In the query above, dbt automatically compiles the `from {{ ref('orders') }}` to `from analytics.jaffle_shop.orders` when the query is sent down to the data warehouse and run in the production environment. If you’re selecting from multiple tables or models, that’s where you’d rely on unions or joins to bring multiple tables together in a way that makes sense to your data. #### FROM statement syntax in Snowflake, Databricks, BigQuery, and Redshift Just as the humble SELECT statement is a SQL fundamental that goes untampered by the data warehouses, FROM syntax does not vary within them. As a result, writing the actual `select…from` statement across Snowflake, Databricks, Google BigQuery, and Amazon Redshift would look the same. --- ### SQL GROUP BY GROUP BY…it’s a little hard to explicitly define in a way *that actually makes sense*, but it will inevitably show up countless times in analytics work and you’ll need it frequently. To put it in the simplest terms, the GROUP BY statement allows you to group query results by specified columns and is used in pair with aggregate functions such as [AVG](https://docs.getdbt.com/sql-reference/avg.md) and [SUM](https://docs.getdbt.com/sql-reference/sum.md) to calculate those values across specific rows. #### How to use the SQL GROUP BY statement The GROUP BY statement appears at the end of a query, after any joins and [WHERE](https://docs.getdbt.com/sql-reference/where.md) filters have been applied: ```sql select my_first_field, count(id) as cnt --or any other aggregate function (sum, avg, etc.) from my_table where my_first_field is not null group by 1 --grouped by my_first_field order by 1 desc ``` A few things to note about the GROUP BY implementation: * It’s usually listed as one of the last rows in a query, after any joins or where statements; typically you’ll only see [HAVING](https://docs.getdbt.com/sql-reference/having.md), [ORDER BY](https://docs.getdbt.com/sql-reference/order-by.md), or [LIMIT](https://docs.getdbt.com/sql-reference/limit.md) statements following it in a query * You can group by multiple fields (ex. `group by 1,2,3`) if you need to; in general, we recommend performing aggregations and joins in separate CTEs to avoid having to group by too many fields in one query or CTE * You may also group by explicit column name (ex. `group by my_first_field`) or even a manipulated column name that is in the query (ex. `group by date_trunc('month', order_date)`) Readability over DRYness? Grouping by explicit column name (versus column number in query) can be two folded: on one hand, it’s potentially more readable by end business users; on the other hand, if a grouped column name changes, that name change needs to be reflected in the group by statement. Use a grouping convention that works for you and your data, but try to keep to one standard style. ##### SQL GROUP BY example ```sql select customer_id, count(order_id) as num_orders from {{ ref('orders') }} group by 1 order by 1 limit 5 ``` This simple query using the sample dataset [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `order` table will return customers and the count of orders they’ve placed: | customer\_id | num\_orders | | ------------ | ----------- | | 1 | 2 | | 2 | 1 | | 3 | 3 | | 6 | 1 | | 7 | 1 | Note that the `order by` and `limit` statements are after the `group by` in the query. #### SQL GROUP BY syntax in Snowflake, Databricks, BigQuery, and Redshift Snowflake, Databricks, BigQuery, and Redshift all support the ability to group by columns and follow the same syntax. #### GROUP BY use cases Aggregates, aggregates, and did we mention, aggregates? GROUP BY statements are needed when you’re calculating aggregates (averages, sum, counts, etc.) by specific columns; your query will not run successfully without them if you’re attempting to use aggregate functions in your query. You may also see GROUP BY statements used to deduplicate rows or join aggregates onto other tables with CTEs; [this article provides a great writeup](https://www.getdbt.com/blog/write-better-sql-a-defense-of-group-by-1/) on specific areas you might see GROUP BYs used in your dbt projects and data modeling work. 👋Bye bye finicky group bys In some sticky data modeling scenarios, you may find yourself needing to group by many columns to collapse a table down into fewer rows or deduplicate rows. In that scenario, you may find yourself writing `group by 1, 2, 3,.....,n` which can become tedious, confusing, and difficult to troubleshoot. Instead, you can leverage a [dbt macro](https://github.com/dbt-labs/dbt-utils#group_by-source) that will save you from writing `group by 1,2,....,46` to instead a simple `{{ dbt_utils.group_by(46) }}`...you’ll thank us later 😉 --- ### SQL HAVING SQL HAVING is just one of those little things that are going to make your ad hoc data work a little easier. A not-so-fun fact about the [WHERE clause](https://docs.getdbt.com/sql-reference/where.md) is that you can’t filter on aggregates with it…that’s where HAVING comes in. With HAVING, you can not only define an aggregate in a [select](https://docs.getdbt.com/sql-reference/select.md) statement, but also filter on that newly created aggregate within the HAVING clause. This page will walk through how to use HAVING, when you should use it, and discuss data warehouse support for it. #### How to use the HAVING clause in SQL The HAVING clause essentially requires one thing: an aggregate field to evaluate. Since HAVING is technically a boolean, it will return rows that execute to true, similar to the WHERE clause. The HAVING condition is followed after a [GROUP BY statement](https://docs.getdbt.com/sql-reference/group-by.md) and optionally enclosed with an ORDER BY statement: ```sql select -- query from <table> group by <field(s)> having condition [optional order by] ``` That example syntax looks a little gibberish without some real fields, so let’s dive into a practical example using HAVING. ##### SQL HAVING example ##### HAVING example ```sql select customer_id, count(order_id) as num_orders from {{ ref('orders') }} group by 1 having num_orders > 1 --if you replace this with `where`, this query would not successfully run ``` ##### CTE example ```sql with counts as ( select customer_id, count(order_id) as num_orders from {{ ref('orders') }} group by 1 ) select customer_id, num_orders from counts where num_orders > 1 ``` This simple query using the sample dataset [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return customers who have had more than one order: | customer\_id | num\_orders | | ------------ | ----------- | | 1 | 2 | | 3 | 3 | | 94 | 2 | | 64 | 2 | | 54 | 4 | The query above using the CTE utilizes more lines compared to the simpler query using HAVING, but will produce the same result. #### SQL HAVING clause syntax in Snowflake, Databricks, BigQuery, and Redshift [Snowflake](https://docs.snowflake.com/en/sql-reference/constructs/having.html), [Databricks](https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-having.html), [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#having_clause), and [Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_HAVING_clause.html) all support the HAVING clause and the syntax for using HAVING is the same across each of those data warehouses. --- ### SQL ILIKE The favorite child ILIKE helps you easily match, find, and filter out string values of a specified pattern by using SQL wildcards *without having to worry about case sensitivity*. If you’re a stickler for case-sensitivity, don’t hesitate to use the not-as-special (but still important) child, the LIKE operator 😆 #### How to use the SQL ILIKE operator The ILIKE operator has a simple syntax, with the ability to have it utilized in WHERE clauses or case statements: `where <field_name> ilike '<pattern>'` or `case when <field_name> ilike '<pattern>'` Some notes on this operator’s syntax and functionality: * The `<pattern>` can use two SQL wildcards (`%` and `_`); the underscore will match any single character and the % matches zero or more characters * Ex. '%j' = any string that ends with the letter j * Ex. 'j%' = any string that starts with a letter j * Ex. 'j%l' = any string that starts with a the letter j and ends with a letter l * Ex. '\_j%' = any string that has a letter j in the second position * Majority of use cases for the ILIKE operator will likely involve the `%` wildcard * The ILIKE operator is case-insensitive, meaning that the casing in the `<pattern>` you want to filter does not need to match the same-case in your column values * The ILIKE operator can be paired with the NOT operator, to filter on rows that are not like a specified pattern Let’s dive into a practical example using the ILIKE operator now. ##### SQL ILIKE example ```sql select payment_id, order_id, payment_method, case when payment_method ilike '%card' then 'card_payment' else 'non_card_payment' end as was_card from {{ ref('payments') }} ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `payments` table is creating a new column to determine if a payment used a type of card (ex. debit card, credit card, gift card) payment based on if the `payment_method` value ends in `card`: | **payment\_id** | **order\_id** | **payment\_method** | **was\_card** | | --------------- | ------------- | ------------------- | ------------------ | | 1 | 1 | credit\_card | card\_payment | | 9 | 9 | gift\_card | card\_payment | | 3 | 3 | coupon | non\_card\_payment | | 4 | 4 | coupon | non\_card\_payment | #### ILIKE syntax in Snowflake, Databricks, BigQuery, and Redshift Most modern data warehouses, with the exception of Google BigQuery, support the ILIKE operator and the syntax is the same across them. Use the table below to read more on the documentation for the ILIKE operator in your data warehouse. | **Data warehouse** | **ILIKE support?** | | ------------------ | ---------------------------------------------------------------- | | Snowflake | ✅ | | Databricks | ✅ | | Amazon Redshift | ✅ | | Google BigQuery | ❌, recommend using regular expressions or the CONTAINS function | #### ILIKE operator example use cases The ILIKE operator has very similar use cases to the [LIKE operator](https://docs.getdbt.com/sql-reference/like.md), so we won’t repeat ourselves here. The important thing to understand when using the LIKE or ILIKE operators is what the casing variations look like in your data: if casing is inconsistent within a column, ILIKE will be your friend; if your backend engineers and analytics engineers rigorously follow a style-guide (and our source data is magically of the same case), the LIKE operator is there for you if you need it. --- ### SQL IN It happens to the best of data people: The `orders` table always needs to filter out `status = employee_order` in order to get the accurate order counts. So you’re data model for the `orders` table looks a little something like this: ```sql select * from {{ source('backend_db', 'orders') }} where status != 'employee_order' ``` What happens one day if there’s an additional `status` that needs to be filtered out? Well, that’s where the handy IN operator comes into play. The IN operator ultimately allows you to specify multiple values in a WHERE clause, so you can easily filter your query on multiple options. Using the IN operator is a more refined version of using multiple OR conditions in a WHERE clause. #### How to use SQL IN operator In the scenario above if you now needed to filter on an additional new `status` value to remove certain rows, your use of the IN operator would look like this: ```sql select * from {{ source('backend_db', 'orders') }} where status not in ('employee_order', 'influencer_order') --list of order statuses to filter out ``` Woah woah woah, what is a `not in`? This is exactly what it sounds like: return all rows where the status is not `employee_order` or `influencer_order`. If you wanted to just use the IN operator, you can specify all other statuses that are appropriate (ex. `where status in ('regular_order', 'temp_order')`). You can additionally use the IN/NOT IN operator for a subquery, to remove/include rows from a subquery’s result: ```sql where status in (select …) ``` Compare columns against appropriate data types The only “gotcha” that really exists in using the IN operator is remembering that the values in your IN list **must** match the data type of the column they’re compared against. This is especially important for boolean columns that could be accidentally cast as strings. #### IN operator syntax in Snowflake, Databricks, BigQuery, and Redshift The IN operator, like most of the SQL operators, are not syntactically different across data warehouses. That means the syntax for using the IN/NOT IN operator is the same in Snowflake, Databricks, Google BigQuery, and Amazon Redshift. #### IN operator use cases Use the IN condition to filter out inappropriate or inaccurate rows from a query or database schema object based on parameters you define and understand. We guarantee there’s an IN somewhere in your dbt project 😀 --- ### SQL INNER JOINS The cleanest and easiest of SQL joins: the humble inner join. Just as its name suggests, an inner join between two database objects returns all rows that have matching join keys; any keys that don’t match are omitted from the query result. #### How to create an inner join Like all joins, you need some database objects (ie tables/views), keys to join on, and a [select statement](https://docs.getdbt.com/sql-reference/select.md) to perform an inner join: ```text select <fields> from <table_1> as t1 inner join <table_2> as t2 on t1.id = t2.id ``` In this example above, there’s only one field from each table being used to join the two together; if you’re joining between two database objects that require multiple fields, you can leverage AND/OR operators, and more preferably, surrogate keys. You may additionally add [WHERE](https://docs.getdbt.com/sql-reference/where.md), [GROUP BY](https://docs.getdbt.com/sql-reference/group-by.md), [ORDER BY](https://docs.getdbt.com/sql-reference/order-by.md), [HAVING](https://docs.getdbt.com/sql-reference/having.md), and other clauses after your joins to create filtering, ordering, and performing aggregations. As with any query, you can perform as many joins as you want in a singular query. A general word of advice: try to keep data models modular by performing regular DAG audits. If you join certain tables further upstream, are those individual tables needed again further downstream? If your query involves multiple joins and complex logic and is exposed to end business users, ensure that you leverage table or [incremental materializations](https://docs.getdbt.com/docs/build/incremental-models.md). ##### SQL inner join example Table A `car_type` | user\_id | car\_type | | -------- | --------- | | 1 | van | | 2 | sedan | | 3 | truck | Table B `car_color` | user\_id | car\_color | | -------- | ---------- | | 1 | red | | 3 | green | | 4 | yellow | ```sql select car_type.user_id as user_id, car_type.car_type as type, car_color.car_color as color from {{ ref('car_type') }} as car_type inner join {{ ref('car_color') }} as car_color on car_type.user_id = car_color.user_id ``` This simple query will return all rows that have the same `user_id` in both Table A and Table B: | user\_id | type | color | | -------- | ----- | ----- | | 1 | van | red | | 3 | truck | green | Because there’s no `user_id` = 4 in Table A and no `user_id` = 2 in Table B, rows with ids 2 and 4 (from either table) are omitted from the inner join query results. #### SQL inner join use cases There are probably countless scenarios where you’d want to inner join multiple tables together—perhaps you have some really nicely structured tables with the exact same primary keys that should really just be one larger, wider table or you’re joining two tables together don’t want any null or missing column values if you used a left or right join—it’s all pretty dependent on your source data and end use cases. Where you will not (and should not) see inner joins is in [staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) that are used to clean and prep raw source data for analytics uses. Any joins in your dbt projects should happen further downstream in [intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) and [mart models](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md) to improve modularity and DAG cleanliness. --- ### SQL LEFT JOIN An analytics engineer favorite: the left join. Without a doubt, this is probably the most regularly used join in any dbt project (and for good reason). The left join returns all rows in the [FROM statement](https://docs.getdbt.com/sql-reference/from.md), regardless of match in the left join database object. Compare this to an [inner join](https://docs.getdbt.com/sql-reference/inner-join.md), where only rows are returned that have successful key matches between the database object in the FROM statement and in the inner join statement. #### How to create a left join Like all joins, you need some database objects (ie tables/views), keys to join on, and a [select statement](https://docs.getdbt.com/sql-reference/select.md) to perform a left join: ```text select <fields> from <table_1> as t1 left join <table_2> as t2 on t1.id = t2.id ``` In this example above, there’s only one field from each table being used to join the two together; if you’re joining between two database objects that require multiple fields, you can leverage AND/OR operators, and more preferably, surrogate keys. You may additionally add [WHERE](https://docs.getdbt.com/sql-reference/where.md), [GROUP BY](https://docs.getdbt.com/sql-reference/group-by.md), [ORDER BY](https://docs.getdbt.com/sql-reference/order-by.md), [HAVING](https://docs.getdbt.com/sql-reference/having.md), and other clauses after your joins to create filtering, ordering, and performing aggregations. You may also left (or any join really) as many joins as you’d like in an individual query or CTE. ##### SQL left join example Table A `car_type` | **user\_id** | **car\_type** | | ------------ | ------------- | | 1 | van | | 2 | sedan | | 3 | truck | Table B `car_color` | user\_id | car\_color | | -------- | ---------- | | 1 | red | | 3 | green | | 4 | yellow | ```sql select car_type.user_id as user_id, car_type.car_type as type, car_color.car_color as color from {{ ref('car_type') }} as car_type left join {{ ref('car_color') }} as car_color on car_type.user_id = car_color.user_id ``` This simple query will return *all rows* from Table A and adds the `color` column to rows where there’s a successful match to Table B: | **user\_id** | **type** | **color** | | ------------ | -------- | --------- | | 1 | van | red | | 2 | sedan | null | | 3 | truck | green | Because there’s no `user_id` = 2 in Table B, there is no `color` available, thus a null result `color` column for `user_id` 2. #### SQL left join use cases Left joins are a fundamental in data modeling and analytics engineering work—they allow you to easily join database objects onto each other while maintaining an original table’s row count (in the from statement). Compared to right joins, that return all rows in a right join database object (and not the from statement), we find left joins a little more intuitive to understand and build off of. Ensure your joins are just ~~left~~ right Something to note if you use left joins: if there are multiple records for an individual key in the left join database object, be aware that duplicates can potentially be introduced in the final query result. This is where dbt tests, such as testing for primary key uniqueness and [equal row count](https://github.com/dbt-labs/dbt-utils#equal_rowcount-source) across upstream source tables and downstream child models, can help you identify faulty data modeling logic and improve data quality. Where you will not (and should not) see left joins is in [staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) that are used to clean and prep raw source data for analytics uses. Any joins in your dbt projects should happen further downstream in [intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) and [mart models](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md) to improve modularity and DAG cleanliness. --- ### SQL LIKE The LIKE operator helps you easily match, find, and filter out string values of a specified pattern by using SQL wildcards. Important to note that the pattern passed into the LIKE operator is case-sensitive, unlike its case-insensitive cousin, [ILIKE](https://docs.getdbt.com/sql-reference/ilike.md). #### How to use the SQL LIKE operator The LIKE operator has a simple syntax, with the ability to have it utilized in [WHERE clauses](https://docs.getdbt.com/sql-reference/where.md) or case statements: `where <field_name> like '<pattern>'` or `case when <field_name> like '<pattern>'` Some notes on this operator’s syntax and functionality: * The `<pattern>` can use two SQL wildcards (`%` and `_`); the underscore will match any *single character* and the % matches zero or more characters * Ex. '%J' = any string that ends with a capital J * Ex. 'J%' = any string that starts with a capital J * Ex. 'J%L' = any string that starts with a capital J and ends with a capital L * Ex. '\_J%' = any string that has a capital J in the second position * Majority of use cases for the LIKE operator will likely involve the `%` wildcard * The LIKE operator is case-sensitive, meaning that the casing in the `<pattern>` you want to filter for should match the same-case in your column values; for columns with varied casing, leverage the case-insensitive ILIKE operator * The LIKE operator can be paired with the NOT operator, to filter on rows that are not like a specified pattern Let’s dive into a practical example using the LIKE operator now. ##### SQL LIKE example ```sql select user_id, first_name from {{ ref('customers') }} where first_name like 'J%' order by 1 ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `customers` table would return all of the customers whose first name starts with the *uppercase* letter J: | **customer\_id** | **first\_name** | | ---------------- | --------------- | | 1 | Julia | | 4 | Jeremy | Because LIKE is case-sensitive, it would not return results in this query for customers with lowercase J-names. If you have a mix of uppercase and lowercase strings in your data, consider standardizing casing for strings using the [UPPER](https://docs.getdbt.com/sql-reference/upper.md) and [LOWER](https://docs.getdbt.com/sql-reference/lower.md) functions or use the more flexible [ILIKE operator](https://docs.getdbt.com/sql-reference/ilike.md). #### LIKE syntax in Snowflake, Databricks, BigQuery, and Redshift Most, if not all, modern data warehouses support the LIKE operator and the syntax is also the same across them. Some data warehouses, such as Snowflake and Databricks, additionally support similar or more flexible operators such as ILIKE, the case-insensitive version of LIKE, or LIKE ANY, which allows you to pass in multiple pattern options to scan for. Use the table below to read more on the documentation for the LIKE operator in your data warehouse. | **Data warehouse** | **LIKE support?** | | ------------------ | ----------------- | | Snowflake | ✅ | | Databricks | ✅ | | Amazon Redshift | ✅ | | Google BigQuery | ✅ | #### LIKE operator example use cases You may see the LIKE operator used in analytics engineering work to: * Bucket column values together based on general requirements using case statements and the LIKE operator (ex. `case when page_path like '/product%' then 'product_page' else 'non_product_page'`) * Filter out employee email records based on a similar email address pattern (ex. `where email_address not like '%@dbtlabs.com'`) This isn’t an extensive list of where your team may be using the LIKE operator throughout your dbt models, but contains some common scenarios analytics engineers face day-to-day. --- ### SQL LIMIT When you’re developing data models or drafting up a query, do you usually need to see all results from it? Not normally. Hence, we LIMIT. Adding the LIMIT clause to a query will limit the number of rows returned. It’s useful for when you’re developing data models, ensuring SQL in a query is functioning as expected, and wanting to save some money during development periods. #### How to use the LIMIT clause in a query To limit the number of rows returned from a query, you would pass the LIMIT in the last line of the query with the number of rows you want returned: ```sql select some_rows from my_data_source limit <integer> ``` Let’s take a look at a practical example using LIMIT below. ##### LIMIT example ```sql select order_id, order_date, rank () over (order by order_date) as order_rnk from {{ ref('orders') }} order by 2 limit 5 ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return these exact 5 rows: | order\_id | order\_date | order\_rnk | | --------- | ----------- | ---------- | | 1 | 2018-01-01 | 1 | | 2 | 2018-01-02 | 2 | | 3 | 2018-01-04 | 3 | | 4 | 2018-01-05 | 4 | | 5 | 2018-01-05 | 4 | After ensuring that this is the result you want from this query, you can omit the LIMIT in your final data model. Save money and time by limiting data in development You could limit your data used for development by manually adding a LIMIT statement, a WHERE clause to your query, or by using a [dbt macro to automatically limit data based](https://docs.getdbt.com/best-practices/best-practice-workflows.md#limit-the-data-processed-when-in-development) on your development environment to help reduce your warehouse usage during dev periods. #### LIMIT syntax in Snowflake, Databricks, BigQuery, and Redshift All modern data warehouses support the ability to LIMIT a query and the syntax is also the same across them. Use the table below to read more on the documentation for limiting query results in your data warehouse. | Data warehouse | LIMIT support? | | --------------- | -------------- | | Snowflake | ✅ | | Databricks | ✅ | | Amazon Redshift | ✅ | | Google BigQuery | ✅ | #### LIMIT use cases We most commonly see queries limited in data work to: * Save some money in development work, especially for large datasets; just make sure the model works across a subset of the data instead of all of the data 💸 * Paired with an ORDER BY statement, grab the top 5, 10, 50, 100, etc. entries from a dataset This isn’t an extensive list of where your team may be using LIMIT throughout your development work, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL LOWER We’ve all been there: * In a user signup form, user A typed in their name as `Kira Furuichi`, user B typed it in as `john blust`, and user C wrote `DAvid KrevitT` (what’s up with that, David??) * Your backend application engineers are adamant customer emails are in all caps * All of your event tracking names are lowercase In the real world of human imperfection, opinions, and error, string values are likely to take inconsistent capitalization across different data sources (or even within the same data source). There’s always a little lack of rhyme or reason for why some values are passed as upper or lowercase, and it’s not worth the headache to unpack that. So how do you create uniformity for string values that you collect across all your data sources? The LOWER function! Using the LOWER function on a string value will return the input as an all-lowercase string. It’s an effective way to create consistent capitalization for string values across your data. #### How to use the SQL LOWER function The syntax for using the LOWER function looks like the following: ```sql lower(<string_column>) ``` Executing this command in a SELECT statement will return the lowercase version of the input string. You may additionally use the LOWER function in WHERE clauses and on join values. Let’s take a look at a practical example using the LOWER function. ##### SQL LOWER function example You can lower the first name and last name of the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `customers` model using the following code: ```sql select customer_id, lower(first_name) as first_name, lower(last_name) as last_name from {{ ref('customers') }} ``` After running this query, the `customers` table will look a little something like this: | customer\_id | first\_name | last\_name | | ------------ | ----------- | ---------- | | 1 | michael | p. | | 2 | shawn | m. | | 3 | kathleen | p. | Now, all characters in the `first_name` and `last_name` columns are lowercase. > Changing all string columns to lowercase to create uniformity across data sources typically happens in our [dbt project’s staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md). There are a few reasons for that: data cleanup and standardization, such as aliasing, casting, and lowercasing, should ideally happen in staging models to create downstream uniformity and improve downstream performance. #### SQL LOWER function syntax in Snowflake, Databricks, BigQuery, and Redshift Google BigQuery, Amazon Redshift, Snowflake, Postgres, and Databricks all support the LOWER function. In addition, the syntax to use LOWER is the same across all of them. #### LOWER function use cases Let’s go back to our chaotic trio of users A, B, and C who all used different capitalizations to type in their names. If you don’t create consistent capitalization for string values, how would a business user know what to filter for in their BI tool? A business user could filter a name field on “John Blust” since that’s what they would expect it to look like, only to get zero results back. By creating a consistent capitalization format (upper or lowercase) for all string values in your data models, you therefore create some expectations for business users in your BI tool. There will most likely never be 100% consistency in your data models, but doing all that you can to mitigate that chaos will make your life and the life of your business users hopefully a little easier. Use the LOWER function to create a consistent casing for all strings in your data sources. --- ### SQL MAX The SQL MAX aggregate function allows you to compute the maximum value from a column. This kind of measure is useful for understanding the distribution of column values, determining the most recent timestamps of key events, and creating booleans from CASE WHEN statements to flatten semi-structured data. #### How to use the SQL MAX function in a query Use the following syntax to find the maximum value of a field: `max(<field_name>)` Since MAX is an aggregate function, you’ll need a GROUP BY statement in your query if you’re looking at counts broken out by dimension(s). If you’re calculating the standalone maximum of fields without the need to break them down by another field, you don’t need a GROUP BY statement. MAX can also be used as a window function to operate across specified or partitioned rows. Let’s take a look at a practical example using MAX and GROUP BY below. ##### MAX example The following example is querying from a sample dataset created by dbt Labs called [jaffle\_shop](https://github.com/dbt-labs/jaffle_shop): ```sql select date_part('month', order_date) as order_month, max(amount) as max_amaount from {{ ref('orders') }} group by 1 ``` This simple query is something you may do while doing initial exploration of your data; it will return the maximum order `amount` per order month that appear in the Jaffle Shop’s `orders` table: | order\_month | max\_amount | | ------------ | ----------- | | 1 | 58 | | 2 | 30 | | 3 | 56 | | 4 | 26 | #### SQL MAX function syntax in Snowflake, Databricks, BigQuery, and Redshift All modern data warehouses support the ability to use the MAX function (and follow the same syntax!). #### MAX function use cases We most commonly see queries using MAX to: * Perform initial data exploration on a dataset to understand the distribution of column values. * Identify the most recent timestamp for key events (ex. `max(login_timestamp_utc) as last_login`). * Create descriptive boolean values from case when statements (ex. `max(case when status = 'complete' then 1 else 0 end) as has_complete_order`). * Establish the most recent timestamp from a table to filter on rows appropriately for [incremental model builds](https://docs.getdbt.com/docs/build/incremental-models.md). This isn’t an extensive list of where your team may be using MAX throughout your development work, dbt models, and BI tool logic, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL MIN SQL MIN, MAX, SUM…the aggregate functions that you’ll live and die by as an analytics practitioner. In this post, we’re going to unpack the SQL MIN function, how to use it, and why it's valuable in data work. The MIN aggregate function allows you to compute the minimum value from a column or across a set of rows for a column. The results from the MIN function are useful for understanding the distribution of column values and determining the first timestamps of key events. #### How to use the MIN function in a query Use the following syntax in a query to find the minimum value of a field: `min(<field_name>)` Since MIN is an aggregate function, you’ll need a GROUP BY statement in your query if you’re looking at counts broken out by dimension(s). If you’re calculating the standalone minimum of fields without the need to break them down by another field, you don’t need a GROUP BY statement. MIN can also be used as a window function to operate across specified or partitioned rows. Let’s take a look at a practical example using MIN below. ##### MIN example The following example is querying from a sample dataset created by dbt Labs called [jaffle\_shop](https://github.com/dbt-labs/jaffle_shop): ```sql select customer_id, min(order_date) as first_order_date, max(order_date) as last_order_date from {{ ref('orders') }} group by 1 limit 3 ``` This simple query is returning the first and last order date for a customer in the Jaffle Shop’s `orders` table: | customer\_id | first\_order\_date | last\_order\_date | | ------------ | ------------------ | ----------------- | | 1 | 2018-01-01 | 2018-02-10 | | 3 | 2018-01-02 | 2018-03-11 | | 94 | 2018-01-04 | 2018-01-29 | #### SQL MIN function syntax in Snowflake, Databricks, BigQuery, and Redshift All modern data warehouses support the ability to use the MIN function (and follow the same syntax). #### MIN function use cases We most commonly see queries using MIN to: * Perform initial data exploration on a dataset to understand the distribution of column values. * Identify the first timestamp for key events (ex. `min(login_timestamp_utc) as first_login`). This isn’t an extensive list of where your team may be using MIN throughout your development work, dbt models, and BI tool logic, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL NOT This will be a not *not* useful page on a helpful SQL operator. Ok we had to get that out of the way. The SQL NOT operator allows you to return results from conditions that are not true. Pretty intuitive, right? In this page, we’ll dive into how to use the NOT operator, demonstrate an example, and elaborate on potential use cases. #### How to use the SQL NOT operator The NOT boolean is kind of similar to an adjective—it’s often put in front of another operator, such as [BETWEEN](https://docs.getdbt.com/sql-reference/between.md), [LIKE](https://docs.getdbt.com/sql-reference/like.md)/[ILIKE](https://docs.getdbt.com/sql-reference/ilike.md), IS, and [IN](https://docs.getdbt.com/sql-reference/in.md), to return rows that do not meet the specified criteria. Below is an example of how to use NOT in front of a LIKE operator: `where <field_name> not like <value>` This syntax can be easily modified for other operators: * `where not between <value_1> and <value_2>` * `where <field_name> is not null` * `where <field_name> is not in (array_of_options)` * …or placed altogether in a different place, such as a case statement (ex. `case when <field_name> is not null then 1 else 0 end`) Let’s dive into a practical example using the NOT operator. ##### SQL NOT example ```sql select payment_id, order_id, payment_method from {{ ref('payments') }} where payment_method not like '%card' ``` This simple query using the sample dataset [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `payments` table is returning all rows whose `payment_method` is not a card-type (ex. gift card or credit card): | **payment\_id** | **order\_id** | **payment\_method** | | --------------- | ------------- | ------------------- | | 3 | 3 | coupon | | 4 | 4 | coupon | | 5 | 5 | bank\_transfer | | 10 | 9 | bank\_transfer | #### SQL NOT syntax in Snowflake, Databricks, BigQuery, and Redshift [Snowflake](https://docs.snowflake.com/en/sql-reference/operators-logical.html), [Databricks](https://docs.databricks.com/sql/language-manual/functions/not.html), [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/operators), and [Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_logical_condition.html) all support the NOT operator, but may not all support secondary operators you would typically use the NOT operator in pair with. For example, `where <field_name> not ilike <pattern>` is valid in Snowflake, Databricks, and Redshift, but the ILIKE operator is not supported in BigQuery, so this example would not be valid across all data warehouses. #### NOT operator example use cases There are probably many scenarios where you’d want to use the NOT operators in your WHERE clauses or case statements, but we commonly see NOT operators used to remove nulls or boolean-identifed deleted rows in source data in [staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md). This removal of unnecessary rows can potentially help the performance of downstream [intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) and [mart models](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md). --- ### SQL OR We tried to come up with something witty about using the OR operator in a query, but couldn’t think of any 🤷 Use the OR operator in a WHERE clause to filter on multiple field values or perform more advanced joins on multiple fields. #### How to use the OR operator The OR operator is technically a boolean operator—meaning it returns results that execute to true. It’s straightforward to use, and you’ll typically see it appear in a WHERE clause to filter query results appropriately or joins that involve multiple possible fields. ##### OR operator example ```sql select order_id, customer_id, order_date, status, amount from {{ ref('orders') }} where status = 'shipped' or status = 'completed' limit 3 ``` This query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return results where the order status is shipped or completed: | order\_id | customer\_id | order\_date | status | amount | | --------- | ------------ | ----------- | --------- | ------- | | 2 | 3 | 2018-01-02 | completed | 20.0000 | | 3 | 94 | 2018-01-04 | completed | 1.00000 | | 4 | 50 | 2018-01-05 | completed | 25.0000 | #### OR operator syntax in Snowflake, Databricks, BigQuery, and Redshift Snowflake, Databricks, Google BigQuery, and Amazon Redshift all support the OR operator with the syntax looking the same in each platform. You may see the OR operator substituted for a more appropriate IN operator. #### OR use cases We most commonly see OR operators used in queries and dbt models to: * Return results for fields of varying values * Joining tables on multiple fields using an OR operator (fair warning: this can be a bit scary and inefficient, so use OR operators in joins very carefully and consider refactoring your work to avoid these scenarios) This isn’t an extensive list of where your team may be using OR throughout your data work, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL ORDER BY The ORDER BY clause allows you to specify the resulting row order for a query. In practice, you use the ORDER BY clause to indicate which field(s) you want to order by and in what type of order you want (ascending or descending). It’s useful to leverage during ad hoc analyses and for creating appropriate column values for partitioned rows in window functions. #### How to use the SQL ORDER BY clause ORDER BY clauses have multiple use cases in analytics work, but we see it most commonly utilized to: * Order a query or subquery result by a column or group of columns * Appropriately order a subset of rows in a window function To use the ORDER BY clause to a query or model, use the following syntax: ```sql select column_1, column_2 from source_table order by <field(s)> <asc/desc> --comes after FROM, WHERE, and GROUP BY statements ``` You can order a query result by multiple columns, represented by their column name or by their column number in the select statement (ex. `order by column_2 == order by 2`). You can additionally specify the ordering type you want (ascending or descending) to return the desired row order. Let’s take a look at a practical example using ORDER BY. ##### ORDER BY example ```sql select date_trunc('month, order_date') as order_month, round(avg(amount)) as avg_order_amount from {{ ref('orders') }} group by 1 order by 1 desc ``` This query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return the rounded order amount per each order month in descending order: | order\_month | avg\_order\_amount | | ------------ | ------------------ | | 2018-04-01 | 17 | | 2018-03-01 | 18 | | 2018-02-01 | 15 | | 2018-01-01 | 17 | #### SQL ORDER BY syntax in Snowflake, Databricks, BigQuery, and Redshift Since the ORDER BY clause is a SQL fundamental, data warehouses, including Snowflake, Databricks, Google BigQuery, and Amazon Redshift, all support the ability to add ORDER BY clauses in queries and window functions. #### ORDER BY use cases We most commonly see the ORDER BY clause used in data work to: * Analyze data for both initial exploration of raw data sources and ad hoc querying of [mart datasets](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md) * Identify the top 5/10/50/100 of a dataset when used in pair with a [LIMIT](https://docs.getdbt.com/sql-reference/limit.md) * (For Snowflake) Optimize the performance of large incremental models that use both a `cluster_by` [configuration](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#using-cluster_by) and ORDER BY statement * Control the ordering of window function partitions (ex. `row_number() over (partition by user_id order by updated_at)`) This isn’t an extensive list of where your team may be using the ORDER BY clause throughout your dbt models, ad hoc queries, and BI tool logic, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL OUTER JOIN SQL full outer joins exist and therefore we have to talk about them, but they’re *highly unlikely* to be a join you regularly leverage in your data work. In plain terms, a SQL full outer join is a join between two tables that returns *all rows* from both tables, regardless of join key match success; compare this to [left](https://docs.getdbt.com/sql-reference/left-join.md), [inner](https://docs.getdbt.com/sql-reference/outer-join.md), or [right joins](https://docs.getdbt.com/sql-reference/right-join.md) that require matches to be successful to return certain rows. In this page, we’ll unpack how to create a full outer join and demonstrate when you might need one in your analytics engineering work. #### How to create a full outer join Like all joins, you need some database objects (ie tables/views), keys to join on, and a [select statement](https://docs.getdbt.com/sql-reference/select.md) to perform a full outer join: ```text select <fields> from <table_1> as t1 full outer join <table_1> as t2 on t1.id = t2.id ``` In this example above, there’s only one field being used to join the table together; if you’re joining between database objects that require multiple fields, you can leverage AND/OR operators, and more preferably, surrogate keys. You may additionally add [WHERE](https://docs.getdbt.com/sql-reference/where.md), [GROUP BY](https://docs.getdbt.com/sql-reference/group-by.md), [ORDER BY](https://docs.getdbt.com/sql-reference/order-by.md), [HAVING](https://docs.getdbt.com/sql-reference/having.md), and other clauses after your joins to create filtering, ordering, and performing aggregations. A note on full outer joins: it may sound obvious, but because full outer joins can return all rows between two tables, they therefore can return *many* rows, which is not necessarily a recipe for efficiency. When you use full outer joins, you often can find alternatives using different joins or unions to potentially bypass major inefficiencies caused by a full outer join. ##### SQL full outer join example Table A `car_type` | user\_id | car\_type | | -------- | --------- | | 1 | van | | 2 | sedan | | 3 | truck | Table B `car_color` | user\_id | car\_color | | -------- | ---------- | | 1 | red | | 3 | green | | 4 | yellow | ```sql select car_type.user_id as user_id, car_type.car_type as type, car_color.car_color as color from {{ ref('car_type') }} as car_type full outer join {{ ref('car_color') }} as car_color on car_type.user_id = car_color.user_id order by 1 ``` This simple query will return all rows from tables A and B, regardless of `user_id` match success between the two tables: | user\_id | type | color | | -------- | ----- | ------ | | 1 | van | red | | 2 | sedan | null | | 3 | truck | green | | 4 | null | yellow | #### SQL full outer join use cases There will inevitably be valid use cases for full outer joins in your dbt project. However, because of the nature of dbt, which heavily encourages modularity and DRY dryness, the necessity for full outer joins may go down (slightly). Regardless, the two primary cases for full outer joins we typically see are around consolidating or merging multiple entities together and data validation. * Merging tables together: A full outer join between two tables can bring those entities together, regardless of join key match. This type of joining can often be bypassed by using different joins, unions, pivots, and a combination of these, but hey, sometimes the full outer join is a little less work 🤷 * Data validation: Full outer joins can be incredibly useful when performing data validation; for example, in the [dbt-audit-helper package](https://github.com/dbt-labs/dbt-audit-helper), a full outer join is used in the [compare\_column\_values test](https://github.com/dbt-labs/dbt-audit-helper/blob/main/macros/compare_column_values.sql) to help determine where column values are mismatched between two dbt models. --- ### SQL RANK There are many different ranking window functions…[ROW\_NUMBER](https://docs.getdbt.com/sql-reference/row-number.md), DENSE\_RANK, RANK. Let’s start off with the most basic (RANK) and talk about what it is, how to use it, and why it’s important in analytics engineering work. The RANK function is an effective way to create a ranked column or filter a query based on rankings. More specifically, the RANK function returns the rank of a value (starting at 1) in an ordered group or dataset. It's important to note that if multiple values executed by the rank function are the same, they’ll have the same rank. #### How to use the RANK function The RANK function has a pretty simple syntax, with an optional partition field and support for ordering customization: `rank() over ([partition by <field(s)>] order by field(s) [asc | desc])` Some notes on this function’s syntax: * The `partition by` field is optional; if you want to rank your entire dataset by certain fields (compared to partitioning *and ranking* within a dataset), you would simply omit the `partition by` from the function call (see the example below for this). * By default, the ordering of a ranking function is set to ascending. To explicitly make the ordering in a descending order, you’ll need to pass in `desc` to the `order by` part of the function. Let’s take a look at a practical example using the RANK function below. ##### RANK function example ```sql select order_id, order_date, rank() over (order by order_date) as order_rank from {{ ref('orders') }} ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return the rank of orders by their `order_date`: | order\_id | order\_date | order\_rank | | --------- | ----------- | ----------- | | 1 | 2018-01-01 | 1 | | 2 | 2018-01-02 | 2 | | 3 | 2018-01-04 | 3 | | 4 | 2018-01-05 | 4 | | 5 | 2018-01-05 | 4 | | 6 | 2018-01-07 | 6 | Some notes on these results: * Orders that have the same `order_date`(ex. Orders 4 and 5) have the same `order_rank` (4). * Order 6’s `order_rank` is 6 (if you wanted the rank to execute to 5, you would use the DENSE\_RANK function). Ranking functions to know RANK is just one of the ranking functions that analytics engineering practitioners will use throughout their data models. There’s also DENSE\_RANK and [ROW\_NUMBER](https://docs.getdbt.com/sql-reference/row-number.md) which rank rows differently than RANK. #### RANK syntax in Snowflake, Databricks, BigQuery, and Redshift Most, if not all, modern data warehouses support RANK and other similar ranking functions; the syntax is also the same across them. Use the table below to read more on the documentation for the RANK function in your data warehouse. | Data warehouse | RANK support? | | --------------- | ------------- | | Snowflake | ✅ | | Databricks | ✅ | | Amazon Redshift | ✅ | | Google BigQuery | ✅ | #### RANK function use cases We most commonly see the RANK function used in data work to: * In [SELECT statements](https://docs.getdbt.com/sql-reference/select.md) to add explicit ranking to rows * In QUALIFY statements to filter a query on a ranking without having to add the rank to the query result This isn’t an extensive list of where your team may be using the RANK function throughout your dbt models and BI tool logic, but contains some common scenarios analytics engineers face in a day-to-day. --- ### SQL Reference #### [Statements](https://docs.getdbt.com/sql-reference/select.md) [5 items](https://docs.getdbt.com/sql-reference/select.md) --- ### SQL RIGHT JOIN The not-as-favorite child: the right join. Unlike [left joins](https://docs.getdbt.com/sql-reference/left-join.md) that return all rows in the database object in [the FROM statement](https://docs.getdbt.com/sql-reference/from.md), regardless of match in the left join object, right joins return all rows *in the right join database object*, regardless of match in the database object in the FROM statement. What you really need to know: You can accomplish anything a right join does with a left join and left joins typically are more readable and intuitive. However, we’ll still walk you through how to use right joins and elaborate on why we think left joins are superior 😉 #### How to create a right join Like all joins, you need some database objects (ie tables/views), keys to join on, and a [select statement](https://docs.getdbt.com/sql-reference/select.md) to perform a right join: ```text select <fields> from <table_1> as t1 right join <table_2> as t2 on t1.id = t2.id ``` In this example above, there’s only one field from each table being used to join the two together; if you’re joining between two database objects that require multiple fields, you can leverage AND/OR operators, and more preferably, surrogate keys. You may additionally add [WHERE](https://docs.getdbt.com/sql-reference/where.md), [GROUP BY](https://docs.getdbt.com/sql-reference/group-by.md), [ORDER BY](https://docs.getdbt.com/sql-reference/order-by.md), [HAVING](https://docs.getdbt.com/sql-reference/having.md), and other clauses after your joins to create filtering, ordering, and performing aggregations. You may also right (or any join really) as many joins as you’d like in an individual query or CTE. ##### SQL right join example Table A `car_type` | **user\_id** | **car\_type** | | ------------ | ------------- | | 1 | van | | 2 | sedan | | 3 | truck | Table B `car_color` | **user\_id** | **car\_color** | | ------------ | -------------- | | 1 | red | | 3 | green | | 4 | yellow | ```sql select car_type.user_id as user_id, car_type.car_type as type, car_color.car_color as color from {{ ref('car_type') }} as car_type right join {{ ref('car_color') }} as car_color on car_type.user_id = car_color.user_id ``` This simple query will return *all* rows from Table B and adds the `color` column to rows where there’s a successful match to Table A: | **user\_id** | **type** | **color** | | ------------ | -------- | --------- | | 1 | van | red | | 3 | truck | green | | 4 | null | yellow | Because there’s no `user_id` = 4 in Table A, there is no `type` available, thus a null result `type` column for `user_id` 4. Since no `user_id` = 2 exists in Table B, and that id is not in the right join database object, no rows with a `user_id` of 2 will be returned. #### SQL right join use cases Compared to left joins, you likely won’t see right joins as often (or ever) in data modeling and analytics engineering work. But why not? Simply because right joins are a little less intuitive than a left join. When you’re data modeling, you’re usually focused on one database object, and adding the supplementary data or tables you need to give you a final dataset. That one focal database object is typically what is put in the `from {{ ref('my_database_object')}}`; any other columns that are joined onto it from other tables are usually supplementary, but keeping all the rows from the initial table of focus is usually the priority. Don’t get us wrong—right joins can get you there—it’s likely just a little less intuitive and can get complex with queries that involve multiple joins. --- ### SQL ROUND If you’re reading this, that probably means you’re a data person. And as a data person who’s likely modeling data for analytics use cases, you’re going to need to round data from time to time. For the unacquainted, "rounding" is making a number simpler so that it's easier to understand while keeping it close to its original value. In data, a common use case for rounding is to decrease the number of decimal places a numeric record has. To round numeric fields or values in SQL, you’re going to use the handy ROUND function. #### How to use the SQL ROUND function The syntax for using ROUND function looks like the following: ```sql round(<numeric column or data>, [optional] <number of decimal places>) ``` In this function, you’ll need to input the *numeric* field or data you want rounded and pass in an optional number to round your field by. For most data warehouses, the number of decimal places is defaulted to 0 or 1, meaning if you rounded 20.00 using `round(20.00)`, it would return 20 or 20.0 (depending on your data warehouse). ##### SQL ROUND function example You can round some of the numeric fields of the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` model using the following code: ```sql select cast(order_id as string) as order_id, order_date, amount, round(amount, 1) as rounded_amount from {{ ref('orders') }} ``` After running this query, the resulting `orders` table will look a little something like this: | order\_id | order\_date | amount | rounded\_amount | | --------- | ----------- | --------- | --------------- | | 1 | 2018-01-01 | 10.000000 | 10.0 | | 2 | 2018-01-02 | 20.000000 | 20.0 | | 3 | 2018-01-04 | 1.000000 | 1.0 | The new `rounded_amount` column is the `amount` fielded rounded to 1 decimal place. For most data warehouses, the returned data from the ROUND function should be the same as the input data. If you input a float type into the ROUND function, the returned rounded number should also be a float. #### SQL ROUND function syntax in Snowflake, Databricks, BigQuery, and Redshift Google BigQuery, Amazon Redshift, Snowflake, and Databricks all support the ability to round numeric columns and data. In addition, the syntax to round is the same across all of them using the ROUND function. #### ROUND function use cases If you find yourself rounding numeric data, either in data models or ad-hoc analyses, you’re probably rounding to improve the readability and usability of your data using downstream [intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) or [mart models](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md). Specifically, you’ll likely use the ROUND function to: * Make numeric calculations using division or averages a little cleaner and easier to understand * Create concrete buckets of data for a cleaner distribution of values during ad-hoc analysis You’ll additionally likely see the ROUND function used in your BI tool as it generates rounded clean numbers for business users to interact with. --- ### SQL ROW_NUMBER In this page, let’s go deep into the ROW\_NUMBER function and talk about what it is, how to use it, and why it’s important in analytics engineering work. The ROW\_NUMBER window function is an effective way to create a ranked column or filter a query based on rankings. More specifically, the ROW\_NUMBER function returns the *unique* row number of a row in an ordered group or dataset. Unlike the [RANK](https://docs.getdbt.com/sql-reference/rank.md) and DENSE\_RANK functions, ROW\_NUMBER is non-deterministic, meaning that a *unique* number is assigned arbitrarily for rows with duplicate values. #### How to use the ROW\_NUMBER function The ROW\_NUMBER function has a pretty simple syntax, with an optional partition field and support for ordering customization: `row_number() over ([partition by <field(s)>] order by field(s) [asc | desc])` Some notes on this function’s syntax: * The `partition by` field is optional; if you want to get the row numbers of your entire dataset (compared to grabbing row number within a group of rows in your dataset), you would simply omit the `partition by` from the function call (see the example below for this). * By default, the ordering of a ROW\_NUMBER function is set to ascending. To explicitly make the resulting order descending, you’ll need to pass in `desc` to the `order by` part of the function. Let’s take a look at a practical example using the ROW\_NUMBER function below. ##### ROW\_NUMBER function example ```sql select customer_id, order_id, order_date, row_number() over (partition by customer_id order by order_date) as row_n from {{ ref('orders') }} order by 1 ``` This simple query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return the unique row number per customer by their `order_date`: | customer\_id | order\_id | order\_date | row\_n | | ------------ | --------- | ----------- | ------ | | 1 | 1 | 2018-01-01 | 1 | | 1 | 37 | 2018-02-10 | 2 | | 2 | 8 | 2018-01-11 | 1 | | 3 | 2 | 2018-01-02 | 1 | | 3 | 24 | 2018-01-27 | 2 | | 3 | 69 | 2018-03-11 | 3 | Because ROW\_NUMBER is non-deterministic, orders per customer that have the same `order_date` would have unique `row_n` values (unlike if you used the RANK or DENSE\_RANK functions). #### ROW\_NUMBER syntax in Snowflake, Databricks, BigQuery, and Redshift Most, if not all, modern data warehouses support ROW\_NUMBER and other similar ranking functions; the syntax is also the same across them. Use the table below to read more on the documentation for the ROW\_NUMBER function in your data warehouse. | Data warehouse | ROW\_NUMBER support? | | --------------------------------------------------------------------------------------------------------------- | -------------------- | | [Snowflake](https://docs.snowflake.com/en/sql-reference/functions/row_number.html) | ✅ | | [Databricks](https://docs.databricks.com/sql/language-manual/functions/row_number.html) | ✅ | | [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_WF_ROW_NUMBER.html) | ✅ | | [Google BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#row_number) | ✅ | #### ROW\_NUMBER function use cases We most commonly see the ROW\_NUMBER function used in data work to: * In [SELECT statements](https://docs.getdbt.com/sql-reference/select.md) to add explicit and unique row numbers in a group of data or across an entire table * Paired with QUALIFY statement, filter CTEs, queries, or models to capture one unique row per specified partition with the ROW\_NUMBER function. This is particularly useful when you need to remove duplicate rows from a dataset (but use this wisely!). This isn’t an extensive list of where your team may be using the ROW\_NUMBER function throughout your dbts some common scenarios analytics engineers face day-to-day. --- ### SQL SELECT My goodness, would there even be modern data teams without SQL SELECT statements? Probably not. Luckily, we live in a world of tabular data, cloud data warehouses, and SQL prowess. Analysts and analytics engineers are writing queries, creating data models, and leveraging SQL to power their [data transformations](https://www.getdbt.com/analytics-engineering/transformation/) and analysis. But what makes these queries possible? SELECT statements. The SQL SELECT statement is the fundamental building block of any query: it allows you to select specific columns (data) from a database schema object (table/view). In a dbt project, a SQL dbt model is technically a singular SELECT statement (often built leveraging CTEs or subqueries). #### How to use SELECT Any query begins with a simple SELECT statement: ```sql select order_id, --your first column you want selected customer_id, --your second column you want selected order_date --your last column you want selected (and so on) from {{ ref('orders') }} --the table/view/model you want to select from limit 3 ``` This basic query is selecting three columns from the [jaffle shop’s](https://github.com/dbt-labs/jaffle_shop/blob/main/models/orders.sql) `orders` table and returning three rows. If you execute this query in your data warehouse, it will return a result looking like this: | order\_id | customer\_id | order\_date | | --------- | ------------ | ----------- | | 1 | 1 | 2018-01-01 | | 2 | 3 | 2018-01-02 | | 3 | 95 | 2018-01-04 | You may also commonly see queries that `select * from table_name`. The asterisk or star is telling the compiler to select all columns from a specified table or view. Goodbye carpal tunnel Leverage [dbt utils’ star macro](https://docs.getdbt.com/blog/star-sql-love-letter) to be able to both easily select many and specifically exclude certain columns. In a dbt project, analytics engineers will typically write models that contain multiple CTEs that build to one greater query. For folks that are newer to analytics engineering or dbt, we recommend they check out the [“How we structure our dbt projects” guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) to better understand why analytics folks like modular data modeling and CTEs. #### SELECT statement syntax in Snowflake, Databricks, BigQuery, and Redshift While we know the data warehouse players like to have their own slightly different flavors and syntax for SQL, they have conferred together that the SELECT statement is sacred and unchangeable. As a result, writing the actual `select…from` statement across Snowflake, Databricks, Google BigQuery, and Amazon Redshift would look the same. However, the actual SQL manipulation of data within the SELECT statement (ex. adding dates, casting columns) might look slightly different between each data warehouse. --- ### SQL SELF JOINS Simultaneously the easiest and most confusing of joins: the self join. Simply put, a self join allows you to join a dataset back onto itself. If you’re newer to data work or SQL, you may be asking yourself: why in the world would you want to do this? Shouldn’t joins happen between multiple *different* entities? The majority of joins you see in analytics work and dbt projects will probably be left and inner joins, but occasionally, depending on how the raw source table is built out, you’ll leverage a self join. One of the most common use cases to leverage a self join is when a table contains a foreign key to the primary key of that same table. It’s ok if none of that made sense—jump into this page to better understand how and where you might use a self join in your analytics engineering work. #### How to create a self join No funny venn diagrams here—there’s actually even no special syntax for self joins. To create a self join, you’ll use a regular join syntax, the only differences is the join objects are *the same*: ```text select <fields> from <table_1> as t1 [<join_type>] join <table_2> as t2 on t1.id = t2.id ``` Since you can choose the dialect of join for a self join, you can specify if you want to do a [left](https://docs.getdbt.com/sql-reference/left-join.md), [outer](https://docs.getdbt.com/sql-reference/outer-join.md), [inner](https://docs.getdbt.com/sql-reference/inner-join.md), [cross](https://docs.getdbt.com/sql-reference/cross-join.md), or [right join](https://docs.getdbt.com/sql-reference/right-join.md) in the join statement. ##### SQL self join example Given a `products` table that looks likes this, where there exists both a primary key (`sku_id`) and foreign key (`parent_id`) to that primary key: | **sku\_id** | **sku\_name** | **parent\_id** | | ----------- | ------------- | -------------- | | 1 | Lilieth Bed | 4 | | 2 | Holloway Desk | 3 | | 3 | Basic Desk | null | | 4 | Basic Bed | null | And this query utilizing a self join to join `parent_name` onto skus: ```sql select products.sku_id, products.sku_name, products.parent_id, parents.sku_name as parent_name from {{ ref('products') }} as products left join {{ ref('products') }} as parents on products.parent_id = parents.sku_id ``` This query utilizing a self join adds the `parent_name` of skus that have non-null `parent_ids`: | sku\_id | sku\_name | parent\_id | parent\_name | | ------- | ------------- | ---------- | ------------ | | 1 | Lilieth Bed | 4 | Basic Bed | | 2 | Holloway Desk | 3 | Basic Desk | | 3 | Basic Desk | null | null | | 4 | Basic Bed | null | null | #### SQL self join use cases Again, self joins are probably rare in your dbt project and will most often be utilized in tables that contain a hierarchical structure, such as consisting of a column which is a foreign key to the primary key of the same table. If you do have use cases for self joins, such as in the example above, you’ll typically want to perform that self join early upstream in your DAG, such as in a [staging](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) or [intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) model; if your raw, unjoined table is going to need to be accessed further downstream sans self join, that self join should happen in a modular intermediate model. You can also use self joins to create a cartesian product (aka a cross join) of a table against itself. Again, slim use cases, but still there for you if you need it 😉 --- ### SQL Strings We can almost guarantee that there is not a single dbt model or table in your database that doesn’t have at least one column of a string type. Strings are everywhere in data—they allow folks to have descriptive text field columns, use regex in their data work, and honestly, they just make the data world go ‘round. Below, we’ll unpack the different string formats you might see in a modern cloud data warehouse and common use cases for strings. #### Using SQL strings Strings are inherent in your data—they’re the name fields that someone inputs when they sign up for an account, they represent the item someone bought from your ecommerce store, they describe the customer address, and so on. To formalize it a bit, a string type is a word, or the combination of characters that you’ll typically see encased in single quotes (ex. 'Jaffle Shop', '1234 Shire Lane', 'Plan A'). Most often, when you’re working with strings in a dbt model or query, you’re: * Changing the casing (uppering/lowering) to create some standard for your string type columns in your data warehouse * Concatenating strings together to create more robust, uniform, or descriptive string values * Unnesting JSON or more complex structured data objects and converting those values to explicit strings * Casting a column of a different type to a string for better compatibility or usability in a BI tool * Filtering queries on certain string values * Creating a new string column type based off a CASE WHEN statement to bucket data by * Splitting a string into a substring This is not an exhaustive list of string functionality or use cases, but contains some common scenarios analytics engineers face day-to-day. ##### Strings in an example query ```sql select date_trunc('month', order_date)::string as order_month, round(avg(amount)) as avg_order_amount from {{ ref('orders') }} where status not in ('returned', 'return_pending') group by 1 ``` This query using the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` table will return the order month as a string and rounded order amount for only orders with statuses not equal to `returned` or `pending` string values: | order\_month | avg\_order\_amount | | ------------ | ------------------ | | 2018-01-01 | 18 | | 2018-02-01 | 15 | | 2018-03-01 | 18 | | 2018-04-01 | 17 | #### String support in Snowflake, Databricks, BigQuery, and Redshift Snowflake, Databricks, Google BigQuery, and Amazon Redshift all support the string [data type](https://docs.getdbt.com/sql-reference/data-types.md#string-data-types). They may have slightly varying sub-types for strings; some data warehouses such as Snowflake and Redshift support text, char, and character string types which typically differ in byte length in comparison to the generic string type. Again, since most string type columns are inherent in your data, you’ll likely be ok using generic varchar or strings for casting, but it never hurts to read up on the docs specific to your data warehouse string support! --- ### SQL SUM The SQL SUM function is handy and ever-present in data work. Let’s unpack what it is, how to use it, and why it's valuable. Jumping into it, the SUM aggregate function allows you to calculate the sum of a numeric column or across a set of rows for a column. Ultimately, the SUM function is incredibly useful for calculating meaningful business metrics, such as Lifetime Value (LTV), and creating key numeric fields in [`fct_` and `dim_` models](https://www.getdbt.com/blog/guide-to-dimensional-modeling). #### How to use the SUM function in a query Use the following syntax in a query to find the sum of a numeric field: `sum(<field_name>)` Since SUM is an aggregate function, you’ll need a GROUP BY statement in your query if you’re looking at counts broken out by dimension(s). If you’re calculating the standalone sum of a numeric field without the need to break them down by another field, you don’t need a GROUP BY statement. SUM can also be used as a window function to operate across specified or partitioned rows. You can additionally pass a DISTINCT statement into a SUM function to only sum distinct values in a column. Let’s take a look at a practical example using the SUM function below. ##### SUM example The following example is querying from a sample dataset created by dbt Labs called [jaffle\_shop](https://github.com/dbt-labs/jaffle_shop): ```sql select customer_id, sum(order_amount) as all_orders_amount from {{ ref('orders') }} group by 1 limit 3 ``` This simple query is returning the summed amount of all orders for a customer in the Jaffle Shop’s `orders` table: | customer\_id | all\_orders\_amount | | ------------ | ------------------- | | 1 | 33 | | 3 | 65 | | 94 | 24 | #### SQL SUM function syntax in Snowflake, Databricks, BigQuery, and Redshift All modern data warehouses support the ability to use the SUM function (and follow the same syntax). #### SUM function use cases We most commonly see queries using SUM to: * Calculate the cumulative sum of a metric across a customer/user id using a CASE WHEN statement (ex. `sum(case when order_array is not null then 1 else 0 end) as count_orders`) * Create [dbt metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) for key business values, such as LTV * Calculate the total of a field across a dimension (ex. total session time, total time spent per ticket) that you typically use in `fct_` or `dim_` models * Summing clicks, spend, impressions, and other key ad reporting metrics in tables from ad platforms This isn’t an extensive list of where your team may be using SUM throughout your development work, dbt models, and BI tool logic, but it contains some common scenarios analytics engineers face day-to-day. --- ### SQL TRIM We’ve been there: pesky blank spaces, weird, inconsistent formats, or unaccountable asterisks hiding at the end of your column value—[strings](https://docs.getdbt.com/sql-reference/strings.md) are one of the most variable data types in your datasets. They likely lack a uniform casing, vary in length, and will inevitably have characters you need to trim from them. Introducing: the SQL TRIM function, which removes the leading and trailing characters of a string. By default, it removes the blank space character from the beginning and end of a string. #### How to use the SQL TRIM function The syntax for using TRIM function looks like the following: ```sql trim(<field_name> [, <characters_to_remove>]) ``` Like we said earlier, the default `<characters_to_remove>` is a blank space, such that if you were to `trim(' string with extra leading space')` it would return `'string with extra leading space'`. You can explicitly specify single characters or a pattern to trim from your strings. ##### SQL TRIM function example ```sql select first_name, concat('*', first_name, '**') as test_string, trim(test_string, '*') as back_to_first_name from {{ ref('customers') }} limit 3 ``` After running this query, the resulting `orders` table will look like this: | first\_name | test\_string | back\_to\_first\_name | | ----------- | ------------ | --------------------- | | Julia | *Julia*\* | Julia | | Max | *Max*\* | Max | | Laura | *Laura*\* | Laura | In this query, you’re adding superfluous asterisks to a string using the [CONCAT function](https://docs.getdbt.com/sql-reference/concat.md) and recleaning it using the TRIM function. Even though I specified one asterisk in the TRIM function itself, it recognizes that as a pattern to remove from the beginning and end of a string, which is why the double asterisks (\*\*) were removed from the end of the `test_string` column. #### SQL TRIM function syntax in Snowflake, Databricks, BigQuery, and Redshift [Google BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#trim), [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_TRIM.html), [Snowflake](https://docs.snowflake.com/en/sql-reference/functions/trim.html), and [Databricks](https://docs.databricks.com/sql/language-manual/functions/trim.html) all support the ability to use the TRIM function. In addition, the syntax to trim strings is the same across all of them using the TRIM function. These data warehouses also support the RTRIM and LTRIM functions, which allow you to only trim characters from the right side and left side of a string, respectively. #### TRIM function use cases If string values in your raw data have extra white spaces or miscellaneous characters, you’ll leverage the TRIM (and subset RTRIM AND LTRIM) functions to help you quickly remove them. You’ll likely do this cleanup in [staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md), where you’re probably standardizing casing and doing other minor formatting changes to string values, so you can use a clean and consistent format across your downstream models. --- ### SQL UPPER UPPER is the counterpart to [LOWER](https://docs.getdbt.com/sql-reference/lower.md) (who would have guessed?)—and they’re probably the most intuitive of SQL functions. Using the UPPER function on a string value will return the input as an all-uppercase string. It’s an effective way to create expected capitalization for certain string values across your data. #### How to use the SQL UPPER function The syntax for using the UPPER function looks like the following: ```sql upper(<string_column>) ``` Executing this command in a SELECT statement will return the uppercase version of the input string value. You may additionally use the UPPER function in WHERE clauses and on join values. Below, we’ll walk through a practical example using the UPPER function. ##### SQL UPPER function example You can uppercase the first name of the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `customers` model using the following code: ```sql select customer_id, upper(first_name) as first_name, last_name from {{ ref('customers') }} ``` After running this query, the `customers` table will look a little something like this: | customer\_id | first\_name | last\_name | | ------------ | ----------- | ---------- | | 1 | MICHAEL | P. | | 2 | SHAWN | M. | | 3 | KATHLEEN | P. | Now, all characters in the `first_name` are uppercase (and `last_name` are unchanged). > Changing string columns to uppercase to create uniformity across data sources typically happens in our [dbt models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md). There are a few reasons for that: data cleanup and standardization, such as aliasing, casting, and lower or upper casing, should ideally happen in staging models to create downstream uniformity and improve downstream performance. #### SQL UPPER function syntax in Snowflake, Databricks, BigQuery, and Redshift Google BigQuery, Amazon Redshift, Snowflake, Postgres, and Databricks all support the UPPER function. In addition, the syntax to use the UPPER function is the same across all of them. #### UPPER function use cases By creating a consistent capitalization format (upper or lowercase) for all string values in your data models, you therefore create some expectations for business users in your BI tool. * Uppercase country codes in data sources to meet user expectations * Create a consistent capitalization format for string values in your data models, also creating expectations for business users in your BI tool There will most likely never be 100% consistency in your data models, but doing all that you can to mitigate that chaos will make your life and the life of your business users hopefully a little easier. Use the UPPER function to create a consistent casing for all strings in your data sources. --- ### SQL WHERE If the humble [SELECT statement](https://docs.getdbt.com/sql-reference/select.md) is an analytics engineer kitchen knife, the WHERE clause is the corresponding knife sharpener: no (good) cooking (or data modeling) is happening without it. The WHERE clause is a fundamental SQL statement—it allows you to appropriately filter your data models and queries, so you can look at specific subsets of data based on your requirements. #### How to use the SQL WHERE clause The syntax for using WHERE clause in a SELECT statement looks like the following: ```sql select order_id, customer_id, amount from {{ ref('orders') }} where status != 'returned' ``` In this query, you’re filtering for any order from the [Jaffle Shop’s](https://github.com/dbt-labs/jaffle_shop) `orders` model whose status is not `returned` by adding a WHERE clause after the FROM statement. You could additionally filter on string, numeric, date, or other data types to meet your query conditions. You will likely see WHERE clauses show up 99.99% of the time in a typical query or dbt model. The other .01% is probably in a DML statement, such as DELETE or ALTER, to modify specific rows in tables. #### SQL WHERE clause syntax in Snowflake, Databricks, BigQuery, and Redshift Since the WHERE clause is a SQL fundamental, Google BigQuery, Amazon Redshift, Snowflake, and Databricks all support the ability to filter queries and data models using it. In addition, the syntax to round is the same across all of them using the WHERE clause. #### SQL WHERE clause use cases WHERE clauses are probably some of the most widely used SQL capabilities, right after SELECT and FROM statements. Below is a non-exhaustive list of where you’ll commonly see WHERE clauses throughout dbt projects and data work: * Removing source-deleted rows from staging models to increase accuracy and improve downstream model performance * Filtering out employee records from models * Performing ad-hoc analysis on specific rows or users, either in a dbt model, BI tool, or ad-hoc query * Paired with IN, LIKE, NOT IN clauses to create more generalized or a group of specific requirements to filter on --- ## Tags ### 13 docs tagged with "dbt v2" #### [BigQuery configurations](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md) Reference guide for Big Query configurations in dbt. --- ### 19 docs tagged with "Metrics" #### [About MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) Learn more about MetricFlow and its key concepts --- ### 2 docs tagged with "Adapter creation" #### [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) Create an adapter that connects dbt to you platform, and learn how to maintain and version that adapter. --- ### 2 docs tagged with "Analyst" #### [Analyze your data in dbt](https://docs.getdbt.com/guides/analyze-your-data.md) Introduction --- ### 2 docs tagged with "API" #### [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) Integrate and use the JDBC API to query your metrics. --- ### 2 docs tagged with "Beginner" #### [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) Clone the Jaffle Shop sample dbt project from GitHub or GitLab. --- ### 2 docs tagged with "Best practices" #### [How to use prompts for dbt Wizard](https://docs.getdbt.com/guides/prompt-cookbook.md) A cookbook of prompts and real-world examples to use dbt Wizard efficiently. --- ### 2 docs tagged with "billing" #### [dbt Wizard billing and AI access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) Answers to common questions about AI being enabled by default, dbt Wizard usage, consumption pools, trials, spend limits, and BYOK billing. --- ### 2 docs tagged with "CI" #### [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md) Learn the benefits of version-controlled analytics code and custom pipelines in dbt for enhanced code testing and workflow automation during the development process. --- ### 2 docs tagged with "CLI" #### [Getting started with the terminal](https://docs.getdbt.com/guides/terminal-guide.md) A short intro to the terminal for dbt users — learn how to open it, navigate folders, and run your first dbt commands. --- ### 2 docs tagged with "dbt Insights" #### [Access the dbt Insights interface](https://docs.getdbt.com/docs/explore/access-dbt-insights.md) Learn how to access the dbt Insights interface and run queries --- ### 2 docs tagged with "dbt MCP" #### [Connect dbt MCP to dbt platform](https://docs.getdbt.com/guides/qs-mcp-oauth.md) --- ### 2 docs tagged with "dbt" #### [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) Clone the Jaffle Shop sample dbt project from GitHub or GitLab. --- ### 2 docs tagged with "DuckDB" #### [DuckDB configurations](https://docs.getdbt.com/reference/resource-configs/duckdb-configs.md) Reference for DuckDB-specific configurations in dbt-duckdb, including secrets, DuckLake, external files, incremental strategies, and more. --- ### 2 docs tagged with "Git" #### [Branches when changing Git providers or repositories](https://docs.getdbt.com/faqs/Git/branch-migration.md) How Git branch names and commit history carry over when you change providers, repositories, or remotes—for dbt v1 and the dbt platform, including platform-only notes for Studio IDE and repository settings. --- ### 2 docs tagged with "Redshift" #### [Quickstart for dbt and Redshift](https://docs.getdbt.com/guides/redshift.md) Introduction --- ### 2 docs tagged with "Reference" #### [Wizard CLI command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) Full reference for all wizard subcommands and global flags. --- ### 2 docs tagged with "static_analysis" #### [Supported BigQuery functions in dbt v2](https://docs.getdbt.com/reference/resource-configs/bigquery-function-support.md) Check which BigQuery built-in SQL functions dbt v2 can typecheck during static analysis. --- ### 2 docs tagged with "Troubleshooting" #### [Debug errors](https://docs.getdbt.com/guides/debug-errors.md) Learn about errors and the art of debugging them. --- ### 20 docs tagged with "dbt State" #### [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) Learn about dbt State, its benefits, and key concepts for running only what has changed in your dbt project. --- ### 20 docs tagged with "Quickstart" #### [Analyze your data in dbt](https://docs.getdbt.com/guides/analyze-your-data.md) Introduction --- ### 27 docs tagged with "dbt platform" #### [Airflow and dbt](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) Introduction --- ### 28 docs tagged with "Wizard" #### [About dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) dbt Wizard in dbt platform helps teams investigate, change, validate, and ship trusted dbt work with warehouse-aware AI. --- ### 3 docs tagged with "APIs" #### [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) Integrate and use the GraphQL API to query your metrics. --- ### 3 docs tagged with "cost insights" #### [Cost Insights in the dbt platform](https://docs.getdbt.com/docs/explore/cost-insights.md) Track warehouse compute costs and understand the impact of optimizations across your dbt projects and models. --- ### 3 docs tagged with "cost savings" #### [Cost Insights in the dbt platform](https://docs.getdbt.com/docs/explore/cost-insights.md) Track warehouse compute costs and understand the impact of optimizations across your dbt projects and models. --- ### 3 docs tagged with "IDE" #### [About the Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) Develop, test, run, and build in the Studio IDE. You can compile dbt code into SQL and run it against your database directly --- ### 3 docs tagged with "models built" #### [Cost Insights in the dbt platform](https://docs.getdbt.com/docs/explore/cost-insights.md) Track warehouse compute costs and understand the impact of optimizations across your dbt projects and models. --- ### 3 docs tagged with "SAO" #### [Cost Insights in the dbt platform](https://docs.getdbt.com/docs/explore/cost-insights.md) Track warehouse compute costs and understand the impact of optimizations across your dbt projects and models. --- ### 3 docs tagged with "Upgrade" #### [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md) A quickstart for moving environments and jobs off legacy dbt versions to a dbt v1 or v2 release track. --- ### 4 docs tagged with "BigQuery" #### [BigQuery configurations](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md) Reference guide for Big Query configurations in dbt. --- ### 4 docs tagged with "Orchestration" #### [Airflow and dbt](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) Introduction --- ### 40 docs tagged with "AI" #### [About AI and dbt](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md) Learn about AI-powered features and agents in dbt that help you build, manage, and consume governed data at scale. --- ### 42 docs tagged with "Semantic Layer" #### [About dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) Learn how to query data and perform exploratory data analysis using dbt Insights --- ### 5 docs tagged with "Agents" #### [dbt Copilot in Insights](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md) Chat with your data using the Analyst agent powered by the dbt Semantic Layer --- ### 5 docs tagged with "dbt Wizard" #### [Get started with Wizard Desktop](https://docs.getdbt.com/docs/dbt-ai/wizard-desktop.md) Download Wizard Desktop and start your first session with rich data visualizations, data diffs, and lineage. --- ### 5 docs tagged with "Platform" #### [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md) Use the dbt Wizard home tab in dbt platform for an agent-native development workflow — iterate in natural language, review inline diffs and DAG previews, and validate changes without leaving the agent. --- ### 5 docs tagged with "Snowflake" #### [Get hands-on with Snowflake Horizon and Apache Iceberg](https://docs.getdbt.com/guides/iceberg.md) Use Snowflake Horizon's open Iceberg REST catalog to access and transform one shared set of governed tables from Snowflake and DuckDB. --- ### 6 docs tagged with "Databricks" #### [Databricks configurations](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md) Configure Databricks-specific settings for models in dbt, including file formats, table properties, and materializations. --- ### 6 docs tagged with "dbt v1" #### [BigQuery configurations](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md) Reference guide for Big Query configurations in dbt. --- ### 6 docs tagged with "Migration" #### [Migrate from dbt-spark to dbt-databricks](https://docs.getdbt.com/guides/migrate-from-spark-to-databricks.md) Learn how to migrate from dbt-spark to dbt-databricks. --- ### 6 docs tagged with "Webhooks" #### [Create Datadog events from dbt results](https://docs.getdbt.com/guides/serverless-datadog.md) Configure a serverless app to add dbt events to Datadog logs. --- ### 7 docs tagged with "scheduler" #### [Deploy dbt](https://docs.getdbt.com/docs/deploy/deployments.md) Use the 's capabilities to seamlessly run a dbt job in production or staging environments. Rather than run dbt commands manually from the command line, you can leverage the 's in-app scheduling to automate how and when you execute dbt. --- ### One doc tagged with "Amazon" #### [Quickstart for dbt and Amazon Athena](https://docs.getdbt.com/guides/athena.md) Introduction --- ### One doc tagged with "Athena" #### [Quickstart for dbt and Amazon Athena](https://docs.getdbt.com/guides/athena.md) Introduction --- ### One doc tagged with "BigFrames" #### [Using BigQuery DataFrames with dbt Python models](https://docs.getdbt.com/guides/dbt-python-bigframes.md) Use this guide to help you set up dbt with BigQuery DataFrames (BigFrames). --- ### One doc tagged with "Canvas" #### [Quickstart for dbt Canvas](https://docs.getdbt.com/guides/canvas.md) Introduction --- ### One doc tagged with "CLL" #### [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) Use column-level lineage to gain insights about your data at a granular level, in dbt Catalog or locally with dbt v2. --- ### One doc tagged with "Configuration" #### [Configure BYOK for dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) Bring your own API key to use dbt Wizard CLI. Supports OpenAI, Anthropic, AWS Bedrock, Google Gemini, Snowflake Cortex (preview), and Databricks. --- ### One doc tagged with "Copilot" #### [dbt Copilot](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) dbt Copilot is dbt's inline AI assistance experience, providing single-click generation of SQL, documentation, tests, and semantic models. --- ### One doc tagged with "cost optimization" #### [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) Learn how to set up Cost Insights to track warehouse compute costs and view realized savings from state-aware orchestration across your dbt projects and models. --- ### One doc tagged with "cost optimizations" #### [Cost Insights in the dbt platform](https://docs.getdbt.com/docs/explore/cost-insights.md) Track warehouse compute costs and understand the impact of optimizations across your dbt projects and models. --- ### One doc tagged with "cost reduction" #### [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) Learn how to set up Cost Insights to track warehouse compute costs and view realized savings from state-aware orchestration across your dbt projects and models. --- ### One doc tagged with "cost reductions" #### [Cost Insights in the dbt platform](https://docs.getdbt.com/docs/explore/cost-insights.md) Track warehouse compute costs and understand the impact of optimizations across your dbt projects and models. --- ### One doc tagged with "dbt Charts" #### [Quickstart for dbt Charts](https://docs.getdbt.com/guides/dbt-charts.md) Build a dbt project locally, write your first dbt Charts board by hand, and preview it in the browser. --- ### One doc tagged with "Dremio" #### [Build a data lakehouse with dbt v1 and Dremio Cloud](https://docs.getdbt.com/guides/build-dremio-lakehouse.md) Learn how to build a data lakehouse with dbt v1 and Dremio Cloud. --- ### One doc tagged with "GCP" #### [Using BigQuery DataFrames with dbt Python models](https://docs.getdbt.com/guides/dbt-python-bigframes.md) Use this guide to help you set up dbt with BigQuery DataFrames (BigFrames). --- ### One doc tagged with "Google" #### [Using BigQuery DataFrames with dbt Python models](https://docs.getdbt.com/guides/dbt-python-bigframes.md) Use this guide to help you set up dbt with BigQuery DataFrames (BigFrames). --- ### One doc tagged with "Governance" #### [Build your metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) Learn about MetricFlow and build your metrics with semantic models --- ### One doc tagged with "Iceberg" #### [Get hands-on with Snowflake Horizon and Apache Iceberg](https://docs.getdbt.com/guides/iceberg.md) Use Snowflake Horizon's open Iceberg REST catalog to access and transform one shared set of governed tables from Snowflake and DuckDB. --- ### One doc tagged with "Intelligence" #### [About AI and dbt](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md) Learn about AI-powered features and agents in dbt that help you build, manage, and consume governed data at scale. --- ### One doc tagged with "Jan-1-2020" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "Jan-1-2021" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "Jinja" #### [Use Jinja to improve your SQL code](https://docs.getdbt.com/guides/using-jinja.md) Learn how to improve your SQL code using Jinja. --- ### One doc tagged with "Local development" #### [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) Introduction --- ### One doc tagged with "MCP" #### [Use MCP servers with dbt Wizard in dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-mcp.md) Understand MCP server support for dbt Wizard in dbt platform. --- ### One doc tagged with "Model" #### [Quickstart for dbt Canvas](https://docs.getdbt.com/guides/canvas.md) Introduction --- ### One doc tagged with "Privacy" #### [dbt Wizard CLI data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md) Understand what dbt Wizard CLI telemetry collects, why it is collected, and how to opt out. --- ### One doc tagged with "SQL" #### [Refactoring legacy SQL to dbt](https://docs.getdbt.com/guides/refactoring-legacy-sql.md) This guide walks through refactoring a long SQL query (perhaps from a stored procedure) into modular dbt data models. --- ### One doc tagged with "Studio" #### [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) Use dbt Wizard in the Studio IDE to write or refactor dbt models from natural language, validate with your dbt engine, and run against your warehouse with full project context. --- ### One doc tagged with "Tenancy" #### [Migrate between dbt platform tenancy types](https://docs.getdbt.com/guides/tenancy-migration.md) A self-serve guide for migrating between dbt platform tenancy types, such as moving from multi-tenant to single-tenant. --- ### One doc tagged with "Teradata" #### [Quickstart for dbt and Teradata](https://docs.getdbt.com/guides/teradata.md) Introduction --- ### One doc tagged with "v0.5.0" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.01" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.02" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.03" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.04" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.05" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.06" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.07" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.08" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.09" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.10" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.11" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.12" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.13" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.14" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.15" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.16" #### [Changelog 2019 and 2020](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2019-2020.md) 2019 and 2020 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.18" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.19" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.20" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.21" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.22" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.23" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.24" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.25" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.26" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.27" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.28" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.29" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.30" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.31" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.32" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.33" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.34" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.35" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.36" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.37" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.38" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.39" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.40" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### One doc tagged with "v1.1.41" #### [Changelog 2021](https://docs.getdbt.com/docs/dbt-versions/release-notes/dbt-cloud-changelog-2021.md) 2021 Changelog for the dbt Cloud application --- ### Tags #### A * [Adapter creation2](https://docs.getdbt.com/tags/adapter-creation.md) * [Agents5](https://docs.getdbt.com/tags/agents.md) * [AI40](https://docs.getdbt.com/tags/ai.md) * [Amazon1](https://docs.getdbt.com/tags/amazon.md) * [Analyst2](https://docs.getdbt.com/tags/analyst.md) * [API2](https://docs.getdbt.com/tags/api.md) * [APIs3](https://docs.getdbt.com/tags/ap-is.md) * [Athena1](https://docs.getdbt.com/tags/athena.md) *** --- ## Terms ### hover-terms ---